We experimentally and theoretically investigate the photoluminescence broadening of different excitonic complexes in single self-assembled GaAs/AlGaAs quantum dots. We demonstrate that the excitonic fine-structure splitting leads to a sizable line broadening whenever the detection is not resolved in polarization. The residual broadening in polarized measurements is systematically larger for the exciton with respect to both the trion and the biexciton recombination. The experimental data agree with calculations of the quantum confined Stark effect induced by charge defects in the quantum dot (QD) environment, denoting the role of the QD spectator carrier rearrangement in reducing the perturbation of the fluctuating environment. (C) 2008 American Institute of Physics.