Issues in weighting for longitudinal surveys
Jul 25, 13:45
The production of survey weights can be particularly challenging and complex in the case of longitudinal surveys. The challenges stem primarily from three sources: a) the dynamic nature of the study population; b) sample attrition; and c) multiple combinations of waves and survey instruments that may be of interest for a particular analysis purpose. In this chapter, we will describe the range of issues involved in developing a weighting strategy for a longitudinal survey and in producing the weights once the strategy is decided. We will review recent research into relevant aspects of survey weighting and will identify issues on which research is lacking. We will also indicate whether an issue is particularly, or only, relevant for a certain type of longitudinal survey.
The issues to be discussed will include, but not necessarily be limited to:
- Weights for different analysis purposes: Weights can, and perhaps should, be tailored to the analysis objectives. But longitudinal survey data is often used for a very broad spectrum of analysis, often carried out by a large and heterogeneous group of analysts, and much of which cannot be specified at the outset of the survey. This presents data producers with a challenge to provide sets of weights that satisfy most users most of the time;
- Non-response adjustments: Many approaches to modelling, and adjusting for, attrition, are possible. For example, one can model each step in the attrition process (wave, instrument) sequentially and apply a series of adjustments, or one can model each cumulative attrition pattern independently. Hybrid approaches are also possible. But each approach has advantages and disadvantages in terms of variance inflation, bias reduction (use of covariates) and resource requirements;
- Representing a changing population: Standard approaches to post-stratification or calibration require modification when the study population is dynamic. If there are no sample entrants after wave 1, one approach is to post-stratify the wave 1 responding sample and thereafter rely on non-response adjustment. An alternative approach is to post-stratify at each wave to the latest population estimates, but this assumes the sample is designed to remain cross-sectionally representative. A third alternative is to post-stratify the responding and out of scope sample at each wave to the wave 1 population. The survey sample design may include features designed to maintain cross-sectional representativeness (following rules, regular samples of new population entrants). If so, the weighting must appropriately reflect these features. There are bias-variance trade-offs to be made when choosing between the various approaches;
- Uncertainty regarding continuing eligibility: Due to population dynamics, sample units that are initially eligible may not remain so for the duration of the survey (e.g. due to death or emigration). It can be difficult to always identify a transition to ineligibility, particularly when the unit is no longer participating in the survey. Inadvertently retaining ineligible units in the denominator of attrition models can lead to systematic errors. Methods for minimising these errors through at either sample level or unit level depend on the nature of sample information and auxiliary information available.