Identifying interviewer effects in panel surveys
Jul 27, 09:00
Early stages of survey research already revealed that interviewers can affect respondent behavior and responses during personal interviews. These interviewer effects have been shown to be present across modes of data collection and surveyed instruments. Interviewer effects reduce overall survey data quality by introducing measurement error and bias into estimates and thereby generally threatening the reliability and validity of survey measures.
Past studies on the topic almost exclusively relied on cross-sectional data and little is known about potential peculiarities of interviewer effects in longitudinal studies. Usually, scholars estimate the intra-interviewer correlation (e.g., Kish’s ρint), reflecting the interviewers’ contribution to the overall variation of a given survey measure. If available, information about the interviewers (mainly socio-demographics) has been used to explain the observed effects and to quantify the bias that is introduced by the interviewer. However, these approaches often face challenges of separating interviewer effects from other confounding factors such as area effects.
In this paper, we suggest an alternative research design to identifying interviewer effects by drawing on panel data. This design considers changes in respondents’ answers in accordance to changes in interviewers’ as an indicator of interviewer effects. Our analysis is based on data from the Socio-Economic Panel Study (SOEP), a longitudinal household survey largely relying on face-to-face interviewing. In the SOEP, longitudinal survey data is available not only for respondents, but for interviewers as well: In the years 2006, 2012 and 2016, nearly all SOEP interviewers took part in an interviewer survey. The questionnaires covered, amongst others, the interviewers’ socio-demographics, personality traits, health, as well as political attitudes. Since questionnaires for SOEP respondents as well as interviewers largely overlap, this provides the opportunity for a 1:1 link of respondents’ and interviewers’ answers on the same items; a data format which offers entirely new research perspectives. Mixed models allow us to analyze within-variation at a respondent and interviewer level for a variety of survey questions and topics.
The paper will review alternative strategies of identifying interviewer effects as well as testing these strategies empirically. More generally, our study will contribute to a better understanding of the underlying mechanisms causing interviewer effects in personal interviews. In addition, we will present a review of the existing literature on interviewer effects in panel surveys. Finally, results might be used to improve fieldwork monitoring and interviewer evaluation processes.