Does the laptop always help? Non-compliance and interviewer effects in cognitive tests
Jul 25, 11:30
Individual performance in a cognitive test does not only reflect inherent ability but also contextual circumstances and differences in interviewers’ styles. One way to scale down the impact of interviewers is to shift administration of a cognitive test to a technical device. However, this approach can introduce mode effects, as some interviewers may deviate from the study protocol and administer the test without the help of the technical device for some but not all respondents.
In this work, we study test scores of the word recall test, collected in Understanding Society. Under the default procedure, respondents hear ten words from the laptop of the interviewer. Nevertheless, for about 20% of respondents in our sample the interviewer instead reads the words herself, leading to the cognitive test being effectively administered in two distinct modes.
We aim to answer three questions. Firstly, we ask which determinants drive imperfect compliance with the study protocol. Next, we turn to the question of what drives the difference in performance between the two modes. Finally, we seek to answer whether administration via the computer successfully reduces interviewer effects.
We find that hearing the words from the interviewer is predicted by respondent characteristics, which have been associated with the propensity to use technologies. Moreover, the share of respondents who hear the words from the interviewer varies greatly across interviewers. Selection effects appear to be the main driver of performance differences in our setting. Those respondents who are read the words to by the interviewer, perform significantly worse not only in the word recall test but also in other cognitive tests. Finally, our results suggest that the use of the laptop does indeed reduce interviewer effects. Therefore, the use of laptops in administration of the word recall test may be preferable, despite the problem of non-compliance.