Measuring cognition in a multi-mode context
Jul 25, 13:45
As longitudinal surveys increasingly integrate alternate modes (telephone, web, mail) to reduce costs, maintaining comparability of measures across modes presents operational and substantive challenges. The measurement of cognitive ability, common in longitudinal surveys of the general population, as well as specific studies of aging, is particularly challenging for a multi-mode design. Some of the standard tests that are designed for self-administration (e.g., tests that use visual stimuli) may be difficult to adapt for telephone administration and those used in interviewer-administered surveys (e.g., immediate and delayed word recall tests) can be difficult to adapt for web. Mode differences may be exacerbated for older respondents who are less technologically sophisticated or have cognitive impairments that could affect the quality and completeness of the data. Despite these challenges and the movement of many longitudinal studies to alternate modes, few multi-mode comparisons of cognition exist in the current literature.
To address this gap, we will first provide a review of the literature and current practice relating to the measurement of cognition in longitudinal surveys. We’ll then present empirical results from mode comparisons of cognitive performance using data from the Health and Retirement Study (HRS). The biennial core HRS interview has been dual mode (telephone and face-to-face) for most of its history, and plans to offer web as a third mode in the 2018 wave. HRS has also conducted periodic web surveys between core waves with some overlap of content. We investigate mode differences for several cognition measures (word recall/recognition, verbal analogies, number series, Serial 7s, and self-rated memory) that were included in the 2013 HRS Internet Survey and either/both of the adjacent core waves. We compare levels of cognitive performance and associations between tests, known predictors, and outcomes across modes (telephone, face-to-face, web); assess differences in data quality using paradata; and identify tests for which mixed-mode approaches may be problematic.
Preliminary results show systematic differences in levels of cognitive performance across all measures, with respondents performing better on the web than either telephone or face-to-face modes and slightly better on telephone than face-to-face. In addition, we also find mode differences in associations between cognition and certain known predictors and outcomes. This raises a concern that substantive conclusions about cognition may be sensitive to survey mode. Results of this study will help inform the development of cognition measures for the HRS moving forward and provide guidance for design and research on cognition in mixed-mode contexts.