Using the seam effect to identify occupations most prone to survey errors
Jul 26, 11:00
Survey research provides imperfect data on occupations, for two reasons. First, the use of open-ended questions increases the scope of interviewing errors. Secondly, coding the verbatim data possibly adds coding errors. Most studies on the quality of survey data on occupations involve estimating inter-coder consistency. As two or more persons are coding the same material, attention focuses on errors that arise during coding, to the detriment of errors resulting from the fieldwork. Using panel survey data can overcome this limitation, since occupations of the same panelists are described by different interviewers.
In this paper I use the Polish Panel Survey POLPAN to analyze inconsistencies in respondents’ occupational codes on the seams between each two successive waves. POLPAN began in 1987, with subsequent waves every five years since. So far, it was administered via face-to-face interviews, using pen and paper. Questions on occupations are detailed, including occupational title, job position, and a short description of job activities. In each POLPAN wave, occupations are coded using the same classification with approximately 400 categories. Since POLPAN did not experience Dependent Interviewing, the interviewing errors typical for cross-sectional surveys have been preserved.
The POLPAN data demonstrate that some occupations, like manager, inspector, or clerk, are more prone to low consistency of coding, if no additional description of occupational characteristics are available, or when these characteristics are insufficient. Verbatim responses from POLPAN provide valuable suggestions on follow-up questions for instances when the occupation title could belong to different occupational categories. Such follow-up questions could be implemented in CAI scripts. Data also show that, when occupational codes are aggregated into seven social classes, most misclassifications occur between upper and lower middle class, and between two strata of manual workers. Similar results were obtained in the US Current Population Survey (Conrad, Couper & Sakshaug 2016). This suggests that problems with the quality of data on occupations have much in common with the organization of production in modern economies.
A different take from these results deals with research on social structure. Theoretical and empirical studies emphasize the importance of hierarchical divisions within the middle and working classes. However, internal divisions within these two classes are most frequently exposed to survey errors.