Interviewer variance and its effects on estimates

Abstract

Purpose

In survey methodology, it is well-known that interviewers can have an impact on the registered answers. This paper aims to focus on one type of interviewer effect that arises from the differences between interviewers in the systematic effects of each interviewer on the answers. In the first case, the authors evaluate interviewer effects on the measurement of alcohol consumption in European countries. The second case is about the interviewer effects on the respondents’ tendency to (non)differentiate their responses and the consequences of this response style for the correlation between variables.

Design/methodology/approach

The interviewer effects are evaluated by means of interviewer variance analysis. Because respondents are nested within interviewers, we can specify a two- or three-level random intercept model to calculate the proportion of variance explained by the interviewers. Data from the seventh round of the European Social Survey are used.

Findings

The results in the first case show that the substantive conclusions about the effect of gender and education on the alcohol measures continue to hold when interviewer effects are taken into account. The results of the second case make clear that interviewer effects on attitudinal questions are considerable. There is also a significant effect of the interviewers on the degree that respondents differentiate their responses. The results also illustrate that correlations between attitudinal variables are influenced. This also implies that the results of statistical procedures using a correlation or covariance matrix can be strongly influenced by the tendency to (non)differentiate and the interviewers’ impact on this tendency.

Originality/value

The results clearly demonstrate that there are considerable differences between countries concerning the impact of the interviewers on substantive variables. Cross-national differences are striking and the importance and necessity to evaluate interviewer effects in a cross-national survey becomes clear.

Citation

Loosveldt, G., Wuyts, C. and Beullens, K. (2018), "Interviewer variance and its effects on estimates", Quality Assurance in Education, Vol. 26 No. 2, pp. 227-242. https://doi.org/10.1108/QAE-06-2017-0030