Scottish Household Survey: response rates, reissuing and survey quality

1 Introduction

1.1 The Scottish Household Survey (SHS) is the largest and one of the most important surveys in Scotland and is central to the Scottish Government's evidence based approach to policy making.

1.2 The SHS is an important source of data for a variety of areas including housing, public services, physical activity, volunteering and cultural participation. It provides one-third of the National Performance Framework Indicators, is the source of evidence for Single Outcome Agreements and a number of Benchmarking Frameworks and informs cross-reaching policy development.

Background

1.3 Like most major face-to-face random probability surveys, response rates on SHS have been declining, albeit slowly.

1.4 The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.

1.5 In 2008, the SHS response rate was 67% and this fell to 64% by 2018. It is, however, taking more and more fieldwork effort and cost to achieve these headline figures. Like many major face-to-face surveys, the SHS has increasingly relied on reissuing non-responding samples at first issue to other interviewers to try and maintain the overall response rate.

Overview of the project

1.6 The primary focus of this analysis is to assess what impact reissuing had on survey estimates, building on the previous work described above to explore the effect of the response rate on the quality of the estimates. This methodological investigation provides important evidence for other surveys grappling with falling response rates and rising cost issues. Reissuing is a widespread practice in surveys, whereby people who have not responded to the first interviewer^[1] are revisited by another interviewer in an effort to get them to take part. It is a common technique for maintaining a high response rate and reducing the risk of non-response bias.

1.7 The analysis in this paper explores how the published results of the survey would differ if reissuing had not been used. It does this by comparing the published estimates from the weighted full survey sample with estimates from first issue interviews only, weighted as if they were the final achieved sample. This is done across a range of key measures at the national level, on estimates for key sub-groups, and for two waves of SHS data (2014 and 2016).

1.8 For SHS, in 2014 reissues increased the response rate from 56% to 67%, and in 2016 from 54% to 64%. Therefore, the analysis presented shows the effect of reducing the response rate by around 10-11 percentage points on SHS.

1.9 The scale of the impact on each estimate is reported in two main ways. Firstly, in terms of the absolute difference. This has been calculated as the published estimate minus the revised lower response rate estimate. Secondly, because the absolute differences are not a good indicator of significance, we also standardise these differences. This has been done by comparing them to the standard error of the published full sample estimate.