Scottish Household Survey 2020: methodology and impact of change in mode

Chapter 1: Introduction

The Scottish Household Survey (SHS) is one of the largest and most important surveys in Scotland. It provides robust evidence on the composition, characteristics, attitudes and behaviour of private households and individuals in Scotland, as well as evidence on the physical condition of Scotland's homes. The national fuel poverty estimates rely on data from both the social and physical elements of the survey. Data from the SHS provides estimates for National Indicators in the Scotland’s National Performance Framework.

Since 1999, fieldwork for the Scottish Household Survey has been conducted annually, with interviews undertaken throughout the year. It has used the gold standard survey methodology of interviewing face-to-face, in people's homes, a random sample of households, to consistently produce high quality estimates. By maintaining a high response rate, it has minimised the potential for non-response bias. And by skilled interviewers carrying out the fieldwork face-to-face, it has also ensured that participants were able to fully engage with the questionnaire.

In March 2020, to help prevent the spread of Covid-19, fieldwork for the Scottish Household Survey was suspended. A revised approach was piloted and adopted for the remainder of the 2020 sample. This involved no interviewer travel, and surveys conducted remotely, either by telephone or by video.

The change in data collection method from the traditional face-to-face interviewing to the push-to-telephone/video approach has the potential to change the accuracy of the estimates and introduce discontinuity into the data series.

This report describes the methodology for the 2020 survey and explores the impact of the change in approach on the survey estimates. It covers the impact of (a) how people are approached to take part and whether this had led to increased bias, and (b) how they are interviewed and whether this had changed the way they responded to questions.

The report is structured as follows.

• Chapter 2 provides an overview of the change in approach
• Chapter 3 gives a short summary of relevant previous literature on survey error and mode effects
• Chapter 4 provides analysis of telephone matching rates and response rate patterns for the different approaches.
• Chapter 5 examines the size of differences between estimates from the telephone-matched sample and the opt-in only sample to previous estimates. This covers a range of key survey measures, before and after weighting, at both the national level and within key sub-groups
• Chapter 6 explores whether the mode of interview has had an impact on how people respond to the survey.

Acknowledgements

The authors take full responsibility for the content of this report, but gratefully acknowledge the contribution of a wide range of people who have provided support and guidance throughout.

Special thanks go to the respondents who agreed to take part in the survey. Without them, the survey would not be possible.