Scottish Household Survey 2020: methodology and impact of change in mode
The methodology report for the Scottish Household Survey 2020 telephone survey which discusses the impact of the change in mode.
Chapter 2: Overview of the change in approach
This chapter provides a brief overview of the changes in approach to the design and execution of the survey.
Summary of change in survey approach
The design of the SHS has been broadly consistent since its inception in 1999. A number of amendments were made in 2012, when the SHS was combined with the Scottish House Condition Survey. However the core approach of a face-to-face, interviewer administered, in-home survey has remained unchanged since the beginning.
In March 2020, SHS fieldwork was suspended in response to the Covid-19 pandemic. Following the suspension of fieldwork, the SHS team at the Scottish Government, together with Ipsos MORI Scotland, assessed different options for restarting the fieldwork in a safe way. There were two key constraints that shaped considerations around the re-design of the fieldwork approach.
Firstly, the method could not involve any interviewer travel. This was to ensure that it complied with all public health guidance on social contact and appropriate working arrangements and avoided reputational and ethical risk. This precluded the use of a 'knock-to-nudge' approach, where interviewers would undertake the actual interviews remotely, but would still call in-person at sampled addresses to encourage people to take part.
Secondly, the method had to be designed to enable, as far as possible, data collected in interviews conducted before lockdown to be compared with data from interviews undertaken by the revised approach.
Approval was given in September 2020 to pilot an alternative approach. Following the pilot[2], this approach was extended to the remainder of the SHS 2020 sample with fieldwork resuming in early January 2021 and completed in March.
Table 2.1 summaries the key elements of the change of approach.
| Pre-lockdown approach | Revised Push-to-Telephone/video approach | |
|---|---|---|
| Survey overview | Target of around 10,450 surveys a year[3]. The target population is the Scottish population living in private households[4]. All parts of Scotland are included including the small islands. A two-part 60-minute social interview, the first part with a householder, the second with a random adult in the household[5]. | The target for the revised approach was to achieve a large enough sample size from the unworked sample to allow for national level analysis. |
| Sample design | The Royal Mail's small user Postcode Address File (PAF) is used as the sample frame for the address selection[6]. The sample has been drawn as a completely un-clustered sample since 2012[7]. It is stratified by local authority with disproportionate sampling to meet minimum target numbers in each area. 18,195 addresses were drawn for the 2020 wave of the SHS. Prior to the break in fieldwork necessitated by the COVID-19 pandemic, interviewing had been continuous, with fieldwork organised into annual waves. | The revised approach used the addresses that had not been worked when the face-to-face interviewing was suspended in March 2020. When fieldwork was suspended, 15,400 addresses in the 2020 SHS sample had not been started or fully worked and were worked using the revised approach. Telephone matching was undertaken to allow interviewers to try to get agreement to interview by telephone. Matching was successful for 23% of addresses. Details of the sub-samples profiles and the telephone matching are given in Chapter 4. |
| Questionnaire | The social survey covers a wide range of topics including the composition, characteristics, attitudes and behaviour of Scottish households and individuals. | The questionnaire was amended in places, partly because of concerns about length, and partly to adapt it for the different modes of interview. Amendments were kept to a minimum to facilitate comparability with data collected face-to-face. Further details are provided below this table. |
| Mode of approach | Householders were sent an advance letter and leaflet in advance of interviewers calling. Interviewers were required to make at least 6 attempts to secure an interview at a sampled address. | With no interviewer travel allowed, gaining consent for interview came either in response to respondents opting-in on receipt of the advance materials, or in response to an approach by telephone. After the initial mail-out, addresses where a phone number had been obtained were followed up by telephone call. For those where we were unable to obtain a telephone number, two reminders were sent after the initial mail-out, a postcard reminder followed by a final letter reminder. Advance letters directed participants to a portal where they could log in using a unique reference and then submit their name and contact details. These were then passed to the interviewer team. Respondents were given a conditional incentive of £20 for completion, to encourage participation. |
| Fieldwork and mode of interview | Interviews were conducted in-home, face-to-face using Computer Assisted Personal Interviewing (CAPI). Pre-pandemic, interviewing would normally take around fourteen months per wave, starting in January and finishing in the February of the following year. Face-to-face interviewing on the 2020 wave was suspended on 17 March. | All interviews were undertaken remotely, either by telephone or video link. Video link interviews used one-way Microsoft Teams, where the respondent could see the interviewer. Fieldwork was undertaken by interviewers from the SHS face-to-face interviewer panel. All interviewers were briefed via video call on the revised approach, prior to the pilot starting. Fieldwork for the push-to-telephone pilot was undertaken in Oct 2020. The rest of the fieldwork was undertaken between January and April 2021. Appendix 3 provides some analysis of seasonal effects on the SHS data. |
| Physical survey | Since 2012, the survey has included a follow-up physical survey for a sub-sample of addresses, to incorporate the Scottish House Condition Survey (SHCS) elements. | As the SHCS physical survey requires an extensive inspection of both the inside and outside of dwellings, this element of the SHS remained suspended. |
| Data Processing | As well as data checks and editing involving range checks, simple logic checks and complex logic checks, the data undergoes three additional processes: calculation of derived variables such as age and gender of the Highest Income Householder; imputation of household income; and imputation of housing costs. | The data processing approach was unchanged. |
| Survey response | An overall response rate of around 63% - 65% was achieved between 1999 and 2019. In total, 1,545 interviews from 2,796 addresses pre-lockdown were achieved. | The response rate for the opt-in only sample was 14%. For the telephone number matched sample, it was 37%. Overall, the response rate achieved using the revised approach was 20%. This is detailed further in Chapter 4. 3,031 interviews were achieved from the 15,399 addresses that used the revised approach. |
| Survey weighting | The SHS incorporates selection weighting to address the unequal selection probabilities and calibration weighting to correct for non-response bias. Calibration weighting derives weights such that the weighted survey totals match known population totals for sex and age band within Local Authorities. | The calibration model chosen for 2020 (model 2 in Appendix 2) calibrates to NRS population and household estimates for 2020 for age band and sex within Local Authorities, but also to SIMD quintiles and Urban-Rural Classification. Age bands used in the calibration for 2020 are wider than in 2019, due to smaller respondent numbers. An additional alternative weighting model was chosen for housing related questions to take specific account of tenure bias in the achieved sample. |
| Limitations of the data | Like all sample surveys, the SHS can only produce estimates and these estimates are limited by factors such as sample coverage, sampling variability, the number of cases that analysis is based on, and the bias in the achieved sample[8]. | The smaller achieved sample size means that the confidence internals around the estimates are wider. The change in approach means that the profile of bias in the achieved sample is different to before. This is discussed throughout the rest of the report. Chapter 3 summarises previous literature on Total Survey Error. Chapter 4 examines the profile of the achieved sample. Chapter 6 looks for evidence of changes to measurement error. |
Amending the questionnaire
At around an hour in length, and often involving relatively complex questions and showcards, the SHS questionnaire in its existing format has relied on interviewer facilitation to encourage full participant engagement and, in turn, the quality of the data captured. The design of the questionnaire was not optimal for interviewing by telephone or video. However, changes to the questionnaire had to be kept to a minimum to facilitate comparisons with the data collected face-to-face prior to lockdown.
The full questionnaire was reviewed and revised prior to the pilot. The changes between the pre-lockdown questionnaire and the pilot questionnaire are fully detailed in the questionnaire documentation[9].
The main consideration was how to adapt the questions that relied on showcards.
A total of 119 showcards were used in the face-to-face SHS 2020 questionnaire. If interviewing using video, the interviewer could use showcards via screenshare, but an alternative strategy was needed for the telephone interviews. Two main approaches were devised. First, in instances where the question was factual
(e.g. ethnicity), interviewers were instructed to read the question, wait for the respondent to answer and then select the corresponding code. Interviewers then confirmed the code they had selected with the respondent before continuing. Second, for questions where the range of response options are not obvious from the question, the interviewer was directed to read out all the response codes along with the question. The strategy for each showcard was added as an interviewer instruction below the question. Online showcards were developed for a small number of questions for the telephone interviews during the push-to-telephone main stage. However, these were not widely used.
The questions were reviewed with the impacts of COVID in mind, so that they still made sense to respondents. Where there might be ambiguity – particularly around whether the question related to the participants' 'usual' pre-COVID circumstances, or their current circumstances – this was clarified in interviewer instructions and/or through tweaks to the question wording. An example of this is given in Figure 2.1, with changes detailed in orange.
There were some changes in relation to the streaming and the proportion of people who were asked particular sets of question. These changes were driven by considerations around interview length, which was expected to increase given the change in approach, and the suspension of the SHCS physical survey.
Two new questions were added to the questionnaire. The first relating to furlough and the second to changes in income that resulted from COVID.
There were also a limited number of questions that were temporarily paused from the survey because of their complexity and heavy reliance on detailed showcards, such as the section on repairs and maintenance undertaken in the last year.
Revisions to the questionnaire and the impact of the change in mode and are discussed in more detail in Chapter 6.
Contact
Email: shs@gov.scot
There is a problem
Thanks for your feedback