Scottish Household Survey 2020: methodology and impact of change in mode
The methodology report for the Scottish Household Survey 2020 telephone survey which discusses the impact of the change in mode.
Footnotes
1. Normally, response rates are adjusted to account for deadwood.
2. https://www.gov.scot/isbn/9781802019568
3. Before 2012, the target was around 15,000. Between 2013 and 2017, the target was 10,678 interviews.
4. The target population of the SHCS is subtly different, excluding dwellings that do not have foundations – eg. static caravans, canal boats etc. – that are used as permanent homes. This has no effect on the SHS social survey.
5. Before 2012, the survey length was 45 minutes.
6. This excludes institutional locations such as prisons, hospitals, military bases, and student halls of residence.
7. Before 2012, a degree of clustering was used in the least densely populated Local Authorities.
8. Further discussion of the limitations of the data using the pre-lockdown approach can be found in Scottish Household Survey 2019: methodology and fieldwork outcomes - gov.scot (www.gov.scot)
9. https://www.gov.scot/isbn/9781802019568
10. The impact was less than half of the standard error for the majority of estimates included in the analysis
11. Such as analysis undertaken by ONS in 2015 on the impact of a lower response rate on the Crime Survey of England and Wales. (Williams & Holcekova, 2015).
12. It did this by re-weighting the results of the sample achieved at first issue and ignored data collected at reissue.
13. The administrative source on the number of social rented households comes from information held by each social landlord on their housing stock. This data is collected from social landlords in Scotland by the Scottish Housing Regulator as part of annual Charter Data and Stock returns, with information published at www.housingregulator.gov.scot/landlord-performance/statistical-information. The Scottish Government also collects aggregate annual statistics from local authorities on levels of council housing stock, which is reported on in annual housing statistics releases at www.gov.scot/collections/housing-statistics, with further information on the data quality, sources and suitability of these statistics available at www.gov.scot/publications/housing-statistics-data-quality-sources-and-suitability. The administrative source on the number of private rented households comes from the Scottish Landlord Register. The Scottish landlord registration scheme provides a register of all private landlords, in which private landlords must register with their local authority and ensure they meet the legal requirements for letting houses. Local authorities are responsible for the administration and enforcement of landlord registration. The figures are not subject to any formal statistical quality assurance checks, and may depend on the quality of the underlying data being recorded and maintained by each local authority. In addition there are risks of duplicated data due to errors in the imported data from a previous IT system, and some double counting of rental properties in certain instances for joint owners. The administrative figures presented may therefore not be as robust as if they were produced as part of a set of official statistics.
14. Note that this is different from deadwood - addresses that are on the PAF but are ineligible because they are vacant, demolished, used as second homes, or ineligible for some other reason. While the deadwood rate has typically been 8-9% in the major Scottish surveys, this has not impacted the representativeness of survey estimates.
15. In this study, the telephone survey element used Random Digit Dialing while the face-to-face approach drew a sample from the PAF.
16. For almost all addresses where both suppliers managed to link in a telephone number, the number was identical.
17. Ofcom technology tracker
18. https://www.gov.uk/electoral-register/opt-out-of-the-open-register
19. The main reason why it was lower than 2018 and 2019 was that the lockdown meant that no reissuing of unproductive addresses was carried out.
20. Total interviews achieved divided by all addresses. No adjustment for deadwood is made. Therefore, the overall figure for response is lower than the published headline figures.
21. Unadjusted response rates by LA by wave are provided in Table A4.1 in Appendix 4.
22. See Appendix 2 for more details.
23. Further details are provided in the tables in Appendix 2 including the unweighted figure and the figures for 2018.
24. For example, the impact of the calibration adjustments to match household totals and how the calibration targets have been set for each variable.
25. The household weight has been applied to this random adult question to obtain an estimate of households. Households with no completed random adult interview are excluded.
26. Additional weighting by tenure was introduced for housing related questions for the 2020 results because of these large changes in the tenure estimate.
27. While age and gender of household members are used in the weighting, we do not weight specifically for the HiH's age and gender.
28. Calibration weighting by gender uses NRS population estimates for sex.
29. 'Minority ethnic groups' includes mixed or multiple, Asian, African, Caribbean or Black, Arab or any other ethnic groups
30. 'White other' includes Irish, Gypsy/Traveller, Polish and other white ethnic groups
31. 'Minority ethnic groups' includes mixed or multiple, Asian, African, Caribbean or Black, Arab or any other ethnic groups
32. Figures for 2018 rather than 2019 for these measures (biennial questions)
33. Figures for 2018 rather than 2019 for these measures (biennial questions)
34. Figures for 2018 rather than 2019 for these measures (biennial questions)
35. A total of 119 showcards were used in the face-to-face SHS 2020 questionnaire.
36. There are some soft checks in the script, for example, if not receiving child benefit and a child under 16 is in the household. Full details of the income imputation routines used in the SHS can be found here: www.gov.scot/binaries/content/documents/govscot/publications/statistics/2020/12/scottish-household-survey-2019-methodology-fieldwork-outcomes/documents/shs-income-imputation-project-adults-income-final-memo-ipsos-mori/shs-income-imputation-project-adults-
37. Logistic regression of any/none cultural attendance and any/none cultural activity. Independent variables included were age band, gender, highest educational qualification, rurality, and SIMD quintile. These were treated as categorical variables.
38. Both neither agree nor disagree and no opinion responses are less common for telephone respondents than for other respondents.
39. Logistic regression of whether smokes. Independent variables included were age band, gender, highest educational qualification, rurality, and SIMD quintile. These were treated as categorical variables.
40. Normally, response rates are adjusted to account for deadwood.
41. https://www.gov.scot/publications/scottish-household-survey-2019-methodology-fieldwork-outcomes/
42. Generally speaking, a calibration model with more constraints will produce weights that are more extreme/have more variation and this will result in larger design effects and confidence intervals. This is another reason for wanting to minimise the number of constraints/variables in a calibration model.
43. It is generally not best practice to calibrate a survey to data from a previous year of the survey. This does nothing to get rid of any underlying non-response bias built into the survey. Furthermore, it is not ideal to calibrate a survey to survey estimates rather than administrative data, because of the sample errors associated with the survey data. When using survey data in a calibration model, the confidence intervals for weighted estimates that arise from these models should reflect the sample error in the calibration targets. This can be done but would involve a lot of effort (see Berger et al. and Estevao and Särndal).
45. Information on gender was collected in the 2020 SHS but the survey was calibrated to NRS population estimates that are based on sex. Where a respondent in the SHS reported their gender identity as 'In another way' or refused to provide an answer then, only for the purpose of calibrating the survey to NRS estimates, they were randomly allocated to a female or male gender.
47. For more details see the 2019 SHS methodology and fieldwork outcomes report: https://www.gov.scot/publications/scottish-household-survey-2019-methodology-fieldwork-outcomes/
50. https://www.gov.scot/publications/scottish-surveys-core-questions-2019/
See Chapter 3 for details of the administrative sources and comparability with SHS estimates.
51. https://www.gov.scot/publications/scottish-household-survey-2019-annual-report/
52. The household weight has been applied to this random adult question to obtain an estimate of households. Households with no completed random adult interview are excluded.
53. 'White other' includes Irish, Gypsy/Traveller, Polish and other white ethnic groups
'Minority ethnic groups' includes mixed or multiple, Asian, African, Caribbean or Black, Arab or any other ethnic groups
54. Figures for 2018 rather than 2019 for these measures (biennial questions)
55. Figures for 2018 rather than 2019 for these measures (biennial questions)
56. Figures for 2018 rather than 2019 for these measures (biennial questions)
58. The survey average design factor for the second calibration model was calculated to be 1.34 compared to 1.41 for the third calibration model.
59. The household weight has been applied to this random adult question to obtain an estimate of households. Households with no completed random adult interview are excluded.
60. 'White other' includes Irish, Gypsy/Traveller, Polish and other white ethnic groups
61. 'Minority ethnic groups' includes mixed or multiple, Asian, African, Caribbean or Black, Arab or any other ethnic groups
62. Figures for 2018 rather than 2019 for these measures (biennial questions)
63. Figures for 2018 rather than 2019 for these measures (biennial questions)
64. Figures for 2018 rather than 2019 for these measures (biennial questions)
65. 'Minority ethnic groups' includes mixed or multiple, Asian, African, Caribbean or Black, Arab or any other ethnic groups
Contact
Email: shs@gov.scot
There is a problem
Thanks for your feedback