Scottish household survey 2017: methodology and fieldwork outcomes

8 Limitations of the Data

Summary

• There are a number of important methodological and data issues that users need to be aware of when using the SHS data.
• Like all sample surveys, the SHS can only produce estimates and these estimates are limited by a number of factors. The factors are Sample coverage; Sampling variability, the number of cases that analysis is based on and the Bias in the achieved sample
• The SHS is also limited in the amount of detail it can collect about some topics. For example, it was not designed to provide reliable "economic" statistics (e.g. employment/unemployment rates and average earnings).
• As a multi-purpose survey of households, the SHS is not designed to provide the kinds of information about economic activity and household income that can be obtained from more specialised surveys such as the Labour Force Survey and the Family Resources Survey.
• Although the SHS has a large sample that covers the whole of Scotland, it has some geographical limitations. Users should not use it to undertake geographical analysis below local authority level. Instead, the Scottish Surveys Core Questions should be used for this.
• Users need to be mindful of the sampling errors for analysis but especially when this is based on breakdowns within a single local authority.

8.1 Introduction

There are a number of important methodological and data issues that users need to be aware of when using the SHS data.

Like all sample surveys, the SHS can only produce estimates and these estimates are limited by a number of factors.

• Sample coverage - although there are no geographical exclusions to the survey, the sampling frame does not cover the whole population because of a combination of inherent limitations and administrative errors and delays.
• Sampling variability - all samples can differ from the population by chance. This is often referred to as sampling error.
• The number of cases that analysis is based on - estimates based on large samples are more accurate than those based on small samples.
• Bias in the achieved sample - if a sample under-represents sections of the population or if a large proportion of people do not answer some questions, the estimates may differ substantially from the population for reasons that are not a result of chance. For example, in 2017, the unweighted sample of adults is 54 per cent female and after basic weighting 52 per cent of the sample is female which is in line with the 2016 population estimate of 52 per cent.^[23] This is an example of bias caused by young males, in particular, being difficult to contact or refusing to take part in the survey.

Although the use of calibration weighting addresses the disparity between the age/sex composition of the sample and the known composition of the population, it does so on the assumption that respondents do not differ in terms of survey measures that do not form part of the weighting. The review of the weighting strategy generally found that calibration brought the survey estimates closer to census estimates but like all surveys, the potential for bias remains a limitation that should be considered.

The SHS is also limited in the amount of detail it can collect about some topics. For example, it was not designed to provide reliable "economic" statistics (e.g. employment/unemployment rates and average earnings).

The SHS's information about the economic status of members of the household reflects the view of the respondent to the "household" part of the interview, and so may not conform to official definitions of employment and unemployment, for example. As a result, the SHS cannot provide estimates of unemployment that are comparable to official statistics of unemployment.

There are several reasons why the SHS data on income may not be completely accurate.

• The SHS only collects information from, or about, the Highest Income Householder and, if there is one, their spouse or partner.
• Information is provided "off the top of the head" as part of an interview on many other topics. There is no requirement to refer to pay slips or bank statements to check the figures.
• Some people may not know the correct figure (particularly in the case of the income of a spouse/partner), and may just provide a guess, perhaps based on a level that they remember from some time ago.
• Other interviewees may under-state their income because they do not want to reveal how much they really earn.
• Because about a third of the households in the sample are unwilling or unable to provide income information, values for some or all of the main components of income have to be imputed.

As a multi-purpose survey of households, the SHS is not designed to provide the kinds of information about economic activity and household income that can be obtained from more specialised surveys such as the Labour Force Survey and the Family Resources Survey, which have questions and procedures which are designed to obtain much more reliable information on those matters than the SHS can collect. The SHS has questions on such topics only for selecting the data for particular groups of people (such as the unemployed or the low-paid) for further analysis, or for use as "background" variables when analysing other topics (such as the means of travel or the frequency of driving).

Although the SHS has a large sample that covers the whole of Scotland, it has some geographical limitations because of the sample sizes in small local authorities and because it is designed to be representative only at national and local authority level.

This means:

• users need to be mindful of the sampling errors for analysis but especially when this is based on breakdowns within a single local authority
• it is not appropriate to undertake geographical analysis below local authority level since the sampling techniques used in some local authorities cannot guarantee representativeness in smaller areas.