Scottish household survey: Frequently Asked Questions

Overview

Surveys such as the Scottish Household Survey (SHS) have recognised limitations in how robust the final survey estimates are.  This section provides a brief overview of the same key issues that users should be aware of.

Representativeness of Data

Although the SHS sample is chosen at random, the people who take part in the survey will not necessarily be a representative cross-section of the population. Like all sample surveys the results of the SHS are estimates of the corresponding figures for the whole population and these results might vary from the true values in the population for three main reasons:

1. The sample source does not completely cover the population because accommodation in hospitals, prisons, military bases, larger student halls etc. are excluded from the sampling frame. The SHS provides a sample of private households rather than all households.
2. Some people refuse to take part in the survey and some cannot be contacted by interviewers. If these people are systematically different from the people who are interviewed, this represents a potential source of bias in the data. Comparison of the SHS data with other sources suggests that for the survey as a whole, any bias due to non-response is not significant.
3. Samples always have some natural variability because of the random selection of households and people within households. 

Each of these sources of variability becomes much more important when small sub-samples of the population are examined. For example, a sub-sample with only 100 households might have had very different results if the sampling had, by chance, selected four or five more households with children, or households including one or two adults of pensionable age and no younger adults.

Statistical Significance

All proportions produced in a survey have a degree of error associated with them because they are generated from a sample of the population rather than the population as a whole.

Any proportion measured in the survey has an associated confidence interval (within which the 'true' proportion of the whole population is likely to lie), usually expressed as ±x%. It is possible with any survey that the sample achieved produces estimates that are outside this range. The number of times out of a 100 surveys when the result achieved would lie within the confidence interval is also quoted; conventionally the level set is 95 out of 100, or 95%.

Comparability with Other Sources

In some cases the SHS is not the official source of statistics on a particular topic: such as income, employment or housing. The interview collects information on these topics to select the data of particular groups for further analysis or for use as background variables when analysing other topics.

Whilst surveys such as the SHS collect information on the main equality strands, it is often difficult to gain large enough sample sizes for minority groups to gain robust estimates.  For example, information on an individuals ethnic group is collected, though many ethnic groups are small in number which could lead to statistical unreliability when analysing data. The problem of small numbers can be overcome by combining categories under a section heading, for example combining the counts of people who responded "Caribbean, Caribbean Scottish or Caribbean British". This is not an ideal solution as it can hide inequalities that occur between each of the separate categories. Often the SHS is required to use a 2-fold classification of "White" and "Mixed or Multiple; Asian; African; Caribbean or Black; and Other ethnic groups".

The question on self-identified sexual orientation was introduced to the SHS in 2011 to provide statistics to underpin the equality monitoring responsibilities of public sector organisations and to assess the disadvantage or relative discrimination experienced by the lesbian, gay and bisexual population.  Despite this positive step in collecting such information, it is felt that the figures are likely to under-report the percentage of lesbian, gay or bisexual (LGB) people within society due to a number of reasons, including the following.

• Asking about sexual orientation/identity is a new development in national surveys and such questions can be seen as intrusive and personal.
• There is still significant prejudice and discrimination against LGB people in society.  In a context where some LGB people will not have told friends and family about their sexual identity, there is a real question about whether LGB people generally would want to be open with an interviewer.
• The default option for being uncertain about one's sexual orientation may be to respond 'straight/heterosexual' rather than to say 'Don't know / not sure'.
• Particular LGB people are still less likely to be open where they belong to groups or communities where an LGB identity is less acceptable.

Despite the uncertainties of the data, it does make sense to collect statistics on sexual orientation, to start to make this a more standard element within data collection. This does not mean that data will necessarily become reliable over the short term, but they may still be able to offer useful insights into the experience of some LGB people in particular areas of policy interest. The Scottish Government is looking at how it can improve its data collection on these issues going forward.

The Equality Evidence website brings together information the Scottish Government and its Agencies collect, analyse and publish on equalities

Information on housing is collected through the SHS to help explore the relationships between living circumstances and the characteristics, attitudes and behaviours of Scottish households.  The House Condition component of the survey (based on the previously separate Scottish House Condition Survey) is the primary source of information about the physical condition of housing in Scotland.  More detailed information on housing is available through the Housing Statistics for Scotland website.

Some information is collected on community safety and perceptions of crime within the SHS, though the preferred source for these are the Scottish Crime and Justice Survey (SCJS) alongside other information collated by the Scottish Government on the Crime and Justice theme. By having the questions in the SHS we have a way of exploring whether the very process of taking part in the SCJS heightens anxiety about crime.  It should also be noted that the Scottish Government conducts several major population surveys that are used to inform the policy debate in Scotland, and in some instances the surveys can be complimentary.   The Long Term Strategy for Population Surveys in Scotland 2009-2019, of which the SHS is a central element, is designed to improve the way population surveys are run and to increase the availability and use of survey data, both at a national and local level.  This work includes asking a core set of questions commonly across the Scottish Government large-scale cross-sectional population surveys, such as the perceptions of crime question, to allow the samples of multiple surveys to be pooled for further analysis (see Population Surveys in Scotland).

The SHS gathers information about the current economic situation and the characteristics of individuals and households in different economic activity categories. The official source of statistics on employment, unemployment and economic activity is the Labour Force Survey for Scotland and the Annual Population Survey at a local authority level - see Labour Market Statistics for further information.

Although the SHS is not the definitive source of information about education and qualifications in Scotland (see School Education Statistics and Lifelong Learning Statistics), it can contribute to the measurement of key education indicators.  Education level is also an important factor that can be used in the wider analysis of the survey's data, for example to explore differences in people's characteristics and attitudes by educational attainment.