Consultation on the design of the Scottish Household Survey (SHS) for 2017 and beyond

Options for consultation

There are two options which the Scottish Government would like stakeholders' views on, namely:

Option A
Biennial topics, i.e. halving the number of topics covered by the survey every year and collecting data on each topic every second year, with a small reduction in sample size (from 10,700 to 10,100).

Option B
Reduction of the overall survey sample size by around a third, from 10,700 to 7,450, with a small reduction in topics covered by the survey.

Both options retain an annual Scottish Household Survey (SHS) social survey and an annual physical survey, and both options would realise a similar level of cost savings.

A description of each option is provided below, including Table 1 which provides an overview of the key characteristics of each option.

Option A maintains a sample size of over 10,000 and the same level of topic coverage, albeit on a biennial basis for most social survey topics. In other words, one half of topics would be covered in an odd year, and the second half of topics in an even year. In line with the small reduction in sample size at national level (from 10,700 to 10,100 or 4%), one third sample topics would have a small reduction in sample size from 3,550 to 3,350. Savings are achieved by reducing the SHS interview time from 1 hour to 40 minutes.

Whilst this option sacrifices the frequency of data collection it retains the full ability to drill down and undertake sub-group analysis in any given year. It should be noted that a small number of key household and key random adult questions are 'protected' under option A, i.e. they would run on an annual basis in order to retain the ability to analyse across household and adult respondent characteristics in both odd and even years. Annex A shows which sections of questions are protected.^[1] This includes all SSCQ questions.

Local Authority level data would continue to be published every year to a similar level of precision, but every second year for most social survey topics (i.e. non-protected questions).

Option B maintains annual data collection for all questions, but reduces the overall sample size by 30%. The majority of topics are also retained in the survey. Under option B savings are achieved by cutting the sample size.

To maintain 'one third sample' questions at a similar sample size in SHS 2017 around 4 minutes of topics/questions (around 7% of questions) would have to be cut in order to maintain the interview at 1 hour in length.^[2] Alternatives to the 4 minute reduction in topics/questions could be making around 9 minutes worth of full sample topics biennial or changing a similar amount of topics to 'one third sample' questions. The consultation questionnaire asks for people's views on options for achieving a 4 minute reduction in interview time.

In essence, option B prioritises annual data collection, but with a reduced sample size for full sample questions, and does so whilst maintaining the majority of topic and question coverage. Sub-group analysis would still be possible for many variables, but to a lower level of precision than currently available under an overall survey sample size of 10,700. Two years or more of data would need to be combined for detailed subgroup analysis.

In line with the national sample size reduction under option B, all local authorities would receive a reduction in their sample size. This means annual local authority results would no longer be available under Option B. Instead the Scottish Government proposes to publish local authority results on a two year rolling average basis each year, i.e. two years' worth of data would be combined on a rolling basis to produce average figures each year.^[3]

The two year rolling average estimate, based on a combined sample size of 15,000, would have higher precision, i.e. smaller confidence intervals, than the current annual estimates based on a sample size of 10,700. This would make it easier to detect 'real' differences between estimates for different local authorities than current annual estimates (due to the increased sample size from creating two year rolling averages), but more difficult to identify change over time in the short term within a local authority.^[4] This is due to rolling averages producing 'smoothing' of estimates, although they do still identify trends that emerge over time.

The SHCS currently publishes its local authority level estimates on a 3-year rolling average basis. Nevertheless rolling average estimates make it more difficult to detect change over time when there are small changes in estimates.

An alternative is to produce and publish data every two years on a two year basis for local authorities, i.e. under option B 2016 and 2017 data would be combined to produce estimates published in summer 2018, followed by 2018 and 2019 data which would be combined and published in summer 2020. Presenting and publishing data this way makes it easier to detect change over time when there are small changes in estimates. The consultation questions in Annex B asks for people's preferred option for the publication of local authority results. However, it should be noted that data every two years on a two year basis, i.e. 2016 and 2017 data combined, and then 2018 and 2019 data combined, would be available for making clearer comparisons over time under the two year rolling average publication option.

For comparison, the main features of options A and B are shown in Table 1 over the page.

Table 1 Options A and B overview

Options	Principles	National sample size and frequency	Local sample size and frequency
Option A Biennial Topics Collect half of the topics in odd year (2017), half of the topics in even year Key household and random adult 'protected' questions (including SSCQ) unchanged on an annual basis	Maintain sample size at over 10,000 households for full sample topics. 'One third sample topics' maintain one third of full sample size (i.e. 3,350) Reduce frequency of data collection for most topics Retain all topic coverage, albeit every two years for most topics	Largely unchanged sample size and precision albeit every other year for most topics Difficulties in combining two years' worth of data for detailed sub-group analysis due to combining non-consecutive years (a particular issue for 'one third sample' topics')	LA data published annually albeit every other year for most topics Key household and random adult data published annually Largely unchanged sample size for LAs Same issue of combining non-consecutive years for some sub-group analysis
Option B Reduce sample size by around one third (to 7,450) All topics have 7,450 sample size except current 'one third sample' questions which slightly increase their sample size to 3,700 (from current 3,550)	Reduce sample size for full sample topics 'One third sample topics' maintain similar sample size (i.e. 3,700) Maintain frequency of data collection Small reduction in topic coverage - around 4 minutes - or equivalent reduction in frequency or sample size to achieve the same time-savings	Reduced annual sample size and precision of results If necessary combine data from 2 years to have sufficient sample sizes for detailed sub-group analysis	Reduced sample size for LAs means publication of annual data no longer possible. LA data published annually but on two year rolling average basis Two year rolling averages at improved precision than current annual estimates

The main implications of option A are that:

• National and local authority data would be published annually, but only every other year for most topics. Key household and random adult questions would be published annually as they retain an annual sample size of over 10,000.
• The Travel Diary would only be asked every second year.
• SHCS social survey questions would only be asked every second year. However, there would be no impact on fuel poverty estimates which are based on the physical survey (which retains its current 3,000 target sample size) and protected questions in the SHS social survey, nor on energy efficiency SAP ratings which are based on the physical survey.
• SHS based National Performance Framework (NPF) Indicators would only be updated and reported every other year.
• Risk of losing ability to cross tabulate variables if split across a different year of the survey.
• Rarely occurring characteristics and/or sub-group breakdowns - either at national or local level - that already require a combination of two years' worth of data to achieve a sufficient sample size would need to combine non-consecutive years of data (e.g. 2017 and 2019 if this design continues for SHS 2018-2021). This would be the case for some national level breakdowns of one third sample questions which currently require the merging of two years' worth of data.
• Precision of equalities breakdowns protected at current levels, but impact of biennial equalities monitoring for most topics.

The main implications of option B are that:

• Annual local authority results would no longer be available. Local authority data would be published every year, but on a two year rolling average basis and at an improved level of precision. This makes it easier to identify 'real' differences between local authorities than current annual estimates (due to the increased sample size from creating two year rolling averages), but more difficult to identify change over time in the short term within a local authority area.
• The Travel Diary would be asked every year but at a reduced sample size and level of precision.
• SHCS social survey questions would be asked every year at around the same sample size as currently (SHCS questions are 'one third sample' questions). There would be no impact on fuel poverty estimates nor energy efficiency SAP ratings which are based on the physical survey which retains its current 3,000 target sample size.
• SHS based National Performance Framework (NPF) Indicators are reported annually but at a reduced level of precision making it more difficult to detect 'real' change on an annual basis.
• Rarely occurring characteristics and/or sub-group breakdowns - either at national or local level - that require a combination of two years' worth of data to achieve a sufficient sample size would have a higher combined sample size than under option A. (15,000 every two years under option B compared to 10,700 every two years under option A.) This particularly benefits one third sample questions that currently require two years' worth of data to produce sub-group breakdowns as it avoids combining two non-consecutive years' of data.
• Retain many national annual equalities breakdowns but at a lower precision than currently. May need to combine two years of data for breakdowns by some rarer equalities characteristics.

Consultation responses will be used to inform the design of the survey for both SHS 2017 and SHS 2018 - 2021 (subject to sign off of procurement by Scottish Ministers). Issues around continuity of the survey will be considered as far as possible when making decisions on 2018 - 2021.

Differential impacts

Options A and B would have different impacts depending on the specific topics and questions. The graphics below show how different sections of questions are treated under each of the options, whilst Annex A provides information on what topics fall under each heading in the graphic.

Option A Biennial Topics

Option B Reduce sample size

It should be noted that current one third sample topics are not illustrated in the graphics as they maintain a sample size of around 3,550 under both options (option A 3,350 and option B 3,700). In option A one third topics are only available every other year, the same as most social survey topics.

A few examples may help illustrate the different impacts.

SHS Key Household and Key Random Adult Questions (e.g. household income estimates)
Under option A, a small number of key household and random adult questions are 'protected' and asked on an annual basis. Therefore annual household income would be published every year at a national level and local authority level. However, under option B, whilst household income estimates would be available on an annual basis nationally every year, albeit at a lower precision than currently, at local authority level there would be no annual results. Instead, they would be published every year but on a two year rolling average basis.

Most social survey topics - random adult topics and other household section topics

For all social survey topics that currently appear in the non-protected random adult and other household sections on an annual basis, option A would mean that data is only available every other year at national and LA level for both full and one third sample topics, whereas for topics that are currently run on a biennial basis (e.g. culture and transport) there would be no change to frequency.

Under option B, annual data would be available nationally for those social survey topics currently run on an annual basis, albeit at a reduced sample size and precision level for full sample topics (current 'one third sample topics' maintain a similar sample size, 3,700, under option B). At local authority level there would be no annual results but data would be published every year on a two year rolling average basis for full sample topics. There would be no change in frequency for current biennial topics.