Scottish Household Survey 2021: methodology and fieldwork outcomes

Methodology of the 2021 Scottish Household Survey and information on fieldwork targets and outcomes.


Sample design

Requirements

The sample for the 2021 Scottish Household Survey (SHS) was designed by the Scottish Government. From 2012, the sample design has been coordinated with the sample designs for the Scottish Health Survey (SHeS) and the Scottish Crime and Justice Survey (SCJS), as part of a survey efficiency project to allow the samples of the three surveys to be pooled for further analysis.

The SHS sample has been designed to allow annual publication of results at Scotland level and for local authorities. To meet these requirements, the target sample size for Scotland was 10,450 household interviews with a minimum local authority target of 250. From 2012, the physical survey of the Scottish House Condition Survey (SHCS) has been incorporated into the SHS. The SHCS has a required sample size of 3,004 for Scotland and a minimum of 80 for each local authority.

Sample design and assumptions

The Scottish Household Survey has a single-stage unclustered sample design. In order to provide annual local authority results without specifying an excessive overall sample size, the sample was disproportionately stratified by local authority (smaller local authorities have a higher sample proportion relative to their populations than the larger local authorities).

To deliver the required local authority precision, the minimum sample size for each local authority was set at 250. In order to estimate the annual target achieved sample size for each local authority, analysis of design effects from the 2007-08 survey was undertaken, since the effective sample size is equal to the achieved sample size divided by the design effect.

As rural areas of local authorities were clustered in the 2007-08 survey, for the 2021 unclustered sample the median design effect from a range of variables for the unclustered parts of local authority samples were assumed for the entire areas in 2021. This allowed the calculation of the target achieved sample size for each local authority, as shown in Table 1.

Normally, the sampling assumptions about eligibility and response rates are based on historic data at the local authority level from multiple previous waves of the face-to-face SHS. In order to provide the full number of social interviews (10,450) across Scotland, and to meet the local authority targets as far as possible, the sampling approach was adjusted to reflect the lower anticipated response rate compared to the traditional face to face approach.

The sample was drawn in two batches. Assumptions used to draw the sample for the first 6 months of fieldwork were based on data from the 2020 telephone pilot. These were revised before drawing the sample for the remaining 6 months, based on initial outcomes from the 2021 fieldwork. The revised assumptions also took into account that telephone matching, which was used for the 2020 fieldwork and the first half of 2021, would not be used for the second half of the 2021 fieldwork.

Table 1: Social survey target sample sizes and selected addresses
Local authority Target Sampled
Aberdeen City 357 3,511
Aberdeenshire 372 2,501
Angus 250 2,574
Argyll and Bute 250 2,199
Clackmannanshire 250 2,065
Dumfries and Galloway 250 2,056
Dundee City 250 2,908
East Ayrshire 250 2,224
East Dunbartonshire 250 1,578
East Lothian 250 1,617
East Renfrewshire 250 1,843
Edinburgh City 783 5,255
Na h-Eileanan Siar 250 2,018
Falkirk 250 2,284
Fife 559 4,936
Glasgow City 985 10,180
Highland 362 2,451
Inverclyde 250 2,237
Midlothian 250 2,605
Moray 250 1,605
North Ayrshire 250 2,379
North Lanarkshire 506 5,676
Orkney 250 1,711
Perth and Kinross 250 2,283
Renfrewshire 283 2,530
Scottish Borders 250 1,404
Shetland 250 1,626
South Ayrshire 250 1,873
South Lanarkshire 486 5,001
Stirling 250 1,698
West Dunbartonshire 250 3,407
West Lothian 257 1,929

Telephone matching

A telephone matching exercise was undertaken to help increase the number of completed surveys. This involved matching names and telephone numbers to addresses using publicly available sources, such as the electoral register and the telephone directory. Telephone matching was undertaken for all the the entire 15,400 addresses remaining in the 2020 sample when face-to-face fieldwork was stopped.

Two different suppliers were engaged to undertake the telephone matching. The process involved linking the addresses, sampled from the PAF, with databases they held to attempt to match in a name of someone within the household and then, if possible, a telephone number. Overall, telephone matches was successful for 34% of addresses. As the initial sampling of addresses had assumed a matching rate of 23% (the success rate for the 2020 addresses not worked when the pandemic hit) only a proportion of these telephone number were used.

Table 2: Matching rates – 2021 H1 sample
N %
No match 17,563 66%
Match from Supplier 1 1,863 7%
Match from Supplier 2 4,043 15.2%
Match from both suppliers 3,142 11.8%
Total 26,611 100%

The sample was drawn in two stages – H1 at the start of the fieldwork year, and H2 mid way through the fieldwork year. A review of the data was undertaken prior to the H2 sample being drawn. Overall, while the telephone matched data resulted in a higher response rate than the opt-in only sample, the profile of the achieved sample was further from prevoius estimates, and it was decided to not undertake telephone matching in the second half of the sample.

Physical survey sub-sample

Completion of the physical survey requires that selected households respond to the main social survey and agree to a follow-up visit for the physical survey to be completed. Similar to the social survey, all assumptions underpinning the sampling approach had to be revised. Pre-pandemic, sample targets were set using estimates of conversion rate from household interview to physical survey by local authority.

However, the 2020 Push to telephone/video suggested that owner-occupiers were more likely to take part that those in the private or social rented sectors. Rather than allocating an address to the SHCS before the interview as had been done prevoiusly, sampling was done within the interivew after tenure had been established rather than the selection being done prior to the fieldwork. The social survey interview script routed a certain proportion into the physical survey based on tenure and local authority.

For the first half of the 2021 all non-owner-occupiers were routed into the SHCS and asked to allow a surveyor visit. For owner-occupiers, the sampling fraction differed by local authority.

Table 3: Propoprtion of social interviews routed into the SHCS elements – first half of 2021 fieldwork
Local authority Owner-occupier Non-owner-occupiers
Aberdeen City 35.4% 100%
Aberdeenshire 35.7% 100%
Angus 45.3% 100%
Argyll and Bute 46.5% 100%
Clackmannanshire 45.4% 100%
Dumfries and Galloway 42.3% 100%
Dundee City 44.8% 100%
East Ayrshire 45.6% 100%
East Dunbartonshire 46.9% 100%
East Lothian 47.5% 100%
East Renfrewshire 47.9% 100%
Edinburgh City 36.4% 100%
Na h-Eileanan Siar 45.4% 100%
Falkirk 46.3% 100%
Fife 34.1% 100%
Glasgow City 37.3% 100%
Highland 35.3% 100%
Inverclyde 48.8% 100%
Midlothian 46.1% 100%
Moray 48.6% 100%
North Ayrshire 47.4% 100%
North Lanarkshire 36.9% 100%
Orkney 42.1% 100%
Perth and Kinross 44.5% 100%
Renfrewshire 43.8% 100%
Scottish Borders 48.8% 100%
Shetland 41.6% 100%
South Ayrshire 48.0% 100%
South Lanarkshire 37.4% 100%
Stirling 45.5% 100%
West Dunbartonshire 46.2% 100%
West Lothian 44.6% 100%

The sample fractions were revised in the second half of 2021 based on updated projections of likely number of returns, with a fixed proportion of all tenures being routed into the SHCS elements.

Table 4: Propoprtion of social interviews routed into the SHCS elements – second half of 2021 fieldwork
Local authority Owner-occupier Non-owner-occupiers
Aberdeen City 24.6% 28.4%
Aberdeenshire 26.2% 26.9%
Angus 32.8% 43.4%
Argyll and Bute 35.4% 38.8%
Clackmannanshire 14.8% 54.8%
Dumfries and Galloway 23.5% 64.4%
Dundee City 31.3% 49.1%
East Ayrshire 32.2% 53.4%
East Dunbartonshire 26.1% 29.0%
East Lothian 32.0% 96.9%
East Renfrewshire 39.2% 65.0%
Edinburgh City 14.7% 27.1%
Na h-Eileanan Siar 45.1% 89.2%
Falkirk 22.3% 64.9%
Fife 16.0% 24.6%
Glasgow City 17.7% 8.3%
Highland 16.5% 27.8%
Inverclyde 16.1% 18.2%
Midlothian 39.5% 63.9%
Moray 19.7% 32.3%
North Ayrshire 34.1% 30.9%
North Lanarkshire 27.7% 14.1%
Orkney 34.9% 39.2%
Perth and Kinross 32.2% 49.0%
Renfrewshire 33.4% 1.9%
Scottish Borders 22.3% 57.2%
Shetland 36.2% 40.6%
South Ayrshire 48.8% 35.8%
South Lanarkshire 29.7% 16.2%
Stirling 33.1% 25.0%
West Dunbartonshire 23.7% 49.0%
West Lothian 10.4% 60.5%

Sample selection

The Royal Mail's small user Postcode Address File (PAF) was used as the sample frame for the address selection. The advantages of using the small user PAF are as follows:

  • It has previously been used as the sample frame for Scottish Government surveys, so previously recorded levels of ineligible addresses can be used to inform sample design assumptions.
  • It has excellent coverage of addresses in Scotland.
  • The small user version excludes the majority of businesses.

The PAF does still include a number of ineligible addresses, such as small businesses, second homes, holiday rental accommodation and vacant properties. A review of the previous performance of individual surveys found that they each recorded fairly consistent levels of ineligible address for each local authority. This meant that robust assumptions could be made for the expected levels of ineligible addresses in the sample size calculations.

As the samples for the SHS, SHeS and SCJS have been selected by the Scottish Government since 2012 onwards, addresses selected for any of the surveys are removed from the sample frame for a minimum of 4 years, so that they cannot be re-sampled for another survey. This helps to reduce respondent burden and facilitates the development of the pooled sample.

Systematic random sampling was used to select the addresses from the sample frame, with the addresses ordered by urban-rural classification, SIMD rank and postcode.

A small number of addresses have only one entry in the PAF but contain multiple dwelling units. Such addresses are identified in the PAF by the Multiple Occupancy Indicator (MOI). To ensure that households within MOI addresses had the same probability of selection as other households, the likelihood of selecting the addresses were increased in proportion to the MOI. For addresses flagged as having multiple dwellings in the PAF, the dwelling to interview was randomly selected as part of the sample selection process.

As the survey is intended to collect information both about the structure and characteristics of Scottish homes and about the people who occupy them, the interview has a two-part structure. The respondent for the first part of the interview must be a householder – generally the Highest Income Householder or their spouse or partner[1]. For the second part of the interview, one adult (aged 16+) member of the household is selected at random by the CAPI script[2].

Finally, addresses were grouped into batches for effective fieldwork. This was done by minimising the distance required to visit each address in a batch. Batches were then allocated to a particular fieldwork quarter. All quarters had, as far as possible, the same number of batches in each local authority to help ensure that the fieldwork was carried out throughout the year.

To meet the need for modularisation, all sampled addresses were randomly assigned to one of 12 sub-samples or interview streams, which could be used as the basis for assigning sub-samples of respondents to particular blocks of questions. The published version of the questionnaire indicates where streaming is used.

Households are usually assigned to take part in the physical survey based on streams. However, in 2021, households from all streams of the social survey were asked to take part in the physical survey, with a set proportion from each tenure being asked to take part. This attempted to compensate for the over-representation of owner occupiers in the social survey when it was carried out over the telephone, and the knock-on effect this would have on the tenure profile of the physical survey sample.

Contact

Email: shs@gov.scot

Back to top