Sample design

Requirements

The sample for the 2023 Scottish Household Survey (SHS) was designed by the Scottish Government. From 2012, the sample design has been coordinated with the sample designs for the Scottish Health Survey (SHeS) and the Scottish Crime and Justice Survey (SCJS), as part of a survey efficiency project to allow the samples of the three surveys to be pooled for further analysis.

The SHS sample has been designed to allow annual publication of results at Scotland level and for local authorities. To meet these requirements, the target sample size for Scotland was 10,450 household interviews with a minimum local authority target of 250. From 2012, the physical survey of the Scottish House Condition Survey (SHCS) has been incorporated into the SHS. The SHCS has a required sample size of 3,004 for Scotland and a minimum of 80 for each local authority.

Sample design and assumptions

The Scottish Household Survey has a single-stage unclustered sample design. In order to provide annual local authority results without specifying an excessive overall sample size, the sample was disproportionately stratified by local authority (smaller local authorities have a higher sample proportion relative to their populations than the larger local authorities).

To deliver the required local authority precision, the minimum sample size for each local authority was set at 250. In order to estimate the annual target achieved sample size for each local authority, analysis of design effects from the 2007-08 survey was undertaken, since the effective sample size is equal to the achieved sample size divided by the design effect.

As rural areas of local authorities were clustered in the 2007-08 survey, for the 2023 unclustered sample the median design effect from a range of variables for the unclustered parts of local authority samples were assumed for the entire areas in 2023. This allowed the calculation of the target achieved sample size for each local authority, as shown in Table 1.

Normally, the sampling assumptions about eligibility and response rates are based on historic data at the local authority level from multiple previous waves of the face-to-face SHS. In order to provide the full number of social interviews (10,450) across Scotland, and to meet the local authority targets as far as possible, the sampling approach reflected lower anticipated response rates than before the pandemic.

For the 2023 wave of the survey, the initial response rate was higher than expected. As a result, batches of addresses were removed at random from the sample in local authorities where it was predicted that interviewers would over-achieve against targets. 

Table 1: Social survey targets and sample sizes

Physical survey sub-sample

For the physical survey, the minimum target sample size over each rolling three year period for each local authority is 240 giving a minimum of 80 per annum. There is also a minimum annual target sample for Scotland of 3,004. An iterative approach was taken to allocate the physical surveys across local authorities.To determine target sample sizes each local authority is first allocated a minimum sample of 80. After these are deducted from the total overall sample of 3,004 the remaining sample is allocated to local authorities proportionate to the number of occupied dwellings in each local authority.

Completion of the physical survey requires that selected households respond to the main social survey and agree to a follow-up visit for the physical survey to be completed. Therefore, in order to achieve the sample targets a conversion rate from household interview to physical survey is required. For each local authority, assumptions for conversion from household interview to physical survey were based on the average conversion rate from the three most recent SHCS with information available. Additional conditions were added to the conversion rate assumptions setting upper and lower limits of 90 per cent and 60 per cent, respectively.

In order to calculate the total number of addresses in the sample to assign to the physical sample, the number of responding households required to yield the physical survey responses is calculated using the conversion rates. The response rate and ineligible address assumptions are then applied.

Table 2: Physical survey targets and sample sizes

Sample selection

The Royal Mail’s small user Postcode Address File (PAF) was used as the sample frame for the address selection. The advantages of using the small user PAF are as follows:

• it has previously been used as the sample frame for Scottish Government surveys, so previously recorded levels of ineligible addresses can be used to inform sample design assumptions.
• it has excellent coverage of addresses in Scotland.
• the small user version excludes the majority of businesses.

The PAF does still include a number of ineligible addresses, such as small businesses, second homes, holiday rental accommodation and vacant properties. A review of the previous performance of individual surveys found that they each recorded fairly consistent levels of ineligible address for each local authority. This meant that robust assumptions could be made for the expected levels of ineligible addresses in the sample size calculations.

As the samples for the SHS, SHeS and SCJS have been selected by the Scottish Government since 2012 onwards, addresses selected for any of the surveys are removed from the sample frame for a minimum of 4 years, so that they cannot be re-sampled for another survey. This helps to reduce respondent burden and facilitates the development of the pooled sample.

Systematic random sampling was used to select the addresses from the sample frame, with the addresses ordered by urban-rural classification, SIMD rank and postcode.

A small number of addresses have only one entry in the PAF but contain multiple dwelling units. Such addresses are identified in the PAF by the Multiple Occupancy Indicator (MOI). To ensure that households within MOI addresses had the same probability of selection as other households, the likelihood of selecting the addresses were increased in proportion to the MOI. For addresses flagged as having multiple dwellings in the PAF, the dwelling to interview was randomly selected as part of the sample selection process.

Where the MOI is correct, this procedure is unproblematic. Sometimes, however, the MOI is incorrect or missing (in about 2% of cases) and the true number of dwellings at an address is discovered only once the survey is in the field.

Where an interviewer finds that the MOI is different from the actual number of dwellings observed (and there is more than one dwelling), he or she contacts the office, where the correct details are used to randomly select one of the dwellings.

As the survey is intended to collect information both about the structure and characteristics of Scottish homes and about the people who occupy them, the interview has a two-part structure. The respondent for the first part of the interview must be a person in whose name the accommodation is owned or rented or who is otherwise responsible for the accommodation – generally the Highest Income Householder or, where the income of householders is the same, the oldest householder. For the second part of the interview, one adult (aged 16+) member of the household (who is normally resident during term time and/or has not been living outside the household for 6 months or more) is selected at random by the CAPI script.

Finally, addresses were grouped into batches for effective fieldwork. This was done by minimising the distance required to visit each address in a batch. Batches were then allocated to a particular fieldwork quarter. All quarters had, as far as possible, the same number of batches in each local authority to help ensure that the fieldwork was carried out throughout the year.

To meet the need for modularisation, all sampled addresses were randomly assigned to one of 12 sub-samples or interview streams, which could be used as the basis for assigning sub-samples of respondents to particular blocks of questions. The published version of the questionnaire indicates where streaming is used.