Scottish Household Survey 2022: methodology and fieldwork outcomes

Weighting

Introduction

Weighting aims to ensure that the Scottish Household Survey results represent the population of Scotland as a whole. The procedures for the implementation of the weighting methodology were developed by the Scottish Government working with the Methodology Advisory Service at the Office for National Statistics. The weighting procedures for the SHS incorporate:

• Selection weighting to address unequal selection probabilities resulting from the sampling methods.
• Calibration weighting to correct for non-response bias. Calibration weighting derives weights such that the weighted survey totals match known population totals. The most recent available Scotland-level estimates for population and households were used. For the 2022 SHS, the population totals used were the National Records of Scotland's (NRS) Mid-2021 Population Estimates Scotland. For households, the NRS Households and Dwellings in Scotland, 2022 estimates were used and these were also used for the physical survey.
• To undertake the calibration weighting the ReGenesees Package for R was used. Within this, to execute the calibration, a linear distance function was implemented.

Resumption of historic weighting method

Up to and including the 2019 SHS, results had been weighted in broadly the same way. However, the shift from face-to-face to remote interviewing in 2020 as a response to Covid-19, an approach continued in 2021, was associated with changes in the profile of the achieved sample that were unlikely to reflect real changes in Scotland's population. As a result, alternative weighting methods to those used pre-2020 were developed that incorporated SIMD quintiles and 6-fold-urban/rual demographics as part of the calibration process. The methodology for this is detailed in full in the 2020 methodology report.

In 2022 face-to-face interviews resumed, and the weighting approach used was the historic weighting approach from 2019 and earlier. Three weights were derived for the main section of the 2022 SHS a household weight; random adult wieght; and a random schoolchild weight. Further weights were required for analysis of the travel diary and physical survey sections. The procedures to calculate the weights are described in the following subsections.

Household weights

This weight is for use with variables that relate to the household. There were three steps to creating the household weight:

1) Selection weights

The address selection weights were calculated to compensate for unequal probabilities of selection of addresses in different survey strata. For the SHS there were 32 strata – one for each local authority. The address selection weight for each stratum was calculated as the proportion of Scottish households (from NRS estimates) in the stratum, divided by the proportion of all responding addresses in the stratum.

2) Calibrated household weight

The stratum selection weight was applied to the survey data to act as entry weights for the calibration. The execution of the calibration step then modified the entry weights so that the weighted total of all members of responding households matched NRS population totals for age bands and sex within each local authority.

3) Households adjustment

The calibration step ensured that survey totals matched the population totals for local authorities but not the household totals. To make the sample representative of households at local authority level, a scaling factor was applied so that the weighted number of households from the sample matched the NRS local authority household estimates.

Random adult weights

Within responding households a random adult was selected to answer individual questions. There were three stages to creating the random adult weights:

1) Stratum selection weight

A new stratum selection weight is required for the random adults as the reference population is all adults within the stratum. Also, not all random adults in households that completed the household section responded to the survey. The stratum selection weight was calculated as the proportion of Scottish adults in the stratum (from NRS estimates) divided by the proportion of all responding adults in the stratum.

2) Adult selection weight

The probability of an adult within a household being selected for the random adult interview was inversely proportional to the number of adults within a household – i.e. in a single adult household the only adult resident must be sampled but in a three adult household each adult only has a one in three chance of being selected. To correct for this unequal probability of selection an adult selection weight equal to the number of adults in the household was applied.

3) Calibrated weight

The stratum selection weight and adult selection weight were multiplied together and applied to the survey data. The execution of the calibration step then modified these combined entry weights so that the weighted total of responding random adults matched NRS adult population totals for age bands and sex within each local authority.

Random schoolchild weights

A separate weight was required for information collected about a random schoolchild within responding households. The weighting procedures for the random schoolchild were similar to those for the random adult:

1) Stratum selection weights

Stratum selection weights were calculated as the proportion of Scottish school age children in the stratum (from NRS estimates), divided by the proportion of all responding school children in the stratum.

2) Random schoolchild selection weight

As with the random adult weight, only one child was selected within each household so a selection weight equal to the number of eligible children in the household was required.

3) Calibration weight

Population estimates for the number of schoolchildren resident in each local authority are not available. Available population estimates are for child age, and the school census gives the local authority totals for place of schooling rather than residence. Therefore, the population of schoolchildren was estimated using the survey data by applying the household grossing weight to calculate the total number of pupils in each local authority by age group. The selection weights were then combined and applied to the data before the calibration was run to match the random schoolchild totals to the target populations by age group and local authority.

Travel diary weight

The travel diary questions were asked as part of the random adult interview. The travel diary collects information on all travel undertaken on the day prior to interview. Over the fieldwork period significantly fewer interviews took place on Fridays, Saturdays and Sundays when compared to other days.

The working status of respondents was also found to vary across day of response, with disproportionately more adults in full-time employment interviewed at the weekend.

These factors resulted in two stages of rescaling the random adult weights for travel diary analysis:

1) To ensure the travel diary was representative of travel patterns for the week as a whole, the random adult weights were rescaled so that the weighted number of interviews was equal for each day of the week. The scaling factor was given by dividing the total number of interviews by 7, and then dividing by the weighted (using the random adult weight) total interviews on that day:

To ensure the travel diary was representative of working status across each day, a second scaling factor was derived such that the working status breakdown for each day was equal.

The scaling factor was given by dividing the total number of responses for each working status by 7, and then dividing by the weighted (using the random adult weight) total interviews for that working status on that day:

The final travel diary weight was then calculated as the random adult weight multiplied by the day scale factor multiplied by the day and working status factor:

Physical survey weight

A subsample of the total SHS sample was allocated to the physical survey. This subsample completed a specific module of the SHS in the main interview and received a visit from a fieldworker to conduct a physical survey of the property. Not all of the properties which completed the interview resulted in a completed physical survey. Previously, two weights were required for the physical survey, one for the interview and one for the completed physical survey paired with the interview. In 2022, there was a change to the fieldwork which means the interview weight cannot be produced. The derivation for the paired physical survey and social interview weight followed these steps:

1) Selection weights

The address selection weights were calculated to compensate for unequal probabilities of selection of addresses in different survey strata. Usually, for the physical survey there were 32 strata – one for each local authority. The address selection weight for each stratum is calculated as the proportion of Scottish households (from NRS estimates) in the stratum divided by the proportion of eligible selected addresses in the stratum.

2) Calibration

The stratum selection weight was applied to the survey data to act as entry weights for the calibration. The execution of the calibration step then modified the entry weights so that the weighted total of responding households matched:

a) The number of households in each local authority

b) Dwelling age at Scotland level

c) Dwelling type at Scotland level

d) Urban-rural classification at Scotland level

The totals for targets (b) and (c) were generated from the sample itself. For almost all of the addresses in the physical survey sample, even where an interview or physical survey were not completed, a visual inspection of the selected address was conducted to record information on dwelling type and age. In previous years a visual inspection for almost all of the selected addresses in the physical survey sample was conducted to record information on dwelling type and age. This visual inspection happened even when an interview or physical survey was not completed. In 2022, this visual inspection was only conducted on a subset of the physical survey sample. The Scotland-level targets were then generated from the frequencies for dwelling age and type from the sample weighted with the selection weight.