Scottish household survey 2017: methodology and fieldwork outcomes

7 Survey Weighting

Summary

• Ideally, a selected sample is a miniature of the population it came from. This should be reflected in the sample being representative with respect to all variables measured in the survey. Unfortunately, this is often not the case. One of the problems is non-response. It may cause some groups to be over- or under-represented within the responding sample. If such problems occur, no reliable conclusions can be drawn from the observed survey data, unless a correction is made to ensure the sample is representative of the population. A commonly applied correction technique is weighting adjustment. It assigns an adjustment weight to each survey respondent.
• Weighting ensures that the sample in the Scottish Household Survey represents the population of Scotland as a whole.
• The procedures for the implementation of the weighting methodology were developed by the Scottish Government working with the Methodology Advisory Service at the Office for National Statistics
• The weighting procedures for the SHS incorporate a selection weighting stage to address the unequal selection probabilities and calibration weighting to correct for non-response bias. Calibration weighting derives weights such that the weighted survey totals match known population totals.
• Three weights were derived for the main section of the 2017 SHS; a household weight; random adult weight; and a random schoolchild weight. Further weights were required for analysis of the travel diary and physical survey sections.
• In the SHS, there are seven weights that can be used but two of these - LA_WT (used for analysis of data about the household and data collected from or about the HIH and spouse in the main SHS sample. This includes all variables asked in the first part of the interview, apart from the questions about the random schoolchild and the random child receiving childcare) and IND_WT (used for analysis of data in derived variables about the random adult or collected from the random adult) are used for most analyses.

7.1 Introduction

This section presents information on the weighting procedures applied to the survey data. From the 2012 SHS onwards, the weighting has been undertaken by the Scottish Government rather than the survey contractor (as had previously been the case), but the methodology applied has been largely consistent with that from previous sweeps of the survey. The procedures for the implementation of the weighting methodology were developed by the Scottish Government working with the Methodology Advisory Service at the Office for National Statistics.^[19]

Weighting procedures for survey data are required to correct for unequal probabilities of selection and variations in response rates from different groups. The weighting procedures for the SHS incorporate a selection weighting stage to address the unequal selection probabilities and calibration weighting to correct for non-response bias. Calibration weighting derives weights such that the weighted survey totals match known population totals. For the 2017 SHS the population totals used were the National Records of Scotland's (NRS) "Mid-2016 Population Estimates Scotland"^[20] and for households the NRS "Estimates of Households and Dwellings in Scotland, 2016" were used.^[21] For the physical survey weighting household total were used from the NRS for households the NRS "Estimates of Households and Dwellings in Scotland, 2017".^[22] To undertake the calibration weighting the ReGenesees Package for R was used and, within this to execute the calibration, a linear distance function was implemented.

Three weights were derived for the main section of the 2017 SHS; a household weight; random adult weight; and a random schoolchild weight. Further weights were required for analysis of the travel diary and physical survey sections. The procedures to calculate the weights are described in the following subsections.

7.2 Household weights

This weight is for use with variables that relate to the household. There were three steps to the household weight:

1) Selection weights

The address selection weights were calculated to compensate for unequal probabilities of selection of addresses in different survey strata. For the SHS there were 32 strata - one for each local authority. The address selection weight for each stratum was calculated as the proportion of Scottish households (from NRS estimates) in the stratum divided by the proportion of the sample in the stratum:

2) Calibrated household weight

The stratum selection weight was applied to the survey data to act as entry weights for the calibration. The execution of the calibration step then modified the entry weights so that the weighted total of all members of responding households matched NRS population totals for age bands and sex within each local authority.

3) Households adjustment

The calibration step ensured that survey totals matched the population totals for local authorities but not the household totals. To make the sample representative of households at local authority level a scaling factor was applied so that the weighted number of households from the sample matched the NRS local authority household weights.

7.3 Random Adult weights

Within responding households a random adult was selected to answer individual questions. There were three stages to the random adult weights:

1) Stratum selection weight

A new stratum selection weight is required for the random adults as the reference population is all adults within the stratum. Also, not all random adults in households that completed the household section responded to the survey. The stratum selection weight was calculated as:

NRS mid-year population estimates were used to calculate the population percentages for each stratum.

2) Adult selection weight

The probability of an adult within a household being selected for the random adult interview was inversely proportional to the number of adults within a household - i.e. in a single adult household the only adult resident must be sampled but in a three adult household each adult only has a one in three chance of being selected. To correct for this unequal probability of selection an adult selection weight equal to the number of adults in the household was applied.

3) Calibrated weight

The stratum selection weight and adult selection weight were multiplied together and applied to the survey data. The execution of the calibration step then modified these combined entry weights so that the weighted total of responding random adults matched NRS adult population totals for age bands and sex within each local authority.

7.4 Random Schoolchild weights

A separate weight was required for information collected about a random schoolchild within responding households. The weighting procedures for the random schoolchild are similar to those for the random adult:

1) Stratum selection weights

Stratum selection weights were calculated as:

NRS mid-year population estimates were used to calculate the population percentages for each stratum.

2) Random schoolchild selection weight

As with the random adult weight, only one child was selected within each household so a selection weight equal to the number of eligible children in the household was required.

3) Calibration weight

Population estimates for the number of schoolchildren resident in each local authority are not available. Population estimates only give estimates in terms of ages and the schools census gives the local authority totals for place of schooling rather than residence. Therefore, the population of schoolchildren was estimated using the survey data by applying the household grossing weight to calculate the total number of pupils in each local authority by age group. The selection weights were then combined and applied to the data before the calibration was run to match the random schoolchild totals to the target populations by age group and local authority.

7.5 Travel Diary weight

The travel diary questions were asked as part of the random adult interview. The travel diary collects information on all travel undertaken on the day prior to interview. Over the fieldwork period significantly fewer interviews took place on Fridays, Saturdays and Sundays when compared to other days.

The working status of respondents was also found to vary across day of response, with disproportionately more adults in full-time employment interviewed at the weekend.

These factors resulted in two stages of rescaling the random adult weights for travel diary analysis:

1) To ensure the travel diary was representative of travel patterns for the week as a whole, the random adult weights were rescaled so that the weighted number of interviews was equal for each day of the week. The scaling factor was given by:

To ensure the travel diary was representative of working status across each day a second scaling factor was derived such that the working status breakdown for each day was equal.

The final travel diary weight was then calculated as:

7.6 Physical survey weight

A subsample of the total SHS sample was allocated to the physical survey. This subsample completed a specific module of the SHS in the main interview and received a visit from a fieldworker to conduct a physical survey of the property. Not all of the properties which completed the interview resulted in a completed physical survey. Therefore, two weights are required for the physical survey, one for the interview and one for the completed physical survey paired with the interview. The derivation for both weights followed exactly the same steps:

1) Selection weights

The address selection weights were calculated to compensate for unequal probabilities of selection of addresses in different survey strata. For the physical there were 32 strata - one for each local authority. The address selection weight for each stratum was calculated as the proportion of Scottish households (from NRS estimates) in the stratum divided by the proportion of eligible selected addresses in the stratum:

2) Calibration

The stratum selection weight was applied to the survey data to act as entry weights for the calibration. The execution of the calibration step then modified the entry weights so that the weighted total of responding households matched:

a) The number of households in each local authority

b) Dwelling age at Scotland level

c) Dwelling type at Scotland level

d) Urban-rural classification at Scotland level

The totals for targets (b) and (c) were generated from the sample itself. For almost all of the addresses in the physical survey sample, even where an interview of physical survey were not completed, a visual inspection of the selected address was conducted to record information on dwelling type and age. The Scotland-level targets were then generated from the frequencies for dwelling age and type from the sample weighted with the selection weight. The following tables show the targets.

Table 7.1: Physical Survey calibration targets

Category	Calibration target
Dwelling type
Detached	553,614
Semi	481,354
Terrace/corner	534,168
Tenement	583,872
Other flat	310,561
Total	2,463,569
Dwelling age
Pre-1919	467,402
1919-1944	290,569
1945-1964	544,384
1965-1982	515,014
1983-2002	393,772
Post 2002	252,428
Total	2,463,569
Urban-Rural
Large urban	882,180
Other urban	858,875
Accessible small towns	225,931
Accessible rural	263,770
Small remote towns	87,918
Remote rural	144,895
Total	2,463,569

7.7 Summary of weighting variables to apply in analysis

In the SHS, there are seven weights that can be used but two of these - LA_WT and IND_WT - are used for most analyses, with the others used for smaller specific subsets of the sample. The table below shows the different types of weights available for the survey.

Table 7.2: Survey weighting variables

	Calibrated weights	Grossing to population estimates
Main sample
Household	LA_WT	LA_GRWT
Random Adult	IND_WT	IND_GRWT
Random Schoolchild	KID_WT	KID_GRWT
Travel Diary	TRAV_WT	-
Physical Survey
Social survey	SWGHT12_N	SWGHT12
Paired social and physical survey	PWGTH12_N	PWGHT12
Household scaling weight	SHSWGHT12_N

• LA_WT is used for analysis of data about the household and data collected from or about the HIH and spouse in the main SHS sample. This includes all variables asked in the first part of the interview, apart from the questions about the random schoolchild and the random child receiving childcare.
• IND_WT which is used for analysis of data in derived variables about the random adult or collected from the random adult. This includes all variables in the second part of the interview.
• KID_WT which is used for analysis of questions related to the random schoolchild - HE10N to HE17 inclusive (see separate Questionnaire publication).
• TRAV_WT, contained in the Travel Diary data, which is used for analysing that data.