Scottish household survey 2018: methodology and fieldwork outcomes

7 Survey Weighting

Summary

• Ideally, a selected sample is a miniature of the population it came from. This should be reflected in the sample being representative with respect to all variables measured in the survey. Unfortunately, this is often not the case. One of the problems is non-response. It may cause some groups to be over- or under-represented within the responding sample. If such problems occur, no reliable conclusions can be drawn from the observed survey data, unless a correction is made to ensure the sample is representative of the population. A commonly applied correction technique is weighting adjustment. It assigns an adjustment weight to each survey respondent.
• Weighting ensures that the sample in the Scottish Household Survey represents the population of Scotland as a whole.
• The procedures for the implementation of the weighting methodology were developed by the Scottish Government working with the Methodology Advisory Service at the Office for National Statistics
• The weighting procedures for the SHS incorporate a selection weighting stage to address the unequal selection probabilities and calibration weighting to correct for non-response bias. Calibration weighting derives weights such that the weighted survey totals match known population totals.
• Three weights were derived for the main section of the 2018 SHS; a household weight; random adult weight; and a random schoolchild weight. Further weights were required for analysis of the travel diary and physical survey sections.
• In the SHS, there are seven weights that can be used but two of these – LA_WT (used for analysis of data about the household and data collected from or about the HIH and spouse in the main SHS sample. This includes all variables asked in the first part of the interview, apart from the questions about the random schoolchild and the random child receiving childcare) and IND_WT (used for analysis of data in derived variables about the random adult or collected from the random adult) are used for most analyses.

7.1 Introduction

This section presents information on the weighting procedures applied to the survey data. From the 2012 SHS onwards, the weighting has been undertaken by the Scottish Government rather than the survey contractor (as had previously been the case), but the methodology applied has been largely consistent with that from previous sweeps of the survey. The procedures for the implementation of the weighting methodology were developed by the Scottish Government working with the Methodology Advisory Service at the Office for National Statistics.^[23]

Weighting procedures for survey data are required to correct for unequal probabilities of selection and variations in response rates from different groups. The weighting procedures for the SHS incorporate a selection weighting stage to address the unequal selection probabilities and calibration weighting to correct for non-response bias. Calibration weighting derives weights such that the weighted survey totals match known population totals. For the 2018 SHS the population totals used were the National Records of Scotland's (NRS) "Mid-2017 population estimates Scotland"^[24] and for households the NRS "Estimates of Households and Dwellings in Scotland, 2017" were used.^[25] For the physical survey weighting household totals were used from the NRS "Estimates of Households and Dwellings in Scotland, 2018".^[26] To undertake the calibration weighting the ReGenesees Package for R was used and, within this to execute the calibration, a linear distance function was implemented.

Three weights were derived for the main section of the 2018 SHS; a household weight; random adult weight; and a random schoolchild weight. Further weights were required for analysis of the travel diary and physical survey sections. The procedures to calculate the weights are described in the following subsections.

7.2 Household weights

This weight is for use with variables that relate to the household. There were three steps to the household weight:

1) Selection weights

The address selection weights were calculated to compensate for unequal probabilities of selection of addresses in different survey strata. For the SHS there were 32 strata – one for each local authority. The address selection weight for each stratum was calculated as the proportion of Scottish households (from NRS estimates) in the stratum divided by the proportion of the sample in the stratum:

2) Calibrated household weight

The stratum selection weight was applied to the survey data to act as entry weights for the calibration. The execution of the calibration step then modified the entry weights so that the weighted total of all members of responding households matched NRS population totals for age bands and sex within each local authority.

3) Households adjustment

The calibration step ensured that survey totals matched the population totals for local authorities but not the household totals. To make the sample representative of households at local authority level a scaling factor was applied so that the weighted number of households from the sample matched the NRS local authority household weights.

7.3 Random Adult weights

Within responding households a random adult was selected to answer individual questions. There were three stages to the random adult weights:

1) Stratum selection weight

A new stratum selection weight is required for the random adults as the reference population is all adults within the stratum. Also, not all random adults in households that completed the household section responded to the survey. The stratum selection weight was calculated as:

NRS mid-year population estimates were used to calculate the population percentages for each stratum.

2) Adult selection weight

The probability of an adult within a household being selected for the random adult interview was inversely proportional to the number of adults within a household – i.e. in a single adult household the only adult resident must be sampled but in a three adult household each adult only has a one in three chance of being selected. To correct for this unequal probability of selection an adult selection weight equal to the number of adults in the household was applied.

3) Calibrated weight

The stratum selection weight and adult selection weight were multiplied together and applied to the survey data. The execution of the calibration step then modified these combined entry weights so that the weighted total of responding random adults matched NRS adult population totals for age bands and sex within each local authority.

7.4 Random Schoolchild weights

A separate weight was required for information collected about a random schoolchild within responding households. The weighting procedures for the random schoolchild are similar to those for the random adult:

1) Stratum selection weights

Stratum selection weights were calculated as:

NRS mid-year population estimates were used to calculate the population percentages for each stratum.

2) Random schoolchild selection weight

As with the random adult weight, only one child was selected within each household so a selection weight equal to the number of eligible children in the household was required.

3) Calibration weight

Population estimates for the number of schoolchildren resident in each local authority are not available. Population estimates only give estimates in terms of ages and the school census gives the local authority totals for place of schooling rather than residence. Therefore, the population of schoolchildren was estimated using the survey data by applying the household grossing weight to calculate the total number of pupils in each local authority by age group. The selection weights were then combined and applied to the data before the calibration was run to match the random schoolchild totals to the target populations by age group and local authority.

7.5 Travel Diary weight

The travel diary questions were asked as part of the random adult interview. The travel diary collects information on all travel undertaken on the day prior to interview. Over the fieldwork period significantly fewer interviews took place on Fridays, Saturdays and Sundays when compared to other days.

The working status of respondents was also found to vary across day of response, with disproportionately more adults in full-time employment interviewed at the weekend.

These factors resulted in two stages of rescaling the random adult weights for travel diary analysis:

1) To ensure the travel diary was representative of travel patterns for the week as a whole, the random adult weights were rescaled so that the weighted number of interviews was equal for each day of the week. The scaling factor was given by:

To ensure the travel diary was representative of working status across each day a second scaling factor was derived such that the working status breakdown for each day was equal.

The final travel diary weight was then calculated as:

7.6 Physical survey weight

A subsample of the total SHS sample was allocated to the physical survey. This subsample completed a specific module of the SHS in the main interview and received a visit from a fieldworker to conduct a physical survey of the property. Not all of the properties which completed the interview resulted in a completed physical survey. Therefore, two weights are required for the physical survey, one for the interview and one for the completed physical survey paired with the interview. The derivation for both weights followed exactly the same steps:

1) Selection weights

The address selection weights were calculated to compensate for unequal probabilities of selection of addresses in different survey strata. For the physical there were 32 strata – one for each local authority. The address selection weight for each stratum was calculated as the proportion of Scottish households (from NRS estimates) in the stratum divided by the proportion of eligible selected addresses in the stratum:

2) Calibration

The stratum selection weight was applied to the survey data to act as entry weights for the calibration. The execution of the calibration step then modified the entry weights so that the weighted total of responding households matched:

a) The number of households in each local authority

b) Dwelling age at Scotland level

c) Dwelling type at Scotland level

d) Urban-rural classification at Scotland level

The totals for targets (b) and (c) were generated from the sample itself. For almost all of the addresses in the physical survey sample, even where an interview or physical survey were not completed, a visual inspection of the selected address was conducted to record information on dwelling type and age. The Scotland-level targets were then generated from the frequencies for dwelling age and type from the sample weighted with the selection weight. The targets are shown in Table 7.1.

Table 7.1: Physical Survey calibration targets

Category	Calibration target
Dwelling type
Detached	551,998
Semi	504,519
Terrace/corner	531,569
Tenement	575,729
Other flat	313,459
Total	2,477,275
Dwelling age
Pre-1919	469,477
1919-1944	281,424
1945-1964	530,362
1965-1982	529,247
1983-2002	401,891
Post 2002	264,874
Total	2,477,275
Urban-Rural
Large urban	884,276
Other urban	862,606
Accessible small towns	227,374
Accessible rural	266,835
Small remote towns	88,884
Remote rural	147,299
Total	2,477,275

7.7 Summary of weighting variables to apply in analysis

In the SHS, there are seven weights that can be used but two of these – LA_WT and IND_WT – are used for most analyses, with the others used for smaller specific subsets of the sample. The table below shows the different types of weights available for the survey.

Table 7.2: Survey weighting variables

	Calibrated weights	Grossing to population estimates
Main sample
Household	LA_WT	LA_GRWT
Random Adult	IND_WT	IND_GRWT
Random Schoolchild	KID_WT	KID_GRWT
Travel Diary	TRAV_WT	-
Physical Survey
Social survey	SWGHT12_N	SWGHT12
Paired social and physical survey	PWGTH12_N	PWGHT12
Household scaling weight	SHSWGHT12_N

• LA_WT is used for analysis of data about the household and data collected from or about the HIH and spouse in the main SHS sample. This includes all variables asked in the first part of the interview, apart from the questions about the random schoolchild and the random child receiving childcare.
• IND_WT which is used for analysis of data in derived variables about the random adult or collected from the random adult. This includes all variables in the second part of the interview.
• KID_WT which is used for analysis of questions related to the random schoolchild (HE1 to HE13N inclusive) (see separate Questionnaire publication).
• TRAV_WT, contained in the Travel Diary data, which is used for analysing that data.