Scottish household survey 2017: methodology and fieldwork outcomes

2 Sample Design

Summary

• From 2012 the three Scottish Government interviewer-led population surveys (SHS, Scottish Health Survey (SHeS) and the Scottish Crime and Justice Survey (SCJS) have coordinated sample designs.
• The SHS sample has been designed to allow annual publication of results at Scotland level and for local authorities.
• From 2012, the physical survey of the Scottish House Condition Survey (SHCS) has been incorporated into the SHS.
• Around one third subsample of the main SHS sample has been allocated to the physical survey, which has a required sample size of 3,004 for Scotland and a minimum of 80 for each local authority.
• In order to provide annual local authority results without specifying an excessive overall sample size, the sample was disproportionately stratified by local authority (smaller local authorities have a higher sample proportion relative to their populations than the larger local authorities).
• For the physical survey, the minimum sample size over each rolling three year period for each local authority is 240 giving a minimum of 80 per annum. There is also a minimum annual sample for Scotland of 3,004.
• Samples of the general population exclude prisons, hospitals and military bases.
• The Royal Mail's small user Postcode Address File (PAF) was used as the sample frame for the address selection.
• Addresses selected for any of the surveys (SHS, SHeS, SCJS) are removed from the sample frame for a minimum of 4 years.so that they cannot be re-sampled for another survey. This reduces respondent burden.
• In order to collect information both about the structure and characteristics of Scottish homes and about the people who occupy them,the interview has a two-part structure, generally the Highest Income Householder or their spouse or partner answers the first part, and one adult (aged 16+) member of the household is selected at random by the CAPI script conducts the second part.

2.1 Requirements

The sample for the 2017 Scottish Household Survey (SHS) was designed by the Scottish Government. From 2012 the sample design has been coordinated with the sample designs for the Scottish Health Survey (SHeS) and the Scottish Crime and Justice Survey (SCJS) as part of a survey efficiency project and to allow the samples of the three surveys to be pooled for further analysis^[4].

The SHS sample has been designed to allow annual publication of results at Scotland level and for local authorities. To meet these requirements the target sample size for Scotland was 10,678 household interviews with a minimum local authority target of 258 (West Lothian). From 2012, the physical survey of the Scottish House Condition Survey (SHCS) has been incorporated into the SHS. Around one third subsample of the main SHS sample has been allocated to the physical survey, which has a required sample size of 3,004 for Scotland and a minimum of 80 for each local authority .

2.2 Sample design and assumptions

2.2.1 Main sample

The 2017 Scottish Household Survey has a single-stage unclustered sample design. In order to provide annual local authority results without specifying an excessive overall sample size, the sample was disproportionately stratified by local authority (smaller local authorities have a higher sample proportion relative to their populations than the larger local authorities).

To deliver the required local authority precision the minimum effective sample size for each local authority was set at 250. For local authorities where an effective size sample of 250 would have decreased estimate precision by more than 25 per cent from the previous sweep of the survey the target effective sample size was increased such that the decrease in precision was less than 25 per cent.

In order to estimate the annual target achieved sample size for each local authority, analysis of design effects from the 2007-08 survey was undertaken, since:

As rural areas of local authorities were clustered in the 2007-08 survey, for the 2017 unclustered sample the median design effect from a range of variables for the unclustered parts of local authority samples were assumed for the entire areas in 2017. This allowed the calculation of the target achieved sample size for each local authority, as shown in Table 2.1.

Table 2.1: Target sample sizes and selected addresses

	Main sample		Physical survey sub-sample
Local Authority	Target interviews	Selected addresses	Target interviews	Selected addresses
Aberdeen City	352	657	91	315
Aberdeenshire	335	552	89	248
Angus	263	433	80	220
Argyll and Bute	263	446	80	220
Clackmannanshire	263	386	80	190
Dumfries and Galloway	264	441	80	221
Dundee City	261	429	80	227
East Ayrshire	266	476	80	240
East Dunbartonshire	273	452	80	228
East Lothian	270	474	80	245
East Renfrewshire	269	472	80	246
Edinburgh City	747	1,385	190	683
Falkirk	275	446	80	203
Fife	261	376	80	196
Glasgow City	553	852	137	354
Highland	982	1,863	246	925
Inverclyde	335	579	88	263
Midlothian	278	479	80	231
Moray	267	475	80	243
Na h-Eileanan Siar	265	440	80	215
North Ayrshire	271	431	80	224
North Lanarkshire	477	810	124	380
Orkney	267	390	80	179
Perth and Kinross	267	439	80	225
Renfrewshire	275	447	80	233
Scottish Borders	263	423	80	207
Shetland	274	405	80	193
South Ayrshire	278	484	80	242
South Lanarkshire	459	775	119	367
Stirling	268	413	80	196
West Dunbartonshire	279	467	80	234
West Lothian	258	425	80	235
Scotland	10,678	18,022	3,004	8,828

2.2.2 Physical survey sub-sample

For the physical survey, the minimum sample size over each rolling three year period for each local authority is 240 giving a minimum of 80 per annum. There is also a minimum annual sample for Scotland of 3,004. An iterative approach was taken to allocate the physical surveys across local authorities. Firstly, the overall sample of 3,004 was allocated to local authorities proportionate to the number of occupied dwellings. Where the allocated number of interviews was below 80, the allocation was increased to 80. The remaining sample was then allocated across the local authorities which had an initial allocation of more than 80.

Completion of the physical survey requires that selected households respond to the main social survey and agree to a follow-up visit for the physical survey to be completed. Therefore, in order to achieve the sample targets a conversion rate from household interview to physical survey is required. Prior to 2012 the Scottish House Condition Survey consisted of a similar structure with social interview followed by the physical survey visit. For each local authority, assumptions for conversion from household interview to physical survey were based on the average conversion rate from the three most recent SHCS with information available. Since the physical survey was a module of the SHS for the first time in 2012, the conversion rate for each local authority was reduced by 2 per cent to be conservative. Additional conditions were added to the conversion rate assumptions setting upper and lower limits of 90 per cent and 70 per cent, respectively.

In order to calculate the total number of addresses in the sample to assign to the physical sample, the number of responding households required to yield the physical survey responses is calculated using the conversion rates. The response rate and ineligible address assumptions are then applied.

Table 2.1 shows the target sample size and the number of selected addresses for the main sample and physical survey by local authority.

2.3 Sample selection

The Royal Mail's small user Postcode Address File (PAF) was used as the sample frame for the address selection. The advantages of using the small user PAF are as follows:

• It has previously been used as the sample frame for Scottish Government surveys so previously recorded levels of ineligible addresses can be used to inform assumptions for 2017 sample design;
• It has excellent coverage of addresses in Scotland; and
• The small user version excludes the majority of businesses

The Assessor's Portal which is the council tax list of all dwellings in Scotland was considered as an alternative sample frame but since it had not previously been used as a sample frame for large scale surveys in Scotland there would have been a greater risk attached to assumptions for response rates and ineligible addresses.

The PAF does still include a number of ineligible addresses, such as small businesses, second homes, holiday rental accommodation and vacant properties. A review of the previous performance of individual surveys found that they each recorded fairly consistent levels of ineligible address for each local authority. This meant that robust assumptions could be made for the expected levels of ineligible addresses in the sample size calculations.

As the samples for the SHS, SHeS and SCJS are all being selected by the Scottish Government from 2012 onwards, addresses selected for any of the surveys are removed from the sample frame so that they cannot be re-sampled for another survey. This will help to reduce respondent burden and facilitate the development of the pooled sample. The addresses are removed from the sample frame for a minimum of 4 years.

The sample design specified above was implemented as follows:

1) Systematic random sampling was used to select the addresses from the sample frame with the addresses ordered by urban-rural classification, SIMD rank and postcode.

2) Once the overall sample was selected systematic random sampling was used to select the subsample for the physical survey.

2.3.1 Selecting households at addresses with multiple dwellings

A small number of addresses have only one entry in the PAF but contain multiple dwelling units. Such addresses are identified in the PAF by the Multiple Occupancy Indicator (MOI). To ensure that households within MOI addresses had the same probability of selection as other households, the likelihood of selecting the addresses were increased in proportion to the MOI. For addresses which are flagged as having multiple dwellings in the PAF the dwelling to interview was randomly selected as part of the sample selection process.

Where the MOI is correct, this procedure is unproblematic. Sometimes, however, the MOI is incorrect or missing (in about 2 per cent of cases) and the true number of dwellings at an address is only discovered once the survey is in the field.

Where an interviewer finds that the MOI is different from the actual number of dwellings observed (and there is more than one dwelling) he or she contacts the office where the correct details are used to randomly select one of the dwellings.

2.3.2 Selecting individuals within households

As the survey is intended to collect information both about the structure and characteristics of Scottish homes and about the people who occupy them, the interview has a two-part structure. The respondent for the first part of the interview must be a householder - generally the Highest Income Householder or their spouse or partner^[5]. For the second part of the interview, one adult (aged 16+) member of the household is selected at random by the CAPI script (see section 3.1). If this person is not available at the time, the interviewer will call back to complete the interview at a later date if necessary^[6].

2.3.3 Allocation of sample to different time periods

Finally, addresses were grouped into batches for effective fieldwork. This was done by minimising the distance required to visit each address in a batch. Batches were then allocated to a particular fieldwork quarter. All quarters had, as far as possible, the same number of batches in each local authority to help ensure that the fieldwork was carried out throughout the year.

2.3.4 Allocation of sample to questionnaire modules

To meet the need for modularisation, all sampled addresses were randomly assigned to one of 12 sub-samples or interview streams, which could be used as the basis for assigning samples of respondents to particular blocks of questions.

For example, the Physical Survey module is intended to provide representative data on the physical condition of Scotland's homes and this is achieved by assigning the module to streams 1 to 4. It should be noted that given difficulties in achieving the target number of physical surveys (see section 6.3), a fifth stream (stream nine) was opened up to the physical survey. In other words, households who were in stream nine were asked to participate in the physical survey from quarter 2 onwards.

A series of "social" questions are asked in the Household section of the survey to understand experiences of households, also in streams 1 to 4. This means that the Physical Survey social questions are based on a random sub-sample of 1 in 3 addresses and (assuming no difference in response rates) 1 in 3 interviews will be directed through those questions.

Other smaller blocks of questions are asked of sub-samples at various points in the questionnaire and the published version of the questionnaire indicates where and at what points in time streaming is used.

2.4 Exclusions

Samples of the general population exclude prisons, hospitals and military bases. While prisons and hospitals do not generally have significant numbers of private households, the same may not be true of military bases. These are classified as special enumeration districts (EDs) in the Census and account for just 0.5 per cent of the population. Interviewing on military bases would pose fieldwork problems relating to access and security so they are removed from the PAF before sampling.

The following types of accommodation are excluded from the survey if they are not listed on the Small User file of the PAF:

• nurses' homes;
• student halls of residence;
• other communal establishments (e.g. hostels for the homeless and old people's homes);
• mobile homes; and
• sites for travelling people.

Households in these types of accommodation are included in the survey if they are listed on the Small User file of the PAF and the accommodation represents the sole or main residence of the individuals concerned. People living in bed and breakfast accommodation are similarly included if the accommodation is listed on PAF and represents the sole or main residence of those living there^[7].

Students' term-time addresses are taken as their main residence (in order that they are counted by where they spend most of the year). However, since halls of residence are generally excluded there will be some under-representation of students in the SHS.