Long term survey strategy: mixed mode research report

4. Representation: coverage and sampling

Introduction

Representation, in the context of social surveys, refers to the extent to which the survey sample (typically of people or households) accurately reflects the characteristics of the wider population the study is designed to capture (the target population). Ensuring your survey allows for representative analysis of your target population is vital for ensuring it is possible to generalise your findings.

The three Scottish Government surveys that are the focus of this report are all general population surveys. As such, their approach to sampling is designed to ensure that the profile of the achieved sample represents the actual profile of the Scottish population as accurately as possible. More specifically:

• The SHS is intended to accurately reflect the profile of private households in Scotland (via the Household element of the questionnaire), adults (aged 16+) living in private households in Scotland (via the random adult element), and private residential properties in Scotland (via the physical survey).
• The SCJS is intended to reflect the profile of all adults (aged 16+) living in private households in Scotland
• SHeS is intended to reflect the profile of both adults and children under 16 living in private households in Scotland.

This chapter and the following chapter discuss the different components that determine how representative any sample is, and how changing or mixing modes may impact on this. The focus of this chapter is on issues relating to coverage and sampling, while the following chapter focuses on nonresponse and adjustment (see Total Survey Error framework, discussed in chapter 3).

Coverage

Coverage error occurs when there is a mismatch between the target population and the sample frame. This section describes this source of error, including in the context of mixed-mode approaches.

Sample frame

The first element that impacts on representation is the data already available about the target population, from which sample can be drawn. Undercoverage occurs when the sample frame is missing units of the target population, while overcoverage occurs when units outside the target population, or duplicated units of the target population, are included in the frame. Undercoverage can introduce coverage error while overcoverage typically adds to costs and introduces extra uncertainty regarding achieved sample sizes (but does not introduce error so long as sample units outside the target population can be identified as such). When considering mixing modes, it is important that the survey design – in particular, the contact strategy adopted for inviting people to take part – does not limit the sample frame options to one with large-scale coverage issues that cannot be adjusted for.

In some countries (e.g. Denmark, Sweden, Norway, the Netherlands, Japan and Israel) population registers^[23] are available, with all citizens or residents in a country required to register with local authorities so that they can maintain an accurate record of the population. These registers can then be used for sampling for government surveys. No such register exists in Scotland or the UK, however. As a result, almost all large-scale, random probability general population surveys in Scotland and the UK, including the three that are the focus of this study, rely on the Royal Mail’s Postcode Address File (PAF).

PAF has very good coverage of addresses, but it does exclude the non-household population, including homeless people. It is also necessary to build in selection rules to correct for the fact that some individuals could be overrepresented (e.g. second-home owners) and others underrepresented (e.g. those living in a house listed as a single address, but which has actually been split into independent flats). These omissions were noted by Scottish Government survey stakeholders as limitations on the three surveys that are the focus of this report. On the SCJS in particular, it was noted that the limitation of the survey to adults in private households meant there were key missing sub-groups whose experience of crime may be very different to the rest of the population – including students, children, prisoners, and homeless people.

An alternative address-based sample frame for general population surveys that could, potentially, be used in the future for Scottish Government surveys is the Scottish Address Directory (SAD). The SAD is designed to provide a more up to date and accurate address directory using the Ordnance Survey AddressBase and additional information from other Geography products.^[24] AddressBase Premium is being used by the Transformed Labour Force Survey as a sample frame and is generally considered to be more comprehensive and accurate than PAF, as it is compiled from a variety of sources, including local authority address gazetteers, Ordnance Survey data, and Royal Mail data. PAF, on the other hand, is compiled solely from Royal Mail data. Because of this AddressBase Premium is more likely to include new build properties and to have more accurate address details. It is also easier to identify communal establishment for exclusion from the issued sample. SAD may be expected to be similarly more accurate than PAF, given it uses AddressBase (and other sources).

The Community Health Index (CHI) is used on SHeS in conjunction with the PAF sample, to identify addresses that contain children for the child boost. This has reduced costs, improved response and reduced the field capacity required to achieve interviews with eligible households, and the scope to use this further could potentially be explored.

For telephone surveys, a sample frame that has been used in the past (including the 2004 telephone Scottish Crime and Victimisation survey) is a list of telephone numbers generated via Random Digit Dial (RDD). RDD allows researchers to create a sample frame, using area codes to ensure representation of key geographic areas and the random generation of landline numbers from this. However, RDD-based samples now suffer from very significant under-coverage issues, as landlines are far less common than they used to be, particularly among younger households.^[25] These are therefore much less commonly considered as an option for large-scale government surveys. There is no comprehensive list of all mobile numbers which can be used in conjunction with an RDD landline sample. Neither is there any comprehensive list of email addresses for the Scottish population (which would, in any case, exclude those who do not have an email address).

The ONS has been exploring developing a ‘population spine’ for the UK, based on integrating various administrative data sources to create a large, integrated dataset on the UK population. One of the uses originally envisaged for this population spine is “as a survey frame for selecting a sample for social surveys”, including potentially “increased sample inclusion of sub-populations”.^[26] At the time of writing, the primary focus to date appears to have been on developing this approach to produce population estimates from administrative data rather than use for survey sampling. Moreover, so far work on developing a ‘demographic index’ which can provide population statistics appears to have focused primarily on England-only data. It is worth noting, however, that the Survey Futures project, funded by the Economic and Social Research Council, has recently announced funding for two projects that aim to shed light on what may be possible in terms of alternative sample frames (including those built from administrative data) for general population surveys in the future.^[27]

At present though, either PAF or another address-based sample frame remain the primary options for probability-based general population surveys in Scotland. However, address-based sample frames do not include names, telephone numbers or email addresses. This means that, in order to use them to support sampling for surveys by modes other than face-to-face or paper (by post), further steps are required. These include:

• Matching telephone numbers collected via other sources to address-based samples. This approach was used on the SHS during the pandemic, when face-to-face fieldwork was suspended. Two different suppliers were engaged to try to match a telephone number, using publicly available sources such as the electoral register and the telephone directory, to the 15,400 remaining 2020 addresses in the sample at the point face-to-face fieldwork stopped. However, they were only able to find a match for 23% of the sample. Moreover, households where a match was obtained were more affluent, rural, and household members tended to be older, meaning that the telephone matched sample suffered from significant sample bias.^[28]
• Postal push-to-web or push-to-telephone approaches. This is the most common approach across the government and other large scale social surveys reviewed for this study that have moved to web or telephone as a primary mode (see case studies in Annex A). It was used on both the SHS and SHeS on their telephone surveys during the pandemic period when face-to-face fieldwork was suspended. The sample is drawn from the PAF, and sampled addresses are written to and invited to take part in the survey online or by telephone. For online surveys, they can simply be sent a unique link (or links) to the survey (linked to their address), often including a QR code that can be scanned to take them to the survey directly. For telephone surveys, respondents need to ‘opt in’ and share their telephone number to take part – the most common options include registering their phone number via a portal, emailing, or phoning a helpline to leave their details.
• A ‘knock to nudge’ approach, whereby a face-to-face interviewer calls at the address in person, but rather than conducing the interview they ‘nudge’ them to take part online or by phone (usually collecting contact details to enable telephone interviewers to ring them and/or to be able to send reminders by email). While a number of government surveys – including SHeS – used a ‘knock to nudge’ approach while face-to-face fieldwork was suspended because of Covid-19 related restrictions, it appears to be less commonly used now. This is largely because it is expensive to send interviewers to call in person, so where surveys do so they generally also try to conduct the interview rather than simply nudging respondents to complete it by another mode. A notable exception to this is the Transformed Labour Force Survey, which continues to use an adaptive knock to nudge strategy, where interviewers call on non-responders to encourage them to complete the survey online or by phone.

Mode-specific coverage error

Coverage error arises due to properties of the sampling frame, and so is not affected by the mode(s) of data collection used. However, an exception to this is where the sample frame is chosen based on the mode of survey administration, as in the case of a list of telephone numbers for a telephone survey, or a list of email addresses for a web survey, each of which may possess coverage error to varying extents. Whether or not particular modes of administration themselves exclude particular groups of the population is a separate issue from coverage and is discussed further in chapter 5 (Nonresponse).

Sampling

Sample design

Once a sample frame is selected, a sample must be drawn from this (since for general population surveys, it is not practical or affordable to take a census approach). The approach taken to sample design also has implications for how representative a survey is.

There are two broad types of sampling approach on surveys: probability sampling and non-probability sampling. In probability sampling, all units of the population have a known chance, greater than zero, of being selected for the sample. Non-probability sampling uses a less stringent approach, where the chance of each unit being selected is unknown. In non-probability sampling, quotas are used to match the sample to population estimates for certain characteristics such as gender, age, and working status. It is assumed that if the sample is representative in terms of these chosen characteristics, it will also be representative in terms of the population parameters the survey seeks to measure.

Research has tended to find that probability surveys produce more representative samples and more accurate survey estimates than their non-probability counterparts (e.g. Cornesse & Bosnjak, 2018; Chang & Krosnick, 2009; Yeager et al., 2011; MacInnis et al. 2018). Government surveys in the UK that are used to produce National Statistics are all based on probability sample designs. This excludes certain options for mixed mode surveys – such as including a web element relying on an ‘open’ link (where it is not possible to measure the likelihood of individual members of the target population participating). However, probability sample approaches are possible for all mixed mode surveys, as long as they start with a sampling frame with high quality coverage, as discussed above. Mixing modes does, however, introduce a number of specific considerations around within household sample selection, sample boosts, and sample clustering.

Within household selection

The PAF is a list of addresses, not households or people. As such, surveys based on PAF require additional steps to select a household at addresses with multiple dwelling units, and then individuals within households. Doing so in a robust way is important as evidence suggests that those who are most likely to take part in a survey without controls on selection are likely to be older and are more likely to be those most interested in the topic (e.g. Cornick et al, 2022).

On the three Scottish Government surveys, all three have similar procedures for selecting dwelling units when the interviewer finds more than one dwelling unit at an address.^[29] However, they differ in terms of the selection of individuals within households:

• The SCJS requires the selection of a single adult within the household to take part. This is done via random selection with the interviewer collecting the names or initials of all adult household members and a single adult selected either using a Kish grid or a random number algorithm in the CAPI script.^[30]
• The SHS requires data collection from a ‘householder’, so the interviewer is required to identify the Highest Income Householder or their spouse or partner to complete the first part of the interview. For the second part of the interview, one adult is selected at random by the CAPI script.^[31]
• On SHeS, all adults in each household are eligible for inclusion. To ease respondent burden, up to two children per household are interviewed for both the main and child boost sample.^[32]

For surveys conducted face-to-face, selection processes for households and individuals, like those applied on the three main Scottish Government general population surveys, are fairly straightforward to implement, as they can be completed by a trained interviewer on the doorstep. A similar approach (including using an algorithm to select a respondent from the initials of all household members) can also be applied at first contact by telephone. There was no change in the proportion of household where an interview was achieved with the random adult in the push to telephone approach on the SHS compared with the pre-Covid wave.

However, for self-complete surveys (web or paper), selection processes for individuals within households especially are more complicated, as it requires the participants to follow instructions without an interviewer to support, or to hold them to account. Experiments with next/last birthday, or other approaches to quasi-random selection, have been conducted, but show that participants often do not follow instructions. For example, the Community Life Survey found a compliance rate of only 35% when using the next/last birthday approach (Williams, no date). This is particularly relevant to the SHS and SCJS, as both involve selection of a random adult within the household. As the aim on SHeS is to interview all adults in the household, the issue for that survey is not how to select but how to motivate multiple household members to participate.

Some web-first surveys, such as the Fundamental Rights Survey, use a two-stage approach to selection of a single individual within households, where a first participant completes a household grid, a random selection is applied within the script, and then the selected participant is invited to complete the full survey (Cleary et al, 2018). However, the approach risks drop-off between sections, and is also likely to lead to non-compliance, especially if an incentive is attached.

Given these challenges, many surveys that use a push-to-web design apply a two-per-household, or all-adults approach, where two or more links to the survey are included in the invitation letter, to allow multiple adults to participate.^[33] This approach was used across a number of the large-scale probability surveys reviewed for this study, including Active Lives, British Social Attitudes, Food and You 2, and the Participation Survey (see case studies in Annex A). Allowing multiple respondents within the same household creates some in-home clustering (since individuals from the same household are more likely to share characteristics, views, and experiences), which needs to be accounted for in weighting. However, it also means that for the majority of participants, no selection is required (as most households have fewer than two adults). An all-adults approach allows for more responses to be received per invitation. However, a two-adults approach reduces the risk of fraud, where additional participants are created to receive additional incentives (although it does not remove it, since there is still scope for a single participant to complete the survey twice to receive an additional incentive). Experiments on the Community Life Survey showed that allowing more than one household member to participate resulted in the most robust responses, as well as being more cost and resource efficient (Williams, 2019).

Sample boosts

Boosted random sampling may be used on probability surveys to increase the numbers of respondents from particular population sub-groups of interest, to ensure that there are enough of these groups in the sample to allow or separate analysis. The SHS, SCJS and SHeS all involve disproportionate sampling to ensure sufficient respondents at particular geographies (e.g. local authority or Health Board) over a given survey period. There was considerable interest among survey stakeholders interviewed for this study in increasing geographic sub-samples further across all three surveys – discussed further in chapter 10, which looks at resource implications and the scope to boost sample size for a given budget by changing or mixing mode. There was also interest among stakeholders interviewed for this survey in boosting various groups across the three surveys, including:

• Victims of crime (SCJS)
• Those with low incidence health conditions (e.g. psychosis)
• Those in particular tenures (SHS/SHCS)
• Those in particular equality groups (e.g. disabled people, minority ethnic groups, or LGBTQ+ people).

On the assumption that, whatever mode of data collection is employed, surveys continued to be based on address-based sample frames (like PAF), boosting geographic sub-samples is relatively straightforward, regardless of mode. However, boosting for characteristics that are not straightforwardly linked to addresses is more complicated. In this case, it is likely to be necessary to screen respondents.

Screening is more difficult with self-completion modes (web and paper); as with sample selection, sample screening instructions can be complex and difficult for participants to complete without an interviewer present.

In general, where the incidence of a particular subgroup in the population is lower, the cost is likely to be higher for a survey involving screening, as many participants will not meet the required criteria. While this is the case regardless of mode, the costs of incentives may be higher when screening using push-to-web approaches. For example, the push-to-web experiments on the Ethnic Minority British Election Study showed that although it was possible to screen by ethnicity, it required collecting data that was unusable for respondents who did not meet the screening criteria, but who still had to be incentivized to provide this (Bodgan et al, 2024). For interviewer-administered surveys, screening is more straightforward, as screening can take place before the participant is required to do anything, meaning that no incentives need to be paid to anyone screened out before the survey begins (although this may still be outweighed by the additional costs of interviewers conducting the screening).

One option for mitigating the challenges of identifying particular sub-samples is to use administrative data to supplement sample information and try and identify households more likely to contain respondents falling into particular sub-samples. This approach is currently used on SHeS to identify households with children. Administrative data has also been used to try and identify households more likely to contain people in particular age groups on the Participation Survey and Community Life Survey (see case study re. the Participation Survey). The scope for using administrative data more often in this context is discussed further in chapter 11.

Sample clustering

Clustering – whereby small-scale geographic areas such as postcode sectors are first randomly selected, followed by addresses within those areas – is often a feature of sampling on face-to-face surveys to reduce interviewer travel distances and costs. However, it can impact the precision of estimates and the ability to conduct analysis in small local areas, so where possible an unclustered design is generally preferred.

Since 2012, the SHS and SCJS have adopted a single-stage, unclustered stratified sample design, but with addresses grouped into batches post-selection, to facilitate effective fieldwork (prior to 2012, both SHS and SCJS used some element of clustered sampling). The SHeS is based on a partially clustered, stratified multi-stage design.

For surveys using push-to-web, push-to-telephone, or paper (postal) approaches only, clustering by location can be completely avoided as clustering brings no logistical advantages. However, where a mixed mode survey involves a face-to-face or knock to nudge element, particularly as a follow-up mode, this may become an issue, since the number of cases being issued to follow-up face-to-face per location is likely to be lower (given some interviews will already have been completed in earlier modes). As a result, for surveys using a mixed mode design with face-to-face as a follow-up mode for non-responders, even if the overall survey remains unclustered, the locations where face-to-face follow-up is employed may need to be clustered for reasons of cost and fieldwork efficiency. For example, on the TLFS knock-to-nudge is used in areas where expected response rates are lower only – focusing on more deprived areas. There is no barrier to using a two-domain approach, for example using a clustered sample in the more rural areas of Scotland and an unclustered approach in urban areas. This approach was employed on the SHS and SCJS prior to 2012.