Long term survey strategy: mixed mode research report

Glossary of key terms

AAPOR response rates: The American Association for Public Opinion Research (AAPOR) has produced standardised methods of calculating and expressing survey response rates, enabling comparison across surveys. There are six formulas; the first three are used in this report and are as follows:

• AAPOR1: This is the minimum response rate. It is the number of complete interviews, divided by the total number of cases in the issued sample less those identified as ineligible.
• AAPOR2: As per AAPOR1, but with partial interviews included in the numerator.
• AAPOR3: As per AAPOR1, but for cases where eligibility cannot be determined, the proportion of these cases that are ineligible is estimated, and these cases are excluded from the denominator (further guidance for this is available in the AAPOR handbook).
• AAPOR4: As per AAPOR3, but with partial interviews included in the numerator.

Calibration: In the context of this report, calibration refers to attempts to control or correct for differences in data between data collection modes. Modelling can be carried out to ‘recalibrate’ the data, whereby weighting is used to match data collected via one mode, to another. This may involve using weighting to match ‘old’ data in a time series collected by one mode, to ‘new data’ collected by another, or vice versa. It can also be used in mixed-mode research to adjust for differences between the different modes used and produce a single dataset.

CAPI: Computer-Assisted Personal Interviewing (CAPI) is the most common type of face-to-face interviewing, involving an interviewer using a digital device such as a tablet or laptop to administer pre-scripted survey questions and record responses.

CASI: Computer-Assisted Self-Interviewing (CASI) sections are typically self-administered sections of CAPI surveys, where the interviewer hands over their laptop or tablet to the respondent to complete a particular section directly. They are typically used for more sensitive sections within face-to-face surveys.

CATI: Computer-Assisted Telephone Interviewing (CATI) is the most common type of telephone interviewing (CATI is often used interchangeably with telephone interviewing), involving telephone interviewers using a digital device to administer pre-scripted survey questions and record responses.

CAWI: Computer-Assisted Web Interviewing (CAWI) involves respondents completing an online survey themselves via a web browser.

Cross-sectional survey: A survey of a representative sample of a defined population at a specific single point in time, or from samples of the same population at repeated points in time.

Data linkage: The practice of linking survey data to other data, such as administrative records, to supplement and enhance data collected from the survey.

Eligible sample: The sample units (e.g. individuals or households) from a survey’s target population that meet specific inclusion criteria set out by researchers/commissioners. For example, for a survey aiming to interview private households, the eligible sample would be all the addresses where a private household was resident – if the sampled addresses included business addresses, these would not be part of the eligible sample.

General population survey: A type of survey designed to collect data from a sample that is representative of an entire population, such as the population of Scotland.

Household reference person: An individual within a household who is selected to provide information for a survey about the whole household.

‘Knock to nudge’: A contact method whereby interviews make in-person visits to a survey sample who have been invited to take part in a survey, to encourage them to participate (typically by another mode, such as telephone or online). Knock to nudge was first used on a widespread basis during the Covid-19 pandemic, when restrictions on face-to-face contact precluded long visits within people’s homes, but eventually allowed for interviewers to have brief, distanced contact on the doorstep to encourage those who had not responded to written invitations to take part in online or telephone surveys.

Longitudinal: A longitudinal survey involves repeated interviews with the same sample over a period of time (distinguishing it from cross-sectional surveys, where those sampled are only interviewed on one occasion).

Non-response bias: The bias that occurs when sample units (e.g. individuals, households) who do not respond to a survey differ in meaningful ways from those who do respond. Non-response bias means that certain types of people are underrepresented, and the accuracy and generalisability of findings may be negatively affected.

Measurement error: When the answers people give do not accurately reflect what the survey was intended to measure – in other words, it is the difference between the data you obtain from your survey, and the ‘true value’ of the thing you were trying to capture. This type of error can arise from various sources such as poorly worded questions, respondent misunderstanding, and data processing errors, among others.

Parallel testing: Parallel testing, or conducting ‘parallel runs’, involves running a survey with a new design simultaneously with running the existing survey with the previous design, so that results can be directly compared.

Population parameter: A value that represents a characteristic of an entire population. It is generally not feasible to measure population parameters directly, since it requires collecting information from all members of the population. General population surveys are designed to provide estimates of population parameters based on samples.

Postcode Address File (PAF): A database of known postcodes in the United Kingdom, managed by Royal Mail. It is used as a sample frame for some UK surveys, including the Scottish Household Survey, Scottish Crime and Justice Survey and Scottish Health Survey.

Push-to-web / telephone: A survey sample is invited to take part in a survey online / by telephone, via a different mode of contact than the survey itself. Typically, push-to-web/telephone invite people to take part by post, writing to a random sample of the population and providing them with a unique weblink and/or a portal to register telephone contact details and preferences.

Random digit dialling (RDD): A probability sampling approach whereby telephone numbers for selecting respondents are generated at random to create a sample. involves using software to randomly generate phone numbers. Landlines can be generated using the first part of area codes, so that they can be targeted at specific geographies. This is more difficult with mobile numbers; although new techniques have emerged that allow randomly generated mobile numbers to be ‘pinged’ to estimate where they were last used, this is much less accurate.

Response rate: The number of sample units (people, households, etc.) who respond to a survey compared to the total number of people who were asked to respond (and who were eligible to do so). The exact calculation for determining response rates varies, depending on how partial interviews and cases of unknown eligibility are treated (see AAPOR response rates, above).

Respondent Centred Design (RCD): A survey design approach that places respondent experience, preferences and needs at the centre. The approach considers aspects of surveys such as ease of use, relevance, and participant feedback mechanisms.

Sampling error: Occurs when a sample does not perfectly represent the target population.

Survey mode: The way in which survey data is collected, e.g. face-to-face, telephone, web/online.

Tailored Design Method (TDM): An approach to designing surveys, developed by Don A. Dillman (2014), that seeks to maximise response rates and data quality by considering participants’ experience and customising surveys to the target population.

Total Survey Error (TSE) framework: The generally accepted approach for assessing survey quality used by survey methodologists. The TSE approach methodically identifies all possible errors which can arise at each stage of the survey process.