Long term survey strategy: summary report and framework to support decision-making

Key messages

• There has been a trend across large-scale government social surveys in England, Wales and elsewhere towards considering moving away from a purely face-to-face approach to data collection. This reflects pressure on public sector budgets, declining response rates for face-to-face surveys, and increasing internet access making online approaches more feasible.
• This research is intended to provide information on the benefits and risks of mixed mode surveys to inform the Scottish Government's next iteration of its Long Term Survey Strategy, which sets out the vision and plans for Scottish Government population surveys, particularly the Scottish Household Survey (SHS), Scottish Health Survey (SHeS) and Scottish Crime and Justice Survey (SCJS).
• 'Mixed mode' surveys involve combining two or more ways of collecting data from respondents, such that different respondents complete either the survey as a whole or the same section of a survey using two or more different modes.
• There are multiple ways of mixing modes, with mixed mode surveys varying in terms of mode of invitation, mode of completion, and sequencing (whether different modes are offered at the same time or in a particular order). The precise characteristics of the mode designs being implemented or tested at the time of writing across large scale surveys in the UK and further afield also varied considerably.
• Existing survey quality frameworks – in particular the Total Survey Error framework – in combination with priorities of Scottish Government stakeholders interviewed for this research indicate that when weighing future mode options the Scottish Government should consider:
  • The impact of different modes on representation - what difference does mode make to the profile of the sample of people that respond to surveys?
  • The impacts of different modes on measurement - what difference does mode make to the nature and quality of data collected from the sample of respondents?
  • And impacts of different modes on wider commissioner and user needs and priorities – including trends, measures of quality and external credibility, resources, and the possible interaction between survey and administrative data.
• This research summarises the evidence on the key issues, potential mitigations, and remaining trade-offs that are likely to need to be considered in each of these areas.
• There is no simple formula that can be applied to determine the optimum future design for any survey: in reality the most appropriate approach in any context will be dependent on a combination of factors such as the available time and budget, number and range of survey stakeholders, and the main aim that changing mode is intended to achieve. Good practice requires being clear on all these elements and being transparent about the likely trade-offs.
• Many of the impacts of mode change apply at the item or question level, rather than the whole-survey level. The implications of potential redesign may therefore need to be considered at the level of individual questions or topics in the Scottish Government's key surveys.
• Trade-offs associated with changing or mixing modes that are likely to be particularly difficult to fully mitigate, and where transparent conversations with stakeholders are particularly important, include:
  • Understanding the likely nonresponse bias associated with different modes and the potential impacts of this on key measures, since weighting and other strategies can reduce nonresponse bias but cannot eliminate it.
  • Agreeing on an alternative approach to response rates for measuring survey quality (or determining an acceptable future response rate if it is decided to retain this as a key measure).
  • Reducing mode effects may require significant question redesign and may reveal issues with longstanding questions.
  • Changing mode, in practice, means accepting an effective break in the time series: there are significant practical and methodological issues with using statistical techniques to try and recalibrate previous data to match a new design.
  • Additional elements and tasks where equivalently robust alternative approaches to face-to-face data collection are not (yet) available (including the physical survey element of the Scottish Household Survey and height and weight data in the Scottish Health Survey).
• Parallel testing and investment in questionnaire development and testing are widely seen as optimal for transitioning surveys in a robust and defensible manner. However, this means that, in the short-term, the costs of delivering surveys are likely to increase.
• The longer-term resource implications will depend on the exact approach and combination of modes adopted, in addition to any other changes made at the same time (e.g. to frequency, sample size, etc). Given this, it is not possible to make a definitive comparison of the resource implications of different mode designs. However, relative cost savings are likely to be greatest where surveys move to predominantly self-completion modes and make little or no use of face-to-face. Cost savings tend to be less substantial where face-to-face remains a significant element.