Long term survey strategy: mixed mode research report

3. Identifying key issues to consider when assessing mode options

This chapter introduces the key issues of relevance to decisions about the future mode(s) of the Scottish Government’s flagship general population surveys and explains how these issues were identified. It begins by summarising the key existing frameworks of relevance to considering options for survey mode.

Existing frameworks for assessing survey quality or design

Total Survey Error (TSE) framework

The TSE Framework is the generally accepted approach for assessing survey quality used by survey methodologists. The TSE approach methodically identifies all possible errors which can arise at each stage of the survey process. In so doing the survey process is divided into two main strands: a representation strand (which primarily relates to the accuracy of the sample in representing the population it is drawn from) and a measurement strand (which primarily relates to whether the responses captured are accurate reflections of the behaviours, experiences or views the questions were aiming to capture).

The relationship between survey process and error type is shown in Figure 3.1, below.

The TSE is generally accepted as the most comprehensive framework for assessing survey quality. Other frameworks – such as NatCen’s REMoDEL approach to considering options for mode change (Cornick, 2023) – draw heavily on TSE as a theoretical framework. The building blocks of the TSE framework clearly provide a useful tool for reviewing mixed mode designs and potential mode transitions, by ensuring that different types of error and the interactions between them are identified and considered. As such, it has strongly informed the key topics considered in relation to the impacts of mode change in this research.

In addition to the TSE framework, other frameworks relevant in assessing survey design include the European Statistical System framework, the Tailored Design Method and Respondent-Centred Design.

European Statistical System (ESS)

The ESS provides a framework for producing, using and disseminating comparable and reliable statistical information across Europe. It was developed with input from national statistical institutes and other national authorities across the European Union's member states, along with Eurostat. The framework identifies five aspects of quality: ‘relevance’, 'accuracy and reliability', 'timeliness and punctuality', 'coherence and comparability' and 'accessibility and clarity'. In comparison to the TSE framework, the ESS framework emphasises not just the accuracy of estimates, but also their fitness for use. In other words, that estimates are not only as accurate as possible, but also meet user need in relation to factors such as data accessibility and clarity, timeliness, relevance, and comparability across repeated surveys and regions of the country, as well as demographic sub-groups.

Tailored Design Method (TDM) and Respondent Centred Design (RCD)

RCD and TDM are not methods for assessing overall survey quality, but rather principles for maximising response rates and improving the quality of survey data. However, they differ in their focus and approach.

TDM, developed in America by Don Dillman (see Dillman et al, 2014), emphasises the importance of social exchange theory, suggesting that respondents are more likely to participate in a survey if they perceive the benefits of participation to outweigh the costs, and if they trust that their responses will be used as promised. It focuses on elements such as respondent-friendly questionnaire design, personalised communication, and a series of contacts (pre-notification, invitation to participate, reminders), all tailored to the specific context of the survey.

RCD (Wilson and Dickinson, 2021) is focused more on the cognitive processes of respondents. It emphasises understanding how respondents perceive, process and respond to survey questions. The design of the survey is centred around ensuring that questions are understood as intended and that respondents are able to accurately recall and report the necessary information. It advocates an iterative approach to survey design, often involving extensive pre-testing and cognitive interviewing to explore how different respondents interpret and respond to survey questions. Key recommended elements include ensuring language that is understandable to the respondent, keeping the survey length reasonable, and ensuring the survey is accessible and comfortable for the respondent to complete.

Both TDM and RCD can help inform optimum mixed-mode designs by providing practical elements to consider, particularly in terms of mitigation strategies that might be available to help offset the potential risks of mixed-mode designs, such as strategies to improve the motivation of respondents to opt-in to or complete surveys where an interviewer is not present to persuade them (TDM) or how the redesign of questions for different modes can take account of respondents’ cognitive processes (RCD).

Additional stakeholder priorities for Scottish Government surveys

While existing frameworks – particularly TSE – provided a starting point for considering the key issues that should be examined when considering options for mixed mode designs on the Scottish Government’s flagship general population surveys, there are some significant considerations that are missing.^[22] In particular, existing frameworks arguably pay less attention to practical considerations – particularly cost – as well as issues relating to user need (e.g. relevance and timeliness). Moreover, where existing frameworks assess the extent to which survey designs enable participation among different groups of the population, this tends to be framed as a question of sample accuracy; ethical issues around exclusion from public data are arguably less prominent.

In addition to more traditional quality issues around sample representativeness and size, and quality of questionnaire design and administration, these additional themes all featured in interviews with stakeholders in the Scottish Government’s general population surveys, conducted for this research:

• Cost – stakeholders tended to frame value for money in terms of the public benefit of the surveys, rather than securing cost savings. However, where stakeholders were responsible for budgets, it was recognised that budget may be a factor outside of their control that impacts on future design – even if they thought the survey was value for money, if the budget was not available, then a significant change would have to be made. In this context, there was strong interest in whether mixed mode designs could reduce (or avoid increasing) the costs of surveys at a time when public sector budgets are under pressures. There was also considerable interest in whether alternative designs could deliver bigger sample sizes within the same budget, without an unacceptable loss of data quality. Costs are considered in detail in chapter 10 of this report.
• Timeliness of data was seen as both a key priority and an area that could be improved across all three flagship general population surveys. Scottish Government policy stakeholders and external stakeholders concerned with service delivery expressed a desire for the surveys to be able to deliver data more quickly, in order to increase their usefulness. Timeliness as an element of survey quality is discussed in chapter 9 of this report.
• Relevance – linked to concerns about timeliness was a desire among some stakeholders for the surveys to be able to be more responsive to emerging policy priorities. It is important to note that there was some disagreement among stakeholders on this point – an alternative view was that the surveys ought to be primarily focused on longer-term trends (indeed, maintaining trend data was a top priority for some – though not all – stakeholders). In the context of mode change, considering the ways in which different designs might impact on the ability of the Scottish Government to provide data that is relevant and timely, and on their ability to maintain historic trend data, are each important elements of assessing their ability to meet stakeholder needs.

In addition to quality, practical and data user considerations, stakeholders also highlighted the importance of maintaining the surveys’ external credibility. The value of the surveys to stakeholders was seen as contingent on them being – and being seen to be – a “tried and tested, recognised source” by other stakeholders, including Ministers and the public. The scope for using administrative data, and the ways in which this might help shape decisions about the future design of Scottish Government surveys was also raised by a number of stakeholders as an important issue to explore. For example, could administrative data remove the need to ask some questions about which administrative data exists, meaning questionnaires could be shorter and different modes potentially more feasible? Finally, there was a desire to better understand the potential impact on inclusion of changing mode – is there any evidence that changing or mixing modes could increase the accessibility of surveys for some groups (e.g. young people) and/or reduce it for others (e.g. those with low literacy levels, or who are digitally excluded)?

Key issues discussed in the remainder of this report

The final list of key issues this research focused on (drawing on both existing frameworks and stakeholder views, as outlined above) can be roughly divided into three overarching areas:

• Impacts of different modes on representation – in other words, what difference does mode of data collection (and mode of invitation to participate) make to the sample of people that respond to surveys? (Issues reflected in the right-hand side of the TSE framework diagram, shown in Figure 3.1). These issues are discussed in detail in chapters 4 and 5.
• Impacts of different modes on measurement – in other words, what difference does mode of data collection make to the nature and quality of data collected from the sample of respondents? (Issues reflected in the left-hand side of the TSE framework diagram, shown in Figure 3.1). These issues are discussed in detail in chapter 6, while implications for other aspects of data collection options (e.g. survey length, structure, etc.) are discussed in chapter 7.
• Impacts of different modes on wider commissioner and user needs and priorities (beyond TSE) – including impacts on trends (chapter 8), measures of quality and external credibility (chapter 9), resources (chapter 10), and the possible interaction between survey and administrative data (chapter 11).

In addition, chapter 12 considers evidence on ‘good practice’ in making decisions about transitioning surveys to mixed mode approaches, drawing on both the literature and on expert interviews and stakeholder views.

These issues shaped the questions included in expert interviews, the focus of the desk-based review of key literature, and the structure of the remainder of this report.