Long term survey strategy: mixed mode research report

12. Good practice when considering mode change

This report has discussed in detail the key issues impacting on data quality and usability that arise when considering to moving to a mixed mode design, and how these issues may apply in the particular context of the three flagship Scottish Government general population surveys. This final chapter considers what ‘good practice’ with respect to reviewing and implementing mode change options might look like, setting out various elements that the Scottish Government may wish to consider in developing its own approach. It is framed in terms of ‘good practice’ rather than ‘best practice’, since in reality the most appropriate approach in any context will be dependent on a combination of factors such as the available time and budget, number and range of survey stakeholders, and the main aim that changing mode is intended to achieve.

Involving stakeholders

Survey experts interviewed for this research who had themselves been involved in mode change emphasised the central importance of not only consulting stakeholders, but engaging them across the whole journey from considering to implementing a change in mode design. As one interviewee put it:

“If you think you have communicated well enough, you haven’t. Multiply that by a thousand. You cannot do enough communication if you are to change it. If you are changing the survey that somebody relies on, you cannot ever think that you have engaged enough.” (Expert interviewee 4)

At the early stages of engagement, understanding stakeholders’ priorities for the surveys is likely to be key. It is worth noting in this regard that, even among the relatively small number of stakeholder interviews conducted for this study, there were a number of competing views on the top priorities for the three Scottish Government surveys, particularly around the extent to which people prioritised maintaining trends versus other goals, such as being able to flex the content to meet current or future policy needs and/or very significantly expanding the sample size. Different stakeholders also prioritise different measures within the three surveys, indicating that it may be difficult to identify a consensus on priorities even if trade-offs are considered at a question-level rather than a survey level. However, understanding what (different) stakeholders value is nonetheless crucial to both development of appropriate mode design options and weighing the impacts of the trade-offs that are likely to be involved whatever option is selected (including maintaining the status quo).

There is no single standard approach for involving stakeholders in discussions about survey redesign. The TLFS invested very significantly in this area, undertaking two years of user engagement involving over 3,000 users before they began transitioning to the new design. Others, particularly those surveys that transitioned during Covid, did not have the time or resource to support this level of engagement. However, whatever approach is adopted, experts suggested a number of elements worth considering in terms of good practice.

• Framing of the issues at stake. Several challenges were raised in relation how to engage stakeholders with relatively technical issues around survey methods. One expert suggested that users were not easy to engage when discussing mode change per se, but instead were interested in issues of data quality, sample size and trend discontinuity. It was also suggested that engagement is easiest when the discussion focuses on tangible outputs - for example, the likely impact of changing mode design on the ability to accurately estimate the proportion of obese adults in Scotland and trends in obesity over time, as opposed to the general possibility that mode change might impact on survey estimates or trends.
• Explaining why methodological issues matter. One expert interview noted that it could be challenging to explain to stakeholders why a particular methodological issue is important. In relation to questionnaire design, for example, it was suggested that using audio or video clips of someone struggling to answer questions that are poorly designed (in general or for a particular mode) was an effective way to illustrate the relevance of these challenges to the accuracy of survey measures.
• Maximising transparency. Several experts emphasised that all development and testing should be published in an accessible way to help evidence decisions and to allow stakeholders to comment and raise issues throughout. This echoes the ‘Accessibility and Clarity’ aspect of quality from the ESS framework. It was also noted that the development of new approaches may uncover previously unnoticed problems with previous approaches (for example, in how people have understood questions asked by the existing mode). Although it might be difficult to be transparent about such issues, especially if data is currently being used to produce national estimates, it was emphasised that in the longer term, honesty will help build trust in the robustness of the approach taken to survey design and redesign. The ONS team involved in the development of the LMS and TLFS made data from parallel runs publicly available, to allow data users to understand and provide feedback on TLFS data, prior to LFS decommission. They also worked with the Office for Statistics Regulation, alongside wider stakeholders, to ensure the process of reviewing and transitioning modes was clear and robust (see case study in Appendix A).
• Allowing sufficient time. While there was considerable variation in the types of testing undertaken to inform mode change decisions (as discussed below), there was consensus among the expert interviewees of the importance of ensuring enough time and resources for stakeholder engagement at all stages of the process.

Clarity of purpose

As discussed above, a key focus of stakeholder engagement around future survey mode(s) is likely to be the trade-offs involved. Both expert and stakeholder interviewees recognised that thinking through trade-offs in terms of quality, budget, and usability is far from straightforward; there is unlikely to be a single design that meets everyone’s priorities fully. Expert interviewees emphasised the importance of transparency about likely trade-offs and recommended guarding against the temptation to underplay them in the early stages of stakeholder engagement.

“’Trade-off’ is the right term … every design decision involves a trade-off” (Expert interviewee 8)

The questions provided in the tables in this report provide a potential framework for engaging in discussion about competing priorities and trade-offs with stakeholders.

However, it was also emphasised by expert interviewees that it is equally important to be clear from the outset on the reasons that mode change is being considered by the commissioner and what any redesign is intended to achieve. While pressures on public sector budgets are clearly a significant factor in discussions about the future of large-scale surveys in the UK, as discussed in chapter 10, reducing budgets was not the main stated aim for all the surveys reviewed for this study. Other stated aims included: reducing cost increases (as distinct from actually reducing survey budgets); increasing the sample size; and future proofing methods, so that surveys are more resilient both in terms of general societal shifts (such as greater use of online and potentially lower willingness to participate in face-to-face surveys) and potential future shocks (such as a future pandemic).

Whatever the primary aim of redesign, it was argued that it is important to be clear about this at the outset and throughout the development and testing process. In the absence of such clarity, it was suggested that focus is likely to drift and the ultimate result is unlikely to deliver the hoped for outcomes.

Finally, experts and stakeholders interviewed for this study commented that it is not only important to be clear on the purpose of any redesign, but also on the overall purpose of the surveys. One view was discussion of changing modes provides an opportunity for a more radical rethink of whether the focus, sample design, structure and content of surveys remain fit for purpose.

Investing in development, piloting and parallel testing

As discussed in chapters 6 and 10, approaching mode redesign in a robust manner is likely to involve significant upfront investment in development, piloting and testing of different options. This may include:

• Questionnaire (re)design and testing, including cognitive testing of both new content and content originally designed for a different mode
• Parallel testing of new mode design(s) alongside the existing approach, in order to assess mode effects and to begin to construct a new time series
• Piloting and experimentation to test various mitigation strategies aimed at addressing some of the known challenges of the new design (or perhaps just of survey research more generally), such as new incentive strategies, or redesigning materials.

Overall, the consensus among experts was that it was worth spending as much time as possible on this development: “Longer and slower is better!” (Expert interviewee 8). Parallel testing was also widely considered the ‘gold standard’. However, a more exceptional expert view was that survey teams sometimes spend too long on parallel runs and would be better investing more in questionnaire design and testing to ensure the content is fit for the survey’s current purpose.

In practice, the nature and scope of the development and testing phase will also depend on the extent of changes being proposed, time frames and budget. It was noted, however, that compared to England (where different UK Government departments are responsible for individual surveys), Scotland has relatively greater centralised control over the major general population surveys. This provides a potential opportunity to leverage learning from across the SHS, SCJS and SHeS. This might involve planning experiments and testing of different options on each survey, in order to build a body of knowledge about likely impacts more quickly. The review of surveys conducted for this study also highlights a wide-ranging methodological work on mixed mode approaches that is already taking place in the UK (and further afield), from which the Scottish Government could draw lessons.

A final consideration in developing an approach to testing new designs is what quality metrics these are judged against. As discussed in chapter 9, assessing survey quality is not necessarily straightforward, particularly as many methodologists are moving away from response rates as the main or primary indicator of nonresponse bias. However, there are various frameworks available for considering the elements of quality, and it is important to be clear at the outset about the criteria against which any new design is to be judged.

‘Future proofing’

A final element of ‘good practice’ in considering future mode design is to assess how ‘future proof’ the different options (including the status quo) might be. Much of the recent focus in terms of the future of survey research has been on the sustainability of face-to-face interviewing. However, there are a number of factors that might impact on the longer-term sustainability of all modes, including:

• Changing patterns of willingness to respond to different modes. Over the past decade it has become increasingly harder to get people to take part in research by telephone. This is partly due to the decline of usage in landlines but also to changing patterns of responding to calls from unknown numbers and use of telephone number blocking. As discussed in chapter 5, face-to-face response rates have declined over the longer-term and more sharply since the Covid-19 pandemic – at this stage, it is unclear what type of response rate it will be possible to achieve face-to-face in the longer-term.
• Availability of different sampling frames. As discussed in chapter 4, with the move away from landline telephone to mobile phones, there is now no reliable and comprehensive database of telephone numbers from which to draw a sample for a probability-based general population survey. Similarly, there is no robust list of email addresses in the UK. There is an assumption, therefore, that the Postcode Address File (PAF) will remain the main low-cost option for a robust sample frame of residential addresses (and, by extension, the general population living in households) whatever mode of interview is used. While it is possible to enhance PAF by adding additional information to addresses, as this is still at the address rather than the individual level this still constrains the options for individual targeting and boosts. However, as discussed in chapter 11, it is possible that as further work to link administrative data together progresses, more sophisticated sample designs may become feasible.
• Risk of disruption. The Covid-19 pandemic disrupted in-home face-to-face interviewing with fieldwork being suspended because of risk of disease transmission. Any future pandemic would likely create similar disruption to face-to-face data collection. Other modes would be susceptible to different types of disruption. For example, any approach relying on advance letters would be at risk from extended postal strikes. Online modes would be at risk from any major outage to domestic online connectivity. All these risks are difficult to plan for, but would have a major impact.
• Cost of different modes. While reducing the use of interviewers on surveys is likely to reduce their total costs, no mode is immune from future cost increases. The cost of interviewer-administered interviewing (face-to-face and telephone) is sensitive to labour market supply of suitable interviewers and changes to the Living Wage. Approaches that rely on large mailouts will be more sensitive to changes in postal rates (the significant increase in postage over the last decade was noted by several expert interviewees). Where incentives are used, these costs are also likely to increase over time as the amount required to have the same impact increases.
• Availability of other data sources to meet information requirements. Any advances in the use of administrative data to supplement survey data will have an impact on what surveys need to collect in the future. As discussed in chapter 11, advances in the use of administrative data have arguably been slower than initially envisaged, but this may change in the future.

While changes to these factors are very difficult to predict with great certainty, it is important they are at least considered when developing mixed mode survey options. One potential advantage of a mixed mode approach is that they safeguard against the risk of a complete break in data collection, since it is less likely that two modes will experience the same level of disruption at once. However, any major disruption would still impact on the balance between modes, and may cause significant issues for data quality, so mixing modes is far from a panacea in this regard.

Finally, in addition to changes that may impact on the practicalities of survey administration, there is also the potential for major changes to the information requirements they are expected to meet. Decisions around the future of the census in Scotland, any major change to the design of the National Performance Framework, and any changes to key Scottish Government targets such as those relating to Fuel Poverty, for example, could change the information requirements for the major surveys – with implications for their optimum future design. As such, any considering of future design options should factor in, as far as possible, how the surveys might need to be used in future, as well as how they can best meet current stakeholder needs.

Summary of good practice suggestions

• Engage stakeholders as much as possible, from as early as possible, as transparently as possible, being upfront about trade-offs and sharing learning and data from testing
• Communicate clearly with stakeholders – including conveying how methodological issues and trade-offs relate to the tangible outputs they value
• Be clear about the aims of considering changing mode – and maintain focus on these aims throughout the development and testing process
• Plan to invest in significant development and testing work. While there is no definitive template, parallel runs are considered the ‘gold standard’, and significant investment in questionnaire development and testing is likely to be required when moving questions to new modes.
• As far as possible, decisions about future mode designs should consider how ‘future proof’ different options are likely to be. This should include considering future as well as current information requirements for the three surveys.