Long term survey strategy: mixed mode research report

7. Implications of mode for data collection options

Introduction

In addition to the direct impacts of mode on measurement error, choice of mode(s) of data collection also has implications for other features of survey design, which in turn may constrain the type and volume of data that can be collected. This chapter discusses the potential implications of changing or mixing mode for survey length, survey structure, options for presenting answer categories, collection of additional data, beyond the main interview, and for accessibility and inclusion.

Survey length

The three main Scottish Government surveys currently involve fairly long interviews – the SHS currently lasts around 45 minutes (including the household and random adult element, but excluding the surveyor visit, conducted separately); the SCJS lasts around 40 minutes^[44]; and the SHeS main interview (including interviewer collected height and weight measures) varies depending on the number of people in the household, but lasts around 45 minutes for a one adult household, while the child interview lasts around 30 minutes (interviews for multiple adults in the same household are sometimes conducted concurrently to reduce the total time).

There was anxiety among the Scottish Government survey stakeholders interviewed for this study that the length of the surveys meant they would be off-putting to complete by other modes, particularly online, meaning that some people might be less likely to take part in the first place while others might drop out part-way through. One view was that to make the SHS, in particular, work online the questionnaire would have to be very substantially shortened and/or broken into a series of shorter surveys.

A key finding from both the literature and the expert interviews conducted for this study is that there is no clear consensus among survey methodologists as to the ‘optimal’ interview length for different modes of administration. In particular, although there is general agreement that face-to-face and telephone interviews tend to allow for longer interviews than do paper or online administrations (due to involving an interviewer who can motivate respondents to complete with the interview), there is considerable variation in views on how long is ‘too long’ for a web-based interview.

When they have been asked explicitly, respondents have said that the optimal length for a web interview is between 10 and 20 minutes (Revilla and Ochoa, 2017), and qualitative research has suggested that just five minutes is more attractive for young people (Lugtig and Luiten, 2021). Survey methodologists interviewed for this study noted that a number of non-UK countries strongly advocated not exceeding 20 minutes for an online interview, including in the Netherlands, Canada and Australia (in spite of the fact that participation in certain government surveys is mandatory there). A key factor in recommending keeping web interviews short is the need to design them to be completed on smartphones, as this is how many people, in practice, will complete them. There is evidence that break-off rates are higher when web surveys are completed on mobile devices (Maveltova and Couper, 2015; Huskinson, in press).

Aside from the threat of non-response there is also evidence that without the motivating presence of an interviewer, longer interviews can damage data quality. Andreadis and Kartsanidou (2020) found that longer web interviews were associated with shorter responses to open-ended questions and more non-substantive answers (as measured by the choice of the ‘neither/nor’ response option at Likert-type scales). Peytchev and Peytcheva (2017) varied whether certain questions were placed near the start or end of a web questionnaire that took on average 18 minutes to complete, finding more evidence of contradictory response patterns for those placed near the end.

However, the survey review and expert interviews conducted for this study also identified a number of surveys that involved web interviews substantially longer than 20 minutes. For example, on Understanding Society, the online version of the questionnaire has very similar content to the face-to-face version, which lasts 45-60 minutes. Understanding Society differs from the three main Scottish Government general population surveys, as it is longitudinal and respondents have already engaged with the study at previous waves. However, the European Social Survey, which is cross-sectional, has also implemented a much longer online version of the questionnaire, lasting around 50 minutes. This was implemented following testing of web content designed to take 35 minutes and 50 minutes, which found little difference in response rates or drop-off and few differences in sample profile:

“Our conclusion, which goes against some other guidance, is that you can have a reasonably long survey in self-completion form and achieve good outcomes. Most people that start the survey go on to finish, and if they drop out it’s often early in the questionnaire, meaning a reduction in length may make minimal difference.” (Expert interview 15)

This view was echoed by some other experts, who argued that a well-designed web questionnaire with appropriate incentives could be longer and still maintain an acceptable response. There was a perception that tolerance for completing things online was increasing in general.

“I would still think, you know, an hour and a half is too long, but an hour and a half is too long for an interview as well. So, I think if anything, people have been more willing to spend longer on devices that is the trend, and much less willing to spend a long, long, time, with effectively a stranger in their house.” (Expert interview 18)

Experimental research has also suggested that, in practice, the relationship between interview length and nonresponse in web surveys is fairly weak (e.g. Lugtig and Luiten, 2021; Kaplowitz et al, 2012; Yan et al, 2011; Galesic and Bosnjak, 2009). This research has, however, varied whether the predicted interview length was stated in the survey invitation and the actual average interview length, with the maximum length lying at some point between the five- and 30-minute mark, so still somewhat below the longer interviews that are typical for face-to-face administration.

Those involved in surveys that had moved from face-to-face to telephone – including the National Survey for Wales and the Health Survey for England (during the Covid-19 pandemic – it has since returned to face-to-face) – indicated that they needed to be shorter by telephone, partly because respondents find them more tiring to complete over the phone. One view was that 35 minutes is around the limit of what can be asked by phone without a significant loss of data quality. The National Survey for Wales was initially cut back from 45 minutes to 25 when it moved to telephone, but has subsequently increased to around 30-35 minutes, while the online self-completion has allowed inclusion of additional content.

Potential mitigations

Those methodologists interviewed for this study who felt that online interviews needed to be kept short discussed the scope for modularising as a mitigating strategy, to maintain shorter interviews while maximising the content it is possible to include. Surveys can either be modularised so that different respondents complete different elements, or so that the same respondents complete different elements over a period of time, rather than all at once.

The former approach has been used on a range of push-to-web surveys that were formerly administered face-to-face, including Food and You 2 (which has a single web questionnaire, and two versions of the paper questionnaire) and British Social Attitudes (which has twelve versions of the web questionnaire, compared to four versions when it was administered face-to-face). The SHS and SHeS also involve modularisation already, with different respondents allocated to different ‘streams’ containing different content, to enable more topics to be covered across the surveys without increasing the overall length for each respondent. However, keeping the three Scottish Government surveys to the kind of maximum length recommended by some (though not all) survey methodologists for web interviews would require either further modularisation and/or modularisation in combination with a radical reduction of content.

Modularisation increases the amount of content a survey can cover but does so at the expense of smaller sample sizes for modularised content (which can often be compensated for by increasing the amount of sample issued), difficulties in cross-analyses of questions from different modules, and complexities in terms of data imputation and weighting strategies (as different sub-samples or streams may end up with different sample biases, requiring more complex weighting strategies). This in turn can have implications for both data processing resources (requiring more input from expert statisticians) and data usability (especially for non-expert users).

An alternative suggestion for managing longer content online is to design the web survey in a way that allows respondents to break off and come back to the survey later, rather than modularising it. While this approach requires careful explanation to the respondent, and the setting up of automatic reminders to ensure respondents do return to complete the interview, it avoids the pitfalls of modularisation noted above (though may increase attrition/partial interviews).

The expert interviews and survey review conducted for this study also identified examples of formerly cross-sectional surveys that had, or were considering, introducing a longitudinal element to their design to enable them to collect more data through telephone or web interview (see, for example, the case study for the Crime Survey for England and Wales in Appendix A). While there is evidence in the literature that this approach can avoid the poorer quality of data arising from long web interviews, it suffers in that the cumulative response across both sections is likely to fall below the response to the single longer interview (Andreadis and Kartsanidou, 2020).

A final potential mitigating strategy relating to survey length is to increase the use of administrative data to reduce the number of questions it is necessary to include in the survey questionnaire. The scope and feasibility of using administrative data alongside or instead of survey data is covered in chapter 11.

Survey structure options

Options for modularisation of survey content are arguably easier to implement on computer assisted modes, which can use sample data or responses to particular questions to allocate people automatically to particular modules of questions. Modularisation is more difficult with paper interviews, requiring multiple different hard copy versions of questionnaires.

Across the survey case studies conducted for this research, those that have transitioned to push-to-web design with a paper option (either for non-responders or on request) have generally had to both shorten and simplify the content and structure of the paper version of the questionnaire compared with the web-version. For example, the Participation Survey has three versions of the web questionnaire, but only one version of the paper questionnaire which contains only the key survey items. Surveys that have more complex structural features – such as looping of questions, complex routing, randomisation and rotations of survey questions, answer codes or both, etc. – will require more radical redesign for any paper version. The three Scottish Government surveys that are the focus of this study all contain these complex features in their main interviews.

Options for presentation of answer categories

As discussed in the previous chapter, while face-to-face, web and paper all allow for easy visual presentation of answer options, telephone is an inherently ‘aural’ mode. Scottish Government survey stakeholders interviewed for this study were concerned about the challenges of asking questions that had been based on (sometimes long) show cards by phone, in particular. One option is to provide showcards or other visual stimulus to support telephone interviews, either by drop-off by an interviewer, post, or via an online link for the respondent to view during the interview. This approach was implemented across all three Scottish Government surveys when they (temporarily) moved to a primarily telephone mode during the pandemic - for example, the SHS used a small number of online showcards when it moved to a push-to-telephone / video, followed by a choice of online or single-use hard copy showcards when using a knock-to-nudge approach during the Covid-19 pandemic (Martin et al, 2023). The Childcare and Early Years survey of parents has also followed this approach since 2021 (they are given the option of taking a hard-copy during a ‘knock-to-nudge’ visit, or accessing them online), to minimise mode effects between interviews conducted face-to-face and by telephone. However, it is more difficult for interviewers to ensure that respondents are using showcards when the interviews are being conducted remotely by telephone, especially when there are a large number of showcards to navigate.

Difficulties using showcards with telephone interviews was cited by one expert interviewee as part of the reason they had not considered it as an option when exploring options for transitioning a face-to-face survey. However, an alternative view was that the reliance of many face-to-face surveys on showcards is problematic for less literate respondents, and that redesigning surveys for telephone so that they work without showcards can make for a better questionnaire in terms of both respondent comprehension and inclusion.

Collection of additional data

The SHS and SHeS both involve the collection of additional data, in addition to questionnaire data – surveyor data on physical features of the respondent’s property, in the case of the SHS, and height, weight and biomeasures, in the case of SHeS. This data feeds into various key Scottish Government indicators.

During the pandemic, SHeS adopted a self-report approach for height and weight, with respondents asked to estimate their own height and weight. Previous analysis of the Health Survey for England (HSE) and Active Lives to compare interviewer-produced and self-report measures found that, on average, respondents overestimated their height and underestimated their weight (see NHSE, 2022). SHeS therefore applied an adjustment formula to their adult self-report height and weight data, based on that developed for HSE and Active Lives. However, even with the adjustment there were still concerns on HSE about the accuracy of estimates for those who were very under- or over-weight, so HSE data collected via self-report was labelled as inconsistent with the previous time series. Similarly, the 2021 SHeS technical report notes that “the mode of data collection requires the continued use of caution when interpreting such data.” Moreover, as no similar analysis was available comparing interviewer-collected and self-report height and weight for children, it was not possible to adjust data for the child sample of SHeS.

SHeS stakeholder interviewed for this study emphasised the importance of reliable height, weight and biomeasures and, in the context of these findings from the pandemic period, expressed scepticism about whether these could be obtained other than face-to-face by trained interviewers. With regard to other biomeasures, there are examples of surveys that do collect certain measures remotely – for example, Understanding Society has included respondent self-collected dry blood spots (which can be used to test for markers of risk factors for cardiovascular disease, like total cholesterol / HDL-cholesterol). However, while technically feasible, as discussed in the previous chapter, they found that the response rate to these types of self-collected biomeasures was much lower when respondents took part in the initial survey online, compared with respondents who had taken part in their initial interview face-to-face.

It is also worth noting (in the context of potential future content for health-related surveys like SHeS) that there have been found to be significant differences in data from cognition tests conducted via different modes^[45]. One view from an expert interview was that the data is so different, it is not possible to compare or combine cognition data collected via different modes.

During 2021, the SHS restarted undertaking surveyor dwelling inspections. As Covid restrictions still initially applied, surveyors carried out an external-only inspection. Data on internal aspects of the dwelling required for energy modelling were obtained from the householder by telephone or in a small number of cases from an Energy Performance Certificate. Analysis of the resulting data noted the impact of the change in approach to key estimates and argued that this approach should not be used in future waves because of mode effects (Scottish Government, 2023).

Similarly to the SHS, the English Housing Survey (EHS) moved to a push-to-telephone approach during the Covid-19 pandemic, with physical inspections of the exterior of the property only. They have experimented with alternatives to surveyor visits to properties, including getting a householder to walk surveyors through their home with a phone or to film inside their home, but the consistency of the data was described as problematic. At the time this study was conducted, it was expected this element would need to remain fully face-to-face because of the technical complexity of the task: “you really need an expert to go in and understand what they’re looking at because (the alternatives) just didn’t get us good data” (Expert interviewee).

Accessibility and inclusion

Ensuring that surveys are accessible to all members of the survey population is methodologically sound in that it leads both to more representative samples, and to lower levels of measurement error to the extent that respondents comprehend survey items in the same way. It was also noted by a Scottish Government stakeholder that government is (and should be) held to a high standard in terms of accessibility and inclusion, so inclusion considerations are also extremely important when thinking about the design of any government survey.

Accessible survey design is embedded in a wide range of principles and recommendations that are relevant to the design of social surveys. The Respondent Centred Design Framework (Wilson and Dickinson, 2022) includes the component ‘Design inclusively’, which notes that inclusive design enhances the survey experience for all respondents, as well as for interviewers.

Different modes are associated with different accessibility features. For example:

• Interviewer administered modes may be more accessible to those with low literacy, or who need additional assistance to complete the survey. An expert interviewee for this study cited concern to ensure that people who are less literate can take part as a key reason for inclusion as a key reason for choosing a push-to-telephone rather than a push-to-web approach when transitioning away from a primarily face-to-face design. Another factor was the desire to retain interviewer support for certain questions that benefit from having an interviewer to guide respondents through them:
  • “These are among the groups that we want our policies to be helping, so we don’t want a survey design that systematically excludes them.” (Expert interview 13)
• Web surveys, by their nature, require that respondents have internet access and are internet-literate. This excludes the offline population, who are typically older and more likely to be living in deprivation. On the other hand, web surveys could have alternative language options built in for respondents’ immediate access (although there would, of course, be cost implications to this), while interviewer administered modes require arranging a translator or interpreter. Web surveys can also be made more accessible to those who are visually impaired by ensuring they are adjustable, so that respondents can control features such as text size, background colours, and are able to use a screen reader with them. They also permit the possibility of incorporating aural elements, though this is relatively rarely implemented at present. One example of this is the Active Lives Children and Young People survey, which includes a brief web survey for pupils in years 1 and 2 (aged 5-7) with audio icons they can use to hear the question read aloud. Self-completion modes are also preferred by many with neurodiversity and those with introvert personalities, while the flexibility of time and location offered makes it more inclusive for those with busy lives, caring responsibilities or unusual work patterns.
• Telephone interviews are arguably less accessible for those who are deaf or hearing impaired, although it is possible to offer text-relay services.

The potential for offering different modes of completion to increase the accessibility of surveys to people with different needs was something that appealed to Scottish Government survey stakeholders interviewed for this study. For example, following up a push-to-web survey with a paper survey or face-to-face interviewer allows participants who may be unable or less willing to take part online to still participate in the survey (although it may still exclude those with low literacy).

It could be argued that there is a greater benefit from an inclusion perspective of offering different modes concurrently from the outset, rather than sequentially, in terms of maximising accessibility from the start. However, as discussed in chapter 10, the more modes that are offered concurrently, the greater the resources required to implement a mixed mode survey. This is particularly the case if alternative modes are actively offered from the outset, rather than simply being available on request for those who are not able (or do not wish to) take part by the primary (usually cheaper) mode. This is not to downplay the inclusion potential of mixing modes, but to note that there will inevitably be trade-offs with the resources required to offer multiple modes at once.