Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): Mode Effect Study Report 2015
Findings of a mode effect study conducted during the 2015 SALSUS survey to see whether the different routes of administration resulted in different data.
6 Does mode affect survey engagement?
This chapter explores the impact of mode on survey engagement as indicated by responses to the open-ended and 'other specify' questions in SALSUS 2015.
The quality and richness of responses to open-ended questions is often used in mode comparison studies to gauge the extent to which respondents are engaged. This is important as engaged respondents are more likely to give thoughtful and accurate answers. There are no real open-ended questions in the current SALSUS questionnaire, but we looked at responses to the cigarette branding question ( 'Please [write/type] in the space below the names of as many makes or brands of cigarettes that you have either seen or heard of') and at a number of questions which allowed for an 'other specify' response.
Survey engagement - key findings
- There were no clear indications that the online mode increased or decreased engagement with the survey
- Lower engagement was suggested by higher non-response to the cigarette branding question (the closest to an open-ended question) among pupils completing the survey online compared to paper
- Higher engagement was suggested by higher use of 'other specify' for those responding online compared to paper
In the cigarette branding question, 50% of pupils did not name any brands in the paper mode, compared with 59% in the online mode [16] . This suggests that pupils found it 'easier' to skip past the question in the online mode.
Once those who provided at least one brand were separated from those giving no answer, analysis showed there was very little difference in the number of brands named in each mode. This suggests that pupils are more likely to not give an answer in the online mode, but among those who do, neither mode prompts more responses.
The 'other specify' options were more commonly used among pupils in the online sample, compared with the paper sample (see Table 6.1). For example, for 'Where do you usually get your cigarettes/tobacco from?', 7.1% of pupils in the online sample selected the 'other' response, compared with only 2.5% of pupils in the paper sample. This suggests a higher level of engagement from the online sample.
Table 6.1 Proportion of 'other' responses by survey mode
| Paper | Online | |
|---|---|---|
| Where do you usually get your cigarettes/tobacco from? | ||
| % 'other' response option | 2.48% | 7.14% |
| Base | 714 | 369 |
| Where were you the last time you used drugs? | ||
| % 'other' response option | 3.61% | 7.81% |
| Base | 1,017 | 590 |
| If you felt that you needed to get help because you were using drugs, who/where would you go to? | ||
| % 'other' response option | 1.45% | 4.84% |
| Base | 9,287 | 6,271 |
A greater use of the 'other specify' options could point to increased engagement with the survey with the assumption that those that are engaged deliberate more on questions, including considering 'other' answers beyond those specified on the list. However, the situation is not clear given that higher levels of non-response to the cigarette branding question suggests that the online mode did not increase the level of consideration that pupils gave to that question.
Contact
There is a problem
Thanks for your feedback