Scottish Social Attitudes Survey 2023 - Technical Report

Mode Change – Effect on Questionnaire Design

Changing the mode so that the respondent completes the survey online rather than face-to-face with an interviewer can have an impact on how a question is adminstered, and this possibility had to be considered for each of the Core Module questions before they were fielded on SSAS 2023. The majority of questions were able to be asked exactly as they were previously, without any changes. However, this chapter looks at those questions that did need to be adapted specifically for the push-to-web mode.

The survey asks respondents whether they think that the standard of the health service and the standard of living has increased, fallen, or stayed the same in the last 12 months. Provided they did not refuse to answer the question, or state that they did not know, they were then asked what they thought the change (if any) was mainly the result of. The options were as follows:

“What do you think this has been mainly the result of?

1. Mainly the result of the UK government's policies at Westminster

2. Mainly the result of the Scottish Government’s policies

3. For some other reason”

4. (Both Westminster and Scottish Government equally)”

When the survey was conducted face-to-face, the first three options were read out, but the fourth was not; the fourth option (‘Both Westminster and Scottish Government equally’) could only be coded if the respondent said it spontaneously to the interviewer. This aspect of the question presented a challenge for the web survey, because respondents cannot provide the answer spontaneously in the same way.

There were two possible approaches to address this. The first was simply include the fourth option alongside the others. However, presenting option four in this way would more accurately reflect a face-to-face design where this option was read out by the interviewer alongside the others, rather than one where the answer was not read out and could only be given spontaneously. The other approach was to hide this response option, and only have it appear should the respondent try and skip the question. This is similar to the way that ‘Prefer not to answer’ and ‘Don’t know’ were presented as response options throughout the survey, but may still be chosen by fewer people than when the answer could be given spontaneously.

To assess the impact of the two approaches on the pattern of response to the question, both were fielded as part of SSAS 2023. Half the sample were presented with the fourth option upfront, while for the other half the option was hidden and only presented if the respondent skipped the question.

Tables 15 and 16 show the distribution of responses between 2017 and 2023 for those who thought the standard of the health service had increased and decreased respectively. Note that in this instance we also report the results for the 2021/2 push-to-phone survey, which, as it was adminstered by an interviewer also enabled respondents to say ‘both Westminster and Scottish Government’ spontaneously.

Only a small proportion of respondents in 2023 stated that the standard of the health service had increased in the previous 12 months, so any differences in the pattern of response may be attributable to the low base sizes rather than the different ways the options were presented. This caveat notwithstanding, there was a small difference in the proportions selecting the fourth option: while no-one selected it when it was hidden, 4% did so when it was presented upfront alongside the other options.

If we examine the results for those who thought the standard of the health service had decreased, the difference in the pattern of response is much clearer. When presented with ‘both Westminster and Scottish Government’ upfront, just over 4 in 10 (42%) people selected it, whereas when it was not presented upfront fewer than 1% did so. Crucially when the fourth option was shown, the proportion selecting it was also notably higher than the results for 2017-2021/22, where the equivalent figures were between 7% and 15%.

A similar pattern is evident in the responses to the questions on who is responsible for changes in the standard of living in Scotland in the previous twelve months. Tables 17 and 18 show the distribution of responses for those who thought the standard of living had increased and decreased respectively.

When the option was presented upfront, 13% stated that ‘both Westminster and the Scottish Government’ were responsible for an increase in the standard of living in the previous 12 months, whereas when the option was hidden no-one selected it. The 13% figure is more in line with the figures for when the survey was conducted face-to-face, though as with the health service questions these results need to be caveated with the fact that the unweighted base sizes are low.

In comparison, for those who stated that the standard of living had decreased, 36% said this was the responsibility of both Westminster and the Scottish Government when this option was presented upfront, whereas only 1% did so when the option was hidden. The equivalent results between 2017 and 2021/22 were between 6% and 17%.

Focussing just on the results for those who said the standard of the health service or standard of living had decreased, where there is no need to caveat the results because of low base sizes, the general trend is that where the option is presented upfront the proportion selecting it is markedly higher than when the survey was conducted face-to-face, and respondents could only provide the response spontaneously. In contrast, where the option is hidden the proportion selecting it is markedly lower than when the survey was conducted face-to-face. Therefore the way the question is asked has a significant effect on the pattern of response – and in neither case does the pattern of response reflect that on previous face to face SSAS.

We would recommend that if SSAS is fielded using this design in future a version of the question that does not present the fourth response option upfront should be used. This is because the wording asks respondents what they think ‘this has mainly been the result of’. Hiding the ‘both’ option therefore prompts the respondent more strongly to make a decision on where they think responsibility lies, and therefore gets somewhat closer to the intended meaning of the question than the version that presents the fourth option upfront. Although a significant proportion of people selected the ‘both’ option when it was not hidden, many of them may still feel that one government is more responsible than the other. Therefore, hiding the response option more closely aligns the way the question is asked with what it was designed to achieve.