Scottish Social Attitudes Survey 2023 - Technical Report

Conclusion

The primary purpose of fielding SSAS using a push-to-web mode in 2023 was to determine whether the results obtained through an online survey would be comparable with those of previous face-to-face surveys. The results presented in this report provide encouraging evidence that SSAS can be fielded as a push-to-web survey in future and the long-standing time series largely maintained.

Sample

Prior to weighting, there were some differences between the demographic composition of the push-to-web sample and that of previous face-to-face surveys. Most notably there was a smaller proportion of over 65s, a larger proportion of women and a larger proportion of those living in the least deprived SIMD quintile. However, after weighting the demographic data are similar to those obtained in the last three face-to-face surveys conducted in 2016, 2017 and 2019. In repect of area deprivation, age and respondent’s sex, the survey provided a final weighted sample that is broadly reflective of the Scottish population – and this has been achieved without significantly stronger weighting than was required for the face-to-face surveys. At 64% the weighting efficiency of the sample was similar to that of recent face-to-face samples.

However, one key apparent disadvantage of the web mode was the noticeable drop in the proportion identifying as having a long-term health condition or disability: the weighted figure was 26% in 2023 compared with 40% when the survey was last conducted face-to-face. It is likely that a significant drop of this kind is at least in part a consequence of the change in mode, though other factors such as a difference in the position of the question in the survey may well have played a role. Unfortunately, it is not possible to tell from the data whether this reduction in the proportion of those who report having a long-term health condition/disability is a result of them being less likely to opt-in to an online survey, or whether those with a long-term health condition/disability were less likely to identify as such after opting-in. However, both issues should be considered in future if SSAS is to be conducted online to ensure the survey is accessible for everyone. For example, the wording of the question and its position in the questionnaire should be reconsidered carefully to reduce any questionnaire design effects to a minimum. Survey materials (invitation letters, leaflets etc.) could also be examined and potentially redesigned to try and improve the participation rate of those with a long-term health condition/disability as much as possible.

Adapted questions

The exercise was instructive with regards to questionnaire design. Questions that, when asked by an interviewer, allowed a response option that was not offered to respondents to be given spontaneously by the respondent present a challenge when asked on an online survey. When the additional response option is shown upfront along with the others, more respondents select it than would typically do so in a face-to-face survey when it is not read out by the interviewer and can only be offered as a spontaneous response. When the option is not shown upfront but only appears if the respondent skips the question (alongside ‘Don’t know’ and ‘Prefer not to say’) fewer respondents select it than would typically do so in a face-to-face survey. We would therefore advise that the results from these questions are not directly comparable with those obtained in the face-to-face surveys, given the significant differences in question design and resulting large differences in the spread of responses. We would also advise that, if these questions are to be fielded in future online surveys, the fourth option (‘Both Westiminster and the Scottish Government’) is hidden and presented to the respondent if they skip the question. This version more closely aligns with the purpose of the question, which is concerned with who is mainly responsible for changes in the standard of living and of the health service

Response patterns

Adminstering the survey via the web also had some impact on the pattern of response to questions on national identity. Respondents, when given a free choice to select as many or as few national identities as they wished, were less likely to select more than one than they were when the survey was conducted face-to-face. A similar pattern was noted when BSA moved to an online mode. It also appears to be the case that in selecting just one identity respondents were more likely to select an identity that appeared towards the top of the list that was offered. It might be wise to randomise the order of the response options, should SSAS be fielded online in future. The majority of questions in the Core Module, however, require minimal to no adaptation for an online survey.

As in the case of demographics, the data collected on party political support and independence were generally encouraging in terms of representativeness. The mean error for the constituency vote in the 2021 Scottish Parliament election was similar to that obtained when the same question was asked face-to-face in 2011. It was also not widely dissimilar to the mean error obtained in 2016, when SSAS was fielded in the same year as the election. The mean error for list vote was notably higher, but this is likely more attributable to recall error and to the current participation of the Greens in the SNP-led Scottish Government than it is to the change in survey mode. The figures obtained for turnout at the election were the most accurate SSAS has obtained in recent years, and results for the left-right scale provide no cause for concern.

Recommendations

On balance, the findings outlined in this mode report are a valuable addition to our understanding of the impact of changes in survey mode and we would advise that, a push-to-web method is a viable option for fielding SSAS in future. We would also recommend the following:

• The nature of the achieved sample and the weighting efficiency figure indicates that all questions in the Core Module, with the exception of those on assigned responsibility for changes in the standard of living/health service, can be compared with existing time series.
• The questions on assigned responsibility for changes in the standard of living and health service cannot be compared with the existing time series, and the version of the question that hides the ‘Both Westminster and Scottish Government’ should be used if SSAS is fielded online in future.
• Consideration should be given to maximise representation of those with a long-term health condition/disability in future SSAS online surveys, given the apparent reduction in their representation when compared with the face-to-face surveys.