Scottish Social Attitudes Survey 2023 - Technical Report

Introduction

The purpose of the Scottish Social Attitudes Survey (SSAS) is to provide robust, high quality and nationally representative data over time on the Scottish public’s attitudes towards a range of social and political issues. SSAS has a modular structure, with a ‘Core Module’ which has been consistently commissioned by the Scottish Government since 2004^[i], and modules on different topics being commissioned by a number of funders each year. The Core Module typically consists of thirty questions on attitudes toward government and public services

SSAS was first fielded in 1999, at the beginning of Scottish devolution, and hitherto has been run as a face-to-face survey with in-home interviewing. During the COVID-19 pandemic, face-to-face interviewing was not feasible due to government regulations on social distancing. SSAS, in common with other large-scale surveys in Scotland, the rest of the UK and further afield, had to use alternative methods. However, this raised potential questions about the comparability of the data collected using these alternative methods with those obtained by previous surveys - and thus the integrity of any time series derived from questions asked repeatedly from year to year. A key consideration is therefore whether alternative methods result in samples with significatly different characteristics or change the way in which people answered questions.

SSAS did not run in 2020, while in 2021/22 it was run for the first time as a push-to-telephone survey, with no interviewing being conducted face-to-face in-home. Addresses were drawn at random from the Postcode Address File and up to two people at each address invited by post to undertake an interview by phone. Unfortunately, the change in methodology had adverse effects on the quality of the sample achieved and the final response rate was lower than anticipated. Even after the data were weighted, compared with previous surveys certain demographic groups, most notably graduates and supporters of the SNP, were over- or under-represented in the final sample. As a result, the results of the 2021/22 survey were not comparable with those of previous survey years. Consequently, one of the key advantages of SSAS as a survey, namely its two-decade time series of data on attitudes in Scotland, was negatively impacted by the methodological change. The 2021/22 SSAS technical report sets out the 2021/22 survey design and impact on the time series in further detail.

With the lifting of restrictions on face-to face interviewing following the pandemic, and again in common with other large-scale surveys in Scotland, the rest of the UK and further afield, consideration was given as to whether to return to face-to-face interviewing in people’s homes or whether to move SSAS to being administered online. While a number of large-scale surveys have returned to face-to-face interviewing, including those with a requirement for physical health measurement or building inspections such as the Scottish Health Survey and the Scottish House Condition Survey, a number of other surveys, including the British Social Attitudes Survey (BSA) and the European Social Survey, have taken the decision to move permanently to being conducted online.

At the beginning of the COVID pandemic, BSA, which has been run since 1983, moved from face-to-face interviewing to a push-to-web survey. Since then, a random selection of addresses have been sent a letter to complete the survey online and offered a monetary incentive for doing so. Up to two adults per household are eligible to take part. The change of mode had some impact on the character of the sample, but analysis of the differences determined that they did not significantly compromise the integrity of the many time series that are collected on BSA. Given these encouraging results, it was decided that a condensed version of the SSAS Core Module would be trialled as a push-to-web survey in 2023 in order to see whether a similar approach would be sucessful.

There are two main ways that the change of methodology may impact on or undermine the SSAS time series. Firstly, the nature of the achieved sample might be markedly different than in previous years. Secondly, the distribution of responses to attitudinal questions asked as part of the Core Module might be affected by the change of mode. In view of these potential risks, this report seeks to answer the following questions:

1) What impact, if any, has the change in survey mode had on the characteristics and representativeness of the achieved sample?

2) What impact, if any, has the change in mode had on the distribution of responses to certain attitudinal questions, especially those that needed to be adapted in order to be asked via the web?

3) To what extent is the time-series of long-running questions that have previously been asked on SSAS as part of the Scottish Government’s Core Module still robust, despite the change in mode?

Although the scope of the report is limited to the questions asked as part of the Core Module in 2023, the findings are of relevance to all future SSAS content fielded via a push-to-web design.