Scottish Social Attitudes Survey 2023 - Technical Report

Mode Change – Effects on Politics and National Identity Questions

This chapter covers effects of the change in mode on responses to the politics and national identity questions asked as part of SSAS. These questions do not form part of the Core Module and are not funded by the Scottish Government. However, attitudes towards the subject matter of the Core Module, that is, government, public services and democratic engagement, can be influenced by political preferences and identity and are often used in analysis of SSAS data for the Scottish Government and others. In addition, many of them have external measures (e.g. election results), with which they can be compared to evaluate the effectiveness of the face-to-face and push-to-web surveys in providing politically representative samples. This chapter therefore analyses whether the change in mode has had any impact on how representative the SSAS 2023 sample is politically.

The chapter first examines the figures for the questions on turnout at the 2021 Scottish Parliament election and the proportion who voted for each party, the accuracy of both of which can be verified by an external source. Then, the questions on national identity and position on the left-right scale are examined. The pattern of response to these questions on SSAS has hitherto been relatively stable over time.

Voter Turnout and Party Political Affiliation

Table 5 compares the weighted figures for the level of participation in the 2021 Scottish Parliament election reported by respondents to the 2023 survey with the actual outcome – together with the same information (collected via an identically worded question) for previous face-to-face SSAS conducted after the 2011 and 2016 Holyrood ballots.

It is common for polls and surveys to find that their respondents are more likely to say that they voted in an election than did so according to the official count. In part this is because the official count understates the level of turnout. This happens as a result of redundancy in the electoral register and because it excludes those who voted but whose ballot was deemed invalid. However, it can also arise either because those who participate in elections are more likely to respond to surveys or because respondents are reluctant to admit their failure to vote. While the latter pattern might be thought less likely to occur when respondents are not giving their answers to an interviewer, we might be concerned that the former trend might be more prevalent in a push-to-web sample than in one obtained face-to-face.

In practice, at 68%, the level of turnout in the 2021 election reported by our push-to-web survey is only five points above that recorded by the official count. This compares with figures of nine points in 2011 and 14 in 2016. This suggests, if anything, that the push-to-web sample is a little more representative on this criterion.

Table 6 compares the weighted distribution of the party for which people said they voted on the constituency ballot in the 2021 Scottish Parliament election with the actual outcome of the election, while Table 7 shows the equivalent figures for the list ballot. The same analysis is also shown for the 2011 and 2016 elections. It should be borne in mind that the information on the push-to-web survey was collected some two years after the 2021 election, whereas the 2011 and 2016 SSAs were both conducted relatively soon thereafter. As a result, there is a greater risk on the push-to-web survey that respondents may have misremembered how they voted at the last election.

Across the four parties that contested all of the constituency seats, together with all others combined, on average there is a difference between the actual and reported result of just over three points in the 2023 sample. Much of this difference arises because people were more likely to report voting for an ‘Other’ party – in practice, primarily the Greens - than did so, perhaps because they did not remember the Greens failed stand in all but 16 of the 73 single-member constituencies. Even so, the mean difference across all parties is little different from that recorded in the 2011 and 2016 surveys.

That said, at just over five points, the mean difference on the list vote is noticeably higher than the equivalent statistic for 2011 and 2016. A key reason is that reported support for the Greens is as much as eight points above that in the official result. Most of those who say they voted Green on the list vote reported having voted SNP on the constituency ballot, while the proportion who say they voted SNP on the list vote is six points down on the party’s official tally on that vote. It may be that in the wake of the coalition between the SNP and the Greens, some of those who voted SNP on the list indicated how, in the light of that alliance, they were now inclined to vote Green on the list vote. We might note that a similar pattern was found on two other surveys conducted by ScotCen after the 2021 election, both of which used a data collection mode other than push-to-web. One was the push-to-phone survey that carried the Scottish Government core module that year, the other a survey undertaken via the NatCen mixed mode random probability panel.^[xii] Given the very different way in which those two surveys and the push to web survey were conducted, it certainly seems unlikely that the discrepancy is evidence of a distinctiveness in the character of the push-to-web sample.

Attitudes towards Scotland’s constitutional status

A key political dividing line in Scotland is people’s attitudes towards the country’s constitutional status, that is, whether it should remain part of the UK or become an independent country. SSAS 2023 contains two measures of people’s attitudes towards the constitutional question, both of which are part of the background questions asked as part of SSAS and are not part of the Scottish Government funded Core Module. The first is how they say they would vote now in response to the question that was posed in the 2014 referendum, ‘Should Scotland be an independent country?’. The second is a question that presents people with five options: independence, either inside or outside the EU; a devolved Scottish Parliament, either with or without taxation powers; and Scotland being ruled from Westminster. This latter question has been asked on every SSAS since 1999. The distribution of responses to both questions on the 2023 survey and information on how they compare with those on the 2016, 2017 and 2019 face-to-face SSAS is shown in Tables 8 and 9. In the case of Table 9, those who chose either independence option have been combined, as have those who chose either devolution option.

Of course, in this instance it is possible that any difference between the distribution of responses on the 2023 survey and that on previous surveys may have been occasioned by a shift of opinion in the general population. However, we can refer to the evidence provided by other sources to assess whether this might be the case. Table 8 shows that respondents to the push-to-web survey in 2023 were evenly divided between those who said they would vote Yes to independence and those who report that they would vote No. This is in line with the evidence of recent opinion polls^[xiv]. Meanwhile, although lower than in 2019, the level of support for independence is similar to that on the 2016 and 2017 SSAS. This is not inconsistent with the evidence of the opinion polls that there has at most been no more than a marginal shift in the distribution of constitutional preferences since the EU referendum.^[xv] As we might anticipate, this pattern is also reflected in the pattern of responses to our long-running question on how Scotland should be governed.

National Identity

People’s attitude towards Scotland’s constitutional status is in part influenced by their sense of national identity. The pattern of response to the questions on national identity asked as part of SSAS has remained relatively stable over time. Consequently, if the weighted sample produced by the push-to-web survey is similar to that obtained by previous SSAS, we would expect the distribution of national identity to be much the same as in previous surveys. Table 10 reports the pattern of response to the ‘Moreno’ question on national identity, in which respondents are invited to identify which of a mixture of being British and Scottish best describes themselves.

There is little in the way of notable difference between the pattern of response in 2023 and that in earlier years. Indeed, in most instances the figures for 2023 are within a point or two of those obtained in 2019, differences that could well, of course, reflect no more than random sampling variation. That said, at 4%, the proportion who do not select any of the main options is five points down on that recorded in any other recent year. However, this is accompanied by similar increases on previous years both in the proportion who say they are wholly or mostly Scottish and in the proportion who say they are wholly or mostly British. Thus, while the different way of administering the survey may have somewhat depressed the proportion who chose to say ‘Other’, it appears not to have made any notable difference in the balance between those who chose primarily a Scottish identity and those who select primarily British identity.

SSAS also gathers information about national identity in a second way. In this instance people are presented with a list of the various national identities associated with the islands of Great Britain and Ireland and invited to choose as many as apply to themselves. In the event that they select more than one identity, they are then asked to state which one best describes themselves.

In practice by far the two most commonly chosen responses are ‘Scottish’ and ‘British’. Table 11, therefore, shows the proportion who included these two options among their answers. It will be seen that in both cases the proportion opting to do so is lower than that recorded on recent face-to-face SSAS. As Table 12 further below shows, this reflects the fact that respondents to the push-to-web survey were less likely than those on previous interviewer administered SSAS to take the opportunity to select more than one identity. Indeed, whereas on the 2017 SSA 77% said they were both British and Scottish, and in 2019 67% did so, in the 2023 push-to-web survey, the proportion fell to 52%.

The British Social Attitudes survey also asks this question on national identity. A similar increase in the proportion of respondents who select just one option is also to be observed on that survey, since its transition to a push-to-web methodology. On average, 57% have selected just one option on the three BSA surveys that have asked this question since the change of method, compared with an average of 53% in the three face-to-face surveys conducted between 2017 and 2019. Moreover, this difference is even more notable – and similar to that in Table 12 - if we focus solely on Scottish respondents to BSA in the same period. On average, 65% of Scottish respondents on BSA have selected one option in the last three push-to-web surveys, compared with an average of just 50% who did so in the last three surveys that were conducted face-to-face.

This suggests that the difference in the number of multiple responses reported in Table 12 is a consequence of the change in the way in which the survey is administered. In face-to-face surveys it is standard practice for interviewers, when reading out multiple response questions, to prompt respondents by asking them if there are any further answers they would like to give to the question. In the absence of these prompts it is understandable that fewer respondents selected more than one option when the survey was administered online.

This pattern also appears to have had some impact if we examine the distribution of people’s ‘forced choice’ identity. For those who initially acknowledged more than one identity, their forced choice identity is the identity they chose when subsequently asked to name just one. For the remainder, their forced choice identity is whichever single identity they selected. Table 13 shows that when calculated in this way, the proportion who are classified as Scottish is somewhat lower than on previous SSAS while the proportion who emerge as British is a little higher.

When this question is administered, the option ‘British’ is the first to be shown in the list of responses, while Scottish is the sixth, a reflection of how they were displayed on a show card on previous SSAS/. One possible consequence is that some respondents selected ‘British’ and then did not necessarily pay as much attention to the other items on the list. Indeed, those who on the push-to-web survey said they were British were 14 points less likely than the equivalent group on the 2019 SSA to select more than one identity. Crucially, this figure is higher than the equivalent figure (ten points) among those who said they were Scottish. Consideration may need to be given to the order in which the possible responses to this question are displayed in future surveys.

Position on left-right scale

Finally, a key dividing line in both Scottish and British politics is between those on the ‘left’ and those on the ‘right’. Those on the ‘left’ are those who are concerned about inequality and believe that the government should reduce it, while those on the ‘right’ feel it is more important to provide the incentives to investment that will encourage economic growth from which all can prosper.^[xvi] Since 1999, SSAS has measured where people stand on this divide by administering a series of agree-disagree questions about the extent of inequality and what the government should do about it. As we might anticipate from a measure designed to obtain an underlying value orientation, for the most part, the pattern of responses has varied relatively little from one year to the next.

Table 14 shows that the proportion of people who emerge as being on the ‘left’ and on the ‘right’ is little different on the push-to-web survey than it has been on the last three SSAs. It would seem that in this respect at least the push-to-web survey has secured a sample whose character is similar to that obtained by previous SSAS.