Scottish Social Attitudes Survey 2023 - Technical Report

Mode Change: Effect on Demographics of Sample

To analyse the potential effect of the change of mode on the achieved sample in 2023, we compare the nature of the online sample across several key demographics with that achieved when the survey was conducted face-to-face. In general, the results are encouraging, and did not show a marked discontinuity. Table 2 provides the weighted and unweighted figures for respondent age on the 2023 web survey and the last three years that SSAS was conducted face-to-face.^[viii] The unweighted figures are presented in brackets.

The most noticeable difference between the online survey and previous SSAS in the unweighted data is the lower proportion of those aged 65 and over. The proportion in this age group was three in ten (30%) in 2016, 31% in 2017, and a third (33%) in 2019. However, it was only around a quarter (26%) in 2023. As might have been expected, a web mode appears to have led to a reduction in the proportion of older respondents. That said, this simply meant that the proportion of older respondents was closer to the population figure. As a result, the weights were smaller for the 65+ group in 2023 than they have been in the face-to-face surveys.

Meanwhile, the age composition of the 2023 survey was similar to the 2016, 2017, and 2019 figures after weighting was applied. The final weighted sample had nearly double (23%) the proportion of over 65s than it did those in the 16-24 bracket (12%). This is similar to when the survey was conducted face-to-face: 23% of respondents were aged 65 and over in 2019 and 10% were aged between 16 and 24, while the equivalent figures for 2017 were 22% and 14% respectively and for 2016, 23% and 11%.

Table 3 shows the weighted and unweighted results for sex of respondent, with the unweighted figures shown in brackets. At 58%, the proportion of women in the sample before weighting was applied is higher than in 2019 (54%), 2017 (53%) and slightly higher than in 2016 (56%). However, after weighting the figures for each of the years are the same. 52% of the respondents were women and 48% were men This means that here the weights are having to do slightly more work in 2023 than in the face-to-face years.

The characteristics of the online and face-to-face samples can also be compared according to the Scottish Index of Multiple Deprivation score of the area in which the participant lived. Household surveys tend to underrepresent those individuals living in areas of high socio-economic deprivation, and overrepresent those from more affluent areas. Before weighting was applied, the proportion of those living in the least deprived quintile in 2023 (26%) was higher than in each of the previous face-to-face years, indicating an overrepresentation of those living in wealthier areas. That being said, the proportion living in the most deprived quintile (22%) was also higher than in the face-to-face samples. The web survey appears in fact to have been responded to least readily by those in the middle of the SIMD quintiles.

However, Table 4 shows that, after weighting, SSAS 2023 achieved a fairly representative sample with respect to SIMD score, as was the case when the survey was conducted face-to-face. As the score is divided into quintiles – a completely representative sample would have a 20% score in each row.

There is, however, one area where the distribution of the data was notably different in the web survey from that in previous SSAS. The unweighted figure for the proportion with a long-term health condition or disability in 2023 is 30%, whereas in 2016, 2017 and 2019 the figure was between 44% and 46%. This suggests that the online survey is potentially underrepresenting those with a long-term health condition or disability. The most recent data from the Scottish Health Survey, which is conducted using face-to-face interviewing, indicates that the prevalence of disabilities/long-term health conditions in the Scottish population was 48% in 2022.^[x]

Weighting the data does not change this picture. After weighting was applied, the proportion with a long-term physical or mental health condition drops to around a quarter (26%). This is significantly lower than in the equivalent figure in previous years. In 2016, the figure was 41%, while the weighted figures for 2017 and 2019 were 43% and 40% respectively. Both before and after weighting SSAS, therefore, came closer to the Scottish Health Survey figure when it was conducted face-to-face than when it was conducted online.

Several factors may have caused this discrepancy, including not least the change in mode from face-to-face interviewing to a web survey. Those living with a disability or long-term health condition are more likely to face barriers to accessing digital and online services than non-disabled people. They are also more likely not to have the skills required to access the devices needed to get online.^[xi] When the interview was conducted face-to-face the respondent did not need to have any digital skills or access to the internet to take part, and the interviewer could guide them through both the questions and any areas of the interview that may not have been clear. The existence of a potential digital barrier and the absence of an interviewer there to help the respondent may have potentially caused a drop in those with a long-term health condition/disability taking part in the online survey.

However, it should be noted that there is no consistent support for this expectation from BSA. Prior to weighting 37% of respondents to that survey in 2018 said that they had a long-term health condition or disability, as did 34% in 2019. In contrast, the figure was 29% in 2020. However, since then it has continued to be between 33% and 34%, in line with the figure in 2019. Meanwhile, after weighting, the figures were 33% in 2018 and 29% in both 2019 and after the move to web in 2020, and between 33% and 34% thereafter.

Another possible explanation for the sharper drop on SSAS is questionnaire order. In the previous face-to-face SSAS in 2016, 2017 and 2019 the question was preceded by one on the respondent’s general health. The same is also true of the Scottish Health Survey. This may have primed the respondent to be thinking more readily about long-term health conditions and disabilities than would otherwise have been the case. In contrast, on SSAS 2023 the question was preceded by one completely unrelated to health, as, incidentally, has always been the case on BSA.

We should also note that the question that was asked to ascertain whether someone had a long-term health condition was somewhat different from that which has previously been used on SSAS. In line with the wording that has been used on BSA surveys since 2021, the question asked in 2023 read, ‘Do you have any physical or mental health conditions or illnesses lasting or expected to last 12 months or more?’. However, until now the question asked on SSAS has read, ‘Do you have any long-term illness, health problems or disability?. By long-term we mean that it can be expected to last for a year or more?’ However, it can only be a matter of supposition as to whether this seemingly modest change of wording might have made a difference to the pattern of response.

Further discussion on the apparent underrepresentation of those with a disability or long-term health condition and how it may be addressed is included in the chapter ‘Relationship between Disability/Long-term health condition and National Identity variables with Core Module Questions’ and in the conclusion to this report.

Weighting Efficiency

The demographic composition of the achieved sample feeds into a ‘weighting efficiency’ score. Weighting efficiency effectively provides an indication of how much adjustment is required to align the characteristics of the unweighted figures in a sample with the characteristics of the population it is aiming to represent. A weighting efficiency score of 100% would indicate that no adjustment was needed by the weights to make the sample align with the wider population. A lower weighting efficiency score indicates the weights are having to do more work to get the achieved sample to reflect the population – in other words, the unweighted sample is more biased. The weighting efficiency score for SSAS 2023 was 64%, which is broadly comparable with the score achieved in 2019 (66%) and 2017 (70%). This slight downward trend in weighting efficiency is comparable with other attitudinal surveys, such as BSA, which had a weighting efficiency score of 76% in 2018 and 77% in 2019, compared with a score of 74% in 2020 followed by scores of between 64% and 67% in 2021-2023. It is also a marked improvement on when the survey was conducted using a push-to-phone methodology in 2021/22, when the weighting efficiency score was 50%. This score provides encouraging evidence that a web design can provide a sample that is as broadly representative of the wider population as that achieved through a face-to-face method.