Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): Mode Effect Study Report 2015
Findings of a mode effect study conducted during the 2015 SALSUS survey to see whether the different routes of administration resulted in different data.
3 Did the mode affect the representativeness of the survey?
This chapter first looks at the response rates for the online and paper samples before going on to discuss the impact this had on the sample profiles.
In survey research, higher response rates tend to increase representativeness i.e. the extent to which the achieved sample reflects the population of interest (in this case, 13 and 15 year olds across Scotland). While this is generally true, lower response rates are only problematic if there is something systematically different about those who respond to the survey compared with those who do not in a way which biases the results.
This is particularly important for the mode effect study as, in order to identify a mode effect, the online and paper samples needed to be comparable. If they were not, then it would not be possible to know whether any differences in the key measures were due to the mode or to a difference in the sample profiles.
Response rates
Response rate - key findings
- The school, class and overall response rates were lower in the online sample than in the paper sample
- There were no differences in pupil response rate between the online and paper samples
- Differences in response rate between the online sample and the paper sample varied by local authority
School response rates
At the start of recruitment, 175 schools were invited to complete the survey online and 176 schools were invited to complete the survey on paper. Eight schools did not want to complete the survey online but were willing to do so on paper. As a result, seven schools from the paper sample were swapped to the online sample to replace these schools. This meant that the final online sample contained 174 schools and the final paper sample contained 177 schools.
The school response rate was 72% for the paper sample (128 out of 177 schools) and 61% for the online sample (107 out of 174 schools). This shows a clear difference in the school response rate between the paper sample and the online sample. While the paper response rate is in line with that of the last wave of the survey in 2013, the online response rate is 11 percentage points lower (see Table 3.1). This suggests that the prospect of completing the survey online is less appealing to head teachers and there is potential for lower response rates in future waves if the survey moves online.
Table 3.1 Response rates broken down by mode and year
| 2013 paper | 2015 paper | 2015 online | 2015 overall | |
|---|---|---|---|---|
| School response rate | 71% | 72% | 61% | 67% |
| Class response rate | 68% | 68% | 53% | 61% |
| Pupil response rate | 90% | 88% | 87% | 87% |
| Overall response rate | 60% | 60% | 46% | 53% |
- The school response rate shows the proportion of schools in each sample that agreed to participate in the survey, and then administered the survey to at least one class. It does not include those who initially agreed but did not complete any surveys.
- The class response rate shows the proportion of the total number of sampled classes that participated in the survey. As class participation is dependent on the school completing the survey, it is largely driven by the school response rate. However, it also takes into account where some sampled classes in a participating school completed the survey and others did not.
- The pupil response rate is the proportion of pupils in participating classes who completed the survey.
- The overall response rate is the product of the class response rate and the pupil response rate. As the survey uses a class based sample design, the class response rate, rather than the school response rate is used.
Schools completing the survey online were also less likely to complete the survey once they had agreed to take part than schools in the paper sample. Eighty-seven per cent of schools in the paper sample who originally agreed to participate went on to complete at least one class, compared with 76% of schools in the online sample. This suggests that it is more difficult for some schools to complete the survey online.
Some local authority areas had the same or similar response rates for both modes but others differed greatly (see Table 3.2). It should be noted however that, due to the small number of schools in some local authorities, the participation of even one school can influence response rates greatly. Perth & Kinross, Falkirk and City of Edinburgh all had particularly low online response rates when compared with their paper response rate. In contrast, Scottish Borders, East Lothian and Argyll & Bute had higher online response rates. There was also a higher online response rate in Glasgow. The relatively high number of pupils in Glasgow means that this may have had some impact on the sample profile.
Table 3.2 School response rates by local authority [9]
| Paper | Online | |||||
|---|---|---|---|---|---|---|
| Schools sampled | Schools completed | % completed | Schools sampled | Schools completed | % completed | |
| Aberdeen City | 7 | 6 | 86% | 8 | 5 | 63% |
| Angus | 4 | 3 | 75% | 4 | 4 | 100% |
| Argyll & Bute | 5 | 2 | 40% | 4 | 3 | 75% |
| Dumfries & Galloway | 8 | 3 | 38% | 8 | 5 | 63% |
| Dundee City | 5 | 3 | 60% | 5 | 3 | 60% |
| East Dunbartonshire | 3 | 3 | 100% | 5 | 2 | 40% |
| East Lothian | 5 | 3 | 60% | 2 | 2 | 100% |
| East Renfrewshire | 3 | 3 | 100% | 4 | 2 | 50% |
| Edinburgh, City of | 17 | 13 | 76% | 17 | 6 | 35% |
| Falkirk | 5 | 3 | 60% | 3 | 1 | 33% |
| Fife | 10 | 8 | 80% | 10 | 8 | 80% |
| Glasgow City | 18 | 8 | 44% | 19 | 12 | 63% |
| Highland | 12 | 12 | 100% | 11 | 6 | 55% |
| Inverclyde | 3 | 3 | 100% | 4 | 4 | 100% |
| Midlothian | 3 | 1 | 33% | 3 | 2 | 67% |
| Moray | 4 | 4 | 100% | 5 | 3 | 60% |
| North Ayrshire | 4 | 4 | 100% | 5 | 3 | 60% |
| North Lanarkshire | 12 | 9 | 75% | 12 | 8 | 67% |
| Orkney Islands | 1 | 1 | 100% | 1 | 1 | 100% |
| Perth & Kinross | 7 | 5 | 71% | 7 | 1 | 14% |
| Renfrewshire | 6 | 3 | 50% | 5 | 2 | 40% |
| Scottish Borders | 5 | 3 | 60% | 5 | 5 | 100% |
| Shetland Islands | 2 | 2 | 100% | 1 | 1 | 100% |
| South Ayrshire | 4 | 4 | 100% | 4 | 3 | 75% |
| Stirling | 4 | 3 | 75% | 4 | 3 | 75% |
| West Dunbartonshire | 2 | 2 | 100% | 3 | 3 | 100% |
Class response rates
The class response rate is driven to a large extent by the school response rate, so it is inevitable that it is lower for the online than the paper sample (see Table 3.1). 345 out of an invited 649 classes completed the survey online and 477 out of an invited 705 classes completed the survey on paper. This means that, in total, 822 classes out of 1,345 classes completed the survey.
The fieldwork monitoring figures suggest that, in addition to fewer schools participating in the online survey, those that did were less likely to follow up on missing classes. One of the main findings from the electronic pilot was that completing the survey in ICT suites required a greater amount of advanced planning than completing the paper survey. It could be that quickly following up on missing classes in response to a reminder from the survey contractor is less feasible when using an online methodology. Furthermore, some schools had prelims for the National 5 and 6 courses in January (the fieldwork contingency period) which meant that many of the ICT suites had already been booked.
Pupil response rates
7,125 out of a possible 8,231 pupils completed the survey online and 9,786 out of a possible 11,170 pupils completed the survey on paper. This meant that a total of 16,911 pupils out of a total of 19,401 pupils completed the survey. There was no difference in pupil response rate by mode (see Table 3.1).
Sample profile
This next section looks at the sample profile for those completing the survey online, compared with those completing on paper and compared with the national profile.
While the overall response rate is lower for the online sample than the paper sample, this is not necessarily an issue. Lower response rates are mainly a problem if this introduces bias into the results by skewing the profile of respondents in ways that affect the survey measures.
Sample profile - key findings
- There were some differences between the online and paper samples in relation to the Scottish Index of Multiple Deprivation ( SIMD), school sector, school denomination and rurality. However, this disappeared when school clustering was taken into account.
- While the online and paper sample differed from each other to some extent, both were reasonably in line with the national profile
- There was greater variation at a local level, with some areas more likely to complete the survey on paper and others more likely to complete the survey online
There were some differences between the online and paper samples (Table 3.3), but these differences were related to how pupils were clustered within schools. This was mainly due to the fact that schools in some local authorities were more likely to agree to the online survey and others to the paper survey (see Table 3.4). However, once the clustering of schools was taken into account, there were no substantial differences in pupil characteristics as a result of data collection method.
Table 3.3 Sample profile
| Scotland | Paper | Online | Combined | |
|---|---|---|---|---|
| Gender | ||||
| Girls | 49.7% | 50.0% | 49.6% | 49.9% |
| Boys | 50.3% | 50.0% | 50.4% | 50.1% |
| Year group | ||||
| S2 | 49.2% | 54.5% | 53.4% | 54.0% |
| S4 | 50.8% | 45.5% | 46.6% | 46.0% |
| School sector | ||||
| State | 94.3% | 94.1% | 94.9% | 94.4% |
| Independent | 5.7% | 5.9% | 5.1% | 5.6% |
| School denomination | ||||
| Non-denominational | 82.3% | 86.9% | 80.3% | 84.2% |
| Roman Catholic | 17.7% | 13.1% | 19.7% | 15.9% |
| Rurality | ||||
| Urban | 81.9% | 76.3% | 81.2% | 78.4% |
| Rural | 18.1% | 23.7% | 18.9% | 21.7% |
| SIMD | ||||
| 1 - most deprived | 20.1% | 17.8% | 25.3% | 21.0% |
| 2 | 18.4% | 18.4% | 18.3% | 18.4% |
| 3 | 19.2% | 17.4% | 14.7% | 16.3% |
| 4 | 21.2% | 21.3% | 19.4% | 20.5% |
| 5 - least deprived | 21.2% | 25.1% | 22.2% | 23.9% |
Analysis was conducted to explore any differences using chi-squared tests. There were no differences between the online and paper sample in terms of the age or gender of pupils. There were statistically significant differences between the two samples in relation to SIMD, school sector, denomination and rurality.
A regression model was used to check whether gender, age, school sector, denomination, rurality and SIMD were significantly related to the probability of having completed the survey online or on paper, while taking into account that pupils were clustered within schools. The results show that these variables did not have a significant impact on the chance of having completed a questionnaire online or on paper.
It is important to note that while the two samples differed on some characteristics, both were reasonably in line with the national profile. For some sample characteristics (rurality, denomination and year group), the online sample was more representative than the paper sample.
Table 3.4 shows the local authority profile of Scotland and of the combined, paper and online survey samples. For most local authorities, the likelihood of completing the survey on paper or online was similar. However, there were a number that showed more notable differences: City of Edinburgh, Perth & Kinross and, to a lesser extent, North Lanarkshire were all more likely to complete the survey on paper than online. Glasgow and, to a lesser extent, Fife were more likely to complete the survey online than on paper.
Table 3.4 Local authority profile [10]
| Scotland | Combined | Paper | Online | |
|---|---|---|---|---|
| Aberdeen City | 4.1% | 4.9% | 4.8% | 5.0% |
| Angus | 2.8% | 2.4% | 1.9% | 3.0% |
| Argyll & Bute | 1.9% | 1.5% | 1.6% | 1.4% |
| Dumfries & Galloway | 3.6% | 2.2% | 1.8% | 2.8% |
| Dundee City | 3.2% | 2.9% | 2.9% | 2.8% |
| East Dunbartonshire | 3.1% | 1.8% | 2.0% | 1.6% |
| East Lothian | 2.4% | 2.6% | 2.6% | 2.6% |
| East Renfrewshire | 3.3% | 2.4% | 2.8% | 1.8% |
| Edinburgh City | 7.9% | 12.4% | 14.7% | 9.5% |
| Falkirk | 3.8% | 2.0% | 2.3% | 1.6% |
| Fife | 8.8% | 9.1% | 7.9% | 10.8% |
| Glasgow City | 11.3% | 14.9% | 10.0% | 21.2% |
| Highland | 5.9% | 5.0% | 6.0% | 3.8% |
| Inverclyde | 1.9% | 2.6% | 2.0% | 3.3% |
| Midlothian | 2.3% | 0.8% | 0.4% | 1.2% |
| Moray | 2.4% | 1.9% | 1.8% | 2.0% |
| North Ayrshire | 3.4% | 2.7% | 3.1% | 2.2% |
| North Lanarkshire | 9.0% | 9.4% | 10.7% | 7.8% |
| Orkney Islands | 0.5% | 0.6% | 0.4% | 0.9% |
| Perth & Kinross | 3.4% | 4.6% | 7.6% | 0.7% |
| Renfrewshire | 4.3% | 2.1% | 2.6% | 1.3% |
| Scottish Borders | 2.8% | 2.9% | 1.9% | 4.1% |
| Shetland Islands | 0.6% | 0.7% | 1.0% | 0.3% |
| South Ayrshire | 2.6% | 3.1% | 3.2% | 2.9% |
| Stirling | 2.5% | 2.2% | 2.1% | 2.3% |
| West Dunbartonshire | 2.2% | 2.5% | 2.0% | 3.2% |
Moving to online administration may result in a reduced response rate but the analyses suggest that this would not affect the representativeness of the sample at a national level. However, a greater impact has been demonstrated at a local level. SALSUS has tended to be run every two years, providing local level results (with a larger sample size) every other wave (i.e. every four years).The local level differences have implications for how the survey is administered depending on whether national results or local level results are required.
Contact
There is a problem
Thanks for your feedback