Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS) 2015: Online Feasibility Study

4 Can SALSUS be run online?

Key points

Overall, the majority of local authorities (24 out of 32) would be able to complete SALSUS 2015 online. While 8 local authorities were classified as possibly being problematic, this was due to caution over a lack of response from liaison teachers.

The most likely means for running SALSUS 2015 will be on PCs in ICT suites.

On balance, the results suggest that an online survey could result in a reduced response rate.

Participants highlighted a number of potential risks when completing the survey online including timetabling issues, a lack of computers and software compatibility. However, a number of options to overcome these obstacles were also mentioned.

Generally, liaison teachers thought that moving the survey online would be more enjoyable for pupils. Only a small proportion thought it would have a negative impact on pupils.

Where can SALSUS run online in autumn 2015?

4.1 All local authority IT officers reported that every secondary school in their council area would have the technical capacity to participate in an online version of SALSUS 2015.

4.2 Assuming there was sufficient advance notice and testing of the survey link, all indicated that administering the survey would be straightforward and should pose no serious problems. Many noted that they had run online surveys in the past, or that schools in their area had successfully taken part in other national online studies (e.g. the Scottish Survey of Literacy and Numeracy) or large-scale online assessments exercises.

4.3 However, headteachers and liaison teachers held more mixed views. While a small majority (59%) of headteachers said that their school would have the technical capacity and capability to do so online in the autumn of 2015, 25% reported that they would not, and 15% did not know.

4.4 Seventy-nine per cent of liaison teachers thought that it would be feasible for their school to administer the survey.

4.5 As the findings from the IT officers and individual schools are not always in agreement, it is unlikely that any one measure will give the true picture. In order to better appreciate the findings across all of the research strands, the survey results were compared for each local authority (see Table 4.1).

Table 4.1 - Summary of feasibility research strands

	Headteacher survey	Local Authority IT officer survey	Liaison teacher survey[9]	Number of schools in the local authority	Ability to complete online
Local Authority 1	78%	Y	100%	>15	Good
Local Authority 2	64%	Y	75%	>15	Good
Local Authority 3	50%	Y	100%	<9	Good
Local Authority 4	100%	Y	100%	9-14	Good
Local Authority 5	100%	Y	100%	>15	Good
Local Authority 6	67%	Y	83%	9-14	Good
Local Authority 7	100%	Y	67%	<9	Good
Local Authority 8	57%	Y	50%	>15	Good
Local Authority 9	50%	Y	67%	9-14	Good
Local Authority 10	75%	Y	50%	>15	Good
Local Authority 11	54%	Y	71%	>15	Good
Local Authority 12	55%	n/r[10]	88%	>15	Good
Local Authority 13	50%	Y	100%	9-14	Good
Local Authority 14	100%	Y	n/r	>15	Good
Local Authority 15	75%	Y	75%	9-14	Good
Local Authority 16	56%	Y	75%	>15	Good
Local Authority 17	50%	Y	50%	9-14	Good
Local Authority 18	67%	Y	50%	9-14	Good
Local Authority 19	0%	Y	100%	9-14	Mixed
Local Authority 20	100%	Y	0%	<9	Mixed
Local Authority 21	27%	Y	80%	>15	Mixed
Local Authority 22	0%	Y	100%	<9	Mixed
Local Authority 23	33%	Y	50%	<9	Mixed
Local Authority 24	33%	Y	50%	9-14	Mixed
Local Authority 25	n/r	Y	n/r	<9	Poor/DK
Local Authority 26	50%	Y	33%	<9	Poor/DK
Local Authority 27	40%	Y	n/r	<9	Poor/DK
Local Authority 28	0%	Y	0%	<9	Poor/DK
Local Authority 29	67%	Y	n/r	<9	Poor/DK
Local Authority 30	50%	Y	n/r	9-14	Poor/DK
Local Authority 31	33%	Y	n/r	9-14	Poor/DK
Local Authority 32	33%	Y	n/r	<9	Poor/DK

4.6 The local authorities were classified into three groups - local authorities in which the majority of schools seem able to complete SALSUS online in 2015 (18 local authorities), local authorities where the picture is mixed and may require further work getting either headteachers or liaison teachers to buy into the survey (6 local authorities) and local authorities in which it may be problematic to administer SALSUS online (8 local authorities). We have been cautious in our assumptions and, for example, have classified Local Authority 29 as potentially problematic because we had no response from liaison teachers in that area (rather than knowing for certain that it will be problematic).

What is the most likely means for running SALSUS online?

4.7 Taking the views of IT officers together with those of liaison teachers (headteachers were not asked about this), the most likely means for running SALSUS online in 2015 will be administering the survey in ICT suites.

4.8 Twenty-one IT officers said this would be the most likely way, seven said it would be a mix of ICT suites and laptops in classrooms and two said it would be mainly laptops in classrooms.

4.9 Seventy-three per cent of liaison teachers indicated that it would be 'definitely' or 'probably' feasible to administer the survey in ICT suites. Fewer thought that it would be possible to administer the survey using either laptops/netbooks (35%) or tablets (17%). Only a very small percentage thought that it would be possible to administer the survey on tablets or laptops but not in ICT suites. See Table 4.2 below.

Table 4.2: Liaison teachers' views on administering SALSUS online in 2015

On balance, do you think it would be feasible for your school to administer SALSUS in the autumn of 2015 in Personal and Social Education (PSE) classes under exam conditions using…?

	Yes - definitely	Yes - probably	No - probably not	No - definitely not	Don't know	Base
ICT suites	16%	57%	15%	8%	4%	79
Tablet devices	2%	15%	25%	57%	2%	61
Netbooks/laptops	6%	29%	22%	40%	3%	63

4.10 Liaison teachers who had said 'yes' to more than one option were asked for their preferred method. Eighty-eight per cent indicated a preference for ICT suites. This reflects past experience, with 83% reporting that the last time they ran an online survey it was administered in an ICT suite while 14% said that it was completed using laptops or netbooks.

4.11 IT officers were not explicitly asked about 'bring your own device' (allowing pupils to use their own personal devices to access the internet in classrooms) but representatives from three local authorities mentioned this spontaneously as a possible method for completing the survey. However, the most commonly used device in these circumstances is a smartphone, which is not suitable for SALSUS as the survey is too long for a small screen.

What is the likely impact on school response rates?

4.12 It is important to bear in mind that, while schools may have the capacity and capability to administer SALSUS online, it does not necessarily mean that they will. As with most school surveys, participation is at the discretion of the headteacher.

4.13 On balance, the results suggest that an online survey could result in a reduced response rate.

4.14 Headteachers and liaison teachers were asked if their school would be more or less likely to participate in an online version of SALSUS. While 39% of headteachers said that a move online would make no difference, 34% reported that they would be less likely to participate and 27% said they would be more likely. Because the SALSUS school response rate is already very high (71% in 2013), this is potentially more negative than it might at first appear because there is more scope for schools to move from taking part to not taking part, than to move from not taking part to taking part.

4.15 There were a number of local authorities where higher proportions of headteachers said that they would be less likely to participate online: East Ayrshire, Eilean Siar and Orkney (all 100%), Renfrewshire and West Dunbartonshire (both 67%), and North Lanarkshire (64%).

4.16 Similarly, 43% of liaison teachers expected that a move to online would make no difference to their school's likelihood to participate, while 28% expected that their school would be less likely to take part, and 21% said that their school would be more likely to take part. Liaison teachers were more likely to report their school being less likely to participate in North Lanarkshire (100%), and Angus and East Dunbartonshire (both 67%).

What are the potential problems or risks posed by running SALSUS online?

4.17 This section examines some of the most likely problems which could arise as a result of administering SALSUS online, how much of a risk these might be, and whether they can be resolved.

4.18 There are two main risks:

• Schools decide not to take part because the survey is online (which affects the response rate)
• Schools take part, but they can't complete the survey or it does not work.

4.19 There are a number of potential problems which underpin both of these risks. Table 4.3 summarises some of the most common problems encountered by liaison teachers when they last administered an online survey in school.

Table 4.3: Liaison teachers: problems encountered in most recent online survey

Q. When the survey took place, did the school encounter any of the following issues?

	Yes - a major issue	Yes - was an issue, but overcome/fixed	No	Base
The survey site was blocked by a firewall or filter	7%	18%	75%	61
There were problems with software compatibility	10%	17%	73%	63
The number of pupils taking part took up too much bandwidth and made the survey run slowly	13%	21%	66%	61
It was more difficult to manage the survey under exam conditions	10%	27%	63%	62
There were accessibility issues for pupils with additional support needs	6%	32%	61%	62
Audio or video elements made the survey run slowly	10%	32%	58%	60
There were problems with computers not working	19%	44%	37%	63
There were not enough computers	21%	50%	29%	62
There were difficulties timetabling classes into ICT suites	23%	52%	25%	61

Timetabling difficulties

4.20 The problem of timetabling classes into ICT suites was the most frequently cited major issue by liaison teachers (23%). A further 52% indicated that it was an issue but they were able to overcome it.

4.21 Most of the liaison teachers with whom depth interviews were conducted noted that it was relatively straightforward to book an ICT suite, provided that there was sufficient notice. In the survey, around half of the liaison teachers said that it was difficult to book an ICT suite (47%), while 35% said that it was easy to do so. Most commonly, 3-4 weeks' notice was required and very few required longer than 6 weeks. The Dartington Social Research Unit indicated that schools required at least one - ideally two - months' notice.

4.22 A further complication stems from the fact that ICT suites tended to be subject-specific rooms, so there were timetabled classes in them. If there is a clash with the PSE time of a sampled class alternative accommodation would have to be arranged for the original class. The NFER Pisa Study trial also found that demands placed on school ICT facilities for coursework completion and exam preparation was a problem.

4.23 The SALSUS main fieldwork period is long enough to be able to accommodate this although it would likely have an impact on schools/classes which require to be chased up at the end of the fieldwork period (and therefore have an impact on response rates). It may therefore be prudent to extend the fieldwork period by a few weeks if moving online.

4.24 It would also help to recruit schools earlier than at present, potentially while timetables are being drawn up for the next academic session in May or June. This would allow schools greater flexibility since they would be informed during recruitment of the number of classes to be sampled and the timing of the fieldwork period, allowing schools to identify the most appropriate time for sampled classes to take part, as well as being able to plan for any alternative arrangements for subject classes which would normally be using the ICT suite.

Lack of available/working computers

4.25 Major problems with computers not working and there being too few computers were cited by one in five liaison teachers. There was quite a bit of overlap, and overall, around a quarter had experienced major issues with one or both of these problems.

4.26 In the liaison teacher survey, around 60% indicated that the capacity of their ICT suites was smaller than the average size of a PSE class.

4.27 Around half the IT officers (n=16) thought capacity in ICT suites would not be a problem and a further five indicated that it would be easy to use laptops in the ICT suites to 'top up'. Three did think that capacity would be a problem. The rest were not sure.

4.28 If topping up with laptops is not possible, there are other potential ways to overcome these problems, such as splitting the class between two ICT suites or splitting the class in half (with half doing the survey one week, the other half another week). However, these solutions require an additional member of staff to provide cover.

Software/website issues

4.29 Liaison teachers were less likely to report the following as major issues: audio or video elements making surveys run slowly, software compatibility problems, or websites being blocked. Around three-quarters thought that these issues were not a problem the last time they administered an online survey in the school (although, depending on the nature of the survey, they may simply not have arisen).

4.30 A number of IT officers mentioned that plug-ins (especially audio and video elements) would have the potential to cause surveys to run slowly, especially if there were a large number of participants. This would be exacerbated by class teachers using webpages with audio or video content as teaching resources at the same time because more bandwidth would be used.

4.31 This should not be problematic as the SALSUS questionnaire in 2015 will not contain audio or video elements.

4.32 A small number noted concerns over software compatibility, particularly where schools across the authority may be using different browsers or different versions of the same browser. All IT officers said that testing the survey site in advance to ensure that there were no such problems would be advantageous to the smooth running of the survey. They also said that unblocking websites was straightforward.

4.33 The Dartington researcher indicated that the main problem to be aware of are PC's crashing while a high volume of pupils are completing the survey - partially due to problems with internet connections/bandwidth (particularly in rural areas). This can be addressed by staggering completion of the survey across schools.

Pupil reaction

4.34 A number of liaison teachers and IT representatives mentioned that children are used to doing things on computers, already do a number of assessments online, and take part in online surveys using Survey Monkey. In addition, 58% of liaison teachers said that pupils enjoyed the experience of completing an online survey more than they would have with a paper survey (Table 4.4)

Table 4.4: Liaison teachers: pupil reaction to their school's most recent online survey

Q. In general, how did pupils react to completing the survey online?

	More than they would have with a paper survey %	The same as they would have with a paper survey %	Less than they would have with a paper survey %	Base
They enjoyed the experience	58	42	0	60
They rushed through their answers without much thought	18	73	8	60
They took it seriously	12	85	3	60
They conferred over answers	10	70	20	61
They were concerned about confidentiality	5	73	22	60

4.35 NFER research on a computer-based pilot of the PISA test in Scottish schools noted that a number of pupils appeared not to take the test seriously. Similarly, Ipsos MORI research on an online pilot questionnaire of 12-15 year-olds found that participating children were more likely to confer over answers than in a paper-administered survey conducted under exam conditions, and were more likely to deliberately provide misleading answers.

4.36 This view was echoed by a number of liaison teachers, with 18% saying that pupils rushed through their answers more than they would have with a paper survey, in the most recent online survey in their school. However, 72% of liaison teachers said that pupils did this to the same extent as they would have with a paper survey. One-fifth of liaison teachers said that pupils were less likely to confer over answers (20%) or to have concerns over confidentiality (22%) with an online survey.

4.37 When asked what effect they thought moving online would have on SALSUS (as opposed to their most recent experience of a survey), 10% of liaison teachers said that pupils would rush through the survey or not take it seriously. This compares with 25% who thought that an online survey would have no effect on pupils' responses, and the 13% who felt that pupils would provide better quality data, or more honest answers.

4.38 Liaison teachers thought that there would less concern about confidentiality than there would be with a paper survey.

4.39 Taking these factors into consideration, moving the survey online would appear to pose little risk to the way in which pupils approach the survey, and could potentially yield better data.