Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS) 2015: Online Feasibility Study

5 Mode Effect experiment

Key points

The current IT infrastructure could not support a full, nationally representative 3-way (paper/PCs in ICT suites/tablets) mode experiment.

However, a 2-way mode experiment (paper versus PCs in ICT suites) would be possible.

While there are some moves towards the use of mobile devices such as tablets, schools appear to be retaining ICT suites for at least the near future. Therefore, it will be worthwhile to test this mode and conduct the mode experiment in SALSUS 2015.

What are the options?

5.1 Many of the potential benefits and risks of completing the survey online are related to the place of administration rather than simply the type of device (e.g. confidentiality concerns because it is harder to put ICT suites into exam conditions, pupils being in a different 'mind set' when taken from the classroom to the ICT suite, limited Wi-Fi coverage for use of classroom-based devices). Therefore, classroom-based administration (on laptops/netbooks or tablets) and ICT suite administration (on PCs) should be treated separately in any mode effect experiment as they are too different to be considered as a homogenous 'online survey' method.

5.2 The ideal would be to conduct a three-way mode experiment comparing paper completion in classrooms, online completion in classrooms (laptops/netbooks and tablets) and online completion in ICT suites (on PCs). This would require a nationally representative sample for each mode with a minimum sample of 8,000^[11] paper and 4,000 for each of the online modes. This is one possibility and the primary research explored the feasibility of this.

Would a three-way mode experiment be possible?

5.3 The current IT infrastructure across Scottish schools could not support a full three-way mode experiment because there are insufficient schools with the ability to conduct the survey online in classrooms.

5.4 Most local authorities have very few tablet devices in school. Those schools that do have tablets tend not to have a full PSE size class set (often only 10-15 devices per set). In the liaison teacher survey, only 17% said that they could administer SALSUS 2015 using tablets.

5.5 The use of laptops/netbooks in class was more widespread than the use of tablets. The interviews with IT representatives suggests that 10 of the 30 local authorities would be able to use laptops/netbooks for the mode effect experiment. Overall, 35% of liaison teachers said that they could administer SALSUS using laptops.

5.6 We estimate that the maximum sample size that we could achieve for pupils completing the survey on laptops/netbooks or tablets would be around 4,000. However, we not are not confident that all of these would complete the survey in classrooms (some would be on laptops/netbooks in ICT suites), so the maximum achievable sample size would likely be somewhat short of 4,000.

5.7 A further problem, raised by a number of IT officers, is the lack of Wi-Fi connections and/or the unreliability of Wi-Fi connections in classrooms. Problems related to this may further reduce the achieved sample size.

5.8 Moreover, the geographic profile of classroom-based devices would not be sufficiently representative of Scotland as a whole (and consequently not comparable to that of the paper sample).

Would a two-way mode experiment be possible?

5.9 A two-way mode effect experiment, comparing paper administration in classrooms and PCs administration in ICT suites, will be possible.

5.10 As outlined in Section 4, although there will still be schools that need to administer the survey on paper (or have a strong preference for doing so), for the most part it is feasible to conduct SALSUS 2015 online using ICT suites. There were 6 local authorities where the transition may be more difficult. However, these tend to be smaller local authorities which would have limited impact on the national picture and, while the schools in these areas may have had reservations, the local authority IT officers were still confident they could participate.

5.11 This would suggest that the geographic coverage for a two-way mode effect experiment would be sufficient.

Would a two-way mode experiment be worthwhile?

5.12 While it is possible to conduct a two-way mode effect experiment comparing paper completion in classrooms with online completion in ICT suites, it would not necessarily be worthwhile if there was evidence of a move towards classroom-based devices in the near future.

5.13 However, although there is something of a move towards classroom-based devices, it is not particularly fast or widespread. Most schools still use a traditional ICT suite set up (28 of 30 would use ICT suites to administer SALSUS 2015).

5.14 Twenty-one out of 30 IT officers said that they would be using ICT suites for the foreseeable future, five said they are moving towards classroom-based devices, three said that it was be too hard to predict and one did not want to answer. One IT representative said her local authority had gone full circle - they had moved away from ICT suites and had PCs in classrooms, but had realised that the 'traditional' ICT suite was more useful.

5.15 This suggests that a two-way mode experiment is worthwhile. If classroom-based devices do become much more widespread, a further mode effect experiment may be required, but this is likely to be several years down the line.