Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS) 2015: Online Pilot

3 Key Findings

• Overall, pilot schools found that administering the survey online was relatively straightforward. Pupil reactions were positive and they would prefer to complete the survey online rather than on paper. There was only one major problem encountered: in two of the twelve schools the Local Authority (LA) firewall blocked the names of specific drugs so the survey was stopped at the drugs section. Now that this issue has been identified, we can liaise with LA IT officers to address this in advance of the main fieldwork.
• Pupils were generally positive about their experience of completing the survey online and the dominant view was that an online survey was ‘easier’, ‘more fun’, ‘less dull’ and ‘more modern’ than a paper survey. In addition, they frequently mentioned the environmental benefits. Teachers also thought that pupils reacted better to an online survey because they found it easier and more engaging.
• On balance, pupils tended to feel that an online survey would lead to more honest answers. However, they perceived advantages and disadvantages to each mode.
• Overall, both liaison and class teachers were positive about administering the survey online and found it straightforward. The need to book ICT suites (or laptops or tablets) meant that aspect was more burdensome than administering a paper survey and required more advance planning – but it was do-able. However, they also reported that once that aspect was arranged, the actual administration of the survey with the class was much easier.
• Topline analysis of the data suggests that there are no major problems with data quality. The responses relating to some key measures (regular smoking, drinking in the last week and ever taken drugs) are broadly in line with what might be expected given the 2013 SALSUS results.

Overview

3.1 Overall, pilot schools found that administering the survey online was relatively straightforward. Pupil reactions were positive and they would prefer to complete the survey online rather than on paper. There was only one major problem encountered: in two of the twelve schools the LA firewall blocked the names of specific drugs so the survey was stopped at the drugs section. Now that this issue has been identified, we can liaise with LA IT officers to address this in advance of the main fieldwork.

Pupil reactions

3.2 Pupils were generally positive about their experience of completing the survey online and the dominant view was that an online survey was ‘easier’, ‘more fun’, ‘less dull’ and ‘more modern’ than a paper survey. In addition, they frequently mentioned the environmental benefits (Figure 3.1).

Figure 3.1 Pupil reactions - advantages of online

3.3 Teachers also thought that pupils reacted better to an online survey because they found it easier and more engaging. There were mixed views on whether pupils took it more or less ‘seriously’ than on paper: one view was that they did take it more seriously because they were more engaged; another view was that they took it a little less seriously because it was ‘less like an exam’; others thought it made no difference.

Perceptions of anonymity

3.4 If reported levels of substance use differ significantly depending on whether the survey is completed online or on paper, this is most likely to be because there is a perceived difference in anonymity. We would predict that pupils would provide more honest answers in the mode that they perceive as more anonymous. In relation to self-reported substance use, more honest answers are generally assumed to mean higher levels of reported use.

3.5 On balance, pupils tended to feel that an online survey would lead to more honest answers. However, they perceived advantages and disadvantages to each mode.

3.6 Although pupils place their completed paper surveys in blank envelopes and seal them before they are collected by the teacher, there was a concern that teachers might open the envelopes and look at the answers (and be able to identify who had completed the questionnaire from handwriting, specific responses or some other means) (see Figure 3.2). Concerns were also raised about envelopes getting lost and about the paper questionnaire being more of a ‘permanent record’ than an online survey.

Figure 3.2 Pupil reactions – disadvantages of paper

3.7 On the other hand, it was generally felt that pupils could see the screens of those next to them (if they wanted to) and teachers could see screens some of the time if they were moving round the room. However, there appeared to be less concern about this than about the possibility of teachers opening the envelopes and being able to read through the whole paper questionnaire. There was also a feeling that, although someone might be able to see one question at a time on the screen, a pupil could answer quickly and move on to the next screen. In contrast, although they might be less easy to see in general (and easier to ‘hide’), responses on a paper questionnaire are potentially visible for longer.

Figure 3.3 Pupil reactions – advantages of paper

3.8 The use of a sheet of sticky labels with the web address and unique log-ins (which teachers handed round for pupils to choose and peel off a label at random) worked well. Pupils generally understood that they could not be identified from their log-in code. However, there were still a few who were concerned that their online responses could be linked back to them (either through the log-in or through the tablet/PC that they used).

3.9 It is not possible to assess from the pilot what impact these perceptions might have on responses. However, if they lead to a sizeable difference, this will be identified in the mode experiment.

3.10 It should be noted that pupils’ main concern in this regard was not about teachers or other pupils seeing their responses, but about the potential to be identified by their postcode. Although this also applies to paper and is therefore not directly related to the move to online, it is discussed in sections 3.18-3.22 below as it appeared to be a significant issue which may be affecting responses.

Teacher reactions

3.11 Overall, both liaison and class teachers were positive about administering the survey online and found it straightforward. The need to book ICT suites (or laptops or tablets) meant that aspect was more burdensome than administering a paper survey and required more advance planning – but it was do-able. However, they also reported that once that aspect was arranged, the actual administration of the survey with the class was much easier: there were fewer materials to deal with; it was quicker; pupils were more engaged and found it easier; and once pupils had completed and submitted their surveys, the class teacher did not need to do any more (with a paper survey, they need to ensure they have collected them all in and then take them to the office for courier collection).

Aspects that worked well

3.12 The data from the pilot was not cleaned and was not analysed in detail: any meaningful comparisons with paper would require the robust sampling and analysis that would be undertaken in the mode experiment. However, a topline analysis of the data suggests that there are no major problems with data quality. The responses relating to some key measures (regular smoking, drinking in the last week and ever taken drugs) and demographics (age and sex) are broadly in line with what might be expected given the 2013 SALSUS results.

3.13 Aspects of the process that worked well and require little or no change in advance of the main fieldwork were:

• the use of sticky labels to randomly distribute unique log-ins
• the overall look of the questionnaire
• the instructions in the questionnaire
• the instructions for liaison teachers and class teachers (some did comment that they were rather long - but others felt they were comprehensive and it was better to provide more rather than less information).

Main problems

3.14 The biggest problem identified was that, in two schools, the LA firewall blocked the names of specific drugs. This meant that the survey stopped at the first drugs question and pupils could not go any further. In one school, a LA IT officer was able to unblock the terms and pupils in the later classes were able to complete the survey. In the other school, the school-based IT officer attempted to unblock the page but was unable to do so.

3.15 Now that this problem has been identified, we can raise the issue when we contact LA IT officers in advance of the main fieldwork. We will also ask liaison teachers to check the survey in advance (and will provide spare log-ins for this purpose).

3.16 In a couple of the schools which piloted the survey with tablets, there were problems with iPads not being charged and with a poor Wi-Fi connection (see Figure 3.4). However, both these schools indicated that they would have chosen to administer the survey on PCs, and only used tablets because they had been asked to do so for the purposes of the pilot. We will warn schools about these issues. In the main fieldwork, it is likely that the few schools which choose to administer the survey on tablets will do so because they use tablets quite frequently and do not anticipate problems.

Figure 3.4 Pupil reactions – disadvantages of online

Postcodes

3.17 The questionnaire asks pupils for their home postcode. It was clear from the focus groups with pupils that there was considerable concern about this because they felt it made them identifiable (particularly in combination with their month and year of birth). They did not understand why it was necessary. There is the option to click ‘prefer not to say’ (and 39% selected this option) but it appeared that many did provide their postcode despite their concerns and suspicions about it – and this may have affected their subsequent responses. One girl said that she gave her postcode but, because she had done so, she did not answer one of the subsequent questions (on drinking) truthfully.

3.18 While this issue also applies to the paper survey (and the aim of the pilot was to identify issues relating to the move to online), we have raised it here because it emerged as the biggest concern around anonymity and it seems likely that, to some extent at least, it is affecting responses. There are a number of ways this could be addressed (or explored further) and we recommend that this issue is given further consideration. Options include:

• adding a short explanation at the postcode question of why the information is useful e.g. ‘We ask for your postcode so we can compare results from different types of area. For example, we can compare rural areas with cities and richer areas with poorer areas
• adding a further reassurance that we will not use it to identify people e.g. ‘Only the research team at Ipsos MORI will see your postcode and we have no way of using your postcode to identify you’
• removing the question on month of birth (since this is little used in analysis)
• separating the postcode question from the other demographic questions (e.g. school year, year of birth, sex) so pupils are not misled into thinking they might be linked with postcode in order to identify them.

3.19 All of these amendments could be applied to both the paper and online questionnaires and so would not affect the mode experiment.

3.20 Consideration should also be given to undertaking a split sample experiment in a future wave (not at the same time as the mode experiment) where half the sample are asked for their postcode and half are not and any differences in responses are measured.

3.21 Pupils concerns about providing their postcode also raises questions about the impact that data linkage (which is being considered for future waves of SALSUS and for other surveys) might have on their perceptions of confidentiality. Overall, pilot schools found that administering the survey online was relatively straightforward. Pupil reactions were positive and they would prefer to complete the survey online rather than on paper. There was only one major problem encountered: in two of the twelve schools the LA firewall blocked the names of specific drugs so the survey was stopped at the drugs section. Now that this issue has been identified, we can liaise with LA IT officers to address this in advance of the main fieldwork.

Specific issues identified

3.22 The list below outlines the list of specific issues that were highlighted by participants in the online pilot.

1. Confidentiality concerns:

• Teacher seeing the screen
• Other pupils seeing the screen
• Where the data goes.

2. Changes in pupil reactions:

• More enjoyable/engaging
• More talking/conferring
• Quicker to complete.

3. Technical issues:

• Firewall blocking drugs questions
• Problems with Wi-Fi connection
• Tablets/laptops not being charged
• Scrolling at long lists/questions running over two pages.

4. Logistical

• Support for pupils with Additional Support Needs
• Difficulties in following up on absent pupils.

3.23 For full details of the issues identified by pilot participants and our recommended actions please see Annex B.