Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS) 2015: Online Pilot

Teacher seeing the screen

There was an awareness among pupils that when their teacher moved around the room (e.g. responding to pupils’ requests for assistance or checking on progress), s/he could potentially see their screen.

In general, there did not appear to be a great deal of concern about this but it may lead to under-reporting. If this has a sizeable impact, it will be detected in the mode experiment.

Medium

Highlight in teacher instructions and ask them to walk around as little as possible.

Other pupils seeing the screen

PCs were close together and pupils were aware that those seated next to them could potentially see their screen.

To a lesser extent, this also applied to tablets: it would be easier for a pupil to position their tablet so the screen was not visible to another pupil but, in practice, many did not.

There appeared to be even less concern about other pupils seeing their screen than teachers (particularly because pupils often sat next to their friends). However, it may lead to under- or over-reporting. If this has a sizeable impact, it will be detected in the mode experiment.

Low

There is nothing that can realistically be done about the set-up.

We will explore the option of making the responses less obvious (e.g. by using a paler font) but this needs to be balanced by clarity for the pupil completing the survey.

Where the data goes

Some pupils would like more explicit information about where the data goes and what it is used for.

Potential impact on informed consent and willingness to participate.

Low

Provide information about the data being held securely by Ipsos MORI.

Give more information about the types of services the results from the survey help to shape.

More enjoyable/engaging

Pupils enjoyed taking part online and said that they enjoyed it more than they would have done had it been on paper. They said they found it easier to engage with and had a generally positive reaction to it. Teachers also felt that pupils were more engaged than they would have been with a paper survey.

Unknown, but presumably a positive impact. If pupils are more engaged, they may think about their answers more and provide more accurate information. If this has a sizeable impact, it will be detected in the mode experiment.

Low

N/A

Talking/conferring more

There was a degree of talking/conferring in each observed class, however the extent to which it happened varied – in some classes it was more or less continuous, whereas in others it only happened at the start, before everyone settled down, and again towards the end once the first few pupils had finished.

We have not observed classes completing the survey on paper. However, teachers felt that there was likely to be more talking with an online survey – at least in part because pupils are more engaged.

Has the potential to influence the way in which pupils respond or can serve as a distraction to other pupils – particularly those who take time over their responses. However, teachers’ perceptions were that pupils were more likely to be discussing the questions rather than their responses. If this has a sizeable impact, it will be detected in the mode experiment.

Medium

We already indicate in the instructions to teachers that pupils should not confer and the survey should be completed under ‘exam conditions’.

Giving pupils an interactive task at the end of the survey (with a balance between educational and fun) should help keep pupils occupied once they have finished and allow others to complete the survey.

Quicker to complete

Once they had logged on, many pupils were able to complete the survey within 25 minutes.

We do not have accurate data on the average time it takes pupils to complete the survey on paper, but from the cognitive testing exercise and from previous feedback from teachers, we know it takes longer (perhaps around 5-10 minutes longer).

Makes the survey easier logistically:

it can be completed in shorter periods;

if schools have to use rooms where there are slightly fewer computers than pupils, some pupils could do something else until the first few pupils are finished (as happened in some classes in the pilot)

No risk – positive impact

If schools are concerned about capacity in their ICT suites, reassure them that some pupils will complete the survey within a relatively short space of time. This information would be included in the instructions.

Firewall blocking drugs questions

At two schools, the survey was stopped when pupils reached the drugs questions. In one school, their LA IT department was able to unblock the survey within 24 hours. In the other, the school IT officer was unable to unblock it (they did not immediately contact the LA IT dept.).

If the survey is blocked, pupils can’t complete the remainder of the questions unless the firewall is re-set to allow the blocked words to pass through.

High

Raise the issue in advance with all LA IT contacts.

Request that liaison teachers test the survey in advance and notify their LA IT department about any problems with the survey being blocked, allowing good time before pupils are due to complete the survey.

Problems with Wi-Fi connection

Slow Wi-Fi connectivity; not coping with a whole class taking part at once; drops in connection.

Disruption to participation. Pupils can’t take part/complete.

Low

Inform schools about potential issues with Wi-Fi coverage if they administer the survey via portable devices.

Tablets/laptops not being charged

One school had problems with iPads which were not charged before the survey was administered. There was disruption while fully charged devices were found or iPads were plugged in.

Hold ups and disruption while devices are charged/plugged in; pupils might not be able to complete survey.

Low

Reminder to teachers to ensure such portable devices are charged.

Scrolling at long lists/Questions running over two pages

Some pupils were selecting ‘No’ automatically without reading the list item at the drugs questions, although this was because they knew that they hadn’t been offered/taken any drugs. Although most questions with long lists were split over more than one page, there were instances when the question and answer options were not visible on the second page. Others couldn’t see the response options further down the page with the Strengths and Difficulties questionnaire.

If pupils forget or get confused about what the response options were, this could affect the accuracy of the data.

Low

Ensure any remaining questions like this are split over more than one page so that response options stay visible.

Pupils with Additional Support Needs

Teachers said that there could be some difficulties enabling pupils with Additional Support Needs to participate, depending on the number of pupils involved and the type of support they require. Those who require a Personal Support Assistant would need to have a private room with a computer in order to complete the survey. This creates additional logistical issues for the school.

Fewer children with Additional Support Needs take part.

Low

Encourage schools to allow children with Additional Support Needs to participate, perhaps by staggering the days on which they are given the survey to do if there are pressures on resources or staff.

Administer the survey via Support For Learning. Remind them that survey can be completed on tablet or laptop (even if most pupils doing on PC). Remind schools of the importance of representative data.

Following up on absent pupils

Some teachers raised potential logistical difficulties posed by following up with absent pupils given the need to book ICT facilities.

Reduced pupil response rates; source of bias.

If this has a sizeable impact, it will be detected in the mode experiment.

Medium

Add the following suggestions to the instructions: group absent pupils together and book a room for them to take part; have individual absent pupils take part in the library when they return to school; remind schools that the survey could be undertaken by individual pupils on a lap-top or tablet (even if most pupils are completing the survey on PC).