Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS) 2015: Online Pilot

Annex A: Topic guides

CLASS/LIAISON TEACHER GUIDE

Introduction

Introduce self, Ipsos MORI, thank for taking the time to speak to us

As you probably know, SALSUS is a Scottish Government survey which has been running since 1982. The survey asks pupils in S2 and S4, in schools all across Scotland, about their health and lifestyle, including their smoking, drinking and drug use.

In the past the survey has always been completed on paper but we now want to see if it can be run online. So, today I want to talk to you about your experience of piloting SALSUS 2015 online. This will help us to advise the Scottish Government on whether or not it is feasible to move the survey online in 2015. Can I just double check, have you administered one of the online pilot classes?

The interview is confidential to the research team. While we might quote what you say (assuming you give your permission) no school or individual will be named in the research report.

Names

Permission to record

General reactions

• Overall, how did you find the process of administering SALSUS online?
• What problems did you face administering the survey online?
  • How serious were the problems?
  • How, if at all, were they overcome?
• What, if anything, do you think worked well about the online methodology?
• Have you administered the paper version in the past? How does administering it online compare? What’s better/worse?

Pupil reactions

• What was the pupils’ reaction to completing the survey online?
  • How easy or difficult did they find it? (vs. paper)
  • How seriously did they take it? (vs. paper)
  • Was there any conferring over answers? (vs. paper)
  • Were there any concerns about confidentiality? (vs. paper)

Administering the survey

What did you think about the class teacher/liaison instructions?

IF YOU NOTICED SOMETHING IN THE CLASS OBSERVATION THAT HAS NOT BEEN MENTIONED

• When I observed one of the classes complete the survey, I noticed … What are your views on this?

IF NOT ALREADY COVERED

• Our feasibility study highlighted some possible issues that might occur while administering an online survey. When the pilot took place did you encounter any of these things?

IF ANY OF THESE THINGS WERE A PROBLEM, WAS IT A SERIOUS PROBLEM AND HOW, IF AT ALL, WAS IT RESOLVED.

• • Not having enough computers for the whole class.
  • Problems with computers not working.
  • Problems with software compatibility or out-of-date browsers.
  • The survey site was blocked by a filter or firewall.
  • Timetabling difficulties
  • Problems with bandwidth or connectivity
  • Problems creating exam conditions in the classroom
  • Accessibility issues for Additional Support Needs pupils
• Could anything be done to make it easier for school to administer the survey online?
• Do you have any other suggestions on how to improve the way the survey is administered online?

Thank you very much for your time.

PUPIL TOPIC GUIDE

Introduction

Introduce self, Ipsos MORI

SALSUS is an important survey which has been running since 1982. The survey asks pupils in S2 and S4, in schools all across Scotland, about their health and lifestyle, including their smoking, drinking and drug use. The results of the survey will help provide information and advice for young people on smoking, drinking and drug use.

In the past the survey has always been completed on paper but we now want to see if it can be done online. We asked your school to test out the online SALSUS survey for us. Can I just confirm that all of you have actually completed the survey online in the last week, it might even have been today?

So, in this group, I want to talk to you about how you found filling in the survey. I’m not so much interested in what your actual answers are as how you found the experience of answering the questions.

As you probably remember, some of the questions are about smoking, drinking and drug use and things you might not want teachers or your parents to know about. But just to reassure you, your answers to the questions and anything you say to me, are completely confidential. I won’t say anything to your parents, teachers or anyone else in the school about what you say.

And please don’t say anything to anyone else about what the other people have said.

So please be as honest as you can. If there is anything you don’t want to answer, you don’t have to.

Now you know a bit more about it, are you happy to take part? Any questions before we start?

Names

Permission to record

Warm up exercise

The last time we ran SALSUS was in 2013 and when we did it was on paper rather than on the computer like you did it. (HAND OUT QUESTIONNAIRES). Here is the questionnaire in paper format. When we run the survey on paper, pupils complete it in their PSE class. The classroom is set up in ‘exam conditions’ and the teacher stays at the front of the class. Once a pupil completes their questionnaire, they put it in blank envelope and seal it. They do not put their name on the questionnaire or envelope at any point. The sealed envelopes are then handed back to their class teacher who puts them into a return bag that gets sent straight to an Ipsos MORI office.

To begin with we’re going to play a game. I’m going to ask you to guess some of the results from the 2013 survey, don’t worry if you get it wrong it’s just for fun. Let’s start with smoking, what % of 13/15 year olds (ONLY ASK FOR PUPILS OF THEIR AGE) in Scotland do you think smoke at least one cigarette a week. Who wants to go first?

WHEN A GUESS IS GIVEN WRITE IT DOWN ON A STICKY AND GO TO THE NEXT PARTICIPANT.

OK, who’s next? Do you think it will be higher or lower? Write down the number and stick it above or below the last guess.

REPEAT UNTIL EVERYONE HAS GUESSED.

Great, so the number that we got from the survey was 2% (if S2)/9% (if S4). POINT OUT WHICH GUESS WAS CLOSEST.

So most of you thought it was higher/lower/about that. We’re going to talk about that a little more in a minute.

This time we’re going to be guessing about drinking, what % of S2/S4 pupils have ever had an alcoholic drink?

REPEAT EXERCISE.

Based on what pupils said in the survey in 2013 the figures are 32%/70% S2/S4. POINT OUT WHICH GUESS WAS CLOSEST AND IF THERE IS A SIMILAR TREND IN GUESSES BEING HIGHER OR LOWER THAN THE ACTUAL RESULT.

And finally drug use, what % of S2/S4 do you think have taken drugs in the last month?

As above. Correct answer 2% and 9%.

So now that you’ve seen everyone’s guesses and the survey result, do you think the findings from the survey are right? Why/why not?

How much do you think young people lie when they complete a survey like this?

Would they say they do things (like smoking a cigarette) more or less than they actually did?

Do you think they would lie more or less in an online survey than in a paper survey, or would it make no difference?

Would anything help make pupils more likely to tell the truth?

So now we’re going to have a chat about the survey session that you took part in earlier this week/today.

• What was completing the survey like?
  • What was good about it?
  • What was bad about it?
• How easy or difficult was it to complete the survey on the computer/tablet?
  • Was anything difficult or hard to understand?
  • Did anything not work?
• Were there any questions you didn’t know how to answer?
  • What was the problem?
  • Were the instructions were clear enough?
  • Probe for difference between technical issues and comprehension issues.
• Was the information given to you before you completed the survey enough to help you decide whether or not to take part? Why/Why not?
  • Is there anything else would you like to know?
• Can you tell me what you thought about how the survey looked on the screen?
  • What was good? What was bad
• • What improvements could be made?
  • Could you see the entire question in every questions/was scrolling a problem?
  • Does the way the survey looks make any difference to how you complete it? Why/why not?
• Where there enough computers to go round all the pupils? If so, how did the teacher get around this problem?
• What did you think about being able to choose your own log-in?

Second exercise

Now that we’ve talked about completing the survey a little more, I want to do another exercise with you. So, I’m going to put up two bits of paper on the wall. One for online surveys, and one for paper surveys. I want you to write down on the post-it notes, what you think is good or bad about each method and when you’re finished, come up and stick them on the bits of papers.

ONCE COMPLETED GO THROUGH POST-ITS AND DISCUSS WITH THE GROUP – CHECK TO SEE IF PEOPLE AGREE OR DISAGREE.

I now want you to go through these cards as a group and decide if the statements on the cards apply more to online survey or to paper surveys.

PROBE ON PLACEMENT OF CARDS AND DISCUSS ANY DISAGREEMENTS/DIFFICULT DECISIONS

IF NOT COVERED IN FIRST TWO EXERCISES

• Was anyone in the classroom talking while filling in the survey? Why/why not?
  • Was the room under exam conditions?
  • Did you talk about any of your answers?
  • If they did, did it change how you answered any of the questions?
• Did you feel like anyone else in the classroom could see your answers? Why/Why not?
  • Could the teacher see?
  • Could other pupils see?
  • If so did it change how you answered the questions?
• Were you worried whether anyone could get access to the answers you typed into the computer?
  • Would that have worried you if the survey was on paper?
  • Who were you worried about seeing your answers?
  • Did it change the way you answered the questions?
  • What would help to reassure you that someone could not access your answers?
• Would you change anything about how the survey is run?

Thank and close.

FEEDBACK FORM FOR CLASS TEACHERS

Overall, how did you find the process of administering SALSUS online?

Where there any problems? What were these? How, if at all, were they overcome?

Was there anything that you thought worked well about the methodology?

What was the pupils’ reaction to the survey?

How helpful were the class teacher instructions? How could they be improved?

Could anything be done to reduce the burden on you/the school?

Our feasibility study highlighted some possible issues that might occur while administering an online survey. When the pilot took place did you encounter any of these things?

1. There were not enough computers Yes ☐ No ☐

If this was a problem, how, if at all, was this resolved?

2. There were problems with computers not working Yes ☐ No ☐

If this was a problem, how, if at all, was this resolved?

3. There were problems with software compatibility Yes ☐ No ☐

If this was a problem, how, if at all, was this resolved?

4. The survey site was blocked by a filter or firewall Yes ☐ No ☐

If this was a problem, how, if at all, was this resolved?

5. There were timetabling difficulties Yes ☐ No ☐

If this was a problem, how, if at all, was this resolved?

6. There was a problem with bandwidth or connectivity Yes ☐ No ☐

If this was a problem, how, if at all, was this resolved?

7. It was difficult to manage the survey under exam conditions Yes ☐ No ☐

If this was a problem, how, if at all, was this resolved?

8. There were accessibility issues for Additional Support Needs pupils Yes ☐ No ☐

If this was a problem, how, if at all, was this resolved?

Thank you very much for your time.