Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): Technical Report 2015

Appendix A: Cognitive Question Testing

Before finalising the questionnaire for the 2015 wave, a cognitive testing exercise was undertaken in November 2014 to test proposed new questions and existing questions which were thought to be problematic.

Methods

Cognitive testing was undertaken with 26 pupils (11 boys in S2, 5 girls in S2, 6 boys in S4 and 4 girls in S4) in three schools. Two of the schools were in areas of high deprivation and one was in an area of moderate deprivation. In order to maximise the chances of identifying problems, the sample intentionally included pupils who were less academically able.

The testing was undertaken by core members of the Ipsos MORI research team. Pupils were interviewed individually or in friendship pairs. Interviews lasted approximately 45 minutes. Pupils first completed the questions on paper, while the researcher observed. This was a shortened version which included the questions to be tested plus some of the questions which immediately preceded those questions (in case that affected their interpretation).

The researcher then questioned the pupil about how they had answered the questions, checking their understanding and interpretation (retrospective verbal probing).

The testing was undertaken in three waves with around a third of pupils involved in each wave. After each of the first and second waves, a report was sent to the Scottish Government Project Manager and relevant policy colleagues, and agreed amendments were made to questions which were then tested in the next wave.

After the third wave, the findings were discussed with the Research Advisory Group and agreement was reached on the final versions for the 2015 questionnaire, or removal of the question in some cases.

Findings

The findings and final decision for each question (or series of linked questions) tested is described in this section. All iterations of questions are described, although not all are presented in full ^[7] .

Question on 'fake fags'

This question was tested because it involved a lengthy explanation and there was concern about the way it was being interpreted (see Figure A.1). Two amended versions were tested before a decision was made to remove the question as it proved to be too problematic.

Figure A.1 : Existing version from 2013 survey

Findings from the first wave of interviews revealed that the question was indeed not well understood and required alteration. Pupils found it too long and confusing. In discussion, they appeared to have a good understanding of the concept of 'fake fags' and were familiar with the term 'fake fags'. However, as the explanation in the question was so long, many did not read it in any detail and made incorrect or incomplete assumptions about what was meant.

It was agreed that the question would be simplified (see Figure A.2). In the second wave of interviews, pupil comprehension was tested again, with a particular focus on whether pupils considered untaxed cigarettes and those containing cheaper ingredients.

Figure A.2: Version 2 of question on 'fake fags'

A test of the second version found that some pupils thought the first part of the question was referring to e-cigarettes and answered on that basis.

Part (b) of the question seemed to be understood and answered in the way intended. One of the pupils who had thought the first part of the question was referring to e-cigarettes understood, once they had read part (b), (although they didn't then go back to Q9 to change their answer). There was also mention of the tobacco being more loosely rolled and being more yellow in colour than 'real' cigarettes.

Pupils were not including 'real' cigarettes of known brands that were brought from abroad and sold more cheaply in their definition of 'fake fags'.

One girl did include cigarettes from Barcelona that she described as 'minging', which did not light properly, and were in paper packaging. However, she admitted that she did not know that they were necessarily 'fake' and may, in theory, have been a 'real' Spanish brand that she wasn't aware of.

It was agreed that further amendments would be made to the question to specifically exclude e-cigarettes and vapour pens as well as add an additional response category of 'they look different' to part (b) of the question (see Figure A.3).

Figure A.3: Version 3 of question on 'fake fags'

None of the respondents from day three of the interviews smoked cigarettes and were, therefore, not routed to this question. When asked to read the question and say what they thought 'fake fags' meant there were various definitions given:

• E-cigarettes (despite the new question wording)
• 'Poisonous tobacco'
• Cigarettes from 'dodgy' shops
• Cigarettes from abroad
• Fake cigarettes from other countries
• Chinese cigarettes with 'chemicals in'.

It is important to note that, as these pupils were non-smokers, they were less likely to know what was meant by 'fake fags'. However, it did raise issues about the variability in pupils' interpretation of the phrase 'fake fags'.

Due to the demonstrated issues with pupil understanding and interpretation of this question, it was removed from the 2015 survey.

FINAL VERSION

Question removed.

Question on whether other people smoke

This question was tested because it had resulted in a high number of missing answers in previous waves. Two amended versions were tested and a final version agreed as the cognitive testing indicated that the amendments had improved levels of response.

Figure A.4: Existing version from 2013 survey:

The test of the 2013 existing version showed that some pupils left particular rows blank because they did not notice the 'Don't have or don't see this person' column. One suggested it was because of the dark line which separates that column from the others. Another said it was because 'Don't have or don't see this person' is quite long so they didn't read it properly.

Others left rows blank because the person did not smoke. It seemed they did not look for the 'does not smoke' option because the question asked 'which of the following people smoke'. They looked down the list of people and only crossed a box if the person did smoke.

A revised version was agreed which changed the order of the response options so that 'does not smoke' was first instead of third. Furthermore, the question wording was changed to 'How often do each of the following people smoke' to try and encourage pupils to answer at each option.

Figure A.5: Version 3 of question on other people smoking :

Despite the reordering of response options and change to the question wording, some pupils still left rows blank, although the level of response was better than for the original version. The reasons given for the blanks from one pupils was because they did not know the answer (and didn't notice the 'don't know' box). Another said that because they did not have a sister they just skipped that row and did not look along it for 'don't have or don't see this person'. Also, pupils with only very young siblings commented that it was odd to be asked, for example, about whether your three year old brother smokes. They did, however, answer correctly after a bit of hesitation.

It was agreed to add further instructions to the question. This was tested in interview wave three and the additional instructions did appear to help as most pupils completed the question correctly. The recommendation was to use this version as the final version.

Question on whether the pupil is a young carer

This question was revised and then tested because, in the previous wave, it has resulted in implausibly high numbers of pupils reporting that they were young carers. As a result, a new question on caring responsibilities was designed (Figure A.7) and tested.

Figure A.6: Existing question on caring responsibilities

Figure A.7: Version one of question on caring responsibilities

The first wave of testing highlighted some confusion among pupils about whether they should include looking after a younger sibling when their parents were not around.

That said, seven out of the eight pupils tested in that wave understood that it should only include looking after an adult if that person was ill or disabled. However, one ticked 'Yes' because he cared about his mum.

Therefore, it was agreed that the text 'because they have a long-term illness or disability' to the question wording to avoid confusion (Figure A.8). There was a concern that the new wording may miss those caring for someone with a mental health problem, so this was explored in more detail in the second wave of testing.

Figure A.8: Version two of question on caring responsibilities

Version two of the question appeared to work better than the first and none of the pupils said 'yes' simply because they cared about someone. There was less confusion about whether looking after younger siblings (temporarily) should be included. Moreover, two pupils who were extensively involved in caring for disabled/ill siblings correctly answered 'Yes'.

However, one girl raised an interesting point: her friend has to look after a younger sibling because, she said, her mother 'is not fit to be a mum'. She said her friend has to 'do everything' - get the brother up, take him to nursery, give him his tea etc. She thought her friend probably would cross 'yes' because she would class herself as a young carer but she wasn't sure if she should because the brother isn't disabled/ill.

For this reason, it was decided that 'for example' would be added to the question in an attempt to ensure that situations like that described above are included. The third wave of testing focused on whether this addition would cause more or less uncertainty.

Figure A.9: Version three of question on caring responsibilities

Version 3 of the question worked well and the addition of 'for example' did not cause any confusion. This version was used in the 2015 questionnaire.

Question on what influences choice of drink

This question was tested because it had resulted in a high number of missing answers in the previous wave.

Figure A.10: Existing question version in 2013 survey

Several pupils said they 'didn't know' at 'How many calories it has in it' because they didn't know how many calories there were rather than they didn't know how much that influenced them. Similarly, some of the 'how easy it was to get' and 'how cheap it was' responses seemed to be a rating of how easy it was to get/how cheap it was, rather than how much that influenced them. In the existing 2013 version of the question, pupils needed to separate how they would rate a drink on different dimensions (cheapness, availability, taste) from whether that aspect actually influenced them, and then estimate how much it influenced them. The results of the first set of interviews indicated that this involved too great a cognitive effort to provide good quality data. The question was simplified for the second round of testing so that it just asked about what factors influenced their choice. In addition, 'last time' was underlined for greater emphasis (Figure A.11).

It is worth noting that all seven of those who answered said 'not very much' to 'how cheap it was'. However, they were all drinking relatively cheap drinks (MD 20/20, Cactus Jack, Buckfast and Blue WKD). This may be because they see their choice of drink in the context of what other (cheap) drinks other young people are consuming so don't feel it is a particular factor. Or because they are aware that even cheaper drinks are available: one pupil mentioned paying £7 for a bottle of MD 20/20 rather than the £6 he could have paid for a bottle of Buckfast (so he said he wasn't influenced by how cheap it was). Hence this question will at best only measure how much pupils perceive that they have been influenced by price (in their context) and not how much the price of alcohol does influence what or how much young people drink.

Figure A.11: Version two of question on what influences choice of drink

The simplified version in A.11 was found to work better than the original version and, in most cases, the responses fitted the story pupils told about why they had drunk that particular drink (which was not the case for the first wave of testing).

However, the issue of pupils tending not to say that price was an influence on their choice of alcohol beverage (referred to earlier) remained.

It was decided that the question would remain unchanged for the third wave of testing. However, further consideration should be given to whether the data will be useful from a policy perspective or whether the question should be dropped to create more space/time for other questions.

The pupils that participated in the third wave of testing were less likely to have drunk alcohol than those on the first or second day of testing. This may be why a further issue emerged. A couple of pupils in this wave of testing found it difficult to specify why they drank a particular type of alcohol and answered the question in terms of why they were drinking at all. This caused some confusion as the response options did not fit with the information they were trying to provide.

Given the problematic nature of the question, after further discussion, the Research Advisory Group decided to drop the question.

FINAL VERSION

Question removed.

Questions on how much their mother and father know about their activities

These questions were tested due to high levels of missing answers in previous waves of the survey.

Figure A.12: Existing version tested from 2013 survey about parental knowledge

In the first wave of testing, this question (and the same question about how much they though their father really knew about their activities) seemed unproblematic.

The only issue was that one pupil interpreted 'free time' as meaning time when his mother was out of the house (so, by that definition, she knew less about it). As a result, the question was amended so that the final row read 'What you do with your spare time'.

During the second wave of interviews, the question remained relatively unproblematic. Further probing on the use 'spare time' found that a couple of pupils still interpreted 'spare time' as being time away from parents (though it did not affect their answers). Furthermore, another pupil felt that 'free time' was a better phrase than 'spare time'.

The decision was taken to revert the final row text to 'free time'. The amendment to 'spare time' does not appear to have addressed the (slight) issue identified on day one so there is not a strong enough justification for making the change.

The final wave of testing confirmed that the original question version worked well and it was retained.

FINAL VERSION

No change to existing version - refer to Figure A.12

Questions on 'other' drugs that have been offered and taken

There was desire to check how well the item on 'other drugs' was working (the last item in Figure A.14) in terms of whether pupils noticed it and how they interpreted it. A similar cognitive test was also conducted on the question asking whether pupils have 'ever taken any drugs' (not shown). The question format is the same for both questions.

Updates to the list of drugs for the questions on 'ever been offered drugs' and 'ever taken any drugs' were not cognitively tested.

Figure A.14: Existing version tested from 2013 survey on if ever been offered drugs

The focus of the first wave of testing was on the last row 'Other drugs….'. All of the pupils said they were confused by this and were not sure what it meant.

One pupil had been offered cannabis. They did not notice it was on the list because they knew it as 'grass' or 'weed' and did not read as far as 'weed' in the list of other names for cannabis. They did not think to write it in at 'other drugs' because they were confused by what that meant and thought it was about drugs that would be given by a doctor, despite the instruction to the contrary.

One pupil had been offered 'base' (i.e. amphetamine) but did not know what that came under and, again, did not think to write it in under 'other drugs' because they thought that was perhaps meant for other drugs that you would get from a doctor, but at a higher dose than from a doctor.

Another pupil wondered if it meant whether you had ever been handed anything by a doctor. Another pupil wrote in 'paracetamol' but then ticked 'No' and said they didn't know what it meant.

Given the confusion of all pupils, it was agreed that the explanation in the last row would be removed and replaced with 'Any other drugs (please cross box and write in below the name of the drug)'.

As the explanation was removed, checks were made to see if pupils included drugs that they had legitimately been offered for medication during the second wave of testing.

Version two of the question worked better than the first and caused much less confusion. Pupils generally understood that if they had been offered or taken anything other than the drugs listed, they should write it in under 'any other drugs'. However, there were still issues, with some pupils either writing a drug name but failing to cross 'yes'; crossing 'yes' but failing to write in a drug name or not responding at all to the 'other' category. Therefore, a further amendment was agreed for testing in the third round of interviews.

The categories for 'Powders or pills that are sold as legal highs' and 'Other drugs that would not be given to you by a doctor or chemist' were separated from the main list and asked separately in new questions (see Final Version). This worked well and the new question design was agreed to be used in the 2015 survey.

Proposed new question on lessons about smoking, alcohol and drugs

This question was introduced as a new question in 2015 and, therefore, required testing.

Figure A.15: Version one question lessons, videos/ DVDs or discussion about substance use topics

In the first wave of testing, this question was unproblematic. The only exception was one pupil who focused just on 'videos/ DVDs' and did not notice 'lessons' or 'discussion'. However, this did not, in practice, affect their answer. Therefore, it was agreed that the question should be left unchanged for the second wave of testing.

The question was, again, unproblematic in the second wave of testing. Furthermore, no other pupils focused on 'videos/ DVDs' when answering the question.

Given that there were no issues with the question, there was no further testing in the third wave of research and the question was included in SALSUS 2015 unchanged.

FINAL VERSION

No change to first version - refer to Figure A.15

Proposed new question on how much learned at school about the risks of substance use

This question was introduced as a new question in 2015 and, therefore, required testing.

Figure A.16: Version one question about information learned at school

During wave one of testing, there was some confusion when pupils were asked about what 'legal highs' meant. For example, one thought it meant drugs like cannabis that were legal in other countries but not in the UK. That said, the confusion did not seem to affect their answers and they just answered in relation to 'drugs' generally.

One pupil forgot/did not notice the 'in school' part, so answered about how much she had learned in general. This affected her response to the risks from cigarettes. She put 'a lot' because her grandmother had died of lung cancer, but would have put 'a little' if she was just thinking about what she'd learned at school.

One pupil interpreted 'The effects that taking illegal drugs or 'legal highs' can have on other areas of your life (e.g. your safety, your behaviour, your relationships, how you feel about yourself)' as being about people turning to drugs because of something bad that had happened in their life.

The last item caused confusion. While most seemed to understand the concept of people's views being affected by the media, they tended to answer about how much people's views were affected, rather than how much they had learned at school about the fact that they can be. The concept is a bit more complex than the others, and requires a longer description. Asking pupils how much they have learned about this (rather than how much does this happen) AND how much of what they've learned was at school, was too cognitively difficult. While the second last item is almost as complex, pupils did talk about 'peer pressure' and gave examples of discussions in class and DVDs they had seen in relation to this issue.

These issues resulted in two amendments being made in advance of the second wave of testing (Figure A.17). First, illegal drugs and 'legal highs' was changed to drugs. The question was not trying to distinguish between the two, and was not specifically trying to find out what they learned about 'legal highs', so it was an unnecessary complication.

Second, the last item in the question was simplified and amended to 'That people's views about smoking, drinking and drug use can be affected by TV and films'.

Figure A.17: Version two question about information learned at school

The last item worked slightly better in the second wave of testing than in version one in the first wave. However, some pupils were still answering in relation to how much they thought people were affected by TV and films rather than how much they had learned in school about that.

In relation to the second last item, some pupils were saying they had learned in school that people have different views about these issues rather than learning that views can be affected by peers.

A further change was made to the question in advance of the third wave of testing. The second last item was changed to 'That people's smoking, drinking and drug use can be affected by the things their friends say or do' to reflect that it is their behaviour that changes rather than their views.

As the final item on the list was still proving problematic, it was recommended that if this was still true during the third wave of testing, it should be dropped from the question.

Figure A.18: Version three question about information learned at school

The third wave of testing highlighted that the changes made to the second last item, seemed to improve comprehension and the decision was taken to retain it. The last item continued to be problematic (pupils answered in relation to how much they thought people's views were affected, rather than how much they had learned at school about this) and the Research Advisory Group agreed that the item should be dropped.

Proposed new question on whether school provides advice and support to make healthy choices

This question was introduced as a new question in 2015 and, therefore, required testing.

Figure A.19: Version one question about advice and support received from school

Version one of this question seemed relatively unproblematic although it should be noted that it came close to the end so was there was not enough time in some interviews to explore it thoroughly.

One girl interpreted the question as being about whether she would go to someone in the school if she wanted advice or support.

The second wave of testing suggested that the 'important decisions' element was being lost and pupils tended to answer in relation to whether the school provides advice and support.

It was decided that an alternative version would be tested (Figure A.20) which focused simply on whether they had sufficient advice and support to take decisions. It was thought that this may help pupils focus more on the decisions element.

Figure A.20: Version two question about advice and support received from school

Version two of the question worked well in the third wave of testing in terms of comprehension. However, discussion at the Research Advisory Group clarified that whether the school was providing enough advice and support was too important an element to drop. It was felt that this was more important than the 'important decisions' element and the final version below was agreed.

Proposed new question on confidence in future wellbeing choices

This question was introduced as a new question in 2015 and, therefore, required testing.

Figure A.21: Version one question about confidence in future wellbeing choices

This question appeared to be straightforward and was understood by pupils. The exception to this was that a couple of pupils were not sure what 'substance related issues' meant.

However, it should be noted, as with the previous question, it came close to the end and was not covered in as much detail as earlier questions.

In advance of the second wave of testing, the question was amended so that 'substance related issues' was changed to 'smoking, drinking and drug use'.

The question was covered in more depth in the second wave of testing and it was confirmed that there were no comprehension issues related to this question. In the end, the original version was used in the final questionnaire.

FINAL VERSION

No change to first version - refer to Figure A.21

Proposed new question on whether would talk to parents if worried about something

This question was introduced as a new question in 2015 and, therefore, required testing.

Figure A.22: Version one question about whether would talk to parents if worried about something

In the first wave of testing, pupils appeared to find this question uncomplicated. However, it should be noted that it came close to the end, so it was not discussed in as much depth as the earlier questions.

Only one amendment was made to the question in advance of the second wave of testing. It was decided that the dark line in front of 'Don't have or see this person' should be removed because of the issue identified with similar formatting on the question about whether other people smoke.

Figure A.23: Version two question about whether would talk to parents if worried about something

One issue came up in the second wave of testing, a pupil left father/carer blank because she didn't notice the last column ('Don't have or see this person').

Therefore, as with the question on whether other people smoke (discussed above) it was decided that the additional instruction 'IF YOU DON'T HAVE OR DON'T SEE THE PERSON, CROSS THE BOX AT THE END OF THE LINE' should be tested.

Figure A.24: Version three question about whether would talk to parents if worried about something

Version three worked well in the third wave of testing and it was confirmed that this version would be used in the SALSUS 2015 questionnaire. In further discussion with the Research Advisory Group, it was subsequently decided to add 'Someone else in your family'.