Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): technical report 2018

Limitations of the data

As with any survey, there are a number of limitations of the data. These include those already mentioned earlier in this report:

• The data are from a sample of the population as opposed to a census and, therefore, subject to sampling error
• Non-response of schools, classes and pupils
• Item non-response

There are other limitations that have not already been covered in detail. These include:

• The self-report nature of the results. As with all surveys, the results can only tell us what respondents say that they do, think or feel. We have to assume that their answers are honest and accurate. Concerns about the honesty of responses are particularly applicable to a survey such as SALSUS which covers sensitive topic areas. Although steps were taken to encourage honest answers, such as pupils sitting in exam conditions and being reassured of the confidentiality of their answers, we do not know how honest pupils were. However, English surveys of smoking, drinking and drug use among secondary school pupils carried out throughout the 1990s used saliva tests to validate the estimates of the prevalence of smoking derived from the questionnaire and smoking diary. The results consistently suggested that pupils were providing honest answers. While there is no such evidence relating to drinking or drug use, it is assumed that the proven high level of honesty extends to these substances.
• Accuracy is also an issue with self-report and accuracy of answers will vary. Some questions will be easy for pupils to answer and should, therefore, produce accurate results; for example, would you like to give up smoking? – Yes, No, Don't know. Other questions rely on the accuracy of pupils' recall and ask them to quantify their behaviour; for example, how many cigarettes did you smoke on each day in the last seven days, ending yesterday? Answers to this type of question are less likely to be accurate.
• The sample only covers pupils in mainstream secondary education. The sample excludes pupils in special schools, secure residential units and those who are home schooled. Substance use may be very different among pupils who are not in mainstream secondary education and hence the results from SALSUS cannot be applied to these pupils.
• Pupils who are supposed to attend mainstream schools but don't (e.g. absent through truancy or exclusion) are potentially less likely to have taken part. Findings from studies show that absenteeism due to truanting and exclusion is correlated with substance use. For instance, in the 2018 survey, 80% of 15 year old regular smokers had truanted in the past year, compared with 46% of 15 year old non-smokers. Similar patterns are evident in relation to alcohol and drugs.