A process evaluation of the implementation of ASSIST in Scotland

Chapter 6: Discussion and Conclusions

This final chapter will discuss our findings and finish with a brief conclusion to summarise the key points and future recommendations.

6.1 Discussion

Chapter 2 in this report set out the research questions for the study. Here we reflect on how each of these have been addressed through our findings, discussing key results for each question in turn. Recommendations are highlighted in italics throughout.

1. What are the barriers and facilitators to the implementation of ASSIST in Scotland?

Barriers and facilitators in this study were categorised as macro (strategic) and micro (operational). At the macro level they were: partnership working; budget; and culture. Delivery of ASSIST in Scotland was led by NHS Boards, but with a significant contribution from Local Authority partners working in education and Community Learning and Development. Local Authority contribution varied across sites, ranging from offering support to recruit schools to in-kind staff time to deliver training. Relationships with schools were key to the successful delivery of ASSIST. One of the reasons schools agreed to take part was previous successful collaborations with the pilot sites. Identifying a school lead who was supportive of the programme was clearly advantageous. The evaluation has highlighted examples where the school staff had taken the time to encourage students to take part who were perhaps unsure and also identified suitable classrooms and organised student consent forms, etc. In addition, establishing a good rapport with the school lead made scheduling the different elements of the programme easier as they could advise on timing (e.g. not arranging slots over lunch or after school). All three pilot sites had an established relationship with their partners, some stronger than others. Future delivery of ASSIST in new areas should ensure enough time is set aside to build these relationships if they do not exist already.

At the micro level barriers and facilitators were: trainers; delivery mode (peer nomination, training and follow-up); and behaviour management. Due to the different delivery models across the pilot sites, some trainers had a delivery role only, while others were responsible for both coordination and delivery. In addition, some trainers delivered ASSIST alongside other responsibilities completely separate to ASSIST. The latter caused some anxiety for trainers with regards to managing these different commitments. Feedback from trainers on their three day training to deliver ASSIST was extremely positive. They found it particularly useful because it equipped them with first-hand experience of delivering the programme and provided an opportunity for trainers to get to know each other prior to delivery which built confidence. Potential suggestions for improvement included more time to go through the follow-up sessions.

Levels of confidence to deliver the programme were influenced by professional background. Most trainers tended to have either a background in youth work or smoking cessation, not both. Naturally this caused some anxiety pre-delivery but the model used in some schools of pairing up trainers with different experience seemed to go some way to address this anxiety. Pairs of trainers with a mixture of smoking cessation and youth work expertise should be applied (where possible) in future.

Peer nomination was only delivered under exam conditions in one of the eight case study schools, resulting in student 'chatter' which could have influenced whom they nominated. There was a suggestion that peer nomination would be better administered on a class by class basis (especially in schools with a large year group). However, this would have resource implications for delivery and create logistical challenges for the school. Future deliverers of ASSIST should adopt a pragmatic approach to balance best practice and fidelity for programme delivery alongside schools being able and willing to sign up to the programme.

The style and content of the peer supporter training were viewed favourably by staff and students, with trainers praised for treating the students like young adults, rather than children. However, there was discussion around behaviour management with some schools suggesting a lack of authority which, at times, resulted in students not engaging or respecting the trainers. This was clearly very challenging for trainers and highlights the difficulty they face trying to balance the informality of the ASSIST model alongside the 'formality' required to keep the students safe and equip them with the skills and knowledge required to fulfil their peer supporter role.

Trainers faced some challenges delivering follow-up sessions in school - classrooms were not ideal and time to deliver the session was often too short. There was also general discussion related to the number of follow-ups with a view that students may have found four sessions too many - showing signs of disinterest and boredom. Diary use was limited, which led to discussion around whether they were the best format for peer supporters to record their conversations, with some sort of phone based device or application viewed more favourably. Fewer follow-ups should be considered in future and diaries, if included, may not need to be paper-based.

2. What refinements are required to implement the ASSIST programme in Scotland?

Overall, we found that very little adaptation was required to implement ASSIST in Scotland, with DECIPHer-ASSIST praised for the quality of delivery manuals, support and training offered as part of the license fee.

Evidence of the acceptability of delivery to S1 or S2 was limited. School leads tended to focus on the reasons why one year was chosen over the other - not how this decision may have influenced delivery of the programme, with no apparent challenges delivering to S1 or S2. Student feedback, however, suggested that friendship groups may have been more established in S2 than S1. Therefore, in light of student feedback, if ASSIST is delivered in S1, this should ideally be in the second half of the school year when friendships are more established.

3. Were essential elements of the ASSIST model maintained during pilot implementation in Scotland?

Overall, delivery appears to have been with a high degree of fidelity to the licensed programme. Fidelity measures for the peer supporter critical mass of 18% were met. All case study schools delivered the four follow-up sessions, but observation fieldwork highlighted that diaries were not consistently checked in two of the three case study schools.

4. How acceptable is the programme from a stakeholder perspective (strategic leads, trainers, students and school staff)?

Stakeholder feedback suggests that delivery of ASSIST in schools in Scotland was acceptable with the (unavoidable) minor disruption to school timetable accommodated. Similar to findings from the process evaluation of the 2008 RCT, school leads indicated support for the peer education model and programme delivery which required a small amount of school resources (Audrey et al 2008). Trainers demonstrated a clear understanding of the theory behind ASSIST and recognised the importance of delivering the programme according to the manual to maintain the fidelity of the intervention. They were particularly clear that schools should not interfere with the peer selection process or delivery. School leads demonstrated a similar commitment to the peer element of the programme but there were examples of anxiety around student selection and an initial concern that the critical mass of 18% might not be achievable, although it was.

The general opinion was that students selected to be peer supporters viewed this positively, but opinion was mixed regarding their motives and suitability to fulfil the role.

5. What changes in smoking-related knowledge, attitudes and behaviour are observed amongst students in the ASSIST Scotland pilot schools?

Findings from this evaluation have clearly shown that ASSIST continues to make a significant and positive contribution to peer supporters and their schools. This contribution goes beyond learning about tobacco harms and discussing tobacco use with peers, to wider benefits in terms of building knowledge and skills amongst peer supporters. Feedback from participants was overwhelmingly positive regarding the benefits of taking part in ASSIST for peer supporters, commenting on the personal skills and social contacts they have gained and the potential for the school and wider communities to benefit. Recognition of the wider benefits of peer supporter training should be included in discussions around the future of the ASSIST programme in Scotland.

Naturally schools were interested in what 'impact' ASSIST had in terms of smoking prevalence, even though this process evaluation was designed to look at acceptability and fidelity, not intervention effectiveness.

From the data we have, it is clear that there is uncertainty regarding the extent of message diffusion between peer supporters and peers in their school year and any impact this may have on adolescence smoking. Findings from the student survey showed no significant change in self-reported smoking prevalence with 1.6% of pupils (n=33) reporting that they smoked one or more cigarettes per week increasing slightly to 1.8% (n=38) at follow-up. In addition, recall of any relevant conversations about smoking with a peer supporter was fewer than one in ten (9%). Despite the caveats around the sampling approach used (discussed in 2.3), this was considerably lower than positive responses to the same question which was asked of students who participated in the original RCT of ASSIST. Indeed in the trial, just under one in four students (23.9%) recalled that a peer supporter had spoken to them about smoking (Audrey et al 2006).

Despite these findings, it is important to recognise that opportunities to have informal conversations about smoking with peers may now be limited due to the ongoing decline in adolescent smoking since ASSIST was first developed. This decline may also partially explain why schools did not have their own clear smokefree policy and why there was limited student awareness of any existing smoking prevention lessons. It may well be that addressing smoking has become less of a priority in schools. It is also worth noting that some peer supporters felt apprehensive or awkward initiating conversations about smoking with their peers and this also may have contributed to the low recall of any conversations relevant to ASSIST.

The original ASSIST trial is now 13 years old. Although our study was not designed to assess effectiveness, our modest findings on the number of relevant conversations (in particular) do raise questions about any appropriate assessment of ASSIST in the future. The programme is still fairly widely delivered in England and Wales. A relevant future step would be an implementation trial (a Phase IV study) of ASSIST, using a similar methodology to the original RCT, to assess current effectiveness in the context of lower smoking prevalence in the target age group.

Another important factor perceived to influence message diffusion was the view that peer supporters were more likely to talk to family members than their school year peers. Findings from peer supporters supported this view suggesting conversations with parents and other family members was a common occurrence. This is an important research area which is currently being explored by a PhD student. The impact of ASSIST in encouraging peer supporters to discuss smoking with smoking parents, other family members and wider social networks is an important area for research.

6. What are the delivery costs of the programme?

The average cost to deliver one cycle of ASSIST ranged from £8,939

£11,357. Between August 2014 - June 2017, 72 cycles of ASSIST were delivered or confirmed for delivery in 2017. The actual total delivery cost for all three sites was £674,360 of which 20% (£136,800) was for the license fee. This equates to a cost per cycle of £9,366. This cost is higher than the comparable cost analysis data conducted as part of the part of the original RCT, with costs per intervention school being £5,662 (Hollingworth et al 2013). However, these costs did not include the license fee which accounted for 20% of the cost per cycle and were for one delivery model only.

7. What lessons can be learned to assist future roll out across Scotland?

Key lessons have been discussed above and we have included recommendations in italics. Overall, we believe that there are five considerations for any future delivery of ASSIST in Scotland, and a sixth relating to future research.

First, if a relationship between programme deliverers of ASSIST and the school is not already established, time should be included to invest in this before delivery. This will help with programme delivery but also add important context in terms of what smoking prevention provision the school already has in place. If delivery is in partnership with NHS Boards, Local Authorities and the third sector, time needs to be built into the delivery timetable to establish these relationships and understand the level of resource required from each partner.

Second, identifying trainers depends on available resources and they are likely to have a variety of professional backgrounds. Using this experience to pair up trainers with different backgrounds (e.g. one with a youth work experience and one with smoking cessation) and developing opportunities to share practice and experience across sites may aid delivery and increase confidence.

Third, if the school timetable permits delivery of ASSIST to S1, students should be targeted during the third term, as this will maximise message diffusion via established friendship groups. Few problems were observed with delivery in S2, so this continues to be viable.

Fourth, consideration should be given to how trainers can be offered further support (e.g. further training, better use of teaching staff who chaperone students) to manage student behaviour if they feel this is required.

Fifth, in terms of the manual and intervention delivery, DECIPHer-IMPACT may want to consider: 1) whether a paper diary is still the best medium for students to record their conversations; 2) if four follow-ups are still productive; 3) how to include content on e-cigarettes in the programme, as appropriate (in particular, making clear that they are far less harmful than tobacco but are not products for teenagers that have never smoked, and informing teenagers of age of sale laws).

Sixth and finally, there may now be a need for a Phase IV trial of ASSIST to determine effectiveness in the context of continued decline in smoking prevalence. This should, if possible, take into account that smoking still remains an issue particularly in more deprived areas.

8. Is there scope to expand the model to look at other risk taking behaviours in Scottish schools in the future, e.g. drugs, alcohol?

There was general agreement (from school lead and students) that, in theory, the ASSIST model could and should be applied to other risk taking behaviours such as alcohol or drugs. Indeed it may have real promise for these behaviours. There was strong support for the peer model of delivery but in the current climate of continued budget cuts, NHS Boards and Local Authorities may be cautious to direct funding to ASSIST because it only covers one risk behaviour. Currently and previously there has been other versions of ASSIST (sexual health, physical activity, drugs). Given the very low prevalence of smoking now in the target age group, using the ASSIST model in relation to these other behaviours may be particularly important. It is worth highlighting that a feasibility trial of the ASSIST model as applied to drug prevention (ASSIST +Frank and Frank friends) has just concluded, conducted by some of the authors of this report. Plans are now underway to seek funding for one component of this from NIHR for a larger trial. Early results look promising. School staff and stakeholders were also interested in how the ASSIST model could address multiple behaviours in one intervention, but this may be far more challenging to deliver. Investigating how/if the ASSIST model could be developed to address more than one risk behaviour is an important area for further research.

School leads were motivated to make the best use of the skills they have acquired from ASSIST. A range of ideas were suggested, but no evidence was gathered of explicit actions taking place or timescales around this. This is perhaps a reflection of the follow-up interviews taking place soon after the programme finished. Finally, it should be noted that suggestions centred on raising the profile of ASSIST within the school. This may have fidelity implications if the school plans to deliver ASSIST to subsequent years - a key component of the programme is that students do not know anything about ASSIST as this may influence who they nominate to become peer supporters.

6.2 Conclusion

Overall, this process evaluation has demonstrated that it is feasible and acceptable to deliver the ASSIST programme in Scottish schools. Despite slight differences in the age of young people participating compared to the original programme in England and Wales, and some organisational and socio-demographic variation between the three participating areas in Scotland (Glasgow, Lothian and Tayside), the programme was delivered to a high degree of fidelity. All the research questions set out in the original proposal were addressed, and recommendations for practice and research have been set out above.

Three different delivery models were piloted in the participating areas. This did not impact on fidelity or acceptability but we have outlined the strengths and weaknesses of these different models. There are learning points from them to apply to other areas that may implement ASSIST in Scotland. For example, the manual states that peer nomination for the whole school year (via a special assembly) should, ideally, be delivered under exam conditions. This was observed in one case study school only. There were various reasons for this and a pragmatic approach is required to balance fidelity of the intervention and securing schools for delivery. Partnership working, from the onset, was viewed as being key to successful delivery and securing school participation. Feedback was overwhelmingly positive regarding the wider benefits of taking part in ASSIST for peer supporters (i.e. personal and communication skills) but also for the school and communities, via message diffusion to wider social networks. In addition, we have documented the cost of delivering the programme in each area, which provides useful information to other areas or schools considering ASSIST.

Our findings show less certainty regarding the extent of message diffusion and any impact this may have had on adolescent smoking. Student survey results showed no significant change in self-reported smoking prevalence between baseline and follow-up and conversation recall with a peer supporter was low at 9%. There are caveats around the interpretation of these results which were not the main focus of this process evaluation. It is also important that the current context (where regular smoking prevalence is 2% overall in 13 year olds in Scotland^[8]) is taken into account. Now may be the time to consider whether, 13 years on from the original RCT, an implementation trial of ASSIST is warranted to determine if it is still effective and cost effective. It may still have an important role to play, particularly in more deprived areas where youth smoking uptake starts in the early teens and where community smoking rates and norms have shown little change in recent years.

Specific (although more minor) adaptations to the existing ASSIST programme should also be considered. In particular: the utility of paper diaries; whether four week follow-ups are required; and how content on electronic cigarettes can be included in a way that makes clear the important distinctions between vaping and smoking.

Further consideration is also merited regarding the best school year for any delivery of the programme in Scotland i.e. S1 or S2. Findings from the process evaluation gave no clear guidance over one year or the other. However, considering the very low rates of smoking amongst young people in their very early teens today, and the relevance of peer groups being formed when the programme is delivered, S2 may be more appropriate.

This process evaluation has demonstrated that it is feasible and acceptable to deliver the ASSIST programme in Scottish schools, although questions remain about the extent of message diffusion. Further consideration is required to assess whether delivery of ASSIST still offers a suitable return on investment and what role it may play in schools in areas of deprivation where smoking rates are higher.