National Trauma Training Programme - local delivery trials: interim evaluation

Outputs from the process evaluation

Theories of Change

A range of ToC models were produced from stage 1 and 2 of the process evaluation. A strategic ToC for the overall Delivery Trial Site programme which included assumptions that underpinned the overall programme theory is in Appendix 2. The ToCs for each individual Delivery Trial Site are in Appendix 3.

The delays and postponements across all sites as a result of Coronavirus (COVID-19) meant that the only training that was delivered was that targeted at senior managers and leaders - STILT training delivered by NES and associated awareness raising roadshows delivered by NES and/or local teams. This evaluation cannot therefore report back on the reach and short-term outcomes or impact of frontline staff training. All that can be reported is learning from the planning phases and delivery of STILT and associated awareness training for senior managers.

Progress in achieving planned activities

Substantial work was undertaken across the sites to progress the activities listed in their ToCs during the 6/7 months they were operational. This included:

• the establishment of the interagency steering/ implementation groups
• conducting needs assessments or workforce mapping of the specific settings and workforce
• considering appropriate roles and methods for engaging and recruiting people with lived experience of trauma
• agreeing content of training courses and any specific tailoring needed to the content recommended by NES for the various levels of training and ensuring this aligned with the results of needs assessment and previous training delivered within the specific contexts
• agreeing communication plans to raise awareness and recruit targeted participants for training
• identifying the best mode of training delivery (in house and/or externally commissioned)
• the delivery of training and agreed support and supervision (delayed in part due to Coronavirus pandemic)
• identifying processes to embed and sustain training in ongoing workforce planning
• considering monitoring and evaluation processes.

Similarities and differences in planned activities across sites

The key contextual and intervention differences across the three sites are illustrated in the figures in Appendix 4. Appendix 5 provides further explanation of similarities and differences in each of the specific activities progressed across the sites.

Learning

The NTTP highlights a number of expectations and principles for local implementation structures intending to develop trauma informed workforces and organisations. These processes and outcomes where change was anticipated include:

1. Provide local oversight, governance and planning of sustainable local training delivery and implementation - Leadership

2. Committing to trauma-informed principles and values – Culture change

3. Use available local evidence and take local needs and priorities into account, including access and equity of access to support, care and interventions, service, organisational needs and cost effectiveness – Needs Assessment

4. Committing to training and translation of training into practice and promote trauma-informed practice (TIP) - Training, TIP and service change

5. Promote cross-sector, partnership working and third sector involvement -Partnership Working

6. Develop staff support, coaching and supervision systems – Staff Support, supervision and self-care

7. Ensure involvement of people with lived experience in local training initiatives and developments - Involvement of people with lived experience and co-production

8. Monitor and evaluate outcomes

Many of these processes and outcomes align with domains within implementation and audit tools in the wider TIP literature^[7],[8]. An example of this alignment is highlighted in Table 2, which compares the above NTTP expectations with domains from Trauma-Informed Care and Practice Organisational Toolkit (TICTOP)⁵ revised in 2018 and currently used to support the roll-out of TIP across Australia’s public services.

Table 2: Alignment of NTTP and Trauma-Informed Care and Practice Organisational Toolkit ( TICTOP) domains

TICTOP Organisational Audit Domains
Domain A	Domain B	Domain C	Domain D	Domain E	Domain F
Governance, management and Leadership	Organisational Structure and Policy	Consumer and Family Participation and Peer Work	Direct Services to Consumers and Families	Healthy and Effective Workforce	Outcomes and Evaluation
NTTP Expectations and Principles
1 & 5	2	7	4	4 & 6	3 & 8
Leadership and Partnership Working	Culture Change	People with lived experience & Co-production	TIP and Service Change	Staff Training Support, Supervision and Self-Care	Needs assessment and Monitoring and Evaluation

Given the consistency of these process and outcome areas with those in the wider TIP literature they are used below to structure the feedback on progress and learning from the Delivery Trial Sites. It should be reiterated that significant challenges including pre-existing workloads and pressures, staff absences, and the outbreak of Coronavirus (COVID-19) combined to limit implementation progress. These barriers also prevented access to participants and data to address the process evaluation objectives. The findings and lessons below are therefore based only on the limited information and data accessed from those research activities achieved - illustrated in Table 1. As such, the findings should be treated with caution and seen as early feedback that should be further informed/refined

Leadership and Partnership Working

Engaging leaders

The main tool used in the Delivery Trial Sites to engage and enthuse leaders was awareness raising roadshows and STILT training. Two of the three sites delivered both their awareness raising and STILT sessions to a range of senior and operational leaders and managers in their areas. A STILT session had been requested for Senior Board members at a third site but had not as yet been delivered. Additional sessions were also planned with elected members in two sites. The STILT and roadshow sessions were intended to raise awareness and achieve a shared understanding of TIP and its impact as well as the benefits of culture change to embed the principles of TIP.

STILT training was positively received although session vibrancy varied slightly depending on the range and mix of attendees. Catering for varying knowledge levels was challenging for leadership/manager awareness training provided by the site teams (rather than NES). Some services, such as criminal justice, were already vary aware of, and working on, trauma whereas other services were less so. Where delivered, these sessions helped secure support for frontline staff training.

There were some challenges encountered in identifying the appropriate levels of leaders to attend STILT and this varied across sites. NES describes STILT as training for senior leaders within partnerships but in fact many who attended were senior operational managers. In the site focusing on a locality, rather than a whole HSCP area, local lead officers were initially not targeted for STILT yet had roles and responsibilities for population groups as large, complex and perhaps more socio and economically deprived as those ‘more senior’ leaders targeted within other sites. Delivery Trial Sites were however all intending to include the most senior partnership and political leaders (e.g. HSCP Board members and elected members) in second waves of leadership training.

Engaging partners

Participation as a Delivery Trial Site brought additional work to partnerships and services that were already experiencing substantial work pressures. One area reported that insecurity of staff contracts and staff turnover added to challenges of progressing the project. In another site staff illness and absence impacted substantially on overall progress. All areas established multi-agency implementation/steering groups to lead the work. This process was at an earlier stage and less stable in Site 2 due to staff sickness absence and/or as a result of the locality focus - given different agencies may have had non-aligned boundaries and roles.

Unsurprisingly, faster progress seemed to be made in areas with consistent project leaders, an established /stable multi-agency implementation group and where project managers were identified with dedicated time allocated to their role.

Engaging third sector partners

Contextual issues such as geography, quality of existing partnerships and prior involvement in TIP influenced the varied approaches to engagement of third sector partners.

In Sites 2 and 3 third sector agencies were involved in the implementation/steering groups. In Site 2 such third sector participation was from an agency with recent experience of the roll-out of a training programme for those working with the sensory impaired. In Site 3 a cohort of third sector partnership staff had participated in the train the trainer programme and were part of ‘in house’ training team for Level 2 input.

Site 1, a more rural area with training capacity issues took a unique approach which saw them ‘commission’ the tailoring of content and delivery of level 2 and 3 training from a consortium of national third sector agencies that were experienced providers of services for CYP with significant experience of trauma (e.g. residential child care providers). Table 3 highlights key aspects of, and learning from, this approach.

An unanticipated issue that arose in this process related to the need for consideration of and negotiated solutions to ownership and management of intellectual property with regard to training content, tools and approaches in joint developments between statutory services and national and local third sector agencies -who brought their own content, experience and expertise to the task, partnership and consortia.

Table 3: Trial Site 1 -Third Sector Training Consortium

Membership

The coalition was brought together through Care Coalition Providers Scotland (CCPS). It included the Scottish Throughcare and Aftercare Forum (STAF), Kibble, Barnardo’s, and Aberlour. The consortium was led and overseen by STAF.

Remits

In partnership with Site 1’s implementation team the consortium was to collate, adapt and tailor content for level 2 (skilled) and level 3 (enhanced) TIP training for CYP’s Services and to provide trainers for the delivery of the agreed package. The training content was to be used going forward by Site 1 to ensure sustainability of training. The Delivery Trial Site implementation team were to coordinate the participant recruitment, timing and venues for training.

Training content and delivery

NES training content was blended with material from the consortium agencies’ existing whole systems TIP change programmes that are delivered within their own organisations and in some instances to other partnerships. A substantial proportion of the content came from consortium members and this was blended with content from Site 1’s implementation partners (NHS and Local Authority) that was specific to their local and service contexts, systems, needs and levels of demand. For example, content took account of the GIRFEC structures and processes locally and service user numbers and access routes for CYP’s Services in rural and remote areas. The content had not, at the time of reporting, been fully signed off.

Training delivery was not progressed due to COVID and associated lockdowns. Delivery was to be face to face. The anticipated throughput was that each of the 4 localities would have 1 skilled (level 1) and 2 enhanced (level 2) courses with level one being delivered over 2 days and level 2 being delivered over 3 days. Attendance was anticipated as circa 20-25 at each course and with the likely need for an additional mop-up session. The key focus was to be on social workers, residential, education and health staff delivering CYP’s Services. The training was intentionally developed as a workshop model where knowledge on TIP would be refreshed and reinforced but where there would be space to reflect on current and future practice and work in multi-disciplinary groups on how to challenge practice, system and cultural issues relating to TIP. The consortium was keen to ensure a focus beyond individual and team practice to issues of leadership, organisation and partner systems, structures and culture to effect change.

Challenges and learning

There appeared to be an initial lack of clarity over whether SG/NES guidance was encouraging third sector engagement in this site as a partner and co-deliverer relationship (where input might have been contracted via SG/NES) or, unfolded, a more traditional commissioner and contractor arrangement. The collaborative and administrative processes in setting up the coalition and delivery arrangements were more complex and time consuming than anticipated in part because of this confusion.

Consortium members contributed aspects of their agencies’ existing training content and experience into the development of the adapted and tailored localised Site 1 materials. In addition, members spent time accessing free and publicly available content and material from other organisation and providers. Site 1, as the ‘commissioners’ requested ongoing access to and ‘ownership/copyright’ of these outputs and materials as part of the contractual arrangement. This contractual arrangement was seen as unusual by and raised challenges for consortium members with regard to existing legal arrangements covering their specific content and indeed the Intellectual Property rights around the contract outputs. Challenges around IP for training content and materials and therefore the immediate and longer-term commercial value of the consortium’s input were not fully resolved pre COVID-19. Key lesson to unfold from this process should inform future sensitivities and issues around contract specification, commercialisation of outputs and appropriate costing of such partnership contributions.

The balance of the consortium’s role between content development for and delivery of training and any potential wider support and influence for structural and culture change as part of the agreement was discussed with the implementation team. The former, more transactional rather than collaborative, relationship was prioritised at least initially and in part due to short timeframes and resource restrictions.

Existing training content that was used to contribute to the specific and tailored outputs had previously included co-produced contributions and oversight from people with lived experience of trauma. Whilst the consortium was keen to have further co-production and involvement of people with lived experience, limited timescales and resources may have challenged or curtailed this in practice.

Evaluation materials for the training had been developed but the overall monitoring and evaluation of the training and other aspect of the contract were still to be finalised.

This type of consortium approach as a mechanism to fully involve third sector partners was unique to Site 1. Whilst it encountered several challenges, the resulting learning if taken on board may have much to offer in terms of collectively addressing TIP challenges and fully utilising the extensive knowledge and expertise of third sector partners. Informing such potential collaborative gains was a key intention of funding the Delivery Trials as part of the NTTP.

Organisations versus multi-agency partnerships

Despite TIP work having started prior to the establishment of the Delivery Trials it was evident in all sites that gaining leadership approval across partnership services, mobilising multi-agency participation and agreeing training content, schedules and roles and responsibilities still took substantial time and effort. This finding concurs with the wider literature which emphasises the long-term nature of the endeavor to influence and change trauma improved practice and culture within organisations^[6,9]. The CCTIC framework⁷ for example, developed to support such culture change in an organisation, suggests it requires between 2-5 years to achieve, embed and sustain such change.

Much of the literature on TIP is informed by research that focuses on driving practice and culture change through single (even if large-scale) health and or care organisations. All three NTTP Delivery Trial Sites were trying to progress TIP across partnerships rather than simply organisations. It is likely that driving such change across multiple organisations with varying purposes (e.g. health, care, education, leisure, etc.) is even more challenging and time consuming.

Culture Change

In each locality implementation built on previous trauma related work. All areas, for example, had previously completed work on Adverse Childhood Experiences (ACES) as part of the SG’s 2017/18 Programme for Government which committed to preventing ACES, helping to reduce the negative impacts of ACES where they occur and supporting the resilience of CYP, families and adults in overcoming adversity. Figure 2 illustrates the relationships between ACES and other aspects of trauma. Delivery Trial Sites used their experiences of ACES to inform their TIP plans and activities.

Esaki 2019^[10] suggests that trauma informed leadership is key to achieving a trauma responsive culture. Leadership is also highlighted as a key driver within the KSF² as illustrated in Figure 3^[11].

Much of the ACES work completed previously in sites would likely have initiated changes in practice, staff support and culture within their partnerships and organisations. Figure 4 shows the many changes needed within organisations as they become increasingly trauma transformed.

Due to implementation delays only one Delivery Trial Site participated in the reflective feedback sessions (see Table 1). There were therefore as yet limited data to inform lessons with regards to influencing culture beyond the STILT/leadership training and co-production of carers training in Site 1 discussed above.

One site suggested that partnership services and agencies with more rigid rules that service users must abide by (e.g. housing, finance, police) sometimes struggled more with aspects of culture change associated with TIP. However, with time, these services were also ‘coming on board’ even within the short time frame of the trials.

Site 3 suggested that to secure agency participation TIP training needed to be sold as a win-win situation. Emphasising that staff will value and enjoy training and benefit from the training in terms of their own experiences and responses to trauma encourages leadership commitment. Further learning about progressing culture change may result from the process of delivering training and as services try to translate training into practice and service change.

German et al⁸ like Esaki⁹ highlights that culture change requires additional focus beyond training and must include staff health and wellbeing actions that ensure appropriate staff support, supervision and self-care as well as policy and practice alignment.

The Delivery Trial Sites (like the wider NTTP) acknowledged the need for these additional cultural elements related to staff supervision, care and wellbeing and involvement of people with lived experience within their plans. Progress towards these are discussed below.

Involvement of People with Lived Experience^[13]

Exposure to adversity and trauma in Scotland and elsewhere is common and can substantially impact upon short and long-term life chances, wellbeing and quality of life^[3,14]. Within some services there are often particularly high rates of people who have lived through trauma: 75% of women and men attending substance misuse services, for instance, report abuse and trauma in their lives^[15]. Among people in prison, studies have found 94% of people report a history of trauma^[16] and in inpatient mental health services 60% of women and 50% of men report being sexually or physically abused in childhood^[17]. Research suggests that many people experience events described as traumatic – rapes, assaults, traffic accidents for example – at some point in their lives^[18]. Given these prevalence figures many people working within public services will also have lived experience of trauma and may be able to use that to influence their own practice, services and organisations.

Whilst Trial Sites had set out in their applications that people with lived experience would be involved at a strategic level within implementation/steering groups the reality of achieving this seemed more challenging. Some participants highlighted that people with lived experience may prefer to be involved in other less strategic ways such as co-training, peer evaluation etc.

Site 1 had plans for involvement of people with lived experience in a specific and bounded area of their programme. They focused on co-production of bespoke training for foster/kinship carers and adoptive parents. Once complete this element of their programme should provide both a useful training product and learning on the co-production process for this important target group for the NTTP.

In Site 2 the HSCP Board was reported to have representation from people with lived experience but the trial site implementation group was not yet stable and it did not as yet have formalised involvement from people with lived experience.

In Site 3 involvement of people with lived experience at a strategic level was sought via a Local Authority wide Advocacy Group but was not yet secured. The project team recognised the need for involvement of people with lived experience who were engaged with more local services but they had not as yet investigated the extent to which those using their service Hub might wish to participate or the type of involvement that would be desired or appropriate for those who might volunteer/contribute.

Trauma Improved Practice and Service Change

Given that the Delivery Trial Sites were at an early stage and had not yet conducted their frontline training nothing can be said about changes in practice and services that may have resulted post training.

Two of the three sites reported that early activity had led to interest and demand for input and training from other services within their partnerships and/or an increased interest from, and opportunities to engage with, strategic leaders (e.g. CEOs and Elected members) to scale and spread TIP and training. Their early work had also raised awareness of the wider NTTP and associated tools as well as local activity.

Staff Training, Support, Supervision and Self-care

Staff Training Content

The NTTP outlined potential content to be covered and tools available for different levels of training. Figure 5 shows the intended alignment of NES animations, existing training content and e-learning tools to the various levels of training.

The resources provided by NES were generally well received. The NTTP and the KSF were used to inform training content and overall site training plans. The KSF booklets were reported as very useful for collaborating with third sector colleagues. The animations were also seen as informative and useful to integrate with training and have resulted in very positive feedback.

Whilst guidance on content and supportive tools from NES were seen as beneficial the fact that resources continued to be developed during the trial period required constant review of already agreed content in local areas. There was also some concern that the recent availability of the e-learning resources for level 2 might dissuade staff from attending face to face training and reduce collective learning and potential links for future collaboration as a result.

Staff Training, Targeting and Delivery Teams

The wider literature such as the CICI framework^[19] highlights how the effectiveness of complex interventions, as well as their success in reaching targeted populations, is influenced by various interrelated aspects of context. They identify three dimensions in their framework, each with multiple elements - context (e.g. geography and socio-economic conditions), implementation (e.g. strategies and delivery agents) and physical settings (e.g. organisations and networks). Many of these elements such as size, geography, workforce stability and previous local experience influenced decisions about training focus and delivery with the Delivery Trial Sites.

Site 1, due to rural geography and associated challenges in terms of recruitment and capacity, choose to focus on a particular sector (CYP) as an initial priority. This site planned to deliver level 1 training ‘in house’ which necessitated agreeing the appropriate mix of e- learning resources and face to face contact and deciding how best to ensure group training opportunities and interaction across a rural area even when using e-learning resources. Site 1 also chose to outsource the delivery of level 2 and level 3 training due to limited training capacity (See Table 3). The content for these sessions was jointly developed and the training was to be delivered by a third sector consortium (containing national third sector providers of residential care for young people) as detailed above. Content was agreed but delivery had not taken place prior to the outbreak of Coronavirus (COVID-19).

Site 2, where services and populations were less geographically spread intended to use in-house trainers although they may have encountered capacity challenges given existing workloads. Challenges for this site included population size, levels of socio-economic deprivation and resultant high incidence of issues such as substance abuse and poor mental health. Given similar funding was available for each site, Site 2 was unable to take a whole HSPC approach and so focused on a single locality within the partnership.

Site 2 and 3 had similar reach target numbers for level 1 training, but the latter had lower targets for level 2. Site 2 could not saturate the whole HSCP and chose to prioritise training for staff in services with clients who experienced high levels of trauma within one locality. The implications of this was that almost the totality of their targeted group required level 1 and also level 2 training.

Site 3 was geographically smaller and less deprived, so the intent was to saturate services across the whole of the partnership. Training was to be provided face to face with e-learning resources being integrated and or used for later cohorts/top-ups. Delivery challenges were fewer in Site 3 due to size and to the presence of a cohort of existing trainers who had gone through the NTTP train the trainers programme. Several of these trainers were from CYP’s Services -previously involved through ACES training - despite CYP not being a specific focus as part of the trial for this site. Even within this smaller locality ensuring consistent contacts with some participating agencies was still challenging.

Due to the Coronavirus (COVID-19) pandemic and the postponement of level 1, 2 and 3 training delivery there is no evidence to confirm whether local recruitment strategies would have been fully integrated with their intended programme theory. For example, it cannot be known whether training uptake would have achieved the intended thresholds and fully addressed the gaps and needs identified. Initial invites were open for self-referral in at least one site - so training places may perhaps have been taken up by the most motivated or usual early adopters rather than necessarily those intended or most in need of training. Whilst this may have been an intentional strategy to seek early adopters this was not made explicit. Similarly, it remains unknown whether chosen recruitment strategies would lead to sufficient numbers of staff trained within specific sectors, services or teams to create a ‘tipping point’ in these key contexts that would accelerate practice changes.

Various aspect of context have undoubtedly influenced decisions about focus and delivery methods within the sites and will likely influence the success and impact of both training and other actions to enhance trauma informed cultures. Tools such as the CICI framework ¹⁹ and learning from reviews^[20] conducted in Scotland concerning how factors that facilitate and hinder progress and effectiveness of similar QI interventions may be useful for NTTP roll-out given they highlight the many contextual issues that should be considered in both planning and evaluating complex intervention such as training and improvement programmes within HSCPs.

Staff Support, Supervision and Self-Care

The applications and plans for the Delivery Trial Sites included contributing to understanding the impact of trauma on staff, the risks of vicarious trauma and how to prevent and decrease its effects. As training (other than STILT) was not yet delivered in any of the sites changes to staff support and supervision and how these changes would further contribute to staff health were not yet well specified.

Plans across sites generally implied that staff support would follow existing supervision processes within services targeted at individuals more exposed to trauma (such as Social Work or Community Mental Health Services). Whilst the trial implementation teams acknowledged that more needed to be done to secure appropriate levels of supervision and prevention of vicarious trauma these areas were not prioritised in advance of training content and delivery. There had therefore been limited progress on these staff health and welfare issues or wider cultural change at the time of writing.

Needs Assessment and Monitoring and Evaluation

Needs Assessment

Varied approaches to assessing needs were used across the three sites and again these were influenced by the contextual factors discussed above ^14,15. Site 1 used learning from the roll-out of their ACES training and from inspection reports for care services and schools for their needs assessment. In addition, Site 1 conducted a bespoke staff survey informed by intended training outcomes agreed locally and prioritised from the national KSF¹.

Site 2 commissioned an external and wide-ranging needs assessment well in advance of their bid for inclusion as a Delivery Trial Site^[21]. This needs assessment highlighted key barriers to TIP that included service designs and environments, buy-in from leaders and managers to culture change, in addition to staff capabilities.

Site 3 used learning from their recent co-location of services for those highly likely to have experienced trauma (e.g. substance misuse, mental health and domestic violence services). The Project Lead also conducted further mapping of needs within this Hub to further their understanding of level 2 and 3 training demand.

As sites have not yet delivered their frontline training it is difficult to know which of these approaches most accurately predicted gaps and needs. Learning from needs assessments along with contextual issues seemed, in the main, to be used to identify the sectors or services to prioritise for initial training delivery. It was less evident that recruitment for training at the level of the individual staff member was influenced by the output from needs assessment processes. For example, recruitment of individuals for level 1 training in at least one site was by open invitation rather than targeted at individuals with expressed need. However, this area was seeking saturation across partnership services. In terms of training at levels 2 and 3, further mapping of skills within more specialist services was still being undertaken in at least two sites.

Figure 6, taken from the NTTP² illustrates the complexity and multiple steps that should be undertaken to comprehensively prioritise the focus and level of training for services, professional roles, teams and individuals. Identifying needs and gaps in training at service, team and individual level is a time consuming and iterative task. In the complex context of service reconfiguration and health and social care integration in Scotland the task should be closely aligned with and embedded in existing workforce planning and recruitment. All sites acknowledged the need for this alignment.

Monitoring and Evaluation

Each Delivery Trial Site had integrated aspects of monitoring and evaluation at the initial planning stages. In the main this involved intentions to monitor the reach/numbers of attendees completing training at all levels. All sites followed a Kirkpatrick Training Evaluation Model approach looking for changes at Kirkpatrick levels 1-3^[22] of the four-level model. Kirkpatrick’s model describes evaluating training outcomes at the following levels:

• Level 1 - Reaction (the degree to which participants found training favourable, engaging and relevant);
• Level 2 - Learning (the degree to which participants acquired the intended awareness, knowledge, skills, confidence
• Level 3 - Behaviour (the degree to which participants changed their own behaviour and practice)
• Level 4- Results (the degree to which participants influenced the wider service and organisations)

The sites intended to use the NES post-training evaluation questionnaires to assess these outcomes for the relevant levels of training. Site 1 agreed priority local learning outcomes that tied into the KSF and sector specific needs for CYP and were considering using a repeat of their initial needs assessment survey to gauge changes in outcomes post training. Detailed monitoring and evaluation plans in Site 2 were still being developed.

Site 3 added further tailored follow-up questions to the level 1 and 2 NES training evaluation sheets for use immediately after and at 3 and 6 months post training. Site 3 was also investigating using coded responses to allow following up individual change pre and post training rather than only reporting changes in awareness, knowledge, confidence or practice at a group level. There were however concerns expressed over ensuring confidentiality as part of this process. This site also identified more nuanced potential evaluation questions about levels of exposure to training within services and across the whole partnerships to achieve a tipping point in practice and culture changes. Plans to address these were not furthered at the point that training delivery was paused.

There was discussion in all sites about the value of investing in monitoring and evaluating at level 1 training given this was ‘light touch’ and involved only short (circa 2 hour) input aimed mainly at awareness raising. The likelihood of achieving practice change via level 1 training alone was questionable given this limited exposure. As substantial time and effort was being expended on level 1 training some feedback on changes in awareness and knowledge would inform whether this investment was of value.

Greater changes in knowledge, confidence and subsequent practice were more likely to result from level 2 and 3 training. Plans for monitoring such changes

were again based on developing existing NES evaluation sheets and tools. The Trial Delivery Sites faced similar challenges to other professional training focused programmes (including those promoting quality improvement in healthcare) where it is more difficult to gather evidence of impact at team, service, culture and system level^[23],[24]. Two sites were explicitly considering QI methods such as small test of change or case study approaches to address these issues within teams and services showing early evidence of practice or service adaptations.

A recent systematic review^[25] of TIP interventions involving training showed positive changes in staff attitudes, knowledge and behavior post training but could not confirm if these outcomes would have been achieved by training alone (rather than the wider organisational aspect of the interventions reviewed). The review also identified the need to:

• identify minimum durations for effective training courses
• provide more detail on content and implementation processes
• understand the extent to which tailoring of training for specific sectors or organisations is needed to ensure achievements of outcomes
• further test and validate existing outcome measures and scales and seek consistency of measures across interventions
• measure outcomes at least 1 month post training and report on survey completion rates (and characteristics of non-completers)
• understand what level of change in outcomes is important to lead on to further impact in people with lived experience and what mechanisms lead to such improvements
• conduct more sophisticated analysis of training outcomes (e.g. aggregate means, multivariate analysis and report rates in service use/uptake rather than counts)
• use more robust and sophisticated evaluation designs with (randomisation or) matched control/comparative designs.

Sites had limited resources and timeframes to embed sophisticated evaluation processes into local implementation. Many of the issues raised in the above systematic review should be consider by NES and the SG if commissioning a wider outcome evaluation of the NTTP programme as a whole.