What works to prevent youth violence: evidence summary

Annex B: Methodological discussion

In line with the report What Works to Prevent Violence Against Women and Girls: A Summary of the Evidence, this review identified existing evidence drawn from reviews and reports, such as those produced by the World Health Organisation (WHO). They were used as a starting point from which to explore evidence on what works to prevent youth violence.

A literature search was also conducted by the Scottish Government Library and covered a wide range of resources, including: IDOX, EBSCOHOST (Academic Search, SocIndex), PROQUEST (Applied Social Sciences Index and Abstracts (ASSIA), ERIC, PAIS International, International Bibliography of the Social Sciences (IBSS), ProQuest Sociology, Social Services Abstracts, Sociological Abstracts) and Web of Science.

While not completely comprehensive, this report aims to highlight the interventions with robust and reliable evaluations, using this evidence to classify their effectiveness (see details below). Drawing on and synthesising a range of sources, this report also looks at moderating factors; that is, potential barriers and facilitators to interventions working effectively.

In addition to reviewing key literature, extensive consultation took place with academics and key experts in the field. Relevant internal and external stakeholders contributed to quality assuring drafts of this report.

Reviewing and assessing available evidence

This report draws upon existing systematic evidence reviews, peer-reviewed academic publications, and a range of high-quality reports; including the most up-to-date evidence possible. In doing so, this report relies upon the classifications that the authors have assigned to their evidence. Where the strength of the evidence is explicit within such reports, the classifications of “weak”, “moderate” and “strong” evidence are used. The publications cited within this report include details of how these authors assessed the evidence presented.

Certain types of studies such as well conducted randomised control trials (RCTs) may be more likely to be classed as providing strong evidence. A definition for RCTs is provided in Box 1 below.

Box 1: RCT definition from National Institute for Health and Care Excellence (NICE) glossary online (no date)

“A study in which a number of similar people are randomly assigned to 2 (or more) groups to test a specific drug, treatment or other intervention. One group (the experimental group) has the intervention being tested, the other (the comparison or control group) has an alternative intervention, a dummy intervention (placebo) or no intervention at all. The groups are followed up to see how effective the experimental intervention was. Outcomes are measured at specific times and any difference in response between the groups is assessed statistically. This method is also used to reduce bias”.

This research approach is sometimes understood as the ‘gold standard’ of evaluations as they use a rigorous and reliable approach which helps researchers to draw conclusions regarding causal relationships (Cleaver et al., 2019).

RCTs are less commonly found within social science research and intervention evaluations. Instead, a range of quantitative and qualitative methods can be used to produce reliable, robust, and high-quality data on both specific outcomes (e.g. attitudes towards gender violence through quantitative methods) and understandings of the process (e.g. understanding the setting, how the programme was implemented through qualitative methods) (see Williams and Neville, 2017:27). Moreover, using qualitative methods within evaluations can allow researchers to consider the unquantifiable processes and factors that might impinge on the success of an intervention; particularly important when researching social behaviours (Cleaver et al., 2019).

An RCT design is not always practical or feasible given constraints of working in real-life situations. As such, quasi-experimental pre-and-post studies are often conducted where there are practical and ethical barriers to conducting randomized controlled trials. These investigations are non-randomised, which means that study participants are not randomly allocated to the experimental (intervention) and control (no interventions) conditions. Outcomes are assessed before and after a planned intervention. However, these estimated effects obtained from these studies are susceptible to a higher level of bias than their experimental counterparts. This means that it is more challenging to infer whether that intervention has caused any change in violence related outcomes and that the results of these studies should be interpreted with greater caution.

Assessment of effectiveness of interventions

Categories of evidence of effectiveness were developed, drawing on definitions/terminologies used by the National Institute for Health and Care Excellence (NICE) for reviewing research evidence and The Department for International Development’s (DFiD) Rapid Evidence Assessment (see Annex C). The inclusion criteria for evidence within this report on preventing and reducing youth violence included^[36]:

• High-quality peer-reviewed studies, evaluations, systematic reviews, and grey literature (including RCTs, cohort evaluations, qualitative studies^[37])
• Studies focusing on interventions intended to prevent violence (primary prevention) or further violence (secondary prevention)
• Studies focusing on the effectiveness of interventions in either preventing/reducing further youth violence
• Studies from high-income countries, published in the English language^[38]

Annex C and D include the decision-making tools (effectiveness classification criteria and decision tree) developed to illustrate the process undertaken in synthesising the available evidence. These tools have been used to ensure a consistent and transparent approach to classifying the effectiveness of interventions to prevent youth violence. In particular, the following aspects are considered in classifying the available evidence:

• The relevance of the evidence: must include outcomes related to violence prevention/reduction or risk factors or intermediate outcomes for violence
• What the evidence says about the effectiveness of the intervention
• The strength of the available evidence

The decision tree leads to the following six categories of effectiveness, which have been colour coded. Annex C provides definitions for each of these evidence classifications:

Effective (Green)

Promising (Amber)

Mixed (Amber)

No effect (Red)

Negative effect/potentially harmful (Red)

Inconclusive (Grey)

It should be noted that the inconclusive category is:

• distinct from the no effect^[39] category
• is based on insufficient evidence to make a judgement on impact of an intervention (e.g. only pilot evaluations available)
• indicates the need for further research and evidence before conclusions can be drawn on the effectiveness of an intervention

Where a respected expert organisation such as, for example, WHO had assigned a particular level of effectiveness to an intervention, this review has used their effectiveness rating, rather than following the decision-making process outlined in the decision tree. Exceptions to this include where robust new evidence has been produced since the publication of ratings by these organisations, or where an effectiveness rating is not relevant to a high-income country like Scotland (e.g. if that rating was only applicable to low income countries in a WHO report).

Caveats

There is the potential for interventions that fall “out of scope” for this review to positively impact on violence prevention in Scotland. Their omission from this report should not be seen as indicative of lack of effectiveness in violence prevention, rather as indicative of violence prevention not being their main aim or focus.

Likewise, there are limited robust evaluations which met the criteria for inclusion into this report. Again, this does not discount the effectiveness of the intervention. There may also be promising interventions that are not included within this report as they have not been evaluated or had evaluations published (Fulu and Kerr-Wilson, 2015).

As highlighted in What Works to Prevent Violence Against Women and Girls: A Summary of the Evidence, it can be hard to draw robust conclusions about what works, due to factors such as variable and low quality evaluations. Moreover, assessing the effectiveness of primary preventative interventions in relation to future violence is challenging.

Whilst an exhaustive discussion of the difficulties around assessing the effectiveness of violence prevention strategies is beyond the scope of this report, it is important to acknowledge these and the implications they may have for our understanding of what works to prevent youth violence. Within the Taking Stock of Violence in Scotland report (2019:61) it is noted that:

The complex and multi-faceted nature of violence makes it difficult to establish an evidence-base in terms of effective interventions. Not only are controlled evaluation designs and randomised trials near impossible in the context of violent street cultures, for example, in the few instances where such designs are implemented it is often difficult to attribute any ‘promising’ results solely to the intervention under evaluation.

When considering the impact of violence prevention interventions, it is important to keep in mind the wider structural, cultural and societal contexts in which violence occurs (Bellis, 2017; Equally Safe, 2016; WHO, 2016; WHO, 2019). The ecological model framework takes these contexts into account alongside the interactions between the individual level, personal relationships, community contexts and societal factors in influencing interpersonal violence, including youth violence (WHO, 2020).

Implementation issues

'Implementation fidelity' is the degree to which an intervention is delivered as intended. A good level of implementation is critical to the successful translation of evidence-based interventions into practice (Breitenstein et al., 2010). Programmes do not always transfer from one geographic or cultural setting to another and the structures for delivering prevention programmes might not always be in place (Breitenstein et al., 2010).

Diminished fidelity may be why interventions that show evidence of efficacy in highly controlled trials may not deliver evidence of effectiveness when implemented in real life contexts/routine practice. Likewise, transferring programmes to substantially different contexts may require adaptation and re-evaluation (Faggiano et. al, 2014). Williams and Neville’s (2017) evaluation of a Mentors in Violence Prevention (MVP) programme in Scotland highlights that caution should be taken regarding “implementation fidelity” to ensure that the US evidence base is utilised, while also ensuring that the programme is appropriately adapted for a Scottish context.

A substantial proportion of the evaluation research presented within this report has examined violence prevention efforts within North America. Although some programmes have been developed (or adapted) for use elsewhere, there is much less evidence relating to the evaluation of these approaches of these approaches in the UK and research within a Scottish context is particularly limited. As the demographics, socio-cultural dynamics, nature of violence, and systems of policing, criminal justice, and education within North America do not map easily onto the Scottish context^[40], the transferability of interventions or their effectiveness cannot be assumed. To illustrate the challenges of transferability in violence prevention initiatives, previous research demonstrates that there are differences between the effectiveness of the Olweus Bullying Prevention Programme in Norway (where it was originally designed and implemented) and the United States. These differences may reflect cultural and social differences between young people in Norway and those in the USA. Therefore, as interventions are dependent on the culture and context where they are implemented, further research evaluating the transferability to, and violence prevention impacts in, different settings is much needed.

Interventions that have been identified as out of scope have been outlined in Annex E.