Preventing violence against women and girls - what works: evidence summary

Annex B: Methodological discussion

This report identified relevant existing evidence drawn from reviews and reports, such as those produced by the World Health Organisation (WHO). They were used as a starting point from which to explore evidence on what works to prevent VAWG.

A literature search was also conducted by the Scottish Government Library and covered a wide range of resources, including: IDOX, EBSCOHOST (Academic Search, SocIndex), PROQUEST (Applied Social Sciences Index and Abstracts (ASSIA), ERIC, PAIS International, International Bibliography of the Social Sciences (IBSS), ProQuest Sociology, Social Services Abstracts, Sociological Abstracts) and Web of Science. The majority of the literature was published within the last five years, although some sources are older.

While not completely comprehensive, this report aims to highlight the interventions with robust and reliable evaluations, using this evidence to classify their effectiveness (see details below). Drawing on and synthesising a range of sources, this report also looks at moderating factors; that is, potential barriers and facilitators to interventions working effectively.

In addition to reviewing key literature, extensive consultation took place with academics and key experts in the field. Relevant internal and external stakeholders contributed to quality assuring drafts of this report.

Prior to presenting interventions in detail, the approach to assessing evidence on interventions is outlined, implementation issues are highlighted, and interventions that are out of scoped are detailed.

Reviewing and assessing available evidence

This report draws upon existing systematic evidence reviews, peer-reviewed academic publications, and a range of high-quality reports; including the most up-to-date evidence possible. In doing so, this report relies upon the classifications that the authors have assigned to their evidence. Where the strength of the evidence is explicit within such reports, the classifications of “weak”, “moderate” and ”strong” evidence are used. The publications cited within this report include details of how these authors assessed the evidence presented.

Certain types of studies such as well conducted randomised control trials (RCTs) may be more likely to be classed as providing strong evidence. Often used in medical settings, a randomised control trial is defined in Box 2 below.

Box 2: RCT definition from National Institute for Health and Care Excellence (NICE) glossary online (no date)

“A study in which a number of similar people are randomly assigned to 2 (or more) groups to test a specific drug, treatment or other intervention. One group (the experimental group) has the intervention being tested, the other (the comparison or control group) has an alternative intervention, a dummy intervention (placebo) or no intervention at all. The groups are followed up to see how effective the experimental intervention was. Outcomes are measured at specific times and any difference in response between the groups is assessed statistically. This method is also used to reduce bias”.

This research approach is sometimes understood as the “gold standard” of evaluations as they use a rigorous and reliable approach which helps researchers to draw conclusions regarding causal relationships (Cleaver et al., 2019).

RCTs are less commonly found within social sciences research and intervention evaluations. Instead, a range of quantitative and qualitative methods can be used to produce reliable, robust, and high quality data on both specific outcomes (e.g. attitudes towards gender violence through quantitative methods) and understandings of the process (e.g. understanding the setting, how the programme was implemented through qualitative methods) (see Williams and Neville, 2017:27). Moreover, using qualitative methods within evaluations can allow researchers to consider the unquantifiable processes and factors that might impinge on the success of an intervention; particularly important when researching social behaviours (Cleaver et al., 2019).

Assessment of effectiveness of interventions

Categories of evidence of effectiveness were developed, drawing on definitions/terminologies used by the National Institute for Health and Care Excellence (NICE) for reviewing research evidence and The Department for International Development’s (DFiD) Rapid Evidence Assessment For Conflict Prevention (see Annex C). The inclusion criteria for evidence within this report on preventing and reducing VAWG included^[124]:

• High-quality peer-reviewed studies, evaluations, systematic reviews, and grey literature (including RCTs, cohort evaluations, qualitative studies^[125])
• Studies focusing on interventions intended to prevent violence (primary prevention) or further violence (secondary prevention)
• Studies focusing on the effectiveness of interventions in either preventing/reducing further VAWG
• Studies from high-income countries^[126], published in the English language^[127]

Annex C and D include the decision-making tools (effectiveness classification criteria and decision tree) developed to illustrate the process undertaken in synthesising the available evidence. These tools have been used to ensure a consistent and transparent approach to classifying the effectiveness of interventions to prevent VAWG. In particular, the following aspects are considered in classifying the available evidence:

• The relevance of the evidence: must include outcomes related to violence prevention/reduction or risk factors or intermediate outcomes for violence
• What the evidence says about the effectiveness of the intervention
• The strength of the available evidence

The decision tree leads to the following six categories of effectiveness, which have been colour-coded. Annex C provides definitions for each of these evidence classifications:

Effective (Green)

Promising (Amber)

Mixed (Amber)

No effect (Red)

Negative effect/potentially harmful (Red)

Inconclusive (Grey)^[128]

It should be noted that the inconclusive category is:

• distinct from the no effect^[129] category
• is based on insufficient evidence to make a judgement on impact of an intervention (e.g. only pilot evaluations available)
• indicates the need for further research and evidence before conclusions can be drawn on the effectiveness of an intervention

Where a respected expert organisation such as, for example, WHO had assigned a particular level of effectiveness to an intervention, this review has used their effectiveness rating, rather than following the decision making process outlined in the decision tree. Exceptions to this include where robust new evidence has been produced since the publication of ratings by these organisations, or where an effectiveness rating is not relevant to a high income country like Scotland e.g. if that rating was only applicable to low income countries in a WHO report.

Caveats

There is the potential for interventions that fall ‘out of scope’ for this review to positively impact on violence prevention in Scotland (see section below). Their omission from this report should not be seen as indicative of a lack of effectiveness in violence prevention, rather as indicative of violence prevention not being their main aim or focus.

Likewise, there are limited robust evaluations which met the criteria for inclusion into this report. Again, this does not discount the effectiveness of the intervention. There may also be promising interventions that are not included within this report as they have not been evaluated or had evaluations published (Fulu and Kerr-Wilson, 2015).

We know from available published evidence that it can be hard to draw robust conclusions about what works, due to factors such as variable and low quality evaluations. Moreover, as Scott (2015) notes within a Health Scotland report on intimate partner violence and abuse, assessing the effectiveness of preventative interventions in relation to future violence is difficult:

• The outcomes of studies are often limited to the impact of interventions on attitudes or educational change rather than any impact on behavioural outcomes. This is in part due to the challenges of assessing domestic abuse outcomes at a community level
• Most interventions focus on young people, with the aim of preventing violence or abuse before it occurs. However, the key time point for effective delivery of primary preventative interventions remains to be identified
• Interventions aimed at adults have tended to be media or awareness-based campaigns, but the evidence for these is inconsistent

Fulu and Kerr-Wilson (2015:9) also highlight that using short-term outcomes as measurements to determine the impact of an intervention upon the occurrence of VAWG: “may over-estimate effect because to sustain impact over the long-term many interventions require effective systems beyond the control of the intervention”.

Therefore, the wider structural, cultural and societal contexts in which VAWG occurs, must be kept in mind when considering violence prevention interventions (Equally Safe, 2016; WHO, 2019). The ecological model framework takes these contexts into account alongside the interactions between the individual level, personal relationships, community contexts and societal factors in influencing interpersonal violence, including VAWG (WHO, 2020).

Challenges in assessing attitudinal change and behavioural change in primary prevention interventions

Throughout this report there is also reference to how change in attitude does not necessarily equal change in behaviour. However, it is important to measure attitudes, and interventions which attempt to change attitudes. As noted in the Scottish Social Attitudes Survey (SSAS) VAWG module (2014);

The relationship between attitudes held by an individual and their behaviour is not always straightforward. However, attitudes held by many individuals, or by powerful individuals, potentially shape broader social norms, which in turn do influence behaviour.

Public attitudes can also provide a culture of support for violence by justifying or excusing it, trivialising or minimising the problem, or shifting responsibility for violent behaviour from perpetrator to victim-survivor. Importantly, attitudes can be seen as a ‘barometer’ of how societies, as well as particular groups, are faring in relation to violence against women.

As indicated here, focusing in part on attitudes can provide an indication of progress relating to addressing violence against women. In particular, interventions that seek to address gender based violence are often focused on primary prevention.

Moreover, as Vladutiu et al. (2011) note, changes in both attitude and intention by primary prevention programme participants are important outcomes. They recognise the limitations of these outcomes as they argue that researchers “will never have full confidence in our prevention programs until they are firmly linked to reductions in violence perpetration and victimization” (Vladutiu et al. 2011:81).

With this awareness, primary prevention interventions are often focused on changing personal and societal attitudes, often directed towards men and boys, attitudes that often inform the violent behaviours of VAWG (see Fulu et al., 2014:17).

Implementation issues

'Implementation fidelity' is the degree to which an intervention is delivered as intended. A good level of implementation is critical to the successful translation of evidence-based interventions into practice (Breitenstein et al., 2010). Programmes do not always transfer from one geographic or cultural setting to another and the structures for delivering prevention programmes might not always be in place (Breitenstein et al., 2010).

Diminished fidelity may be why interventions that show evidence of efficacy in highly controlled trials may not deliver evidence of effectiveness when implemented in real life contexts/routine practice. Likewise, transferring programmes to substantially different contexts may require adaptation and re-evaluation (Faggiano et. al, 2014). Williams and Neville’s (2017) evaluation of a Mentors in Violence Prevention (MVP) programme in Scotland highlights that caution should be taken regarding “implementation fidelity” to ensure that the US evidence base is utilised, while also ensuring that the programme is appropriately adapted for a Scottish context (see section on MVP).

For example, the authors note that “while MVP is specifically designed to address GBV, participants in the current study expressed a desire for the programme to additionally cover other forms of bullying” (Williams and Neville, 2017:25). However, implementation fidelity may not be achieved if broadening out, or degendering^[130], interventions that have been designed to focus on tackling VAWG such as GBV broadly or sexual violence specifically (see Anitha and Lewis, 2018 on prevention in university communities).

Williams and Neville (2017:29) also suggest that a “process of continual development/refinement” is required to “ensure age and cultural appropriateness”. By this, they refer to the process of adapting USA-based scenarios to situations that would be relevant and applicable to young people in Scotland (see section on MVP programmes)^[131]. Within their MVP evaluation Williams and Neville (2017:29) also highlighted the importance of both flexibility and adaptability, as well as ongoing evaluation of best practice in using this programme with young people in Scotland. Within a Scottish context, MVP programmes have been translated from international contexts and have identified actions to mitigate implementation challenges (see for example MVP Scotland Progress Report 2018-19, page 48).

Interventions that have been identified as out of scope have been outlined in Annex E.