Caledonian System Evaluation: Analysis of a programme for tackling domestic abuse in Scotland

1. Introduction and background

1.1 Addressing domestic abuse of women in Scotland

Domestic abuse is widespread and often hidden. The most recent findings from the Scottish Crime and Justice Survey ( SCJS) show that, since the age of 16, 14.1% of adults had experienced 'partner abuse', ^[1] with 2.9% experiencing it in the last 12 months (Murray/Scottish Government, 2016). ^[2]

Both men and women experience domestic abuse. However, women in Scotland are much more likely than men to report having experienced domestic abuse at some point during their adult lives (18.5% compared with 9.2% of men in the 2014/15 SCJS). Murray et al argue that this finding is consistent with other research which distinguishes between situational violence and 'coercive control'. Coercive control refers to the sustained use of a range of tactics (including physical violence but also financial, sexual and behavioural control) over a period of time to control and dominate the other partner. Analysis of the Crime Survey of England and Wales indicates that 'coercive control is highly gendered and is significantly more damaging to its primarily female victims than is situational violence' (Myhill, 2015).

The Scottish approach to tackling the domestic abuse of women by men is framed by the Scottish Government and COSLA's joint strategy for preventing and eradicating violence against women and girls, 'Equally Safe' (Scottish Government, 2016). The long-term aim of Equally Safe is 'a strong and flourishing Scotland, where all individuals are equally safe and respected, and where women and girls live free from such abuse'. However, the strategy also recognises that gender-based violence is a deep-rooted problem, requiring significant cultural and attitudinal change, and that women and girls will therefore 'continue to experience violence in all its forms today, tomorrow and for some time to come'. In this context, intervention services remain key. The Caledonian System, which is the focus of this report, reflects Equally Well's strategic focus on interventions which: maximise women's safety; hold men to account for their violence; are early; and address men's re-offending.

1.2 The Caledonian System

The Caledonian System is an integrated approach to addressing domestic abuse. It combines a court-ordered programme for men, aimed at changing their behaviour, with support services for women and children. The Caledonian System has its origins in various Domestic Violence Perpetrator Programmes ( DVPPs) developed in Scotland in the 1980s and 1990s, including CHANGE, based in Central Region, and the Lothian DVPP. The Caledonian System was developed from 2004, following a call from the Scottish Executive Effective Practice Unit to develop an accredited domestic violence intervention. The Caledonian Men's Programme was subsequently the first offender-rehabilitation programme to be accredited by the Scottish Accreditation Panel for Offender Programmes ( SAPOP ^[3] ) in August 2009. In operation (initially in Edinburgh) since 2011, the Caledonian System is currently operating in four Community Justice Authorities by five 'hubs' (Aberdeen; Dumfries and Galloway; Lothian, Edinburgh and Borders; Forth Valley; and North Ayrshire), covering 13 participating local authority areas.

1.2.1 Key principles underpinning the Caledonian System

The development of the Caledonian System was informed by research and best practice evidence on what works in preventing domestic violence. Among the key theoretical and practical guiding principles underpinning the system are:

• A 'systems approach' - the combination of services for men, women and children. Working together with the whole family is central to the Caledonian System's ultimate aim of reducing the risk of harm to women and children. As outlined in the Theory Manual, research 'makes clear that working with men in isolation is potentially dangerous in terms of raising the risk of harm to women partners', for example because men may resent having to attend and blame their partner for the fact they are on the programme. The systems approach also encompasses being embedded in a wider system of multi-agency working as a pre-requisite for successful intervention.
• Working towards 'good lives' - in working with men, the System does not focus only on their deficits, but also on their personal goals for a 'good life' and how they could achieve these as a means of motivating them towards positive change.
• An 'ecological model' of behaviour - 'ecological models' of behaviour, including abusive behaviour, argue that behaviour is influenced at various levels, from the individual, to the familial, to broader social and cultural contexts. This model influences how the programme works with men - for example, examining social stereotypes about gender roles as well as exploring the specific factors in individual men's lives that may have contributed to their propensity to abuse, such as their own exposure to violence and their use of alcohol and drugs.

1.2.2 Core elements of the Caledonian System

The Caledonian System is supported by a series of detailed manuals which set out the background to the programme, its core elements, and how each element should be delivered. In summary, the System comprises:

• A Men's Programme lasting at least two years and comprising a minimum of 14 one-to-one preparation and motivation sessions ( Pre-Group stage), a Group Work stage of at least 26 weekly three-hour sessions, and further post group one-to-one work ( Maintenance stage). The Men's Programme is highly structured, with each session detailed in the Men's Programme manual. It is delivered by Case Managers (who deliver the one-to-one sessions and manage individual men throughout their time on the programme) and Group Workers (who deliver the Group Work stage). Participation in the Men's Programme is a mandatory requirement of a statutory order or licence: they are referred by court order if they have been convicted of offences involving domestic abuse and are assessed as suitable candidates in terms of risk and readiness to change.
• A Women's Service , which provides safety planning, information, advice and emotional support to women partners and ex-partners of men referred to the Men's Programme. It is provided by dedicated Women's Workers, who aim both to reduce the risk to women and their children, and to improve women's social and emotional wellbeing. In contrast with the Men's Programme, the Women's Service is voluntary - women are not obliged to accept the support they are offered.
• A Children's Service , which aims to ensure that the needs of children (whose father or whose mother's (ex) partner is on the Men's Programme) are met and their rights upheld. It is supported by Caledonian Children's Workers, who do not necessarily work with children directly but rather ensure their rights and needs are being considered both within the Caledonian System and by wider services.

1.3 Evaluation aims

Ipsos MORI Scotland were commissioned by Scottish Government Justice Analytical Services in February 2016 to evaluate the Caledonian System. The evaluation is intended, as far as possible, to provide learning about both processes and outcomes from delivery of the System to date in order to feed into application for reaccreditation by SAPOR. It has four main aims:

1. To examine whether the System meets SAPOR's design standards (specifically Standard 7, which stipulates a process and outcome evaluation)

2. To assess to what extent, and how, the planned activities have taken place

3. To assess to what extent, and how, the short and medium (and, where possible, long) term outcomes have been realised (including which aspects of the system work, for whom, and under what circumstances), and

4. To propose a data collection framework for a future evaluation.

The evaluation was not designed to assess differences in success or impact between Caledonian hubs, although the report does include some summary information about the overall numbers of Caledonian System participants across different local Hubs and some discussion of variations in local practice.

1.4 Evaluation design

The evaluation adopted a mixed method approach, combining analysis of quantitative monitoring data supplied by each of the participating Caledonian System hubs, with qualitative interviews with:

• Participants in the Men's Programme (21 in total)
• Users of the Women's Service (19 in total)
• Staff involved in delivering the Caledonian System (Men's, Women's and Children's Workers and Delivery Managers - including 25 Men's Workers, 11 Women's Workers, 3 Children's Workers - one of whom was also a Women's Worker - and 3 Delivery Managers)
• 4 stakeholders from other services, including Children and Families Social Work and Scottish Women's Aid.

Analysis of the monitoring data focused on examining the level of participation and attrition, as well as changes in key outcome measures. Qualitative interviews focused on exploring perceptions of the delivery and impact of the Caledonian System and on identifying areas for improvement.

Participants in the Men's Programme and Women's Service and stakeholders from other services were all interviewed individually, while staff were interviewed in small groups. ^[4] All interviews were conducted by members of the Ipsos MORI research team, using flexible topic guides to ensure key topics were covered while enabling the researchers to follow-up on particular issues as they arose (see Annex A for copies of topic guides). With participants' permission, interviews were audio-recorded, transcribed and summarised for further analysis using a thematic framework based around key research questions.

1.5 Challenges and limitations

Evaluations are always subject to ethical, methodological and practical limitations, but evaluations of DVPPs raise more challenges than most. This section outlines the challenges associated with this evaluation and the limitations this places on some of the conclusions that can be drawn. In reporting the evaluation findings, the report attempts to strike a balance between drawing out evidence that might point to programme success or failure, while at the same time reminding the reader of the various limitations that apply to the conclusions that can be drawn from the data.

1.5.1 Lack of a control group

The most reliable way of establishing the impact of an intervention is to compare outcomes for participants with outcomes for similar individuals who did not go through the intervention (a 'control' group). However, given that finding a sizable matched control group of families would have been challenging given the programme design and impossible given the evaluation timeframe (the evaluation took place from mid-February to mid-June 2016), we were unable to compare the findings with a comparison group of families affected by domestic abuse but not involved in the Caledonian System in this evaluation. As such, any quantitative changes observed in this report cannot be conclusively attributed to the Caledonian System.

1.5.2 Determining what counts as evidence of 'success'

Deciding on appropriate outcome measures for DVPPs is far from straightforward. SAPOR are concerned primarily with offender rehabilitation and 'desistance' (stopping offending). However, for DVPPs in particular police call-outs and reconviction rates are not necessarily reliable indicators of this, since a consequence of supporting women to recognise the signs of abuse can be to increase police call outs and/or charges. At the same time, focusing solely on whether or not men's behaviour has changed as a result of the Caledonian System runs the risk of missing out on important impacts for women and children, that may occur whether or not their (ex) partner's behaviour has improved. In this report, we use the monitoring data to explore whether there is at least tentative evidence that the Caledonian Programme is associated with moves in the right direction in terms of the risk profiles of men, while the accounts of men, women and staff are used to explore perceptions of a range of potential impacts of the programme for men, women and children, including risk behaviours, safety, and wider wellbeing.

1.5.3 Gaps in the monitoring data

Data entered into the Caledonian monitoring database from late 2010 to mid-April 2016 was provided to the research team. This data includes a wide range of measures that could be used to look at outcomes (albeit with the caveat above that without a control group it is not possible to definitively attribute these to the Caledonian System). However, there were some substantial gaps in both the overall number of cases included in the monitoring data and the number of records for specific outcome fields. This limits the conclusions that can be drawn from the data about participation, attrition and whether specific outcomes were apparent. Issues relating to the monitoring data are discussed in more detail in relevant sections of the report and in Appendix B.

1.5.4 Barriers to triangulating accounts

In their review of published studies of DVPPs, Kelly and Westmarland (2015) note that the 'first generation' of studies of DVPPs in the US, Australia and the UK which focused on men's behaviour change as measured by convictions and/or men's self-reports came in for heavy criticism, with later studies that also interviewed women finding significant disparities in men and women's assessment of change. However, studies which have tried to address this through tracking women's perceptions of change have experienced considerable challenges around recruiting and retaining women in the research (see both Kelly and Westmarland, 2015 and Eckhardt et al's 2013 evidence review). Moreover, as Kelly and Westmarland highlight, women who are no longer in a relationship with the man who committed the abuse may not be in a position to assess change on some dimensions.

In this evaluation, while both men and women were interviewed there was no attempt to 'match' these interviews - it was not considered appropriate to interview men and their partners/ex-partners as this could increase risk for the women concerned. As such, it is not possible to directly triangulate the individual accounts of men with those of the women interviewed for this evaluation. Moreover, of the 19 women interviewed, 11 had no contact at all with their ex-partner, two had only occasional contact, while one was a new partner, who had not been with the man at the time of the offence that led to him being on the Caledonian System. As such, their ability to comment on the impact of the programme on men (rather than on them or their children) was inevitably limited. The same issue also means that parts of the monitoring data that might in theory be used to examine women's perspectives on men's behaviour change are in fact too incomplete to be provide a reliable measure of change over time.

A further limitation is the absence of children's views in the evaluation. Given that the Caledonian System is intended to benefit children as well as their parents, ideally the evaluation would have incorporated data on their views and experiences. However, the design of the System (whereby only a minority of children actually work with Caledonian Workers directly) and the timetable for the evaluation presented barriers to including children appropriately and ethically. Some children may be relatively unaware that their parents are involved in the Caledonian System, or of indirect support it has provided to them - framing interviews appropriately would have been challenging. At the same time, identifying and recruiting children to the evaluation would have required a longer lead in time than was available, given the need to go through gatekeepers and to ensure additional ethical scrutiny of recruitment and interview protocols. Wherever possible this report tries to triangulate evidence from different sources. However, it is important to keep in mind that participants' views were not always consistent with one another and that no one view of the System can be taken as definitive on its own.

1.5.5 Barriers to 'engaging the disengaged'.

It was not possible within the evaluation timescale to adopt a prospective approach to the research (whereby participants are recruited at the start and interviewed repeatedly as they progress through - or leave - the programme). As a result, it was decided to focus on men and women who were (or whose partners were) nearing the end of the Group or Maintenance stages of the programme so that they could comment on the impact of the programme as a whole. This inevitably meant that those we interviewed tended to be those who were more engaged with the programme. Moreover, as the researchers had no means of directly contacting participants, the evaluation was reliant on Caledonian System staff to recruit men and women for interviews. While the research team worked with staff to attempt to minimise recruitment bias, ^[5] accessing a sample of men who had left the Men's Programme or women who had little or no contact with the Women's Service was not feasible. As such, while interviews with staff, women and men identified a wide range of views and experiences, it is possible that there are further variations that are not fully captured - particularly around the experiences of men who do not complete the Caledonian programme, and of women who decline the offer of support from the Women's Service.

1.5.6 Limitations on the number of areas visited and interviews conducted.

Although the evaluation interviewed participants and staff in all five of the 'Hubs' in which the Caledonian System is being delivered, it was not possible within the evaluation timescale or resources to recruit people from every team or every local authority area within these Hubs. There may, therefore, be further variations in local Caledonian practice and experience that are not captured in this report.

In addition, the evaluation team were only able to speak to a limited number of stakeholders within the resources and time available. Stakeholders in other geographic areas, or from other sectors, may have had different views on the Caledonian System. In particular, we had hoped to include Sheriffs and Police Scotland representatives in the evaluation but attempts to recruit them for interview proved unsuccessful. ^[6]

1.6 Report structure and conventions

The remainder of this report is structured as follows:

• Chapter 2 describes the operation of the Caledonian System in Scotland. It reviews variations in how it is being delivered in different areas and explores perceived facilitators and barriers to effective partnership working with other organisations.
• Chapter 3 examines participation in the Caledonian System. It examines uptake, engagement and attrition in the Men's Programme and uptake of and engagement with the Women's Service.
• Chapter 4 examines the perceived impact of the Caledonian System on women and children, including impacts on safety, parenting, and wider wellbeing.
• Chapter 5 examines the impact of the System on men's behaviour, attitudes, and needs.
• Chapter 6 summarises recommendations for improvement of the System in general and discusses how to strengthen monitoring and evaluation of the System specifically.

1.6.1 Report conventions

As discussed above, Caledonian participants, staff and stakeholders were interviewed for this evaluation using a qualitative approach. Qualitative samples are generally small, and are designed to ensure a range of different views and experiences are captured. It is not appropriate given the number of interviews conducted to draw conclusions from qualitative data about the prevalence of particular views or experiences. As such, quantifying language, such as 'all', 'most' or 'a few' is avoided as far as possible when discussing qualitative findings.

In order to protect anonymity, participants in the Men's Programme and Women's Service are identified using anonymous reference numbers or letters only, while quotes from staff are not attributed to specific areas (given the small numbers of staff employed in each area, a job title in combination with an area reference could easily be identifying).