Caledonian System Evaluation: Analysis of a programme for tackling domestic abuse in Scotland - research findings

Research Findings 63/2016: Crime and Justice: Caledonian System evaluation: Analysis of a programme for tackling domestic abuse in Scotland

This paper summarises the key findings and policy messages from an evaluation of the Caledonian System, an integrated approach to addressing domestic abuse which combines a programme for male offenders with support services for women and children affected by domestic abuse.

Key messages

• Interviews with male participants, staff and female partners showed that the programme is rated very highly.
• Women reported that they felt safer, attributing this to: safety planning; support to contact the police about breaches of no-contact orders; and being better able to keep track of men's behaviour because of their involvement with the Men's Programme.
• Men who complete the programme were judged by case workers as posing a lower risk to partners, children and others by the end of the programme.
• While overall the Caledonian System is being delivered in line with its core principles and design, the evaluation identifies a number of areas for further reflection and/or improvement, including:
  • variations in the role of the Children's Worker
  • whether or not the system would be improved by more direct input from psychologists
  • updates to the manuals to improve usability and to reflect changes since they were written
  • enhanced training and learning opportunities for staff, and
  • discussion around what staffing structures and locations can best support delivery.
• Although the evaluation provides evidence of positive perceived impacts, limitations of timescale and available data mean that it cannot conclusively demonstrate impact.
• In terms of future evaluation, there is a clear need to improve the quality, relevance and usability of the monitoring data being collected to support the System, and to consider the feasibility of a longer-term, prospective evaluation with a control group.

Background

The Caledonian System is an integrated approach to addressing the domestic abuse of women by men. It is a 'systems approach', working with the whole family by combining a court-ordered programme for men with support services for women and children. The Men's Programme comprises a structured two-year programme of one-to-one and group sessions, focused on changing negative attitudes and abusive behaviours. At the same time, the Women's Service offers flexible support to the partners or ex-partners of men on the programme, while the Children's Service aims to ensure that the needs and rights of children affected by abuse are being met within both Caledonian and wider services.

Evaluation aims

Ipsos MORI Scotland were commissioned by Scottish Government Justice Analytical Services in February 2016 to evaluate the Caledonian System. The evaluation aimed to:

• examine whether the System meets the Scottish Advisory Panel on Offender Rehabilitation's design standards
• assess to what extent, and how, the planned activities have taken place
• assess to what extent, and how, the short and medium (and, where possible, long) term outcomes have been realised, and
• propose a data collection framework for a future evaluation.

Methods

The evaluation involved quantitative analysis of monitoring data (collected and provided to the evaluation team by the five regional Caledonian 'Hubs') and qualitative interviews with 21 men participating in the Caledonian men's programme, 19 women supported by the Women's Service, 42 staff delivering the service (Men's, Women's and Children's Workers and Delivery Managers), and a small number (4) of additional professional stakeholders. Wherever possible the report tries to triangulate evidence from different sources. However, it is important to keep in mind that participants' views were not always consistent with one another, and that no one view of the System can be taken as definitive on its own.

Main findings

Delivery of the Caledonian System

Overall, the Caledonian System is being delivered in line with its core principles and design. There were, however, some variations in delivery and team structure across local areas. These included:

• Examples of deviations from the Men's Programme manual, reflecting either local resourcing issues or deliberate decisions by management and staff to vary content or delivery (primarily to try and better match it to men's perceived needs).
• Some variations in whether/how Men's Workers worked with both Women's Workers and women themselves, and the service provided to new partners of men on the programme.
• Differences in whether or not those delivering the programme were focused solely on the Caledonian System, or whether they combined this with other roles, such as general criminal justice social work case management, or delivering other offender intervention schemes.
• Differences in the tasks Children's Workers undertake, particularly whether or not they work directly with children and families themselves.

Participation and reach

Uptake of the Men's Programme and Women's Service is difficult to quantify precisely, because of limitations to the Caledonian System monitoring data. However, from the data available, at least 941 men had started the Men's Programme and 598 women had taken up the offer of support from the Women's Service.

There is considerable variation across Hubs in the levels of attrition and completion recorded in the men's monitoring data. It is unclear to what extent this reflects genuine differences in completion rates vs. differences in how the monitoring data has been completed. There is a need for further examination of this, to better understand the reasons for variation and what can be learned from this.

A strong relationship with their Case Manager and men's own motivation to change were identified as the key factors influencing programme engagement. The vast majority (81%) of men had a problem with alcohol when they started the programme, while well over half (57%) had a problem with drugs. There was a perception that men with chaotic lifestyles, including alcohol and substance use problems, as well as those with mental health issues could be more difficult to keep engaged. It was suggested that having more direct input from professional psychologists built into the programme might be helpful to support work with these groups.

(Perceived) impacts on women and children

The monitoring data provides only a partial picture of changes in the risk faced by women over time. However, there was a strong belief across women interviewed for the evaluation that the Women's Service, and the fact that it works together with the Men's Programme as a system, had both contributed significantly to making them feel safer. Key elements of the System that women identified as enhancing their safety were: the advice and support they received around safety planning; support and encouragement to contact the police about breaches of no-contact orders; and being better able to keep track of men's behaviour because of their involvement with the Men's Programme.

Staff and stakeholders also highlighted the ability of the System to provide women with a broader perspective on their partners' behaviour which could help them make more informed choices about the future of their relationships. For women with particularly controlling partners, being able to coordinate appointments with Men's Workers also helped Women's Workers support women safely (by enabling them to see women while their partners were with their Case Managers). Even when it was thought that the man's behaviour had not changed, these aspects of the System were viewed as enhancing women's safety.

Women identified a range of positive benefits for children, from increased safety, to improvements in problem behaviour, to increased emotional and mental wellbeing. Where children had received support directly from Caledonian Children's or Women's Workers, this was viewed very positively by women. However, there was some evidence of a gap in services available to work directly with children around domestic abuse.

(Perceived) impacts on men

Although the monitoring data cannot be used to conclusively assess the impact of the Men's Programme on behaviour, it does indicate that those men who completed it posed a lower risk to partners, children and others by the end of the programme.

Psychometric data on changes in men's attitudes presents a more mixed picture (and is more difficult to interpret, given wider debates about the use of psychometrics). There was some evidence that participants make progress in terms of general attitudes and feelings that may be predictors of abuse, and in reduced tendencies to blame their problems on either chance or other people. However, there was less clear evidence of any change in whether men feel they have control over their own lives. The psychometric data also indicates that men may display a greater tendency to exaggerate positives about themselves by the end of the programme.

Men interviewed for the evaluation reported positive impacts from the programme in terms of: equipping them with techniques to better control their behaviour and reactions; helping them learn to communicate more positively with their (ex) partners; improved understanding of the nature of abuse and of appropriate behaviour in relationships; a greater awareness and understanding of the inequalities that exist between men and women; and a more 'positive mindset' about both their relationships and themselves.

Women interviewed for the evaluation expressed more mixed views about whether the Caledonian programme had any impact on their (ex) partner. In some cases, they were unable to comment at all since they no longer had any contact with their ex-partner by the end of the Men's Programme.

Conclusions and recommendations

The data that was collected provides evidence that women feel safer and that men who complete the programme pose a lower risk to partners, children and others by the end of the programme. However, limitations of timescales and available data mean that this evaluation cannot conclusively quantify the impact of the Caledonian System.

The evaluation identifies a number of areas where more reflection on delivery and resourcing may be required - particularly in terms of the role of the Children's Worker, precise team structures, training and learning resources (including refreshing the manuals), and a potential need for additional psychological input to support delivery.

However, the main recommendations are around strengthening monitoring and evaluation going forwards. There is a pressing need to improve the content, structure and usability of the monitoring database. At the same time, more conclusive evidence of impact requires establishing the feasibility of a longer-term, prospective evaluation, ideally including a control or comparison group.

How to access background or source data

The data collected for this social research publication:

☐ cannot be made available by Scottish Government for further analysis as Scottish Government is not the data controller.