Caledonian System Evaluation: Analysis of a programme for tackling domestic abuse in Scotland

Policy messages and key findings

This report presents findings from an evaluation of the Caledonian System, an integrated approach to addressing domestic abuse which combines a programme for male offenders with support services for women and children affected by domestic abuse as both victim and witnesses.

Policy messages

• The programme is clearly rated very highly by participants, staff and (female) partners interviewed for this evaluation. There is also evidence that women feel safer and that men who complete the programme pose a lower risk to partners, children and others by the end of the programme. Although the evaluation provides evidence of these positive perceived impacts, limitations of timescale and available data mean that it cannot conclusively demonstrate impact.
• While overall the Caledonian System is being delivered in line with its core principles and design, the evaluation identifies a number of areas for further reflection and/or improvement, including:
• reflecting on the role of the Children's Worker and how they work with families. Is there a need for (more) direct work with families or can children's needs be consistently met by referrals to other services? More widely, is there a gap in the services available to children affected by domestic abuse?
• considering whether more direct input to the System (in addition to that provided through existing referral networks) from psychologists or other specialists would enhance the support offered, particularly for men and women with needs relating to mental health or substance use issues
• updating the manual to improve usability (e.g. by simplifying the language), and to reflect changes in the world and in how domestic violence manifests in relationships
• enhancing training and ongoing learning opportunities for staff (including building in further discussion around information sharing and joint working)
• discussion of the staffing structures and physical locations that would best support effective delivery across current/future Caledonian Hubs.
• There is also a clear need to improve the data being collected to support monitoring and evaluation of the System. In particular, there is a need to consider:
• Whether psychometric tests should be retained as key components of the men's monitoring data. If they are, there may be a need for further training on how they should be interpreted and used in practice by staff delivering the men's programme
• Whether partner behaviour checklists should be retained as key outcome measures in the women's monitoring data
• Whether different outcome measures should be used with women, which would be equally relevant regardless of contact with (ex) partner
• Whether the structure of the monitoring data needs changing to (a) provide more timely data on participation rates and (b) to better reflect women's actual patterns of engagement with the System.
• What the explanations might be for the considerable data variations across hubs (e.g. in attrition rates) and how best to address these.

There is also a need to provide guidance and consider the resources required to facilitate more regular and consistent analysis and use of the monitoring data to inform policy and practice going forward.

• As noted above, timing constraints meant that this evaluation was reliant on monitoring data collected for all participants and qualitative interviews with a smaller number of participants, staff and stakeholders. To provide more conclusive evidence of impact, there is a need to consider whether it is feasible to source a sizable control or comparison group of families and conduct a longer-term (3-4 year), larger-scale prospective evaluation.

Key findings

The evaluation involved quantitative analysis of monitoring data (collected and provided to the evaluation team by the five regional Caledonian 'Hubs') and qualitative research with staff, men participating in the Caledonian men's programme, women supported by Caledonian System, and a small number of additional professional stakeholders. Wherever possible the report tries to triangulate evidence from different sources. However, it is important to keep in mind that participants' views were not always consistent with one another and that no one view of the System can be taken as definitive on its own.

Delivery of the Caledonian System

• Overall, the Caledonian System is being delivered in line with its core principles and design. There were, however, some variations in team structure and delivery across local areas.
• Delivery of the Men's Programme largely followed the structure provided by the manual. However, there were some examples of deviations from the manual reflecting either local resourcing issues or deliberate decisions by management and staff to vary content or delivery, primarily to try and better match it to men's perceived needs.
• Delivery of the Women's Service also largely appeared to reflect the aims and design of the System. However, there were again a few examples of local variations, particularly in relation to how Men's Workers worked with both Women's Workers and women themselves, and the service provided to new partners of men on the programme.
• Staff expressed a range of views on the impact of variations in team structure. While one view was that the ideal was for staff to be focused solely on the Caledonian System, another was that being able to draw on experience of delivering other related programmes brought a 'broader perspective'.
• The Children's Worker role was described as still developing. It was suggested that there is a need for greater consistency across areas in what Children's Workers offer - in particular whether Children's Workers should work directly with children or not.

Participation and reach

• Uptake of the Men's Programme and Women's Service is difficult to quantify precisely, because of limitations to the Caledonian System monitoring data. However, from the data available, at least 941 men had started the Men's Programme and 598 women had taken up the offer of support from the Women's Service.
• It is similarly difficult to quantify completion and attrition within the men's programme. There is considerable variation across Hubs in the levels of attrition and completion recorded in the monitoring data. It is unclear to what extent this reflects genuine differences in completion rates vs. differences in how the monitoring data has been completed. There is a need for further examination of this, to better understand the reasons for variation and what can be learned from this.
• Engagement with the women's service may fluctuate or tail-off over the course of the two years the service is offered for depending on: levels of control experienced in relationships; anxieties about the impact of participation on partner's cases; changes in women's own circumstances; and improvements in women's self-confidence. The level of engagement at the two-year point is therefore arguably a less relevant outcome measure for the women's service (since they are under no obligation to stay engaged for this length of time).
• 9 in 10 men were assessed at the start of the Men's Programme as posing a moderate or high risk of future domestic abuse to their partner, indicating that participants generally reflect the target group for Caledonian in terms of risk-levels.
• Men who successfully completed the programme had slightly lower levels of previous convictions and police call-outs for domestic abuse compared with those who did not complete it. This may suggest that more prolific offenders are more difficult to engage in behaviour change.
• A strong relationship with their Case Manager and men's own motivation to change were identified as the key factors influencing programme engagement.
• The vast majority (81%) of men had a problem with alcohol when they started the programme, while well over half (57%) had a problem with drugs. There was a perception that men with chaotic lifestyles, including alcohol and substance use problems, as well as those with mental health issues could be more difficult to keep engaged. It was suggested that having more direct input from professional psychologists built into the programme might be helpful to support work with these groups.

(Perceived) impacts on women and children

• The monitoring data provides only a partial picture of changes in the risk faced by women over time. However, there was a strong belief across women interviewed for the evaluation that the Women's Service, and the fact that it works together with the Men's Programme as a system, had both contributed significantly to making them feel safer.
• Key elements of the System that women identified as enhancing their safety were: the advice and support they received around safety planning; support and encouragement to contact the police about breaches of no-contact orders; and being better able to keep track of men's behaviour because of their involvement with the Men's Programme.
• Staff and stakeholders also highlighted the ability of the System to provide women with a broader perspective on their partners' behaviour which could help them make more informed choices about the future of their relationships. For women with particularly controlling partners, being able to coordinate appointments with Men's Workers also helped Women's Workers support women safely (by enabling them to see women while their partners were with their Case Managers).
• Even when it was thought that the man's behaviour had not changed, these aspects of the System were viewed as enhancing women's safety.
• Other perceived benefits (from the Women's Service in particular) included: improved self-confidence; improved physical health; reductions in substance use; reductions in women's own criminal behaviour; and positive impacts on income and work.
• Women identified a range of positive benefits for children, from increased safety, to changes in problem behaviour, to increased emotional and mental wellbeing. Where children had received support directly from Caledonian Children's or Women's Workers, this was viewed very positively by women. However, there was some evidence of a gap in services available to work directly with children around domestic abuse.
• Men and women reported the ways in which they felt the Caledonian System had made them better parents: by improving their confidence (women), and by increasing their understanding of the impact of partner abuse on children and equipping them with skills to better control their reactions to their children and (ex) partners (men)

(Perceived) impacts on men

• Although the monitoring data cannot be used to conclusively assess the impact of the Men's Programme on behaviour, the evidence indicates that those men who completed it posed a lower risk to partners, children and others by the end of the programme.
• Psychometric data on changes in men's attitudes presents a more mixed picture (and is more difficult to interpret, given wider debates about the use of psychometrics). There was some evidence that participants make progress in terms of general attitudes and feelings that may be predictors of abuse, and in reduced tendencies to blame their problems on either chance or other people. However, there was less clear evidence of any change in whether men feel they have control over their own lives. The psychometric data also indicates that men may display a greater tendency to exaggerate positives about themselves by the end of the programme.
• Men said the programme had equipped them with techniques to better control their behaviour and reactions and helped them learn to communicate more positively with their (ex) partners. The group sessions gave them the opportunity to practice new skills.
• Men also reported improved understanding of the nature of abuse and of appropriate behaviour in relationships; a greater awareness and understanding of the inequalities that exist between men and women; and a more 'positive mindset' about both their relationships and themselves.
• Other perceived impacts included: helping men to address substance misuse problems (an issue for a majority at Pre-group stage); improvements to health; and general improvements to confidence, particularly as a result of learning 'positive self-talk'.
• Women interviewed for the evaluation expressed more mixed views about whether the Caledonian programme had any impact on their (ex) partner. In some cases, they were unable to comment at all since they no longer had any contact with their ex-partner by the end of the Men's Programme.