Caledonian System Evaluation: Analysis of a programme for tackling domestic abuse in Scotland

3. Participation in the Caledonian System

Key findings

• Uptake of the Men's Programme and Women's Service is difficult to quantify precisely, because of limitations to the monitoring data. However, from the data available, at least 941 men had started the Men's Programme; and 598 women had taken up the offer of support from the Women's Service.
• There is a need for further examination of patterns of engagement and attrition across Hubs, and to better understand the reasons for any variations (and what can be learned from this).
• 9 in 10 men were assessed at the start of the Men's Programme as posing a moderate or high risk of future domestic abuse to their partner, indicating that participants generally reflect the target group for Caledonian in terms of risk-levels.
• Men who successfully completed the programme had slightly lower levels of previous convictions and police call-outs for domestic abuse compared with those who did not complete it. This may suggest that more prolific offenders are more difficult to engage in behaviour change.
• A strong relationship with their Case Manager and men's own motivation to change were identified as the key factors influencing programme engagement.
• There was a perception that men with chaotic lifestyles, substance use problems and mental health issues could be more difficult to keep engaged, and that further professional psychological input might be helpful to support work with these groups.
• Engagement with the Women's Service is voluntary and may fluctuate or tail-off over the course of a man's two-year order depending on: levels of control experienced in relationships; anxieties about the impact of participation on partner's cases; changes in women's own circumstances; and improvements in women's self-confidence.

3.1 Uptake, engagement and attrition in the Men's Programme

3.1.1 Referral and assessment process

Men are referred to the Caledonian Men's Programme following receipt of a court order, either as part of a Community Payback Order or Probation Order or as a requirement of post release supervision. On receipt of a court referral, the manager from the relevant service allocates an assessor (typically one of the case managers working with men on the programme) to undertake the man's assessment. The evaluation and assessment manual states that assessment of suitability should be based on a combination of information sources including:

• assessment interviews with the men
• joint interviews conducted by the Women's Service worker and the programme assessor with the partner / ex-partner, including the administration of the behaviour checklist
• the first administration of the Spousal Assault Risk Assessment ( SARA ^[13] ), which assesses men's level of imminent risk of violence to partners, children, and others. Men are considered to be suitable for the Caledonian Men's Programme if their level of risk of future domestic violence is assessed as 'moderate' or 'high'.
• information from the general needs/risk assessment conducted for the purpose of preparing the Court Report
• information from other relevant agencies such as the Police, child protection agencies and GPs
• Scottish Criminal Record Office records
• records already held by the social work department and those made available to them from other sources (e.g. NHS).

Across these information sources, a range of factors are considered when making the assessment, including housing stability, health, substance use, family history, domestic abuse history and motivation to change, all of which may impact on whether or not men are assessed as 'ready' to engage with the programme.

Although consistency of assessment was not explored in detail in evaluation interviews, feedback from Caledonian staff generally indicates that the process is being delivered in line with the guidance. One reported exception related to courts allowing insufficient time for staff to meet with the woman before writing the assessment report. However, there was a general perception that the assessment process was as thorough as it could be, and that triangulating reports from different professionals as well as accounts from both women and men meant that the right men were usually identified. This view was confirmed by stakeholder interviewees, who felt the level of time and detail associated with Caledonian assessments, in combination with the experience of those carrying out the assessments, meant that they were identifying the right people.

Staff acknowledged that men who were not ready or willing to be held accountable can manipulate Caledonian assessments to avoid a custodial sentence as ' it's easy to be someone else for a short time', but felt that this was the exception rather than the rule, especially when the staff conducting assessments were experienced in doing so. It was suggested that assessment of suitability was, in practice, ongoing and that if staff realised that a man was not genuinely suitable for the programme, they could return their case to court. However, there was a suggestion that it would be helpful to have a review point after a few Pre-Group sessions before finally deciding on suitability.

3.1.3 Participation in and attrition from the Men's Programme

As indicated in Chapter 1, gaps in the Caledonian Monitoring data make it an imperfect guide to programme participation. The reasons for this are discussed in more detail in Appendix B, but in summary, discussions with Delivery Managers and Data Champions (responsible for entering data for each Hub) indicate that the monitoring data may underestimate the number of men who have participated in the Caledonian System by between 10% and 30%. However, while acknowledging this limitation, the monitoring data is the only available source of information that provides an indication of either the profile of participation or patterns of attrition in the Men's Programme.

The data provided to the evaluation team includes records for 941 men who have started the Caledonian Men's programme since mid-September 2010 (see Table C.1 in Appendix C for breakdown by hub).

This figure includes cases that are still open (that is, men who may not yet have reached the Group or Maintenance stages of the programme). Of the 941 men included in the database as of April 2016, an estimated 583 should have completed the programme (based on their date of order). However, this can only be an estimate - in some cases orders may be extended beyond two years.

If only those men who are recorded as either having 'completed successfully', having 'breached' or had their order 'revoked', or who are recorded as 'closed' on the system but not as having completed successfully are included, then overall 37% are recorded as having completed the full programme successfully. There was, however, considerable variation in recorded completion rates between areas - from 18% to 67% (see Appendix C, Table C.2). Analysis by stage suggests that there is relatively low attrition at Pre-group stage, and that the programme experiences greatest attrition at Group stage.

Discussions with staff across hubs have raised concerns about the reliability of these figures - several hubs have indicated that their own separate data shows successful completion rates considerably higher than those indicated in Table C.2. ^[14] Both the overall completion rate, and variations in recorded levels of completion of the Men's Programme by area, therefore need to be treated with considerable caution - given known issues with the monitoring data, it is unclear whether or not these differences are genuine, or if they are simply a reflection of differences in how the monitoring data has been completed across Hubs. This issue clearly needs further exploration and discussion - establishing robust completion rates by area should be a priority going forward (see Chapter 6 for further discussion).

Given the Caledonian Men's Programme is a court ordered programme, where a participant does not complete the programme this almost always reflects a breach in order. However, the Caledonian database does not include sufficient information to assess the reasons for breach. ^[15]

In interpreting attrition completion rates from the Men's programme, it may be helpful to consider comparisons with other DVPPs in the UK and internationally. These show a wide variation in attrition rates. In part, this appears to reflect differences between court-mandated and community-based (voluntary) programmes. For example, in his systematic review Losel (2011) reports that while 68% of men completed a court-ordered programme in the West Midlands, only 23% completed a community-based programme in Devon (with a further 14% still on the programme at the time of the post-intervention evaluation assessment). However, drop-out rates tend to be fairly high across the board - attrition rates for the 11 programmes included in Losel's systematic review were rarely below 30%. Similarly, a recent study of two DVPP programmes in the US found that only 51% and 41% respectively of men referred to the programmes completed the programmes (Mills et al, 2013). A review of DVPP probation and prison programmes in England and Wales found that staff delivering the probationary programmes reported high levels of attrition, particularly in the pre-group stages, where a lack of real motivation to change was apparent (Bullock, et al, 2010).

Profile of participants in the Caledonian Men's Programme

The Caledonian Monitoring Data includes various fields capturing different aspects of participants' characteristics. Two-thirds of men on the programme (67%) were aged between 25 and 44 and almost all (98%) were white (See Appendix C, Tables C3 and C4). The monitoring data does not allow the identification of sub-groups within the 'White' ethnic category who may have specific needs - for example, in Edinburgh the Respekt programme delivers the Caledonian programme to Polish speaking men. There were no significant differences in age between those who completed the programme and those who did not, and no clear difference in ethnicity. ^[16]

Men may be referred to Caledonian after more than one conviction. However, the database includes a 'main index offence' for each case. The most common index offences were 'assault to injury' (36%), an offence under the Criminal Justice and Licensing (Scotland) Act 2010 ^[17] (28%) and simple 'assault' (17% - Table 3.1 ^[18] ). Those who completed the programme successfully were slightly more likely than those who did not complete to have 'assault' or 'assault to injury' recorded as their main index offence (65% compared with 49%), indicating that those convicted of violent offences are certainly no less likely to engage with the programme once they accept it. Non-completers were more likely to have (mainly unspecified) 'other' offences recorded.

Table 3.1: Type of offence for all men starting Men's Programme

Type of offence	% of total	% of successful completers	% of non-completers
Assault to Injury	36%	43%	34%
Criminal Justice and Licensing (Scotland) Act 2010	28%	20%	26%
Assault	17%	22%	15%
Breach of the peace	3%	3%	2%
Communications Act 2003 (Sec 127)	3%	4%	4%
Breach of Bail	2%	1%	3%
Criminal Law Consolidation (Scotland) Act 1995	1%	1%	1%
Assault to severe injury	1%	1%	1%
Other	8%	5%	12%
Base (all starting Men's Programme)	941	187	355 1

Notes to table:
¹ This figure is slightly higher than that shown in Table C.2, as it is based on all those who had not completed, including some who breached their order within the last two years.

The record of prior police involvement with men on the Caledonian programme varies quite widely. A quarter of men (25%) had no previous convictions for domestic abuse offences before starting on the programme, a similar proportion (27%) had one, a third (33%) had two or three previous convictions, and 15% had four or more previous convictions (Table 3.2). The number of police call-outs for domestic abuse on record for participants tended to be higher than the number of actual convictions (unsurprisingly) - while around a quarter 22% had no previous police call-outs on record prior to their most recent offence, 10% had one previous call-out, 20% had two or three, and 47% had four or more previous call-outs on record.

The mean number of previous convictions was 1.9 among all men that started the programme; among those that completed it was lower at 1.6, while among non-completers it was 2.1. A similar pattern was apparent for previous police call-outs for domestic abuse. The mean number of previous police call-outs was 5.7; this figure was slightly lower (5.4) among those that completed the programme, but increased to 6.6 among those that dropped out of the programme.

Given the 'hidden' nature of domestic abuse, which does not always result in police involvement, there is no easy read across from levels of prior conviction or police involvement to severity of previous offending behaviour. However, keeping this qualification in mind, the differences in mean previous convictions and call-outs between men who completed Caledonian successfully and those that did not may indicate that those who have already been more prolific offenders at the time they are referred to Caledonian are more difficult to engage in behaviour change.

Table 3.2: Number of previous convictions and police call-outs for domestic abuse, all men starting Men's Programme

Number of previous convictions for domestic abuse	% of total	Number of previous police call-outs for domestic abuse	% of total
0	25%	0	22%
1	27%	1	10%
2-3	33%	2-3	20%
4-5	11%	4-5	14%
6-10	3%	6-10	19%
Over 10	1%	Over 10	14%
Mean no.	1.9	Mean no.	5.7
Base (all starting Men's Programme)			941

As discussed above, the Men's Programme is targeted at those who are assessed as 'moderate' or 'high' risk of future domestic abuse. Three categories of SARA risk scores are recorded in the monitoring data: risk to partner, risk to child and risk to others. As shown in table 3.3, the majority (90%) of men starting the programme were considered at assessment stage to have a 'moderate' or 'high' risk of future domestic violence against their partner. However, 10% (91 men) were considered to pose a 'low' risk to their partner. While it could, of course, be the case that this group were rated 'moderate' or 'high' risk to children or others, in fact further analysis identifies 72 men in the monitoring data who appeared to be categorised as 'low risk' at assessment on all three of these measures. So while the vast majority of men who receive Caledonian orders appear to fall into the target group for the programme, there also appear to have been a small number of lower risk men involved. The reasons for this are not clear from the monitoring data or from discussions with staff conducted for this evaluation, but may warrant further investigation. There does not appear to be any significant difference in the risk profile of those who completed the programme successfully and those who did not complete.

Table 3.3: SARA scores for men at assessment stage

	SARA 1 - Partner	SARA 2 - Children	SARA 3 - Others
High	30%	5%	4%
Moderate	60%	31%	27%
Low	10%	60%	69%
N/A	0	4%	0
Base: all with scores available	940	940	937

3.1.3 Engagement with the Men's Programme

In addition to providing data on programme participation and attrition, the monitoring data also includes measures of men's 'responsiveness' to the programme, with appropriateness of behaviour in sessions, participation, and 'therapeutic alliance' each rated by case managers from 0 ('poor') to 3 ('very good') at the end of the Pre-group and Group stages.

Analysis of this data suggests that, while men who remain on the programme tend to display fairly consistent and high (satisfactory or very good) levels of programme responsiveness on all three measures across the Pre-group and Group stages, those who leave before the end of Group stage are more likely to have poor scores on each dimension at Pre-group stage (Table 3.4). Programme responsiveness at Pre-group stage thus appears to be a good predictor of whether or not men will continue to engage with the programme.

Table 3.4: Programme responsiveness for men at Pre-group and Group stages

	Appropriateness of behaviour in sessions			Participation			Therapeutic alliance
Score	Pre-Group scores (those who left before end of Group)	Pre-Group scores (those who completed Group)	Group scores	Pre-Group scores (those who left before end of Group)	Pre-Group scores (those who completed Group)	Group scores	Pre-Group scores (those who left before end of Group)	Pre-Group scores (those who completed Group)	Group scores
Poor (0)	23%	3%	4%	24%	3%	4%	24%	4%	5%
Marginal (1)	13%	9%	11%	14%	11%	12%	19%	15%	16%
Satisfactory (2)	40%	50%	51%	42%	48%	45%	36%	48%	47%
Very good (3)	24%	38%	34%	20%	39%	38%	21%	33%	32%
Base	273	308	308	273	308	308	273	308	308

As discussed in Chapter 1, the men interviewed for this evaluation tended to be highly engaged with Caledonian, and reported rarely missing any meetings or group sessions. However, their accounts did suggest more variation in levels of enthusiasm for the programme over time - for example, in some cases they had turned down the programme when it was initially offered, or had attended meetings only reluctantly to begin with. While the main reason for accepting the programme after initially turning it down appeared to be avoiding prison (after Sheriffs made clear this was the alternative), developing a strong relationship with their Case Manager and starting to see benefits from the programme were key factors in overcoming initial reluctance or anxiety about participation.

She built up a relationship with me. It's a bit like the doctor in the medical centre, if you're getting a different one every single time you go, he doesn't get to know you. Whereas if you're seeing the same one every time you can speak to them, you can let things out that you maybe wouldn't have let out and things like that.

(Men's Programme participant B)

First and foremost, I thought it was just to punish me, just to, "don't do this, don't do that". But I see that different now. (…) I wouldn't say I was enjoying it but I know what it's about now. It's, obviously it's to try and better me as a person, which I'm quite happy with.

(Men's Programme participant S)

Staff, stakeholder and participant interviews also highlighted the importance of men's own motivation to change in underpinning engagement with the programme.

It's a fabulous programme, but it can only really work when men acknowledge that they have a problem and want to change. Much of it is down to them.

(Stakeholder, Women's Aid)

Staff from one area discussed a period when they felt they had been pressured to accept men who were not as motivated to change onto the programme, which they believed had led to a lot of breached orders. These comments reinforce the importance of taking account of readiness to change in assessing suitability for Caledonian.

While one view among Men's Workers was that there was no particular pattern to the kinds of men who did and did not remain engaged with Caledonian, another was that those with chaotic lifestyles (particularly those with alcohol or drug problems) and those with mental health issues and personality disorders could be particularly difficult to keep engaged. While the assessment process is intended to screen out men with issues that would completely prevent them with engaging with Caledonian, in practice substance use and mental health issues which appear manageable at assessment may fluctuate over the two years of the programme. ^[19] While Caledonian staff work closely with other services, including mental health and substance use services, given the prevalence of these kinds of issues among their client group, it was suggested by staff that further support, such as more direct input to the System from professional psychologists might help them to manage their approach to men with more complex needs in these areas.

3.2 Uptake and engagement with the Women's Service

3.2.1 Referral and assessment process

According to the manual, Women's Service workers should be allocated to present and/or ex-partners at the point at which a man is given a court referral for assessment for the programme. Initial contact is made by the Women's Worker, by telephone or by letter, offering the woman an appointment for a home visit from both the Women's and Men's Worker. At this appointment, the woman is offered the support of a Woman's Worker for the duration of her partner or ex-partners time on the Men's Programme. The Women's Worker can also use this opportunity to engage in urgent safety planning with the woman if necessary. The accounts of women and Women's Workers interviewed for this evaluation indicate that this process generally appears to be followed in practice across Hubs.

3.2.2 Uptake of the Women's Service

Monitoring data for the Women's Service includes only those women whose partner was actually given an order to attend the Men's Programme - women whose partners were not given orders may be offered time limited support (up to four sessions according to the manual, although interviews with Women's Workers suggested it could sometimes be more in practice), but this is not recorded in the monitoring data.

The monitoring data up to April 2016 includes 1,116 women who were offered the Caledonian Women's Service (Table 3.5). Of these, the Women's Service had been able to establish direct contact with 78% (n = 865). Among women with whom contact was established, 69% accepted some support (n =598, 54% of all women recorded on the database). The remainder had either declined the service altogether (15%, n = 168), declined it at that point in time (leaving open the possibility that they might accept support later, 3%, n = 35), or were recorded as undecided on whether or not they wanted the service (6%, n = 64).

There was considerable variation across Hubs in the levels of women recorded as having been contacted, ranging from 95% to 61% contact. The proportion accepting support once contacted also varied, from 60-80%. Given known issues around the accuracy and completeness of the monitoring data, these figures should again be treated with some caution. While perceptions of reasons for variation in take-up were explored in staff interviews for this evaluation, these identified generic factors behind differences in women's engagement with Caledonian (see section 3.2.3, below), rather than anything that would explain variation across areas.

Table 3.5: Level of uptake by women to the Women's Service

Stage	TOTAL	Lothian and Borders	Dumfries and Galloway	Forth Valley	Aberdeen	Ayrshire
All identified by the Caledonian System [20]	1,116	294	95	269	261	197
All with whom contact was made	865 (78%)	264 (90%)	90 (95%)	164 (61%)	180 (69%)	167 (85%)
All accepting the Women's Service (% of those with whom contact made)	598 (69%)	158 (60%)	72 (80%)	110 (67%)	129 (72%)	129 (77%)

3.2.3 Engagement with the Women's Service

Women's Workers are asked to record women's level of engagement with the service in parallel with the key stages of the Men's Programme. They are assessed as falling into one of four categories:

• Assessment - when contact has not yet been established, or when a woman has been out of contact for a while and their engagement is unclear
• Active - actively engaged with the service
• Passive known - no longer actively engaged with the Women's Service, but Caledonian staff are still in occasional contact (the woman may, for example, be in touch with the Men's Worker to be kept up to date on their partner's broad progress)
• Passive unknown - is not in contact with the service and is no longer able to be contacted.

As shown in table 3.6, the proportion of women recorded as actively engaged with the System is highest at the early, Pre-group stage (43%), and reduces at Group (33%) and Maintenance (24%) stages. Meanwhile, the proportion of women recorded as 'passive' (known or unknown) increases sharply over time: (46% at Pre-group, 62% at Group and 74% at Maintenance stage).

Table 3.6: Level of engagement by women at each stage of Men's Programme

Stage	Pre-group	Group	Maintenance
Assessment	11%	4%	2%
Active	43%	33%	24%
Passive known	33%	46%	57%
Passive unknown	14%	16%	17%
Base (all with engagement records in monitoring data at this stage)	569	315	242

The figures in Table 3.6 are based on those women with an engagement record at each stage in the monitoring data - if all women in the monitoring data were included, then the proportion recorded as actively engaged at each stage would be lower. However, although data on women is recorded in parallel with the stages of the Men's Programme, this is arguably a misleading picture since the Women's Service is voluntary and women are not required to stay engaged for any particular length of time. Both participants and Women's Workers identified reasons why their engagement might fluctuate or tail-off over the course of a man's two-year order, including:

• High levels of control in a relationship, which may mean that they find it difficult to engage at particular points (especially near the start)
• Anxieties (particularly in the early stages) about the impact of their engagement on their partner's case or sentencing
• Changes in their circumstances, particularly following separation which may mean they either feel more in control of their own safety or want to move on with their lives rather than be reminded of their partner's behaviour
• Improvements in women's own levels of self-confidence, which might mean they feel less need for support over time.

Staff interviews also discussed perceptions of the reasons some women decline the support offered by Caledonian altogether. In addition to the factors noted above, fear of professional intervention in general (and fear of children being removed in particular) was also believed to be a factor.

Most of the factors above are individual factors, associated with the women's own circumstances and feelings about their situation. Staff did not identify any particular changes to the Caledonian System that they felt would enable them to engage women more effectively. Rather, they emphasised that persistence in keeping in contact with women is key. This was confirmed by interviews with women participants:

I couldn't tell you how regularly but, a phone call or a message just to say, "look it's OK, I completely understand, but the support's here if you need it, just keeping, just letting you know that we're here", and we'd leave it at that … Now if [Women's Worker] hadn't done that I wouldn't have engaged because I'd already decided, "oh my God, no way, there's just no way." I want to put all that behind me. But, without [Women's Worker] doing that I wouldn't have come as far as I have, so I think it was quite an important approach that she took.

(Women's Service participant 7)

At a minimum, Women's Workers send a letter to women whose partners are on the programme every three months to remind them that the service is still available to them, but Women's Workers also described much more proactive approaches (particularly where there were ongoing concerns about risk), including accompanying other services when they are visiting the woman, and trying to speak with them privately if they accompany their partner to a meeting with his man's worker.

Women's Workers in one area suggested that their involvement in taking victim statements for court reports had helped increase the numbers accepting the Women's Service by involving Women's Workers earlier on.

3.3 The Children's Service

As discussed in Chapter Two, in most cases, the System does not work directly with children but rather works to ensure that their needs are being considered and met by wider services - for example, by referring families to services that can help their children, or by attending multi-agency meetings to ensure that children's needs are taken into account. Quantifying the number of children 'supported' by the Children's Service is therefore problematic, since it was not designed to offer direct support to the same extent as the Men's Programme and Women's Service.

However, the men's monitoring data does include data on the number of children known to be linked to the man. This indicates that 686 children were linked with the 941 men recorded in the monitoring data. The number of children recorded as linked with men in the monitoring data varied considerably by area - the proportion of men recorded as having no children at all varied from 32% to 100%. This may, in part, explain the reasons for variations in the resourcing and delivery of the Children's Service by area - if the men referred to the service to date have had few children between them, there may have been less perceived need for a full time Children's Worker or for the Children's Worker to engage directly with families.