Caledonian System Evaluation: Analysis of a programme for tackling domestic abuse in Scotland

5. Perceived impacts of the Caledonian System on men

Key findings

• Although the monitoring data cannot be used to conclusively assess the impact of the Men's Programme on behaviour, it does indicate that those men who completed it posed a lower risk to partners, children and others by the end of the programme.
• Psychometric data on changes in men's attitudes presents a more mixed picture (and is more difficult to interpret, given wider debates about the use of psychometrics). There was some evidence that participants make progress in terms of general attitudes and feelings that may be predictors of abuse and in reduced tendencies to blame their problems on either chance or other people. However, there was less clear evidence of any change in whether men feel they have control over their own lives. And, if anything, the psychometric data indicates that men may display a greater tendency to exaggerate positives about themselves by the end of the programme.
• Men said the programme had equipped them with techniques to better control their behaviour and reactions and helped them learn to communicate more positively with their (ex) partners. The group sessions gave them the opportunity to practice new skills.
• Men also reported improved understanding of the nature of abuse and of appropriate behaviour in relationships; a greater awareness and understanding of the inequalities that exist between men and women; and a more 'positive mindset' about both their relationships and themselves.
• Other perceived impacts included: helping men to address substance misuse problems (an issue for a majority at Pre-group stage); improvements to health; and general improvements to confidence, particularly as a result of learning 'positive self-talk'.
• Women interviewed for the evaluation expressed more mixed views about whether the Caledonian programme had any impact on their (ex) partner. In some cases, they were unable to comment at all since they no longer had any contact with their ex-partner by the end of the Men's Programme.

5.1 Introduction

This chapter explores, the impact of the Caledonian system on men - their behaviour, attitudes, motivations and knowledge about domestic violence, as well as any wider impacts on their lives and wellbeing. It draws on a combination of the men's monitoring data and qualitative interviews with men, women and staff. ^[21]

5.2 Impact on men's behaviour

As the Spousal Assault Risk Assessment ( SARA) questionnaire is administered at both Assessment and the Maintenance stage ^[22] , it can be used to assess any changes in the level of risk men are assessed as posing to their partners over the course of the Men's Programme - an indication of behaviour change. Changes in SARA scores suggest that, among those men who stayed on the Men's Programme until Maintenance stage, the risk they posed to their partners decreased substantially over time (Figure 5.1). The proportion assessed as 'high risk' to their partner decreased from 26% to 8%, while the proportion classed as 'moderate risk' fell from 62% to 32%. By Maintenance stage, the proportion classed as 'low risk' increased from 12% to 60%.

Figure 5.1: SARA 1 (Risk to partner) at Assessment and Maintenance stage

Base: All with a SARA 1 score at Assessment and Maintenance stages (195)

Although, at Assessment stage, more men were classed as 'low risk' to children and others compared with partners, SARA scores for these also moved in a positive direction. The proportion classed as 'high risk' to children fell from 6% to 1%, while those classed as 'moderate risk' decreased from 31% to 15% (Figure 5.2). Only a small minority were classed as 'high risk' to others at either Assessment or Maintenance stages (3% and 1% respectively), but the proportion classed as moderate risk decreased from 25% to 16% ( Appendix C, Figure C1).

Figure 5.2: SARA 2 (Risk to children) at Assessment and Maintenance stage

Base: All with a SARA 2 score at Assessment and Maintenance stages (195)

In the absence of a control group, changes in SARA scores cannot be interpreted as conclusive evidence that the Caledonian programme has effected behaviour change. However, they indicate that for those men who remain on the programme there is considerable improvement in risk levels over time.

The positive impact on men's behaviour suggested by the SARA scores was echoed in interviews with men and staff. Men who were interviewed were overwhelmingly positive about the impact they believed the programme had on them, providing a range of examples of positive behaviour change. In particular, they reported:

• Improved ability to control their behaviour and reactions. Men described how useful they had found some of the techniques introduced early on in the Men's Programme, such as 'taking a breather' and 'self-calming' in helping them to de-escalate or remove themselves from situations that might lead to arguments. In addition to reporting that they were no longer physically violent, men also felt they were shouting less at their (ex) partners as a result. The role of the group workers was highlighted by men as key in helping them learn how to manage their behaviour, as was the use of stimulus materials designed to help them reassess how men should behave in particular scenarios.

The Caledonian programme probably learnt me to stop and think just for that tenth of a second, and for me that's life changing. It might not be much but that tenth of a second is enough sometimes between saying yes and no, or doing the wrong thing, doing the right thing, just stopping and thinking.

(Men's Programme participant M)

The group workers are very good at making you think about the situation that you were in, what you could have done differently. Points in time where you could have taken yourself away from the whole situation totally. So I would say a lot of it is down to the group workers, they were great.

(Men's Programme participant D)

• Improved communication with (ex) partners. Men reported that they had learned to listen more and to give their partner the opportunity to get their point across. These improvements in communication were attributed both to discussions with case managers and group workers, and also to group sessions, which they felt had helped them learn to open up and listen to the views of others. Men also gave examples of communication techniques that they had learned from group work which they had applied at home with their partner, such as taking turns holding an object when they wanted to talk, so that the other person has a chance to have their say.

[I am now] listening to how she feels for a change, rather than me moaning and groaning and walking out in the middle and stuff like that.

(Men's Programme participant K)

Women interviewed for the evaluation expressed more mixed views on whether their (ex) partners' behaviour had changed. Some simply felt unable to comment, as they no longer had any contact. Those who were still in contact did identify some examples of positive changes in behaviour which echoed the accounts of men, including their partner acting more calmly, listening more, controlling their anger, and reflecting more on their behaviour.

I feel that he tries to listen or approaches me more when he notices that something is wrong or thinks something is up. He'll ask me, instead of being awkward or causing an argument over it, which is a big major. So, I feel like that is changing, he is trying to listen and be more understanding.

(Women's Service participant 17)

However, other women who were still in contact with their (ex) partner felt that there had been no noticeable change, including reporting that their (ex) partner had committed further offences while on the Men's Programme. This was particularly the case where women reported that their (ex) partner had problems with alcohol or drugs. In some cases, women viewed the man's problems as beyond the abilities of the Men's Programme to influence - a view echoed to some degree by staff, who suggested that motivating men to change could be more challenging when they had serious issues with drugs or alcohol, or wider mental health issues. As discussed in Chapter 3, it was suggested that the programme might benefit from more support from additional specialist professional input to help manage some of these complex issues.

It is important to remember here that the women we interviewed were not related to the men we interviewed. As such, their more mixed views on the perceived impact of the Men's Programme do not necessarily contradict those of men - it may be that the men we interviewed did indeed change, but that the partners of women interviewed for the study (who were not interviewed for this evaluation) did not.

Staff suggested that the length of the programme was important in effecting lasting behaviour change, since the stage at which men realise the impact of their behaviour varied widely between individuals. However, although lasting change can take a long time to achieve, staff also felt that the tools the programme provides early on and more 'generic' changes such as improvements in how they communicate can be a significant step:

Sometimes it will take two cycles [of group modules] for there really to be behaviour change. But just to make it a wee bit easier to express yourself, that is a massive change for some people.

(Men's Worker)

5.3 Impact on men's attitudes, motivations and knowledge

While the ultimate aim of the Caledonian Men's Programme is behaviour change, given the relationship between behaviour, feelings and values, it also works intensively around men's beliefs and attitudes. The specific areas covered are numerous, but there is a particular focus around: understandings of and attitudes to abuse; gender roles; relationship orientations; empathy; responsibility; and self-image.

5.3.1 Quantitative measures: psychometrics

The Caledonian Men's Programme includes a variety of 'psychometric tests' ^[23] , administered early in Pre-group stages and again at the Maintenance stage. ^[24] These tests include:

• The Propensity for Abusiveness Scale ( PAS), which measures men's attitudes, self-assessed emotions and past experiences across a range of areas known to be related to having a propensity to abuse, including: feelings of anger; beliefs about how others view you; attitudes to relationships; and experiences of punishment and violence as a child.
• Levenson Locus of Control ( LOC), which measures the extent to which individuals feel in control of their lives. It is made up of three sub-scales relating to: internality (the extent to which people believe they have control over their lives); powerful others (the extent to which people believe that powerful others control their lives); and chance (the extent to which they believe chance affects their lives). It is included to help staff understand men's relationship styles - a key area the Caledonian programme aims to work with.
• Paulhus Deception Scales (Balanced Inventory of Desired Responding, BIDR), which measures: impression management (the tendency to purposefully describe oneself in overly positive terms) and self-deception (the tendency to attempt to be honest but still exaggerate positive virtues). Presentation style is again one of the areas the Caledonian Men's Programme seeks to understand in order to assess men's accounts of their behaviour.

Interpreting the data from these tests is complicated for a variety of reasons. The value of psychometric tests is somewhat disputed in the academic literature. With the exception of the PAS scale, the relationship between high or low scores on these scales and domestic violence could be contested (for example, does a continued tendency to 'impression manage' necessarily correlate with being unable/unlikely to make progress in terms of attitudes and behaviours more directly linked to domestic violence?). As reported in Chapter 2, Men's Workers themselves were not always clear about the practical value of these assessments for their work with men. Perhaps as a reflection of this, there are some issues around missing data - of the 187 men recorded in the monitoring data as successfully completing the programme, only around 130 have completed psychometric tests at Maintenance stage. However, taking all these qualifications into account, we have included the scores here to provide a tentative indication of whether or not the Caledonian Men's Programme appears to be correlated with attitudinal change as measured by these scales.

Scores for each of the psychometric tests administered at Pre-group and Maintenance stages are shown in Appendix C, table C.5 (for those that have data recorded at both the pre-group and Maintenance stages). Each of these psychometric tests are measured on different scales, but in each case, a lower score reflects a more desirable position than a higher score. Among those for whom data was available, a positive change had occurred across three of the psychometric measures: Propensity for Abusiveness, LOC 'Powerful others' and LOC 'Chance' (each of these had a reduction in the overall score between Pre-group and Maintenance stages). In the other three measures, scores had either remained largely the same ( LOC Internality and BIDR impression management) or had increased ( BIDR self-deception). Although we cannot conclusively attribute change to the Caledonian Men's Programme, this suggests that participants do make progress in terms of general attitudes and feelings that can be predictors of abuse, and in reduced tendencies to blame their problems on either chance or other people.

However, it is less clear that they make any progress in terms of believing they have control over their own lives, which makes it difficult to draw conclusions based on the Locus of Control data on whether overall men accept more responsibility for their own behaviour in general by the end of the programme. Meanwhile, it appears that, if anything, men may exaggerate positives about themselves to a greater extent by the end of the programme (as measured by BIDR self-deception - designed to capture an unconscious bias to narcissism).

Given the general issues around interpreting psychometric measures discussed above, rather than viewing these findings on their own as positive or negative, further discussion and reflection is needed to explore why particular patterns are appearing and what they might mean for either programme delivery or the value of particular psychometric tests as measures of progress within the Caledonian System.

5.3.2 Views of qualitative interviewees

Men and staff (and to a lesser extent women) interviewed for the evaluation identified a wide range of positive changes in men's attitudes and understanding that they believed had resulted from the Caledonian Men's Programme. Interviewees identified changes in attitudes towards:

• Relationships and abuse. Men reported an increase in their understanding of what constitutes acceptable and unacceptable behaviour within a relationship, including acknowledging that their own past behaviour had been abusive. They attributed this to their own discussions with Men's Workers, and to the role playing exercises used in the group sessions, which encouraged them to understand different types of abusive behaviour and the impacts of that behaviour.

It's realising that something like shouting at somebody is exactly the same as hitting them in terms of the abusive side of things … you're abusing them in some way, you're using your abusive power.

(Men's Programme participant F)

It has certainly made me think a lot more about what's acceptable in the relationship. Certainly, somebody getting drunk and hitting their missus isn't.

(Men's Programme participant L)

• Gender roles. Men described having a greater awareness and understanding of the roles of men and women in society generally and the social inequalities that women had experienced in particular. They made reference to the 'Men and Women' model and the historical timeline exercise which helped to illustrate the different levels of rights afforded to men and women over time. They suggested that reflecting on historical/social gender inequalities had, in turn, made them reflect on their own relationship and the need for greater partnership and equality of roles.

I think the old fashioned values people have, like that the woman should cook and do everything round the house and that the man should go out to work, obviously times have changed and woman work now. I think a relationship should be a 50/50 thing and I don't think it always is, it tends to be more in the man's favour and it shouldn't be.

(Men's Programme participant N)

• Self and emotions. Men reported having a more 'positive mindset' about both their relationships and themselves. This was partly as a result of techniques they learned in group sessions such as positive self-talk, helping them to turn negative thoughts into positive ones, which in turn help them to control their anger and jealousy. The Men's Workers were also seen as playing a key role in helping them to open up about, and understand, their emotions. The importance of group sessions in changing attitudes was also stressed by both men and staff - men can change their attitudes and perspectives by learning from each other's experiences.

I like it when you've got somebody that's been on the group for a while and you're about to explain something and then one of the other men actually does it for you, and you kind of just take a back seat and you let the other men teach the new men in the group what it's about.

(Men's Worker)

Women interviewed for the evaluation felt even less able to comment on whether men's attitudes had changed than whether their behaviour had changed, though as noted above, those that were still in contact with them did give examples where they felt men were calmer and were over-reacting less.

5.4 Perceived impact on men's wider needs

The Caledonian Men's Programme aims not only to change behaviours and attitudes linked to abuse, but also to improve men's lives in a wider sense. This is in keeping with the 'good lives' model, which argues that people will be more strongly motivated to make positive changes (and more likely to maintain change) if they can see these goals in the context of working towards a better life as a whole.

5.4.1 Alcohol and drug use

The monitoring data includes measures of men's drug and alcohol misuse at both Pre-group and Maintenance stage. Although comparisons between the two are complicated by the use of different measurement scales at each stage ^[25] , the monitoring data indicates that, among those men who stayed on the programme to Maintenance stage the proportion with a drug of alcohol problem had reduced substantially - the proportion with an alcohol problem had almost halved (from 81% to 43%), while the proportion with a drug problem had more than halved (from 57% to 23% - see Appendix C, Table C.6 for full figures).

Of course, it is not possible based on this data to attribute this change to the Caledonian System. While men's own accounts indicate that in some cases the support they had received from the Caledonian Programme had helped them address substance misuse, in others they reported that they had already begun to address their problems prior to starting on the Programme, or that they were receiving help from elsewhere with these issues. However, the self-reflection that the Men's Programme encourages was viewed as a contributory factor in some cases, helping men to understand that they had a problem and to take steps to address it, as were lessons learned from the experiences of other men in the group.

I don't need anything anymore, completely drug free and alcohol free…. I think sitting there in the group, there was a few boys had problems with alcohol, but sitting watching them and how they've turned their lives around [has helped]. And as I say, a bit of self-reflection in yourself, you can see yourself and the damage you're doing.

(Men's Programme participant P)

5.4.2 Other impacts

Men also identified wider impacts from their participation in the Caledonian system on their confidence, work prospects and health. For example, one man had described how his Men's Worker had supported him as he decided to start his own business, by motivating him and helping him to feel more confident in his abilities. Another had suffered a series of health problems before being on the programme, but as a result of the programme he felt he had learned how to open up and communicate more, and to be more comfortable speaking with his doctor and seeking help for his health issues as a result. The positive self-talk techniques were again seen as a useful tool in contributing to these kinds of changes, in addition to the general support men had received from their workers.

I'm back to what I was like before. I'm happy, I just lead a normal life… and I'm more confident about my ability to deal with situations.

(Men's Programme participant J)

This programme has helped me to move on from the issues that I had and move on from the offence that I done, and realise, "okay, you've done the offence, you've made a mistake, put your hands up and try to contribute to the group as much as possible, now it's time for you to go and make a life for yourself".

(Men's Programme participant A)

5.5 Suggestions for development or improvement

Although the men interviewed for this evaluation were extremely positive about the Caledonian Men's Programme in general, they did have a few suggestions for improvement:

• Changes to meeting frequency / duration of Maintenance stage. It was suggested that it was important to make sure men were able to attend the Maintenance stage for at least four or five months, as it was an important ' bridge' between group work and returning to 'real life' to apply what had been learned. There was a concern that when men took longer to get to Group stage, or where they needed to repeat a module, they might run out of time for a proper Maintenance stage within their two-year order. One man described feeling 'let go' towards the end of the programme, with the move from three-hour weekly group meetings, to shorter monthly one-to-one meetings. He felt this was too much of a change, and that it might be better to move to fortnightly meetings first. These views may highlight a need to ensure approaches to 'exit-planning' within Caledonian are regularly discussed and reviewed, both within staff teams and within case management meetings.
• Additional professional psychological input. Interviews with men and with Caledonian staff both suggested a potential need for more input from professionals with expertise in psychology. While one view was that the Caledonian System gives staff the tools to identify men with particular personality disorders or traits, and provides guidance on approaches to working with them, staff nonetheless highlighted that men with borderline personality disorders could be particularly difficult to engage. At the same time,one view from the men's interviews was that given the high proportion of men with mental health issues on the programme, the programme would benefit from more professional psychological input in general. Findings on the extent of alcohol problems among men on the Caledonian programme also reinforce the suggestion (discussed in Chapter 2) that there may be a need for the programme to address alcohol issues more directly or in more detail.
• Changes to practical aspects. For example, in one area men found the location for the groups extremely inconvenient and expensive to travel to.

In addition, men suggested it might be helpful to include talks at the Group stage from men who had completed the programme and to have partners attend at least one session (to give them a greater understanding of what the programme is about). Men also expressed a desire for greater opportunity to explain their own situation or their 'side of the story' within the programme, relating an associated feeling of being blamed or being made to relive the incident week after week. Rather than necessarily indicating a need to amend the programme structure or content, this is perhaps a reminder of the challenges of holding men to account effectively while also building an effective therapeutic relationship and helping them to see a positive future, and may be a theme to revisit in (refresher) training.