Caledonian System Evaluation: Analysis of a programme for tackling domestic abuse in Scotland

6. Future development

Key findings

• Overall, the Caledonian System is highly regarded by participants, staff and stakeholders. Many suggestions for improvement relate to the details of specific exercises or tools rather than the overall design of the programme. However, there may be a need for further reflection on how the Children's Service should operate.
• In terms of future resourcing, there was a perception that where Men's Workers have wider criminal justice caseloads, they may be over-stretched. Women's Service resourcing may need reviewing to ensure that all areas are able to offer the same level of service to new partners. Given staff and participant comments, it may also be worth reviewing whether there is a need for professional psychological input to the System.
• Other suggestions for change included: improvements to venues used in particular areas; more or enhanced training; and access to other services' information databases.
• Key recommendations for monitoring and evaluation include: an assessment of whether it is feasible to conduct a longer-term (3-4 year), prospective evaluation design with a large control group of families; improving the structure, content and management of monitoring data; and establishing the feasibility of data collection from children (potentially as part of a prospective qualitative element).
• Suggested improvements to the monitoring data include: considering whether to drop some items that have proved problematic to collect and interpret (particularly psychometric tests for men and partner behaviour checklists for women); considering whether alternative data should be collected, in particular revised outcome measures for women that focus on changes in their safety and wellbeing; reconsidering the time points at which key data can or should be entered (including potentially de-coupling the women's data collection timeline from the men's); providing clear guidance on (and resource for) data analysis and use; and re-launching the monitoring data, to build staff completion, use of and trust in the figures.

6.1 Introduction

This final chapter draws together findings on how the Caledonian System could be improved in the future. Suggestions for change focus on three main areas: the design of the programme; the resources available to support programme delivery; and future monitoring and evaluation. The first two sections draw primarily on suggestions from staff, stakeholders and participants and refer to more detailed suggestions already included at the end of previous chapters. The third section, on monitoring and evaluation, also picks up on issues raised earlier in this report, but goes beyond these to consider how future monitoring and evaluation might contribute more to the ongoing development of the Caledonian System.

6.2 Changes to programme design

Overall, the accounts of participants, staff and stakeholders indicate that the Caledonian System is a highly-regarded and well-designed intervention. The length of the Men's Programme, the systems approach to working with whole families, and the in-depth assessment process were all identified as key strengths. Many suggestions for improvement related to the details of specific exercises or sessions in the manuals or to the specific tools used to assess men and women's needs, rather than to the overall shape and design of the programme. As these specific suggestions have already been discussed (primarily in Chapter 2), they are not reiterated here. However, there were two suggestions for change that touch on the broader shape of the programme:

• What is offered by the Children's Service. As discussed in Chapters 2 and 4, staff and women both raised issues around whether or not the Children's Service is consistently meeting children's needs in all Hub areas. The Caledonian Children's manual, which was substantially revised in 2013, is prefaced by the advice that ' This manual provides guidance rather than instruction because local areas have their own resources and approaches for working with children.' However, these findings raise the question of whether, in practice, there is too much variation in the service available to children.

A central issue is the extent to which the Children's Workers ought to be involved with families directly, rather than simply advising other Caledonian staff or wider services on how to ensure children's rights and needs are met. The revised manual strongly emphasises that ensuring children's needs are met cannot and should not be the role of one worker, but should be central to the work of all staff in the Caledonian System. However, interviews with staff and women suggest that, nonetheless, there may be unmet need for dedicated support for children affected by domestic abuse. While there will undoubtedly continue to be a need for local flexibility around precisely how children are supported within the Caledonian System, these findings may suggest a need for further discussion about the roles and resources needed (either within or outwith Caledonian) to ensure children's needs are met.

• The length of time the Women's Service keeps in contact. As discussed in Chapter 2, being able to offer support to women for two years was seen as a strength of the Caledonian System (a view confirmed by the women we interviewed, most of whom had been engaged for the full two years). Staff were strongly of the view that keeping in touch with women over an extended period of time is important in ensuring that those women who may not be ready to engage earlier on are still given the opportunity to access support. However, there was also a belief among staff that, for women who did not need as much support, it was sometimes unnecessary or even potentially harmful to keep re-contacting them to talk about past experiences of abuse. In part, this was linked to the specific issue of the appropriateness of using the behaviour checklist at the end of the programme. If this issue could be addressed (see 6.4 for further discussion), it may be that keeping in occasional contact by letter or phone would be less problematic.

Finally, in terms of programme design, it is worth noting that stakeholders expressed a desire for the Caledonian Men's Programme - or something like it - to be available more widely, including men who were not the subject of court orders. In general, there was a perception that in most areas of Scotland, there is nothing available to men who carry out domestic abuse until they have been convicted, when earlier intervention could well be successful. A non-compulsory version of the Caledonian System is in fact currently being run in Edinburgh (but was not part of this evaluation).

One stakeholder view was that similar interventions should also be in place for sex offenders, women perpetrators of domestic abuse, and same-sex couples affected by domestic abuse.

6.3 Changes to resources to support delivery

The data collected for this evaluation does not allow a detailed assessment of the costs associated with delivering the Caledonian System. However, interviews with staff and participants did identify a number of issues around the resources available to support delivery that may need to be taken into account in considering its future funding and development. In particular, they raised issues around: staff resources; physical locations; training and support; and information sharing.

6.3.1 Staff resources

Given the variation in staffing structures and delivery models across areas (discussed in Chapter 2), unsurprisingly, there was no consensus among staff on whether or not additional staff resources were needed to support delivery. However, where staff suggested more resources were needed, this appeared to reflect:

• Issues around the role of the Children's Worker - where they were working directly with families, one view was that more resource was needed to support this work.
• Issues around whether or not Men's Workers (particularly Case Managers) were in dedicated Caledonian roles or whether they were also taking on wider criminal justice work. Where Case Managers also had general criminal justice caseloads, there was a perception that they could be over-stretched and lack the time to develop expertise in managing Caledonian cases.
• A perception that the Women's Service in particular areas needed more resources to ensure that all women - including new partners - can be offered the support they need.

Staff and participant accounts also suggest that there may be a need to consider whether additional professional psychological input is needed to ensure the programme is able to work effectively with men and women with complex mental and emotional issues.

6.3.2 Physical locations

Staff and participants suggested a number of improvements to the physical locations used by the Caledonian System, including:

• Changes to the venues for groups in one area to a more convenient and accessible location
• A more suitable venue for meeting women (when it was not possible or appropriate to meet them at their homes), since one view (among Women's Workers) was that social work offices were not particularly comfortable or appropriate for them
• Changes to where Caledonian teams are (co)-located - in addition to a belief that the System works most effectively where whole teams are co-located, it was also suggested that it would be ideal if they could be located alongside relevant staff from the other key services they work with on a regular basis - for example, police, health and housing.

6.3.3 Training and support

As noted in Chapter 2, staff expressed a desire for more training, particularly refresher training, to support delivery of the Caledonian System. This training could address some of the specific delivery issues identified by this report, including:

• Information sharing/joint-working between Men's and Women's Workers and the role of meetings with women in informing Case Managers' work
• Information sharing with women around the Men's Programme - how and to what extent should they be kept updated on men's progress and/or the content of the Men's Programme at particular stages?
• How can Men's Workers best manage men's desire to 'give their side of the story' in the context of a programme aimed at enhancing their accountability for their behaviour?
• How psychometric tests should be used and interpreted (something already covered in Caledonian training, but which this evaluation indicates may need further ongoing attention).

Delivery Managers interviewed for this evaluation (whose views were, in the main, similar to those expressed by other staff) suggested that there may also be a need to build in more comprehensive support for staff to help them cope with potential 'vicarious trauma' from dealing with domestic violence cases on a daily basis. This was not a specific focus of interviews with staff, so may merit further discussion to inform decisions about what is needed going forward. However, insofar as staff did discuss the support they received from management, they were very positive:

I've never worked anywhere so supportive. (…) There is the respect there, so they are more likely to listen and similarly with seniors [senior staff]. I've worked in various jobs in the past and that's not been present and if that's not there, it just makes it a toxic environment. This is such a good environment.

(Men's Worker)

6.3.4 Information sharing

Although information sharing between Caledonian teams and their organisational partners was generally reported to be effective, there were examples where it was felt to be working less well or could be improved. In addition to a need to ensure information-sharing protocols were being adhered to by other services (for example, in relation to passing on police reports), it was also suggested that having access to other databases, particularly from other social work teams, would help support more effective and safe delivery.

6.4 Future monitoring and evaluation

This final section of the report outlines the research team's specific recommendations for strengthening the future monitoring and evaluation of the Caledonian System. It is not a comprehensive evaluation plan - the suggestions here need further refinement and development (ideally involving Caledonian Teams, the Scottish Government and SAPOR in discussion around feasibility and resourcing implications). However, it outlines the research team's views of the key elements that need to be in place going forward.

6.4.1 Incorporating a prospective evaluation approach

A key limitation of this evaluation was the lack of feedback from men who had not completed the programme and women who had either turned down or dropped out of the Women's Service. To address this gap, a much longer-term (3-4 year) evaluation is required, involving recruiting men and women on referral to the System and tracking them through repeated interviews over the course of their involvement (or non-involvement - though this is often very challenging in practice, since people who leave interventions are often difficult to keep in touch with and may be unwilling to continue participating in research).

6.4.2 Assessing the feasibility of establishing a control group

Perhaps the key question which has dogged debate about evaluating DVPPs is whether or not experimental or quasi-experimental designs ought to be regarded as the 'gold standard'. On the one side are those who argue that, without a control group of men and women who are not referred to the intervention to compare with, any evidence of impact is inevitably inconclusive and flawed. On the other are those who argue that the ethics of randomised control trials (viewed as 'denying' women and children support for the sake of research design), the fact that professionals often do not implement them by the book, and issues around study recruitment and attrition mean that the evidence they provide is, in any case, far from perfect.

Given the difficulties numerous researchers have had in establishing an effective control or comparison group for similar interventions, ^[26] we would strongly recommend a scoping study to inform any decisions about its feasibility in a future prospective evaluation of the Caledonian System. While, in our view, ethical and practical barriers probably rule out a 'pure' experimental approach, whereby families are allocated on a random basis to Caledonian or not, a scoping study could explore whether it is possible to construct a 'matched comparison sample' of families from areas where the Caledonian System is not currently operating. However, there would still be substantial challenges around this, particularly around identifying, accessing and recruiting control families. Potential participants would need to be screened after convictions for domestic abuse, to ensure that they were equivalent in profile to those assessed as suitable for the Caledonian project. There would be multiple issues of access - how and when would researchers (or other gatekeepers) be able to contact men for screening and recruiting? And how would they access their partners in a manner that is safe and does not increase their risk? While these issues may not be insurmountable, they would need careful exploration before any prospective study including a control group is commissioned.

6.4.3 Improving the structure, content and management of the monitoring data

The structure, content and completeness of the monitoring data has presented various challenges for this evaluation. We would strongly recommend that, going forward, Caledonian staff are regularly involved in discussions about the monitoring data - what it shows and what changes or improvements to the System it might point to. If the monitoring data is used for continuous improvement, then staff are more likely both to complete the required measures and to identify any further changes required to ensure this data can more effectively measure outcomes.

However, based on our own understanding of the current dataset, we would suggest the following changes in order to improve its usability (for Caledonian Teams, future evaluators and the Scottish Government):

• Considering whether some items of data should become optional rather than compulsory, in particular:
  • Whether all the psychometric tests for men should be retained as key components of the monitoring data, rather than as tools for Men's Workers to draw on when relevant. If they are all retained, each tool need to be accompanied by clear guidance on how particular scores - and changes over time in scores - should be interpreted in the context of outcomes relating to domestic abuse. If expert psychological input was also built into the System, as discussed above, professional psychologists may be well placed to advise staff on the use of psychometric tools, as well as providing specialist input directly for those participants who need it.
  • Whether it is worth recording the partner behaviour checklist data at all as an outcome measure. As noted repeatedly in this evaluation (and elsewhere - for example, Kelly and Westmarland, 2015), treating women's accounts as the gold standard for measuring changes in men's behaviour (rather than changes in their own feelings of safety and wellbeing) is extremely problematic given that women may no longer be in contact with their (ex) partner in many cases. At the same time, there is a clear need to retain some triangulation or validation of men's accounts of changes to their behaviour - which leads to the next suggestion.
• Considering whether alternative or additional data should be collected, in particular:
  • Whether a different set of outcome measures should be used with women, reflecting changes in their perceived safety and wellbeing (which is a key aim of the System as a whole and is something women should be able to comment on regardless of contact with their (ex) partner). These could draw on questions developed for Project Mirabel (Kelly and Westmarland, 2015), which measured a wider range of outcomes for women.
  • Adding an open 'suggestions for improvement' field to both the men's and women's data at each key stage of data collection, in order to provide ongoing feedback on participants' views of the System.
• Reconsidering the points at which key data can be / should be entered. At the moment, there is a 'lag' built into the database, in that data is only entered for each stage as they move into the following stage. If it were possible to enter key data - such as the date at which the participant starts each stage - closer to the point in time this actually happens, this would help provide a more accurate and timely picture of participation and attrition.

At the same time, the timings for collecting and recording data from women should be re-considered in light of comments about the varying patterns of women's engagement. It may make more sense in future for their data to be 'de-coupled' from the stages of the Men's Programme, and instead to capture information about their safety and well-being and their level of contact with the System at 4-6 month intervals.

• Providing clear guidance on data analysis and use. As noted in Appendix B, the structure of the monitoring data is somewhat counter-intuitive. However, if it is to be used to provide a clear and consistent picture of participation, attrition, outcomes and (potentially) suggestions for improvement in future, very clear guidance is required on exactly what fields should be used as indicators of each of these and exactly what the base should be for each measure (including, for example, guidance on how to restrict analysis to men who could actually have completed the programme by that point in time). Ideally, there should be a standard template for regular reports of participation, attrition and outcomes and guidance about how to produce each figure, so that these are produced consistently over time and across areas.
• Building in resource to analyse (and reflect on) the monitoring data on an ongoing basis. Ensuring that there are resources in place to support collation of and reflection upon key figures on participation, attrition and outcomes on a regular (we would suggest twice yearly) basis will help ensure that the reasons for any differences between areas can be discussed, and that any issues with how the data is being collected or used can be identified and addressed quickly. As described above, this could be done using a standard proforma for each area.

If the monitoring data is to be genuinely useful in informing practice, there will need to be resource to produce these reports for individual Hubs, teams and (potentially) individual workers. There will also need to be space within team meetings and wider Caledonian networking events to reflect on patterns and differences in the data and what these might mean for future programme delivery or development.

• Re-launching the monitoring data. This evaluation has identified considerable scepticism among staff about the usefulness of the monitoring data, which in some cases is viewed as ' box ticking' which distracts from the overall purpose of the programme. In part, the changes suggested above are intended to address some of this. However, any changes will also need to be accompanied by discussions with staff (alongside further training) to explain the rationale and to convince them that implementing these changes will both reduce their workloads and provide them with more useful information going forward.

6.4.4 Establish the feasibility of data collection from children

Finally, a key gap in this evaluation is the lack of any data from children themselves. In considering the scope for involving them in future evaluations, there remain some key challenges. In particular, children may be unaware of the System unless they are receiving direct support from a Children's Worker, meaning the scope for asking children directly about their views of what difference, if any, the System has made to them remains limited in most cases. However, it may be possible, within a sensitively designed prospective qualitative study, to recruit a sample of children for repeat interview and to explore their understandings of the abuse, their feelings about their own safety, and their general wellbeing, and to examine how these change over time.