Caledonian System Evaluation: Analysis of a programme for tackling domestic abuse in Scotland

Appendix B: Caledonian System monitoring data: issues and limitations

The main sources of data available to the evaluation team on how many men and women have taken up the programme, engagement and attrition at different stages, and the main quantitative measure of outcomes were excerpts of data from the Caledonian Monitoring Database. These were provided to the evaluation team in mid-April 2016, and included all data entered onto the Caledonian Data System from 2011 up until to this point.

The men's monitoring data includes details of: their basic demographic characteristics (age, employment status, ethnicity); their offending record; any issues with drugs and alcohol; their risk profile at various stages of the programme (as measured by the Spousal Assault Risk Assessment Guide or SARA); their programme responsiveness; their scores across a range of psychometric measures at various stages of the programme; whether or not they started and completed different stages of the programme (Pre-group, Group and Maintenance); and the reasons for attrition.

The women's monitoring data includes information about: their characteristics; their contact with/acceptance of the Women's Service; their engagement with the Women's Service at different stages; and their assessment of their partner's behaviour and attitudes at different stages of the programme. Monitoring data for men and women was provided in Excel format by each of the five hubs to Ipsos MORI and transferred to SPSS for analysis.

A pre-evaluation review of the Caledonian System conducted for the Scottish Government in early 2015 identified a number of limitations to this database as a tool for evaluating the Caledonian System (Burman and MacQueen, 2015). In particular, it flagged:

• high levels of missing data, both in terms of whole cases that are missing from the data altogether and large quantities of missing data at particular fields, particularly the assessment tools used with men at later stages of the programme, Women's Behaviour and Partner Scaled Checklists, and Women's Experience questionnaires (which directly seek women's views on the Caledonian System).
• limited data on the reasons for variations in women's level of engagement with the service (which could, in fact, reflect positive outcomes - for example, indicating that they have moved on from the relationship and do not feel the need for support). They also highlight challenges around using a database with a linear structure (following the Men's Programme) to explore women's engagement, which may fluctuate over time for good reason.

The report authors also highlight the fact that the database was not originally intended to be the main source of information for evaluation purposes, and that staff had expressed concern about its use as an evaluation tool.

Since the pre-evaluation report, work has been conducted to try and improve the completeness of the dataset - for example, by trying to address some of the issues around mandatory data fields and 'controls' in the system that were in some cases preventing Hubs from entering all the information they had collected. However, analysis of the data provided to Ipsos MORI for this evaluation team and discussions with Caledonian staff highlighted ongoing issues. In particular:

• Missing cases from the men's data. Delivery Managers and Data Champions (who are responsible for entering data for each Hub) reported that the actual number of participants was higher than indicated by the monitoring data for a number of reasons, including the fact that 'closed cases' are currently deleted from the system altogether after a period of time (cited as 3 years in one Hub) for data protection reasons, and as a result of issues around mandatory data fields (for example, one area reported that 27 cases were missing from the data included in this report as a result of the fact the database insists on an LSCMI score, which were introduced in that area after Caledonian) . There is also something of a time lag in entering data, so some men who have started Pre-group but not finished it yet will not have been entered.

Estimating the number of missing cases is difficult, but discussions and additional data provided to the evaluation team by Hubs indicate that there may be in the region of 100-180 men who have started the programme but are still at Pre-group stage, and 40-80 men who started on the Caledonian programme but are missing from the monitoring data for a variety of other reasons.

It is also worth noting that the data does not include any information about men who were assessed for Caledonian but who were not, in the end, given orders to attend - it is not therefore possible to use the monitoring data to assess how the profile of men assessed but deemed unsuitable for Caledonian compares with that of those who start the programme.

• The scope for inclusion in the women's monitoring data misses some women who are provided with (limited) support by Caledonian. The monitoring data only includes women where their partner is assessed as suitable for Caledonian. However, as noted in Chapter 2, Caledonian staff meet with women during the assessment stage and provide some (more limited) support to them even when their partner is not in the end given an order to attend the Men's Programme. These women are not included anywhere in the monitoring data.
• The women's monitoring data cannot be used to evaluate the impact of the Men's Programme. As discussed in Chapter 2, Women's Workers reported significant difficulties around completing the Women's Behaviour Checklist and Partner Behaviour Checklist when women were no longer in contact with their partner. The level of missing data in these fields prevent their being used to evaluate the Men's Programme - the monitoring data to mid-April 2016 included only 133 partner checklist recorded at Pre-group, 55 at Group and 33 at Maintenance stage. The discussion in Chapter 2 indicates that this is not simply a case of the monitoring data having been inadequately completed - rather, it indicates substantive and real challenges around basing evaluations of the Men's Programme on the views of (ex) partners on their progress, given the level of contact they may have as the programme progresses.
• Ongoing issues with missing data from particular fields. While in some cases (as with the Women's Behaviour Checklist), discussions with staff suggest clear reasons for this, in other cases it is not completely self-evident why data has not been entered. For example, in the men's data there are 382 cases under the field "started group work", but only 350 with a date for when group work started, and 310 with a "programme responsiveness" score. While the reasons for this are not wholly clear, interviews with staff for this evaluation confirm that staff have mixed views on the purpose and use of the monitoring data - as discussed in Burman and MacQueen (2015), there remained a perception that Caledonian teams do not get anything back from the (very large volume of) monitoring data they collect and that it is just ' form filling'. This may, in part, explain why some fields still have missing data.

In addition to these problems, it is also worth noting that the dataset has a counter-intuitive structure and, in part as a result of this, it is not straightforward to identify which variable should be used to try and assess particular outputs or outcomes. In particular, because data for each stage is only entered as men enter the following stage, this means that, for example, rather than looking at the variable that apparently shows the number of men starting the Group stage to identify this figure, the actual number starting Group stage is more accurately estimated by identifying all those men for whom some Pre-group data has been entered. This structure introduces a high risk of potential error in attempting to construct an accurate (or as accurate as is possible) picture of participation and attrition.