Evaluation of Sixteen Women's Community Justice Services in Scotland

10 Evaluation and Future Research

Introduction

This section describes lessons learned from the evaluation approach, and areas for future research.

Evaluating services

10.1 The Commission's Report highlighted that effective services should have in place "robust performance management and evaluation arrangements to be able to demonstrate impact and value for money".^{[i], [102]}

10.2 WCJSs all held considerable amounts of information about the women they worked with. However, services appeared to vary considerably in their available resources, expertise and/or systems to demonstrate the difference they made. None of the 16 interventions were allocated funds to develop their performance management and evaluation arrangements, as building local evaluation capacity was within the remit of the national evaluators.

10.3 One of the aims of the evaluation was to support WCJSs to build their local capacity for self-evaluation. The scale of the national evaluation meant that the contribution to local evaluation capacity was modest, but included: logic model development sessions, support to use Excel data collection tools and a dashboard to present data, feedback on data quality and completion, presentations at the practitioner's forum, and a self-evaluation workshop to support WCJSs to 'tell their story' using evidence.

10.4 The specific enablers and barriers experienced in this evaluation are highlighted in existing literature^{[xvi],[xvii], [xxix]} and summarised in Table 12. Barriers and enablers have been drawn from existing literature but reflect the observations of evaluators in this current evaluation, and evidence from WCJSs' secondary documents.

Table 12: The barriers and enablers to building evaluation capacity in WCJSs

10.5 This evaluation (and that of the Reducing Reoffending Change Fund mentoring services) provided an opportunity to trial a standardised outcomes tool (the service user questionnaire). The tool captured 'distance travelled' of women against known criminogenic needs and enabled measures to be reported at a national (aggregate) level to demonstrate the contribution of WCJSs to women's desistence journey. This was a need recommended in previous evaluations of women's centres.^[xvi] The lessons learned from this evaluation should inform future improvements to the tool's content and implementation.

10.6 In general, practitioners were receptive to evaluation activities. However, some expressed valid frustrations relating to the initial delays in receiving the service user questionnaire (particularly where they were funded from 2013), and ironing out unforeseen issues as a result of 'trialling' it across different services.

10.7 Challenges in introducing the service user questionnaire tended to be greater among large, established WCJSs where it introduced another layer of data gathering to existing processes and systems. New services were typically smaller, less complex, and had a 'clean slate' on which to introduce the tool. Overall, regardless of size, WCJSs experienced challenges relating to the time, knowledge and technical skills required to interpret their data and link activities to outcomes at a service level. We understand that some WCJSs intended to continue to tailor and use tools from this evaluation to inform evidence about the impact of their service in future.

Areas for further research

10.8 This evaluation identified a number of areas for future research, including:

• Longitudinal studies to understand the impact of WCJSs on long-term outcomes, including women's reintegration into the community and desistance (this may include linking WCJS records with centrally-held criminal proceedings data, such as reconviction rates)
• A case-control study to explore the extent to which outcomes can be attributed to particular interventions (including groupwork)
• A comparison of breach and compliance rates by different sentence types and approaches to engagement (e.g. structured deferred sentences, and the effectiveness of voluntary versus statutory engagement in WCJSs)
• In-depth analysis of the characteristics of those who find progress difficult or disengage from services (including for example interviews with women who have declined holistic support, have reoffended since attending WCJSs/ are in prison)
• Further qualitative work with women and staff to understand why some outcomes appear to be hard to achieve in practice
• Interviews with sentencers to gain an understanding of their perspectives of WCJSs and when they would use them
• Studies that include partner or mainstream agencies to explore the impact (including cost benefit) of WCJSs upon their work.