Evaluation of Sixteen Women's Community Justice Services in Scotland

2 Evaluation Approach

Introduction

This section briefly describes the purpose and aims of the evaluation, the methods used, and the scope and limitations. Further details are provided in Annex C.

Purpose

2.1 The evaluation was commissioned to:

• Examine the implementation of WCJSs at a national level to understand how resources were used, what services were delivered, and with whom.
• Identify the extent to which WCJSs contributed to outcomes associated with reducing reoffending during the funding period.
• Support WCJSs to build their local capacity for self-evaluation.

Evaluation questions

2.2 The evaluation was guided by four key questions and associated sub-questions (see Annex A):

• What was the need or reason for developing WCJSs?
• How were WCJSs implemented?
• What progress was made towards the intended outcomes?
• How do the findings inform future practice and policy decisions?

Methodology

2.3 The evaluation used mixed methods and was underpinned by a realist approach which aimed to understand 'what worked, for whom, and in what circumstances'.^[iv]

2.4 The evaluation was conducted between September 2013 and December 2014. Findings were drawn from multiple sources of quantitative and qualitative information, including: service-user data; secondary documents (e.g. individual WCJS logic models and reports); observations; and interviews and focus groups with women, practitioners, and partner organisations.

Quantitative data

2.5 Fifteen WCJSs gathered individual-level data about the women in their WCJS between 1 April and 31 December 2014 using a standardised 'service user questionnaire'.^[7] Data included:

• A profile of women when they entered (e.g. demographics, source of referral, offence-related details and presenting needs)
• Progress data on 14 outcomes (see Annex D), which measured changes in women's circumstances (e.g. housing) and attitudes ('readiness to change' indicators) at exit, and/or six months, using 'direction of travel' statements ('got better, 'stayed the same', or 'got worse') compared to their state at entry.^[8]
• The type of support delivered and reasons for exit from the WCJS (for those who left during the evaluation period).

Qualitative data

2.6 Qualitative fieldwork was undertaken in two phases - mid-way (June 2014) and towards the end of the evaluation period (September to November 2014).

2.7 In June 2014, key practitioners (e.g. project leads or funded single posts, such as a Criminal Justice Link Nurse) in all 16 WCJSs were interviewed by phone or in-person. Interviews were semi-structured and focused on experiences implementing WCJSs to date.

2.8 In September-November 2014, fieldwork was conducted in nine WCJSs (representing all three models and unique projects, selected by stratified purposeful sampling). Fieldwork explored the experiences of the participation, activities and outcomes elements of the national logic model. These nine WCJSs therefore feature more strongly in the quotes and examples in this report. This phase of fieldwork included:

• Semi-structured interviews with 37 women (lasting 20-40 minutes per interview)
• Eight focus groups and three individual interviews with practitioners
• Interviews with eight participants from partner agencies or CJSW teams linked to seven of the nine WCJSs (e.g. social workers from teams outwith the WCJSs, strategic leads, a Sheriff and a Procurator Fiscal).

2.9 The women who participated in interviews included individuals who had a range of presenting needs, experiences of the justice system and of WCJSs. Their ages ranged from early 20s to early 60s (60% were in their 20s and 30s) and most attended WCJSs on a statutory basis (59%). Care was taken to ensure participants' characteristics were broadly representative of all women in WCJSs and the sample was reviewed between fieldwork visits (see Annex C for methodology details). However, in the absence of random, representative sample, the fieldwork underrepresented younger women and those engaging in WCJSs on a statutory basis. The length of time that interviewees were in the WCJS ranged from three weeks to over 18 months. Women's quotes in the report do not use their real names.

2.10 Qualitative data from interviews were analysed alongside observational data (from site visits) and secondary data (e.g. funding proposals).

Analysis

2.11 The quantitative data was cleaned and verified with WCJSs as needed. Key descriptive statistics were calculated in Excel at a national (aggregate) level. Further in-depth quantitative analysis explored themes identified in the qualitative data, and links between inputs, activities, and outcomes across WCJSs and service user groups.

2.12 Qualitative data was analysed in NVivo using a thematic approach based on Framework methodology.^[v]

2.13 Information from both the qualitative and quantitative streams was consolidated against the evaluation questions and national logic model. Emergent findings across all data sources were triangulated within the context of existing literature.

Scope and limitations

2.14 A key limitation of the evaluation was the short timeframe for assessing outcomes. Many WCJSs had only recently implemented the funded changes to their service, which meant there was limited time for developments to have 'bedded in' to allow longer-term outcomes in women to materialise. Progress assessments, using the service user questionnaire, provided only a 'snapshot' of women's progress (between April and December 2014) in what is otherwise an iterative, complex and long-term journey for women towards desistance.

2.15 Exploring the effect of WCJSs on long-term outcomes (e.g. women lead better lives, are empowered and integrated in their community and reduce (re)offending) was outwith the scope of the evaluation. Given the large number and diversity of WCJSs, the evaluation was not designed to disentangle the effects of specific interventions or models. Attributing change to specific aspects of WCJSs would not have been methodologically robust without a suitable control group and/or representative sample.

2.16 Women who participated in interviews were engaged in WCJS activities (leading to a positive sampling bias) as opposed to women who were not engaging in WCJSs at all. Evidence from external partners (e.g. sentencers, SPS, third sector organisations) to validate their awareness and experience of WCJSs was limited, except where external partners participated in WCJS focus groups.