Evaluation of Sixteen Women's Community Justice Services in Scotland

Annex C: Methodology Details

1. The 16 projects varied in their proposals to enhance services for women, with a variety of contextual factors (e.g. urban and rural, statutory and voluntary, and types of partner agencies). They also varied in terms of pre-existing service provision, service type and scope (e.g. women received targeted/ intense support in specific areas, or support across many), locally intended outcomes, and target groups.

2. Acknowledging this diversity, the evaluation took a national approach to draw common themes and lessons across all models rather than a direct comparison (akin to comparing 'apples, oranges and pears').

Quantitative data

3. A questionnaire ('logic model questionnaire') captured basic aggregate data from the first 11 WCJSs funded in 2013/14. More substantially, an evaluation tool ('service user questionnaire') was available from April 2014. Practitioners in 15 WCJSs^[104] used this to capture individual-level data on the 1,778 women in their WCJSs between April and December 2014. Practitioners returned anonymised data to the evaluators in a standard Excel database.

4. Demographic and offending history information was provided for most of the total 1,778 women in WCJSs. In addition, practitioners recorded women's presenting needs, progress measures and exit information.

5. To assess progress, practitioners were asked to select a statement across 14 outcomes that best represented the individual when she entered the service. At six months and/or when women exited (anytime between 0 to 9 months), the assessment was repeated and a 'direction of travel' statement was selected for each outcome to describe whether she 'got better', 'stayed the same' or 'got worse' since her initial assessment (see Figure 4).

6. Overall, presenting needs information was available for 737 women and progress data for 406 women who entered between 1 April and 31 December 2014 (for quality purposes, retrospective assessments for women who entered pre-April 2014 were not included).^[105] Exit information was available for 644 women who left WCJSs between April and December 2014, including their assessed state at exit and type of support they'd received (available for 506 women). Exit data for women who entered pre-April 2014 was used as it had not been completed retrospectively.

7. The large sample size and national use of a standardised progress tool, while not perfect,^[106] gave 'direction and distance travelled' data for individuals that had been reported as lacking in earlier evaluations of women's community services.^[xxx]

Figure 4: Example question from service user questionnaire

8. In analysis of women's progress, descriptive statements in each outcome measure (Figure 4) were grouped by 'negative' or 'positive' state (see Annex D for definitions per outcome). Progress was explored by women's state at their initial assessment (i.e. positive or negative) in conjunction with the direction of travel (i.e. got better, worse or stayed the same) at most recent assessment. Since women's circumstances are varied and may only become evident once relationships are established (i.e. after the initial assessment is completed), analysis emphasised 'direction of travel' rather than specific descriptive statements.

9. Where necessary in the report, percentages are provided alongside the numerator (n) and denominator (N) in a footnote. Denominators may vary due to unknown/incomplete assessments.

Qualitative data

10. The evaluation included two phases of qualitative fieldwork. Phase 1 (phone interviews with key practitioners) focused on implementation (e.g. key drivers for developing their WCJS, experiences of delivering activities so far, extent to which they're reaching their target group, and any lessons learned to inform others looking to develop a similar service model).

11. Phase 2 (semi-structured focus groups or interviews with women (service users) and practitioners) focused on outcomes so far for women and local ways of working. Nine WCJSs were purposefully sampled based on WCJSs best-placed to illustrate the three main models of WCJSs and the two unique projects focussed on support pre-sentence (Kilmarnock and Ayrshire).

12. Prior to fieldwork, practitioners were encouraged to inform all women of the opportunity to participate in an interview (e.g. posters in drop-ins or centre) and invite individuals, with an emphasis on women who will represent the characteristics of women most typical in the WCJS (informed by quantitative data) and whom could inform key aspects of interest (e.g. use of particular multi-disciplinary support or in hubs, statutory/voluntary engagement). Participant sampling was reviewed during fieldwork to ensure adequate variation across other characteristics (e.g. age, length of engagement, level of risk/need). Participation was voluntary. Compared to the population of women in WCJSs overall, very young women and those engaging on a statutory basis were slightly underrepresented in interviews, and women who participated in interviews were mostly engaged in WCJS activities, leading to a positive sampling bias (as opposed to women not engaging in WCJSs at all).

13. All participants were issued with an information sheet and an opportunity to ask questions before informed consent was taken. Ethical approval, strategies regarding welfare and disclosure, and research access requests were developed or completed in conjunction with Scottish Government's Justice Analytical Services Department.

14. Interviews and focus groups with practitioners and partners explored the WCJSs target group, what had worked well/not in delivering activities, what changes had been observed, in whom and to what was attributed, lessons learned, e.g. critical features of success, areas for development and any unintended consequences. Questions were tailored to each WCJSs and/or role(s) of practitioners who participated.

15. Interviews with women explored their experience and reflections of the service. Questions were tailored to circumstances of individual women, but generally covered women's background (e.g. reason they attended, how any previous experience of CJSW compared), what they did in WCJS (e.g. support they valued/not), experience of various characteristics of the service (e.g. environment, practical accessibility to WCJS, relationships with workers and other women, service user involvement, voluntary nature of engagement, any changes they would make), and what changes (if any) they had noticed in areas of their life since attending (prompted in key areas if necessary, including whether offending had stopped or reduced) and their perceived reasons for change.

16. Transcripts and notes were cleaned and checked for accuracy. The data was reduced into new and existing themes (based on national logic model) through initial coding using NVivo data management software. Data was displayed in matrices, which facilitated understanding of women and practitioners/partners responses within each main theme. Patterns and connections within and between themes were identified and interpreted across WCJSs and service user groups. Interpretation was informed by quantitative data (and vice versa). Data was recoded where necessary as the understanding of the data evolved.