Evaluation of Community Payback Orders, Criminal Justice Social Work Reports and the Presumption Against Short Sentences – Appendices Report

1 Appendix One: Research Methods

Summary of methods used

1.1 In recognition of the complex and multi-faceted character of the reforms and the research aims, the evaluation adopted a wide range of methods. At the heart of the study was a series of qualitative interviews with key actors - including Criminal Justice Social Work (CJSW) managers and staff, Sheriffs, other practitioners (e.g. Addictions workers, Mental Health professionals, professionals from third sector Drug and Alcohol Services, Sheriff Clerks, Managers of charity shops where some individual unpaid work placements take place), and offenders - in four case study areas. This was complemented by analysis of existing monitoring data, bespoke surveys of Sheriffs and Criminal Justice Social Work Managers, and a participative audit of Criminal Justice Social Work Reports (CJSWRs) allowing the evaluation to examine the implementation and operation of the reforms across Scotland as a whole, and to balance the more detailed picture emerging from the case study areas.

Ethical approval and research access

1.2 Ethical approval was obtained from NatCen Social Research (NatCen) Ethics Committee.

1.3 Access was granted to conduct the research with Criminal Justice Social Work staff from the Association of Directors of Social Work (ADSW). Access to Sheriffs was granted by the Lord President in the first instance and then by the Sheriffs Principal in each Sheriffdom.

Orientation and data scoping

1.4 Orientation interviews were conducted between April and June 2013 with representatives (n=six) of the following Scottish Government and practitioner groups:

• Scottish Court Service
• Crown Office and Procurator Fiscal Service
• ADSW
• Community Justice Authority (Glasgow)
• Judicial Studies Institute
• Scottish Government Justice Directorate policy

1.5 The aim of these interviews was to identify broad issues and perspectives on the reforms, to explain the purpose and form of the evaluation, and to seek views on the appropriateness of the specific data collection activities proposed. They were not treated as primary data but as a means of informing subsequent stages of the evaluation.

1.6 Additionally, initial discussions were carried out with statisticians from the Justice Analytical Services Division about the availability and reliability of data on Community Payback Orders (CPOs), other community penalties, short prison sentences and CJSWRs.

Sheriffs' survey

1.7 The purpose of the Sheriffs' survey was to explore whether the views of the relatively small number of Sheriffs interviewed in the case study areas resonated with those of Sheriffs working across Scotland as a whole.

1.8 The specific aims of the survey were to examine:

• Sheriffs' overall views of the introduction of CJSWRs
• How useful CJSWRs were to Sheriffs in their sentencing decisions
• How CJSWRs compare with Social Enquiry Reports (SERs) in terms of quality and consistency
• Overall use of Community Payback Orders (CPOs) by Sheriffs
• Views on CPOs as a sentencing option and in comparison with community sentencing options previously available
• Perception of any change in Sheriffs' use of short sentences and community penalties in light of the reforms

1.9 The survey consisted of a short (eight sides of A4) paper questionnaire sent to all 141 permanent Sheriffs (both resident and floating) then in post in Scotland. The questionnaire was sent by post (with pre-paid return envelope) and an accompanying letter explaining the purpose of the survey and its relationship to the wider evaluation of CJSWR, CPO and Presumption Against Short Sentences (PASS). The questionnaire was sent to Sheriffs on 7th April 2014 and they were initially asked to complete and return the questionnaire by 25th April.

1.10 Following early feedback from some Sheriffs and the Scottish Court Service suggesting concern about anonymity, the research team sent an email to all Sheriffs on 11th April to clarify that the serial numbers on the questionnaires were purely for internal administrative use within ScotCen to allow the team to monitor response and issue individual reminder letters if appropriate. The email also informed potential participants that they could return the questionnaire with the serial number torn off if they wished (three eventually did so). One further blanket email reminder was issued on 25th April, extending the final deadline by a week to 2nd May.

1.11 The questionnaire largely contained closed questions, though participants were also given the opportunity to add comments on particular responses or in relation to wider questions (e.g. how they felt CJSWRs might be improved).

1.12 In total, 72 completed questionnaires were returned to ScotCen, representing a response rate of 51%. This exceeded the target set in our original research proposal (of 40%) and provides a strong basis for examining the picture at a national level.

Survey of Criminal Justice Social Work Managers

1.13 A survey of Criminal Justice Social Work Managers was conducted to build a picture of key issues and progress in the local implementation of the reforms and, as a result, to assess the extent of variation across Scotland as a whole. We worked in consultation with the Scottish Government and ADSW with additional input from the Chiefs of Community Justice Authorities.

1.14 The specific aims of the survey were to examine:

• Views of training and development for staff in relation to the reforms
• How the new CJSWR template is working
• CJSW working arrangements and practices with the courts
• Perceptions of whether the CPO requirements are being used in accordance with the legislation and practice guidance
• The extent to which consultation and engagement have taken place within the community, and the main purpose of this

1.15 The survey consisted of a short (eight sides of A4) paper questionnaire sent to all 32 Local Authorities. The questionnaire was sent by post (with pre-paid return envelope) and an accompanying letter explaining the purpose of the survey and its relationship to the wider evaluation of CJSWR, CPO and PASS. The questionnaire was sent to CJSW managers in each Local Authority in July 2014 and they were initially asked to complete and return the questionnaire by 2 August 2014. Two reminder emails were sent to each Local Authority and the final deadline was extended to the end of August 2014.

1.16 The questionnaire largely contained closed questions, though participants were also given the opportunity to add comments on particular responses or in relation to wider questions (e.g. any additional comments about the implementation of the CJSW report).

1.17 Of the 32 Local Authorities that were contacted to take part in the survey of Criminal Justice Social Work Managers, 30 took part.

Participative Audit of CJSWRs

1.18 A participative audit of Criminal Justice Social Work Reports (CJSWR) was conducted in the four case study areas. This was done by collecting a number of CJSWRs from each of the four local authorities being studied (144 reports in total) and passing them on to a different case study area. CJSWs from this area then used a simple pen and paper auditing tool to assess different aspects of each report's quality.

1.19 The aim of the participative audit was to assess perceptions of the quality of current CJSWRs and the extent to which measures designed to improve report quality (making comparisons to Social Enquiry Reports, where relevant) appear to have been successful.

1.20 The participative audit tool was developed and designed by Simon Noble and ScotCen in consultation with the Scottish Government. Its design was informed by the relevant outputs and outcomes from the logic model for the evaluation, previous tools used by ADSW and Glasgow CJSW, and emerging findings from the fieldwork during the first phase of the evaluation. The final drafts of the tool were sense-checked with case study area lead managers and members of the audit teams before sign-off from the Scottish Government.

1.21 The actual auditing was carried out by CJSW staff within the four case study areas. Each area was paired with another with the purpose of auditing the other's reports: Area 1 was paired with Area 4, while Area 3 was paired with Area 2. The audit teams numbered between four and six staff and consisted of a mix of middle managers, court-based and report-writing practitioners, and staff development/quality assurance staff.

1.22 A sample of CJSWRs (n=144) submitted to court by the four case study areas between 1st September 2013 and 24th March 2014 was audited. The precise number of reports audited by each team varied according to the relative size (and thus capacity to resource the audit) of the LA.

• Area 1 audited 40 reports from Area 4
• Area 4 audited 32 reports form Area 1
• Area 3 audited 40 reports from Area 2
• Area 2 audited 32 reports from Area 3

1.23 The protocol for the participative audit involved:

• Each auditor reading their assigned number of reports, then
• Assessing the quality of the reports using the audit tool.

1.24 This process was supplemented by a follow-up "validation" meeting, where auditors' views on emerging results and the process of the audit itself were sought.

1.25 The reading of reports, although structured by the content of the audit tool, also relied on individual auditors' subjective assessment of many aspects of the sampled reports. A number of questions called for an opinion or a score on a defined scale. So, while the audit was extremely useful in drawing on practitioner expertise and understanding - and provided a valuable snapshot of report quality two years after the introduction of the Government's reforms - the findings should not be read as 'hard' measures of area performance. (The relatively small sample sizes are another reason for caution in this regard.) While individual case study areas may wish to review the findings for their own area in relation to the sample as a whole, this should be seen as a way of identifying practice development opportunities and not, in any sense, as an objective ranking exercise.

Case Study interviews

Selection of case study areas

1.26 The orientation and data scoping phase of the evaluation were used to help inform the selection of the case study areas. The following criteria were considered when selecting the case study areas:

• Type of area e.g. urban, rural, mixed.
• Overall use of CPOs
• Proportion of cases with CJSWRs submitted
• Use of different requirements within CPOs
• Time taken to complete CPO requirements
• Perceived quality of relationship with the court

The resulting case study selection is specified below. (In order to maintain the anonymity of the case study areas we are restricted in what we can say about each one):

• Case study Area 1 - Urban area
• Case study Area 2 - Mixed urban/rural
• Case study Area 3 - Urban area
• Case study Area 4 - Covers a wide geographical area and is mainly rural, although includes some larger towns

Recruitment and data collection in case study areas

1.27 Recruitment of professionals within CJSW, other practitioners who work alongside CJSW, offenders and Sheriffs were carried out via gatekeepers - a single liaison contact in each area local authority (or in the case of members of the judiciary - the Sheriff Principal). The gatekeeper contacted potential participants on behalf of the evaluation team to find out if they were happy to be invited to take part in an interview with ScotCen.

1.28 The purpose of the evaluation and why they had been invited to take part was explained to all potential participants. Verbal consent was recorded before commencing interviews with professionals, and written consent from offenders was obtained prior to commencing those interviews.

1.29 One-to-one interviews were carried out with offenders. Professionals were given the option of taking part in a one-to-one, paired or small group (n=3) interview.

1.30 Fieldwork for the case study areas was conducted in two phases so that time was built in to reflect on progress and identify any areas where further exploration was needed. Phase one commenced in October 2013 and continued until January 2014, and Phase two took place between May and July 2014.

1.31 The broad aims of the interviews were to build an understanding of how the reforms are operating in specific local contexts and criminal justice cultures. The main topics covered in the interviews with the different stakeholder groups are summarised below:

CJSW managers and staff

• Any challenges associated with the introduction of CPOs and how they are actually being delivered (CJSW managers only)
• Delivery of CPOs (CJSW staff only)
• Views about how offenders are responding to CPOs (CJSW staff only)
• Perceived and actual levels of compliance associated with different CPOs and reasons for this
• Views on the new CJSWR template and how effectively this is being used by CJSW staff
• Barriers to the effective use of the template and the generation of reports
• Perceptions of how Sheriffs are using CJSWRs and CPOs

Other practitioner groups (Due to the wide range of professionals in this group, some topic areas were covered to a greater or lesser extent depending on their role)

• How CPOs are actually being implemented on the ground
• Views about how offenders are responding to CPOs
• Views about the new CJSWR template and how effectively this is being used
• Any barriers to the effective use of the template and the generation of reports
• Perceived and actual levels of compliance associated with different CPOs and reasons for this
• Views about the operation of PASS

Sheriffs

• Decision-making around the use of CPOs and the factors behind it
• Barriers to Sheriffs making greater use of CPOs
• Perceptions of the extent to which offenders take seriously and comply with the requirements of CPOs, understand what is expected of them and are motivated to change their behaviour
• Views and use of CJSWRs

Offenders

• Views and experiences of being on a CPO (making comparisons, where possible, with previous orders)
• To gauge the extent to which individuals take seriously the requirements of CPOs, understand what is expected of them and are motivated to change their behaviour
• Barriers to understanding, engagement, compliance and behavioural change

1.32 Table A1.1 provides a breakdown of the number of participants interviewed by stakeholder group and case study area. Due to the low number of Sheriffs who agreed to take part in the evaluation in Case study Area 4, in agreement with the Scottish Government (and the relevant Sheriff Principal), two Sheriffs from a similar geographical area (to Area 4) were also interviewed. This ensured that our achieved sample included a wide range of views.

Table A1.1: Achieved sample

Stakeholder type	Area 1	Area 2	Area 3	Area 4	Total
CJSW managers	4	3	3	3	13
CJSW staff	6	6	7	6	25
Other practitioner groups	5	8	5	5	23
Sheriffs	5	3	4	4*	16
Offenders	5	5	5	5	20
Total	25	25	24	23	97

∗ The figure includes two Sheriffs that were recruited from an additional area (geographically similar to case study area 4).

Qualitative data management and analysis

1.33 Interviews were digitally recorded and transcribed, then coded using an analytical framework based on the key research questions for the evaluation and the key themes discussed by interviewees. This process of coding facilitates systematic analysis of the range of experiences and views expressed, similarities and differences between and within groups, and emergent explanations for particular experiences or opinions.

Analysis of national quantitative monitoring data

1.34 Analysis of the anonymised unit-level dataset underlying the aggregate statistics was carried out in SPSS to explore the outcomes for individual orders. A particular focus was given to examining the levels and patterns of completion and breach, overall and for different types (and combinations) of requirements.

Logistic regression analyses on CPO terminations

1.35 Logistic regression analyses were done using SPSS v18 and the CPO unit level dataset. All CPOs included in the dataset which were completed in the year to 31st March 2013 were included in the analyses. Binary outcome variables were used, in the first analysis with a value of 1 for all CPOs which were successfully completed (else 0), and in the second analysis, with a value of 1 for all CPOs which were revoked due to breach.

1.36 Fourteen independent variables were entered into each model, 12 binary variables and two treated as continuous (age, and the order number^[1]). The level of multicollinearity between these variables was low and therefore not sufficient to exclude any of the variables from the models.

1.37 Binary variables entered into the models are shown in tables A6.6 and A6.7 in Appendix 4. The odds ratio should be interpreted as the factor by which the odds of a CPO being successfully completed (in the first model), or revoked due to breach (in the second) are multiplied if that requirement or characteristic is present, compared with when it is not.

1.38 The significance value should not be interpreted as statistical significance, as the dataset includes all CPOs completed, not a sample.

Validation event

1.39 A research validation event was held at the end of June 2014. The purpose of this was to bring together a rich mix of participants - drawn from a range of professions and positions - to discuss the implementation of the CPOs, CJSWRs and PASS, the emerging key findings of the evaluation, and the extent to which these resonated with those professionals working within and outwith the case study areas.