Moving Forward Making Changes: evaluation of a group-based treatment for sex offenders - summary

Moving Forward: Making Changes - An Evaluation of a Group-based Treatment Programme for Sex Offenders

This paper summarises the key findings and policy messages from an evaluation of Moving Forward: Making Changes, an intensive group-based treatment programme for sex offenders.

Key Findings

• The report provides evidence that most men who completed the programme (69%) were rated as having a lower risk score by the end of the programme. Among the men rated as ‘high risk’ at the start of MF: MC, 53% remained high risk at the end of the programme, 38% had moved into the ‘moderate’ category and 9% were rated low risk. Among those initially rated as ‘moderate’ risk, 72% remained in this category, but 26% moved into the ‘low risk’ category, while a very a small number (3%) were rated ‘high risk’ at the end of the programme. However, there are serious question-marks over the reliability of some of the risk data and without a control group we cannot be certain that these shifts in risk scores were caused by MF: MC or by other factors.
• Alongside reduced assessed risk levels, other perceived benefits identified by staff and men interviewed for this evaluation included improved ability to sustain healthy social relationships, regulate their emotions, cooperate with supervision, and understand and change problematic attitudes. 85% of men who completed an exit survey on leaving MF: MC said they thought it would stop them reoffending in a similar manner.
• Overall, the programme is being delivered broadly as intended with respect to length of group sessions, group size, staff roles, staff participation in mandatory MF: MC training, and practitioner supervision and support. However, in the community, the programme is typically only being delivered once a week rather than twice as recommended in the manual. This was primarily attributed to a lack of staff resourcing.
• The main areas identified for improvement were: staff resources; access to psychological support; the clarity of some aspects of the MFMC manuals; processes for assessing programme suitability; the programme’s ability to meet the needs of all types of men who commit sexual offences; and data collection and use.
• At least 911 men were recorded as starting MF: MC (81% in the community) although this is likely to be an under-estimate due to data collection issues. Participants generally met the target group criteria, with 98% assessed (at the start) as moderate or high risk of future sexual offending . However , some concerns were raised over whether the current assessment process is always identifying those most suited to the programme.

Background

MF: MC was designed by the Scottish Prison Service and the Scottish Government’s Community Justice Operational Practice Unit . It was introduced in 2014, following accreditation by the Scottish Advisory Panel on Offender Rehabilitation ( SAPOR). It is aimed at adult (18 and over) male sexual offenders assessed as medium-high risk of re-offending. MF: MC is delivered in 4 prisons and in the community (currently 11 sites). MF: MC’s implementation and delivery is supported by a set of manuals, covering the programme theory, structure, content, management requirements, and framework for collecting evaluation data.

Evaluation aims

The Scottish Government commissioned Ipsos MORI Scotland to conduct an evaluation of MF: MC to inform considerations on SAPOR’s reaccreditation of the programme in 2018. The evaluation was intended to address three main questions:

1. How does the programme work in practice, from referral to exit, and to what extent does the programme follow the manuals? Is programme integrity maintained?
2. Are there difficulties with delivering any aspects of the programme and are there improvements that could be made?
3. What are the main outcome measures and to what extent have outcomes been realised?

Methods

The evaluation used a mixed method approach to address the questions above. Data routinely entered into the MF: MC IT system was analysed. In addition, qualitative interviews were conducted with: 19 participants across 5 case study sites (2 custody and 3 community); 18 Treatment Managers across the 15 sites; 4 Case Managers; 3 stakeholders from Police Scotland; and 1 psychologist working with community-based MF: MC teams. The research also drew on evidence from a half-day workshop with 23 Practitioners from 12 MF: MC sites.

Main findings

Programme outcomes and improving impact

• There are significant limitations to the data available to quantify the impact of MF: MC, including the lack of a control group (recruiting a control group was not feasible within the evaluation timescale and resources, and would also have raised considerable ethical challenges), perceived issues around inter-rater reliability, concerns around the validity of some measures, and high levels of missing data.
• However, the monitoring data does indicate a number of positive changes among participants (though these cannot be conclusively attributed to MF: MC), including:
  • A reduction in risk scores over the course of the programme, and
  • Positive changes in scores measuring a range of psychological traits potentially associated with risk of re-offending.
• Overall staff felt that MF: MC had a ‘better chance’ of being effective compared with previous sex offender interventions, as they viewed its design as being more evidence-based. While this evaluation cannot robustly assess whether all elements of MF: MC are best practice, in general the programme design appears to be informed by the current best evidence on ‘what works’ in terms of appropriate treatment targets and approaches to working with sex offenders.
• While staff and stakeholders gave examples where they believed MF: MC had impacted positively on participants, there was also a strong feeling among staff that it was very difficult for the current programme to fully meet the diverse presenting needs of all participants.
• Interviews for this evaluation (particularly with Treatment Managers) identified a range of suggestions for improvements aimed at enhancing MF: MC’s impact, including:
  • providing more external expertise to support delivery
  • assigning national leads on specific approaches and issues to ensure the programme is kept continually up to date in terms of best practice
  • providing guidance on ‘over-treatment’ (where offenders have attended programmes numerous times with limited evidence of progress), and
  • considering the feasibility of establishing separate groups for men with lower cognitive functioning.

Improving programme delivery

• Overall, MF: MC is being delivered broadly in line with programme design with respect to: length of group sessions; group size; staff roles; staff participation in mandatory MF: MC training; and Practitioner supervision and support.
• However, the evaluation identifies a number of concerns around staff resourcing. Most community teams are currently unable to deliver MF: MC at the recommended intensity due to staffing constraints. Both Community and Prison-based Treatment Managers identified significant challenges covering leave or finding time for staff development. The role of psychological support for delivery within the community, and any inconsistencies in accessing it, also requires clairfication.
• The Programme Manual was viewed by staff involved in delivery as ‘a little vague’. There is also a lack of clarity about the level of deviation from the manuals that is permissible. In practice, sites were adapting content and delivery in a range of ways. While these adaptations were aimed at improving delivery, they introduce scope for inadvertent deviation from the programme design. The MF: MC manuals should be reviewed, taking account of the various issues and suggestions for improvement identified in the evaluation report.
• Consideration should also be given to developing an MF: MC ‘knowledge hub’, where additional (approved) materials to support delivery can be shared and added to as new evidence on ‘what works’ in treating sex offenders emerges.
• While MF: MC appears, for the most part, to be successfully targeting men rated as ‘high’ or ‘moderate’ risk, some concerns were raised over whether the assessment process is always identifying those most suited to the programme. Guidance on assessing internet offenders and deniers, in particular, should be updated in light of emerging evidence on managing and treating these groups.
• Sites varied in whether they offered ongoing support to participants after the group stage of the programme. This suggests that guidance on ending the programme should be clear, and resourcing sufficient to ensure a consistent and appropriate approach to post-programme support.

Improving monitoring and future evaluation

• The monitoring data currently available for MF: MC has some significant weaknesses in terms of the robustness of the outcome measures included, the IT system’s perceived usability for recording key data, and the timeliness of data collection and entry. Suggestions for improvement include:
  • A systematic review of the outcome measures included within MF: MC, taking account of concerns raised in this evaluation about: the utility and appropriateness of the current psychometric battery; the reliance on self-reported data; and completion of the ‘significant others’ questionnaire.
  • Considering whether any further guidance and training is required around the completion of SA07 for MF: MC specifically, given the concerns about inter-rater reliability and accuracy raised by interviewees.
  • Reviewing the structure and content of the MF: MC IT system with a view to simplifying data entry, clarifying data outputs, and enhancing data usability for both evaluators and MF: MC teams.
• Any changes resulting from this review process should be supported by refreshed guidance and training on how monitoring data should be entered and used by MF: MC teams, and by the Scottish Government, SPS and any external evaluators.
• In addition to reviewing and improving the current monitoring dataset, there should be consideration of how to monitor longer-term outcomes and whether a control group can be established. As there is a relative dearth of evidence on the effectiveness of sex offender interventions, there is an argument for assessing the feasibility of an experimental or quasi-experimental evaluation of MF: MC. However, an experimental approach would require sufficently large sample sizes to be robust and men would have to be randomly assigned into the programme or into a control group. Alternatively, a large matched sample of offenders who were not on the programme would have to be identified. Both approaches would be challenging to implement in practice.

Conclusions

Overall, the evaluation indicates that MF: MC is being delivered with fidelity to the Good Lives Model on which it is based. Sessions are focused on helping participants re-evaluate their life goals and develop skills to achieve those goals without harming others. It also appears to be delivered in an engaging and responsive manner, with staff tailoring the content where necessary to address individual needs. Acknowledging the concerns over the reliabiity of the risk data and the limitations in being able to quantify the impact of MF: MC, the available data is generally quite positive in terms of perceived impact. Overall, level of risk as scored by Stable 2007 decreased for a majority of men over the course of the programme, while a significant proportion of men also saw positive changes in scores measuring psychological traits potentially associated with risk of reoffending. Men expressed positive views about the programme’s impact, and staff and Case Managers reported that feedback from others and their own observations indicated it was having a positive impact across a range of risk factors. However, the views of some stakeholders struck a slightly more sceptical note, reinforcing the need to reconsider what data is collected on outcomes going forward.

Suggestions for improvement to maximise the potential impact of MF: MC and to assess this more effectively in the future centre around: programme scope and resourcing – including the expertise required to deliver MF: MC, the overall level of resourcing, and whether there is a need to supplement MF: MC with alternative or additional input for groups of men whose needs are not currently perceived to be fully met; improvements to the clarity and content of programme manuals; and improvements to the validity, quality and relevance of monitoring and evaluation.