Community justice: research and analysis

The 5 Step Approach to Evaluation

Evaluation will play a  fundamental part of the Community Justice OPI Framework especially for assessing person-centric outcomes. In April 2016 The Scottish Government published two evaluation packs aimed at both service providers and funders who aim to promote behaviour change.  One pack is specifically targeted at those who aim to reduce crime and reoffending.  With the broad range of partners involved in community justice, both packs should be considered and drawn from.

The packs includes a summary of ‘what works’, effective practice and desistance research, examples of logic models, links to research resources  and offers advice on how to collect and analyse data.

The 5 Step Approach:  Designing and Evaluating Interventions to Reduce Reoffending (full version)

The 5 Step Approach: Designing and Evaluating Behaviour Change Interventions (full version)

The Outcomes, Performance and Accountability Working Group determined that this tool would be a useful component of the OPI Framework, allowing partners – both service providers and funders – to evaluate their services.  The Group viewed it to be both an essential part of the improvement journey and also a key element of strategic commissioning.

How should these be used?

For funders and partners, the packs aim to:

• Offer a strategic, evidence-based and outcomes-focused planning tool.

• Demonstrate the role you can play in promoting and enabling high quality evaluations from those you fund.

• Offer guidance on how to assess evaluations from service providers and therefore direct funding to greatest effect.

For service providers, the packs aim to:

• Provide guidance on planning an evidence-based service with a 'built in'

  evaluation process.

• Provide guidance and resources for you to effectively assess, understand and

  demonstrate how well your service is working in relation to your aims.

• Offer an alternative to randomised control trials, using a 'logic model' approach to

  evaluation, which any service provider can use to evaluate any intervention,

  regardless of size.

• Encourage continual review and improvement of services.

Other audiences

The packs are primarily aimed at funders, commissioners, partnerships  and service providers with a focus on reducing the risk of crime and reoffending or behaviour change. However, they are likely to be relevant to others with an interest in effective evaluation (such as inspectorates and auditors) and the approach can easily be adapted for projects that do not primarily seek behaviour change.

The 5 step approach – A summary

• Identify the problem

If your ultimate aim is to change people’s behaviour, you need to be clear what it is you are trying to change and why there is currently a need for this to happen.

• Review the evidence

What you intend to do should be grounded in the evidence of ‘what works’ and why. Service providers should review the available evidence in order to plan activities which can be expected to achieve the intended behaviour change. The evidence should guide what you do and help you to understand the process through which it should work.

• Draw a logic model

A logic model is a diagram which shows, step-by-step, why the activities you plan should achieve your aims. The logic model forms the basis for evaluating the whole project – you are going to test whether these steps happened as you predicted.

• Identify Indicators and monitor your model

Use the logic model to identify indicators (i.e. measurements or observations) that things actually happen as you predicted. You will need to collect data about your project FROM THE START on inputs, activities, users, short, medium and long-term outcomes. Be proportionate when selecting what data to collect. Seek agreement on what is a priority to report on.

• Evaluate logic model

Analyse the data you’ve collected on your various indicators to evaluate how well your project worked for your various users. Report on whether your data suggests the logic model worked as planned. Be honest about any areas which were less effective. Use this information to improve your service.

If you are trying to support self-evaluation, Evaluation Support Scotland also have a wealth of useful evaluation resources on their website.

http://www.evaluationsupportscotland.org.uk/evaluation/