Community Justice Outcomes, Performance and Improvement Framework

4 The 5 Step Approach to Evaluation

" The Outcomes, Performance and Accountability Working Group determined that the 5 Step Approach to Evaluation would be a useful component of the OPI Framework, allowing partners - both service providers and funders - to evaluate their services."

Background and purpose

In May 2015, the Scottish Government published two evaluation packs aimed at both service providers and funders who aim to promote behaviour change. One pack is specifically targeted at those who aim to reduce crime and reoffending. With the broad range of partners involved in community justice, both packs should be considered and drawn from.

"Designing and Evaluating Interventions to Reduce Crime and Reoffending" is available at: http://www.gov.scot/Publications/2016/05/3241

"Designing and Evaluating Behaviour Change Interventions" is available at: http://www.gov.scot/Publications/2016/05/1967

The Outcomes, Performance and Accountability Working Group determined that the 5 Step Approach to Evaluation would be a useful component of the OPI Framework, allowing partners - both service providers and funders - to evaluate their services. The Group viewed it to be both an essential part of the improvement journey and also a key element of strategic commissioning.

How should these be used?

For funders and partners, the packs aim to:

• Offer a strategic, evidence-based and outcomes-focused planning tool;
• Demonstrate the role you can play in promoting and enabling high quality evaluations from those you fund;
• Provide a focus on person-centred outcomes for service users;
• Offer guidance on how to assess evaluations from service providers and therefore direct funding to greatest effect.

For service providers, the packs aim to:

• Provide guidance on planning an evidence-based service with a 'built in' evaluation process;
• Provide guidance and resources for you to effectively assess, understand and demonstrate how well your service is working in relation to your aims;
• Offer an alternative to randomised control trials, using a 'logic model' approach to evaluation, which any service provider can use to evaluate any intervention, regardless of size;
• Provide a focus on person-centred outcomes for service users;
• Encourage continual review and improvement of services.

Other audiences

The packs are primarily aimed at funders, commissioners, partnerships and service providers with a focus on reducing the risk of crime and reoffending or behaviour change. However, they are likely to be relevant to others with an interest in effective evaluation (such as inspectorates and auditors) and the approach can easily be adapted for projects that do not primarily seek behaviour change.

The 5 step approach - A summary

Identify the problem

If your ultimate aim is to change people's behaviour, you need to be clear what it is you are trying to change and why there is currently a need for this to happen.

Review the evidence

What you intend to do should be grounded in the evidence of 'what works' and why. Service providers should review the available evidence in order to plan activities which can be expected to achieve the intended behaviour change. The evidence should guide what you do and help you to understand the process through which it should work.

Draw a logic model

A logic model is a diagram which shows, step-by-step, why the activities you plan should achieve your aims. The logic model forms the basis for evaluating the whole project - you are going to test whether these steps happened as you predicted.

Identify Indicators and monitor your model

Use the logic model to identify indicators (i.e. measurements or observations) that things actually happen as you predicted. You will need to collect data about your project FROM THE START on inputs, activities, users, short, medium and long-term outcomes.

Evaluate logic model

Analyse the data you've collected on your various indicators to evaluate how well your project worked for your various users. Report on whether your data suggests the logic model worked as planned. Be honest about any areas which were less effective. Use this to improve your service.

Figure 1 shows how the 5 step approach to evaluation fits with the rest of the OPI Framework, the vision from the National Strategy and the Local Context.

When considering the indicators at a service level, you will wish to focus on the impact on service users. This will require establishing baselines and distance-travelled measures.