The 5 Step Approach to Evaluation: Designing and Evaluating Interventions to Reduce Reoffending
Updated guidance on how to use the 5 Step approach to design and evaluate criminal justice interventions.
Background: The tricky business of measuring impact in a messy world
How the 5-step approach came to be.
How was the pack developed?
Who developed this pack?
This pack has been developed by Scottish Government
researchers in Justice Analytical Services with the aim of
promoting and supporting the effective evaluation of criminal
justice interventions.
What's in the pack?
We describe a 5-step approach to designing and evaluating
interventions and services. It includes comprehensive summaries of
the reducing (re)offending evidence-base and subject-specific logic
models to support practitioners working in the field of crime and
desistance.
Is the 5-step approach being used in practice?
The approach described in this pack is already being used
widely by services, interventions and funders including
Third-Sector organisations, The Robertson Trust, the Scottish
Prison Service and the Scottish Government .
A Scottish approach to evaluation
Co-production
Our approach to evaluation enables funders and service
providers to work together in pursuit of their shared aims - to
improve outcomes for service users and communities. The 5-step
approach also engages with service users' views as a resource for
evaluation rather than seeing users solely as an object to be
measured. In fact, most complex social outcomes can ONLY be
achieved if we make a distinctive, yet joined-up contribution over
a sustained period of time.
Asset-based
The 5-step approach focuses on ways in which evaluation
is possible for services of any size, rather than
expecting all services to use an experimental evaluation method
which may not be appropriate or possible for smaller,
community-based organisations. The 5-step approach allows even the
smallest service to demonstrate the contribution they are making to
change.
An Improvement Culture
Evaluation enables improvement and even the most successful
service can always be developed further. Furthermore, with the
5-step approach, evaluation is an on-going process, not something
to be saved for last. This means that services can be continually
improved in order to best meet the needs of their users.
What are impact evaluations/RCTs
What is an impact evaluation or RCT?
An impact evaluation is designed to answer the specific
question 'did my project / service work?'
An impact evaluation or RCT is a much like a scientific experiment and is often considered the 'gold standard'. One group (the 'treatment' group) experience your intervention and one group (the control group) does not. You then compare the outcomes for both groups to see if your intervention made any difference. In other words, if you really want to know if you've made a difference, you need to know what would have happened if the same (or similar) users DIDN'T receive your service. This enables you to ATTRIBUTE changes in users to YOUR service rather than other factors like motivation, another programme or family influences.
The control group must either be selected completely at random or otherwise be very carefully selected to have very similar characteristics. Otherwise, you cannot be sure that any apparent differences in results at the end are not the result of differences that were already there at the start and therefore nothing to do with your intervention.
Cost benefit analysis
Some funders ask for a cost benefit analysis which is an economic assessment that places a monetary value on the costs and benefits of an intervention. It is another way to determine the value of an intervention and convince others that it has public value.
What is a cost-benefit analysis?
- CBA usually builds on a rigorous RCT and typically measures a wide range of outcomes.
- It usually measures the public benefits to society but may also consider benefits to individuals and families.
- It is both an art and a science especially when assigning monetary values to social benefits such as better parenting or securing accommodation .
- CBA allows for comparisons across interventions, policies, and other types of interventions.
What data do you need?
- Cost estimates may be based upon well-documented impacts (i.e. evidence-based interventions)
- Cost estimates may be based upon well-documented impacts and future projections upon these documented impacts
- Cost estimates may be based on undocumented assumptions that the intervention works and hypothetical projections or 'what if' analysis - but no hard data is available
Excellent guidance on CBA can be found here and Justice Analysts are happy to provide advice: http://whatworks.uwex.edu/attachment/whatworks_cost_benefit.pdfa
The difficulty with RCTs
You need a large sample
RCTs are only meaningful IF there is a large control group
with very SIMILAR CHARACTERISTICS to the users (the
counterfactual). Scotland is a relatively small nation and
behaviour change projects often target small or localised
populations, making them hard to carry out.
They can be expensive
Funding may be a barrier since RCTs may be expensive to run
and therefore not cost-effective as a means of evaluating
small-scale projects.
They can't tell you everything
RCTs can't tell you WHY something is effective (or
ineffective) so learning anything about HOW a project worked is
tricky using this method.
Do impact evaluations / RCTs even ask the right question? Contribution not attribution
Example - contribution to achieving outcomes
Like most social outcomes, reducing crime and reoffending are long term, complex goals and hard for any standalone service to achieve. For example, we know that many studies show that the most effective way to reduce reoffending is through a well-sequenced, holistic approach which can address multiple needs such as the provision of quality accommodation, positive relationships and recovery from drug abuse. The question then becomes….if these combined services achieve a reduction in reoffending, which service is responsible? The answer is, of course that all of them have a distinctive role in contributing towards achieving the outcome……so it follows that any evaluation of a single service should assess the extent of their particular contribution (defined by their own objectives). Impact evaluations (RCTs) put all the pressure on single services to 'prove' they have reduced reoffending rather than evaluate the contribution they are making.
An alternative to RCTs
A "middle ground" approach
Rather than carrying out a small RCT which might be
impractical and would only deliver meaningless results, or
unreliable anecdotal research we recommend that small-scale project
organisers carry out a
5-step approach to evaluation. This is summarised
in the following pages and detailed in the remainder of this
pack.
This approach to evaluation is practical for projects of any size but does rely on providers having a clear sense of what they're hoping to achieve or change and how they're going to get there - a theory of change. For this reason, using the 5-step approach to evaluation, we must begin at the planning stage.
What is evaluation really for?
Although doing evaluation requires the use of techniques and tools, bear in mind that its overall purpose is to help you (re) design services, ask questions, gather evidence, interpret the evidence, communicate important information about your service and take informed decisions. In this sense, the ability to ask relevant questions and clearly communicate the answers at the right time to the right people are key skills in making evaluation useful.
Contact
There is a problem
Thanks for your feedback