The 5 Step Approach to Evaluation: Designing and Evaluating Interventions to Reduce Reoffending

Step 2: Review the evidence

What is 'the evidence base?'

For the purpose of evaluation and planning, "the evidence base" refers to all available information which might guide what you do in pursuit of your particular aims.

Evidence can come in many different forms, including anecdotes or personal experience. However, when we talk about evidence in this context, we are usually talking about empirical evidence - that derived from purposively designed research studies. However, be aware that because the evidence base is derived from multiple studies, is not always obvious what will work. Studies can have contradictory findings or may ask different kinds of questions.

The following short guide, produced by the Centre for Research on Families and Relationships, Inspiring Scotland and Evaluation Support Scotland, explains what it means to say a programme is "evidence-based:"

http://www.crfr.ac.uk/assets/CRFR_ESS_IS_Evidence_base_briefing.pdf

In this pack, we have summarised some of the empirical and theoretical evidence for you.

Why review the evidence base?

Crucial for Planning
A well-designed project will be based on the available evidence about 'what works,' and what doesn't, in relation to your aims. Reviewing the evidence base as part of the planning process will give you the best chance of achieving change in your users.

Crucial for Evaluation
However, following the 5-step process, reviewing the evidence is also a crucial phase in the evaluation process. Assuming that an experimental design (i.e. RCT) has not been possible, the 5-step process allows you to evaluate the project by assessing the quality of evidence behind a project's theory of change - what reason do you have to believe that the project's activities should lead to the outcomes envisaged? In addition, it is important that you have a clear idea of the causal processes which underlie the logic of your project so you can plan how you will gather evidence about whether or not they actually took place (see step 4).

Sources of evidence

Research Evidence
Including results of randomised control trials (RCTs), surveys and qualitative studies (e.g. interviews or focus groups). Systematic, literature or evidence reviews synthesise research evidence on a particular topic.

Evidence from Prior Evaluation
If your service (or a similar one) has already been running for a period of time, your own previous evaluations may provide evidence as to whether the approach works or not, how and for whom.

Anecdotal Evidence
Over years of working in a particular field, your own experiences and those you hear about from others can be a further source of evidence. However, whilst valuable, it is important to remember that such evidence may be particularly subject to bias since it will not have been collected systematically.

Research and/or evaluation evidence should be used where available.
However, there is no a simple answer to what counts as "good evidence." It depends on the question you are trying to answer. For more detail see these short videos from the Alliance for Useful Evidence: http://www.alliance4usefulevidence.org/

For best results use a range of evidence

To draw the most robust conclusions about 'what works,' and why, you should take account of evidence produced through a range of methods.

• Quantitative studies (including the results of RCTs and impact evaluations) might help you to establish what usually works and for whom.
• Qualitative work (e.g. interviews with users who 'succeed' and 'fail' and/or with practitioners) might help you to understand the processes through which interventions work or don't work and consider why barriers may exist to achieving your aims.

TIP! If you are short on time and resources, systematic and/or literature reviews are an excellent source of evidence. They often analyse both quantitative and qualitative studies on a particular topic and should do the work of summarising all this evidence for you.

Finding evidence

When time and resources, are limited, evidence reviews (also called systematic reviews or literature reviews) are a realistic solution - enabling an overview of the evidence in a relatively short time.

Online databases and archives are the most convenient means through which to locate evidence reviews. The following provides a summary of the evidence on reducing crime and reoffending and links to full reviews. However, the following databases can be of general help in locating relevant evidence:

Search academic databases:

http://www.mendeley.com/dashboard/
http://scholar.google.co.uk/

Search government archives:

http://www.gov.scot/Publications/Recent
https ://www.gov.uk/government/publications

TIP! Try searching for "evidence/literature/systematic review" + your behaviour change aim (i.e. "reoffending", "parenting", "collective efficacy", "impulse control" or "motivation").

What does the evidence say?

Using evidence is an important INPUT to your service or intervention. Consistent results that show that certain activities or approaches help offenders to desist should work but only if you deliver the service to a high standard. For example, evidence shows that CBT programmes (which as a strong evidence-base for higher risk men) delivered in the community are less effective at achieving outcomes than prison-based interventions due to poorer delivery and less focus on quality assurance.

To make the results from robust studies more accessible, the following pages summarise some of the key evidence from the 'what works', desistance and best practice literature. Links to full reviews are also included.

Reducing Crime

The full evidence review 'What Works to Reduce Crime' can be found here:
http://www.gov.scot/Publications/2014/10/2518

Tackle the root causes of crime

• Low self control in children is linked to offending
• Parenting programmes are effective in improving self-control
• Social skills training designed to improve emotional intelligence, may help reduce delinquent behaviour
• Offending is linked to abuse, neglect, exposure to domestic violence and parental substance misuse
• Quality of care of children and young people and protection from abuse and neglect are key
• Identification of abuse and neglect at the earliest stage

Address key social factors

• Retain an attachment to school
  • Staying at school is a protective factor
  • Exclusion could be a significant risk factor
  • Behavioural boundary setting is key
  • Diversion activities e.g. sport play an important role
  • Enable children to realise their potential
• Holistic employment programmes that also provide social and educational support can be effective
• Minimise the impact of criminal justice sanctions on family bonds
• Restrict access to alcohol
• Tackle drug and alcohol abuse (improving social control through effective parenting may play a part)

Deterrence and Changing the Situation

• Detection and punishment used alone are ineffective.
• People are more likely to comply with rules if they are perceived to be fair and legitimate not because they fear punishment.
• Tackling areas of 'concentrated disadvantage' is the most important step to take to reduce crime
• People offend less when communities look after their areas - 'collective efficacy' has been found to be an important factor in reducing crime.
• The certainty of punishment (increasing the likelihood of detection) is more effective as a deterrent than the severity of punishment

What factors are related to reoffending?

Criminal history (previous convictions), age, gender, disposal type, index crime, age and sentence length are all strong static predictors of reoffending. Although they are unable to be changed by interventions, this information can be used to target intervention resources towards medium-high risk offenders who require the most support to desist.

If an intervention aims to reduce reoffending, it needs to target dynamic criminogenic needs. These are characteristics that have been found by a number of research studies to be associated with reoffending.

Criminal attitudes and values	Anti-social lifestyle
Criminal peers	Poor problem-solving/impulsive behaviour
Lack of employment, volunteering or leisure activities	Homelessness
Substance misuse	Low motivation, lack of hope and low self-efficacy

Offenders usually have multiple needs and thus interventions that tackle a range of problems will be more effective. The extent that needs have been addressed can be defined and measured as intermediate outcomes (short and medium term)

Women's criminogenic needs overlap with men's although women prisoners are more likely to also experience non-criminogenic needs such as depression, anxiety and learning difficulties. They also are more likely to face accommodation problems, financial crisis, trauma arising from abuse and self-harming behaviour than male prisoners. The quality of relationships with family and friends is also more strongly linked to reoffending in women.

Desired intermediate outcomes

Desired intermediate outcomes based on criminogenic needs

What works evidence matrix

Reducing Reoffending - 'What works' evidence

The following table describes the findings from the international 'what works' evidence on reducing reoffending.

The results are generated by quantitative randomised controlled trials of programmes and interventions.

Links to full evidence reviews can be found later in this section and at the end of this pack.

Risks to reoffending (evidence-based)	Indicator that the risk is present in an individual	Desired intermediate outcomes	Approaches that work to address the risk	Promising approaches but more evidence needed
Limited social skills, problem-solving skills and poor emotion management	Impulsive, pleasure-seeking, irritable, poor recognition of problems, poor problem-solving skills, poor social skills, lack of awareness of consequences of actions	Skills in problem-solving and perspective taking Emotion management skills	Structured CBT programmes such as cognitive skills training Restorative Justice Conferencing	No evidence identified but trained supervisors/mentors could help offenders engage in CBT programmes
Criminal attitudes	Rationalisations for crime, negative attitudes towards the law, negative attitudes to supervision and to society as a whole	Development of pro-social attitudes and a non-criminal identity	Structured CBT programmes such as cognitive skills training and cognitive restructuring techniques	Pro-social modelling, positive supervisor/mentor and staff interactions Supervisors/mentors challenge anti-social attitudes
Criminal friends	Criminal friends, isolation from pro-social others, easily influenced by criminal associates	Criminal friends replaced by prosocial friends and associates	More evidence needed	Mentoring, circles of support and accountability (for sex offenders)
Lack of positive recreation or leisure activities /anti-social lifestyle	Lack of involvement and satisfaction in prosocial recreational activities. Regular activities encourage offending, recklessness and risk taking behaviours	Participation in pro-social recreational activities, sense of reward form pro-social recreation and sustained involvement in pro-social lifestyle	More evidence needed	No evidence identified but supervisors/mentors could aim to engage offenders in pro-social activities
Drug misuse	Uses drugs, injects drugs, unmotivated to tackle drug misuse, drug use and obtaining drugs a major occupation	Substance use reduced or stopped	CBT programmes, detox, opiate substitution therapy (for acquisitive opiate-addicted offenders) psycho-social support to maintain abstinence,12 step programmes, structured, therapeutic communities for drug misuse.	No evidence identified but supervisors/mentors could help offenders engage with drug programmes
Alcohol misuse	Binge drinking, long term alcohol misuse, violent when intoxicated	Reduced alcohol use or stopped drinking, reduced through disturbances promoted by drinking	More evidence needed	Supervisors/Mentors could help offenders engage with 'promising' programmes which address the interaction between alcohol and violence
Dysfunctional family relationships	Poor family relationships, no current relationship, no previous experience of close relationships, manipulative lifestyle	Conflict reduced, positive relationships, enhanced warmth and caring, reintegration into (non-criminal) social and family groups Strengthened family ties improving family and intimate relationships, improving parenting behaviours and increasing acceptance into communities and social networks	Therapeutic approaches for young adult offenders that involve the family	No evidence identified but supervisors/mentors could help young offenders engage with therapeutic approaches Supervisors/mentors could also help offenders engage in 'promising' approaches, namely relationship coaching interventions and they could also facilitate family visits to prison
Unemployment	Poor performance, low satisfaction in work, lack of work-related skills, poor attitude to employment, lack of qualifications	Work skills, good interpersonal relationships at work, reward and satisfaction at work Long term employment and increased employment skills	Employment-focussed programmes in which offenders can secure real jobs they enjoy.	Gaining work related qualifications, gaining employability skills Work related support/mentoring
Homelessness	No fixed abode or transient	Finding and keeping suitable housing	More evidence needed	No evidence identified but supervisors/mentors could assist homeless offenders find homes and retain them
Low motivation and/or self-efficacy	Unmotivated to desist and/or the belief that they do not possess the skills to desist from crime	Offenders are highly motivated to engage with supervisors and interventions and offenders are confident they have the skills to desist from crime	Offenders build positive trusting relationships with skilled, empathetic and flexible mentors, collaborative goal-setting	No evidence identified

Effective practice - interventions

• Short term sentences are not effective at reducing reoffending
• Respectful, skilled, participatory and flexible contact with a supervisor can trigger positive changes in offenders.
• The effectiveness of prison-based interventions is enhanced when aftercare support is provided following release.
• Holistic interventions that target offenders' multiple needs and involve work with offenders' families and the wider community (e.g. employers) are more likely to be effective at reducing reoffending.
• Cognitive Behaviour Therapy (CBT) programmes can reduce reoffending by 8-10% especially for violent and higher risk offenders but quality assurance is key to success. The evidence for CBT for women is mixed.
• Interventions for women offenders are more likely to be successful if they target financial and family needs.
• Important to know where offenders are on their 'journey'. Cognitive approaches may be more effective with those more resistant to change.
• Random drug testing combined with treatment and swift and certain (but not severe) responses to breach
• Prison-based drug interventions such as therapeutic communities, psycho-social (CBT) and abstinence based approaches. Crucial to ensure quick access and support for as long as is needed.
• High intensity drug programmes are more effective than low intensity ones
• Specific restorative justice models, especially with property and violent offenders (where there is a clear identifiable victim) and a plan to support the offender to reintegrate
• Face-to-face conferencing model of restorative justice is can be effective at reducing reoffending and highly cost-effective. One UK scheme found reductions in reoffending of 14% when this model was used compared with a matched group.
• Stable and quality employment protects against reoffending especially is accompanied with other forms of support.
• One-size does not fit all - Risk Needs Responsivity (RNR) programmes have better outcomes
• Intervention goals should match the needs of clients
• Programme integrity and integration is crucial and emphasis on quality assurance
• Focusing on an single outcome (e.g. employment) increases likelihood of failure creating feelings of despondency
• Well integrated multi-level interventions (but has negatives for users due to being passed from pillar to post)
• Well-sequenced interventions
• See progress as a series of small steps
• Sharing case management information between partners - more effective and efficient support (HMP Peterborough)

Effective practice - host organisations

The following organisational factors have been found to be important for interventions to work effectively

• The organisation that hosts the intervention has a history of adopting new initiatives
• Some decentralisation to allow a flexible approach to problematic areas
• No task or organisational conflict particularly at managerial level
• Low staff turnover to ensure stability and consistency
• Formal programme of training (e.g. SEED) and instruction
• Formal links with educational or academic institutions
• Funding should originate from the host organisation to increase accountability
• Funds should be sustainable
• Enhance professional credentials
• Focus on achieving clear intermediate goals and don't over-reach
• Evidence-base is widely understood and embedded into programmes

Effective practice - practitioner skills

The following practitioner skills and techniques have been found to be important for interventions to work effectively

• Excellent interpersonal skills so workers can adapt to individual diversity and adapt styles
• Open, caring, warm enthusiastic and empathetic, understanding, listening, giving
• Provide problem-solving advice and practical help
• Practitioners who are persistent and demand change are seen as showing genuine interest and concern
• Practitioners still need to take the lead and use advocacy skills to resolve practical problems and remove obstacles
• Modelling/praising non-criminal behaviour and disapproving of criminal behaviour
• Users tend not to value general 'how are you doing?' conversations
• Time to spend with people to support effectively
• Involve offenders to develop release and treatment plans - helps to motivate and develop a sense of agency and self-determination
• Goals must join up and both parties must bring effort
• Know how to predict and respond to relapse
• Formal training and courses for practitioners is absolutely key and they should be supervised, supported and motivated
• Practitioners should be involved in designing the programme

More evidence on effective practice and 'what works' to reduce reoffending

Here are the links to the full reviews of 'what works' to reduce crime and reoffending

What Works to Reduce Crime
http://www.scotland.gov.uk/Publications/2014/10/2518

Strengthening Transnational Approaches to Reducing Reoffending - University of Cambridge
http://www.cepprobation.org/uploaded_files/Rep%20STARR%20ENG.pdf

Reducing Reoffending Evidence Review - Justice Analytical Services
http://www.scotland.gov.uk/Resource/0038/00385880.pdf

Transforming rehabilitation - A summary of evidence on reducing reoffending - Ministry of Justice
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/243718/evidence-reduce-reoffending.pdf

The quality of probation supervision - A literature review
https://www.shef.ac.uk/polopoly_fs/1.159010!/file/QualityofProbationSupervision.pdf

Effective practice - throughcare

Although there is very limited evidence on 'outcomes', an international review found that features of effective throughcare were perceived to be based on the following features, according to users and practitioners and some research studies.

Targeting the 'right' people

• Key workers should be consistent to enable trusting and flexible relationships to develop
• Pre-release plans are crucial and prisoners should be involved in devising these plans.
• Practitioners believed that services need to be available and accessible at the point when a service user is ready to make changes in their lives.
• The intensity of supervision needs to be relative to risk of reoffending - the greater the risk, the more intense the support should be.

Effective partnership working

• Multi-agency working that takes a holistic approach is important. Agencies should have distinct functions, shared objectives, adequate resources and a strong working relationship
• A liaison officer can help agencies work together
• Opportunities should be created to share ideas and to understand functions and remits of the agencies involved e.g. Link Centres
• The statutory monitoring role and support role should be separated to avoid tensions arising.

Addressing release gaps

• There should be continuity in service provision on release, especially for short term prisoners and those on remand.
• Early contact is crucial and should start at the point of sentencing.
• Day release to go to jobs and temporary accommodation are important for motivating users and avoiding disruption on release. Access to welfare provision and housing is crucial but can be extremely difficult due to bureaucratic barriers.
• Services should be NEEDS LED rather than service led.
• Short-term funding and heavy case-loads can create fragmentation, instability and decrease the quality of services.

Staff selection, remit and skills

• A strong relationship between users and providers is key to changing behaviour but not enough in isolation. Needs have to be addressed too.
• Mentors may be an effective way to support service users but only as part of a wider network of services.
• Interpersonal skills and the ability to be flexible as well as practical was important according to practitioners.
• Service users response better to workers who identify individual strengths and positive features.
• Practical issues should be addressed before more complex needs as this maintains motivation.
• Addressing practical basic needs is important but is not sufficient to trigger change. More complex needs such as attitudes, social skills and emotion management are also important to address.
• Desistance is more likely if throughcare includes work with families and forges links with the wider community e.g. employers.

Effective practice - mentoring

• Mentoring schemes should have robust working links with existing and developing interventions in their area of operation. Where feasible, these links should be underpinned by written protocols and/or care pathway agreements.
• The mentoring schemes should sit alongside cognitive restructuring treatment (changing destructive and anti-social attitudes) and cognitive skills (social and problem-skills) training and behaviour modification. These interventions have the strongest evidence in reducing reoffending because they internalise change in the offender so they acquire the skills to desist after the external support is withdrawn.
• The mentoring schemes should be designed around clear objectives and intended outcomes. These should be the outcomes which the evidence suggests mentoring can help offenders to achieve
• Mentors and mentees should be carefully matched.
• Mentors should undergo at least 20 hours of training prior to matching, and should be provided with ongoing training.
• To underpin the development of a strong, meaningful, supportive relationship between mentors and mentees they should meet at least once a week, and the mentoring relationship should last for at least six months.
• Mentoring projects should have a quality assurance system, a strong structure and overall coordination of the programme. A coordinator should help with selecting appropriate mentors, ensuring that mentors receive training, providing on-going monitoring of mentoring relationships and monitoring effective networks of organisations.

Effective practice - women offenders

• Relationships with others have a stronger influence on women's offending than on men so they are key to desistance: Women desisters say they have strong social support from others and employ strategies for avoiding situations which could lead them back into offending.
• Interventions should be delivered by interpersonally skilled staff who build a consistent and trusting relationship with offenders.
• Interventions are most effective if they start in prison and continue when women are released, address criminogenic and non-criminogenic needs simultaneously and are well targeted and sequenced.
• To reduce reoffending, interventions should help women improve their financial situation, secure suitable and safe housing, establish loving bonds with children, tackle drug abuse in a residential setting and help women form positive relationships.
• Women offenders value help to solve practical problems such as accommodation, childcare and welfare benefits. These short-term needs may have to be addressed before women are ready to engage with interventions or address longer term needs such as education or employment.
• Substance misuse has a stronger relationship with reoffending in women, and women are more likely offend to support others' drug misuse as well as their own. However, some research has shown that recreational and occasional drug use are not strong predictors of reoffending in women, which suggests that intensive interventions should be targeted at drug use that is criminogenic
• Some social conditions that promote desistance in women are outside the control of some formal interventions - maturation, support from family and friends and establishing healthy personal relationships.
• Most of the studies on women's needs and effective interventions derive from the US or Canada so this may limit how transferable the studies are to offenders in the UK as offenders in these countries may have different demographic characteristics.

Effective practice - desistance theory

Some research is beginning to shed light on the process of desistance from crime, and (to a lesser extent) on the potential role of criminal justice social work supervision in facilitating that process. Although there has been relatively little empirical research on the latter subject, a body of theorising has emerged which, follows the idea that probation practice should become 'desistance-focused' seeks to interpret desistance research for practice. Reviewing the available research , these efforts to interpret desistance research for practice tend to stress (albeit to varying degrees) eight central themes:

• Desistance is likely to involve lapses and relapses. There is value, therefore, in criminal justice supervision being realistic about these difficulties and to find ways to manage setbacks and difficulties constructively. It may take considerable time for supervision and support to exercise a positive effect.
• Since desistance is an inherently individualised and subjective process, approaches to criminal justice social work supervision must accommodate and exploit issues of identity and diversity. One-size-fits-all interventions will not work .
• The development and maintenance not just of motivation but also of hope become key tasks for criminal justice social workers .
• Desistance can only be understood within the context of human relationships; not just relationships between workers and offenders (though these matter a great deal) but also between offenders and those who matter to them.
• Although the focus is often on offenders' risks and needs, they also have strengths and resources that they can use to overcome obstacles to desistance - both personal strengths and resources, and strengths and resources in their social networks. Supporting and developing these capacities can be a useful dimension of criminal justice social work.
• Since desistance is in part about discovering self-efficacy or agency, interventions are most likely to be effective where they encourage and respect self-determination; this means working with offenders not on them.
• Interventions based only on developing the capacities and skills of people who have offended (human capital) will not be enough. Probation also needs to work on developing social capital, opportunities to apply these skills, or to practice newly forming identities (such as 'worker' or 'father').

Writing an evidence-based proposal

A service that is underpinned by a clear justification for why it is needed and by a strong evidence-base is more likely to be effective.

The next page shows a short example of how to show that your service or intervention is grounded in the findings from a number of robust studies.

Appendix 1 gives an example of a fuller evidence-based proposal for an intervention.

TIP! A similar document could be provided to a funder. The extent to which an intervention is based on evidence could help assess the strength of a proposal for funding.

Intervention (what are we doing?)	Evidence (why are we doing this?)
This project aims to increase support and interventions for short term prisoners released from prison.	Several international reviews, drawing on randomised controlled trials and qualitative research have demonstrated the positive impact of one-to-support from highly skilled practitoners and needs-led interventions on desistance from crime (see Scottish Government Literature Review, 2011 and 2015). There is also some evidence from impact evaluations that a lack of pre-release planning and poor access to employment, support and accommodation after leaving prison leads to reoffending (Scottish Govt review 2015).
The project is targeted at male short term prisoners.	Although male prisoners are at a higher risk of being reconvicted than women and longer-term prisoners, they are less likely to take up voluntary throughcare (see Throughcare review 2012).
Contact by a fully trained throughcare officer will be made 1 month after sentencing. They will spend the first month building a relationship with prisoners before the first of 3 needs assessments are conducted and 'whole person' pre-release plan is developed.	A systematic review of the international literature on throughcare and resettlement highlighted that needs assessments are higher quality if practitioners give prisoners time to settle into prison, build a trusting relationship and if the needs assessment considers the whole person including family and influences. Research with offenders also shows that trained practioners who use a flexible approach and strong interpersonal skills are able to keep offenders motivated and engaged.
Practitioners will accompany prisoners through the gate to link them with services and for 3 months after release.	A Canadian review and the international review mentioned above highlighted the need for practitioners to connect prisoners with services once they return to the community. The highest risk of reoffending is 3 months after release from custody (Howard, MoJ 2011).