Designing and Evaluating Interventions to Reduce Crime and Reoffending Summary

Step 4: Monitor your logic model

Identify indicators to monitor progress of the logic model

Use the logic model of identify indicators

Once the logic model is completed, you need to figure out how you will be able to tell if your model works as predicted, or not. Todo this, you should:

1. Devise 'evaluation questions' – specific questions that you need to answer in order to test whether the model is working as predicted.

2. Identify specific indicators (measures or signals of some kind) that can answer these questions and therefore provide evidence that your model is or isn’t working as expected.

See next page for an example of how this works in relation to a specific logic model.

Warning! Measuring outcome indicators for national strategic programmes/reform is not easy. The reality of collecting outcomes data from thousands of individuals who flow in and out of services and systems across the country can be prohibitively difficult. The following questions need to be addressed:

• What outcomes data is relevant to measuring performance?
• How is the data going to be collected and how frequently?
• Who is responsible for collecting the data  and analysing it?
• Can data be collected and analysed consistently across a range of areas?
• Are outcomes completely within the sphere of influence of the organisation(s) who is being evaluated/performance managed or are outcomes heavily influenced by external factors?

If it is not feasible to collect outcomes data, then collecting information on the delivery of activities and outputs as per the logic model is advised.

Identify indicators - Parenting skills example

Use the logic model to set evaluation questions to identify indicators. This will guide the collection of data.

Data collection principles

Now you’ve identified your indicators, you need to decide on a way of measuring or observing these things. There are lots of different methods you can use to collect this data (see 'data collection methods') but some basic principles to observe are:

• Collect data for every stage of your logic model, including resources and activities as well as outputs.
   
• Collect data at a unit level (i.e. about every user of the service) and at an aggregate level (i.e. about the service as a whole). Unit level data can be very useful as it can tell you who the service is working for and who it isn’t, and you can follow the progress of individuals over time. It can also be combined to give you overall data about your service. But remember, if you only collect aggregate data you will not be able to disaggregate it and therefore collect evidence about particular individuals.
   
• Follow users through the project. You should collect data about users at the very start, throughout and ideally beyond completion of the project. This will enable you to evidence whether users have changed, in terms of their attitudes, behaviour or knowledge.

TIP Focus on finding indicators that measure the quality of what people do (activities) — unless people deliver a service to a high standard, it is unlikely that outcomes will materialise. Also, if outcomes are hard to measure, focus on quality assurance indictors.

• Make use of numbers and stories . Collect qualitative as well as quantitative evidence. Averages and percentages can help you to assess overall trends and patterns in outcomes for service users. Talking to people, hearing about the views and experience of users and stakeholders will help you to explain these patterns.
   
• Don’t reinvent the wheel. Standardised and validated (pre-tested) tools are available to measure such things as self-esteem, wellbeing and employability. Using these will enhance the reliability of your evidence and save you valuable time.  Freely available tools are detailed here:
  http://www.clinks.org/sites/default/files/UsingOffShelfToolstoMeasureChange.pdf
  http://www.evaluationsupportscotland.org.uk/resources/tools/
  http://inspiringimpact.org/resources/ (follow link to “List of Measurement Tools and Systems”)
   
• Be realistic and proportionate. Expensive and/or experimental projects should collect greater amounts of data than well-evidenced and established, cheaper projects. You might want to give questionnaires to all users but it would usually be sensible to carry out in-depth interviews with just a smaller sample of your users.

Data collection methods

Various methods can be used to collect data in relation to your evaluation questions. Data can be collected from service users, staff or outside agencies. Not all methods will be suitable for all projects. Evaluation Support Scotland have produced excellent guidance on using different approaches.

• Using Interviews and Questionnaires http://www.evaluationsupportscotland.org.uk/resources/129/
• Visual Approaches http://www.evaluationsupportscotland.org.uk/resources/130/
• Using Qualitative Information http://www.evaluationsupportscotland.org.uk/resources/136/
• Using Technology to Evaluate http://www.evaluationsupportscotland.org.uk/resources/131/
• More general advice on generating useful evidence can be found in the 'Evidence for Success' guide http://www.evaluationsupportscotland.org.uk/resources/270/

TIP The most rigorous evaluations will be based on data collected using a range of methods