Designing and Evaluating Behaviour Change Interventions Summary

Step 4: Monitor your logic model

Identify indicators to monitor progress of the logic model.

Use your logic model to identify indicators

Once the logic model is completed, you need to figure out how you will be able to tell if your model works as predicted, or not. To do this, you should:

1. Devise 'evaluation questions' – specific questions that you need to answer in order to test whether the model is working as predicted.

2. Identify specific indicators (measures or signals of some kind) that can answer these questions and therefore provide evidence that your model is or isn’t working as expected.

See below for an example of how this works in relation to a specific logic model.

Example - Increasing physical activity

Data collection principles

Now you’ve identified your indicators, you need to decide on a way of measuring or observing  these things. There are lots of different methods you can use to collect this data (see page 15) but some basic principles to observe are:

• Collect data for every stage of your logic model, including resources and activities as well as outputs.
   
• Collect data at a unit level (i.e. about every user of the service) and at an aggregate level (i.e. about the service as a whole). Unit level data can be very useful as it can tell you who the service is working for and who it isn’t and you can follow the progress of individuals over time. It can also be combined to give you overall data about your service. But remember, if you only collect aggregate data you will not be able to disaggregate it and therefore collect evidence about particular individuals.
   
• Follow users through the project. You should collect data about users at the very start, throughout and ideally beyond completion of the project. This will enable you to evidence whether users have changed, in terms of their attitudes, behaviour or knowledge.
   
• Make use of numbers and stories. Collect qualitative as well as quantitative evidence. Averages and percentages can help you to assess overall trends and patterns in outcomes for service users. Talking to people, hearing about the views and experience of users and stakeholders will help you to explain these patterns.
   
• Don’t reinvent the wheel. Standardised and validated (pre-tested) tools are available to measure such things as self-esteem, wellbeing and employability. Using these will enhance the reliability of your evidence and save you valuable time. Freely available tools are detailed here:
• http://www.clinks.org/sites/default/files/UsingOffShelfToolstoMeasureChange.pdf
• http://www.evaluationsupportscotland.org.uk/resources/tools/
• http://inspiringimpact.org/resources/ (follow link to 'List of Measurement Tools and Systems')
   
• Be realistic and proportionate. Expensive and/or experimental projects should collect greater amounts of data than well-evidenced and established, cheaper projects. You might want to give questionnaires to all users but it would usually be sensible to carry out in-depth interviews with just a smaller sample of your users.

Data collection methods

Various methods can be used to collect data in relation to your evaluation questions. Data can be collected from service users, staff or outside agencies. Not all methods will be suitable for all projects. Evaluation Support Scotland have produced excellent guidance on using different approaches.

• Using Interviews and Questionnaires http://www.evaluationsupportscotland.org.uk/resources/129/
• Visual Approaches http://www.evaluationsupportscotland.org.uk/resources/130/
• Using Qualitative Information http://www.evaluationsupportscotland.org.uk/resources/136/
• Using Technology to Evaluate http://www.evaluationsupportscotland.org.uk/resources/131/

More general advice on generating useful evidence can be found in the 'Evidence for Success' guide http://www.evaluationsupportscotland.org.uk/resources/270/

TIP The most rigorous evaluations will be based on data collected using a range of methods