Publication - Advice and guidance

Designing and Evaluating Behaviour Change Interventions

Published: 10 March 2015
ISBN: 9781785441660

Easy-to-use guidance on designing and evaluating any behaviour change intervention using the 5-step approach

AN UPDATED VERSION OF THIS GUIDANCE IS AVAILABLE HERE http://www.gov.scot/Publications/2016/05/1967

Supporting documents

Guidance for service providers, funders and commissioners

Appendix

Example data collection framework

Part of logic model analysed	Indicators	Data collected from (data source)	Data recorded in…	Data entered into database as…
INPUTS
Were there sufficient resources to run the intervention and how were they deployed? The evaluation should show what resources were required to run the intervention and whether they were sufficient to deliver the intervention as intended.	The total cost of the intervention. Average £ spent on each user. What were funds spent on? How many staff were required, staff, staff case loads, costs of running sessions, cost of materials, venues etc. Gather views on whether resources were sufficient. To what extent was the evidence base embedded into the intervention?	Manager and staff Annual accounts	Intervention level database	Costs, values and views Costs can be reviewed periodically (e.g. annually)

Part of logic model analysed

Indicators

Data collected from (data source)

Data recorded in…

Data entered into database as…

INPUTS

Were there sufficient resources to run the intervention and how were they deployed?

The evaluation should show what resources were required to run the intervention and whether they were sufficient to deliver the intervention as intended.

The total cost of the intervention.

Average £ spent on each user.

What were funds spent on? How many staff were required, staff, staff case loads, costs of running sessions, cost of materials, venues etc.

Gather views on whether resources were sufficient.

To what extent was the evidence base embedded into the intervention?

Manager and staff

Annual accounts

Intervention level database

Costs, values and views

Costs can be reviewed periodically (e.g. annually)

Part of logic model analysed	Indicators	Data collected from (source)	Data recorded in…	Data entered into database as…
PARTICIPANTS Collect information on your users to check that you reach your intended target group. Set up the database so you can collect data on each user. Data can then be aggregated to provide important quantitative data on users e.g. percentages, averages etc. You can also see whether the intervention worked for some users but not others by breaking down outcome data into different types of users (e.g. different ages, offence types). Numbers have to be large for this to be meaningful though.	User ID number	N/A	User level database	Entered as 01, 02, 03 etc
	Name	User survey	User level database	Name
	Date of birth	User survey	User level database	Date of birth
	Age at start of programme	User survey	User level database	Age
	Gender	User survey	User level database	Column - Gender Male= 1, Female= 2
	Current levels of physical activity	User survey	User level database	Coded e.g. 1 = none, 2 = minimal… 6 = regular cardio exercise
	Previous experience of physical activity	User survey	User level database	Coded e.g. 1= none, 2 = school-based, 3 = sports club…
	Attitude towards exercise	User survey	User level database	Coded on scale of 1-5 (very positive to very negative)

Part of logic model analysed	Indicators	Data collected from (data source)	Data recorded in…	Data entered into database as…
ACTIVITIES
What did users experience? Information on activities is important because if activities didn’t happen or were poorly delivered, then it is unlikely that outcomes will occur, if they did, something external to the intervention might be responsible. Work out the number and % of users who complete and did not complete the project as a whole and which activities they took part in.	Number of potential participants informed by school visits	School records	Intervention level database
	Number of participants recruited	Weekly register	User level database	This can be calculated via the total number of participants entered into your database
	Number of participants at each session choice of activity	Staff observation and weekly register	User level database and intervention level database	For each user record which sessions they attended and activities undertaken. Can code as: Week 1: 1= attend 2=did not attend Week 2: 1= attend, 2= did not attend etc. Also record total numbers for each session and activity in a separate record of activities. This will enable you to identify patterns in individual behaviour as well as analyse overall attendance and participation each week.
	Number of sessions run by outside clubs and classes	Staff observation and weekly register		In your record of sessions, record the sessions which were run by, or had visits from, outside organisers. This will enable you to analyse the impact these visits had on take up of outside clubs and classes.
How did users experience the project? Gather user accounts of what they actually did. The extent to which users valued the content of the project and their views on the way the it was delivered are important.	User perspectives on what happened in the sessions, the length of sessions, the format, quality of relationships with organisers and peers, what they learned and skills they developed. What did they enjoy most and least? Did they come every week? If not, why not? Did they engage with social media? Why/why not?	User survey	User level database	Assign codes to closed responses For example, user views on relationships with organisers: Very poor = 1, poor = 2…….. Very good = 5 and enter into database. E.g. 2 - enjoyment: Very enjoyable = 5, enjoyable = 4…very unenjoyable = 1 Analyse qualitative open questions by theme (not entered into data base)

Part of logic model analysed	Indicators	Data collected from	Data recorded in…	Data entered into database as…
SHORT and MEDIUM-TERM OUTCOMES
Did change happen? Quantitative measures of change Obtain a pre-intervention base line and post-project assessment end-line. (see info. collected on participants). Short term outcomes tend to be changes to attitudes, knowledge, learning, motivation or skills. Medium term outcomes show evidence of individual behaviour change. Measure the same outcomes at the start and exit point to see if change occurred. As well as scales, ask the users, supervisors and family as to whether they think users have changed and in what way. If there is no control group but you want to explore contribution you could elicit views on the relative impact of the intervention by asking users and family about perceived impact other interventions or support has had. Could also observe sessions at the exit to see if progress has been achieved.	The difference between the baseline situation and the end situation is the measure of whether change happened. Record the results of tests and surveys with users at the start and end of the intervention: Nb. Depending on your aims and approach, it might be appropriate to use psychometrics tests, physical tests (e.g. fitness or BMI), as well as surveys to measure attitudes, behaviour, competencies etc. Did users friends and family think progress was made? Professional judgement of progress (i.e. from teachers, doctors) User and family views on the contribution of external factors to outcomes, relative to intervention.	User survey Family and friends survey Survey of relevant professionals User and family survey	User level database	Create two columns-one for the outcome variable before and one column for after the intervention . For example: Attitude towards physical activity (before): Very positive = 5, positive = 4…..Very negative = 1 and attitude towards physical activity (after): Very positive = 5, positive = 4…..Very negative = 1 Could code answers e.g. Teacher’s views of participant’s progress: Very good = 5, good =4.. none/minimal = 1 And/or transcribe interviews for more depth information
LONG TERM OUTCOMES
(Difficult to assess without RCT) Did the intervention increase physical activity into adulthood? Did the intervention improve long-term health and fitness?	Did users think they had made sustained progress? Do stakeholders (eg. teachers and parents) perceive long-term changes in the group who engaged with the activity?	Longitudinal surveys of user and stakeholder views	Separate analysis conducted

Evaluation report structure

Structure and Content of the Report

Section 1: Executive Summary

Provide a brief overview of the project itself and it’s overall aims.
Summarise your main findings and recommendations from the evaluation.

Section 2: Intervention description

Explain why the project was required/funded. For example, was there a gap in provision?
Describe the project, including costs, target group and aims.
Describe how the project was intended to work, using your logic model (a diagram may be helpful). You should explain how your plans were informed by evidence of ‘what works’ elsewhere, show in detail how funds were therefore spent on the content of the project and set out the short-, medium- and long-term outcomes that you expected to materialise.

Section 3: Evaluation questions and methods

First set out what questions you were aiming to answer when you collected your evaluation data. E.g.:
- Inputs - How much did the intervention cost and how funds were spent?
- Activities - Were activities carried out as planned? Was the target group reached? How many of the eligible group completed and what did activities consist of?
- Short- and Medium-term (intermediate) outcomes - How many/what percentage of users changed attitudes or behaviour?
Describe what data was collected (quantitative and/or qualitative) in order to answer each evaluation question and describe how the data was collected, for example by questionnaire, observation or through the use of standardised tests.
Describe how the data was analysed (i.e. using Excel for numerical data or by identifying key themes in qualitatitive data).

Section 4: Findings/Results

Results should be set out to answer each of your research questions and must at least include the following results as a minmum

The cost/resources used and whether it was sufficient to run the activities.
Which aspects of the project were evidence-based and which were not.
How were users selected and was this effective at reaching the target group.
Characteristics of the eligible group and eventual users (not just completers).
Throughput – how many of the eligible group started, dropped out and completed and what were their characteristics?
Were activities carried out as planned, what was their specific content and how many participated in them?
How many made progress on different measures? Who did /did not and what were their characteristics?
What were users views and experiences of the project and did they perceive it as contributing to change?

Section 5: Interpretation and recommendations

Use your results to comment on the successes, challenges and lessons learned.
Reflect on the relative contribution of your project in relation to other potential influences.
Reflect on which parts of your logic model did and didn’t work as predicted and consider why.
List suggestions for modifying or supplementing the project in the future to better meet its aims (don’t be afraid to comment on areas for improvement – this lends credibility to your evaluation).
Conclusions must to be backed up by your results.

TIP Short chapter summaries are extremely helpful for readers who don’t have time to read the full report or who want to get a sense of the evaluation before reading it in detail.

This summary was drawn from excellent guidance on what to include in an evaluation report which can be found here: www.uic.edu/depts/crwg/cwitguide/05_EvalGuide_STAGE3.pdf

Contact

Email: Catherine Bisset

There is a problem

Thanks for your feedback

Was this helpful?

Yes

Your comments

Your feedback helps us to improve this website. Do not give any personal information because we cannot reply to you directly.

Choose a reason for your feedback

Your comments

Your feedback helps us to improve this website. Do not give any personal information because we cannot reply to you directly.

Yes, but

Choose a reason for your feedback

Your comments

Your feedback helps us to improve this website. Do not give any personal information because we cannot reply to you directly.

Information

Appendix

Example data collection framework

Evaluation report structure

Structure and Content of the Report

Contact

There is a problem