Evaluation for policy makers - A straightforward guide

Evaluation Jargon Buster

Evaluation

The rigorous collection and analysis of information that determine the merit or worth of the policy.

Evaluability

A formal set of steps that assesses whether a policy can be evaluated in a reliable and credible fashion.

Outcome

The short-medium term changes or results that a policy or intervention should or has brought about. For example a change in attitudes or behaviour.

Impact

Impact is when you can determine if outcomes have occurred as a direct result of the policy or intervention.

In a logic model, 'impact' can mean the long-term, cumulative effect of policies over time on what change they ultimately aim to contribute to such as change in the national crime rate or morbidity and mortality.

Logic model

A logic model is a graphic that explains how policy activities should lead to a series of outcomes that, in turn should contribute to an ultimate outcome or impact . It can be developed for any level of intervention - an event, a project, a programme, a policy, a strategy or an organisation.

Baseline

This is evidence you gather ideally before an intervention has been delivered. Therefore, if you gather similar information afterwards, you can compare the findings before and after the intervention and hence assess to what extent and whether the outcomes may have been achieved

Note: Just because there is a change from start to finish doesn't necessarily mean that the policy has 'caused' that change.

Monitoring data

Data and information that is collected continuously during the lifetime of a policy or service. Data can be collected on user numbers, completion rates, activities and progress towards achieving outcomes. Monitoring data should be more relevant to practitioners and managers than evaluations as it can be analysed at any given point in time.

Validated

A validated questionnaire, scale or inventory has been tested to make sure it measures the constructs that it is supposed to measure e.g. 'anxiety' or 'motivation'. It has also been tested for reliability (the ability of the questionnaire to produce consistent results).

Sample size

Is the number of units from which data will be gathered in an evaluation e.g. the number of participants in a service who answer a questionnaire.

Control group

The control group is the 'untreated' group who don't experience the policy or intervention, but from which data is still collected. The 'treated' group do experience the policy or intervention. Having a control group means you can compare the results of the treated group with untreated group to see if there is a real difference between them. Another way of seeing the control group is as a baseline to compare groups and assess the effect of the policy or intervention.

Effect size

'Effect size' is a statistical term. It is simply a way of quantifying the size of the difference between two groups. For example, it can calculate the size of the difference between test scores in a group of people who experienced your policy and the test scores of a control group. If you have a small sample, the effect size (the size of the difference between the two groups) would have to be very large to have not occurred by chance (be statistically significant).

Statistical significance

Another statistical term. If a result (for example the difference in average test scores between two groups) is statistically significant then the difference is statistically large enough not to have occurred by chance or by fluke. We use a test to determine this.

Attribution

The process by which the causes of outcomes are explained.

Method

The particular way that an evaluation is carried out.

Scale or inventory

A list of items that are known to measure a particular construct e.g. level of confidence or anxiety.

Quantitative

Data is captured by counting and measuring things and analysed by making numerical comparisons. Quantitative approaches can allow us to generalise results from a sample to the wider population of interest.

Qualitative

Data is captured through a person's views and observations and reported in the language of the people interviewed.