Evaluation for policy makers - A straightforward guide

Chapter 4: What does the evaluation process look like?

At what stage should I consider an evaluation?

The quick answer is at policy development stage because you need to make sure that early policy decisions don't limit the evaluation you need to do later on. Even minor aspects of the way a policy is formulated can impact significantly on the ability to evaluate it. It is important to ensure that evaluation is considered and planned at the same time as the policy is being formulated so that the links between the policy design and evaluation options can be recognised and accounted for.

The importance of considering evaluation early: an example

Imagine you want to conduct a pilot to test a policy to decide if there is a case for rolling it out nationally so you decide to fund 3 pilots in comparatively low populated areas. Because the number of people using the pilot is relatively small, analysts tell you that it's only possible to do a qualitative evaluation (using interviews). You're disappointed as this won't provide the information required to make decisions about national roll out. If you had discussed evaluation requirements with analysts at pilot development stage, then you could have considered expanding or extending the pilot to ensure the type of data you need to evaluate it could be collected.

Deciding what type of evaluation is required

For large, complex and expensive evaluations, it might be useful to conduct a formal 'evaluability' assessment in collaboration with stakeholders, analysts and experts. The general process looks like this (Figure 1). Even if the evaluation is smaller scale, conducting a lighter touch evaluability assessment is always a very good idea. This process can ensure that evaluations are useful, and avoids committing significant resources to an evaluation if there is little realistic expectation of benefit.

Figure 1: Evaluability assessment steps

The step by step evaluation process

If you all agree that conducting an evaluation is feasible and useful, then analysts, in collaboration with policy and stakeholders, are likely to follow a general step by step process that looks something like the process set out in Figure 2.

Figure 2: The evaluation process

Top Tip: Combine evaluation with strategic planning

As you set longer term goals and a vision of the future, ask analysts to help identify ways to measure these goals and objectives and how you might collect, analyse, and use this information. This process will help ensure that your objectives are measurable and that you are collecting information that you will use. Strategic planning is a good time to create a list of questions you would like your evaluation to answer.