Annex 1: appraisal and evaluation checklist
1. Define the objectives
Objectives and outputs should be set out clearly and relate explicitly to policy or strategy. They should be defined so that it can be established by evaluation after the event whether and to what extent objectives have been met. It is important that objectives are not described in such a way as to exclude options. Ideally they should be specific, measurable, agreed, realistic and time-dependent (SMART).
2. Consider the options
- Consider the options (i.e. the alternative ways of meeting the policy objectives). These must include a "do nothing" or "do minimum" option which provide a benchmark against which other options can be judged. It also helps demonstrate the need for the action and exposes the consequences of no action. Factors below could influence the choice of alternatives:
- risk;
- timing;
- scale and location;
- scope for shared service arrangements with other public bodies;
- degree of private sector involvement;
- capacity of the market to deliver the required output;
- alternative asset uses;
- use of new or established technology; and
- environmental equality.
For Major Investment Projects as wide a range of options as possible should be considered before preparing a short list for full appraisal. Where some options are dismissed before a full appraisal the reasons should be explained.
Human rights due diligence checks should be carried out before entering into an investment relationship or agreement or with a third party. Guidance on this is available in the Due Diligence Good Practice Guide.
3. Identify, quantify and, where possible, value costs, benefits, risk and uncertainties of options
Identify all significant costs and benefits likely to accrue from each option. Where possible, value in real terms on an "opportunity cost" basis (generally the market value of the resource). Costs and benefits should include adjustments for optimism bias, differential tax treatment (eg in comparisons of public private partnerships versus conventional procurement), and distributional implications. Where costs and benefits cannot be valued in monetary terms, record and, where possible, quantify them. Market values may not be available for some costs and benefits involving quality issues, including some relating to the environment. Assess associated risks and uncertainties, and who is best placed to manage each risk. Assumptions about the future are subject to a margin of error (eg the risk that the demand for the service to be provided will fall off). The risk inherent in the proposal should be identified and valued wherever possible.
4. Analyse the information
Provide a discounted cash flow comparison of the monetised streams of costs and benefits. Also, provide an overview of the important issues, an assessment of costs and benefits that cannot be valued, and an examination of the risks and uncertainties.
5. Decide what evaluation should be performed at a future date
This should include an assessment of criteria against which outcomes can be compared to show that agreed objectives have been met.
6. Present the results
Summarise the objectives, options considered, results obtained and what the results imply for the final management or policy decision. Provide information on underlying assumptions and calculations, for example:
- causal relationships between immediate, intermediate and ultimate objectives and outputs;
- costs and benefits not valued;
- timing of costs and benefits;
- a sensitivity analysis of the effects of changing key assumptions;
- the price basis used;
- the base date for discounting or the discount rate used.
and describe the information needed to inform later evaluation.
The Pre-Expenditure Assessment Guidance includes a useful 'template' for summarising the results of appraisals. Further advice can be obtained from the SG's Analytical Services Divisions.
Evaluation
7. Establish what is to be evaluated and how past outturns can be measured
Specify the activity (project, programme, policy, particular aspects of one activity, or common issues affecting several activities) to be evaluated. It should be related to policy or management aims and objectives. It should not be so narrowly defined as to preclude alternative methods of meeting those aims and objectives. Define and quantify objectives and outputs as precisely as possible. Review the availability of output and performance measures and other monitoring data, and how they relate to objectives. Consider the need for additional data. [The time at which the evaluation is to be undertaken, and the data required, should be well planned in advance, preferably when the policy programme or project is being formulated.]
8. Choose alternative "states of the world" and/or alternative management decisions, for comparison with outturns
Define carefully what is being compared with what. Be aware that the outturn being better or worse than expected may be attributable to factors outside the control of the relevant body / organisation.
9. Compare the outturn with the target outturn, and with the effects of the scenarios chosen for alternative "states of the world" and/or management decisions
As with appraisal, identify and measure where possible both the direct and indirect costs and benefits of the project, programme, or policy. Compare what happened with target outturn and with one or more alternative outturns which would have occurred if the state of the world and/or management or policy decisions had been different. Assess the success of the project, programme or policy in achieving its immediate and intermediate objectives, and how this achievement has contributed to the ultimate objectives.
10. Present the results and recommendations
Summarise the following:
- why the outturn differed from that foreseen in the appraisal;
- how effective the activity was in achieving its objectives;
- its cost-effectiveness; and
- what the results imply for future management / policy decisions.
11. Disseminate and use the results and recommendations
Results and recommendations from an evaluation should be disseminated widely and fed into future decision making. Evaluation reports should generally be placed in the public domain.
Page updated: July 2018