Scottish Government Evaluation Action Plan

About the SG

Background

This plan restates the value the Scottish Government (SG) places on evaluation, and sets out our ambitions for more effective and strategic use of evaluation.

Evaluation is essential to the work of the SG, and can help us:

• understand what policies and interventions (including spending and taxation) are working,
• make improvements when they're needed, or stop doing some things if they are not working,
• allocate resources where they can make the most difference,
• understand which of our policies work best, for whom, and why,
• ensure policies benefit the people who they are designed to help and don’t have unintended consequences,
• where appropriate, highlight whether interventions offer value for money,
• through the outputs from our evaluation activity, highlight to the people of Scotland what’s working.

Overall, evaluation provides accountability for SG policies and interventions, allows us to improve them and deliver better outcomes for the people of Scotland.

This Action Plan also links to, and complements, a number of other Scottish Government activities and priorities including:

• The National Performance Framework
• The Scottish Government’s Open Government Action Plan
• The First Minister’s four priorities for Scotland (eradicating child poverty, growing the economy, tackling the climate emergency and improving public services)
• A strengthened Performance and Delivery System
• Implementation of a programme of public service reform
• The First Minister’s National Advisory Council on Women and Girls (NACWG)

Key approaches to Evaluation in the Scottish Government

The work of the Scottish Government covers many different policy and operational areas. As a result there is a requirement for different types of evidence to inform the design, and operation, of these policies and interventions. This evidence and analysis is provided by the SG’s Analytical Service Divisions or Units which typically comprise analysts from the statistics, social research, economist and operational research professions.

As part of this support, SG analysts regularly conduct evaluations of the key policies and interventions for which the Scottish Government has responsibility. Evaluations can either be conducted in house, or commissioned from external contractors. To inform our evaluation activity we follow the principles outlined in the HM Treasury’s (HMT) guidance on evaluation - the Magenta Book. In addition, we may also use the HMT’s Green Book, which provides guidance on the design and use of monitoring and evaluation before, during and after the implementation of policies, programmes and projects.

Both the Green and Magenta Books reference the ROAMEF (Rationale, Objectives, Appraisal, Monitoring, Evaluation, Feedback) policy cycle which is used to illustrate the steps for policy design and implementation. Evaluation is a key element of this cycle (and in other models which illustrate the key elements of designing, implementing, and learning from government interventions).

Evaluation and the policy cycle:

As set out in the HM Treasury Green Book evaluation fits in to the policy cycle, and is a core part of developing, implementing, and learning from policy

Appraisal is another core component of the ROAMEF policy cycle which is closely interlinked with evaluation. Appraisal helps decision makers choose between options for a policy or intervention before implementation; evaluation can provide evidence of how an intervention has performed if piloted and once implemented.

The HMT Magenta Book sets out the three main types of evaluation used by governments to assess their interventions, and the SG regularly carries out one or more of these types of evaluations. These are process, impact and value-for-money evaluations[i] and each is focused on answering different types of questions. For a full understanding of whether an intervention worked, how, why and for whom, and at what cost, all three types of evaluation are required.

Drawing upon the Magenta and Green Books, as well as from other evaluation guidance, the SG has developed its own evaluation guidance. This includes plain English guidance on evaluation for policy makers[ii], and a handbook for conducting evaluations of rail projects in Scotland[iii].

In 2023 we conducted an Evaluation Skills Survey of all staff to understand existing levels of knowledge and skills on evaluation in the SG. We have already used the data collected to further enhance our in-house training provision for staff on evaluation. This has included the launch of the Evaluation Academy. This is a new training programme for SG analysts being delivered by other specially trained SG analysts. More recently we have developed and launched new training resources for SG policy professionals on evaluation, which has been graded according to level of competence (beginner, developing, and practitioner).