Scottish Government Evaluation Action Plan

Delivering the vision

Culture of Evaluation

The SG already has a long-standing culture of evaluation. For instance, we routinely commission and publish findings from evaluations of key policies, we have our own guidance on evaluation, and offer our own training and knowledge exchange activities on evaluation for staff. However we recognise that we need to do more. So we plan to strengthen this by:

• Continuing to use findings from the 2023 Evaluation Skills Survey of SG Staff to adjust our existing internal evaluation training offer so that all relevant staff receive the training they require. For analysts this will include the further rolling out of the SG Evaluation Academy training, the provision of additional advanced and specialist evaluation training, and a regular seminar series for all staff on evaluation.
• For non-analysts, and particularly those in the SG Policy Profession continuing to rollout targeted training material which will highlight the role and use of evaluation in policy development.
• Establishing a virtual Evaluation Network for SG staff using MS Teams, which will provide a peer support and an evaluation review function.
• Setting up a network of evaluation champions from across the SG who have hands on experience of conducting evaluations.
• Explore options to establish a Centre of Expertise on Appraisal and Evaluation of Spending Programmes to be led by the Chief Economist and Chief Social Researcher. Despite the apparent similarities in the ultimate purpose of appraisal and evaluation, both forms of analysis are often carried out independently of each other. By integrating appraisal and evaluation into a single Centre of Expertise we will ensure synergies are made between the two forms of analysis. The proposed Centre will be charged with improving the quality of both appraisal and evaluation in Government, and for promoting evaluation to Ministers and the SG Executive Team.
• To measure changes in SG staff skills and experience in evaluation during this period, repeat the SG Evaluation Skills Survey in late 2025.

Quality of Evaluations

As well as ensuring evaluations are carried out routinely in the SG we also commit to the evaluations being designed robustly and appropriately, and their outputs being of a high quality. As has already been highlighted, training of SG staff is essential for the delivery of this vision, particularly as one of the key outcomes of the training will be to enhance the quality of evaluation activity. A suggested key function of the proposed Centre of Expertise on Appraisal and Evaluation would be to ensure both ethical and quality standards are followed. In addition we will:

• Roll out an internal peer review network to assess plans for the design, and outputs of SG commissioned evaluations.
• Where appropriate use external peer reviewers to assess evaluations plans and outputs from these studies.
• Develop and roll out an Evaluation Plan template to be completed by all policy and intervention leads at the start of the project / policy development, and which can be returned to throughout the duration of the project. Guidance and training will be provided on completing the template.
• Conduct periodic auditing of Evaluation Plans for quality and coherence. This role will be carried out by the SG senior lead and their team.
• Ensure that policy makers and analysts complete a Theory of Change (ToC) or Logic Model for every new policy or intervention. Theory of Change / Logic Model are useful techniques to assist with the planning, monitoring, and evaluation of any policy or intervention^[9]. Training on using them will be provided to all applicable staff
• Develop new processes to ensure that all SG spending review / budget decision making exercises are informed by appropriate consideration of evaluation evidence, and robust evaluation plans.
• Recommend that an appropriate allocation of the budget of a major policy / intervention be ring-fenced to be spent on evaluating it. Consideration of the costs of evaluation should be made as part of the initial cost estimates at the project / intervention design phase.

Use of Evaluation Evidence

In addition to ensuring that steps are in place for the SG to plan high quality, robust evaluations, and that this planning is a key part of the SG spending review / budget decision making process, we will also take forward plans to ensure that evidence collected is fully utilised.

By ensuring this happens the SG will be able to make better business decisions on whether the intervention should be: continued, expanded, modified or stopped; whether it is meeting objectives; can have impacts for particular groups, organisations or the environment; and if it is offering value for money. Having such evidence will inform the development of similar projects in the future.

To ensure that evaluation evidence is fully utilised we will:

• Ensure that in project Evaluation Plans full consideration is shown to types of outputs and when these will be produced. This will be achieved by including an evaluation use and dissemination plan section in the Evaluation Plan. This will set out how the evaluation evidence will be used, and highlight key decision points when evidence will be required.
• Through engagement with colleagues across the SG, ensure that evaluation evidence is built into and used within the SG performance and delivery reporting systems. As part of the periodic auditing of Evaluation Plans, as well as assessing whether evaluations are delivering or have delivered as planned, we will include an audit of outputs and assess how the evidence has been utilised.
• Review and strengthen internal guidance for completing Impact Assessments to ensure evaluation evidence is used more routinely and effectively when conducting impact assessments, and in the subsequent monitoring of these assessments.

Openness and Transparency

There is a strong link between ensuring that the evaluation evidence which we collect is fully utilised (as set out above) and our next commitment on openness and transparency. In accordance with the SG Social Research Publication Protocol[iv] we will publish findings from all our evaluations and, where possible, data collected on behalf of the evaluation will be made available. To encourage greater openness in relation to our evaluation activity we will:

• Launch a Scottish Government Evaluation web page, which will include links to recently published evaluations, links to our guidance, as well as forthcoming SG evaluations.
• Ensure that all SG evaluations are regularly added and updated on the UK Cabinet Office’s Evaluation Registry. This is a cross-government database of recently completed and on-going government evaluations in a user-friendly fully searchable format. The Evaluation Registry will also be promoted to all SG staff as a resource to be consulted for learning lessons from previous interventions when designing new policies. Furthermore, when the Evaluation Registry is made open to the public, a link to this database will be included on the SG evaluation web page.
• Announce new evaluations when they are commissioned using the Evaluation Registry.
• Commit to pro-actively engaging with interested stakeholders when planning, carrying out and publishing evaluations.
• Ensure outputs from evaluations are accessible, and where appropriate use different approaches to disseminate findings for instance plain English summary documents, infographics, animations and video.
• To encourage wider dissemination of evaluation outputs internally within the SG and ensure that findings / lessons from our evaluations are used in future SG policy making, we will organise regular ‘Evaluation in Practice’ seminars to showcase findings. Other internal dissemination activities will also be taken forward.
• Continue to work closely with devolved and UK governments to share good practice in relation to evaluations and to contribute to shared networks.