Climate change - Scottish National Adaptation Plan 2024-2029: monitoring and evaluation framework

Evaluating Climate Adaptation

What is the role of evaluation in SNAP3?

Monitoring the outcomes and objectives in the Plan through our indicator sets will allow us to assess whether we are making progress towards achieving the impacts we want SNAP3 to deliver. This is important because we know that, if these are being achieved, they will contribute towards improved resilience for people and places in Scotland. A lack of progress in certain indicators could indicate that additional action is needed to meet our objectives and outcomes.

What the indicators do not tell us, however, is to what extent and in what ways particular policies in the Plan are contributing to the achievement of these outcomes and objectives. While our indicators can build understanding of where additional action may be required, there is a need to understand what sorts of actions are particularly effective and what might not work so well. It is also important to understand how policies are working for specific groups of people in Scotland. We know that some groups are especially vulnerable to climate change risks and impacts. Policy evaluation can help to build our understanding of these sorts of questions. Stronger evidence from policy evaluations across priority SNAP3 areas can strengthen our understanding of what works, and what doesn't, for whom, in what contexts and why in successfully adapting to the risks and impacts of climate change. It can improve targeting of policies to particular groups by building evidence around specific barriers and enablers groups face in accessing the benefits of specific policies.

Taken together, evaluation and monitoring can play a key role in the development of better policies. The Magenta Book, which provides UK government wide guidance and best practice on evaluation, presents this through the ROAMEF cycle (shown below in figure 7). As demonstrated, when used together, monitoring and evaluation can allow us to assess whether the Plan is meeting its aims as intended (in this case through our monitoring framework) and then understand why this change is happening (through robust policy evaluation).

Evaluation evidence produced from key SNAP3 policies will feed into and strengthen the CCC's independent assessment and reporting. It will also play a key role in informing ongoing and future policy development so that our policies are better placed to deliver the outcomes we want to see. This framework sets out an initial plan for how we will develop an evaluation approach for climate adaptation across key policies in the plan. This will be developed further during the SNAP3 delivery period, to produce a detailed policy evaluation framework for adaptation policies. The framework will contain information and guidance such as:

• Key evaluation principles in adaptation policy
• Key research questions for understanding contribution towards adaptation outcomes and objectives. These will include process/implementation and impact questions so that all elements of the monitoring maps can be explored through evaluation research
• Guidance on methods for evaluating adaptation policies (i.e. when to use qualitative and quantitative methods, guidance on developing an appropriate counterfactual, monitoring data needs and advice on collecting this, information on particular groups most vulnerable to the impacts of climate change)
• Ethics and data protection in evaluating adaptation policies

How will the evaluation framework be developed?

Collaboration and engagement

Policies in SNAP3 are designed, delivered and implemented by a large and diverse group of stakeholders, within Scottish Government and beyond. Key stakeholders include local government and national and local delivery bodies such as SEPA and Scottish Water. It is important that those who are delivering and have the most detailed understanding of the relevant policies can feed into the evaluation framework development. This engagement and collaboration will help facilitate a sense of shared ownership and greater uptake use of the tools set out in the evaluation framework. In turn, this means that consistent evaluation evidence, which addresses the key identified research questions and evidence needs, can be collected across priority policy areas in the Plan.

Identification of key policy areas

SNAP3 contains many different policies. While all of these policies should have some sort of data or evidence behind them, it is not proportionate, nor feasible, to have a full and detailed evaluation for every policy in the Plan. Part of the process of developing the evaluation framework will be to identify priority policy areas for evaluation. Criteria for assessing these priority policies for evaluation will be agreed with stakeholders but may include:

• Areas of substantial spend within the Plan
• Particularly new or novel policies without an existing evidence base
• Likely impact of the policy on the outcome area(s)
• Policies which we know face particular barriers or challenges in their implementation and delivery
• Cost, feasibility and value of a robust evaluation of the policy on adaptation outcomes

Support to collect data and monitoring information

For evaluation to be successful there should be regular and ongoing data collected in order to demonstrate delivery and impact. While the framework will provide guidance on how to do this, it will also be important to recognise where there may be challenges in implementing robust and ongoing data collection mechanisms. Engagement will continue beyond the publication of the evaluation framework to support delivery partners in building their skills and capacity to collect data of the necessary quality for evaluation at the right points in the implementation and delivery of their intervention.

Reporting on evaluation findings

Having an agreed evaluation framework in place should support the collection and reporting of useful evaluation evidence across key policies within SNAP3. This will improve our evidence base around what works, doesn't work, for whom and in what ways in adaptation policy. We will consider the best way to bring this evidence base together to provide a package of learning which can inform future policy development and the next National Adaptation Plan.