Climate change - Scottish National Adaptation Plan 2024-2029: monitoring and evaluation framework

Introduction and Context

Scottish National Adaptation Plan

The third Scottish National Adaptation Plan (SNAP3) makes a commitment to develop and implement a monitoring and evaluation (M&E) framework for climate adaptation in Scotland. As the Plan sets out, there are a range of different elements which need to work together for Scotland, its people and places to become more resilient to and adapt to the impacts of climate change. Understanding whether and in what ways these different elements are working is complex. Having effective monitoring and evaluation in place can help the Scottish Government and other stakeholders to make sense of this complexity and understand to what extent, why and for whom progress is being made against the objectives and outcomes set out in SNAP3.

This document sets out an initial framework to fulfil the commitment to monitoring SNAP3. The monitoring framework is designed around the theory-based outcomes and objectives described in SNAP3 and sets out how we intend to measure and understand progress towards achieving these outcomes and objectives.

Why monitor and evaluate climate adaptation in Scotland?

By establishing indicators to monitor progress towards SNAP3 outcomes and objectives, and gathering evaluation evidence and learning, we can understand whether the actions we are taking are contributing towards the results we want to see across Scotland.

While most OECD countries have developed a national adaptation strategy, many struggle to design and set up robust measurement systems to evaluate progress on adaptation. A recent OECD review notes that, globally, only 40% of countries that have adopted a national adaptation policy have started to track its implementation.^[1] Adaptation does not have a universal, objective, quantifiable measure of success or one single measurable target (such as 'net zero' or metric tonnes of GHG emissions for our climate mitigation actions). There are a range of conceptual, methodological, empirical and practical challenges associated with measuring adaptation action and effectiveness at a national scale. Multiple comparative reviews have also shown that there is no single blueprint for national adaptation monitoring and evaluation systems, since each system is tailored to specific conditions and priorities, place-based, geographical experiences and political or governmental contexts^[2].

In the context of a rapidly changing climate and associated impacts, it is increasingly important to attempt to overcome these challenges so we can understand how well we in Scotland are adapting to the risks we face. Scottish Ministers have a statutory duty to report annually on progress towards the current adaptation plan. Improved monitoring and evaluation will improve the robustness and usefulness of this statutory reporting. This will support the earlier identification of where progress may be slower or facing barriers over the lifetime of SNAP3. It will also allow us to see where things are going well. Policy evaluation can build on these assessments of progress. It can help us to understand why changes have been seen or not, identify specific barriers and enablers in the implementation of policies, assess to what extent individual policies contribute towards observed impacts, and tell us more about how these impacts are being experienced among different groups of people and why this might be.

The M&E framework set out in this document is an important step towards gathering the evidence we need. It provides a framework for tracking key indicators of progress towards successful adaptation, sets out how we plan to evaluate and understand the impacts of key policies within the plan and how we will share knowledge and learning from the evidence gathered.

Elements of adaptation monitoring and evaluation in Scotland

The National Performance Framework (NPF) is Scotland's Wellbeing Framework. The National Outcomes set the vision for the type of nation Scotland wants to be. The NPF promotes collaboration across Scotland to achieve the National Outcomes, and measures progress through the National Indicators. The National Outcomes are currently being reviewed to ensure that they reflect the priorities of communities in Scotland. As part of the Review, the Scottish Government has heard from communities across Scotland about what matters most to them. A report setting out the activity undertaken as part of the Review of National Outcomes, and the proposed changes to the National Outcomes arising from it, was laid in the Scottish Parliament on 1 May 2024. This included a proposed new National Outcome on Climate Action:

• 'We live sustainably, achieve a just transition to net zero and build Scotland's resilience to climate change'

In line with this updated national outcome, the overall strategic aim of the SNAP3 is to build Scotland's resilience to climate change. It is expected that the revised National Outcomes will be published in early 2025.

To effectively and efficiently understand progress toward its strategic aim, this SNAP3 M&E framework has been designed to build on and complement existing monitoring and evaluation processes in Scotland that assess resilience and evaluate progress:

• Assessing Scotland's climate resilience: The UK Climate Change Risk Assessment (CCRA) sets out the risks and opportunities facing the UK from climate change. The CCRA assesses Scotland's resilience to climate change though a scored set of 61 risks and opportunities. CCRAs are published every five years following production of an extensive independent evidence base.
• Assessing progress of adaptation plans: Twice in every five-year period, the Scottish Government asks the Climate Change Committee (CCC) to provide an independent assessment of the implementation and progress of Scotland's current adaptation plan. This independent assessment is carried out using the CCC's adaptation monitoring framework for assessing the effectiveness of adaptation action in UK nations.

Through the new M&E framework for SNAP3, the Scottish Government will provide significantly better data to maximise the value of these CCC assessments, improve the overall transparency and effectiveness of adaptation action and support decision-making and delivery monitoring during the Plan's implementation period.

Development of the monitoring and evaluation framework

Using a theory-based approach with outcomes and objectives linked to the Plan's activities and enablers, the M&E framework aims to provide a way of understanding the complex nature of climate adaptation. A collaborative approach was key to the framework's development, with input from analysts and policy officials across key Scottish Government policy areas, as well as a range of external stakeholders. This ensured a robust approach to data mapping and the identification and agreement of indicators. This collaborative approach will continue over the Plan period to support ongoing engagement with data collection and to ensure the evidence gathered through the framework is presented and used widely across the Scottish Government and other stakeholders.

As well as complementing the existing Climate Change Committee (CCC) reporting structure, the framework has been developed to align with a range of other related M&E frameworks across Scottish Government. These include the monitoring activities for Scotland's Just Transition Plans, Heat in Building Strategy, Scottish Biodiversity Strategy, Environment Strategy and Climate Change Plan. Indicators in the SNAP3 M&E framework have also fed into the development of draft indicators for the newly proposed Climate Action outcome in Scotland's National Performance Framework. Crossover with other relevant M&E frameworks will be highlighted and linked to in the monitoring and evaluation element of the annual SNAP3 progress reports.

The work to develop an M&E framework for SNAP3 is intended to be iterative. We know that there are gaps in our monitoring indicators. This is partly the nature of monitoring such a complex system. It will not be possible to design a framework which covers every element of the system, nor is this proportionate. By building on the SNAP3 theory-based approach, the proposed indicators have been identified as the best available metrics to capture the most important elements of the SNAP3 objectives and outcomes. However, there are things we know we would like to track but where we do not currently have access to data that is robust enough to enable us to monitor accurately. The process of mapping data and engaging with stakeholders to develop this framework has helped us to identify these key data gaps. Over the lifetime of the Plan we will work with stakeholders to prioritise and identify areas where new data can be developed and collected or where existing data can be better used or adapted to tell us more about the adaptation outcomes and objectives we need to understand. We also have to consider the timeframes over which impacts of policies on Scotland's resilience will take to achieve. This framework is being developed to fit within Scotland's 5-year adaptation planning cycle and SNAP3 delivery period. However, the longer term nature of adaptation processes mean that ongoing monitoring and measurement beyond this period will be important to fully assess Scotland's resilience to climate change impacts. Finally, the set of indicators developed to monitor SNAP3 do not have quantifiable targets set against them. Instead a desired direction of travel for each indicator has been assigned. This decision was taken recognising the challenges of assigning a single quantifiable value to 'success' in a complex adaptation space. However, over time and with further data development it may be possible to work with individual areas to agree targets or standards in line with their policy ambitions.

As well as the work to develop our monitoring approach detailed above, this framework also sets out an initial plan for how we will develop an evaluation approach and framework for climate adaptation across key Scottish Government policies contributing to the Plan's outcomes. This will be developed during the SNAP3 period with stakeholders so that various forms of evaluation evidence can be gathered. This evaluation evidence and learning will support better understanding of how and why actions in the Plan are leading to any impacts seen.