Cancer strategy 2023-2033 and cancer action plan 2023-2026: monitoring and evaluation framework - August 2023

2. Evaluation Purpose and Approaches

2.1 Evaluation for accountability and learning

What are the main purposes of a monitoring and evaluation framework?

Our monitoring and evaluation framework has two main purposes:

1) Accountability;

2) Learning.

As reflected in the Magenta Book, which provides UK-wide government guidance on evaluation, evaluation has an important role in both accountability and learning. We have adopted a theory-based framework to inform the overall approach. This is based on a theory of how the Cancer Strategy is anticipated to create change (or impact) for people affected by cancer. Further information on 'theory of change' is provided in Section 2.2 below.

What is the role of evaluation for accountability and learning purposes?

Evaluation activities can generate evidence to demonstrate the effectiveness, impact or wider outcomes of policy actions associated with the Cancer Strategy. This will provide accountability. Evaluation activities can support learning by generating evidence on what works, for whom, in what context, and why. We will consider evaluation that can contribute to these purposes. The Cancer Strategy is operating within complex systems, with uncertainty and variation inherent in the environments in which policy actions are being implemented. Understanding whether policy actions are working as expected, in specific contexts, and/or for particular groups, is likely to be relevant for evaluation, especially with the strategy's aim to tackle health inequalities.

How will we evaluate the Cancer Strategy for accountability and learning?

Impact evaluation will be considered in order to understand progress towards the outcomes that are articulated in the strategy. Outcomes will take time to emerge. Evidence will be relevant to demonstrate: changes in attitudes and knowledge (short term outcomes), changes in behaviours and decision-making (medium term outcomes), and overall policy impact or significant changes at population/ systems level (long term outcomes). If feasible, the 'counterfactual' should be considered, i.e. comparing the observed results with what would have happened if the intervention had not been implemented. Although positive outcomes are intended, there is always the possibility of unintended consequences or negative outcomes, which will also be considered across the duration of the strategy.

Process evaluation will be considered in order to understand how policies are operating in practice, and for different groups. This can provide evidence of howthe inputs/ activities/ outputs are delivered to achieve the outcomes, with the aim of capturing learning to refine the design of any interventions or inform future policy design. For instance, to deliver equality of outcome and access, it may be important to assess if and how the strategy is reaching target populations for key interventions.

It may be appropriate to undertake economic evaluation, to understand whether a particular policy action is cost-effective and providing value for money.

Evidence from these different types of evaluation can be combined to inform decisions about whether to stop, change or extend policies in the next Cancer Action Plan. For instance, evidence from pilot projects can be used to inform further roll out or scaling up of initiatives. Evidence of whether a policy is impacting on inequalities and variation, and how, can be used to adapt initiatives in order to better serve people experiencing poorer outcomes and to tackle intersections of inequality.

It is important to note that evidence will always be partial with gaps and potential dissonance across different data sources. Firstly, this is due to the complexity of the systems and environments in which the Cancer Strategy is operating. This means that it will be impossible to measure and understand all potential factors that may have contributed to any observed change, and it may not always be possible to construct a 'counterfactual' (what would have happened if the Cancer Strategy was not operating). Secondly, data availability, data collection tools and the capacity of data management systems will evolve over time, which means that there will be gaps in evidence at earlier points in the strategy. Thirdly, proportionality is an important factor in considering what and how to evaluate, which means that we cannot and should not evaluate everything. These contextual issues frame the evidence that we will gather and synthesise. We can still draw reasonable conclusions as to how policy actions are likely to be contributing to change as we consolidate evidence and articulate key learning.

What types of evaluation questions will support early evidence building?

Early evidence for the Cancer Strategy is likely to focus on the following process and impact questions:

• Process evaluation: Are the interventions being implemented and/or delivered as intended? How are they operating in practice? Are they working differently for different groups? If yes, why? How is the context influencing delivery?
• Impact evaluation: What measurable contribution to change is the intervention making? What is the scale of that change? Have different groups been impacted in different ways? Are there any unintended consequences or negative impacts?

These questions may also be useful for guiding local level evaluations to support learning against key indicators of success that can be shared across Scotland.

How will evaluation approaches evolve over the ten-year strategy?

Since the Cancer Strategy will be delivered over ten years, it is likely that we will develop nested or hybrid evaluation designs that combine approaches. This will require a review of evaluation approaches and their feasibility at set timepoints, with processes in place to learn and respond, which will be outlined in Section 5 below.

These approaches will build deeper understanding across time to determine how policy actions are contributing to change, and the extent of that change. Initial evaluation activities will focus on intervention or programme level change. As the evaluation evolves and evidence builds, we will seek to understand the impact of the totality of programme activities, including any impact on health and care systems.

2.2 Understanding and measuring change

How will we measure progress in relation to the strategy's ambitions?

As outlined in the Cancer Strategy, its success will be judged on the realisation of seven outcomes set out in the Outcomes Framework (Figure 1).

We will assess progress towards these seven key outcomes by measuring progress towards delivering on the 11 priority ambitions (cross-cutting and pathways) that are anticipated to create change. In many cases we would expect to see changes in attitudes/ knowledge in the short term, changes in behaviour/ decisions in the medium term, and wider impacts for policy/ society in the long term. Assessing progress is likely to involve a mixture of process and impact evaluation, and potentially economic evaluation, to understand if change is happening, and how.

How have we identified the key steps for achieving and measuring change?

The Cancer Action Plan 2023-2026 sets out 133 actions that are grouped around ten of the 11 priority ambitions, with actions on 'tackling inequalities' merged with actions for other ambition areas (a further three actions relate to governance rather than policy implementation). To measure progress effectively, a theory of change has been developed for each of these ten ambition areas. A theory of change is a model of how a policy or initiative is expected to contribute to change (or impact) through a sequence of early and intermediate outcomes. Theories of change help navigate complexity. A theory of change is never perfect, since it is necessarily perspectival and evolving. However, it is a useful tool with which to achieve the following results:

• Identify the key steps to produce change, and when they must happen,
• Stress-test assumptions with those who design and implement policy,
• Agree collectively what defines success and what it takes to get us there,
• Provide a timeline for outputs and expected outcomes from policy actions,
• Establish what we can (and cannot) measure in order to track progress,
• Map and prioritise the most useful indicators to measure progress,
• Set up a framework for what we need to track to assess progress,
• Establish any evidence gaps that may benefit from further research,
• Identify potential harms or unintended consequences to assess.

In developing these theories of change, analysts have worked with policy teams to map the outputs that we expect to deliver and the short term outcomes that we expect to achieve. The associated medium term and long term outcomes that follow logically from these short term outcomes have also been mapped and align with the periods covered by the second and third Cancer Action Plans. Assumptions and external factors that may influence these outcomes have been considered as part of the process.

How does theory of change development relate to the overall strategic vision?

Assessing progress towards the strategic ambitions will provide a step towards understanding impact in relation to whether and how the overarching vision for the Cancer Strategy is being achieved over time in relation to more cancers being prevented, better survival, excellent care, and reduced inequalities. These ambition-level theories of change are live working documents that will be kept under review and updated on an ongoing basis as evidence emerges over the course of the action plan.

The ambition-level theories of change have fed into an overarching theory of change that summarises the key steps of change that we anticipate happening over ten years to achieve the strategic vision. Appendix 4 provides a summary of these anticipated changes, showing how each ambition will lead to the high level summary outcomes that are set out in Figure 1 above. The advisory group (see Appendix 1) reviewed and contributed to this overarching theory of change. The theory of change will necessarily evolve over time as interventions are implemented and assumptions are refined in light of emerging evidence.