Cancer strategy 2023-2033 and cancer action plan 2023-2026: monitoring and evaluation framework - August 2023

3. Assessing Progress towards the Strategic Ambitions

3.1 Evaluation prioritisation and decision-making

How will we make decisions about which policy initiatives to evaluate?

Table 1 shows an Evaluation Decision Tool^[1] with criteria that will assist us to determine where evaluation activities would be best focused during the timeline of the first Cancer Action Plan 2023-2026. The questions for each of the criteria will be used to assess the appropriateness and value of evaluation, and are intended as a discussion-based tool to support decision-making.

Decisions, including a pre-screening of actions, will be based on the strategic priorities and evidence gaps established through the theory of change work. Evidence gaps may be linked to either the ambition-level or overarching theories of change, or to underpinning assumptions about how change may happen. The tool also helps us to take account of other practical considerations, such as feasibility, when deciding whether and what to evaluate.

These priority criteria will be reviewed when the second and third Cancer Action Plans are published to ensure that the criteria continue to represent the most pressing data and evidence needs at those timepoints. Establishing evaluation priorities for each action plan will provide the flexibility to design the best evaluation approaches and methods to address accountability and learning requirements as they emerge across the timeline of the Cancer Strategy.

Criteria for interventions that are likely to require substantial evaluation will include strategically important interventions, high cost interventions, where there are high levels of uncertainty/ risk, and evaluations with high potential for learning. Low-risk and well-evidenced interventions will require light-touch monitoring and evaluation to ensure they have been delivered as intended and achieved the predicted outcomes. This may include strengthened performance monitoring.

Some interventions that are low-risk and well-evidenced may still have high potential for filling a strategically important evidence gap. For example, it may be important to understand the impact of a well-evidenced intervention that aims to reduce health inequalities where that evidence is not sensitive to the local context in which the intervention is operating. Routine monitoring of many actions is possible through existing processes and governance. We will use existing data sources whenever possible, such as Public Health Scotland cancer datasets and Cancer Quality Performance Indicators (QPIs), to take a proportionate approach to minimise reporting burden.

Once priority evaluation areas are agreed, we will develop broad evaluation questions for commissioning evaluations. Thematic evaluations may be commissioned to address evaluation priorities across a number of interventions or a broad thematic area. This will include consideration of strategic evidence gaps and high priority data beyond intervention or programme level, such as evidence for overall progress and impact of the Cancer Strategy. Scoping work may be required to assess the evaluability of a particular intervention or programme of activity, or to establish the existing evidence base on which to design evaluation activities around an identified data need or evidence gap. The most relevant methodological approaches and specific methods for answering the evaluation questions will be selected, with attention to the evaluation purposes and criteria set out in Table 1.

3.2 Data and evidence to track progress over time

How will we track the progress of change over time?

The ambition-level theories of change will help us to track progress by monitoring planned outputs and evaluating outcomes associated with the 133 policy actions. They provide a bridge between the policy actions and the data we will need to track progress over time.

As part of the theory of change development work, we have identified available data and potential forthcoming data that would indicate progress in relation to short term outcomes, i.e. those outcomes we anticipate happening by the end of the first Cancer Action Plan (2026). Key indicators for assessing progress in relation to the strategic outcomes and vision are outlined in Section 4.2, with detailed information about existing data and development priorities for these metrics provided in Appendix 2.

Quantitative data will show the extent of any change. Qualitative data will provide depth of understanding about the complexity of context and mechanisms. This will include evidence about the contribution of policy to any changes, and whether interventions are having the desired effects and reaching the people we want to reach. In the context of inequalities, qualitative evidence will be important to determine whether interventions are having an effect on reducing barriers and improving access for people who experience disadvantage, inequity and poorer outcomes.

What will be our approach to gathering and analysing data?

We will refresh these metrics at regular review points to enable reflection and updating in light of developments, including work that is ongoing in the Scottish Government to improve data (see Appendix 3). Further work will be required at the end of the first Cancer Action Plan to establish what data and evidence will indicate progress in relation to medium and long term outcomes. It is important to note that we cannot measure every aspect of change since not all changes are either collectable or measurable. Further, it is not feasible to measure all those things that are collectable or measurable. We will therefore focus on identifying those data or evidence that are most crucial to establish whether and how change is happening in line with the priority criteria outlined in Section 3.1.

In line with a proportionate approach that minimises reporting burden, we will address data needs via routine data collection, such as repeated surveys, whenever possible. For instance, the Cancer Prehabilitation Survey will be repeated to evaluate any changes in how prehabilitation and rehabilitation services are operating. Where new analysis or bespoke research is required, approaches will be designed to answer evaluation questions proportionately to minimise burden, using the Evaluation Decision Tool referenced in Section 3.1.