Cancer strategy 2023-2033 and cancer action plan 2023-2026: monitoring and evaluation framework - August 2023

4. Assessing the Overall Impact of the Strategy

4.1 Overarching evaluation questions for the ten-year strategy

As indicated in Section 3.1, evaluation questions to assess progress towards the strategic ambitions will be decided when evaluation activities are prioritised and commissioned with a focus on high priority data and urgent evidence gaps. Across the duration of the strategy, it will be important to consider how evidence is building to understand the overall impact of the strategy.

How will we understand the overall impact of the ten-year strategy?

Overarching impact evaluation questions are set out below:

1) To what extent are we making progress towards each high-level outcome?

2) To what extent is survival improving?

3) What progress has been made in achieving excellent care?

4) What progress has been made in achieving equitably accessible care?

5) What impacts have national programmes of cancer care and linked investment had on people affected by cancer and what matters to them?

6) What impacts have national programmes of cancer care and linked investment had on reducing health inequalities?

7) What impacts have national programmes of cancer care and linked investment had on the workforce who are delivering services and support?

8) What impacts have national programmes of cancer care and linked investment had on system capacity in NHS Boards?

9) What factors have supported or hindered the effectiveness of new models of care (including local contextual variation, external factors, unforeseen events)?

10) What impacts have national programmes of cancer care had on other parts of the wider health and social care system? And wider system changes on cancer care?

These questions relate to outcomes for people affected by cancer and the workforce, as well as to wider system level outcomes. The key steps for change shown in Appendix 4 focus on outcomes for people affected by cancer as articulated in the Outcomes Framework (see Figure 1). As discussed in Section 2, part of impact evaluation will involve understanding the impact of the strategy as a whole. In addition to building evidence on contribution from the evaluation of interventions and programmes of work, system mapping or modelling work may be required to understand the overall impact of the strategy within complex systems. Such approaches will be considered as part of regular reviews of evaluation approaches that will occur at set timepoints. Review points are discussed further in Section 5 in relation to the monitoring and reporting of evaluation evidence.

4.2 Key headline indicators to measure the extent of progress

How will we measure the extent of impact on survival and excellent care?

In addition to combining evaluation findings of individual interventions or programmes, key headline indicators have been established to measure progress towards the intended high-level outcomes and vision for the ten-year strategy. We will monitor each indicator at national level to assess if outcomes are improving, and to what extent. These metrics will provide evidence to address Questions 1 and 2 above. The indicators selected are focussed primarily on outcomes (e.g. survival, quality of life), given the nature of the Outcomes Framework (see Figure 1). However, there are some indicators that by necessity are focussed on system level metrics (e.g. waiting times statistics).

Table 2 sets out the headline indicators for assessing the extent of progress. These indicators have been developed in conjunction with analysts in Public Health Scotland (PHS). Many of the indicators selected are based on data already collected by PHS and the National Records of Scotland (NRS). The use of existing datasets enables us to set a baseline position prior to the launch of the Cancer Strategy although the impact of COVID-19 will mean careful interpretation for some data. These indicators will be reviewed annually. Monitoring will not be limited to headline indicators and we will consider other important data.

Appendix 2 provides detailed metadata for all currently published data that will be used as key headline indicators for tracking the extent of progress in these areas. The following information is provided in that metadata document:

• Indicator description,
• Baseline/ comparability across time,
• Collection frequency and details (including time lag),
• Data source (or data gap if no current sources of existing data),
• Data breakdowns,
• Robustness and data limitations.

Appendix 2 also provides further information on metrics that are in development, as indicated in Table 2. This includes further information about data quality and comparability across time. These metrics will be updated at the first annual review point for the Cancer Action Plan 2023-2026 (see Section 5.1 below for further information on our plans for reporting).

How will we measure impact on reducing health inequalities?

Many areas of the public sector, including different parts of the Scottish Government, are progressing work to improve their equality evidence. Where possible, headline indicators and other metrics to be used in our monitoring and evaluation framework will be disaggregated by equalities, socioeconomic, and geographic characteristics, reflecting the cross-cutting aim to reduce health inequalities in all areas. Further strategic work is in development, set out in Appendix 3, which will improve our ability to monitor and evaluate data and evidence on health inequalities relating to cancer.