Cancer strategy 2023 to 2033

Appendix 1: Monitoring and Evaluation

National Cancer Strategy 10-Year Outcomes Framework

An Outcomes Framework has been developed to identify the outcomes that matter as expressed through consultation and engagement with people affected by cancer. This Framework sets out the 11 cross-cutting ambitions that are anticipated to create change around seven key outcomes to achieve the strategic aim of improved cancer survival and excellent, equitably accessible care.

These seven outcomes that we expect to see at the end of 10 years connect with the overarching vision for the strategy:

More cancers are prevented, and our compassionate and consistent cancer service provides excellent treatment and support throughout the cancer journey, and improves outcomes and survival for people with cancer.

Key Principles for Evaluating the National Cancer Strategy

Evidence gathered through monitoring and evaluation will be required to assess how far the activities associated with the ambitions are progressing. A Monitoring and Evaluation Framework is being designed to cover the duration of this 10-year strategy. A strategic, evidence-based approach will assess progress towards the strategy’s seven key outcomes via the 11 cross-cutting ambitions. The focus is on national (macro) level activities and how these are translated at NHS Board (meso) level to take a Once for Scotland approach to strive for consistent and sustainable services.

Given the system pressures in the NHS and economic pressure on public finance, the evaluation approach will be proportionate and connect with existing monitoring and evaluation activities in NHS Boards. It will focus on measuring actions that are fundamental to achieving the overarching vision. This includes attention to areas where less progress has been made, such as less survivable cancers, and ongoing inequalities in screening and diagnosis, as documented in the National Cancer Plan: Progress Report. It also requires attention to outcomes for people on Waiting Lists and for those who are referred for an Urgent Suspicion of Cancer, some of whom will start a cancer pathway but will not be diagnosed with cancer.

Developing high quality evidence is a fundamental principle underpinning guidance on policy evaluation as set out in The Magenta Book: Guidance for Evaluation. High quality evidence can increase our understanding of what works, maximise the chance of achieving the strategy’s ambitions, and reduce delivery risk. Given the system pressures highlighted above, it will be important to use existing collection and analysis of routine datasets where possible to minimise the burden of reporting for NHS Boards. The evaluation will also use data already being gathered to support commitments in the National Cancer Plan (2020).

New evaluation activities will focus on analysing high priority data and addressing evidence gaps that can demonstrate the measurable contribution of the strategy to improving services, care and outcomes. The question of what is most significant to measure to understand success will be considered across all decisions about data and evidence. Evaluation will also engage with potential unintended consequences.

The success of the strategy will be judged on the realisation of the seven outcomes as evidenced through data and indicators. This will include quantifiable metrics for what is measurable. Qualitative data will capture evidence of experiences, why something works (or not), and the impact for people affected by cancer and the workforce. In prioritising analysis and evidence, evaluation will focus on new models of care or changes that involve significant investment, systemic change or risk.

Where gaps in evidence are identified, new data collection will be proportionate and focus on evidence that can satisfy us that progress is being made in relation to the outcomes as they connect with the strategic aim of the strategy. Consideration will be given to how any new data collection aligns with existing monitoring and surveillance requirements for other health care policies and strategies. This approach will support coordinated policy development across health conditions with the objective of delivering high quality services and care for people with cancer in Scotland.

Steps for Developing the Evaluation Framework

The work to develop an Evaluation Framework will be an iterative process that engages stakeholders in mapping out intended and potential unintended consequences of planned activities, and identifying how change can be measured using management information, research and evaluation.

Stakeholders in this process include, NHS Planners, Regional Cancer Network Managers, Public Health Scotland and the Scottish Cancer Coalition. A working group of analysts, clinical advisors, managers, and planners and policy officials has been established to deliver this Evaluation Framework.

Theory of Change

‘Theory of Change’ describes how and why change is expected to happen and considers the evidence and assumptions underpinning the theory. It focuses on the bigger picture. A Theory of Change model will underpin the development of this Monitoring and Evaluation Framework, to understand how and why a desired change is expected to happen in a particular context. The Realistic Medicine Approach in Scotland recognises that health services alone cannot tackle all the factors that influence good population health. Therefore, evaluation activities will take account of this complexity as reflected in Guidance for Handling Complexity in Policy Evaluation.

A Logic Model can be created from a Theory of Change to map out the expected pathways or steps to realise the intended outcomes and the evidence required to demonstrate whether change has happened. A Logic Model specifies what goes in (Inputs), what is delivered (Activities), what comes out (Outputs) and what the results will be from these (Outcomes). While a Logic Model is by necessity a simplified graphic that sets out key inputs and activities that will contribute to outputs and outcomes, it is important

to recognise that pathways are not linear. With attention to inequalities across all aspects of the strategy, it will be important to understand variation within pathways for different groups of people and communities.

Mapping pathways to change will involve setting out the inputs, activities and outputs associated with the outcomes that will contribute to realising the strategy’s aim. It will take time for evidence of progress to build. Therefore, outcomes will be considered in the short, medium and longer term. Short term outcomes of the 10-Year strategy will be linked to the first 3-year National Cancer Action Plan.

The following steps outline the approach that will be used to develop the Evaluation Framework:

• Setting out the short and medium term outcomes that will follow from the inputs, activities and outputs to contribute towards the long term outcomes.
• Identifying key indicators and evidence to track progress, assess whether outcomes are achieved and measure potential unintended consequences.
• Creating a visual ‘Logic Model’: a simplified schematic that demonstrates how inputs, activities and outputs relate to the anticipated outcomes and aim.
• Linking indicators and evidence to existing data sources, and identifying any significant gaps in evidence that are high priorities for new data collection or analysis.

This step-by-step process will involve working through the detail of how a change in outcomes is anticipated to happen in the short, medium and long term.

Definitions:

• Inputs: resources needed to deliver a programme of change (what we invest)
• Activities: actions required to produce the desired outcomes (what we do)
• Outputs: direct, tangible effects of activities that are delivered (what we get)
• Outcomes: desired results of this programme of change (what we achieve).