Tackling child poverty pathfinders: evaluability assessment

Introduction and context

Overview

The Child Poverty (Scotland) Act of 2017 set new legal targets for child poverty in Scotland, including for less than 18% of children to be living in relative poverty by 2023/24 and for less than 10% to be living in relative poverty by 2030.^[3] As part of this overarching target, the Scottish Government created the 2018-2022 Tackling Child Poverty Delivery Plan, and in March 2022 released the second stage of the delivery plan for 2022-2026.^[4] This second delivery plan includes a commitment to support the work of two Local Pathfinders – one operating in Glasgow and one in Dundee.^[5]

These Pathfinders, designed and implemented at a local level, aim to provide person-centred support to families most at risk of poverty. Critically, they also bring together all the support services that vulnerable families require into one place, in a proactive attempt to imaove the system as a whole and move away from a disaggregated service provision. As such, they aim to tackle child poverty both by directly supporting families at risk, and also by driving systemic change around the way in which families are supported.

As the Pathfinders represent a relatively new approach for tackling child poverty in Scotland, it will be important to conduct a thorough evaluation in order to assess their impact and extract learnings for future expansion or adaptation. This evaluation will take place in two phases. Phase 1 is making use of the early stages of the Pathfinders to develop understanding around how the Pathfinders operate, what their aims are, and how their impact can be evidenced. This in turn will be used to scope the methodology and lay the foundations for conducting the evaluation in Phase 2 as set out below.

Evaluation overview

Phase 1: Sept 2022 – March 2023

• Development of a clear theory of change for the Pathfinders
• Development of a monitoring framework clearly setting out what data will need to be collected for each of the Pathfidners to support effective evaluation
• Production of an evaluability assessment for the longer-term process, impact and economic evaluation
• Gathering of process study information on the early implementation and delivery of the Pathfinder interventions

Phase 2: 2023 – 2025

• An impact evaluation to assess the impact of Pathfinders on poverty rates, child poverty drivers and on any other relevant outcomes, ideally using quasi-experimental methods to robustly assess causal evidence of impact
• An economic evaluation using social cost benefit analysis to measure the relative value for money of the Pathfinders
• A process evaluation to explore perceptions of how Pathfinders are being implemented and experienced – including join up and alignment across ‘the system’, looking at what has worked well and for whom, what could be improved and what barriers and enablers exist

The Scottish Government has commissioned Rocket Science and Matter of Focus to carry out Phase 1 of the evaluation. This report sets out the recommendations in relation to the evaluability assessment and is accompanied, and informed, by two separate reports covering the Theory of Change (ToC) and monitoring and evaluation (M&E) framework.

The fourth element of Phase 1 – an early implementation process study – is ongoing and will be completed in March 2023. Outputs from the process study will also provide useful context and data for the impact evaluation. However, the focus of this process study is the early implementation phase of the pathfinders to inform the delivery of the existing Pathfinders and the development of future Pathfinders, the focus is not on assessing the impact of the Pathfinders. This report therefore specifically outlines our findings with respect to the evaluability assessment for the impact and economic evaluation and is informed by the work undertaken with the Pathfinders to develop ToCs and M&E frameworks (see Child Poverty Pathfinder ToC and MEL report).

The Child Poverty strategic and policy context

As set out above, Scottish Government has enacted an ambitious child poverty delivery plan, and current projections anticipate that the interim target of 18% of children living in relative poverty by 2023-2024 will be met.

In 2022, the Scottish Government published its second Tackling Child Poverty Delivery Plan (TCPDP), Best Start, Bright Futures (2022-2026), recognising the need to work differently and focus on outcomes to achieve the transformational change needed to meet the 2030 targets for child poverty reduction.

The plan acknowledges feedback from families that more needs to be done to support parents and families to understand and navigate the often complex and fragmented child poverty support system and to access the services and support they need to thrive. Specifically, the plan recognises the need to provide integrated and holistic support to parents to drive forward a reduction in child poverty. The plan focuses on three elements intended to directly impact on the drivers of poverty reduction (income from employment, costs of living, income from social security and benefits in kind):

• Providing the opportunities and integrated support parents need to enter, sustain and progress in work by increasing investment in employability support and focusing on key enablers and infrastructure (for example childcare and transport).
• Maximising the support available for families to live dignified lives and meet their basic needs by delivering public services in a holistic way and supporting parents and families to maximise their income and get access to the benefits, support and services they need.
• Supporting the next generation to thrive, focusing on supporting children and young people to get the best start in life, to learn and grow, and progress from school.

To achieve these goals, the plan recognises that transformational change and new ways of working are needed. A key way forward has been to set up Local Pathfinders to deliver "a new phased approach to whole system, person-centred support,"^[6] aimed at meeting the specific needs of families in need and most at risk of poverty. These priority family groups include: lone parents; ethnic minority families; families with a disabled adult or child; families with a young mother (under 25); families with a child under 1; larger families (3+ children).

The aims of the Pathfinder approach aligns with the Scottish Government's overarching Covid Recovery Strategy which focuses on addressing the systemic inequalities heightened by Covid-19, making progress towards becoming a wellbeing economy, and accelerating inclusive person-centered public services. In addition, the Pathfinders are taking place alongside other interventions aimed at catalysing system-wide and local changes including the No One Left Behind approach and the implementation of the Whole Family Wellbeing Fund.

In this wider child poverty reduction policy context, the Local Pathfinders are specifically aimed at contributing to "a new phased approach to whole system change"^[7] focusing on innovation and testing, refining, adapting, and implementing new approaches to delivering person-centred solutions that may be scaled, or replicated in different localities. Critically important then, is the need to gather evidence and learning from the Pathfinders on understanding whether and how they are effective in achieving system change and delivering holistic support which meets the specific needs of people locally to inform national policy and approaches for transforming the wider child poverty system.

Considering this wider context, there are a number of implications to take into account when evaluating Child Poverty Pathfinders. First, the Pathfinders programme is being delivered at a time of high costs of living and economic uncertainty, while still emerging from the pandemic. The economic situation may mean employment is harder to come by and, combined with the cost of living crisis, this likely means that more families are facing the threat of poverty, while those already in poverty are experiencing this more severely. While the Pathfinders are therefore being rolled out at an appropriate time, their impact may relate more to preventing or slowing additional poverty, rather than alleviating existing poverty.

Second, there are other interventions which are taking place at the same time which overlap the Pathfinders in terms of their aims and strategies. Like the Pathfinders, the No One Left Behind initiative and Whole Family Wellbeing Fund aim to create a more holistic support system for vulnerable families and improve employment opportunities for those who struggle the most. These have different aims than the Pathfinder programme – the No One Left Behind initiative is a high-level policy which drives change in the structure of employment services in Scotland, while the Whole Family Wellbeing Fund is a large (£500m) investment in health and social care with the specific purpose of reducing crisis intervention and shifting towards early intervention and prevention. However, there will be inevitable overlap between what they and the Pathfinders do, particularly as the Pathfinders provide tailored support which may include employment or health and social care. On one hand, this can provide a better platform for the Pathfinders to achieve their aims, but at the same time it may introduce more challenges for understanding the specific impact of the Pathfinders alone.

Third, articulating whether, in the long run, the Pathfinders lead to savings for public budgets will need to be read in the context of ongoing budget constraints, under-investment in services, and the presence of formal and informal waiting lists for many supports and services that are available. Improved outcomes for families by reducing child poverty are likely, in the short term, to lead to more families being able to receive higher quality and more intensive support, and then in the longer term are more likely to see these outcomes flow through into possible tangible public sector savings.

Aims of the Evaluability Assessment

Evaluability assessments can occur at various points in the programme implementation cycle. This evaluability assessment takes place relatively early on in the implementation phase of the Child Poverty Pathfinders which commenced activities in the first quarter of 2022. The overarching aim is to provide a plan for conducting a full evaluation of the Child Poverty Pathfinders programme and recommend the best approach for this evaluation. The output from this assessment will be a proposed evaluation plan, including key parameters such as methods used, timescales, and budgetary considerations. This will involve:

• Making explicit the assumptions and the ToCs underpinning the Pathfinders to support a future evaluation
• Assessing whether and how the Child Poverty Pathfinders can be evaluated in a reliable and credible way
• Identifying data gaps and informing the Monitoring, Evaluation and Learning systems of the Child Poverty Pathfinders to support any evaluation going forward.

In order to achieve this, our evaluability assessment considers high-level questions:

• What are the factors to consider prior to implementing an evaluation of the two Pathfinders?
• To what extent it is possible to evaluate the Glasgow and Dundee Pathfinders?
• What aims and research questions should the evaluation focus on?

It also aims to identify the most suitable methodology for the evaluation:

• Which evaluation techniques and methodologies are desirable and possible
• How the impact of the programme could be evaluated, including impact on families, child poverty, systems change and economic and fiscal impact
• How a process evaluation can capture the successes and areas for improvement in the Pathfinders' implementation
• How existing data and evidence will be used
• What the main limitations are of the recommended approaches.

Lastly, the assessment considers practical points relating to the implementation and procurement of any evaluation including phasing, cost and procurement.

Answers to these questions have been developed through a mixed approach of combining: a series of evaluability workshops; one-to-one discussions with relevant staff and stakeholders; a literature review of the policy area and evaluation best practice; and building on the outputs from the Theories of Change and monitoring and evaluation framework.

Structure of the report

This report is structured as follows:

Section 2 outlines the approach and methodology adopted to undertake the evaluability assessment, including the questions that the evaluability assessment aims to address

Section 3 examines the overall findings of the evaluability assessment structured against the questions identified in section 2

Section 4 provides an overall evaluation plan with key recommendations for the Scottish Government to consider when commissioning the evaluation of the Child Poverty Pathfinders

The appendices set out the additional detail underpinning the above sections, including: the ToCs; our best practice literature review; an evaluation ethical framework; a longlist of evaluation research questions; a discussion of how theory-based evaluation methods can be applied to Child Poverty Pathfinders; suggested costs and benefits to include in an economic evaluation; and an initial bank of financial proxies for an economic evaluation.