Tackling child poverty pathfinders: evaluability assessment

Executive Summary

The Child Poverty (Scotland) Act of 2017 set new legal targets for child poverty in Scotland, including for less than 18% of children to be living in relative poverty by 2023/24 and for less than 10% to be living in relative poverty by 2030.^[1] To deliver against these targets, the Scottish Government has made a commitment to support the work of Local Pathfinders.^[2] Two Pathfinders are currently being implemented in Dundee and Glasgow. As these represent a new approach at this scale in Scotland, the Scottish Government intends to commission an evaluation in 2023 to understand whether the initiatives are contributing to reductions in child poverty. This evaluability assessment will support the framing and planning of the evaluation.

The Pathfinders are seeking to effect change in complex child poverty systems. This has implications for evaluation, and requires a complexity-informed approach, which places emphasis on understanding how change happens. This evaluability assessment aims to understand and make recommendations around whether an evaluation of the Pathfinders could or should be conducted, and, if so, what the best approach to this would be.

Before an evaluation can be conducted, we have identified some key factors that need to be considered and addressed:

Agreement needs to be reached on a single definition of what is meant by "a Pathfinder". We have provided a suggestion as part of this evaluability assessment.

At this stage, we consider that each individual Pathfinder model is a separate subject of the evaluation, however at some point the programme as a whole could be evaluated.

Scottish Government's role in the Pathfinders needs to be more clearly established to ensure that all processes, data sources and causal linkages are comprehensively covered in the evaluation.

Other similar programmes happening at the same time might have spill-over into Pathfinder and evaluative activities, which creates additional challenges for isolating and attributing the impact of the Pathfinders.

There is a pre-existing local Pathfinder evaluation underway in Glasgow which must be taken into account so as to avoid unnecessary duplication and capitalise on those findings.

An understanding of existing evidence on the approaches taken by the Pathfinder models is essential.

We propose there are four broad aims of the evaluation, namely, to understand: the impact of the Pathfinders on families, child poverty, and the systems that support them; the value for money of the Pathfinders; how effectively the processes of the Pathfinders are working; and how learning can be captured from the Pathfinders and applied.

To evaluate the impacts on child poverty, families and systems, a key method of interest is a quasi-experimental study. Due to the complexities around child poverty, systems change, and the wider context in which the Pathfinders are operating, it is not clear that a quasi-experimental approach to the evaluation would be feasible. Therefore, should this approach be adopted, we recommend that a quasi-experimental feasibility study is first conducted to confirm the parameters and practicalities of this. In addition, a quasi-experimental study should also be supplemented with either a mixed-methods impact assessment using contribution analysis, or with qualitative-based case studies. A learning partner could be procured as a way of embedding continual monitoring and learning into the Pathfinders. Child poverty is a multi-faceted topic; due to this complexity, and based on previous examples of research and evaluation in this area, the impact evaluation should not rely solely on measuring changes in absolute or relative poverty, but include other key indicators which contribute to poverty.

In relation to evaluating value for money, we recommend a social cost benefit analysis as the best approach to doing this. Our evaluability assessment has identified a range of costs and benefits to capture in this – but implementing this in a robust way will depend on the data that is available to measure these. Gathering this data – where not already collected – should form part of the impact evaluation.

Learning is a key aim of the evaluation and is an integral part of the programme Theory of Change. Learning must be captured from the Pathfinders to ensure wider impacts on child poverty are achieved and to support further national efforts to reduce child poverty at scale. Part of this commission has involved setting up Theories of Change and monitoring and evaluation frameworks for the Pathfinders which will provide a strong foundation for learning to be embedded in delivery. Additionally, any investment in developing robust monitoring and evaluation processes by the Pathfinders in the implementation phase will not only support a future evaluation, but also the ability of the Pathfinders to adapt and improve.

This evaluability assessment represents part of Phase 1 of the Pathfinders' evaluation. Phase 2 will involve conducting the evaluation, building on the recommendations here. Currently, Phase 2 is due for completion by March 2025. Based on our assessment, it is unlikely that an evaluation would be completed within this time if a quasi-experimental approach is adopted. Therefore, if this approach is taken, an extension of the evaluation timelines may be necessary. If any of the proposed alternatives to quasi-experimental approaches is implemented, these may also require a minor extension.

Key Recommendations

1. The evaluation strategy and approach should be guided by the core aim of gathering learning from the Pathfinders.

2. The evaluation questions should focus on the 'difference' the Pathfinders have made in terms of impact, as well as on the processes of change. For this, the Theories of Change of the Pathfinders are of central importance in this evaluation.

3. The Pathfinders should primarily be evaluated using a mixed-methods approach, building on the Theories of Change and evidencing these with theory-based, quantitative and qualitative approaches.

4. If implemented, a quasi-experimental approach should be combined with a theory-based evaluation using contribution analysis or qualitative case studies, and should be preceded by a feasibility study.

5. The Pathfinders should be sufficiently supported and resourced to maintain robust monitoring evaluation and learning processes, to ensure learning is effectively captured to improve delivery and to inform Scottish Government decision-making processes.