Tackling child poverty pathfinders: evaluability assessment

Our Approach and Methodology

Overview of our Approach

We have adopted an analytical and participatory approach to the evaluability assessment, involving gathering stakeholders together to co-produce key components, including the Theories of Change and monitoring and evaluation frameworks, and to discuss and debate possible evaluation methodologies and options as part of the evaluability assessment. Key data and outputs gathered from these participatory workshops have been examined and analysed to support the findings in this report. This process involved the following stages.

Phase 1: Inception and scoping. We held several inception meetings with the Scottish Government. This was an opportunity for Scottish Government to orient the team, discuss the context of the commission, objectives as well as issues and concerns. We also had separate inception meetings with the Glasgow and Dundee Pathfinders. During the inception phase, the team reviewed the shared programme and Pathfinder documentation, identified key stakeholder groups for each element of the work, designed the research tools, and agreed key deadlines, outputs, and deliverables.

Phase 2: Theory of Change (ToC) development. To get a better understanding of the Pathfinders, and the subject of evaluation (the evaluand), we held three ToC workshops, for the Glasgow and Dundee Pathfinders (see Child poverty ToC and MEL report for further information) and for programme staff at the Scottish Government. At each of the Pathfinder workshops we invited relevant staff from the Pathfinders, key stakeholders, and Scottish Government staff (for a full list see ToC and MEL report). Staff from several divisions from the Scottish Government attended the programme level workshop. There was also some representation from the Dundee Pathfinder.^[8] These workshops allowed for collaborative creation of ToCs for each of these aspects of the work, which have been refined and developed to inform the M&E frameworks, and to inform this evaluability assessment.

Phase three: Monitoring and evaluation (M&E) framework development. To develop the M&E frameworks we undertook a 'data audit' exercise with each of the Pathfinders to identify and understand the data already collected and held that might be useful for ongoing monitoring or evaluation. A separate data workshop was held with Scottish Government programme staff to get a better understanding of data held at the national and local level that could support impact evaluation. The Pathfinder M&E frameworks have been developed in a separate document and have been critical in supporting this evaluability assessment report.

Phase four: Evaluability assessment. The process for the evaluability assessment – the subject of this report – involved interviews with senior Scottish Government staff, two in-person workshops and an online workshop with Scottish Government programme and Pathfinder staff. The interviews centred around high-level questions relating to the overarching aims of the Pathfinder programme and consequently what is most important to get out of an evaluation. This then led into the first in-person workshop, which provided an opportunity to reflect on the aims of the proposed impact and economic evaluation, highlight potential ethical issues, and discuss the challenges and practical considerations associated with a variety of methods to evaluating impact. The second workshop built on this by focussing specifically on economic evaluation methods and how these could be applied in a Pathfinder context. A draft evaluation plan was then developed, and in an online workshop this was refined and mapped against the key data sources identified from our data audit workshops. The discussion involved getting a better understanding of potential data held at the national and local level that could support an impact evaluation, and specifically a quasi-experimental approach.

Phase five: Analysis and reporting writing. The team analysed the data and outputs from the ToC, MEL and evaluability workshops and undertook a literature review to address the evaluability questions below.

As noted previously, in addition to these five stages we are also undertaking an early implementation process study. This largely sits separately from the ToC, M&E framework and evaluability assessment, but will build on the findings where possible. It aims to identify early learning on implementation to inform future delivery of the existing Pathfinders and the development of additional Pathfinders. To conduct this evaluation, we are currently undertaking field research with partners, stakeholders and families, with the report being finalised in March 2023.

Evaluability Assessment Questions

The evaluability assessment will address the following questions:

1. What is a Child Poverty Pathfinder?

2. What factors need to be considered before evaluating the Pathfinders?

3. What ethical framework should guide an evaluation?

4. What key aims should an evaluation focus on?

5. What are the methodological options for evaluating the Pathfinders?

6. How can learning be assessed and integrated?

7. What is the best way of procuring the evaluation?

In the following chapter we set out the findings from the assessment, taking each of these questions in turn. The findings in relation to each of these points are then drawn together to provide recommendations and an evaluation plan at the end of this report.