Tackling child poverty pathfinders: evaluability assessment

What key aims should an evaluation focus on?

In this section, we set out our recommendations for the aims which are used to guide the evaluation. In determining these aims, it has been important to account for the complexity around what the Pathfinders do and try to achieve. While these aims have been designed considering the two existing Pathfinders in Glasgow and Dundee, they also encapsulate the policy context and objectives of the broader Pathfinders programme. Therefore, we feel that they can be used to evaluate the Pathfinders in Glasgow and Dundee, as well as act as a framework for the consideration of future Pathfinders.

We have suggested four aims for the evaluation. These are presented separately, but – as detailed in the remainder of this section – have inherent overlap between them.

1. Impact evaluation: To understand the impact Pathfinders have on families, child poverty, and the system that supports them

2. Value for money evaluation: To understand the Pathfinders' costs, benefits, and impact on the economy and public expenditure

3. Process evaluation: To understand the effectiveness of the way Pathfinders have been implemented

4. Learning: To understand how lessons from the Pathfinders can be captured to support scaling up and rolling out elsewhere

These evaluation aims can be used to support conversations around the purpose of the different Pathfinders to help ensure that partners are on the same page about what they are trying to achieve with their Pathfinder related collective action. For example, there are currently conversations in Dundee about the relative importance of the system change aim within what they are trying to achieve – i.e. to what extent are they looking to create change to the way organisations work together as a system. Systems change should therefore be an important aspect of the evaluation, but it is important to understand what it means and where it fits within the evaluation.

Systems change relates to structural and procedural changes to the organisations which support families, which in turn improve the services provided to families. This can refer to (but is not limited to) changes in:

• The types of services available
• The ways in which families are contacted and brought into the system
• The extent to which the right families are reached
• Methods of identifying and targeting families that need support
• The complexity and length of families' journey through the system.

In the design, implementation and evaluation of the Pathfinders, there are two important simultaneous concepts to consider: how the support models help people, as well as how the broader Pathfinder approach drives systems change. For example, the Glasgow model provides support through a helpline and a series of follow-up touch points depending on the person's type and intensity of needs. At the same time, the rationale behind the model drives systems change by shifting away from a support structure of numerous distinct support roles for separate issues, and into a more general support worker role of reaching out to people and arranging help for them in whatever way they need. This changes the attitude around how services are accessed and provided, and leads to a more holistic support system – the latter point being a primary aim of the Pathfinders.

Therefore, the evaluation aims are designed to acknowledge both these elements of the Pathfinder, in order to assess the extent to which both have been achieved. To do this, we suggest that evaluating systems change forms part of the first evaluation aim (the impact evaluation) because, as with assessing the impact on families and child poverty, evaluating systems change is primarily a case of comparing the situation before and after the intervention.

However, evidence to support the impact on systems change is likely to also be accessed through the process evaluation (aim 3). Process evaluations look at the way the service operates, how it is used, and the experiences of delivery staff and families that use it. These are factors which also relate to systems change, and so there will be value in drawing on the process evaluation findings to inform the assessment of whether the Pathfinders have driven systems change.

From our workshops and specific feedback from the Scottish Government policy team, we have identified a set of research questions associated with the above aims. These are provided as a longlist of detailed evaluation questions in the proposed evaluation aims and methods for evaluators to consider, refine and prioritise in a future evaluation.

We expand on how the evaluation should address each of these aims in the remainder of this section.