Child poverty - monitoring and evaluation: policy evaluation framework

Why this Framework has been developed

This paper establishes the shared evaluation framework which will play a role in supporting consistent evaluations around the impact of Scottish Government policies on child poverty.

The framework was a key commitment made in the revised evaluation strategy as part of the second Tackling Child Poverty Delivery Plan. It is intended to be an open source that can be used by policy areas delivering interventions which will help to reduce child poverty. Ongoing reviews will take place to ensure that this document continues to fit into policy development as new initiatives or policies are developed and delivered.

Why this Framework has been developed

Over the life of 'Every Child, Every Chance' (2018-22) we noted a need for improving the consistency of data and evidence collection across the various policies which form part of our overarching approach to tackling child poverty. This base data is important for monitoring policy performance.

In addition, although plans are in place to evaluate policies to see if they are fulfilling their own objectives, they may need adapting to understand the contribution they are making to child poverty. There are a number of reasons why this might not be happening:

• Policies which relate to large system changes take time to implement. For several policies, implementation had only just started when the pandemic hit impacting both policy delivery and the collection of monitoring data.
• There can be challenges for policy implementation agencies to collect additional information, especially if it is confusing their policy narrative, or adding burden to service providers or clients/beneficiaries. For example, a key message of Early Learning and Childcare is that it is about children's development and that placements are, of course, not conditional on the parent being in, or taking up, work. Asking parents about work can confuse this message.
• It is not always possible or proportionate for specific policy evaluation to consider the wider factors which may be relevant to enabling some low income families to enhance their interaction with wider drivers of child poverty reduction. For example, provision of childcare may enable parents to engage in employment support. However, other barriers may persist which restrict their ability to access or progress in employment.
• For households in poverty, or with multiple adversities, data collection can feel overly intrusive, affecting response rates and learning. It also often does not fit with a more dignified experiential approach. Sometimes, detailed characteristics information cannot be collected.

Bearing all this in mind, we need to get better at:

• Understanding and articulating the importance of child poverty alongside other primary policy outcomes for each policy.
• Identifying how and why policies are working and/or the barriers to their effectiveness – both overall and for priority families.
• Articulating and understanding how policies are working together to improve the lives and outcomes of families. For example by increasing a parent / carers ability to participate in employment, education or training.
• Ensuring that evaluations consider not just effectiveness but scalability. For example, a policy that gets 100 parents into work could be effective on its own terms but this needs to consider whether it can be scaled up in a cost-effective way.
• Ensuring that evaluations increasingly contain economic considerations.

What this framework is about

This framework sets out to create a shared understanding of how we measure the impact of individual policies on child poverty. Namely, around:

• setting common definitions
• identifying child poverty outcomes
• setting the rationale for data collection
• presenting options for measuring impact

While successful evaluations will provide important contextual information, the framework is not about:

• cumulative impact assessment. The cumulative impact of the package of policies on targets is estimated separately via UKMOD. It is used to model the impact of any moderate to large scale policy that has an impact on household incomes or housing costs. It is also used to project the child poverty rate forward to future years.
• systems evaluation. This is a separate stream of work which explores how the system is working for families. This includes, how well the policies have been designed, and whether they are being implemented as a coherent package. See Annex B for further details on this work.