Tackling child poverty delivery plan 2022-2026 - annex 2: child poverty evaluation strategy - updated

The Assessment Approach

The first delivery plan, Every Child, Every Chance, set out the three key elements of the assessment approach. These 3 elements remain core to the evaluation approach, although as noted below we will be seeking improvements in each of them based on experiences to date.

Element 1 - Monitoring child poverty

Statistics to monitor the four child poverty targets are published annually, usually in March of each year. The data for the relative poverty, absolute poverty, and low income and material deprivation indicators come from the Family Resources Survey (FRS), which is administered by the UK Government's Department for Work and Pensions (DWP). The Scottish Government provides funding to make sure that enough Scottish households are surveyed to give us robust data at the Scotland level. Statistics are produced in March each year, providing annual updates on a three year rolling average. The data for the fourth target, relating to persistent poverty, come from the longitudinal Understanding Society survey. Both sets of statistics (from FRS and Understanding Society) are usually published together, with the date for publication set by DWP.

These statistics are highly robust at the national level and can provide good assessment for many sub-groups, but there are some weaknesses which we are trying to mitigate:

• It is a sample survey and will never be large enough to pick up small populations, sub-groups or local data. In 2021/22 and 2022/23 we agreed with DWP to further increase the sample of the FRS in Scotland to try to provide additional sub-group analysis. However, COVID-19 pandemic has had a severe impact on survey response rates. Moving forward, we will need to consider whether increased sample boosts can ever provide the refined detail we would ideally like or whether other approaches, such as data linkage, could provide feasible alternatives.

• The FRS data is highly lagged which can be problematic in understanding the impact of policy action in the short to medium term. It is important to continue to monitor a range of additional 'weather vane' data through the measurement framework to help us monitor the trajectory of the drivers of poverty.

Element 2 - Monitoring the drivers and impacts of child poverty

Reviewed child poverty measurement framework

The measurement framework was originally developed and published in 2018 alongside the first Tackling Child Poverty Delivery Plan to support the measurement of how the actions being taken were impacting on the drivers of poverty for children in Scotland.

It is reported on annually, although trend data is given to help contextualise annual changes. It is intended to help us understand why performance against the targets is improving, worsening or remaining static, which drivers are moving in the right direction, and which are not – potentially signalling the need for additional action or a change in approach.

The framework has recently been reviewed to check that it continues to use the best indicators and data. An updated measurement framework has been published alongside this document to support the progress reports for the second delivery plan.

Additional periodic review of wellbeing

To support our refined understanding of child poverty, we have decided to do a separate periodic (every three years) review of wellbeing outcomes for low income households. This will include child poverty but not be limited to it. The first report will be experimental and we will seek views of stakeholders. Depending on the result and usability of outputs, we may then provide reports every three years, starting with 2022/23. Subsequent reports will then be published in 2025, 2028 and potentially 2031 as part of a final evaluation of the Act. The review would consider 'do no additional harm' indicators as well as trigger indicators and wellbeing outcomes.

Examples might include:

• wellbeing of low income parents versus all parents;
• wellbeing of low income young people versus all young people (preventative before they become parents);
• harmful behaviours (low income parents versus all parents);
• satisfaction with home (e.g. policies that move minority ethnic families to more affordable social rented homes in areas where there is racial tension may not be beneficial);
• satisfaction with neighbourhood (poverty also relates to ability to use kin and friend networks for childcare);
• experience of harassment of low income versus other (improvements here would show cultural changes which are likely to feed through to reduced discrimination in other areas of life);
• parent relationships (loneliness is related to wellbeing) and child-peer relationships (a long-term preventative factor);
• child attainment (preventative).

These are just examples and potential indicators will be fully explored.

Continued support for local partners to access and interpret data

The national targets cannot be disaggregated at local level. Local partners therefore need to use a range of data to consider levels of poverty within their areas and we have been working with the National Child Poverty Co-ordinators group to help local partners do this. The child poverty dashboard provides a range of data that can help local areas to think about which drivers are weaker or stronger in their areas as well as the prevalence of different priority groups. The Scottish Household Survey, which does provide local authority breakdowns, now includes a poverty variable and statistics from the DWP/HMRC administrative linked data set Children in low Income Families (CILIF) is a valuable resource with data available for local authorities, data zones and wards. In addition many local partners such as local authorities and health boards will have a range of administrative and local data that can give them a very good sense of levels, types and location of poverty in their area, for example free school meals eligibility, council tax reduction etc. Examples can be found in Glasgow or Inverclyde.

Further consideration of priority family data

Various evidence from reviews of literature and lived and learned experience has also been provided around the priority families to help policymakers understand the range of differing barriers and enablers they faced. Evidence for the further two priority groups (young mothers aged under 25 and families with babies under 1) will be published alongside the annual report in June 2022.

We will continue to consider how to provide further driver evidence for the priority families and continue to update this evidence bank over time.^[2]

Element 3 - Assessing the impact of policies and external factors on poverty and its drivers

Element 3 examines the depth of the complexity of child poverty and our efforts to tackle it. Monitoring the drivers helps to tell us how they are changing over time, but not necessarily why they have changed. To understand the contribution of particular policies to changes in poverty and its drivers, we need to evaluate the relative contribution of different policy actions.

Performance data is critical but qualitative evaluation evidence is also important in helping to unpack the reasons for any observed changes (or lack of change), identify any unintended side-effects, and understand how the actions could be refined and improved. Qualitative evidence includes detailed information about how each action is being implemented (which is likely to differ across service providers and locations), and the views of people supported by actions to find out what has and hasn't been helpful from their perspective. Finally, as more policies come on stream it is important to understand not just whether a policy works, but to understand whether it is the best option. Economic evaluations are complex but will be increasingly important through the second delivery plan^[3].

There are 3 different types of evaluation activity within this element:

A. Individual Policy monitoring and evaluations including economic evaluation

B. System evaluation (quantitative)

C. System evaluation (qualitative)

A. Evaluating implementation and impact of individual policies

Given the size and scope of the first delivery plan, it was not feasible or justifiable to attempt a comprehensive evaluation of the impact of every action on child poverty. As with the evaluation of Scotland's alcohol strategy, decisions on where to focus evaluation efforts took into account a range of criteria, including: likely impact of the intervention on the four child poverty target measures; size of the investment from Government; and feasibility, cost and value of a robust evaluation of the intervention's impact on child poverty.

The shortlisted policies were subject to evaluation by the relevant policy and analytical teams. Key policies evaluated from the first delivery plan include Fair Start Scotland; expanded Early Learning and Childcare; Best Start Grant; Scottish Child Payment; Connecting Scotland; Money Talk Team; Private residential tenancy.

Many policies from the first delivery plan will be continuing in some form allowing development of longer-term evaluation material. For new policies, we will use the same approach for the second delivery plan, shortlisting new key policies based on the criteria set out above.

Each new or significantly refocussed intervention featured as part of the second Tackling Child Poverty Delivery plan is analysed in the Annex 5 (Impact of policies on child poverty).

We show how each policy links to the targets, via the drivers of income from employment, cost of living and income from social security or benefits in kind. We describe the type of impact anticipated, and the numbers impacted. We also look across the priority groups to show where there are direct links, as well as considering geographical factors and wider impacts on gender and inclusive growth.

This analytical structure ensures consistency and transparency, and provides a clear explanation of how our assessment of the impact on child poverty has been arrived at. In so doing it provides a good basis to move forward with evaluating the policies.

Lessons from evaluating individual policies from Every Child Every Chance

Over the last four years we have noted that whilst policies are being delivered there is often a lack of regular capture of key implementation activity/outputs. This base data is important for monitoring policy performance. Lack of data means that it is difficult to undertake policy evaluation or the cumulative impact assessment discussed below.

In addition, although plans are in place to evaluate policies to see if they are fulfilling their own objectives, they may not be collecting data to understand the contribution they are making to child poverty, and specifically in the context of the targets or priority families. There are a number of reasons why this might not be happening:

• Policies which relate to large infrastructure changes take time to implement. For several policies, implementation had only just started when the pandemic hit impacting both policy delivery and the collection of monitoring data.
• There can be challenges for policy implementation agencies to collect additional information, especially if it is confusing their policy narrative, or adding burden to service providers or clients/beneficiaries. For example, a key message of ELC is that it is about child wellbeing and that placements are, of course, not conditional on the parent being in, or taking up, work. Asking parents about work can confuse this message.
• During the pandemic, monitoring had to be light touch because of the capacity constraints on all public and third sector agencies and the constraints on data collection modes available.
• For households in poverty, or with multiple adversity, such data collection can feel overly intrusive, affecting response rates and learning. It also often does not fit with a more dignified experiential approach: Putting it bluntly, at what point does someone with little trust of government or organisations, turning up to play football in StreetSoccer become someone who could be open to interview as a data point?

From a tackling child poverty perspective, we will work with colleagues to continue to improve our understanding of child poverty from individual policy evaluations. Specific areas for improvement include to:

• Understand and articulate the importance of child poverty alongside other primary policy outcomes for each policy.
• Consider and articulate the appropriate nature and scale of data collection depending on the importance of the policy to tackling child poverty.
• Continue evaluation work to identify how and why policies are working and/or the barriers to their effectiveness, overall and for priority families. This will require collecting regular monitoring data that helps identify parental work and household income and ideally information to identify priority families.
• Ensure that evaluations consider not just effectiveness but scale. For example, a policy that gets 100 parents into work could be effective on its own terms but this needs to be considered against a scale of 60,000 parents without paid work.
• Ensure that evaluations increasingly contain economic considerations.

The original evaluation strategy set out that many of the individual actions would probably not be sufficient on their own to lift families with children out of poverty – rather, a package of policies would be required that fit seamlessly together. Work over the last few years has reinforced these messages. As a result many policies in the new delivery plan reflect the complex system. Rather than an employability policy there is an employability offer which is linked into other services such as childcare and transport. Cumulative impact work will be critical as we move forward.

B. Evaluation of the cumulative impact of the package of policies

Cumulative impact evaluation (quantitative). The Act requires us to be able to assess the impact of the package of policies on the targets. When the first delivery plan was published we were not in a position to do this. However we have now built the capability to use UKMOD, an open-access microsimulation tax and benefit model hosted by University of Essex. UKMOD is based on the Family Resources Survey and is therefore capable of estimating the impacts of various policies on child poverty. It can be used to model the impact of any moderate to large scale policy that has an impact on household incomes or housing costs. Small policies can be modelled but their impact on poverty cannot be accurately assessed individually due to the underlying sample size of the FRS – they are better modelled as part of a cumulative package.

Some kinds of policies are easier to model than others. Since UKMOD is based on individual-level data, it is necessary to define who benefits from a policy, in what way, and to what extent. Most social security payments are relatively straight forward, where we can define who is eligible and apply a take-up rate. Employment and other policies require a range of assumptions to be made: eg. how many parents are enabled to move into work, who these parents are, and how many hours they will work and at what pay level. Any policy must have an attributable, additional impact on household incomes or housing costs in order to be modelled.

This cumulative impact modelling has allowed us to broadly estimate the impacts of a range of policies from the first plan period. This differs from simply adding up the individual impacts, since policies interact with each other both in their actual impacts (for example, benefit payments often decrease when someone moves into work or when their housing costs are reduced) and in their impacts on measured levels of poverty (since some households may need multiple policies to cross the poverty line). The cumulative impact assessment also projects the child poverty rate forward to future years, incorporating the impacts of external factors including UK Government policies and macroeconomic trends. This allows us to assess how close or far we will be from meeting the targets in future years.

We will continue to use UKMOD to allow scenario development and cumulative impact assessment of policies to help us move towards the targets.

Lessons from quantitative cumulative modelling during the first plan period

• No single model can do everything. The inputs and assumptions into the model are equally as important as the model itself. This underscores the importance of obtaining good quality data and investing in analysis prior to the modelling stage.
• Similarly, how the outputs are presented and what questions we are trying to answer are equally as important as the model itself.
• Modelling takes time – not every variation can be modelled.
• Small changes in poverty are not reliable due to the nature of the model. Single policies have to be very substantive to 'show' in impact data. Cumulative approaches are often better. By the same token, small policies can add up to create large effects, particularly given their interactions.
• UKMOD is based on FRS which is always at least 18 months out of date and needs regular updates in line with government changes.

C. System evaluation of the implementation and impact of the package of policies

The quantitative modelling can tell us the theoretical impact of policies. However, whether theoretical impacts are realised will depend on how much friction is in the system. The third element of evaluation is to examine how well the system is working for families. This includes how well the policies have been designed, and are being implemented, as a coherent package. For example, is suitable childcare available to support employability services? Do the policies, and their implementation at a local level, seem joined-up from the perspective of the service user? Are there gaps in the coverage of the package of policies, for example are there particular groups who are not being reached?

This is an area where we have taken some initial steps: identifying and mapping the child poverty system (Child Poverty System Map - Introduction (data.gov.scot)); understanding the problems for people with lived experience^[4]; and reviewing published evidence of 'what works to tackle child poverty'. Substantial evidence shows the logistical difficulties of linking up employment, care, transport and financial support for people who have limited resources, opportunities and choice, coupled with often immovable requirements due to health or care responsibilities.

Taking forward evaluation in this space will respond to the transformational place based policy approach in the plan. It is a priority for the second plan period. It is clear that this next stage needs to analyse the process effectiveness and the cost effectiveness of the system at a local level from the perspective of families in poverty and from service providers. It also needs to draw learning from the different approaches being tested in pathfinders and other place based responses. This is complex and we will not get it right first time. Ideally this learning, good or less good, will be openly and constructively shared with other services and/or local contexts to better understand how local systems work in practice and to improve that practice. It also needs to grasp the difficult issues around economic evaluation looking at costs and return on investment. Evaluation in this space can only work if it is taken forward as a creative collaborative effort with partners at national and local level.