Child poverty - monitoring and evaluation: policy evaluation framework

Theme 3 – Impact on drivers

In the previous Theme 2, you will have identified which driver or drivers of poverty your policy is likely to impact. With that clarity, you can now focus on the specific drivers relevant to you. The questions in this theme allow for a more nuanced understanding of how the driver is moving (or not) and how it is impacting (or not) on drivers of poverty and potentially targets.

This theme will take each of the three drivers of poverty in turn. That is, income from employment, cost of living, and income from social security and benefits in-kind.

Your policy may also have a system-change element with the aim of changing the child poverty support system and creating better join up between support services. We are developing an evaluation to examine how well the system in working for families living in poverty. Further information about this work and its associated research questions can be found in Annex B.

Income from employment

The following questions have been taken from the employability evaluation framework. This is to ensure that we maximise consistency in evaluation efforts.

Intended Outcome: income from employment for low income families is maximised

Evidence to understand: what specific outcomes relate to increasing income from employment (research questions 3.1) (short/medium/long terms, research question 3.2) and what is the likelihood of achieved said outcomes (research question 3.3)

Key questions:

Are people progressing, if so, in what way? Covering both progression towards employment as well as in employment

Rationale:

Understanding the distance travelled and progression whilst receiving employability support.

Understanding in-work progression

There will be policies which are providing the infrastructure to allow people get closer to the labour market. For example, childcare interventions.

Data collection:

• Recording income information
• Recording how many people enter employment / progress in the pay scale
• Record programmes that support parents to prepare themselves for employment
• Record successful entrants into labour market

Key questions:

Are people's goals/milestones being achieved within the expected timeframe?

Rationale:

Ensuring that, as far as possible, we're supporting people to work towards and achieve their aspirations. The need for an expected timeframe is driven by the experience of some users that get "stuck" in a cycle of particular support with no forward progression.

Data collection:

• Setting specific goals and tracking progress over time.
• Combination of administrative data and research

Key questions:

Are people entering and sustaining employment, training, and education?

Rationale:

Ensuing that policies do focus on long terms improvements on people's lives.

Data collection:

• Management information from services

Key questions:

For those policies increasing job supply/creation, are these developing focusing on reducing child poverty?

Rationale:

For example, are jobs created focusing on low income parents? That is considering the location, recruitment practices, flexibility, offering real living wave, etc…?

Data collection:

• Labour market analysis

Key questions:

What is the value for money of our investment?

Rationale:

Considerations of value for money need to take account of wider societal impacts on our services.

Data collection:

• Approach to this work will be driven by existing approaches (e.g. Social Cost Benefit Analysis used in the Fair Start Scotland Economic Evaluations)

Income from social security and benefits in-kind

Intended outcome: income from social security and benefits in-kind for low income families is maximised

Evidence to understand: What specific outcomes relate to increasing income from social security and benefits in-kind (research question 3.1) (short/medium/long terms, research question 3.2) and what is the likelihood of achieved said outcomes (research question 3.3)

The following questions have been taken from the Devolution of disability benefits: evaluation strategy - gov.scot (www.gov.scot) and the Social Security Scotland - Charter Measurement Framework. The rationale for these questions is underpinned by the Benefits take up strategy to maximise consistency in evaluation efforts.

Key questions:

Who is eligible? Are there any gaps in eligibility? (This links to Theme 4, targeting)

Rationale:

Ensuring that social security and benefits in-kind can support children in families in poverty and identify any gaps in provision

Data collection:

Measuring eligibility of income-related benefits is established through the Benefits take up strategy.

Key questions:

What is the take up of social security payments / benefits in-kind?

What are the factors influencing take up?

Rationale:

Under the provision of the Social Security (Scotland) Act 2018, Scottish Ministers are committed to promoting take-up of Scottish benefits.

Data collection:

Based on the Benefits take up strategy. Take-up refers to the extent to which people receive benefits they are eligible for. This is the number of benefit recipients divided by the number of people eligible to receive the benefit.

Qualitatively could explore: experience of stigma, application process, awareness and understanding of eligibility.

Key questions:

Is the policy reducing financial pressure on households?

Rationale:

Tackling both the affordability to pay bills and the stress or worry associated with paying these.

Data collection:

Some examples include:

• Records of what people spend the money on
• (Qual) reductions in financial stress/worry
• Relevant questions from client survey, e.g. 'How much did the benefit payment a) help you to pay for what you needed? And b) help make a difference to your life?' ; scale of 0-10, not at all to a lot).

Key questions:

Are clients achieving financial resilience or reductions in financial insecurity?

Rationale:

Longer-term security including reducing the need to enter into further debt.

Data collection:

Relevant question form client survey. For example, 'How much did benefit payment help you to control your finances?' (scale of 0-10, not at all to a lot)

Key questions:

Who is eligible? Are there any gaps in eligibility? (This links to Theme 4, targeting)

Rationale:

Ensuring that social security and benefits in-kind can support children in families in poverty and identify any gaps in provision.

Data collection:

Based on the Benefits take up strategy. Take-up refers to the extent to which people receive benefits they are eligible for. This is the number of benefit recipients divided by the number of people eligible to receive the benefit.

Qualitatively could explore: experience of stigma, application process, awareness and understanding of eligibility.

Reducing cost of living for low income families

The following questions have been developed to produce a generic framework on what to consider when measuring impact on cost of living.

Importantly, this should be done in the context of your target audience and for universal or wider reach policies, highlighting the specific impact for families living in poverty or on a low income.

Intended outcome: cost of living for low income families is minimised

Evidence to understand: What specific outcomes relate to decreasing cost of living (research question 3.1) (short/medium/long terms, research question 3.2) and what is the likelihood of achieved said outcomes (research question 3.3)

Key questions:

How is the cost of living changing for low income families?

How, and by how much, will your policy reduce the cost of living for low income families?

How are low income families specifically supported to reduce their cost of living relative to other families?

Rationale:

There are various ways policies can reduce the cost of living for families, be it by providing benefits in-kind or by directly giving money out to families to cover living costs.

Data collection:

Calculate value of benefit provided.

As part of the overarching child poverty measurement framework, indicators track cost of transport, childcare, food insecurity and fuel poverty.

Key questions:

How is the policy increasing accessibility to financial stability? (i.e. access to affordable credit, increase in savings, reducing unmanageable debt)

Rationale:

In what ways is your policy aiming to alleviate the poverty premium of low income families and increase financial stability to allow for longer term planning.

Data collection:

Calculate savings achieved for families.

Key questions:

How is the policy increasing access to affordable transport, childcare, housing or energy?

Rationale:

Sometimes, policies are focus on improving infrastructure to improve people's opportunities.

Data collection:

Quantify how access has been increased or improved (e.g. transport links established, new childcare settings opened/expanded, housing provided, increases in energy efficiency or access to digital connectivity/devices).

Measure reach and/or take up of new services

Enhancing life chances for households with children in poverty

The following questions have been developed to produce a generic framework on what to consider when measuring long term policies that aim to enhance life chances for people in the context of child poverty. These questions support the understanding that firstly, poverty is not a fixed state, and secondly that there are various conceptual layers of poverty that require a variety of support. (See Annex C for the Conceptual layers of poverty).

Intended outcome: enhanced life chances for individuals or families, with the ultimate aim of impacting on one of the three drivers of poverty (income from employment, cost of living, and/or income from social security)

Evidence to understand: What specific outcomes relate to enhanced life chances (research question 3.1) (short/medium/long terms, research question 3.2) and what is the likelihood of achieving said outcomes (research question 3.3)

Key questions:

Is the policy supporting improvements in wellbeing outcomes? And how are these improvements achieved?

Is the focus on parents or children or both?

Rationale:

Analysis of wellbeing outcome indicators to capture poverty-related evidence on longer term preventative action and to get early warning of potential adverse impacts from policies.

Data collection:

The Children, Young People ad Families Outcomes Framework core wellbeing indicators can be used as the basis for data collection/monitoring.

Further work to be develop to consider how to widen understanding of parental wellbeing.

Key questions:

Is the policy building capabilities for individuals/families or for organisations?

Are the capabilities linked to driving income form employment, reducing cost of living, or increasing income from social security?

Rationale:

There are policies with wellbeing and capabilities at the core of their delivery. This is particularly important and necessary for those deepest in poverty and to ensure long term benefits for families and individuals.

(see Annex C: Conceptual layers of poverty for detail as to what areas are included in the concept of capabilities)

Data collection:

Clear understanding of what capabilities policies are aiming to build.

Link capabilities with drivers of child poverty, while still long term, measure whether anticipated progress is achieved in line with expectations.

Key questions:

Is the policy supporting a systems change with a focus on reducing child poverty?

Rationale:

There are a range of interacting factors that influence child poverty. A number of different forms of support are needed to help people move out of poverty. A joined-up approach between and within policies at national and local levels is needed to facilitate life for families.

Data collection:

Based on research questions developed for 'systems evaluation', see Annex B: Evaluating the effectiveness of system-change initiatives