Scottish Child Payment: interim evaluation

Methodology

This chapter provides an overview of the evaluation approach for Scottish Child Payment. It introduces the logic model and research questions driving the evaluation activities, and gives a summary of the different data sources used.

Overview of evaluation design and logic model

The Scottish Child Payment policy evaluation is based on a “theory of change” logic model. The model shows the mechanisms whereby interventions (such as Scottish Child Payment) have a chain of immediate, short-term, and medium-term outcomes. If these are met, they can generate longer-term outcomes and contribute to wider Scottish Government policy impacts.

Scottish Child Payment’s medium-term outcomes, and the Scottish Government’s long-term policy impacts (e.g. reduced child poverty), will take time to determine, and may require access to robust quantitative data that is not currently available. The long-term impacts in particular will be affected by a range of factors in addition to Scottish Child Payment, making it difficult to measure and attribute changes specifically to that payment. However, the achievement of shorter-term policy outcomes (e.g. increased child-related spend and reduced pressure on household finances) could reasonably be expected to contribute to these wider outcomes.

The logic model for Scottish Child Payment is below at Figure 1.

The immediate outcomes of Scottish Child Payment relate to the delivery process of the benefit – i.e. promoting and administering the benefit. The short-term and medium-term outcomes relate to policy outcomes, which are linked with the intended aims of Scottish Child Payment.

Immediate outcomes (Process Evaluation)

• Scottish Child Payment is well promoted
• Scottish Child Payment and its eligibility criteria are well understood
• Scottish Child Payment is taken up
• Making an application is clear and easy
• Applications are processed in a timely manner
• Awareness is raised about other forms of support
• Clients feel they have been treated with dignity, fairness and respect

Short-term outcomes (Policy Evaluation)

• Increased child-related spend
• Reduced pressure on household finances
• Reduced money-related stress
• Children able to participate in social and educational opportunities
• Improved position of main carers within households

Medium-term outcomes (Policy Evaluation)

• Reduced incidence of debt
• Improved health and wellbeing
• Reduced incidence of material deprivation
• Reduced barriers to education and labour market
• Positive impact on Scottish economy

Long-term outcomes and impacts

The long-term impacts in the logic model (reduced child poverty, reduced inequality of outcomes for children, and reduced incidence of social exclusion) relate not only to Scottish Child Payment but to the wider government outcomes for children and their families, and are influenced by all social security interventions, as well as other interventions designed to support families with children across the Scottish Government. As such, Scottish Child Payment will play an important, but not exclusive, role in contributing to these.

Evaluation questions

Below are the key questions that informed the evaluation design:

1. To what extent did Scottish Child Payment achieve its immediate, short-term, and medium-term policy outcomes?

2. Is there any evidence of Scottish Child Payment contributing to the long-term government policy impacts of (a) reduced child poverty, (b) reduced inequality of outcomes for children, and (c) reduced incidence of social exclusion?

3. What are the implications of the evaluation findings for future policy development?

Summary of data sources

In accordance with the evaluation strategy^[4], the evidence used in this report was drawn from multiple data sources, described below:

Bespoke commissioned research

Ipsos MORI Scotland was commissioned by the Scottish Government to conduct qualitative research with Scottish Child Payment recipients and third sector representatives who provide support to recipients. The research mainly focused on people’s experience of receiving the benefit, and how this mapped on to Scottish Child Payment policy objectives (i.e. the immediate, short-term and medium-term outcomes outlined above). However, it also explored additional aims, such as potential barriers to applying for the benefit.

In-depth qualitative interviews, lasting around 30-45 minutes, were conducted with 39 Scottish Child Payment recipients across Scotland. Additionally, 9 interviews with 10 third sector representatives were held. Fieldwork took place via telephone or online video chat between August 2021 and December 2021.^[5] The recipients were recruited based on key priority groups outlined in the first tackling child poverty delivery plan^[1] - i.e.

• Lone parents
• Families with a disabled adult or child
• Mothers aged under 25
• Minority ethnic families
• Families with a child under 1
• Larger families (with three or more children)

The main findings from the commissioned research are presented in this report to provide a fuller understanding of the implementation and impact of Scottish Child Payment at this early stage of delivery. The full report from Ipsos MORI Scotland is available at Annex B.

Official Statistics

Social Security Scotland collects information on applications, payments, and clients in the process of delivering the benefits. Some of this information is published online as Official Statistics. The following Official Statistics publications are used as sources of evidence in this report:

• Scottish Child Payment – high level statistics to 31 March 2022
• Social Security Scotland client diversity and equalities analysis to May 2021
• Social Security Scotland feedback statistics to 30 September 2021

Of these three publications, the high level statistics data is used most frequently, and is hereafter referred to as “Official Statistics”. The other publications are named in full when they are cited.

Please also note the following technical points about how Official Statistics are presented throughout this report:

• Figures are rounded for disclosure control and may not sum due to rounding
• Where stated, secondary analysis has been conducted on rounded figures from published Official Statistics
• Most results are presented to zero decimal places. ‘0%’ should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by ‘-‘.

More detailed figures and information about the Official Statistics used in this report are provided at Annex A.

Social Security Scotland Client Survey

The Social Security Scotland Client Survey ran August/September 2020 (round 1) and May/June 2021 (round 2).^[6] It was open to everyone who at that time had received either (a) a Social Security Scotland benefit, or (b) a successful decision on a benefit application from Social Security Scotland’s inception in September 2018 to March 2021.

The Client Survey collected equalities and socio-economic information from respondents. It also asked about their experience of Social Security Scotland and receiving benefits. In total, the survey received 10,575 responses (around 4% of the total number of invites sent), of whom:

1. 769 had applied for Scottish Child Payment only (i.e. as opposed to having applied for Scottish Child Payment and other Social Security Scotland benefits)

2. 1,188 had received Scottish Child Payment only (i.e. as opposed to having received Scottish Child Payment and other Social Security Scotland benefits)

Throughout this report, Client Survey findings are based on respondents who only applied for or received Scottish Child Payment. This is because their views and experiences specifically relate to applying for Scottish Child Payment, as opposed to potentially applying for numerous benefits.

Please also note the following technical points about how Client Survey findings are presented throughout this report:

• The number of respondents providing a valid answer to each individual question/statement varied slightly, within the ranges shown.
• Most results to the closed questions are rounded to whole numbers. As such, results (e.g. those presented in tables) may not sum to 100% due to rounding.
• Most results are presented to zero decimal places. ‘0%’ should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by ‘-‘.

Limitations

This section explains what can and cannot be determined from the available data sources, and how this influences the extent to which conclusions can be drawn about the early impact of Scottish Child Payment.

Role of qualitative research: The evaluation is largely dependent on findings from qualitative research commissioned by the Scottish Government. This provides a rich and detailed insight into the impact of Scottish Child Payment on recipients with a range of personal and demographic characteristics, and also the views of third sector stakeholders. However, the research also has the following limitations:

1. The findings are not representative of all Scottish Child Payment recipients, because (a) while diverse, the overall sample of participants was small, and (b) participants were self-selecting, meaning that they actively chose to take part, as opposed to being randomly selected.

2. Findings are based on the participants’ perceptions of impact, rather than objective measures of impact.

These are standard limitations of qualitative work. More detail on limitations has been provided at Annex B.

Role of Social Security Scotland research: The Client Survey statistics contained in this report are based on up to 1,188 responses from clients who applied for Scottish Child Payment (and the number of respondents providing a valid answer to each individual question/statement varies within the ranges shown throughout the report).

Although the survey results provide insight into the views and experiences of a substantial number of Social Security Scotland clients, it should be cautioned that this represents a small fraction of the 157,755 Scottish Child Payment applicants up to 31 March 2022, and that views are drawn from a self-selecting sample of applicants. No weighting had been applied to counteract potential response bias. Therefore, it cannot be assumed that the results represent the views of Scottish Child Payment clients as a whole.

It should also be noted that the Client Survey was designed for all Social Security Scotland clients, not just Scottish Child Payment applicants. This means that questions are worded in a general way to make them applicable to all benefits. Therefore, while it is a useful source of supplementary evidence, it is not specifically designed to gather the views and experiences of Scottish Child Payment recipients about the benefit.

Medium and long-term impacts will take time and additional data to determine: Understanding the true impact of Scottish Child Payment would involve measuring (a) progress towards medium-term policy outcomes, and (b) its lasting contribution to wider Scottish Government outcomes. Doing so requires suitable time to have passed, and for the latter in particular, it would involve isolating the influence of Scottish Child Payment from other contributing factors, such as wider social security benefits and other government interventions designed to support families with children. A step in this direction would be to gain access to data with appropriate outcome variables e.g. from population surveys, or further bespoke research. Note that, at the time of writing, the latest iterations of the Scottish Household Survey and Family Resources Survey do not contain sufficient samples of Scottish Child Payment recipients to conduct a meaningful analysis for this evaluation.

More detail on options for extended policy evaluation is provided in the evaluation strategy report^[4]. This will inform the approach to future evaluations of Scottish Child Payment, once the payment is fully rolled out.