Best Start Grant: interim evaluation

Methodology

This chapter provides an overview of the evaluation approach for BSG. It introduces the logic model and research questions driving the evaluation activities, and gives a summary of the different data sources used.

Overview of evaluation design and logic model

The approach to evaluating the policy impact of BSG uses a "theory of change" model. This approach develops logic models to show the mechanisms whereby interventions (such as BSG) have a chain of short and medium-term outcomes that, if met, can generate longer-term outcomes and contribute to wider Scottish Government policy impacts in the long-term.

The long-term impacts that the Scottish Government are trying to achieve for children, such as reduction in child poverty and inequality of outcomes, or reduced incidence of social exclusion, will take time to determine and are affected by a range of factors of which social security is only one. Some of the intended outcomes of the BSG (i.e. improved health and well-being for children and families, or reduced incidence of debt and material deprivation) may contribute to those wider impacts in the long-term but they are also likely to be affected by a range of other SG government policies.

As a result it is difficult to measure and attribute change in these long-term outcomes and impacts to a single benefit, such as BSG. However, we can reasonably expect that if success against short and medium term policy outcomes associated with BSG is achieved, then this could contribute (to some extent) to better outcomes in the future. Moreover, we do present some evidence (albeit very limited) of how BSG may have been achieving its longer-term intended outcomes and thus contributing to the wider Scottish Government policy impacts with regards to children and their families.

The logic model for BSG is below at Fig. 1.

The short-term BSG policy outcomes are highlighted in grey. The first three (light grey) are outcomes related to process evaluation and the others are related to policy impact evaluation. The focus in this evaluation report is primarily on the policy impact though any findings related to process are discussed too.

Short-term outcomes (Process Evaluation)

• Grants are well promoted
• Application process is clear and easy
• Payments are well administered

Short-term outcomes (Policy Evaluation)

• Grants are taken up
• Grants reach people at key transition points in child's life
• Low-income families supported at key transition points in child's life
• Increased regular income

Medium-term outcomes

• Reduced pressure on household finances
• Increased child-related spend
• Children able to participate in social and educational opportunities

Long-term outcomes and impacts

The long-term outcomes and impacts in the logic model relate not only to BSG but to the wider government outcomes for children and their families, and are influenced by all social security interventions, as well as other interventions designed to support families with children across the Scottish Government. As such, BSG will play an important, but not exclusive, role in contributing to these.

Evaluation questions

Below are the key questions that informed the evaluation design:

1. To what extent did BSG achieve its short-term, medium-term and longer-term policy outcomes?

2. Is there any evidence of BSG contributing to positive development against the wider, long-term government policy impacts in relation to outcomes for children?

3. What are the implications of the evaluation findings for future policy development?

Summary of data sources

In accordance with the evaluation strategy, multiple data sources fed into the evidence collected and these are descibed below:

Bespoke Commissioned Research

ScotCen Social Research was commissioned by the Scottish Government to carry out qualitative research with BSG recipients to explore any impact BSG may have had on their lives.

The qualitative research mainly focussed on families' experience of receiving the benefit and the ways, and extent to which, the impacts mapped onto BSG policy objectives. The findings have been used alongside the other data sources in this report to provide a comprehensive understanding of the implementation and impact of BSG.

In-depth qualitative interviews were conducted by telephone^[2] between March and July 2020 with 36 people who have received any of the BSG payments for their child (some of them claimed more than one payment but for different children). The full report from ScotCen is available at Annex B and the key conclusions have been incorporated into the findings below.

Official Statistics

Social Security Scotland collects certain information on BSG applications, payments, and clients in the process of delivering the benefits. Some of this information is published as official statistics and is used in this evaluation report supplemented with additional data tables based on the same information. All official statistics discussed in this report are the latest available and cover the period up to 31 August 2020, with the exception of equalities data that is available for the period up to 31 May 2020.

Limitations

This section explains what we can and cannot determine from the data available, and how this influences the extent to which we can draw conclusions about the overall impact of BSG.

Limited quantitative data: Existing survey datasets (e.g. Family Resources Survey and Scottish Household Survey) were inspected to see if they could be used to assess the performance of BSG against its intended outcomes in a more robust way, for example allowing attribution of the role of BSG in observed changes in the long-term outcomes and impacts generated by the sum of all government policies. However, the subset of BSG recipients in those surveys was too small for any robust analysis. Therefore, the only quantitative data used are official statistics on applications and payments.

Role of qualitative research: This evaluation is largely dependent on the findings emerging from bespoke qualitative research commissioned by the Scottish Government. As would be expected with qualitative work, findings are not representative across the entire BSG caseload. There were limitations on the diverse representation across subsets of clients and inherent reliance on recipients' perceptions of impacts, which are outlined in further detail in the full qualitative findings report available in Annex B.

Longer-term impacts will take time to determine: To understand the true impact of BSG, we would ideally measure whether there is lasting change in the longer-term outcomes and impacts, and be able to isolate the influence of BSG from the other factors contributing to this. This is difficult due to the considerations regarding attribution outlined above.

Future research: Some information, particularly around BSG recipients' experience of Social Security Scotland, is still to be analysed and so cannot feed into this evaluation. In August 2020, the Social Security Scotland Satisfaction Survey was sent to over 165,000 people who had received a Social Security Scotland benefit, or reached decision stage on a benefit application. The survey collects equalities information as well as client experiences of receiving the relevant benefits and interacting with Social Security Scotland. Depending on response, this may contribute to outstanding data gaps in this area.

As it was not possible (by the time this evaluation was conducted) for families to have received all three payments in relation to the same child (if they remain eligible), we intend to commission a further evaluation to report in 2025 on these families' experiences.