Young Carer Grant: interim evaluation

Methodology

This chapter provides an overview of the evaluation approach for Young Carer Grant. It introduces the logic model and research questions driving the evaluation activities, and gives a summary of the different data sources used.

Overview of evaluation design and logic model

The approach to evaluating the policy impact of Young Carer Grant uses a "theory of change" model. This approach uses logic models to show the mechanisms whereby interventions (such as Young Carer Grant) have a chain of short and medium-term outcomes that, if met, can contribute to longer-term outcomes.

The high-level outcomes that the Scottish Government are trying to influence with regards to carers, such as improved health and wellbeing, increased societal participation, improved quality of life, increased feelings of control and empowerment, and increased societal recognition, will take time to determine and are affected by a range of factors of which social security is only one.

As a result, it is difficult to measure and attribute change in these outcomes to a single benefit, such as Young Carer Grant. However, we can reasonably expect that if success against short- and medium-term policy outcomes associated with Young Carer Grant is achieved, then this could contribute (to some extent) to better outcomes in the future.

The logic model for Young Carer Grant is below:

The short-term Young Carer Grant policy outcomes are relevant to the process of delivery and are highlighted in light blue in the logic model while the medium-term policy outcomes are highlighted in light pink. These outcomes are those which are linked to the intentions of Young Carer Grant itself and are outlined below:

Short-term outcomes (process)

• Carers feel the application process is clear and easy
• Payments are administered well
• Payments are made to as many eligible carers as possible
• Carers feel they have been treated with fairness, dignity and respect
• Carers have a positive experience of Social Security Scotland

Medium-term outcomes (policy)

• Carers are better off financially
• Carers can engage in opportunities that are the norm for their non-caring peers
• Carers feel a sense of choice and control
• Carers feel the grant has had a positive impact on their lives
• • Carers feel they have been recognised for the care they provide

Long-term outcomes

The long-term outcomes in the logic model are highlighted in dark pink and relate to wider government outcomes for carers. These are affected by all social security interventions, as well as other interventions designed to support carers across the Scottish Government. As such, Young Carer Grant will play an important, but not exclusive, role in contributing to these. These draft outcomes map on closely to those that were developed by Scottish Government officials with the Carer Benefit Advisory Group^[6] and those that are outlined in the Carers strategic policy statement^[7]. This wide approach has resulted in some outcomes which are benefit specific and/or focussed on financial entitlement while others are more carer focused and take in to account of carers' life outside of caring.

• • Carers can participate fully in society and have a life outside of caring
• • Carers have a sense of control and empowerment over their lives
• • Carers feel supported to look after their own health and wellbeing
• • Carers feel that they have a good quality of life
• • Carers feel recognised by society for the role they provide

Evidence will be collected as to the extent to which Young Carer Grant has achieved the short and medium-term policy outcomes. We can expect that if success has been achieved, this might have a positive contributory impact on the wider outcomes for carers in the long-term. However, we will be unable to measure this in any robust way with the data available, given that Young Carer Grant is just one intervention that we may expect to be feeding in to these wider outcomes.

Summary of data sources

In accordance with the evaluation strategy, multiple data sources fed into the evidence collected and these are described below.

Official Statistics

Social Security Scotland collects certain information on Young Carer Grant applications, payments, and clients in the process of delivering the benefits. Some of this information is published as official statistics and is used in this evaluation report. All official statistics discussed in this report are the latest available and cover the period from 21 October 2019 to 30 April 2021^[8] (approximately 18 months of applications). Figures are rounded for disclosure control and may not sum due to rounding. Data on client diversity and equality of application outcomes for clients applying to Social Security Scotland covers the period from June 2020 to November 2020^[9].

Social Security Scotland research activity

The first Social Security Scotland Client Survey ran from 17 August to 25 September 2020^[10]. It was open to everyone who had received a Social Security Scotland benefit or had received a decision on an application for a benefit from the beginning of Social Security Scotland in September 2018 to July 2020. The survey collected equalities and socio-economic information on applicants and asked about key aspects of respondents' experience of Social Security Scotland, and of receiving the benefits.

3,253 people responded to the survey (around 2% of those who were sent invites) - and 2% of these had applied for Young Carer Grant which equates to 71 responses. This will be referred to as the "Client Survey" for the remainder of this report. Please note the number of respondents providing a valid answer to each individual question/statement varied slightly in the ranges shown.

Bespoke Commissioned Research

Ipsos MORI Scotland was commissioned by the Scottish Government to conduct qualitative research with young carers, and stakeholders that work with them, to explore the experience of, and early impact of, Young Carer Grant on the lives of those that receive it.

The research mainly focussed on carers' experience of receiving the benefit and the ways, and extent to which, the impacts of receiving Young Carer Grant mapped on to Young Carer Grant policy objectives (and short-term and medium-term outcomes outlined above). However, within this there were additional aims, such as exploring barriers to claiming Young Carer Grant and the impact of certain eligibility criteria.

The findings have been used alongside the other data sources in this report to provide a comprehensive understanding of the implementation and impact of Young Carer Grant at this early stage of delivery.

In-depth qualitative interviews, lasting around 45 minutes to 1 hour, were conducted with 22 young carers across Scotland and 6 interviews with key stakeholders were also held. Fieldwork took place via telephone or Zoom between December 2020 and February 2021. The full report from Ipsos MORI Scotland is available at Annex A and the key conclusions have been incorporated into the findings below.

Limitations

This section explains what we can and cannot determine from the data available, and how this influences the extent to which we can draw conclusions about the early impact of Young Carer Grant.

Role of qualitative research: This evaluation is largely dependent on the findings emerging from bespoke qualitative research commissioned by the Scottish Government. As would be expected with qualitative work, findings are not representative across the entire Young Carer Grant caseload. There were limitations associated with the recruitment of young carers (especially attempts to ensure diverse representation across subsets of clients), and inherent reliance on recipients' perceptions of impacts. These limitations are outlined in further detail in the full qualitative findings report available in Annex A.

Role of Social Security Scotland research: The Social Security Scotland Client Survey statistics contained in this report are based on 71 responses from clients who applied for Young Carer Grant (and the number of respondents providing a valid answer to each individual question/statement varies within the ranges shown). While the survey results provide rich insight into some of the experiences of Social Security Scotland clients, this small sample size means that we cannot assume that the results represent the views of Young Carer Grant clients as a whole, and are not generalisable. Furthermore, just 26 of the 71 respondents only applied for Young Carer Grant, while the remainder said they had experience of applying for or receiving other benefits from Social Security Scotland, (for example, 40 said they had received Carer's Allowance Supplement). These respondents may have been basing answers on their experience of applying for/receiving both benefits, which should be taken into account when interpreting the findings.

Focus on contribution rather than attribution: As noted above, Young Carer Grant is just one intervention across social security and wider governmental support for carers. As such there are other factors that we may expect to feed into wider outcomes for young carers - including impacts of the Young Scot young carers platform and carers' rights to advice and support under the Carers (Scotland) Act 2016. Therefore, we only have a partial understanding of how Young Carer Grant is related to longer-term outcomes and we can only assess the contribution that Young Carer Grant may have had by assessing the extent to which it has achieved its own policy objectives. We cannot directly attribute long-term outcomes to Young Carer Grant, and overall, we are unable to rule whether any impacts we may see have occurred as a result of other factors.

Longer-term impacts will take time to determine: To understand the true impact of Young Carer Grant, we would ideally measure whether there is lasting change in the longer-term outcomes and impacts, and be able to isolate the influence of Young Carer Grant from the other factors contributing to this. This is difficult due to the considerations regarding attribution outlined above. However, the full evaluation of Young Carer Grant scheduled for 2023/24 will provide further learning of the impact of the benefit once it has been live for at least three years.