Carer's Allowance Supplement: evaluation

Methodology

This chapter provides an overview of the evaluation approach for CAS. It introduces the logic model and research questions driving the evaluation activities, and gives a summary of the different data sources used.

Overview of evaluation design and logic model

The approach to evaluating the policy impact of CAS uses a theory of change model. This approach uses logic models to show the mechanisms whereby interventions (such as CAS) have a chain of short and medium-term outcomes that, if met, can contribute to longer-term outcomes.

The high-level outcomes that the Scottish Government are trying to influence with regards to carers, such as improved health and wellbeing, improved quality of life, increased feelings of control and empowerment and increased societal recognition will take time to determine and are affected by a range of factors of which social security is only one.

As a result it is difficult to measure and attribute change in these outcomes to a single benefit, such as CAS. However, we can reasonably expect that if success against short and medium-term policy outcomes associated with CAS is achieved, then this could contribute (to some extent) to better outcomes in the future.

The logic model for CAS is below.

The short-term and medium-term CAS policy outcomes are highlighted in grey in the logic model. These outcomes are those which are linked to the intentions of CAS itself and are highlighted below:

Short-term outcomes

• Payments were made to as many eligible people as possible
• Carers feel that the process of receiving carer benefits is user friendly and simple
• Carers understand why the payment was made
• Carers feel they have been treated with dignity and respect and have a positive experience of Scotland's social security system

Medium-term outcomes

• Carers feel CAS has had a positive impact on financial wellbeing
• Carers feel CAS has had a positive impact on their quality of life (including physical health, mental health and wellbeing and feelings of control and empowerment)
• Carers feel CAS has made them feel more recognised for the role they provide

The long-term outcomes in the logic model are highlighted in pink and relate to wider government outcomes for carers. These are impacted by all social security interventions, as well as other interventions designed to support carers across the Scottish Government. As such, CAS will play an important, but not exclusive, role in contributing to these. These outcomes map on closely to those that were developed by Scottish Government officials with the Carer Benefit Advisory Group^[12] and those that are outlined in the Carers strategic policy statement: consultation document^[13].

Long-term outcomes

• Carers feel supported to look after their own health and wellbeing
• Carers feel supported to improve their own quality of life
• Carers have an increased sense of control and empowerment over their lives
• Society recognises and values the role that carers fulfil

Evidence will be collected as to the extent to which CAS has achieved the short and medium-term policy outcomes. We can expect that if success has been achieved, this might have a positive contributory impact on the wider outcomes for carers in the long-term. However, we will be unable to measure this in any robust way with the data available, given that CAS is just one intervention that we may expect to be feeding in to these wider outcomes.

Evaluation questions

The evaluation questions were developed in partnership with key stakeholders, such as the Carer Benefits Advisory Group, to address the objectives of this evaluation^[14]. Below are the three key questions that informed the evaluation design:

• 1. To what extent did CAS achieve its short-term and medium-term policy outcomes?
• 2. Is there any evidence of CAS contributing to positive development against the wider long-term outcomes for carers?
• 3. What are the implications of the evaluation findings for future policy development?

Summary of data sources

In accordance with the evaluation strategy, multiple data sources fed into the evidence collected and these are described below.

Official Statistics

The DWP runs a scan of the information they hold on those in payment of Carer's Allowance twice a year to identify those eligible for CAS on each of the eligibility dates. Scans are run around six weeks after each CAS eligibility date. DWP then securely transfers the necessary data to Social Security Scotland to allow them to make CAS payments.

A cut of data is extracted from the Social Security Scotland payments system in the month following CAS payments being made. This includes information about each carer that received a payment for any one of the eligibility dates going back to 2018/19. For each carer the data extract includes their title, postcode, whether they have received a payment for each of the eligibility dates, and their age at each of those eligibility dates. This data is used to produce statistics on CAS available on the Scottish Government website^[15].

Experience Panels Research

The Experience Panels are made up of people who have recent experience of at least one of the benefits that are being devolved to Scotland. Over 2,400 people registered as panel members when the Experience Panels launched in 2017. The Experience Panels includes around 500 people with experience of Carer's Allowance.

In May and June 2019, these panel members were invited to take part in a survey by Scottish Government researchers to get some feedback on their experience of CAS^[16]. In total, 129 responded to the survey (a response rate of 25%). Of these, 114 panel members had experience of CAS, and therefore completed the whole survey.

Bespoke Commissioned Research

Ipsos MORI was commissioned by the Scottish Government to carry out qualitative research with carers exploring any impact CAS may have had on their lives.

At the time of comissioning, the Scottish Government's overarching aim was to "support carers to protect their health and wellbeing, so they can continue to care if they wish, and have a life alongside caring"^[17]. This aim has undergone a degree of change in the intervening periods and, at this time the Scottish Government's overall aim is that "carers are supported on a consistent basis to allow them to continue caring, if that is their wish, in good health and wellbeing, allowing for a life of their own outside of caring"^[18]. However, it is the initial aim that governed commissioned evaluation activity.

The qualitative research mainly focussed on carers' experience of receiving the benefit and the ways, and extent to which, the impacts of receiving CAS mapped on to CAS policy objectives (and short-term and medium-term outcomes outlined above). The findings have been used alongside the other data sources in this report to provide a comprehensive understanding of the implementation and impact of CAS.

In-depth qualitative interviews, lasting around 45 minutes to 1 hour, were conducted with 41 carers across Scotland who had received CAS within the last year. Fieldwork took place in March and April 2020. The full report from Ipsos MORI is available at Annex A and the key conclusions have been incorporated into the findings below.

Limitations

This section explains what we can and cannot determine from the data available, and how this influences the extent to which we can draw conclusions about the overall impact of CAS.

Official Statistics: Official Statistics for CAS are produced using data extracted from the Social Security Scotland payments system. As a result there are limitations as to what Social Security Scotland can provide on client characteristics across the caseload - including data on protected characteristics (with the exception of gender and age) and socio-economic data (such as income or employment status).

Role of qualitative research: Because of the limitations associated with the data used for Official Statistics this evaluation is largely dependent on the findings emerging from bespoke qualitative research commissioned by the Scottish Government. As would be expected with qualitative work, findings are not representative across the entire CAS caseload. There were also limitations to the methodology of the commissioned research itself, including lack of diverse representation across subsets of clients and inherent reliance on carer perceptions of impacts rather than definitive impacts, which are outlined in further detail in the full qualitative findings report available in Annex A.

Focus on contribution rather than attribution: As noted above, CAS is just one intervention across social security and wider governmental support for carers. As such there are other factors that we may expect to feed into wider outcomes for carers - including impacts associated with carers' rights to advice and support under the Carers (Scotland) Act 2016. Therefore, we only have a partial understanding of how CAS is related to longer-term outcomes and we can only assess the contribution that CAS may have had by assessing the extent to which it has achieved its own policy objectives. We cannot directly attribute long-term outcomes to CAS, and overall, we are unable to rule whether any impacts we may see have occurred as a result of other factors.

Longer-term impacts will take time to determine: Changing financial and health and wellbeing outcomes is a complex process that requires time. To understand the true impact of CAS, we would ideally measure whether there is lasting change in the longer-term - and be able to isolate the impact of CAS from the other factors contributing to this. This is difficult due to the considerations regarding attribution outlined above.

Future research: Some information, particularly around carer experience of Social Security Scotland, is still to be collected and so cannot feed into this evaluation. In August 2020, the Social Security Scotland Satisfaction Survey was sent to over 165,000 people who had received a Social Security Scotland benefit, or reached decision stage on a benefit application. The survey collects equalities information as well as client experiences of receiving the relevant benefits and interacting with Social Security Scotland. Depending on response, this may contribute to outstanding data gaps in this area.