Examining outcomes associated with Social Security Scotland spending: an evidence synthesis

6. Building on and improving the evidence base

Context

As set out in the background section, the Scottish Government has taken a different approach to the rest of UK in designing its new devolved social security system^[66]. Social Security investment represents a sizeable proportion of the overall Scottish budget^[1];. There is a great deal of interest in the new Scottish approach and some bold claims^[67],^[68],^[69],^[70] from some external commentators about the potential impact of SCP in particular.

It is therefore important to invest in gathering high quality evidence to understand how each of these benefits contributes to the achievement of wider Scottish Government outcomes, as well as whether they are achieving their stated short and medium term outcomes. The majority of social security spending over the next five years is currently forecast to be on ADP, CDP and SCP (see 'Distribution of social security spending across benefits' ). Robust monitoring and evaluation needs to be in place for these three benefits.

Most evaluation evidence relates to the first wave of devolved benefits, often from interim evaluations. Ongoing and future evaluation work is summarised in Annex B.

As set out in the background section, this review synthesises evidence of impact to date in relation to: how security spending can help people move into, or stay in, work and participate in society; provide protection and act as a safety net in times of need; and act as early intervention to give people the best possible chances in life. Much of the social security evaluation work conducted thus far has focused on the effectiveness of benefits delivery and operations. This is because a key priority has been to ensure the safe and secure transfer of cases so that clients already in receipt of benefits do not experience an interruption in their payments. Evidence in relation to the effectiveness of delivery and value for money are outside the agreed scope of this review and are being examined in other work. It should also be noted that typically, benefit payments do not undergo value for money assessments because they represent transfer payments within the economy (e.g. from government to citizens), and are normally excluded from the social benefits included in cost benefit analysis as set out in HM Treasury Green Book guidance^[71].

Although outside the scope of this review, it should be noted that policy action in other areas, e.g. employability, housing, mental health, also has the potential to reduce – or increase - demand for social security benefits. It is therefore not possible to evaluate the outcomes of social security policy in isolation, and robust evaluation of these closely-linked policy areas is also important.

Challenges

Producing high quality evidence of the type needed is methodologically challenging and requires significant investment. Figure 3 below summarises some of the main challenges when considering the strengths and limitations of the evidence available to date.

Figure 3 Summary of main evaluation challenges

• Assessing long term change. Assessing the impact of benefits on longer term outcomes (and preventative savings) will take time to observe, even where plans for doing so are in place.
• Need for quantitative data. Robust quantitative evidence, including data linkage, is needed to fully assess impact but is not yet available. Extracting such data requires resources. Qualitative interview data with benefit recipients can help to describe how change happens, but findings cannot be generalised to the wider population.
• Demonstrating causality. Social security benefits are just one element of a wider package of policy interventions and support; this makes it challenging to directly attribute any changes either to a specific benefit, or to benefits more generally.
• Ethical sensitivities. Data collection can be intrusive, particularly for households living in poverty and facing multiple challenges. The use of control groups to measure the causal impact of benefits is also problematic and usually only possible in instances where benefits are rolled out to different areas or groups over a period of time. Collecting more client data can also conflict with the principle of making access to benefits as easy as possible.

Strengths of the current evidence base

The overarching approach to evaluating the outcomes of the social security benefits is set out in two evaluation strategies; the first in 2019^[84] considered devolved benefits and a more recent strategy in 2021 covers plans for evaluation of the devolved disability benefits^[72]. A benefit take-up strategy was also published in 2021^[73] and Scottish Government analysts produce annual estimates of take-up rates for a range of Scottish benefits^[74].

Analytical work is being undertaken to understand the experiences of social security clients. Logic models have been developed for the Carers Allowance Supplement, YCG, FSP, JSP, CWHP and the Five Family Payments. These articulate theories of change and set out the anticipated short and medium term outcomes and longer term impacts of each benefit. Commissioned evaluations have been completed for CAS, FSP, CWHP, JSP and BSF and interim evaluations completed for the Five Family Payments (BSG, BSF and SCP), and YCG. Modelling has also been conducted and published that estimates the impact of benefits on child poverty rates^[75] using micro-simulation, as well as internal work linking the Five Family Payments to the National Performance Framework.

There is clearly a great deal of analysis underway to understand the experiences of social security clients and how the delivery of social security benefits could be improved.

Building on and improving the evidence base

A number of evidence gaps emerge from this evidence synthesis. Addressing these could be the focus of a longer term evaluation framework for the devolved benefits system in Scotland. Identified gaps to date include the extent to which benefits administered by Social Security Scotland support:

• Impact on employment outcomes (beyond initial promising evidence from the Job Start payment and Young Carer Grant)
• Participation in early years education by children and in further education or training by parents (and any contributions to reducing the education attainment gap)
• Economic impacts more broadly
• Longer term health and wellbeing
• Preventative savings over time as a result of reduced demand for public services e.g. health and social care, and also for benefits themselves over time due to factors such as a reduction in future social problems that result from investment in benefits now

To date there is also a lack of evidence of how outcomes vary by equality group; the 2019 benefits evaluation strategy envisages this detailed evaluation will be undertaken^[84] These points are expanded on below.

Impact evaluation – areas for future development

Several evaluations note that a full assessment of progress towards medium and long term outcomes for recently devolved benefits will require more time to have passed since the benefit was implemented as well as access to robust quantitative data. However, although plans for long term evaluation need to be in place, progress against short term outcomes may provide an important indication of progress. For example, the BSG evaluation^[12] stated ‘we can reasonably expect that if success against short and medium term policy outcomes associated with a specific payment is achieved, then this could contribute (to some extent) to better outcomes in the future.’

The long term outcomes that the Scottish Government is trying to achieve, such as improved health, will take time to determine and are affected by a range of factors and other SG government policies and packages of support. Demonstrating causality is methodologically difficult and ultimately assessing the impact of social security benefits in terms of contribution rather than attribution may be a more realistic approach.

There is currently a lack of robust quantitative evidence relating to the wider outcomes associated with benefits administered by Social Security Scotland. Most of the evaluation evidence collected to date is made up of qualitative interviews with benefit recipients and while valuable, remains a subjective assessment of impact.

There is also currently a lack of longitudinal evidence tracking the outcomes for benefit recipients over time although, as set out in Annex B, there are plans to commission longitudinal qualitative research with a small number of recipients as part of the planned Five Family Payments evaluation.

Well-evidenced logic models are an important foundation for evaluation. Because there have been few changes to eligibility or payment amounts (outside of annual uprating), the focus of the evaluation work for these new disability benefits has so far focused on evaluating key reforms related to delivery of the disability benefits. There are therefore currently no detailed impact logic models for the new adult and child disability benefits of the type developed for most of the earlier low-income and carer benefits that set out the anticipated short, medium and long term outcomes of the benefit itself for individuals and society^[76]. A logic model is currently being developed covering both the impact of the reforms on delivery and the relationship between the benefits and how they are delivered with wider societal impacts associated with the new disability benefits; this is intended to complement the low income benefits logic models.

There is relatively little evidence available to understand the impact of benefits on child development, employment, housing, educational attainment, and outcomes associated with the economy. This may be partly due to the type of evaluation evidence available to date and also the demographic characteristics of the benefit recipients who took part in the evaluations of BSG, BSF and SCP. Many were parents of pre-school children who were not actively seeking employment. Internal analysis is being developed to understand how benefits may influence whether individuals move into paid work, work more hours, or earn more through employment. Given that income from employment is one of the three main drivers of child poverty, the lack of published evidence so far relating to the impact on employment was identified as an area where further analytical work would be beneficial.

More detailed evidence in relation to application approvals between groups with different protected characteristics is required to better understand why some groups appear to be more successful in their applications. For example, a 2023 statistical report^[77] analysed application approval rates by equalities groups and found that these can differ by much as 35 percentage points between some ethnic groups (e.g. Hindu 48% compared with Jewish 83%) and by gender (men 69% compared to women 83%). Although such breakdowns are useful, the report does not state whether these differences are meaningful, or what factors might explain them. Additional analysis to provide context and possible explanations for any unexpected differences (e.g. differences in English language ability) would assist with equalities monitoring. Currently it is not possible to say what might be driving such differences, or if action is needed to address them.

Economic impact – opportunities and challenges for future learning

Economic modelling is complex and requires robust quantitative data. Nevertheless, economic analysis techniques have been used in the social security context in Scotland to understand the expected economic impact of benefit expenditure policies. For example, the Outline Business Case^[78] for the agency in Scotland published in 2017 compared the costs and benefits of a range of options of delivering social security assistance in Scotland. This analysis found favourable results for each, compared with the do nothing/continuation of DWP delivery option. It compared the costs of delivery (e.g. staff costs) with the expected benefits of re-distributing revenue from taxes to low income households in receipt of benefits. HM Treasury Green Book guidance assigns a higher true value to expenditure to low income households due to the theory of diminishing marginal utility of income – where the value of an additional pound of income is higher for a low income recipient and lower for a high income recipient^[79].

The Social Security Programme Business Case was also published by the Scottish Government in February 2020 in line with HM Treasury Green Book guidance. It set out a view of the whole life costs and benefits of the Programme over a 30 year time horizon to 2050^[80]. Different options for delivering SCP were also assessed against policy objectives^[81] and a Multi Criteria Analysis was conducted and published that compared the effectiveness of a range of options for delivering Carer Support Payment, against a set criteria of policy objectives^[82].

Although there was empirical and qualitative evidence to show that most families spend - rather than save - benefits payments, thus boosting consumer spending, a more comprehensive economic assessment would be needed to assess the overall net impact (e.g. considering both the costs and benefits) of social security spending on the wider economy).

Improving outcomes for people is likely to require investment in services and benefits. For example, income from social security is only one of the three key drivers related to reducing child poverty set out within the Child Poverty Delivery Plan. With finite resources Ministers need to carefully assess what the optimum and most beneficial package of benefits and services is to achieve defined outcomes. This involves assessing trade-offs and using evidence to understand how spending on both services and benefits contribute to outcomes.

Ongoing and future analysis

Ongoing and planned analysis to evaluate the impact of social security spending is summarised in Annex B – Ongoing and future evaluation plans. Of particular note in relation to impact are the evaluations of the Five Family Payments planned for 2024-25 and including longitudinal research with recipients. The possibility of linking social security data to existing datasets is also being explored^[83]. This could allow the impact of SCP on health and wellbeing outcomes for children and young people to be better assessed.

Moving forward it will be important to continue to ensure that the development of social security policy in Scotland is informed by high quality evidence relating to both the delivery of benefits and the outcomes associated with Social Security Scotland spending.