Devolved benefits: evaluating the policy impact

Equality and Welfare : Evaluating the Policy Impact of Devolved Benefits

The social security programme represents an ambitious agenda for changing the benefits support that is made available to the people of Scotland. This strategy sets out how policy evaluation of the first wave of benefits can help assess their impact against set goals and on achieving the key principles governing our social security system, to ensure continuous improvement.

Initially this strategy focusses on the benefits being introduced up to early 2020. These are:

• Universal Credit Scottish choices – administrative changes to a reserved benefit, Universal Credit, giving the ability for clients to request more frequent payments and payment of the housing element direct to their landlord
• Carer's Allowance Supplement – a top-up to the main Carer's Allowance, increasing this by 13%
• Best Start Grant, replacing Sure Start Maternity Grant for pregnancy and birth payment, and with additional payments at early learning and school stages
• Funeral Support Payment, to assist relatives on low-income benefits with the cost of funerals
• Young Carer Grant, to enable 16-18 year-old carers to improve their quality of life
• Job Start Payment (previously called Job Grant), to support young people who are returning to work after a period of unemployment

Further benefits, notably disability assistance that will replace Disability Living Allowance, Personal Independence Payment and Attendance Allowance, and the new Scottish Child Payment, will be introduced from 2020. The approach set out below is likely to also apply to these benefits, but their level of spending and the complexity of their delivery processes mean that there may be additional considerations that will need to be developed in due course.

The role of policy evaluation

The devolved benefits are being delivered in line with the Social Security Principles. Figure 1 below sets out how we gather information on how we have delivered the Principles. Social Security Scotland has a large programme of engagement with those who receive benefits in order to show whether the service has been designed with their needs in mind and delivered with dignity and respect. Further details of how this is done, and the eight principles, are set out in Our Charter, which explains what the principles mean in practice and what people are entitled to expect from the Scottish Government and Social Security Scotland.

'Policy evaluation' is the focus of this strategy, and explores the impact of the benefit system in achieving the government's objectives, particularly whether social security has been an investment in people, reduced poverty, and brought value for money. Evaluation itself is a key commitment set out in Our Charter and the evidence we gather will contribute to delivering the Charter Measurement Framework which shows how we have met our commitments. Evaluation also plays a key role in ensuring that we fulfil our obligations under equalities legislation, both by measuring how the impact varies by equality group, and how this and implementation of policy accord with a benefit's Equality Impact Assessment.

Figure 1 – The contribution of policy evaluation and other types of analysis to delivering social security principles

The overall outcomes that we are trying to change in society are affected by a range of factors of which social security is just one. It is not the level of income alone that makes it a key factor, but also how the benefit is delivered. This needs to be done at the right time and tackle the factors which affect quality of life, whether this is through extra income when costs are heightened, or a grant to unlock potential at key moments.

Social security payments are, therefore, also a tool to empower people at key moments. The evidence on the steps that people can take when they have support when they need it, and how their quality of life changes, is therefore an important link between the policies implemented and their longer-term aims. Figure 2 illustrates mechanisms whereby benefits have a chain of impacts. Each benefit can have its own 'theory of change' model, but this one demonstrates the general idea.

Most of the benefits introduced to date are one-off payments, so the outcomes that we look for will primarily be at a more immediate level, although the intent is that this will promote better outcomes in the long term.

We acknowledge that there are impacts in the way the benefit is delivered, in that we want benefit recipients to be treated with dignity and respect. These impacts will be examined through process evaluation and the activities of Social Security Scotland in gathering information on their clients' experiences.

The questions that are for policy evaluation to answer, then, are:

• How do we measure these intermediate impacts?
• How do we identify the contribution that our benefits in particular have made?

As we look at longer-term outcomes, it is more difficult to attribute change to a specific policy. But we can find out whether the benefit did what we hoped in terms of more immediate impacts, and use other evidence to help us establish whether the policy helped to improve the higher level outcomes (the contribution).

Figure 2 – Theory of Change for Wave 1 social security benefits.

The components of policy evaluation

Evaluation is a collection of approaches, rather than a single methodology. We make use of a range of data sources and tools in order to build up the picture of effectiveness. Figure 3 sets out the various types of data and what they are designed to show.

Figure 3: Policy Evaluation data roles

The three components of our policy evaluation strategy are designed to make the best use of resources, and perform different roles.

Management or Agency Information is designed to be collected routinely wherever possible while the benefit is delivered and can provide immediate information on effectiveness of delivery, and the characteristics of clients, including their membership of equality groups, so that we can ensure that benefits fairly reflect those they are trying to help. In addition we intend to report on Scottish government activity in fulfilment of the "A Better Future" section of the Social Security Charter Measurement Framework.

Survey data measure long-term outcomes across society, and show whether the outcomes we want to change are moving in the right direction at the pace we wish to see. In themselves, they are not likely to be sufficient to disentangle the overall impact of our benefits from other factors, but they are crucial in informing the long-term strategy.

Commissioned data can fill gaps in management and survey data, deploying tools such as qualitative research, which can explore user experience and outcomes. They could provide the insight into the chain of events that leads to longer-term outcomes.

Models of evaluation

In the short term, the type of evaluation strategy we use is influenced by the nature of the policy and the availability of both resources and data. There is a trade-off between investment in data collection (time and money) and the quality of evaluation (insight and breadth).

In this respect and in very broad terms, the two options available to us are standard and extended models of evaluation. The main difference is in the extent to which we decide to invest in additional data and information collection specifically for the purposes of evaluation.

The standard model of evaluation largely uses data that is collected for other purposes, supplemented by qualitative data collection. More specifically it uses:

• Existing management data collected for the purposes of delivery, limited to overall volume of cases, some geographical breakdown and some information collected in order to determine eligibility (e.g. qualifying benefits such as Universal Credit)
• Data collected for the purposes of gathering customer insights. This includes the Social Security Experience Panels of people with lived experience of benefits, which helps to inform our design of services, and Social Security Scotland client surveys. Changes can be made to these to gather some limited information on the impact of a benefit on recipients' lives, and seek permission for qualitative research (see below)
• Existing population survey data on employment, poverty, health and wellbeing
• Small-scale qualitative research studies with clients and stakeholder organisations.

This will tell us about the numbers of people claiming, give us some information on the profile of applicants, and provide research from clients on whether the short-term changes are beginning to happen.

After a period of ensuring the policy has been fully embedded, it should be possible to report some information on the short- and medium-term impact of these benefits around 18-24 months after launch date. This will be the approach we deliver as a minimum.

However, systems need to develop over time in order to ensure that this picture can be enhanced for more significant insight into impact. If resources and ethics allow, the three building blocks of evaluation may be improved to move towards an extended model which gathers more data in greater breadth and depth as follows:

• Management data can be expanded through using additional questions in the application process on applicants' circumstances or requesting additional data from partner organisations such as HMRC and DWP solely for the purposes of evaluation

• Another, more ambitious, approach is to link data held by Social Security Scotland with other datasets, for example in the NHS, HMRC and DWP. Data linkage is a promising approach, but is a significant resource commitment, not least because of data protection challenges. There are data linkage projects being explored across government, with significant engagement from academia, that we plan to tap into

• Additional questions can be added to population surveys to tailor them to policy-specific questions. The process usually takes a few years to develop and test, and there is a financial cost and trade-offs to consider

• Commissioned work can be extended to surveys to gather representative samples of benefit claimants and explore their experiences in detail, which enables differences between groups to be compared.

Work is underway to improve data collections, both for the benefits in this strategy, and for the disability benefits to come, and we will implement small changes as quickly as possible. We will keep our approach under review, including when new resources may help to get valuable new information.

Timeline

The timeline in Figure 4 shows when we anticipate reporting on each of the devolved benefits under the standard approach. Two benefits, Best Start Grant and Young Carer Grant, extend over a number of years, as individuals may be eligible for a number of one-off payments over time for the same child or caring episode. However, interim reports would identify short-term impacts, and follow up with a final report when recipients have experienced all payments of the benefit.

During 2019-20, the main focus will be to improve management data and commission qualitative work on the evaluations of Universal Credit Scottish Choices, Best Start Grant and Carer's Allowance Supplement.

Figure 4 – timeline of evaluation of Wave 1 benefits