Universal Credit Scottish choices: evaluation

Introduction

Universal Credit Scottish choices is a flexibility within Universal Credit (UC) which enables eligible claimants in Scotland to choose to be paid twice monthly (More Frequent Payments) and/or have the housing costs in their award paid directly to their landlord in both the private and social rented sectors (Direct Payments to Landlord). The intention of these flexibilities is that recipients have more control over their budgeting, and to help ensure that their rent is paid on time.

This evaluation is designed to assess the evidence for a range of intended impacts for Scottish choices, including people's awareness of Scottish choices, reasons for choosing whether or not to use the flexibilities, impact of Scottish choices on claimants' ability to manage household budgets and any related impacts on rent arrears or evictions. The report will provide evidence for policy-makers in the Scottish Government to inform whether the Scottish choices policy has had a positive impact on clients and what could be done to make it more fit for purpose.

Background and context

UC Scottish choices resulted from the Scottish Government using the devolved powers detailed within the Scotland Act 2016 to make changes to the administration of UC, the overall delivery of which remains reserved to the UK Government. The purpose of Scottish choices was to provide greater flexibility and control over personal income for people in Scotland in receipt of Universal Credit. As at November 2020, 470,000 were on Universal Credit in Scotland^[1]. The delivery of Scottish choices was the culmination of extensive and successful joint working between the DWP and SG.

UC Scottish choices give people the ability to decide whether to receive their UC award either monthly or twice monthly and to have the housing costs in their UC award paid to themselves or directly to their landlord in both the private and socially rented sectors. Scottish choices were implemented on 4 October 2017 for new full service UC claims in Scotland, and availability was extended to all full service claims from 31 January 2018. Full service claims are UC claims made online which gradually replaced claims for legacy benefits (Income-based Jobseeker's Allowance, Income-related Employment and Support Allowance, Income Support, Working Tax Credit, Child Tax Credit, Housing Benefit) in areas referred to as 'full digital service area' or 'full service area'. As of December 2018 full service areas were rolled out across all local authorities in Scotland.

While not a specific benefit, Scottish choices represents a key flexibility in the devolved powers of the Scottish Government, which is designed to improve the experience of recipients of the benefit system in Scotland.

Evaluation methodology

The Scottish Government sets out its methodology for evaluating devolved benefits in Evaluating the Policy Impact of Devolved Benefits, published in November 2019. This approach has been taken for all devolved benefits to date, and this report is the third of the series to be published, after evaluations of Carer's Allowance Supplement and Best Start Grant.

Each benefit evaluation follows a similar approach, drawing on a range of sources to identify impact, including management information, population survey data and commissioned qualitative work. In order to identify the impacts that are to be assessed, each benefit has a "theory of change" which sets out the chain of impacts that are intended to result from the benefit, through to long-term improvements in people's lives. This exercise or "logic model" provides clarity on what the benefit is intended to achieve. In an evaluation context, it identifies the short- and medium-term outcomes which are indicators that the benefit is achieving what is intended at an early stage. As more data is collected, evidence accumulates over time to verify longer-term outcomes, and the theory of change, combined with evidence gathered on immediate impacts, helps to confirm progress is related to the benefit.

The "logic model" for Scottish choices is set out in Figure A. From left to right, the outcomes that may arise over time are set out, beginning with short-term immediate impacts, and finally with longer-term change. An additional issue with long-term outcomes is that other factors will be at work in determining the situation of individuals with regard to their income, employment, health and wellbeing and inequality. These include other benefits, changes in employment and wider economic change. As such the ability to attribute change to the original intervention is reduced. However, the value of the logic modelling approach is that it sets out whether the benefit made a contribution to the overall outcome, and thus whether it has been a valuable addition to policy.

The logic modelling for UC Scottish choices identifies a number of outcomes, which have been tested in evaluation. For the short term, immediate impacts are often related to the processes associated with applications, and awareness raising during administration of UC and through advice services. These are often referred to as "process evaluation", and not only shed light on how effective administration has been experienced by recipients, but may also be associated with later outcomes which are taken into account in "policy evaluation". This evaluation combines aspects of both these approaches.

Short term outcomes assessed in the "process" include the following:

• Choices are well promoted among eligible clients and landlords
• Application process is clear and easy
• Choices are administered well

As we see later in this report, some of these are more straightforward to measure than others – there are objective measures for the numbers receiving a benefit, but other aspects, such as adequate publicity, are may only be measurable by asking stakeholders what their perception was.

These processes may have immediate impacts on perceived awareness shown by clients and landlords.

• Eligible clients and landlords are aware of choices
• People find accessing the choices straightforward
• Landlords make tenants aware of choices

These should then lead to an immediate key outcome:

• Choices are taken up if suitable

Once a client has taken up either More Frequent Payments or Direct Payments to Landlords, then a range of medium-term outcomes are hoped for:

• People are able to manage household budgets better
• Reduced risk of unsustainable debt
• People able to manage debt or rent payments better
• Reduced risk of rent arrears, evictions and homelessness
• Landlords receive rent on time
• Improved housing security for tenants

Again, objective measures of these outcomes vary in their availability of data that measures this, or even an agreed indicator.

Finally, there come longer term outcomes that are of wider interest across government and society. Attributing changes in these outcomes to a single policy such as Scottish choices would be difficult, given the range of societal factors at work. However, it may be that for certain individuals, greater control of finances that comes with Scottish choices, may be crucial to building a platform for increased prosperity and further outcomes.

• Reduced poverty
• Reduced inequality of outcomes for families and children
• Improved health and wellbeing

Data Sources

The evaluation approach draws upon different sources of data to shed light on the outcomes discussed above. It makes use of existing data that is collected for various purposes, and supplements it with commissioned work to ensure that evidence gaps are filled, and firmer conclusions can be drawn from other sources.

Management Information

Unlike devolved benefits that are paid by Social Security Scotland, Scottish choices is a flexibility that is implemented by the Department of Work and Pensions (DWP) through its payment of Universal Credit to claimants in Scotland. The DWP provides the Scottish Government with management information on Scottish choices, including the number of people who are offered the choices and the numbers who took them up. The rate of take up is determined as the proportion of those offered Scottish choices who took up that offer. Data are also provided by DWP on the amount of reversion to default payment methods for UC (monthly payments and/or payment of the housing element to the recipient). The main benefit of these data for evaluative purposes is primarily for indicating the scale and penetration of Scottish choices with the target group (UC recipients), who claims it, and how, and how sustained the choices are.

Survey data

Longer-term outcomes can be measured by population surveys, and data are usually gathered from respondents on their financial situation. In the case of Scottish choices, the surveys of primary interest are the Scottish Household Survey and the Family Resources Survey. While these can be used to build up a population picture of outcomes such as poverty and employment, the fact that they are sample surveys means that the level of statistical precision of estimates drawn from respondents must be quantified so that only an appropriate level of confidence is given in findings when making comparisons. In this instance, the sample sizes that can be used are a sub-group of the population because we are looking at UC recipients only. This means that the statistical error assigned to estimates of outcomes means that small differences are unlikely to be statistically significant. Further comparison with previous years is difficult, because UC was implemented selectively in different areas and to different groups over time, meaning that the groups on UC may vary by characteristics, year to year. A limited comparison may be allowed with those who receive legacy benefits, due to larger numbers, however this also must be treated with caution, as the characteristics of the two groups may vary.

Commissioned research

As noted above, the role of commissioned work is to provide a fuller picture to complement existing evidence. In the case of this evaluation, it is primarily to provide a more in-depth understanding of clients' experience, their attitudes and their perception of the outcomes that have been secured through Scottish choices. Qualitative work is crucial to providing client testimony on the range of short- and medium-term outcomes which build up to long-term impact, in particular, whether recipients perceive they are better able to manage their finances and landlords feel that that their rental income is more secure.

Qualitative data, while it cannot be held to representative on its own, can provide insight into how Scottish choices are experienced, and whether there is evidence that it works as intended, even if quantitative data is required to tell us how widespread it is. When data are provided on the scale of flexibility exercised, and long-term outcomes on poverty and indebtedness, the commissioned work may outline whether the intentions and assumptions behind Scottish choices were plausible.

In this evaluation, the Scottish Government appointed Ipsos MORI Scotland to carry out the qualitative element of the evaluation involving in-depth telephone interviews, conducted between May and early September 2020, with:

• 45 people who were claiming Universal Credit in Scotland
• 20 Landlords (including a mix of private and social landlords), and
• 14 key informants, working with either landlords or people on Universal Credit.

In addition, a short online survey was completed by 260 landlords, including 178 private and 81 social landlords.

UC claimants were required to have been on the benefit for at least 3 months, and the majority (31) were recruited as earlier respondents to the Scottish Household Survey. Accordingly the vast majority of Scottish choices will have been made by interviewees prior to the Coronavirus (COVID-19) pandemic. Their decisions about Scottish choices are thus are less likely to have been affected by any issues regarding face-to-face access to advisory services, or changes in the client base for UC that may been associated with economic difficulties as a result of lockdown.

Interviews were conducted by telephone in line with Scottish Government Social Research guidance for safe working during the pandemic (which precludes face-to-face interviewing).

The questions explored with UC claimants include the following:

1) Are those eligible aware of the choices available to them?

2) Do they find accessing the choices straightforward and understand how they would affect their UC award?

3) If people were aware of the choice but chose to receive their UC in the standard fashion, what was the reason for this and are they open to the possibility of making a different choice later?

4) If people chose to receive UC award twice-monthly, why did they make this choice?

5) If people chose to have the housing costs in their UC award paid directly to their landlord, why did they make this choice?

6) If people chose to be awarded UC twice-monthly and/or to have the housing costs in their UC award paid directly to their landlord, but then reverted back to the default payment method(s), what was the reason for doing so?

7) If people decided to accept the UC Scottish choices, what influence on their ability to manage household budgets did this have and did it have any knock on effects on debt, rent arrears, evictions or homelessness?

8) From the perspective of eligible recipients, is there a better way to operate the offer of UC Scottish choices, and if so, how?

9) Are there other UC flexibilities that people would like?

Questions explored with landlords were as follows:

1) Do landlords make their tenant(s) aware of UC Scottish choices?

2) Do landlords encourage or discourage their tenant(s), who are eligible to choose direct payments of housing costs to landlords and do not already have Alternative Payment Arrangements in place, to take up this option? Why?

3) Is there any evidence – even if anecdotal - among landlords that UC Scottish choices help UC recipients pay rent on time?

4) From the perspective of landlords, is there a better way to operate the offer of UC Scottish choices and, if so, how?

5) Are there other choices landlords would want to be available to people in receipt of UC?

6) Are there differences in terms of the above depending on a type of landlords (i.e. social, private)?

The full Ipsos MORI report is published under separate cover as at Annex B to this report.

In the remainder of this report, findings from the data and qualitative research have been organised thematically by the objectives that have been set out above in the "Findings" chapter. The final chapter discusses overall conclusions and identifies policy implications for Scottish choices.