Disability benefits: evaluation report

Methodology

Logic Model

As outlined in the evaluation strategy, evaluations within this programme of work are supported by the development of a theory of change. This is represented graphically within a logic model. It shows the trajectory from the direct impacts of the policy commitments (short-term outcomes) to the outcomes that arise as a result of these, and other policies related to the delivery of disability benefits (medium-term outcomes). Finally, the long-term outcomes relate to broader impacts on views and experiences with Social Security Scotland.

The logic model for this evaluation was developed collaboratively with members of the Research Advisory Group^[6], and is shown below.

Short term outcomes:

• Guidance and communication about applying for ADP is clear and easily understood
• Case managers, client advisors and local delivery colleagues know how to support disabled people and their representatives effectively
• ADP payments happen at the time people expect them and on the same day they would ordinarily receive/expect to be paid CDP
• Application determinations happen promptly
• A rights-based and person-centred approach are embedded in the CDP-ADP process
• Clients and their representatives understand the information about the process and the options available to them and what is involved in the application process
• Clients and their representatives are reassured that they will continue to receive CDP until they reach the upper age limit, regardless of the outcome of their ADP application
• Clients and their representatives who meet the criteria experience a straightforward journey from CDP to ADP that minimises anxiety and uncertainty about the process
• Clients and their representatives feel empowered and in control of their benefits
• Clients and their representatives feel treated with dignity, fairness and respect.
• Clients and their representatives whose applications are unsuccessful understand the process, the reasons why their application was unsuccessful and their rights to a redetermination and appeal after their determination.

Medium term outcomes:

• Barriers to applying for disability benefits are reduced
• Clients feel that they have a positive relationship with Social Security Scotland
• Trust in Social Security Scotland increased
• Clients have financial stability

Long term outcomes:

• Human rights are respected and protected
• The take up of benefits is maximised
• Poverty is reduced

Data Sources

The evaluation draws on data from multiple data sources to evidence the outcomes outlined in the logic model. Collectively these sources aim to provide a comprehensive understanding of the experiences of clients.

Commissioned research

The Lines Between (TLB), a social research agency, was commissioned by the Scottish Government to conduct research with those making the transition from CDP to ADP. The full report of the commissioned work is published in Annex A.

The commissioned research combined both surveys and interviews. The groups and recruitment approach are outlined below.

Interviews with Social Security Scotland staff

Five interviews took place with Social Security Scotland staff who process CDP to ADP applications in December 2023. This included one group interview with four staff members and one individual interview with a fifth staff member.

Survey of organisations

An online survey was carried out with organisations that support clients and formal representatives applying for ADP while receiving CDP. The Scottish Government contacted third sector organisations that support clients and formal representatives applying for ADP while receiving CDP and requested their participation. TLB emailed the survey link to those who responded. The survey was also disseminated to Social Security Scotland's Local Delivery Teams. The survey was open from January to March 2024 and received 12 responses, 10 from Social Security Scotland local delivery team staff, and two from other organisations that support clients.

Research with clients and formal representatives

The commissioned work focused on the experiences of five main groups:

1. Individuals who applied for CDP then applied for ADP earlier on purpose (early appliers to ADP).

2. Individuals who applied for CDP then applied for ADP to meet the deadline (later appliers for ADP).

3. Individuals who had their DLAC benefit transferred to CDP before they were 16 under Social Security Scotland then applied for ADP earlier on purpose.

4. Individuals who had their DLAC benefit transferred to CDP before they were 16 under Social Security Scotland then applied for ADP to meet the deadline.

5. Individuals who had their DLAC benefit transferred to CDP close to their 18th birthday under Social Security Scotland and then applied to ADP.

Survey of clients and formal representatives

An online survey was sent to clients and formal representatives who had applied for ADP while receiving CDP, or were considering applying. Invitations were sent via Social Security Scotland to a sample of CDP and ADP clients who had recently applied for ADP while receiving CDP, as well as CDP clients who had not yet applied for ADP but are eligible to do so. The first invitation was sent in mid-February 2024. Further invitations were issued to additional samples of clients in late February and mid-March 2024, and the survey closed in early April, and received 89 responses.

The survey focused on clients’ and formal representatives’ views and experiences of applying for ADP while receiving CDP and the reasons that influence the timing of their application. The survey was mostly closed-ended with space for free text comments.

Interviews with clients and formal representatives

Follow-up interviews were conducted with 30 respondents to the survey of clients and formal representatives to explore their experiences in more detail.

Survey respondents were given the option to express interest in taking part in an interview with a member of the TLB research team. Over half (54 per cent, 49) of the 89 survey respondents expressed willingness to participate in an interview.

TLB selected survey respondents to invite to take part in an interview based on a sampling plan. Each interviewee received a £30 shopping gift card as a thank you for their help. Interviews were completed between February and April 2024.

Official Statistics

Social Security Scotland collects information on applications, payments, and clients in the process of delivering the benefits. Some of this information is published online as Official Statistics in development. Official Statistics in development are new or existing statistics which will be tested with users in line with the standards of trustworthiness, quality and value in the Code of Practice for Statistics. The following Official Statistics publications are used as sources of evidence in this report:

• Adult Disabilily Payment – high level statistics to 30 April 2024

This publication is hereafter referred to as “Official Statistics”. Please note the following technical points about how Official Statistics are presented throughout this report:

• Figures are rounded for disclosure control and may not sum due to rounding
• Where stated, secondary analysis has been conducted on rounded figures from published Official Statistics

Considerations and Limitations

This section discusses what can and cannot be determined from the data sources that were available for this evaluation. It also highlights how this influences the extent to which robust conclusions can be drawn regarding the impacts of the policy commitments.

Timing of the research

This research coincided with a change in regulations^[7] which took effect from 20 November 2023 when amendments to sets of principal Regulations for CDP and ADP came into force. Although fieldwork with people who had applied to ADP was not until February 2024, their experience of moving from CDP to ADP may have occurred before the changes in regulations took place meaning that some participants may have had a different experience to others. The researchers have noted throughout this report where this may have impacted the findings of the research.

Commissioned surveys

Participants in the survey were self-selecting, meaning that they actively chose to take part, as opposed to being randomly selected. This may bias the sample towards those more willing or able to take part in research.

There is no aggregated demographic data related to ADP and CDP recipients, so data was unavailable to estimate target quotas for variables such as age, ethnicity and type of disability.

Conclusions about differences in experiences between clients and formal representatives who had received DLAC before CDP and those who had not are limited. While 74 survey responses were received from clients and formal representatives whose award had transferred from DLAC, only 15 had only been in receipt of CDP. Similarly, only five survey respondents had submitted unsuccessful applications compared to 84 successful applicants. Quantitative comparisons of responses based on those variables are not reliable but, where possible, commentary is included on any differences in experiences between those groups emerging from the survey or interviews.

Subgroup numbers were small in some cases, meaning that considerations are sometimes based on individual experiences.

Commissioned qualitative research

The qualitative research aimed to capture a variety of experiences across clients and formal representatives. However, the findings are not representative of clients. This is because the overall sample was small, and participants were self-selecting, meaning that they actively chose to take part, as opposed to being randomly selected.

Medium- and long-term outcomes

These are outcomes that are likely impacted by other factors. The report aims to provide an insight into how policy changes have contributed to these outcomes where possible by linking up the policy commitments and short-term outcomes with the medium- and long-term ones. However, some long-term outcomes necessarily require more long-term data. An evaluation of these outcomes therefore needs to be explored in combination with the full suite of policy changes across the disability benefits.