Devolved disability benefits evaluation: research

Appendix 2: Methodology

Details about each stage of the methodology are provided below.

Phase 1: Inception and foundation

Inception meeting

An inception meeting between TLB and the Scottish Government took place in October 2023. In this meeting, project managment approaches and the research methodology were agreed and discussed. TLB produced an inception report after the meeting.

Contextual review

The research team reviewed relevant research reports and strategies to strengthen our understanding of CDP, ADP and the policy changes related to applying for ADP while receiving CDP. This included information about the Scottish Government's Disability and Carer Benefits Expert Advisory Group.

Logic model workshop

In November 2023, TLB facilitated a meeting with Research Advisory Group members to identify the outcomes the CDP to ADP process is expected to achieve over the short-, medium- and long-term. These aligned with the CDP to ADP policy commitments and guided by the Theory of Change approach^[8],

The output of this meeting was a logic model (included in Appendix 1). This underpinned the research study, and identifying the outcomes at this stage informed the research tools and meant the fieldwork could explore how far they are being achieved.

Development of research tools

The team designed the research tools for the study in discussion with the Scottish Government. This included interview questions for Social Security Scotland, survey and interview questions for clients and formal representatives, and a survey for support organisations.

All the research tools covered the core research questions and built upon the learning generated through the contextual review and logic model exercise.

Invitation text, participant information sheets (including easy-read versions for clients and formal representatives) and privacy notices were also designed by TLB with the Scottish Government.

Phase 2: Fieldwork

Social Security Scotland staff dedicated to processing CDP-ADP applications

The first step of the research involved interviews with Social Security Scotland staff dedicated to processing CDP-ADP applications. This included one group interview with four staff members and one individual interview with a fifth staff member. These interviews took place in December 2023.

Support organisations

An online survey for support organisations was disseminated. The Scottish Government contacted third sector organisations that support clients and formal representatives applying for ADP while receiving CDP and requested their participation. TLB emailed the survey link to those who responded. The survey was also disseminated to Social Security Scotland's Local Delivery Teams. The survey was open from January to March 2024 and received 12 responses.

Clients and formal representatives

Invitations were sent via Social Security Scotland to a sample of CDP and ADP clients who had recently applied for ADP while receiving CDP, as well as CDP clients who had not yet applied for ADP but are eligible to do so. The first invitation was sent in mid-February 2024. Further invitations were issued to additional samples of clients in late February and mid-March 2024, and the survey closed in early April.

The survey focused on clients' and formal representatives' views and experiences of applying for ADP while receiving CDP and the reasons that influence the timing of their application. The survey was mostly closed-ended with space for free text comments.

In total, 89 survey responses were received from people who opted to take part in the survey.

Survey respondents were given the option to express interest in taking part in an interview with a member of the TLB research team. These interviews allowed the team to explore the themes raised in the survey in more depth. Over half (54%, 49) of the 89 survey respondents expressed willingness to participate in an interview.

TLB selected survey respondents to invite to take part in an interview based on a sampling plan. Interviews with 30 clients and formal representatives were completed. Each interviewee received a £30 shopping gift card as a thank you for their help. Interviews were completed between February and April 2024.

Considerations in research design for clients and formal representatives

Key considerations included a focus on remote fieldwork (surveys and telephone or online interviews) because of the potential health risks of in-person research for people with disabilities and health conditions, especially those who may be immunocompromised. Interviewees were given the option of taking part by telephone, online or in-person. All interviews took place by telephone or online.

Accessibility was another important consideration:

• Social Security Scotland's invitations to take part were sent using each client's preferred format (email or text message).
• The survey was primarily online, but options to complete the survey over the phone or on paper were also available.
• Similarly, information was available in alternative formats on request.

Phase 3: Analysis and report writing

The final stage of the research involved analysing all data collected. This included a thorough thematic analysis of all qualitative data using a robust coding process and quantitative analysis of the closed-ended survey questions. Frequency tables were created to analyse the responses to each question and cross-tabulations were compiled to identify any differences in responses based on respondents' demographic characteristics including age, ethnicity, deprivation, urban-rural classification, as well as whether they received DWP disability benefits before CDP, and whether their application was successful.

This report aims to satisfy the research questions. It considers the views and experiences of the CDP to ADP journey among clients, formal representatives, Social Security Scotland and support organisations, as well as identifying aspects of the process that are working well and areas for improvement and development.

TLB presented the research findings to the RAG shortly before the report was submitted.

Methodological limitations

The research involved a few methodological challenges.

Comparing experiences between different groups of clients and formal representatives

We are limited in the conclusions possible about differences in experiences between clients and formal representatives who had received DLAC before CDP and those who had not because, while 74 survey responses were received from clients and formal representatives whose award had transferred from DLAC, only 15 had only been in receipt of CDP. Similarly, only five survey respondents had submitted unsuccessful applications compared to 84 successful applicants. Quantitative comparisons of responses based on those variables are not reliable but, where possible, we include commentary on any differences in experiences between those groups emerging from the survey or interviews.

Subgroup numbers were small in some cases, meaning that considerations are sometimes based on individual experiences. These are indicated throughout the report as appropriate and the framework outlined in Chapter 1 is used to convey the most to least commonly identified themes to assist the reader in interpreting the findings.

Recruitment of clients and formal representatives

The Scottish Government liaised with Social Security Scotland to contact people who had applied for ADP while receiving CDP, or who were eligible to do so, on our behalf to invite them to take part in the research.

Therefore, recruitment was focused on individuals listed on an extract of the Social Security Scotland client database available to the Scottish Government.

The TLB research team thanks the Scottish Government and Social Security Scotland teams for their help in publicising the study among clients and formal representatives.

It is also important to note that the views expressed by the clients and formal representatives who took part in our research do not necessarily reflect the views of all clients and formal representatives.

Sampling clients and formal representatives

There is no aggregated demographic data related to ADP and CDP recipients, so base data was unavailable to estimate target quotas for variables such as age, ethnicity and type of disability. However, with support from the Scottish Government, we identified other data sources, such as Social Security Scotland's client survey, and calculated soft quotas based on this.

Participants were asked to provide their postcode. This information was used to determine which Scottish Index of Multiple Deprivation 2020 quintile their neighbourhood belonged to, and whether their area is categorised as urban or rural on the Scottish Government's two-fold urban-rural classification^[9].