Disability benefits evaluation - case transfer: evaluation

Methodology

As noted in the published evaluation strategy, evaluations under this programme of work are supported by the development of a theory of change. This is represented graphically within a logic model. The logic model (see Figure 3) was developed collaboratively with members of the Research Advisory Group. It shows the trajectory from the direct impacts of the case transfer policy commitments (short-term outcomes) to the outcomes that arise as a result of these and other policies related to the delivery of disability benefits (medium-term outcomes). Finally, the long-term outcomes relate to broader impacts on views and experiences with Social Security Scotland.

Data sources

The evaluation draws on data from multiple data sources. Collectively, these sources provide a comprehensive understanding of the experiences of clients with awards transferring to either Child Disability Payment or Adult Disability Payment. There are different and overlapping fieldwork dates for these sources but taken together these experiences largely cover the period of April 2022 to July 2023.

Commissioned qualitative research

Ipsos Scotland was commissioned by the Scottish Government to conduct qualitative research exploring the experiences and outcomes of clients in relation to the case transfer and review process. The research questions focussed on clients’ overall experience of case transfers and reviews, differences in experiences across clients transferring from different benefits and how clients experience the process compared to their previous experiences with the DWP.

In addition to exploring the short-, medium- and long-term impacts of the case transfer and review experience, the commissioned research assessed whether the policy had been implemented as intended. An overview of this assessment is provided in the final chapter of this report and more detail can be found in the full commissioned report (Annex A).

The commissioned research involved both clients and staff in Social Security Scotland and was conducted over two phases. This was designed to allow sufficient time to have passed to capture the experiences of the first review post-case transfer for those with awards transferring from both PIP and DLA.

Recruitment of clients was undertaken through Social Security Scotland via an opt-in process. Clients were sent a letter, email or text message based on their contact preferences with details about the research and invited to get in touch with Ipsos Scotland if they were interested in taking part. Respondents were asked a few screening questions to maximise the diversity of the sample. Participants were offered £30 as a ‘thank you’ for their time.

A full breakdown of the demographic characteristics of those taking part is provided in Annex A. A summary of the profile of clients is provided below.

• 14 DLAC to CDP; 15 PIP to ADP and 19 DLA to ADP clients
• 11 clients at end of award/review; 18 who reported a change in circumstances; 17 managed case transfer or unsure of reason and 2 DLA clients who asked for their case to be transferred
• 36 from urban postcodes; 12 from rural postcode
• 3 from ethnic minority backgrounds
• Clients also reflected a range of different types of health conditions and disabilities

Interviews with support organisations

Scottish Government researchers conducted single and group interviews with 13 different support organisations between May and June 2023. A list of organisations that were known or expected to directly support clients through the transfer of their benefits and/or their first review under Social Security Scotland was created drawing on knowledge and research into organisations that had been involved in previous research, contributed to public consultations or been part of advisory/stakeholder groups. These organisations were contacted by email or phone and invited to take part in online interviews via Microsoft Teams.

The participants that took part reflected a mix of third sector and local authority-based organisations with experience of supporting clients from across the benefit transfer types.

Interviews lasted from 45 to 75 minutes (longer in duration for group interviews) and undertaken by two researchers, one interviewer and one note-taker. With consent, interviews were audio recorded using an external Dictaphone. Topic guides were created that explored: understanding, views and experiences of clients through the case transfer and review process; differences across clients according to type and circumstance of transfer; impact of the experience on clients; and areas that would improve clients’ experiences.

Notes were recorded as soon as possible after the interviews into an analytical grid split according to the section and question from the topic guide. The interviewer then added any of their own notes. Useful or illustrative quotes were roughly transcribed and entered into the grid; the accuracy of the quotation was then checked by returning to the original audio recording. Analysis of the data included the following steps:

• The analytical grid was coded to identify recurring areas of discussion
• Under each recurring area of discussion, the specific points made and the anonymised organisation that made these points was noted
• These areas were mapped onto key themes and cross-cutting themes
• Another researcher who was involved in the interviews then reviewed the coding structure and key themes; points of divergence were discussed and agreed

The resulting key themes were finally mapped onto the relevant short-, medium- or long-term outcomes and included within the relevant section of this report.

Client survey

This report also draws on data collected via the client survey run by Social Security Scotland. This survey is administered to all those who have received a decision or payment for a benefit delivered by Social Security Scotland.

Data included in this report comes from three waves of fieldwork that captured clients who had completed case transfer between 1^st April 2022 and 31^st March 2023. This included 2478 clients who had their award transferred to Child Disability Payment and 2826 clients who had their award transferred to Adult Disability Payment.

The client survey includes some questions which are specific to the case transfer process and it is primarily data from these questions which have been included in the report. Where possible, analysis has been conducted to explore differences in responses by demographic group, including by the benefit clients were transferring from, age, gender, ethnicity, SIMD, and type of disability. Notable differences are highlighted in the report. A full breakdown of responses is to questions used in this report is provided in the data tables in supporting documents.

Considerations and limitations

This evaluation provides rich insight into the experiences of clients who have had their awards transferred from the DWP to Social Security Scotland. A key strength comes from the integration of data from a variety of sources; individually they provide different perspectives on the outcomes of interest and collectively they allow common themes to be identified. However, there are some considerations to acknowledge.

We cannot assume that the findings represent the views and experiences of all case transfer clients. Both the commissioned qualitative research and the client survey rely on clients opting into the research. This may bias the sample towards those more willing or able to take part in research. To some extent, we have tried to mitigate this through the qualitative research with support organisations who can perhaps provide reflections on a wider range of clients’ experiences.

Particularly for client survey findings, there is a possibility that some of the clients also applied for other benefits and are basing their responses to the survey on these experiences as well as experiences of the case transfer process. However, the majority of questions used in the analysis for this report refer to questions about the case transfer process specifically.

Figure 3: Case transfer logic model

Inputs

• Scottish Government funding
• Scottish Government staff time (policy and programme teams)
• Social Security Scotland staff time
• DWP staff time
• Data sharing and other process agreements between Scottish Government and DWP
• Case transfer technical systems
• Communications with individuals – letters explaining the transfer process

Policy commitments

• Correct payment made at the correct time
• No re-applications
• No face-to-face DWP reassessments
• Complete as soon as possible while maintaining “safe and secure”
• Clear communications with individuals
• Reviews are right first time, where required
• Reviews, where required, are undertaken in line with relevant policy commitments

Short-term outcomes

• Individuals understand what is happening and when
• Individuals feel throughout that the transfer is safe and secure (i.e. the right amount is paid on time)
• Individuals know who to contact for help and support
• Individuals understand what they need to do (when appropriate) or are reassured that they don’t need to do anything
• Individuals do not feel undue stress/ anxiety about the transfer and review elements of the process
• Experience of the process is in line with dignity, fairness, and respect
• Individuals have a positive experience of the review process

Medium-term outcomes

• Individuals develop a positive relationship with Social Security Scotland
• Individuals understand what happens next
• Individuals experience reduced stress and/or anxiety about the prospect of future reviews or assessments
• Individuals feel that decision-making is fair and transparent
• Individuals feel treated with dignity, fairness and respect by Social Security Scotland

Long-term outcomes

• Individuals trust in Social Security Scotland
• There are open and effective communications between Individuals and Social Security Scotland
• Public services treat people with dignity and respect
• There is reduced stigma around illness and disability
• Individuals understand their responsibilities and feel more comfortable about reporting a change in circumstances
• The right decision is made first time by Social Security Scotland

Risk factors/external factors

• communications not clear
• errors in communications
• transfers taking longer than intended
• missed or delayed payments
• incorrect payments
• reviews not right first time