Disability benefits evaluation: supporting information

Methodology

This section provides an overview of the evaluation approach for the supporting information aspect of the application process. Below is an illustration of the logic model that underpins the approach.

Risk factors/ external factors: Policy changes are not implemented as intended; other parts of the application and/or decision-making process negatively outcome on experience of providing supporting information; Individuals unaware of policy commitments and approaching applications/ reviews similar to the way they would with the UK system

Note: SI = Supporting information

As can be seen, the first column contains the various policy commitments for supporting information. These are also detailed above on page 9. The second, third, and fourth columns represent anticipated short, medium, and long-term outcomes that align with Social Security Scotland Principles and wider government objectives.

The model is underpinned by the following assumptions:

• if the policy commitments are implemented as intended, then their impacts will reflect the short, medium, and long-term outcomes.
• The short-term outcomes are seen as immediate policy impacts and are directly related to the policy commitments.
• Going from the short to the medium to the longer-term outcomes represents going from direct, immediate impacts to less direct, wider impacts of the policy commitments. The medium and long-term outcomes are therefore assumed to be impacted by other policy areas and factors that are outside of supporting information.

Summary of Data Sources

In accordance with the Devolved Disability Benefits Evaluation Strategy (PDF, 582.6kB), the evidence used in this report was drawn from multiple data sources including externally commissioned qualitative research and the Client Satisfaction Survey (from now on referred to as Client Survey). These data sources are explained more clearly in their corresponding subsections below.

Commissioned qualitative research

Ipsos Scotland was commissioned by the Scottish Government to conduct qualitative research with people who had applied for ADP or CDP, Social Security Scotland staff who make entitlement decisions on applications and existing awards (Case Managers), Social Security Scotland local delivery staff, and third sector organisation staff. The latter two groups of participants work directly with individuals and have experience of supporting them through the application process. This includes supporting them with the provision of supporting information for their application.

The aim of the research was to address the following questions:

1. Do applicants understand what supporting information they should or could be providing to support the decision-making process as much as possible?

2. Do applicants feel that their supporting information has been considered fairly?

3. What impact does supporting information have on Case Managers' decision-making process?

4. Is the process of requesting and obtaining supporting information done in line with policy commitments?

These questions help to increase understanding of how the supporting information policy commitments are being implemented in practice, whether they are being implemented as intended, and what the consequent impacts are of these commitments.

A targeted approach was taken to the recruitment of people who had previously applied for ADP and CDP. This was to ensure a variety of voices were heard and therefore that a variety of experiences could inform the findings. It should be noted that the aim of sampling in qualitative research is not to obtain a representative sample of the population of interest, but to obtain a mix of people with a range of different characteristics. This maximises the chance of identifying different issues. The two steps that were taken were:

• Targeting on the sample to approach for recruitment. Characteristics that were targeted on prior to approaching potential participants were age (for ADP applicants only), Scottish Index of Multiple Deprivation (SIMD), rurality, ethnicity (by whether English was first language), application outcome (successful vs. unsuccessful).
• Screening interested participants. Screener questions included: confirmation that the applicant had heard the outcome of their application, award determination (positive vs. negative), SIMD, rurality, age (for ADP applicants only), ethnicity, and information about an individual's condition or disability (i.e., physical vs. mental health condition).

Semi-structured interviews lasting around 45 minutes were conducted with 22 CDP applicants (i.e., parents/carers/guardians applying on the child's behalf) and 20 ADP applicants. These interviews were held over the phone, by video call or face-to-face. Fieldwork was conducted between 6 September and 17 November 2022. Their applications for disability benefits were submitted between 2 January and 1 March 2022 (for CDP) and between 21 March and 15 April 2022 (for ADP).

A mixture of interviews and focus groups lasting around an hour to an hour and a half were conducted with 31 members of staff, with a balanced mixture of those who work on ADP and those who work on CDP. All fieldwork was conducted face-to-face and participants were made up of 18 Case Managers, 8 Decision Team Managers, and 5 Operations Managers (see Annex A - Glossary of staff roles). Fieldwork was conducted in Social Security Scotland offices in Dundee and Glasgow in early November 2022.

Four video call interviews were also conducted with local delivery staff who are employed by Social Security Scotland. They work within local delivery teams and provide face-to-face, video and phone support to applicants. Furthermore, two video interviews were conducted with staff from advice and advocacy charities who provide support to applicants in relation to the overall application process, including supporting information. Fieldwork with these latter two participants groups took place in December 2022 and January 2023. The full report from Ipsos Scotland is available at Annex B – Qualitative report.

Client survey

This report draws on data conducted by the client survey team in Social Security Scotland. The survey is administered to all those who have applied for benefits under Social Security Scotland or who have had their benefits transferred from the DWP. It is therefore an ongoing piece of research led by social research colleagues in Social Security Scotland.

The data that is being used for this report reflects one round of the survey. Specifically, all applicants who had received a decision on their CDP or ADP application between 1 September 2022 and 31 December 2022 were invited to take part. The survey ran from 13 February 2023 to 12 March 2023.

The survey collects equalities and socio-economic information from respondents. It also asks about their experience of Social Security Scotland and receiving benefits, including, for this specific evaluation, their experience of the supporting information aspect of the application process.

In total, there were 2977 applicants to either ADP or CDP. Of which,

• 2314 had applied for ADP, and
• 663 had applied for CDP.

Annex C – Tables from Client Survey Analyses contains the output from the analyses conducted on the Client Survey data that were used to feed into this report.

Please also note the following technical points about how Client Survey findings are presented throughout this report:

• The number of respondents providing a valid answer to each individual question/statement varied slightly, depending on who was exposed to the question and whether respondents "skipped" the question by choosing not to answer.
• Most results to the closed questions are rounded to whole numbers. As such, results (e.g., those presented in tables) may not sum to 100% due to rounding.
• Some percentages quoted in the report relate to questions that allowed respondents to choose more than one response. These percentages will not sum to 100%.
• Most results are presented to zero decimal places. '0%' should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by '-'.
• Unless otherwise presented, proportions and numbers have excluded those who reported "can't remember/don't know" or who have skipped the question through their own choice or through a routing condition placed on that question.
• Numbers and proportions are based on those who reported either applying to CDP only or applying to ADP only. Those who applied for other benefits or for both ADP and CDP were excluded from the sample.
• Unless otherwise stated, statistics for those who applied to ADP and CDP were combined throughout the report.
• Some questions asked about individuals' experience since they received a decision on their application. This does not necessarily include going through re-determinations or appeals.
• The vast majority (89%) of people who took part in the survey were successful in obtaining an award for CDP or ADP. The results are therefore under-representative of those who were unsuccessful.
• Only significant differences were included in this report (however, see Annex C – Tables from Client Survey Analyses for a full breakdown of the output including both significant and non-significant findings).
• Significant differences between groups were tested using the online GIGA calculator. A difference is reported as significant when the p value < .05, in line with best practice for social research. However, if the p value was between .02 and .05, the difference was referred to as "small" or "marginal".

Equalities breakdowns for Client Survey

The following table illustrates the proportions of those taking part who represented different equality groups.

* All respondents who indicated they have a physical or mental health condition or illness lasting or expected to last 12 months or more were asked to indicate whether it affects them in these areas (e.g. Vision, Hearing, Mobility etc).

**This factor was created as a new variable based on the "type of disability" data where, for example, if an individual thought their condition or illness impacted on their vision only, then this would be computed as them having one area affected under this new variable.

Source: Client Survey Data

Considerations and limitations

This section discusses what can and cannot be determined from the data sources that are available for this evaluation. It will also highlight how this influences the extent to which robust conclusions can be drawn regarding the impacts that the supporting information policy commitments have.

Commissioned qualitative research: This evaluation is largely dependent on findings from externally commissioned work. The work aimed to capture a variety of experiences based on rurality and socioeconomic status in order to provide context and narrative to the quantitative findings from the other data sources on supporting information. However, the work did have the following limitations:

• Though not the goal, the findings are not representative of all applicants to CDP and ADP because the overall sample was small, and participants were self-selecting, meaning that they actively chose to take part, as opposed to being randomly selected.
• Due to the sample being self-selecting, there is also the likelihood that these participants are more likely to be able to deal with administrative tasks (including people with higher levels of literacy; those less severely affected by their own or their child's condition, whether physical or mental; and those with less chaotic lives and with more resource or means to take part). Consequently, this also means the sample is likely to be biased towards those more able to understand supporting information requirements and to supply supporting information.
• Only people who had completed an application were included in the research. The findings therefore do not provide insight into people who might be eligible but did not apply or did not complete their application, and whether their reasons for not doing so relate to supporting information.
• The sample contained only five applicants who were unsuccessful. While this is not hugely out of line with the proportion of all applicants who are unsuccessful (based on Social Security Scotland ADP High Level Statistics and CDP High Level Statistics), it is a small number from which to draw conclusions.

Social Security Scotland research: While the commissioned research was explicitly conducted for the current evaluation, Social Security Scotland research (the Client Survey) was conducted to provide information relevant to operations and policy and does not therefore serve as a direct measurement of the policy goals concerning this evaluation.

On a similar note, the Client Survey is currently targeted towards those who applied to disability benefits as well as those who applied to a range of low-income benefits, there is a possibility that some of the applicants to CDP and ADP had also applied for other benefits. This might therefore mean that responses to the survey are referring to other benefit experiences as well as experiences of applying to disability benefits. Nonetheless, Social Security Scotland staff have tried to minimise this risk by framing questions in relation to the benefit i.e., ADP or CDP.

Other limitations of the Client Survey apply that are similar to those described in relation to the commissioned qualitative data: (i) Again, survey recipients are self-selecting. They are therefore more likely to be those who are more willing to take part in research and those who are more likely to be able to deal with administrative tasks. The results are also therefore not representative; (ii) Only people who had completed an application were included in the research.

Supporting information for undergoing a review: The sample from which all the data was drawn, including interview data and survey data, for this evaluation consists of applicants only. It should be noted that individuals might also be asked to provide supporting information when they are undergoing a review or have requested a re-determination on a recent entitlement decision. This could include individuals who have had their existing DWP benefits transferred to Social Security Scotland and individuals who applied directly to Social Security Scotland. However, because the timing of this evaluation was just after the launch of ADP (29 August 2022), there would have been no opportunity to capture those who had undergone a review for ADP. It was therefore decided to only focus on the application process specifically with regard to the supporting information experience. Nevertheless, the policy is the same for supporting information across different situations, so learning from this report can also be applied to overall learning on supporting information policy.

Medium- and long-term outcomes: These are outcomes that are likely impacted by other factors as well as supporting information. Nonetheless the report aims to provide an insight into how supporting information has contributed to these outcomes by linking up the policy commitments and short-term outcomes with the medium- and longer-term ones.