Accessible Vehicles Scheme: disability benefits evaluation

Methodology

This section provides an overview of the evaluation approach for the AVE Scheme.

Logic Model

As outlined in the published evaluation strategy, evaluations under this programme of work are supported by the development of a theory of change. This is represented graphically within a logic model. The logic model for the AVE Scheme was developed collaboratively with members of the Research Advisory Group.^[1] It shows the trajectory from the direct impacts of the policy commitments (short-term outcomes) to the outcomes that arise as a result of these, and other policies related to the delivery of disability benefits (medium-term outcomes). Finally, the long-term outcomes relate to broader impacts on views and experiences with Social Security Scotland.

Aim

The AVE Scheme aims to support disabled people to become more independent and improve their quality of life. The Scheme will achieve this by providing disabled people with the choice of a range of affordable accessible vehicles and equipment.

Inputs

• Scottish Government funding and facilitation
• Scottish Government accreditation Scheme
• Providers' finances (e.g. reimbursing SSS admin costs)
• Systems for approval at dealerships
• Scottish Government and Social Security Scotland staff time
• Communications and accreditations with providers
• Communications with individuals and stakeholders
• Providers' and dealers' staff time
• Providers' and dealers' IT systems to interface with SG
• Accessible vehicles and equipment.

Policy commitments

• Individuals have a choice about what provider they choose to go with under the AVE Scheme.
• Individuals can choose an accredited provider who meets their mobility needs.
• Additional providers will have the opportunity to join the Scheme at regular intervals.
• Individuals who currently have a lease with Motability will not lose their vehicle and will continue to enjoy the same standards of service.
• No individual will be subject to a credit check to access a vehicle or equipment.
• Individuals will pay no more than their weekly mobility component to lease a vehicle.
• Individuals can choose to pay an advance payment to lease specific vehicles or equipment that meet their needs or choose from a range of vehicles or equipment with no advance payment.
• All providers will be required to offer the same high standards of service.
• We will make communications about the AVE Scheme as simple and as clear as possible and tell clients about how to access the AVE Scheme.

Short-term Outcomes (direct impacts of the Scheme)

• Individuals are aware of the AVE Scheme.
• Individuals understand how to access the AVE Scheme.
• Individuals feel they have adequate choice through the AVE Scheme.
• Barriers that prevent individuals taking up the AVE Scheme are addressed and reduced.
• Providers are able to access an efficient application process at regular intervals.
• All providers offer the same high level of service.
• Experience of the AVE Scheme is in line with values of dignity, fairness, and respect.
• Individuals' mobility needs are met through the AVE Scheme
• Individuals feel the AVE Scheme offers value for money.
• Individuals have a positive view of the AVE Scheme.
• Individuals experience a smooth transition from the previous pre-AVE Motability Scheme to the AVE Scheme.

Medium-term Outcomes (Changes that occur partly because of the Scheme, but there are also some other contributory and causal factors)

• Disabled people who access the AVE Scheme have improved independence and quality of life.
• Individuals trust in the AVE Scheme.
• Values of dignity, fairness and respect are embedded.
• Anyone involved in the delivery of the Scheme offers individuals a positive experience (including dealers, providers, and others in the supply chain, e.g. breakdown cover providers, mechanics, tyre/windscreen replacement technicians).
• Providers deliver effective support in the case of accidents or breakdowns.
• Providers respond to members' changing needs/circumstances over time, including any deteriorations or improvements in health or disabilities, and any changes in eligibility.
• The AVE Scheme supports people to access and sustain employment, education, and training opportunities.
• The AVE Scheme supports members' wellbeing through improved or sustained access to health and other public services, and to social, cultural and leisure activities.

Long-term Outcomes (Wider outcomes that AVE contributes to but with many different contributory and causal factors)

• Reduced poverty in Scotland
• Reduced inequalities and discrimination
• Individuals trust in Social Security Scotland
• The Scottish social security system is efficient and delivers value for money.
• Scottish Government de-carbonisation objectives are met through provision of accessible, environmentally friendly, and sustainable modes of transport for disabled people.
• Bridging the mobility gap
• Public services treat people with dignity and respect.

Risk factors/external factors

Policy changes are not implemented as intended; Other parts of the Social Security Scotland application and/or decision-making process negatively impact on experience of AVE; Unclear communications, errors in communications; Errors or poor service standards among providers; Providers fail to meet their contractual responsibilities; Vehicle/equipment availability/rising cost; AVE intended outcomes and aspirations are not achievable due to factors beyond SG/SSS control; Providers' business models; SG has limited control over individuals' experience of discrimination; SG has limited control over the cost of living.

Data Sources

The evaluation draws on data from multiple data sources to evidence the outcomes outlined in the logic model. Collectively, these sources provide a comprehensive understanding of the experiences of clients and providers within the AVE Scheme, and those clients who are eligible but have not taken up the Scheme.

Commissioned research

The Lines Between (TLB), a social research agency, was commissioned by the Scottish Government to conduct research with members and non-members of the AVE Scheme, as well as with providers of vehicles and equipment, and dealers who work with Motability Operations to deliver the AVE Scheme. The full report of the commissioned work is published in Annex A.

The commissioned research combined both a quantitative survey and interviews with the four groups. The groups and recruitment approach are outlined below.

Survey of eligible non-members

Invitations were sent via Social Security Scotland to a sample of CDP and ADP clients who were in receipt of the higher or enhanced mobility component but not using the Scheme in November 2023 to complete an online survey. Another invitation to an additional sample of clients was sent in December 2023.

The survey focused on the clients' reasons for not using the Scheme and their interest in potential future participation. The survey was mostly closed-ended with space for free text comments.

In total, 614 survey responses were received. This included one blank response which was removed from the analysis. One individual submitted two nearly identical responses. These were combined to create one composite response. Thirty-two responses were received from individuals who were included in the sample of non-members but who identified themselves as having ordered or received a vehicle or equipment through the Scheme in the period between the sample being drawn and the survey invitation being issued.

In total, therefore, 580 valid survey responses were received from non-members. The 32 individuals who had joined the Scheme were removed from the quantitative analysis of survey responses, but their free text responses were included in the analysis.

Interviews with eligible non-members who responded to the survey

TLB selected survey respondents to invite to take part in an interview based on a sampling plan, to ensure a diverse range of interviewees (more information on the demographic profile of respondents can be found in the Annex). Interviews with 27 non-members were completed. Each interviewee received a £30 shopping gift card as a thank you for their help. Interviews were completed between December 2023 and February 2024.

Interviews with Scheme members

Invitations were sent via Social Security Scotland to a sample of AVE Scheme members to complete an online screening questionnaire (designed by TLB using Snap Surveys) to ascertain if they would be interested in taking part in an interview about their views and experiences of the AVE Scheme. The first invitation was issued in December 2023 and another invitation was sent to an additional sample in January 2024.

Members were invited to contact TLB if they preferred to complete the screening questionnaire in an alternative format or if they had any questions about the research. One member reported they were unable to take part in an interview due to mental health issues. However, they emailed detailed responses to a list of questions sent by TLB.

TLB selected members to be invited to interview based on the sampling plan.

Interviews were completed with 25 members (including the one email response noted above). These interviewees also received a £30 shopping gift card as a thank you for their help. Interviews were completed between December 2023 and February 2024.

Interviews with representatives of providers of accessible vehicles and/or equipment

TLB interviewed nine providers of accessible vehicles and equipment involving 12 individuals, to learn about their views and experiences of the AVE Scheme accreditation process. The providers interviewed fall into one of three categories: (1) an accredited AVE provider, (2) organisations that have applied for AVE accreditation, and (3) organisations that decided not to apply for accreditation. Fieldwork took place in July and August 2023.

Interviews with individuals representing dealers who work with Motability Operations to deliver the AVE Scheme

The Scottish Government provided details of 15 dealers who work with Motability to deliver the AVE Scheme to TLB, including five car dealers, five powered wheelchair and scooter dealers, and five wheelchair-accessible vehicle dealers.

TLB interviewed 12 individuals representing 11 dealers. These include three car dealers, five powered wheelchair and scooter dealers and three wheelchair-accessible vehicle dealers. These interviews focused on the dealers' experiences working with Motability to provide vehicles and equipment to eligible people. Fieldwork took place between September and November 2023.

Client Survey

The report draws on quantitative and qualitative data from the Social Security Scotland Disability Payments Client Survey. The Client Survey is administered to all those who have applied for benefits under Social Security Scotland or who have had their benefits transferred from the DWP. It collects equalities and socio-economic information from respondents. It also asks about their experience of Social Security Scotland and (if applicable) the AVE Scheme.

The Client Survey data presented in this report is based on responses from clients who received an application decision between April 2023 and August 2023, or received a first payment following a Case Transfer during the same period. In total, 481 respondents indicated that they used their ADP or CDP Mobility component to pay for a vehicle through the AVE Scheme. The findings here are based on data provided by these respondents.

Please also note the following technical points about how the quantitative Client Survey findings are presented throughout this report:

• The number of respondents providing a valid answer to each individual question/statement varied slightly. The results do not include respondents who skipped the question/statement.
• Most results to the closed questions are rounded to whole numbers. As such, results may not sum to 100% due to rounding.

Please also note that the qualitative Client Survey findings are based on data from a single open-text question. The question was answered by respondents who indicated that they used their ADP or CDP Mobility component to pay for a vehicle through the AVE Scheme. The question was as follows: 'Is there anything else you would like to tell us about your experience with the Accessible Vehicles and Equipment (AVE) Scheme?' A total of 130 respondents answered the question, and the findings are based on the data provided by these respondents.

Official statistics

Social Security Scotland collects information on applications, case transfers, payments, and clients in the process of delivering the benefits. Some of this information is published online as Official Statistics in development. Official Statistics in development are new or existing statistics which will be tested with users in line with the standards of trustworthiness, quality and value in the Code of Practice for Statistics. The following Official Statistics publications are used as sources of evidence in this report:

• Adult Disability Payment – high level statistics to 31 October 2023
• Child Disability Payment - high level statistics to 30 September 2023

These publications are hereafter referred to as "Official Statistics". Please note the following technical points about how Official Statistics are presented throughout this report:

• Figures are rounded for disclosure control and may not sum due to rounding.
• Where stated, secondary analysis has been conducted on rounded figures from published Official Statistics.
• Most results are presented to zero decimal places. '0%' should therefore be interpreted to mean less than 0.5%.
• A [c] indicates that figures have been suppressed for disclosure control.

Considerations and Limitations

This section discusses what can and cannot be determined from the data sources that were available for this evaluation. It also highlights how this influences the extent to which robust conclusions can be drawn regarding the impacts of the policy commitments.

Commissioned survey

Participants in the survey were self-selecting, meaning that they actively chose to take part, as opposed to being randomly selected. This may bias the sample towards those more willing or able to take part in research.

Commissioned qualitative research

The qualitative research aimed to capture a variety of experiences across members, non-members, dealers, and providers. However, the findings are not representative of all members, non-members, providers, or dealers. This is because the overall sample was small, and participants were self-selecting, meaning that they actively chose to take part, as opposed to being randomly selected. Most research participants were ADP clients and there were more clients who had transferred from DWP than were new applicants to Social Security Scotland.

Social Security Scotland research

While the commissioned research was explicitly conducted for the current evaluation, Social Security Scotland research (the Client Survey) was conducted to provide information relevant to operations and policy and does not therefore serve as a direct measurement of the policy commitments concerning this evaluation.

Other limitations of the Client Survey apply that are similar to those described in relation to the commissioned qualitative data: again, survey recipients are self-selecting. They are therefore more likely to be those who are more willing to take part in research and those who are more likely to be able to deal with administrative tasks. The results are also therefore not representative as only people who had completed an application were included in the research.

Medium- and long-term outcomes

These are outcomes that are likely impacted by other factors as well as the AVE Scheme. Nonetheless the report aims to provide an insight into how the Scheme has contributed to these outcomes where possible by linking up the policy commitments and short-term outcomes with the medium- and longer-term ones. However, some of the long-term outcomes necessarily require more long-term data. An evaluation of these outcomes therefore needs to be explored in combination with the full suite of policy changes across the Disability benefits.

Interpretation and terminology

This report synthesises different sources of information, some qualitative and some quantitative.

Where quantitative data is discussed, percentages and base numbers (n) will be provided. Qualitative analysis does not permit the quantification of results. However, a framework is used to convey the most to least commonly identified themes to assist the reader in interpreting the findings:

• The most common/second most common theme: the most frequently identified.
• Many respondents: around two thirds
• Several respondents: around half
• Some respondents: under half
• A few / a small number of respondents: around a quarter

Due to small numbers and to protect anonymity, when 'providers' are referred to this includes providers of accessible vehicles and equipment who fall into three categories: (1) an accredited AVE provider, (2) organisations that have applied for AVE accreditation, and (3) organisations that decided not to apply for accreditation.