Accessible Vehicles and Equipment Scheme evaluation - devolved disability benefits: research

Appendix 2: Methodology

Details of the research methodology are presented in this Appendix.

Our methodology was underpinned by ‘Realist Evaluation’^[9] theory which assumes that the same intervention may not work everywhere and for everyone. Instead, it focuses on what works, for whom, under what circumstances and how. Our approach was designed to allow us to explore individuals’ experiences of the Scheme and the factors that have made the Scheme a success or have limited its value for the individuals involved.

Details about each stage of the methodology are provided below.

Phase 1: Inception and foundation

Inception meeting

The research began with an inception meeting which included the Research Advisory Group (RAG). This meeting laid the ground for an effective working relationship between the RAG and the research team, and involved discussion and agreement of our methodology. TLB produced an inception report after the meeting.

Contextual review

Our team reviewed relevant research reports and strategies to strengthen our understanding of the AVE Scheme. This included previous reports about the Scheme produced by the Scottish Government. We also met with a member of the Scottish Government team who was key in designing the Scheme, which further enhanced our understanding.

Theory of Change meeting to identify AVE Scheme outcomes

We facilitated a meeting with RAG members to identify the outcomes the AVE Scheme is expected to achieve over the short-, medium- and long-term. These aligned with Social Security Scotland principles, the AVE Scheme policy commitments and broader governmental objectives. The output of this meeting was a logic model (included in Appendix 1) for the AVE Scheme. This underpinned the research study and the broader evaluation of the Scheme. Identifying the outcomes at this stage informed our research tools and meant our fieldwork could explore how far they are being achieved.

Development of research tools

We designed the topic guides to use in interviews with providers and dealers in discussion with the Scottish Government.

TLB developed an online survey for non-members in collaboration with the Scottish Government. It was then piloted by a RAG member with lived experience of disability and was amended and finalised following insightful feedback.

Topic guides for members and non-members were also produced by TLB in discussion with the Scottish Government. These were reviewed by the TLB research team following the first five interviews, but no changes were required.

All the research tools covered the core research questions and built upon the learning generated through the contextual review and Theory of Change exercise.

Invitation text, participant information sheets (including easy-read versions for members and non-members) and privacy notices were also designed by TLB with the Scottish Government.

Phase 2: Interviews with providers

We interviewed nine providers, involving 12 individuals, to learn about their views and experiences of the AVE Scheme accreditation process. This includes providers who applied and those who considered applying but decided not to. Fieldwork took place in July and August 2023.

We analysed the data collected through these interviews and presented the emerging findings about providers’ experiences of the accreditation process to the RAG. The results are outlined in Chapter 2.

Phase 3: Interviews with dealers

The Scottish Government provided details of 15 dealers who work with Motability to deliver the AVE Scheme, including five car dealers, five powered wheelchair and scooter dealers, and five wheelchair-accessible vehicle dealers.

We interviewed 12 individuals representing 11 dealers. These include three car dealers, five powered wheelchair and scooter dealers and three wheelchair-accessible vehicle dealers. These interviews focused on the dealers’ experiences working with Motability to provide vehicles and equipment to eligible people. Fieldwork took place between September and November 2023.

Phase 4: Fieldwork with members and non-members

Online survey and interviews with non-members

Invitations were sent via Social Security Scotland to a sample of CDP and ADP clients who were in receipt of the higher or enhanced mobility component but not using the Scheme in November 2023 to complete an online survey. Another invitation to an additional sample of clients was sent in December 2023.

The survey focused on the clients’ reasons for not using the Scheme and their interest in potential future participation. The survey was mostly closed-ended with space for free text comments.

In total, 614 survey responses were received. This included one blank response which was removed from the analysis. One individual submitted two nearly identical responses. These were combined to create one composite response. Thirty-two responses were received from individuals who were included in the sample of non-members but who identified themselves as having ordered or received a vehicle or equipment through the Scheme in the period between the sample being drawn and the survey invitation being issued.

In total, therefore, 580 valid survey responses were received from non-members. The 32 individuals who had joined the Scheme were removed from the quantitative analysis of survey responses but their free text responses were included in the analysis.

Survey respondents were given the option to express interest in taking part in an interview with a member of the TLB research team. These interviews allowed the team to explore the themes raised in the survey in more depth. Nearly two-fifths (220, 38%) of the 580 respondents who submitted valid survey responses expressed willingness to take part in an interview.

TLB selected survey respondents to invite to take part in an interview based on a sampling plan. Interviews with 27 non-members were completed. Each interviewee received a £30 shopping gift card as a thank you for their help. Interviews were completed between December 2023 and February 2024.

Interviews with members

Invitations were sent via Social Security Scotland to a sample of AVE Scheme members to complete an online screening questionnaire (designed by TLB using Snap Surveys) if they would be interested in taking part in an interview about their views and experiences of the AVE Scheme. The first invitation was issued in December 2023 and another invitation was sent to an additional sample in January 2024.

Members were invited to contact TLB if they preferred to complete the screening questionnaire in an alternative format or if they had any questions about the research. One member reported they were unable to take part in an interview due to mental health issues. However, they emailed detailed responses to a list of questions sent by TLB.

TLB selected members to be invited to interview based on the sampling plan.

Interviews were completed with 25 members (including the one email response noted above). These interviewees also received a £30 shopping gift card as a thank you for their help. Interviews were completed between December 2023 and February 2024.

Appendix 3 contains a profile of the members and non-members who took part in the research.

Considerations in research design for members and non-members

Key considerations included a focus on remote fieldwork (surveys and telephone or online interviews) because of the potential health risks of in-person research for people with disabilities and health conditions, especially those who may be immunocompromised. Interviewees were given the option of taking part by telephone, online or in-person. All interviews took place by telephone or online.

Accessibility was another important consideration:

• Social Security Scotland’s invitations to take part were sent using each client’s preferred format (email or text message).
• The survey was primarily online, but options to complete the survey over the phone or on paper were also available. Four non-members requested a paper copy of the survey. TLB provided these by post along with a stamped reply envelope. Two postal responses were received.
• Similarly, information was available in alternative formats on request.

Phase 5: Analysis and report writing

The final stage of the research involved analysing all data collected. This included a thorough thematic analysis of all qualitative data using a robust coding process and quantitative analysis of the closed-ended survey questions. Frequency tables were created to analyse the responses to each question and cross-tabulations were compiled to identify any differences in responses based on respondents’ demographic characteristics including age, ethnicity, deprivation, urban-rural classification and whether they are CDP or ADP clients.

This report aims to satisfy the research questions. It considers the views and experiences of the AVE Scheme among providers, dealers, members and non-members as well as identifying aspects of the Scheme that are working well and areas for improvement and development.

TLB presented the research findings to the RAG shortly before the report was submitted.