No One Left Behind and the Young Person's Guarantee: implementation evaluation

2. Methodology

A mixed-methods approach was taken to the implementation evaluation of No One Left Behind and the Young Person's Guarantee, drawing on quantitative and qualitative research with local employability stakeholders, staff within employability service providers and service users. The aim was to incorporate as broad a range of views and perspectives as possible to address the evaluation questions. An overview of the data collected to inform the evaluation is provided in Figure 2.1 and further details are provided in the sections that follow.

A Research Advisory Group (RAG) was set up by the Scottish Government to oversee and guide the design and delivery of the evaluation. The RAG was made up of representatives from across Scottish Government policy and social research teams as well as local government. The RAG inputted to each stage of the evaluation including inception, scoping and design, development of research tools and analysis and reporting of interim and final findings.

Figure 2.1: Overview of data collection

Stakeholders and staff

• 20 Interviews with local employability stakeholders
• 172 Responses to an online survey of employability staff
• 6 Focus groups with employability staff

Service users

• 713 Responses to a national survey of service users
• 43 Interviews with service users
• 12 Online diaries with service users

The remainder of this chapter details each stage of data collection before discussing the approach to analysis and synthesis of the findings and methodological limitations.

2.1 Data collection within five local authority areas

Stakeholder interviews

The first stage of fieldwork involved semi-structured interviews with 20 local employability stakeholders within five local authority areas. The areas were selected to ensure a mix by geography (island, rural, medium / large urban), size and stage of implementation of No One Left Behind and the Young Person's Guarantee. The selection was informed by feedback from local and national stakeholders consulted during the scoping stage of the evaluation.^[4] It was agreed with the RAG that the five areas selected would not be identified in the reporting of the findings to protect the anonymity of those who participated and to facilitate open and honest reflections on their experiences.

The interviews were conducted in August and September 2022. They incorporated 14 individual interviews, one paired interview and one focus group involving four participants.

The RAG shared contact details for the employability leads within each of the five selected areas who were contacted by Ipsos to participate in an interview. Employability leads were asked to identify up to five additional local stakeholders involved in the design, delivery, management and / or coordination of employability services within their area. They were asked to identify a range of public, private and third sector partners, as well as local employers. Subsequent interviews were conducted with:

• Local authority management staff involved in the implementation of No One Left Behind and the Young Person's Guarantee (12)
• Third sector delivery partners (4)
• Employers using ERIs and offering work placements (4)

Interviews were conducted virtually on Microsoft Teams except for the mini group, which was conducted in person. Each interview lasted around 60 minutes. The discussion guide was designed by Ipsos, in consultation with the RAG (see Annex A). An audio recording of each discussion was transcribed for use in analysis.

Survey of staff within employability service providers

The second stage of fieldwork involved an online survey of staff within employability service providers within the five selected case study areas. The survey was developed by Ipsos, in consultation with the RAG (see Annex B). It was designed to take 10-15 minutes to complete. The survey launched on 21 September 2022 and closed on 17 October 2022.

The survey was aimed at staff who were involved in direct delivery of employability services. It was distributed by local authorities on behalf of Ipsos. The five local authorities were each provided with a unique link to send out to staff in their area as well as an accompanying information sheet, privacy notice and covering email. A total of 172 responses were received, which included frontline staff, managers, work / careers coaches, training providers and employer engagement leads (Table 2.1). It is not known how many staff work in employabiltiy service providers within each of the five case study areas. As a result, it is not possible to assess how representative the findings are of all employability staff.

It is worth noting that, whilst the survey was clearly framed to be about services delivered through No One Left Behind and the Young Person's Guarantee, we cannot know for sure if staff responded in relation to other services they were delivering.

Focus groups with staff

Employability provider staff who completed the online survey were asked if they would be willing to be recontacted to participate in additional research to inform the evaluation. All those who provided consent were contacted by Ipsos and invited to take part in an online group discussion. A total of six online focus groups were held with employability staff in November 2022, with a combined total of 16 participants. This included representatives from local authorities, public agencies and third sector organisations.

The focus groups were conducted on Microsoft Teams by the Ipsos evaluation team. The timing varied from 60 to 90 minutes depending on the number of participants. The discussion guide was designed by Ipsos, in consultation with the RAG (see Annex B). An audio recording of each discussion was transcribed for use in analysis. The discussions built on the online survey responses by covering views on progress implementing No One Left Behind and the Guarantee, the extent to which services aligned with the values and core principles of No One Left Behind, and the extent to which there has been a notable change in how services are designed and delivered locally due to No One Left Behind and the Guarantee.

2.2 Data collection at national level

National online survey

An online survey was distributed to employability service users across Scotland. The survey was developed by Ipsos, in consultation with the RAG (see Annex B). It was designed and tested to take no more than 10 minutes to complete to minimise burden and maximise responses. The survey launched on 12 October 2022 and closed on 25 November 2022.

It is worth noting that a nation-wide telephone survey of users was the original methodological approach for this research. However, it was discovered that the relevant permissions, informing service users that their contact details could be shared to an independent third-party contractor for evaluation purposes, were not referenced in some Local Authorities' Privacy Information Notices.

To compensate, a range of alternative options were explored and, in consultation with the RAG, it was agreed that the most proportionate approach to reaching the highest numbers of service users would be for the survey to be distributed via email by local authorities. Contact details for the Ipsos team were included in the covering email for the survey for anyone who was having difficulties accessing or completing this.

When compared to a telephone survey, a key limitation of the online survey approach was ceding control to local authorities to issue the survey to service users on behalf of Ipsos. To address this, Ipsos ran two information sessions for local authorities on 10th and 12th October 2022 to explain the purpose of the survey, the process for distributing this and to answer any questions they had. A recording of the session was shared with those who were unable to attend. Ipsos tracked completion rates by local authority and followed up by email and telephone with the employability leads in those areas where there were no completions to offer additional support and to encourage participation.

All 32 local authorities were provided with a unique link to the survey for distributing to their service users along with a covering email and text for issuing reminders. The survey was aimed at all people who had accessed employability services in their area since 2019. A total of 713 responses were received to the survey, with the majority of Scottish local authority areas^[5] being represented. A full breakdown of the profile of survey respondents is provided in Table 2.2. It included representation from service users who were over and under the age of 25, with limiting health conditions or disabilities, from minority ethnic backgrounds, who were care experienced, single parents and those with criminal convictions.

The survey explored service users' initial engagement with the service including how they found out about it, what they were hoping to get out of it, the types of support accessed and how easy or difficult they found it to access services. The survey also gathered feedback on levels of satisfaction with the support received and views on what could be done to further improve employment support services in their area.

Whilst the survey was clearly framed to be about services delivered through No One Left Behind and the Young Person's Guarantee, we cannot know for sure if they were responding in relation to other services.

Interviews with service users

Service users who completed the online survey and provided consent to be recontacted were invited by Ipsos to take part in a follow-up interview. A total of 43 interviews with service users from 19 local authorities^[6] were conducted by Ipsos between November and December 2022. Whilst no fixed quotas were set, the aim was to recruit participants with a range of characteristics (see Table 2.3 for a full breakdown of the profile of participants).

Interviews lasted 20-30 minutes and participants received £30 as a voucher or bank transfer as a thank you for their time. The discussion guide was designed by Ipsos, in consultation with the RAG (see Annex A). An audio recording of the discussions and researcher notes were taken for use in analysis. The discussion built on the themes covered in the online survey to explore in more detail service users' experiences of accessing the support, engagement and recruitment to the service, types of support accessed and the difference this had made / was expected to make to them.

Online diaries with service users

The final stage of fieldwork involved online diaries with 12 employability service users from ten local authorities. These were conducted using AppLife, an online app-based research tool proprietary to Ipsos. AppLife allows participants to contribute to research in a flexible way by providing text, picture, video or audio responses to questions and tasks set by the research team.

Service users who participated in an interview were asked if they would be willing to be recontacted to participate for these diaries. All those who provided consent were invited to take part, but not all decided to do so.

Over a two-week period, participants were set eight tasks. These were made available to participants on set days but could be completed at any time during the remaining fieldwork period. Each task was voluntary. An Ipsos moderator reviewed responses daily and prompted for more detail where appropriate. The schedule of tasks was designed by Ipsos, in consultation with the RAG (see Annex C). Participants received a £50 as a voucher or bank transfer as a thank you for their time.

A detailed breakdown of the profile of AppLife participants is provided in the table in Annex C. It included representation from service users who: were under the age of 25; had a limiting health condition or disability; were from an ethnic minority background; were care experienced; were a single parent; or had a conviction.

In terms of engagement, there were 118 distinct responses received from the 12 participants who took part in the online diaries, by: text (78 entries), audio (7 entries), photo (32 entries) and video (1 entry). Responses covered service users' daily experiences of employability support services and barriers or challenges they have experienced on their employability journey. The text entries were combined with the qualitative data gathered through the interviews with service users and analysed against the same framework (described in more detail in Section 2.3 below). The audio and video data was transcribed and also combined with the other qualitative data for analysis. The pictures were used as prompts for further discussion between moderators and service users.

2.3 Analysis and synthesis

The quantitative data collected through the online surveys of employability staff and service users was analysed based on results tables produced by the Ipsos data processing team. The tables were designed to include:

• Crossbreaks with key variables of interest – enabling comparison of responses by different subgroups of interest, such as type of support accessed (careers advice, volunteering, support for CV / interview skills) and demographic profile (age, gender, protected characteristics). This included significance testing of differences between subgroups on key variables such as overall satisfaction with the service. Only subgroup findings that are statistically significant are reported.
• Net responses – such as combining 'Agree' and 'Strongly Agree' or 'Satisfied' and 'Very Satisfied' to gauge overall levels of agreement / satisfaction.
• Summary tables – for example showing net agreement with a series of statements within the same table for ease of comparison (followed by individual tables with more detailed breakdowns).

Analysis of the qualitative data collected through the interviews with staff and service users and online diaries with service users was based on a systematic thematic approach. It involved the following stages:

• Initial interviewer observations – whereby researchers took notes immediately after each interview or focus group outlining their initial impressions or key points.
• Analysis meetings – during and after fieldwork to identify (emerging) themes and issues under each research question.
• Systematic summarising – of the key findings from each interview into a 'thematic framework matrix' in Excel, developed from the research questions and analysis meetings.
• Systematic analysis of the data – to identify the range of views and experiences, including similarities and differences in views and experiences between different sub-groups (such as by different equalities characteristics).

Where findings are based on qualitative data, the report avoids the use of quantifying language (including terms such as 'most' or 'a few') as far as possible, since the purpose of qualitative data is to identify the range of views and experiences on an issue, rather than to estimate prevalence.

2.4 Methodological limitations

The combination of survey findings and qualitative research allows us to comment on how widespread particular views or experiences are likely to be (based on the survey data), and to provide a more in-depth account of the potential reasons for particular views or experiences (drawing on the qualitative research). However, as with any evaluation, there are limitations to the methodology that should be kept in mind when interpreting the findings.

First, fieldwork with stakeholders and staff was conducted in five local authority areas. While these areas were chosen to ensure a mix in terms of geography, location, size and stage of implementation, it is likely that staff in other local authorities have different experiences of the implementation of No One Left Behind and the Young Person's Guarantee. Furthermore, while Ipsos provided local authorities with information on how to distribute the survey to frontline staff in their area, we cannot be sure that the survey reached all intended recipients.

Second, the original intention had been for Ipsos to conduct a telephone survey of service users for reasons already covered in Section 2.2. The shift to an online survey distributed to service users via local authorities represented a less robust approach as the evaluation team were not in control of who the survey was sent to or when reminders were issued. Whilst mitigations were put in place, including information sessions for local authorities and follow up calls and emails to those who did not response, not all local authorities took part and we cannot be sure that the survey reached all intended recipients for those who did take part. Moreover, we cannot be sure that respondents did not receive employability support that was not funded through No One Left Behind / the Young Person's Guarantee.

It should also be noted that there are relatively small numbers of service users with protected characteristics represented in the survey and so findings for these groups should be interpreted with caution.

Third, in terms of the qualitative fieldwork, participants were self-selecting which may have biased the sample towards those who have stronger views (positive or negative) on their experiences of delivering or accessing employment support services.