Fair Start Scotland - evaluation report 5: participant phone survey - years 4 and 5 - November 2023

2. Methodology

2.1 Research aims

A telephone survey was carried out with FSS participants, in a continuation from the methodology of previous waves of research. This quantitative research aims to broadly explore the experiences and associated outcomes for Year 4 and Year 5 FSS servicce participants. This research incorporated a particular focus on previously identified groups who have significant barriers to employment and are further from the labour market. This includes participants from families at highest risk of child poverty, participants from minority ethnic background and disbaled participants.

2.2 Survey methodology

Participants were sent an advance letter two weeks prior to fieldwork to notify them of the research and offer them the opportunity to decline to take part. 3,000 participants belonging to the 2021-22 cohort (i.e. those who started between July 2021 and June 2022) were identified to be contacted. An additonal 663 participants belonging to the 2020 cohort (i.e. those who took part in Wave 3 survey) were identified for the purose of the longitudinal part of the survey. 2,956 of the 2021-22 cohort were contacted, while 632 of the 2020 cohort were contacted. Telephone fieldwork was conducted between 15th December 2022 and 27th January 2023, with a break for Christmas and New Year from 24^th December 2022 to 2^nd January. Among the 2021-22 cohort, 305 participants declined to participate (10 per cent of the starting sample). 57 of the 2020 cohort declined to participate (9 per cent of the starting sample). The average survey length for the 2021-22 cohort respondents was 23 minutes 18 seconds while for the 2020 respondents was 11 minutes 6 seconds. An ‘assisted’ interview was offered in the most common minority languages spoken in the UK (e.g. French, German, Spanish, Polish, Bengali, Gujarati, Punjabi, Urdu, Yaruba and Hindi). In an assisted interview, the interviewer administers the questionnaire in English, but provides assistance in the respondent’s first language where necessary to aid understanding.

In this report, FSS participants who took part in the surveys are referred to as the 2018 cohort (individuals first interviewed as part of Wave 1 survey), 2019 cohort (individuals first interviewed as part of Wave 2 survey), 2020 cohort (individuals first interviewed as part of Wave 3 survey) and 2021-22 cohort (individuals first interviewed as part of Wave 4 survey), according to the calendar year they joined FSS. From Wave 2 survey onwards the survey had a longitudinal element, meaning that a proportion of the earlier cohorts were recontacted in the subsequent wave(s). Figure 2 shows which respondents were surveyed over the four survey waves.

This report details findings from Wave 4 of the survey that was carried out in December 2022 and January 2023. The first seven chapters focus predominantly on participants from the 2021-22 cohort, and within each chapter comparisons are made between the current 2021-22 cohort and past cohorts (i.e. the 2018, 2019 and 2020 cohorts) where relevant. The final chapter focuses mainly on the experiences over time of the 2020 cohort who were first surveyed in May 2021 as part of Wave 3 survey and were then surveyed again in December 2022 – January 2023 as part of this current Wave 4 survey. It should be noted that the base size for the 2020 cohort (250 participants) is low, so sub-group comparisons should be treated with caution.

Differences between the different year groups, survey waves and sub-groups within them have been tested and only those that are significant are reported on. Generally, questions where the base size was lower than 30 were not used^[7]. For a full explanation of the significance testing undertaken, and a full list of sub-groups tested, please refer to the technical appendix at the end of this report.

Longitudinal sample

All new FSS participants from the Wave 3 cohort were recontacted in Wave 4 (except those who had opted out and declined further contact), with the aim of conducting 250 interviews in total within this group. These participants had joined the FSS service between January and December 2020. The final chapter examines this group in detail and compares their answers from the Wave 3 survey and the Wave 4 survey for the 250 individuals who took part in both Waves. This allows a comparison on key measures over time and examine longitudinal outcomes.

For the 250 individuals who took part in both waves, a RIM weight was applied to the data collected at Wave 3 and at Wave 4. The RIM weight was based on gender, age and geographical Lot from the full population proportions of 2020 FSS starters. The purpose of this was to bring the data back in line with the original population proportions of the 2020 FSS starters and to correct for any non-respose bias^[8]. In addition a RIM weight was also applied to Wave 4 longitudinal data based on the employment status of the 2020 cohort at Wave 3, to further account for any non-respose bias between Wave 3 and Wave 4 of the survey.

A note on differences between geographical Lots

One of the sub-groups tested to explore any statistically significant differences was geographical Lot. This was undertaken in order indicate any differences related to geography, but we do not have insight into the underlying reasons behind any differences. Differences between Lots should not be interpreted as indicative of the performance of local service providers, simply that a difference exists between the geographical areas. Any interpretation of results for geographical Lots should be undertaken with extreme caution.

A note on eligibility for participation in Fair Start Scotland during 2021-22

In response to the labour market context during 2021/22, and participant feedback collected during the Year 3 Evaluation, some changes were made to the participation and eligibility criteria for 2021-22:

• reduction of the criteria for “length of time unemployed” from 24 months to 12 months
• allowing a “right of return” to previous participants; we have termed this group “re-joiners” in this report
• continuing to allow participants to “pause” their engagement with the service

As a result, the profile of individuals surveyed as part of the 2021-22 cohort reflects these changes. For example, just over a quarter (26%) of the 2021-22 cohort were re-joiners, having received support from the service perviously. More information on re-joiners can be found in the following “Profile of Respondents” section.