Fair Start Scotland - evaluation report 5: participant phone survey - years 4 and 5 - November 2023
Part of a series of reports on the evaluation of Fair Start Scotland (FSS) employability service. The report presents findings from a representative phone survey with FSS participants. The report covers years 4 and 5 (April 2021 to March 2023) of FSS delivery.
Technical Appendix
Survey approach
The Fair Start Scotland (FSS) Wave 4 telephone survey was carried out in December 2022 and January 2023. This included new participants who had not previously been interviewed, as well as longitudinal participants who had previously been interviewed at Wave 3 in 2020.
Sampling
The sample was made up of two distinct groups:
1. new sample of participants who joined the FSS service in 2021-22 (between July 2021 and June 2022)
2. longitudinal sample who joined the FSS service in 2020 (between January and December 2020) and took part in the Wave 3 survey
For the new sample, IFF were provided with data consisting of all starts on the FSS service between July 2021 and June 2022, which totalled 12,194 participants. After sample records with no contact details or duplicate contact details were excluded, 3,000 records were drawn. The sample was drawn in proportion with the distribution of participants by Lot (the nine FSS contract areas across Scotland) across all starts between June 2021 and July 2022. Lots 4,7,8 and 9 slightly oversampled to ensure a minimum base for subgroup analysis of this region. After opt-outs and records matching longitudinal contact details were removed, 2,956 useable records remained, as shown in Table 1.
For the longitudinal sample, all participants from the 2020 cohort who had taken part in the Wave 3 survey were sampled for Wave 4. This totalled 663 respondents, as shown in Table 2. Once those who had declined further contact and opt-outs were removed, 632 useable records remained. Respondents from the 2018 and 2019 cohorts were not contacted due to the likelihood of gaining insufficient response rates from each cohort for robust analysis.
Fieldwork
Participants were sent an advance letter two weeks prior to the fieldwork to notify them of the research and offer them the opportunity to decline to take part. They were able to do this via email, or by calling a telephone voicemail service. Telephone fieldwork was conducted between 15th December 2022 and 27th January 2023, with a break for Christmas and New Year from 24th December 2022 to 2nd January. The survey was programmed into Unicom survey software, which interviewers used to call respondents and enter their survey responses.
The project aimed for, and achieved 1,000 completed interviews, made up of 750 from the new sample and 250 from the longitudinal sample. A full breakdown of sample outcomes are shown in Table 1 and Table 2.
The survey included fewer questions for the longitudinal sample (2020 cohort) than the 2021-22 new sample. One reason for this is that these respondents had previously completed a full survey at Wave 3, there was no need to ask duplicate questions in areas that were unlikely to have changed, such as their employment history prior to FSS or their demographic information.
The average survey length for the 2020 respondents was 11 minutes 6 seconds, and for the 2021-22 cohort respondents it was 23 minutes 18 seconds.
| Number | Proportion of starting sample (per cent) | |
|---|---|---|
| Total sample | 3,000 | 100% |
| Opted out | 30 | 1% |
| Matched contact details of Longitudinal records | 14 | 0.5% |
| Total useable sample | 2,956 | 98.5% |
| Unusable (for example, wrong number, participant moved away, participant deceased) | 249 | 8% |
| Call attempted, no final outcome | 1480 | 49% |
| Declined to participate | 305 | 10% |
| Respondent does not recall participating in FSS | 94 | 3% |
| Respondent stopped or disconnected during survey | 78 | 7% |
| Total surveys completed | 750 | 25% |
| Number | Proportion of starting sample ( per cent) | |
|---|---|---|
| Total sample (2020 cohort who completed Wave 3) | 663 | 100% |
| No permission to recontact | 29 | 4% |
| Opted out | 2 | 0.3% |
| Total useable sample | 632 | 95% |
| Unusable (for example, wrong number, participant moved away, participant deceased) | 53 | 8% |
| Call attempted, no final outcome | 258 | 41% |
| Declined to participate | 57 | 9% |
| Respondent stopped or disconnected during survey | 15 | 2% |
| Total surveys completed | 250 | 39% |
Analysis approach
The survey data was checked and processed using SPSS, verbatim responses were fully coded for analysis purposes, then combined and tabulated. The data was weighted, significance testing was undertaken, and differences between subgroups identified.
Weighting
A rim weight based on age, gender and Lot was applied to the 2021-22 cohort data to bring the oversampled Lots 4 and 8 back in line with population proportions of 2021-22 FSS starters, and to correct for any non-response bias.
A rim weight based on age, gender and Lot was also applied to the longitudinal 2020 cohort data to ensure the data was in line with the 2020 population proportions. Furthermore, a rim weight based on the 2020 cohort’s employment status at Wave 3 was applied to correct for any non-response bias.
In order to directly compare the 250 longitudinal respondents to their previous responses, a subset of the Wave 3 2020 cohort data was selected, including only these respondents who also participanted in Wave 4 survey. A rim weight based on age, gender and Lot was applied to this data to ensure proportions were in line with the population proportions of 2020 FSS starters.
Table 3 and Table 4 below show the overall sample, the number of records drawn, the number of surveys achieved, and the weighted propiortions from each Lot.
| Lot | All sample | Sample drawn | Unweighted responses | Weighted responses | ||||
|---|---|---|---|---|---|---|---|---|
| N | % | N | % | N | % | N | % | |
| Lot 1 Glasgow | 2641 | 22% | 510 | 17% | 134 | 18% | 138 | 18% |
| Lot 2 Lanarkshire | 630 | 5% | 420 | 14% | 86 | 11% | 115 | 15% |
| Lot 3 Tayside | 2237 | 18% | 240 | 8% | 57 | 8% | 63 | 8% |
| Lot 4 Forth Valley | 535 | 4% | 210 | 7% | 62 | 8% | 39 | 5% |
| Lot 5 East | 1871 | 15% | 600 | 20% | 155 | 21% | 162 | 22% |
| Lot 6 South West | 835 | 7% | 360 | 12% | 80 | 11% | 87 | 12% |
| Lot 7 North East | 1415 | 12% | 210 | 7% | 64 | 9% | 51 | 7% |
| Lot 8 Highlands and Islands | 1030 | 8% | 210 | 7% | 53 | 7% | 33 | 4% |
| Lot 9 West | 1000 | 8% | 240 | 8% | 59 | 8% | 62 | 8% |
| Total | 12,194 | 100% | 3000 | 100% | 750 | 100% | 750 | 100% |
| Lot | Sample | Unweighted responses | Weighted responses | |||
|---|---|---|---|---|---|---|
| N | % | N | % | N | % | |
| Lot 1 Glasgow | 98 | 15% | 39 | 16% | 45 | 18 % |
| Lot 2 Lanarkshire | 70 | 11% | 26 | 10% | 31 | 12 % |
| Lot 3 Tayside | 46 | 7% | 16 | 6% | 20 | 8 % |
| Lot 4 Forth Valley | 48 | 8% | 23 | 9% | 8 | 3 % |
| Lot 5 East | 142 | 22% | 55 | 22% | 66 | 26 % |
| Lot 6 South West | 81 | 13% | 30 | 12% | 38 | 15 % |
| Lot 7 North East | 51 | 8% | 24 | 10% | 15 | 6 % |
| Lot 8 Highlands and Islands | 49 | 8% | 22 | 9% | 11 | 4 % |
| Lot 9 West | 49 | 8% | 15 | 6% | 17 | 7 % |
| Total | 634 | 100% | 250 | 100% | 250 | 100 % |
Effect of weighting
The data has been weighted to ensure that it is representative of the target population. As weighting should only ever lead to minor corrections in the data the impact of weighting on significance tests should be minimal.[48] Our tables therefore apply significance testing to weighted data.
For the fresh sample of the 2021-22 cohort, the effective sample size, following weighting, was 720 (from an unweighted sample of 750).
Significance testing
As part of our analysis processes, we created data tables which tabulate the data question by question against key measures of interest such as the subgroups listed below. Within these tables we have applied significance testing to make it easier to identify which relationships are significantly different from a null hypothesis (that there is no relationship between the data observed) and that we can therefore conclude that a relationship does exist; accepting a five percent chance of being wrong (the typical level of confidence applied when interpreting statistical significance).
The statistical significance tests used within our tables are two-tailed z-tests.
The z-test is a commonly used significance test for comparing differences where the data follows a normal distribution (approximately) and is particularly suited to comparisons where there are multiple categories e.g. age, gender and ethnicity. The z-test estimates the distance in standard deviations of each data point from the mean of the data. It is typically used where the sample size is large (over thirty observations). Where the sample size is greater than thirty the distribution of the standard deviation starts to resemble a normal distribution. The larger the sample size the closer the distribution of the standard deviation to a normal distribution. As one of the aims of this survey was to examine any differences between particular subgroups, for example reported health conditions and priority families, the z-test was the most appropriate for our data.
Margins of error
For statistics reported on the total base of 750 2021-22 cohort participants, the maximum standard error (at the 95% confidence interval) is +/- 3.6%.
Sub-group comparisons
The following subgroups were analysed across the data, and are shown in Table 5 for the 2021-22 cohort and Table 6 for the 2020 cohort:
- age
- gender
- education level
- employment status (at the time of the interview)
- activity before FSS: whether seeking work or economically inactive (note this was only asked of the 2021-22 cohort)
- support stage: Whether receiving pre-employment or in-work support
- health condition or disability
- ethnicity
- priority family status
- early leaver status
- support status (whether currently receiving or not)
- parental status
- whether a re-joiner (note that this was only done for the 2021-22 cohort as the ability to re-join the service was not offered prior to this)
- Lot
| Demographics | Subgroup* | Unweighted responses | Weighted responses | ||
|---|---|---|---|---|---|
| N | % | N | % | ||
| Age | 16-24 | 122 | 16% | 120 | 16 % |
| 25-34 | 190 | 25% | 190 | 25 % | |
| 35-49 | 225 | 30% | 229 | 30 % | |
| 50-65+ | 213 | 28% | 211 | 28% | |
| Gender | Male | 471 | 63% | 467 | 62 % |
| Female | 278 | 37% | 281 | 37 % | |
| Education level | None | 109 | 17% | 109 | 15% |
| National 1-5 or equivalent | 236 | 36% | 236 | 31% | |
| Highers / Advanced Highers or equivalent | 140 | 21% | 142 | 19% | |
| Degree or above | 167 | 26% | 170 | 23% | |
| Employment status (at the time of the interview) | Working | 298 | 40% | 302 | 40% |
| Not working | 408 | 55% | 405 | 54% | |
| Other | 39 | 5% | 38 | 5% | |
| Activity before FSS (whether seeing work or economically inactive) | Actively searching for work | 484 | 65% | 484 | 65% |
| Economically inactive | 227 | 30% | 227 | 30% | |
| Other | 39 | 5% | 38 | 5% | |
| Support stage | Pre-employment | 481 | 64% | 477 | 64% |
| In work | 269 | 36% | 273 | 36% | |
| Health condition / disability | Has a health condition which limits day-to-day activities | 360 | 51% | 357 | 48% |
| Has a health condition, no limitations | 109 | 15% | 107 | 14% | |
| No health condition | 239 | 34% | 245 | 33% | |
| Ethnicity** | White | 623 | 83% | 623 | 83% |
| Summary: Ethnic Minority | 95 | 13% | 96 | 13% | |
| Priority family status | Yes | 158 | 21% | 158 | 21% |
| No/unknown | 592 | 79% | 592 | 79% | |
| Early leaver status | Yes | 146 | 19% | 147 | 20% |
| No | 604 | 81% | 603 | 80% | |
| Support status (whether currently receiving or not) | In work, receiving support | 111 | 16% | 112 | 15% |
| In work, left the service after completing pre-employment support | 117 | 16% | 120 | 16% | |
| In work, early leaver (left the service before the end of pre-employment support period) | 31 | 4% | 31 | 4% | |
| Not in work, receiving support (pre-employment support) | 129 | 18% | 127 | 17% | |
| Not in work, left the service after completing (pre-employment) support | 215 | 30% | 213 | 28% | |
| Not in work, early leaver (from pre-employment support) | 112 | 16% | 113 | 15% | |
| Parental status | Yes | 174 | 23% | 174 | 23% |
| No | 571 | 77% | 570 | 76% | |
| Whether a re-joiner | Yes | 198 | 29% | 196 | 26% |
| No | 487 | 71% | 489 | 65% | |
| Lot | Lot 1: Glasgow | 134 | 18% | 138 | 18% |
| Lot 2: Lanarkshire | 86 | 11% | 115 | 15% | |
| Lot 3: Tayside | 57 | 8% | 63 | 8% | |
| Lot 4: Forth Valley | 62 | 8% | 39 | 5% | |
| Lot 5: East | 155 | 21% | 162 | 22% | |
| Lot 6: South West | 80 | 11% | 87 | 12% | |
| Lot 7: North East | 64 | 9% | 51 | 7% | |
| Lot 8: Highlands & Islands | 53 | 7% | 33 | 4% | |
| Lot 9: West | 59 | 8% | 62 | 8% | |
*Please note that some subgroups may sum to 99% or 101% within the demographic category due to rounding of percentages **The Ethnicity category does not sum to 100% of respondents because some preferred not to answer this question
| Demographics | Subgroup* | Unweighted responses | Weighted responses | ||
|---|---|---|---|---|---|
| N | % | N | % | ||
| Age | 16-24 | 38 | 15% | 63 | 25% |
| 25-34 | 58 | 23% | 67 | 27% | |
| 35-49 | 75 | 30% | 67 | 27% | |
| 50-65+ | 79 | 31% | 53 | 21% | |
| Gender | Male | 143 | 57% | 150 | 60% |
| Female | 107 | 43% | 100 | 40% | |
| Education level | None | 28 | 11% | 25 | 10% |
| National 1-5 or equivalent | 69 | 285 | 71 | 28% | |
| Highers / Advanced Highers or equivalent | 56 | 22% | 57 | 23% | |
| Degree or above | 68 | 27% | 70 | 28% | |
| Employment status (at the time of the interview) | Working | 159 | 64% | 157 | 63% |
| Not working | 77 | 31% | 79 | 31% | |
| Other | 14 | 6% | 15 | 6% | |
| Support stage | Pre-employment | 106 | 42% | 105 | 42% |
| In work | 144 | 58% | 145 | 58% | |
| Health condition | Has a health condition which limits day-to-day activities | 109 | 44% | 113 | 45% |
| Has a health condition, no limitations | 44 | 18% | 48 | 19% | |
| No health condition | 91 | 36% | 85 | 34% | |
| Ethnicity** | White | 201 | 80% | 202 | 81% |
| Summary: Ethnic Minority | 34 | 14% | 33 | 13% | |
| Priority family status | Yes | 48 | 19% | 45 | 18% |
| No/unknown | 202 | 81% | 205 | 82% | |
| Early leaver status | Yes | 20 | 8% | 21 | 8% |
| No | 230 | 92% | 229 | 92% | |
| Support status (whether currently receiving or not) | In work, receiving support | 10 | 4% | 7 | 3% |
| In work, left the service after completing pre-employment support | 126 | 50% | 129 | 52% | |
| In work, early leaver (left the service before the end of pre-employment support period) | 7 | 3% | 9 | 4% | |
| Not in work, receiving support (pre-employment support) | 13 | 5% | 12 | 5% | |
| Not in work, left the service after completing (pre-employment) support | 76 | 30% | 76 | 30% | |
| Not in work, early leaver (from pre-employment support) | 13 | 5% | 12 | 5% | |
| Parental status | Yes | 55 | 22% | 51 | 21% |
| No | 194 | 78% | 196 | 79% | |
| Lot | Lot 1: Glasgow | 39 | 16% | 45 | 18% |
| Lot 2: Lanarkshire | 26 | 10% | 31 | 12% | |
| Lot 3: Tayside | 16 | 6% | 20 | 8% | |
| Lot 4: Forth Valley | 23 | 9% | 8 | 3% | |
| Lot 5: East | 55 | 22% | 66 | 26% | |
| Lot 6: South West | 30 | 12% | 38 | 15% | |
| Lot 7: North East | 24 | 10% | 15 | 6% | |
| Lot 8: Highlands & Islands | 22 | 9% | 11 | 4% | |
| Lot 9: West | 15 | 6% | 17 | 7% | |
*Please note that some subgroups may sum to 99% or 101% within the demographic category due to rounding of percentages **The Ethnicity category does not sum to 100% of respondents because some preferred not to answer this question
How to access background or source data
May be made available on request, subject to consideration of legal and ethical factors. Please contact <marta.krasuska@gov.scot> for further information.
Contact
There is a problem
Thanks for your feedback