Fair Start Scotland - evaluation report 3: year two - overview
Third report in a series of evaluation reports of Fair Start Scotland employability services which covers the second full year of service delivery (April 2019 - March 2020) and summarises findings from a participant phone survey, local area case studies and analysis of management information.
Appendix 1: Evaluation Methodology
Telephone survey
The Fair Start Scotland (FSS) Wave 2 telephone survey was carried out in May 2020 by IFF Research Ltd.
The sample was made up of two distinct groups:
- New sample of participants who joined the FSS service in 2019 (between January 2019 and December 2019). IFF were provided with a sample of 11,828 participants, from which 1,782 were drawn for the survey.
- Longitudinal sample who joined the FSS service in 2018 (between April and December 2018) and took part in the Wave 1 survey. All participants who took part in the survey and agreed to be contacted again for further research were included in the sample for Wave 2.
A total of 1,007 surveys were completed at Wave 2, made up of 607 from the new sample and 400 from the longitudinal sample, as shown in Table 1.1. An additional 10 respondents from the new sample took part in the survey but said that they had never received a service from FSS and were removed from the final data.
| Sample group | Starting sample available | Sample drawn | Responses achieved |
|---|---|---|---|
| 2019 cohort (new sample) | 11,828 | 1,797 | 607 |
| 2018 cohort (longitudinal sample) | 940 | 940 | 400 |
For the new sample, IFF were provided with data consisting of all starts on the FSS service during this period. From this, 1,782 pieces of sample were drawn. Sample was drawn in proportion with the distribution of participants by lot, with Lots 4,7,8 and 9 slightly oversampled to ensure a minimum base for subgroup analysis of this region. Table 1.2 below shows the number of records drawn and the number of surveys achieved from each lot.
| Lot | All sample | Sample drawn | Unweighted responses | Weighted responses | ||||
|---|---|---|---|---|---|---|---|---|
| N | % | N | % | N | % | N | % | |
| Lot 1 Glasgow | 2398 | 20.3% | 300 | 16.7% | 101 | 16.6% | 123 | 20.2% |
| Lot 2 Lanarkshire | 1804 | 15.3% | 213 | 11.9% | 71 | 11.7% | 93 | 15.3% |
| Lot 3 Tayside | 1161 | 9.8% | 150 | 8.3% | 48 | 7.9% | 60 | 9.9% |
| Lot 4 Forth Valley | 453 | 3.8% | 150 | 8.3% | 50 | 8.2% | 23 | 3.8% |
| Lot 5 East | 2568 | 21.7% | 327 | 18.2% | 112 | 18.5% | 132 | 21.7% |
| Lot 6 South West | 1648 | 13.9% | 207 | 11.5% | 71 | 11.7% | 85 | 14.0% |
| Lot 7 North East | 461 | 3.9% | 150 | 8.3% | 50 | 8.2% | 24 | 3.9% |
| Lot 8 Highlands and Islands | 527 | 4.5% | 150 | 8.3% | 50 | 8.2% | 27 | 4.4% |
| Lot 9 West | 808 | 6.8% | 150 | 8.3% | 54 | 8.9% | 41 | 6.7% |
| Total | 11,828 | 100% | 1,797 | 100% | 607 | 100% | 608 | 100% |
Participants were sent an advance letter two weeks prior to fieldwork to notify them of the research and offer them the opportunity to decline to take part. Telephone fieldwork was conducted between 27th April and 29th May, and 1,007 completed interviews were achieved. A full breakdown of sample outcomes is shown in Tables 1.3 and 1.4.
| Number | Proportion of starting sample (%) | |
|---|---|---|
| Total sample | 1,797 | 100% |
| Opt outs | 18 | 1% |
| Unusable (for example, wrong number) | 124 | 7% |
| Contact attempted, no final outcome after minimum number of attempts | 935 | 52% |
| Refusal | 99 | 6% |
| Respondent stopped or screened out during survey | 14 | 1% |
| Total surveys completed | 607 | 34% |
Table 1.4: 2018 cohort (longitudinal) sample outcomes
| Number | Proportion of starting sample (%) | |
|---|---|---|
| Total sample | 940 | 100% |
| Opt outs | 11 | 1% |
| Unusable (for example, number not in use) | 21 | 2% |
| Contact attempted, no final outcome after minimum number of attempts | 475 | 51% |
| Refusal | 24 | 3% |
| Respondent stopped or screened out during survey | 9 | 1% |
| Total surveys completed | 400 | 43% |
The survey included fewer questions for the 2018 longitudinal sample than the 2019 new sample. The average survey length for the 2018 respondents was 13 minutes 36 seconds, and for the 2019 respondents it was 21 minutes 21 seconds.
The data was checked, tabulated and verbatim responses were fully coded for analysis purposes. A rim weight based on age, gender and lot was applied to the 2019 data to bring the oversampled Lots 4,7,8 and 9 back in line with population proportions of 2019 FSS starters, and to correct for any non-response bias. A rim weight based on age, gender, lot and employment status was also applied to the 2018 data to correct for any non-response bias and bring the proportions in line with the Wave 1 weighted data, which reflected population proportions of 2018 FSS starters.
More detail on the breakdown of respondent profile by Lot, after weighting, is set out in the original report. See: Fair Start Scotland Evaluation Report 3: Participant phone survey (year 2).
Local Area Case Studies
The local area case study research focuses on developing case studies in 9 localities across Scotland over the three years, one in each contract Lot area, undertaken by Rocket Science UK Ltd and Blake Stevenson. The research involves carrying out the following tasks in each of the case study areas:
- Conducting desk-based area analysis of the socio-economic and employment trends in each of the localities to understand the local labour market context that FSS is operating in
- Analysing the management and performance data from FSS in each of the localities to understand the profile and numbers of participants and outcomes achieved in the area
- Conducting interviews with participants in each locality to understand their experience of and views on FSS
- This year this was complemented by with follow up interviews with 9 of those interviewed in the locality case studies last year – this included 3 from each of Alloa, Irvine and Wick. The purpose of these interviews was to explore the longer-term impact of the experience of FSS on participants.
- Conducting interviews with employers in one locality to understand their experience of and views on FSS. Because of the impact of COVID-19 arranging employer interviews this year proved difficult as employers had appropriate staff on furlough or it was difficult to get hold of very busy staff on in HR roles, so the number of employer interviews was limited.
- Conducting interviews with service staff including managers and frontline staff delivering FSS locally to understand their experience of and views on the service
- Conducting interviews with staff in partners of FSS providers in the locality to understand their experience of and views on FSS
- Conducting interviews with relevant other stakeholders in the area to understand their experience of and views on FSS.
FSS participants were contacted through a database of all service participants supplied by Scottish Government. All participants living in Drumchapel, Dundee, and Peterhead and Fraserburgh who had taken part in the service for at least 6 months were emailed, and an introductory letter to participants across all three areas was issued. This was followed up with a telephone call to arrange an interview, making a maximum of three attempts to contact each participant.
In total, 30 interviews with participants were conducted, along with 18 key workers who – between them – worked with 28 of these participants. In addition, 9 of those participants who were first interviewed last year – 3 from each of the three areas (Alloa, Irvine and Wick) – were re-interviewed.
Contact
Email: socialresearch@gov.scot
There is a problem
Thanks for your feedback