Fair Start Scotland: evaluation report 4 - year 3 overview

Appendix 1: Evaluation methodology

Telephone survey

The FSS Wave 3 telephone survey was carried out between 22 March 2021 and 16 April 2021 by IFF Research Ltd. A total of 1, 027 telephone surveys were completed at Wave 3.

The sample was made up of two groups:

• New sample of 663 participants who joined the FSS service in 2020 (between January and December 2020).

• Longitudinal sample that was made up of two groups, those who joined FSS in 2018 (between April and December 2018) and took part in Wave 1 and Wave 2 surveys and those who joined FSS in 2019 (between January and December 2019) and took part in Wave 2 survey. In total, 364 FSS participants were interviewed as part of longitudinal sample.

For the new sample, IFF was provided with the data for all the FSS participants who started on FSS in 2020. From this IFF drew a sample of 2,364 FSS participants to be invited to take part in the survey in such a way that it reflected the distribution of FSS participants by Lot, with Lots 4, 7, 8 and 9 slightly oversampled to ensure a minimum base for subgroup analysis for these Lots.

FSS participants selected to take part in the telephone survey were sent an advance invitation letter two weeks prior to the fieldwork to notify them of the research and allow them an opportunity to decline the invitation.

The average survey length was 21 minutes 30 seconds for the new sample (the 2020 cohort) and 11 minutes 32 seconds for the longitudinal sample. The data was checked, tabulated and verbatim responses were fully coded for analysis purposes.

A rim weight based on age, gender and lot was applied to the 2020 cohort data to bring the oversampled Lots 4,7,8 and 9 back in line with population proportions of 2020 FSS starters, and to correct for any non-response bias. A rim weight based on age, gender, lot and employment status was also applied to the 2018 and 2019 cohort data to correct for any non-response bias.

Further methodological details can be found in the Participant Telephone Survey that has been published as a separate report.

Local Area Case Studies

The local area case study research focused on developing case studies in nine local areas across Scotland, one in each contract Lot area over the three years period and have been undertaken by Rocket Science UK Ltd and Blake Stevenson.

The key objectives of the local area case study research were:

• Understand how FSS is being implemented across the different Lot areas in Scotland
• Understand the experience of FSS for lead providers, partner organisations, participants and employers
• Identify what is working well and less well in the implementation of FSS
• Identify the lessons learnt and recommend changes to consider for the remainder of the FSS contract period as well as shaping what the next iteration of employment support in Scotland might look like

In Year 3 of the local area case study research the case studies were undertaken in: Fife, in Greenock in Inverclyde and in Motherwell in North Lanarkshire.

The following tasks were undertaken in each of the case study areas:

• Conducting desk-based area analysis of the socio-economic and employment trends in each of the case study localities to understand the local labour market context that FSS is operating in. (This analysis was streamlined in Year 3 to enable a fuller analysis of the messages across Fair Start Scotland bringing together the three years of case studies)
• Analysing the Management Information data and performance data from FSS in each of the case study localities to understand the profile and numbers of participants and outcomes achieved in the area
• Conducting interviews with FSS participants in each locality to understand their experience of and views on FSS.
• Conducting interviews with employers in to understand their experience of and views on FSS.
• Conducting interviews with service staff including managers and frontline staff delivering FSS locally to understand their experience of and views on the service
• Conducting interviews with staff in partners of FSS providers in the locality to understand their experience of and views on FSS
• Conducted interviews with relevant other stakeholders in the area to understand their experience of and views on Fair Start Scotland.

In Year 3 of this research the desk-based area analysis was streamlined to enable a fuller analysis of the messages across Fair Start Scotland bringing together the three years of case studies. In addition, the number of employer interviews in Year 3 (as well as Year 2) was limited as a result of the impact of COVID-19 and resulting difficulty to arrange interviews as employers had staff on furlough or it was difficult to get hold of very busy staff in HR roles.

Finally, the local area case study research had an additional, Scotland wide element that took place primarily in Year 3 and comprised of interviews with senior stakeholders from across the Scottish Government, Jobcentre Plus and providers, survey of the FSS providers' frontline staff (across the nine areas where case studies were undertaken) and a survey of Jobcentre Plus Work Coaches across Scotland.

Early Leavers Survey

The early leavers survey aimed to seek views of those FSS participants who left the service early (i.e. those who left the service before the end of pre-employment support period of 12-18 months and without having sustained employment for at least 3 months). The survey was undertaken by social researchers from the Scottish Government.

This was an online survey that comprised of five questions:

1. Why did you stop taking part in Fair Start Scotland? (please select all that apply)? The following answering options were available:

• 'I didn't find the service useful / relevant to my needs'
• 'I got a job'
• 'I've had a Work Capability Assessment and was put into the Support Group / found not fit for work'
• 'I got a place on a course at a college or training centre'
• 'I got a place on a university course'
• 'I got a place on another employment support service or training programme'
• 'I couldn't travel to meet advisors'
• 'I didn't feel ready to move into work'
• 'I have caring responsibilities'
• 'I was worried that work would have a negative impact on my health or disability'
• 'I didn't feel well enough to return to, or start work'
• 'Another personal reason'
• 'I thought I had to join the service, then later found out it was voluntary'
• 'I had been on a similar programme before, and didn't like it'
• 'I didn't like the activities on offer'
• 'I didn't think it would help me'
• 'I was worried it would affect my, or my family's benefits'
• 'Taking part was too stressful'
• 'The service wasn't being adapted to meet my needs'
• 'Another reason, please tell us why…

2. Which of the following options best describes what you are doing now?
The following answering options were available:

• 'I am self-employed'
• 'I am working full time (at least 35 hours a week)'
• 'I am working part time'
• 'I am in education or training (such as college or university)'
• 'I am not working, but due to start work in the next 30 days'
• 'I am not working, and not claiming a benefit'
• 'I am not working, and claiming an out of work or disability benefit'
• 'I am a full time parent or carer'
• 'I am doing something else, please tell us what...

3. Is there anything Fair Start Scotland could have done that would have stopped you leaving the service? With the answering options:

• 'Yes'
• 'No'

4. What could Fair Start Scotland have done to have stopped you leaving the service?

5. Do you have any other comments about Fair Start Scotland?

An invitation email to take part in the survey was sent out to all FSS participants in Year 1, 2 and 3 who left the service early and for whom an email address was available. In total, the invitation email was sent out to 8,347 FSS participants who left early, 451 invitation emails bounced back, 105 FSS participants opted out and 349 completed the survey. The survey was open for 3 weeks and closed on 30 May 2021. All the survey responses were anonymous.

Responses to questions 1 and 4 were analysed for the purpose of this FSS evaluation report.

• For question 1, the free-text responses were first either recoded back into the answering categories provided or new categories to reflect participant's free-text responses were created. The five new answering categories included: 'COVID-19', 'Issues with advisor/lack of contact', 'Did not leave early', 'Retired' and 'Wasn't eligible/No longer eligible'. In the next step the frequencies for each of the answering categories was calculated.
• For question 4, the thematic analysis was used to explore the key themes for the recommendations of what could have been done differently to stop FSS participants from leaving early.

Participant Interviews

The qualitative semi-structured interviews aimed to explore FSS participants views and experiences of taking part in FSS to find out what worked and what didn't work (and why) in terms of providing employability support and what might be the possible recommendations for future delivery of FSS and other employability services. The qualitative interviews were undertaken by social researchers from the Scottish Government.

The participant interviews focused on reaching five groups of individuals taking part in FSS that have been identified as having particular barriers to getting into work and/or were underrepresented in FSS:

• Lone parents
• Refugees
• Those with convictions
• Those who are limited a lot by a health condition
• Ethnic minority participants
• Those who are care experienced

The three key research questions the interviews aimed to address were:

Research Question 1:
What are participant's experiences of taking part in the FSS and of receiving support addressing particular barriers to employment they might have (in relation to belonging to one or more of the five groups summarised above)?

Research Question 2:
What are participants views / opinions on what worked well and what didn't work well (and why) in providing support including specialist support for the specific barriers through FSS to get closer to or into work (and to stay in work for those who got into job)?

Research Question 3:
What are participant's recommendations regarding provision of FSS service (and other employability services) in the future, especially in relations for providing support for specific barriers and, attracting more people from groups experiencing specific barriers , helping to prevent dropping out from FSS and get and keep more people like you in jobs.

Box 1.1 describes the interview guide that was used to conduct the interviews:

Box 1.1 Interview guide for the interviews with FSS participants

Questions About Your Experiences

I'd like to ask you about your views on what was good and not so good about FSS, and what you think could have been improved on. But to start off I'd like to ask you about your experiences of taking part in FSS.

1. How did you first hear about Fair Start Scotland?

2. At the time, why did you take part in the service?

3. Can you tell me what happened initially when you joined and what were your first impressions?

a. Prompts:

• i. What type of information did you initially receive?
• ii. Do you feel your needs were listened to?
• iii. How did you find working with your personal advisor / key worker?
• iv. experience of joining the service?

4. What kind of support were you provided through the service?

5. Can you tell me what is your current support situation in terms of involvement with the FSS?

a. Prompts:

• i. What sort of support are you currently receiving?
• ii. How long have you been taking part?
• iii. Did you drop out / stop participating in the service? If yes, was there a particular reason to it?

6. If that is okay with you, can I ask here if there are any specific barriers / difficulties (other than you've just mentioned) that you experience in relation to getting into and staying in work?

7. Have you received any kind of specialist support to address the particular barriers/difficulties you might have experienced getting into or staying in work?

a. Prompt:

• i. What kind of support was that?

Questions About Views and Recommendations

I'd like to now ask some questions about your views on what support worked well and what worked less well, and also your thoughts on how the FSS service could be improved.

8. What did you like about taking part in FSS and what worked well?

Prompts:

• i. How do you think you benefited from taking part in FSS?
• ii. Any other things you particularly liked about FSS
• iii. How about receiving support for the particular barriers you received?
  

9. How about things that have not worked well when working with FSS?

a. Prompts:

• i. Where there things that have not helped you get into job?

10. Do you have any thoughts or advice in relation to providing FSS services in the future?

Prompts:

• i. What do you think could be changed / improved to better help people like to get into jobs and to support them while in work?
• ii. What else do you think FSS could do to support people before they start looking for a job?
• iii. What do you think would encourage other people to take part in FSS?
• iv. What do you think can be done to better so that people don't drop out / stop taking part in FSS?
• v. We know that about half of people who start taking part in FSS drop out early from the service, why do you think that might be?

11. Is there anything else we haven't talked about today that you think we should know or that you would like to cover?

Potential participants were selected from the list of FSS participants who belonged to the groups of interest (see above) for whom contact details were available. Prior to contacting potential participants we checked with FSS providers to ensure that these individuals were appropriate to be contacted. The potential participants were then invited to take part in the interviews either by email or through a phone call. If a person agreed to take part, the interview was conducted by the researcher right away or a convenient data for an interview was arranged.

Interviews took place over the phone between June-August 2021. In total, 19 interviews were conducted and of these 18 were recorded (using an audio recorder placed next to the phone). For the remaining on the researcher that was conducting the interview took detailed notes during the interview.

Interviews were transcribed verbatim by an external transcription services company and the transcripts were analyses using thematic analysis to draw out key themes in relation to: barriers to employment experienced by FSS participants, experiences of pre-employment and in-work support views on strengths and weakness of the services and recommendations for improvement.

Survey of FSS providers - FSS Employment Outcomes

The aim of this research was to gain insight into the characteristics of employment outcomes for FSS participants across the first three years of FSS.

For this purpose the FSS providers were asked to provide details on each employment outcome for FSS participants (i.e. each job FSS participant started while taking part in the service) including:

• Job start date
• Job end data (if applicable)
• Reason for job end (if applicable)
• Job title
• Job number (i.e. whether it was a first, second or any subsequent job a FSS participant started after joining the service)
• Type of contract
• Contract hours
• Earnings
• Was the person self-employed
• Was this job paying a National Living Wage
• Employer's name
• Employer's sector

The dataset that contained on the employment outcomes characteristics was prepared in July 2021.

For the purpose of the third year FSS evaluation report, we focused on analysing the characteristics of the first jobs FSS participants started after joining the service, although we note that some FSS participants started more than one job while taking part in the service. It is important to note that the dataset was characterised by a high proportion of missing data which limited the scope of the analyses that were undertaken.

We analysed three characteristics of first jobs FSS participants started after joining the service:

• type of contract,
• working full-time vs. part time (based on the 'Contract hours' variable in the dataset, working full-time was defined as working 35 hours per week or more)
• earning below the National Living Wage vs. earning The National Living Wage or above (based on the 'Earnings' variable)

In total, the dataset that formed the basis for the analyses consisted on 32,506 unique entries for employment outcomes. Of these, 10,229 entries described characteristics of the first jobs FSS started after joining FSS. Where appropriate the data was recoded to allow the calculation of the values of interest.

Economic Evaluation

The independent economic evaluation was carried out by Alma Economics and it related to the first three years of the FSS delivery between April 2018 and March 20201.

The three key objectives of the economic evaluation were:

• To understand the value for money of the service by comparing costs and benefits including from the perspective of the tax payer and the society and comparing the realised costs and benefits against the business case and assessing the values for money
• To understand the value for money of the service by employing wider measures including the average cost per job outcome and comparing the FSS service with other employability programmes and the business case (e.g. by comparing he performance, cost-effectiveness, and efficiency indicators)
• To understand the wider social impact of the FSS service including through social cost-benefit analysis, assessing whether the service has contributed to inclusive growth and wellbeing ambitions and considering the distributional impacts of the service including the impacts on particular groups supported by the service

The economic evaluation used the latest Department for Work and Pensions (DWP) Social Cost Benefit Analysis (SCBA) model that has been developed to assess cost effectiveness of labour market policies taking into account wider impacts on the economy and society.

Further details on the economic evaluation methodology can be found in the section 9.1 (Economic Evaluation: Impact, Performance and Value for Money - Methodology) of this report and in the separate report on the Economic Evaluation.