Fair Start Scotland evaluation report 2: local area case studies - November 2019

Appendix 2 – Evaluation Methodology

This evaluation focuses on providing an insight into how delivery is playing out in practice and seeks to inform future iterations of employability services in Scotland through understanding what works.

This evaluation focuses on developing case studies in 9 localities across Scotland over the three years, one in each contract Lot area. This year one report focuses on our first three case studies -Alloa (Forth Valley), Irvine (South West) and Wick (Highlands and Islands).

This appendix breaks down the evaluation objectives and methodology.

Evaluation objectives
The key objectives of the evaluation are to:

• Understand how FSS is being implemented across the different lot areas in Scotland

• Understand the experience of FSS for lead providers, partner organisations, participants and employers

• Identify what is working well and less well in the implementation of FSS

• Identify the lessons learned and recommend changes to consider for the remainder of the FSS contract period as well as shaping what the next iteration of employment support in Scotland might look like

Evaluation methodology
We are carrying out the following tasks in each of the case study areas:

• Conducting desk-based area analysis of the socio-economic and employment trends in each of the localities to understand the local labour market context that FSS is operating in
• Analysing the management and performance data from FSS in each of the localities to understand the profile and numbers of participants and outcomes achieved in the area
• Conducting interviews with participants in each locality to understand their experience of and views on FSS
• Conducting interviews with individuals in each locality who would be eligible for FSS but have not participated in the service to understand potential barriers individuals face in participating in FSS
• Conducting interviews with employers in each locality to understand their experience of and views on FSS
• Conducting interviews with service provider staff including managers and frontline staff delivering FSS locally to understand their experience of and views on the service
• Conducting interviews with staff in partners of FSS providers in the locality to understand their experience of and views on FSS
• Conducting interviews with relevant other stakeholders in the area to understand their experience of and views on FSS.

FSS participants were contacted through a database of all service participants supplied by Scottish Government. We emailed all participants living in Alloa, Irvine and Wick who had taken part in the service for at least 6 months and issued an introductory letter to 112 participants across all three areas – 76 by email and 36 by post. We followed this up with a telephone call to arrange an interview, making a maximum of three attempts to contact each participant. We experienced various challenges in contacting participants, for example:

• Phone numbers were either incorrect or disconnected
• Participants responded to our introductory letter indicating they did not wish to take part in the evaluation
• Participants did not answer and did not respond to voicemail messages left
• Participants agreed to take part at a scheduled time but did not answer when contacted for their interview.

We recruited non-participants by contacting 20 public and third sector organisations with employability services operating in Alloa, Irvine and Wick. These included: local authority-led employability services; community hubs; housing associations; charities that provide employability support; and third sector interfaces. In addition, we liaised with the FSS providers to explore whether they were delivering other services involving people who were eligible for FSS but not taking part.

As anticipated, identifying eligible non-participants proved challenging. Half of the organisations we contacted did not respond and all but one of the responding organisations were unable to identify eligible non-participants. This was due to various reasons, for example:

• The organisation did not collect data on who had or had not taken part in FSS
• The organisation did not collect data on who was or was not eligible to take part in FSS
• The short timeframe of the research
• The contact could not identify anyone suitable to take part in the research.

In total, we conducted 30 interviews with participants and non-participants – 18 participants and 12 non-participants. This figure is broken down by participant group and type of interview below.

Table 5.1: Interviews conducted with participants and non-participants

While both of these figures are below the planned level, the lead-in time for recruitment was shorter than had originally been intended and we are confident that numbers will increase in future years, ensuring that we reach our target number by the end of the evaluation. The interviews we did conduct have, however, already begun to give us a good sense of people’s experiences. Our key findings from these are outlined in Chapter 5

How to access background or source data

The data collected for this <statistical bulletin / social research publication>:

☐ are available in more detail through Scottish Neighbourhood Statistics

☐ are available via an alternative route <specify or delete this text>

☒ may be made available on request, subject to consideration of legal and ethical factors. Please contact socialresearch@gov.scot for further information.

☐ cannot be made available by Scottish Government for further analysis as Scottish Government is not the data controller.