Fair Start Scotland: evaluation report 4 - year 3 overview

10. Reflection on 3 years of Evaluation

The following chapter offers an appraisal of the performance of FSS against its stated aims^[26] which are reiterated below:

• A high quality service that maximises delivery of real and sustained job outcomes for individuals
• A service which treats participants with fairness, dignity and respect and where participation is voluntary
• A programme of service integration and alignment that will seek to join up public employability services
• Providing support for those further removed from the labour market
• Providing person-centred support

Evidence for this chapter is drawn from across this year's evaluation, as well as previous evaluation reports and is assessed against the original business case for the service where relevant (see the standalone economic evaluation for a more detailed assessment of performance against the business case).

Maximising Delivery of Real & Sustained Job Outcomes

In the first three years of delivery, FSS had the stated ambition of supporting 38,000 participants as outlined in the business case with the expectation that:

• 36% of individuals would reach the 13 week job outcome
• 30% of individuals would reach the 26 week job outcome
• 25% of individuals would reach the 52 week job outcome

In comparison to this the service has had 32,664 successfully individuals start on the service by March 2021, which represents a shortfall of 5,336 participants from the original target enrolment figure. As per findings from the Year 2 evaluation report, it is thought that the onset of the COVID-19 pandemic at the end of Year 2 and throughout Year 3 is likely to explain at least part of this shortfall.

With regards to achieved job outcomes, data is not yet available for the entire cohort across the three years of delivery as sufficient time has yet to elapse to allow for recording of longer term outcomes. For those for whom such data is available, we can see in the table below that with regards to actual performance that 23%, 18% and 15% achieved the 13 week, 26 week and 52 week outcomes respectively. In practice therefore actual job outcome rates are lower across all three time periods than those which were originally anticipated. It should however be noted that the unprecedented impact of the COVID-19 pandemic and it's consequent impacts on the labour market, including closure of a number of businesses during lockdown is likely to have had an impact on performance with regards to achieved job outcomes.

Table 10: Percentage of individuals achieving job outcomes comparing the business case to actual performance-to-date*

Job Outcome	Business Case Expectation	Actual Achieved Performance-to-Date
13 week job outcome	36%	23%
26 week job outcome	30%	18%
52 week job outcome	25%	15%

* It should be noted that achieved job outcomes figures are affected by a time delay and therefore the figures above do not represent a finalised outcome, as discussed in chapter 8.

As per the findings from the telephone survey when followed up as a longitudinal sample that it appears that certain individuals achieve job outcomes beyond the reporting period. This may suggest that FSS has a more positive longer term effect than is currently accounted for and would require a more in-depth longitudinal follow up to assess.

With regards to types of work there is some evidence that the types of jobs that participants move into after receiving support from FSS have improved with regards to job quality over the three years of delivery. This is evidenced via increased percentages of individuals in full-time work and receiving pay at or above the national living wage between Year 1 and Year 3.

Fairness, Dignity, Respect & Voluntary Participation

The concepts of a service which embodies fairness, dignity and respect as well as an entirely voluntary approach to participation is a key ethos behind the service design of FSS and sets it apart from many previous employability services.

Evidence gathered from across the three years of evaluation strongly supports that participants view the service as having successfully embodied these principles. Notably on average across the three years, approximately 93% of participants contacted by the telephone survey agreed that they were treated with dignity and respect by FSS providers and 93% on average also stated that they were aware that their participation was entirely voluntary. It should be noted that in relation to this latter point that there is some degree of variation depending on participants' demographics with certain groups such as those from minority ethnic groups less likely to recognize the voluntary nature of the programme. This suggests that while FSS has been successful overall in relation to this aim, that there is scope to improve the way that this ethos is communicated equally to all participants.

Service Integration & Alignment

As noted throughout the three years of evaluation, particularly via the local area case studies, providers and stakeholders have consistently fed back that FSS continues to operate in a cluttered employability landscape with providers in particular reporting that they feel that they are competing with other services for participants. This issue has also been exacerbated by existing funding arrangements which have resulted in consistent reports of difficulty working with other employability providers due to the risk of double funding. It should also be noted that these findings have been reported across all case study areas and are not isolated to particular areas.

Whilst there have been challenges noted in relation to service integration and alignment, there have also been positive lessons learned which can be taken forward, primarily to do with the positive impact that practices such as co-location and the use of dedicated liaison staff can have on fostering productive relationships between local organisations involved in the delivery of employability services.

It should also be noted however that that under the FSS approach there are limitations to what can be achieved at a local level given that there are a number of existing services in place which are not directly funded by the Scottish Government. However, in summary it is difficult to determine whether FSS has achieved the stated goal of promoting integration and alignment across the employability sector.

Furthermore it should also be recognised that since the launch of FSS the Scottish Government has undertaken additional work in this area, notably through the development of the No One Left Behind (NOLB) approach which aims to deliver employability services in partnership with Local Government and which is explicitly designed to support integration and alignment across the sector.

Support for those Further from the Labour Market

Numerous measures could be deployed to assess how well FSS has done in relation to its goal of supporting those further from the labour market. In particular use can be made of the groupings that participants are placed in as a proxy for barriers to employment, where Core represents those closer to the labour market and Advance and Intense represent those further away.

Compared to the original business case FSS had a much higher number of participants deemed Core than anticipated at 41% compared to 14% in the business case. As a consequence the number of participants in the Advance and Intense groups was significantly lower than planned at 33% realised compared to 50% in the business case and 24% realised compared to 36% in the business case respectively. This would therefore suggest that FSS has supported a higher proportion of participants who were deemed closer to the labour market, and a lower proportion of those further from the labour market than was originally intended.

Whilst acknowledging issues around missing data which have worsened during Year 3, looking at demographics of participants over the three years also allows us to begin to identify the emergence of potential trends over time. As per the earlier reach chapter of this report we can see that there is some indication of trends emerging which suggest that those with significant barriers to employment are joining the service at a lower volume over time. In particular:

• The percentage of individuals joining the service with disabilities dropped from 55% to 32% between Year 1 and Year 3.
• The percentage of individuals joining the service who have disabilities and have been unemployed for two or more years dropped from 21% to 9% between Year 1 and Year 3.
• The percentage of individuals joining the service who are from the 15% most deprived areas has dropped from 38% to 31% between Year 1 and Year 3.

Whilst recognising that these groups of individuals continue to be well represented in the service against the unemployed population of Scotland the results suggest that there is an ongoing trend in decreasing proportions joining the service. There are many potential explanations for this including that this shift may have been exacerbated by the onset of the pandemic. We know for instance that one of the primary barriers that participants report in relation to moving into work is a lack of confidence, which is something that could have been made worse by the onset of the pandemic and its broad detrimental effect on the economy. This in turn may have had a discouraging effect on individuals with significant barriers joining the service during Year 3. Furthermore those with serious health conditions may have been shielding and generally less willing to engage with services such as FSS during periods of lockdown.

These findings can be cross-referenced with a number of other findings from this year's evaluation, including those from the most recent local area case studies where providers noted a shift in clients towards those closer to work. It should be noted that while these findings are not comprehensive they are however indicative of a potential ongoing shift in the demographic of participants away from those the service was originally focussed on.

The shift seen away from jobcentres as a major source of referrals during the pandemic period may also have a role to play. Providers have been relying more on generating referrals through online marketing such as via social media. This approach may favour individuals with a higher degree of self-efficacy given the requirement to engage with FSS without formal support from JCP staff.

More broadly, given the tight nature of the labour market over the three years of delivery, with at times record levels of employment, it is possible that the pool of potential FSS participants has remained relatively stable. This may in turn partly explain year on year reductions in those with significant barriers as they may have either been offered an opportunity to join the service or actually taken part in earlier years. However further analysis would be required to quantify the degree to which this effect is taking place.

It should also be acknowledged that analysis of trends in participant demographics is complicated for Year 3 due to increased levels of missing data in Management Information reports as previously mentioned. As such findings should be interpreted with a degree of caution.

Person-Centred Support

Two of the key factors within the service design of FSS which contribute towards its aspiration of being person-centred are the use of dedicated key workers and the development of Personalised Employment Action Plans for participants.

In order to assess how effective FSS has been with regards to deploying a person-centred approach we can therefore utilise evidence gained from participants via the telephone survey. As part of the survey respondents were asked whether they recall being offered each of these elements of support and to what extent they found them useful. It should be noted that the findings in this section are to be treated as indicative as the survey findings used in this section were not specifically designed as a measure of performance in terms of whether FSS providers offered these specific elements of support to participants.

Ahead of discussing the findings it should also be noted however that the offer of a Key Worker and an Action Plan are mandatory elements of the pre-employment service offer that providers must make to participants.

Table 11: Percentage of individuals receiving pre-employment support who were offered a key worker and who found this useful

Year	% Receiving Pre-Employment Support Who Were Offered a Key Worker	% Receiving Pre-Employment Support from a Key Worker Who Found this Useful
Year 1	88%	79%
Year 2	88%	78%
Year 3	87%	77%

As can be seen in table 11 above, with regards to the use of a Key Worker 88% of respondents in the pre-employment stage of support stated they were offered this form of support on average across the three years, with numbers remaining consistent across this time period. Of those who took up the offer of a Key Worker, 78% on average stated that they found it useful.

Table 12: Percentage of individuals receiving in-work support who were offered a key worker and who found this useful

Year	% Receiving In-Work Support Who Were Offered a Key Worker	% Receiving In-Work Support from a Key Worker Who Found this Useful
Year 1	58%	91%
Year 2	49%	85%
Year 3	38%	79%

For those receiving in-work support, we can see from table 12 that 48% on average across the three years stated that they were offered Key Worker support with numbers declining from 58% in Year 1 to 38% in Year 3. Of those who took up this offer 85% on average stated that they found it useful, with numbers again declining from 91% in Year 1 to 79% in Year 3.

While it is clear that the majority of individuals who took up the offer of support from Key Workers found it useful, the evidence suggests that this offer has not been made consistently and moreover that the offer of this form of support may itself be declining for those in work in particular. While there is evidence from the survey that individuals closer to the labour market may feel less in need of certain forms of support compared to those further away and therefore may decline, it is less clear why individuals report year on year reductions in the actual offer of such support from providers.

Table 13: Percentage of individuals receiving pre-employment support who were offered a personalised employment Action Plan and who found this useful

Year	% Receiving Pre-Employment Support Who Were Offered an Action Plan	% Receiving Pre-Employment Support in the form of an Action Plan Who Found this Useful
Year 1	71%	77%
Year 2	61%	77%
Year 3	49%	76%

With regards to the development of employment Action Plans we can see from table 13 that 60% of respondents in the pre-employment stage of support stated they were offered an Action Plan on average across the three years, with numbers declining from 71% in Year 1 to 49% in Year 3. Of those who took up this support, 77% on average stated that they found it useful, with numbers remaining consistent across this time period.

Table 14: Percentage of individuals receiving in-work support who were offered a personalised employment Action Plan and who found this useful

Year	% Receiving In-Work Support Who Were Offered an Action Plan	% Receiving In-Work Support in the form of an Action Plan Who Found this Useful
Year 1	30%	89%
Year 2	24%	92%
Year 3	19%	77%

For those receiving in-work support, 24% on average across the three years stated that they were offered support in the form of an Action Plan with numbers declining from 30% in Year 1 to 19% in Year 3. Of those who took up this offer 86% stated that they found it useful, with numbers again declining from 89% in Year 1 to 77% in Year 3.

These findings suggest that while the use of a personalised Action Plan remain relatively highly rated by respondents, both in-work and in pre-employment support, that the offer of such support may not have been consistently applied and moreover appears to be declining across the three years of delivery.

These findings may reflect changes in the service delivery approach caused by COVID-19 with participants being less clear on the type of engagement they have received as the result of providers necessarily adopting a virtual support approach. The findings may also reflect changes in the demographic of participants over time with providers feeling that it is less necessary to offer such support to certain individuals based on their proximity to the labour market. However if FSS is to meet its stated aims of delivering a person-centred service then the offer of such support should be consistently applied across the range of participants accessing the service. Further work may be required on this issue to better understand the indicative changes that are reflected in the survey findings.

What worked well?

There is robust evidence from across the three years of evaluation to demonstrate that FSS has successfully implemented a service based on principles such as fairness, dignity, respect and voluntary participation.

What were the challenges? / How could we improve?

While there is some evidence that FSS provides a person-centred service there remain scope for improvement in relation to this. In addition, there remains room for improvement with regards to FSS's goals around delivering sustainable job outcomes, decluttering the employability landscape and supporting those furthest from the labour market.

What is Scottish Government doing?

The key principles and values that underpin Fair Start Scotland are shared by No One Left Behind by virtue of both being grounded in the Scottish Approach to Employability that was developed in preparation for devolution of employability powers in 2017. As a result, the findings from the past three years of evaluation are a valuable source of learning, not just for improving FSS services, but also for the future development and delivery of No One Left Behind.

We are disappointed not to have made better progress on improving our service, particularly with respect to participants facing more complex and / or multiple barriers. However, our focus for the past year has, by necessity been on ensuring a basic continuity of support for existing and new participants in the face of the significant practical, financial and health challenges pose by the COVID-19 pandemic.

With regard to the remaining 18 months of FSS delivery in its current guise, we restate our commitment to continuous improvement of our national service. We are working closely with service providers, JCP, local government and other partners to drive better engagement with those who are currently underrepresented and to enhance our support offer for participants who are further from the labour market.

We are already incorporating learning from this and previous evaluations across a range of activities, including:

• Service Provider Action Plans and Tasks & Finish Groups to identify and overcome delivery challenges;
• ongoing reviews of operational guidance, job outcome definitions, performance indicators and elements of the delivery model in light of COVID-19 and labour market changes; and
• test and learn pilots to explore and overcome the challenges of service delivery for specific groups of participants, or in specific circumstances.

We openly acknowledge that the FSS model has both strengths and weaknesses, and we will use all participant, provider and partner feedback to improve both the current service and future delivery of devolved employability support.

Moving forward, user experience in planning and improving employability services will remain of crucial importance for future services. Our commitment to embedding the Scottish Approach to Service Design, which sees empowering users to participate actively in design as a critical component of planning delivering and improving services, will help us strengthen the voice of those accessing and participating in services.

This ongoing commitment to the involvement of users has been embedded in the early stages of No One Left Behind Phase 2, as people with experience of accessing employability support have been working with us to shape the expectations of services and the products that will accompany live delivery.

In addition to this, we are working to develop a Shared Measurement Framework which places user experience as a key measure of success for services, alongside indicators that are more commonly used, such as job outcomes, to allow us to build a more holistic picture of service impacts.