Fair Start Scotland: economic evaluation

Benchmarking Fair Start Scotland's performance

In order to put the Fair Start Scotland evaluation in context, the results of the analysis will be compared to the business case and other similar programmes. The former will allow for assessing the actual performance of Fair Start Scotland compared to the expected performance, while the latter will help benchmark the experience of Fair Start Scotland against comparable programmes.

Comparison with the business case

Overall, as illustrated in Tables 3 and 5, Fair Start Scotland performed much better than expected compared to the business case. This is mainly due to the lower implementation cost of Fair Start Scotland than initially expected, as the benefits are remarkably similar ex-post and ex-ante. There are several differences between the expected and actual performance of Fair Start Scotland. Some of these are shown in Table 8.

Table 8. Comparing business case indicators with the actual performance of Fair Start Scotland (Year 1 to Year 3)

Indicator	Business case	Actual performance
Share of referrals that turned into participants and started on the service	70%	67%
Number of participants that started on the service	38,000	32,504
Number of participants that started on the service per year	12,642 per year	10,063 (2018) 12,085 (2019) 10,356 (2020)
Percentage of Core group participants[15]	14%	41%
Percentage of Advanced group participants	50%	33%
Percentage of Intense group participants	36%	24%
Percentage of participants who achieved the 13-week outcome	36%	23% (for the 2018 cohort)[16]
Percentage of participants who achieved the 26-week outcome	30%	18% (for the 2018 cohort)
Percentage of participants who achieved the 52-week outcome	25%	14% (for the 2018 cohort)
Programme cost per start	£2,928	£2,186 (for the 2018 cohort)

Source: Business case, management information data, and performance data.^[17]

Fair Start Scotland had lower overall participants starting on the service compared to the business case (38K vs 32.5K). It also had a much higher representation among the Core group in the actual performance than the business case. Subsequently, it had lower participation in the Advanced (33% compared to 50% in the business case) and Intense groups than anticipated (24% compared to 36% in the business case). Finally, fewer participants achieved the 13-, 26-, and 52-weeks outcomes (for the 2018 cohort) than anticipated in the business case.

These discrepancies can explain why costs were lower while benefits were higher when compared to the business case. This is best illustrated when comparing the management information to the survey results regarding how long the benefits last. For the 2018 cohort, the share of people who achieved the 52 weeks outcome is 14%. But when people were surveyed (up to) three years later, 25% reported that they had a job. This provides evidence of job outcomes being achieved but without triggering incentive payment to providers. Therefore, despite perceived underperformance compared to business case expectations, evidence suggests that Fair Start Scotland may have achieved higher benefits once long-term benefits are considered.

An additional reason for the discrepancy between the actual and anticipated performance is the reduced costs due to the higher share of Core participants. Providers are paid less for Core participants than for the other two groups, resulting in similar benefits but lower costs. This has an implication for the inclusive growth aspect of the programme, as it was less able to target individuals with high barriers to labour market entry than it set out to do. This will be discussed in more detail in the inclusive growth section.

Comparison with other programmes

It is important to be very careful when comparing the Fair Start Scotland evaluation results with those of other employment programmes. Even if programmes have similar overarching goals, they may still differ in their specific aims, design, and reach. The way outcomes are defined and measured is likely to be heterogenous. The scale of the programme can also distort results; for example, bigger, nation-wide programmes that operate for longer are likely to benefit from economies of scale.

In addition to specific programme characteristics, evaluation design and context may also hinder direct comparisons. For example, other programme evaluations are likely to include or exclude different aspects of cost-benefit analysis compared to the choices made in this evaluation. Some programmes may use control groups to assess the counterfactual or may follow participants for a longer period of time. Additionally, in many cases, the relevant information is not shared clearly and transparently, making it unclear whether comparisons are on a like-for-like basis.

Nevertheless, it is important to benchmark Fair Start Scotland against similar services to understand how well it performed relative to others. This section compares the performance of Fair Start Scotland to other programmes on several measures, including value for money, performance, reach, costs, and job outcomes achieved. The programmes used for comparison are either UK-wide or Scotland based. The reason international comparisons are not undertaken is to ensure that the context is as similar as possible.

In terms of cost-benefit analysis, the most suitable comparator to Fair Start Scotland identified in the literature is the Work Programme. This is mainly because the evaluation of the Work Programme uses the DWP SCBA, meaning BCR comparisons are relatively straightforward.^[18] The Work Programme also aimed to move people into employment and has a payment-by-results model similar to Fair Start Scotland.

However, there are also many key differences. The Work Programme was a very large, UK-wide programme that operated for 6 years with close to 2 million participants.^[19] Unlike Fair Start Scotland, the Work Programme was not voluntary and did not target people with particular characteristics.^[20] These differences mean that comparisons need to be interpreted carefully, and any conclusions made need to be caveated within this context. Table 9 shows a comparison between the different BCRs of Fair Start Scotland and the Work Programme.

Table 9. Comparison between Fair Start Scotland and the Work Programme evaluation results

Comparison measures	Society 'financial' BCR	Public Finance BCR	Participant BCR
Work Programme	2.7	2.4	1.3
Fair Start Scotland	2.0	1.6	1.4

Source: Analysis of management information, Wave 3 survey data, cost data, post-2021 forecasts
and the Work Programme's quantitative impact assessment.

The results are relatively similar, with Fair Start Scotland performing slightly less well than the Work Programme in terms of BCR measures of value for money. This is likely due to the differences in design and scope, as discussed previously. This includes distance and the size of the population. Having to work in remote and scarcely populated areas of Scotland may mean that employment provision is more difficult and costly when compared to the Work Programme. Additionally, Fair Start Scotland is specifically designed to work with participants who have high barriers to finding a job and are further from the labour market. This means that the cost per participant is likely to be much higher than that of the Work Programme. Finally, as previously mentioned, the Work Programme was a very large programme that lasted 6 years and included more than 2 million participants. This will have led to economies of scale that are likely to affect value for money positively.

The voluntary nature of Fair Start Scotland may also have implications for the results. Unlike the Work Programme, participation in Fair Start Scotland is not a condition for receiving or continuing to receive benefits. This may have several effects. On the one hand, this means participants may have less incentive to remain in the programme until they find employment, potentially implying higher incurred costs for Fair Start Scotland that are not offset by benefits. However, because participants voluntarily decide whether they want to participate in Fair Start Scotland or not, the programme likely attracts those who are genuinely motivated and want to find employment. This means that, compared to the Work Programme, Fair Start Scotland is likely to help fewer people achieve job outcomes but those who do achieve better and longer-lasting outcomes.

Beyond BCR measures, there is some scope for comparison between Fair Start Scotland and the Work Programme regarding participants' experience. Participants with experience in the Work Programme felt that Fair Start Scotland key workers are more supportive and respectful than Work Programme advisers. Participants also disliked the compulsory nature of the Work Programme and the risk of sanctions for non-completion or non-participation.^[21]

It is also possible to benchmark how well Fair Start Scotland performed relative to other programmes by using alternative measures of performance. Given the design and the available information on performance and reach, the Work and Health Programme and the Work Choice Programme are identified as suitable comparators. The two programmes share some similarities with Fair Start Scotland. All three aim to help people find and stay in employment. They also target similar (but not identical) groups, namely people with disabilities and those farther away from the labour market. All three programmes are, for the most part, voluntary.^[22]

Table 10 compares Fair Start Scotland with the Work and Health Programme and the Work Choice Programme on reach. Other information about the programmes is also shown, including years and location of operation and target population. This is to ensure that the context is taken into account when the comparison is undertaken.

Table 10. Fair Start Scotland's reach performance compared to the Work and Health Programme and the Work and Choice Programme

Comparison measures	Number of starts	Number of years in operation	Average number of starts per year	Target population	Operation locations
Work Choice Programme	158,440	2010/11 -2017/18 (8 years)	19,805	People with disabilities	UK wide[23]
Work and Health Programme	150,104	2017-2021 (3.5 year)	42,887	People with disabilities, long term unemployed, early access[24]	England and Wales
Fair Start Scotland	32,505	2018/19-2020/21 (3 years)	10,835	As per definitions in Table 1	Scotland

Source: Performance data of the Work Choice Programme,^[25] Work and Health Programme,^[26] and Fair Start Scotland.

Given the significantly narrower operation of Fair Start Scotland compared to the other two programmes, it performed remarkably well in terms of reach. While both the Work and Health Programme and the Work Choice Programme reached a larger number of people per year, they operated at a much larger scale than Fair Start Scotland.

In order to draw firm conclusions about Fair Start Scotland's reach relative to the other programmes, performance should be benchmarked against population measures. However, given the diversity of groups, locations, and years, this is not straightforward and needs to be interpreted carefully. Population measures are illustrated in Table 11.

Table 11. Fair Start Scotland's reach of unemployed people compared to the Work and Health Programme

Comparison measures	Average annual number of starts	Average number of unemployed people in the region/time period of operation	Percentage of starts against unemployed population
Work and Health Programme(2017-2021)	42.9K	1.3m[27]	3%
Fair Start Scotland(2018-2020)	10.8K	110K[28]	10%
Comparison measures	Average annual number of starts with disabilities	Average number of unemployed people with disabilities in the region/time period of operation	Percentage of starts with disabilities against unemployed population with disabilities
Work Choice Programme[29](2013-2018)	18.2K	404K	5%
Work and Health Programme(2017-2021)	32K	351K[30]	9%
Fair Start Scotland(2018-2020)	4.7K	29.5K[31]	16%

Source: Performance data of the Work Choice Programme, Work and Health Programme, and Fair Start Scotland
and national statistics on the number of unemployed people.

Table 11 shows that the reach of Fair Start Scotland was higher than the Work and Health Programme. However, these results need to be interpreted carefully as the population figures are imperfect. Additionally, there may be some inconsistencies across definitions of the targeted population across the years, locations, and programmes. Suffice to say that Fair Start Scotland performed comparatively well in terms of reach.

It is also possible to compare the performance of Fair Start Scotland to the other two programmes in terms of job outcomes achieved. It is important to note that each one of the programmes defines job outcomes very differently, as noted in the table, so any conclusions made need to be considered carefully.

Table 12. Fair Start Scotland's referral to start and job outcome performance compared
to the Work and Health Programme and Work Choice Programme

Comparison measures	Referral to start rate	Job outcome 1	Job outcome 2	Job outcome 3
Work and Health (2017-2021)	62%	5% (Reached a level of earnings once in employment within 6 months)	14% (Reached a level of earnings once in employment within 12 months)	-
Work Choice Programme(2010-2017)	75%	34% (Job is sustained for 3 months in a 12-month period)	21% (Job is sustained for 6 months in a 24-month period)	-
Fair Start Scotland(2018)	67%	23% (In work for 13 weeks in a 16-week period)	18% (In work for 26 weeks in a 30-week period)	14% (In work for 52 weeks in a 60-week period)

Source: Performance data of the Work Choice Programme, Work and Health Programme, and Fair Start Scotland.

Fair Start Scotland performed relatively well compared to the other two programmes. In terms of retention, Fair Start Scotland's referral-to-start rate was between that of Work Choice and Work and Health. Job outcomes for Fair Start Scotland and the Work Choice Programme are more comparable than the Work and Health Programme because the latter takes into account the level of earning, not just the longevity of the job outcome. Fair Start Scotland achieved a lower rate of job outcome 1 than the Work Choice Programme. This is likely due to the time needed for the job outcomes to be achieved, which is much tighter for Fair Start Scotland (16 weeks) than the Work Choice Programme (12 months). The rates of achieving job outcome 2 confirm the evidence that Fair Start Scotland leads to stable jobs; 78% (or 18 out of 23) of Fair Start participants who achieved the first job outcome went on to achieve the second one, as opposed to 62% (21 out of 34) for the Work Choice Programme.

Finally, it is possible to compare Fair Start Scotland with other programmes in terms of cost per job outcome. Similar programmes identified in the literature that use these measures are the Working for Families Fund and New Futures Fund. Table 13 compares the cost per participant and job outcome across the three programmes. The cost per participant and cost per job outcome 1, 2, and 3 are calculated based on information for the 2018 cohort only for Fair Start Scotland. This is because the programme starts of the participants from the Fair Start Scotland 2018 cohort had been fully observed, and job outcome and programme cost figures for the 2018 cohort were near final at the time this analysis was undertaken.

Table 13. Comparing Fair Start Scotland to the Working for Families Fund and New Futures Fund

Comparison measures	Cost per start/participant	Cost per job outcome 1	Cost per job outcome 2	Cost per job outcome 3
Working for Families Fund (2004-2008)	£1,642 nominal £2,430 real	£3,382 nominal £5,004 real	-	-
New Futures Fund (1998-2005)	£2,100 nominal £3,395 real	£6,100 nominal £9,863 real	£9,300 nominal £15,037 real	£21,100 nominal £34,116 real
Fair Start Scotland (2018 cohort)	£2,186 real	£9,361 real (outcome 1) (£6,754 real per job start of any type)	£11,827 real	£15,394 real

Source: Performance data of the Working for Families Fund,^[32] New Futures Fund,^[33] and Fair Start Scotland (2018 cohort).

The two programmes are suitable comparators to Fair Start because all three operated in Scotland and targeted people with similar profiles. However, since the other two programmes are significantly older than Fair Start Scotland, it is important to use real costs when making the comparison.^[34] Additionally, job outcomes are defined very differently in the three programmes. As mentioned, Fair Start Scotland defined job outcomes 1, 2, and 3 as achieving 13, 26, and 52 weeks in employment, respectively. The Working for Families Fund defined job outcomes as the transition into employment. The New Futures Fund defined job outcomes 1, 2, and 3 as moving into education, supported employment, and employment, respectively.

Taking all of this into account, Fair Start Scotland had lower costs per programme start compared to the other two programmes. Regarding costs per specific job outcome, when comparing real values, Fair Start Scotland performed better than the New Futures Fund but worse than the Working for Families Fund.

This is not surprising, given Fair Start Scotland's much stricter definition of a job outcome than that used in the Working for Families Fund. Indeed, accounting for differences in job outcome definitions, it may be fairer to compare the cost per job outcome 1 of Working for Families Fund (£5,004) to the cost per job start of Fair Start Scotland (£6,754). Considering this, Fair Start Scotland still had a higher cost per job start but the difference between the two programmes is much lower.