Fair Start Scotland: evaluation report 4 - year 3 overview

9. Economic Evaluation: Impact, Performance and Value for Money

For this year's report the Scottish Government commissioned an independent economic evaluation which was undertaken by Alma Economics. This component of the evaluation had three broad objectives:

• To understand the value for money of the service by comparing costs and benefits.
• To understand the value for money of the service through wider measures such as unit costs to allow comparisons with other similar employability programmes.
• To understand the wider social impact of the service, including wellbeing and inclusive growth.

A summary of the key findings as well as their relationship to findings from the broader evaluation are detailed below. A complete report of findings from the economic evaluation has been published separately as: Economic Evaluation of Fair Start Scotland.

9.1 Methodology

The economic evaluation utilised the DWP Social Cost Benefit Analysis (SCBA) model as the primary mechanism for analysis. This model is based on work undertaken by Fujiwara (2010)[19] and has been utilised by the DWP for evaluation of a number of programmes including the Work Programme evaluation, Work Experience evaluation and the evaluation of the Future Job Fund.

In order to use the model, several parameters have to be set, details regarding which can be found in the aforementioned standalone report. It is worth noting that these parameters were established using a number of sources including findings from other components of the evaluation (most notably the participant telephone survey), Management Information data and by consulting parameters used by other similar evaluations. Assumptions built into the model were tested for robustness by the contractor via undertaking a variety of sensitivity checks as part of the analysis.

As inputs the model uses programme costs per year, employment outcomes (hours worked, wage received, and number of days at work), as well as participant characteristics (age, marital status, disability status, and number of children).

Outputs from the model detail costs and benefits from the perspective of participants, society, public finance and employers. For the purposes of this evaluation, findings have been focussed on the first three of these groups,^[20] with analysis primarily directed at the societal perspective as this provides the most nuanced picture of impacts. The outputs are in turn available for analysis at a range of levels, including breakdowns by participant group and by geographical Lot. It should be noted that benefit-cost ratios are shown both with and without the inclusion of Quality Adjusted Life Years (QALYs)^[21] and wider distributional effects^[22] (results labelled 'financial' are those without and 'total' those with QALYs and distributional impacts included). Outputs are presented this way to allow comparison with both the business case for Fair Start Scotland as well as to other similar programmes where possible.

9.2 Results

Cost-Benefit Analysis – Results

A summary of the results of the cost-benefit analysis (CBA) are presented in Table 2, Table 3 and Table 4 below from society, participant, and public finance perspectives respectively. The tables also include comparisons against the FSS business case.

Table 2: Results of the cost-benefit analysis from the perspective of society

Society's perspective	Business Case	Current CBA
Total benefits	£177.8m	£167.5m
Total costs	£117.3m	£82.8m
Programme Costs	£109.2m	£69.9m
Financial BCR	1.5	2.0
Total BCR	-	3.6

Table 3: Results of the cost-benefit analysis from the perspective of participants

Participant's perspective	Current CBA
Total benefits	£199.3m
Total costs	£140.4m
Financial BCR	1.4
Total BCR	2.6

Table 4: Results of the cost-benefit analysis from the perspective of public finance

Exchequer's perspective	Business Case	Current CBA
Total benefits	£128.7m	£110.7m
Programme Costs	£109.2m	£69.9m
Total BCR	1.2	1.6

Source: Analysis of Management Information (MI) data, Wave 3 survey data, cost data and post-2021 forecasts.

The results show that FSS is a net positive to society, public finance and from the participants' perspective. For every £1 spent the estimated benefit is:

• £2 in 'financial' terms for society and £3.60 in terms of overall benefits.
• £1.40 in 'financial' terms for participants and £2.60 when QALYs and distributional impacts are incorporated.
• £1.60 in benefits to public finance.

As shown in the above tables, the current BCR's of FSS exceed the business case expectations. From the perspective of society, the business case BCR was 1.5 and the current BCR 2.0 and from the perspective of public finances the business case BCR was 1.2 and the current BCR 1.6. Reasons for this are discussed in the following sections.

Unit Costs

While benefit-cost ratios provide a relatively comprehensive assessment of value for money it is recognised that it is not always possible to compare BCR's with other programmes due to complexities associated with analysis. Therefore in addition to BCR's, the costs per job start was also calculated and can be found disaggregated in the tables below by cohort and participant group.

Table 5: Cost per job start for Year 1 across participant group and Lot

Participant group	Cost per job start
Core	£4,849
Advanced	£8,516
Intense	£10,261
Lot	Cost per job start
Glasgow	£6,586
Lanarkshire	£5,469
Tayside	£5,461
Forth Valley	£8,897
East	£6,005
South West	£7,189
North East	£9,129
Highlands and Islands	£11,442
West	£8,886
Total	£6,754

While average costs per job equalled £6,754 it is recognised that there is significant variation across characteristics. For example, costs are lower for individuals within the Core group compared to those in Advanced or Intense, due to the fact that providers are paid at a lower rate for Core group participants as detailed in the introduction section of this report.^[23]

Comparison with Similar Programmes

Due to variation in programme design, aims, reach, scale and in the definition of outcomes, caution has to be applied when making comparisons between FSS and other employability programmes.

The most suitable comparator which has been identified in relation to cost-benefit analysis is DWP’s Work Programme^[24] which also aimed to move participants into employment, used a similar payment-by-results model and was also evaluated using DWP's SCBA model. It should however be noted that there are significant differences between the two programmes. Most notably the Work Programme was a UK-wide programme, which ran over six years and had close to two million participants - a large difference in scale compared to FSS. Furthermore, The Work Programme was also not voluntary and did not target people with particular characteristics. These differences should be kept in mind as they are likely to have had some impact on the variation seen in results.

Table 6: Comparison between Fair Start Scotland and the Work Programme evaluation results

Comparison measures	Society 'financial' BCR	Public Finance BCR	Participant BCR
Work Programme	2.7	2.4	1.3
Fair Start Scotland	2.0	1.6	1.4

As can be seen in the table above are broadly similar, FSS performs slightly less well than the Work Programme with regards to BCR's. It is likely that some of this variation can be explained by differences in scope and design as mentioned above.

Additional non-BCR value for money comparisons with other employability programmes are available in the previously linked standalone economic evaluation report.

Inclusive Growth Considerations

The Scottish Government defines inclusive growth as economic growth which "combines increased prosperity with greater equality, creates opportunities for all, and distributes the benefits of increased prosperity fairly".^[25] It is recognised that the aims and design of FSS inherently led to some aspects of inclusive growth given it seeks to increase economic activity of low-income groups.

In order to determine how well FSS has contributed to inclusive growth, the economic evaluation disaggregated the cost-benefit analysis by three variables, namely by geographic Lot, participant group (Core, Advanced, Intense) and gender.

Table 7: Results of the cost-benefit analysis disaggregated across gender

Gender	Women	Men
Total benefits	£54.7m	£112.5m
Total costs	£29.6m	£53.2m
Total benefits (QALYs, redistributive effect)	£102.0m	£197.3m
Financial BCR	1.9	2.1
Total BCR	3.4	3.7

The table above shows the outputs of the cost-benefit analysis by gender. While the BCR for both is very similar, men achieved both higher costs and higher benefits. This is due to the fact that more men achieved job outcomes than women, reflecting the higher level of men who joined FSS and the higher number of men in the unemployed population of Scotland. While this suggests that the higher levels of outcomes for men is partly explainable by the characteristics of the unemployed population, it suggests that there may be additional work to do to increase participation levels and outcomes for female participants as substantiated by findings from the rest of the evaluation.

Table 8: Results of the cost-benefit analysis disaggregated by Lot

Lot	Financial BCR	Total BCR	Public Finance BCR
East	2.2	3.9	1.7
Forth Valley	1.3	2.4	0.9
Glasgow	2.1	3.7	1.7
Highlands and Islands	1.5	2.6	1.1
Lanarkshire	2.0	3.6	1.5
North East	1.7	3.0	1.3
South West	2.1	3.9	1.7
Tayside	2.2	3.9	1.7
West	1.7	3.1	1.3

The above table shows the breakdown of the cost-benefit analysis by geographic Lot. It demonstrates that there is significant variation in BCR by geography, with the lowest financial BCR being 1.3 in Forth Valley and the highest being 2.2 in both the Tayside and East Lots. This is explainable by the fact that the areas with the lowest BCR's had the lowest percentage of job starts achieved, as well as the highest percentage of participants within the Intense group, both of which contribute to higher costs and lower benefits. This is discussed further below.

Table 9: Results of the cost-benefit analysis disaggregated by participant group

Group	Financial BCR	Total BCR	Public Finance BCR
Core	2.5	4.4	2.0
Advanced	1.7	3.1	1.3
Intense	1.3	2.5	1.0

The table above provides a breakdown of the BCR by participant group and shows that Core participants have a significantly higher financial BCR at 2.5 than either Advanced or Intense participants, who achieve BCR's of 1.7 and 1.3 respectively. This is due to the fact that payments to providers for achieving outcomes for Core participants are lower than for the other two groups.

This wide variation in BCR by participant group is a significant explanatory factor for variations seen at Lot level as areas with higher levels of Core participants achieved higher value for money than those with higher levels of Advanced and Intense participants. Potential reasons for variation in participant type at Lot level include:

• Variation in service delivery model at the Lot level. Notably the area with the lowest BCR, Forth Valley, is Local Authority led and is focussed on providing support to those in the Intense group.
• Variations in referral source, with referrals generated via marketing and social media campaigns tending to fall into the Core group versus those gained from JCP's. Urban Lots have been more successful at generating these Third Party Referrals thereby outperforming rural areas with regards to BCR's.
• There is some degree of interpretation that can be applied by providers when assigning individuals to a group, given that there is some overlap in criteria. Providers may therefore make different decisions on which group a given participant best fits.

It should be noted that while areas which saw more Core participants achieved a higher BCR, that this is not entirely in keeping with the overall ethos of Fair Start Scotland which was designed with the express purpose of targeting those further away from the labour market. More broadly the combination of significant variation in BCR by participant group coupled with the much higher than expected number of participants joining as part of the Core group for the service as a whole, and thereby signalling a greater proximity to the labour market, is the key contributing factor to the overall BCR achieved by FSS and explains why the service outperformed expectations with regards to the original BCR's set within the business case. This is explored in more depth in the following chapter.

More detailed findings regarding the Inclusive Growth related impacts of FSS, including analysis of job quality and stability can be found detailed in the standalone economic evaluation report.

What worked well?

The economic evaluation clearly demonstrates that FSS provides a net-positive return on investment from the perspectives of society, participants and public finances. Furthermore FSS exceeded expectations with regards to value for money as articulated in the original business case.

What were the challenges? / How could we improve?

While FSS performed well in relation to value for money this was found to be due to a larger than anticipated proportion of individuals with fewer barriers to employment joining and benefiting from the service.

As per findings within the more detailed economic evaluation publication additional consideration could be given towards reviewing the manner in which providers are paid for job outcomes with an appraisal around the potential for increased flexibility regarding this.

What is Scottish Government doing?

As detailed elsewhere in the report, SG is already delivering continuous improvement activity with service providers aimed at addressing the under-representation of key participant groups. We are also developing a number of "test and learn" pilots, through which we will gather and share learning on what works to better engage and support participants with multiple / complex barriers to employment.

With only 18 months remaining on the extension of current FSS contracts, we are limited in the extent to which the FSS financial model can be amended. However, the recommendations from the economic evaluation will be used, along with the wider evaluation evidence to inform the development of more locally-determined and flexible service standards for future delivery of No One Left Behind.