Fair Start Scotland: economic evaluation

The DWP social cost-benefit analysis model

Overview of the literature

There are several impact assessments conducted by the UK government which contain a cost-benefit analysis element that relies on the DWP SCBA framework. These include the Work Programme evaluation,^[3] Sector-Based Work Academies,^[4] the Work Experience evaluation,^[5] and the Future Job Fund evaluation.^[6] All these evaluations use a very similar methodology following the work of Fujiwara (2010), ^[7] which will be discussed in detail in the next section. This section will outline some of the key decisions undertaken in this evaluation and how and why they diverge from those undertaken in the aforementioned impact assessments.

In terms of the parameters used, this work follows similar steps to the other DWP SCBA evaluations. It makes use of redistribution effects, costs of childcare and transport, and benefits accrued from improvements in health outcomes. The magnitude of these effects and how they are modelled within the DWP SCBA framework are discussed in Annex A and in the next section.

Similar to the other evaluations outlined in this section, central to this work is estimating a realistic timeframe for how long the benefits of the intervention last for based on the best available data (the persistence effect). Since the participants of Fair Start Scotland are observed for a short period of time, assumptions on their future labour market behaviour need to be made to approximate the persistence effect. The other evaluations outlined in this section do the same, but since the results are based on assumptions, they vary the timeframe for the sensitivity checks. Following this approach, this evaluation will also estimate the results for a pessimistic and an optimistic scenario based in part on varying assumptions on the length of the persistence effect.

There are several areas in which this evaluation diverges from the rest. Firstly, in terms of parameters used, the other evaluations make use of the social cost to public finance to conduct sensitivity checks. This is excluded from this analysis due to Scottish Government precedents for treatment of public finance costs and concerns about how robust the underlying parameters are for this aspect of modelling. Secondly, in terms of defining the counterfactual, most of the studies use an Intention to Treat methodology to capture the additionality of their programmes. Since such data is not available for Fair Start Scotland, this analysis makes use of assumptions built into the model to account for the counterfactual. Again, since this is based on a level of uncertainty, it is varied in the sensitivity checks.

Finally, it is important to highlight that there are some implications due to Fair Start Scotland being a Scottish policy and not a UK-wide one like the other programmes that use the DWP SCBA model. This is especially the case when it comes to the meaning of public finance costs in the Scottish context. Despite this, public finance improvements from the reduction in Universal Credit are still included in the DWP SCBA to accurately capture the performance of Fair Start Scotland and to enable its comparability to other policies.

Overview of the DWP Social Cost Benefit Analysis

This chapter outlines and summarises the DWP SCBA framework and its methodology for estimating and incorporating the wider social and economic impacts of employment programmes.

Fujiwara (2010) provides the framework of the DWP SCBA, discussing the different factors that movement into employment are likely to impact from a society, individual, and public finance perspective. Based on that, the report gives recommendations on parameters to include and why, citing evidence for the inclusion, exclusion, and caution surrounding each parameter. The DWP SCBA gives the user a choice of which parameters to include in their evaluation and which to exclude. The evidence for each of the parameters as discussed by Fujiwara (2010) are summarised and discussed in this section. The implications on the Fair Start Scotland evaluation are discussed in the next section.

In order to capture the full effect of an employment programme, Fujiwara (2010) discusses evidence on the impact of employment programmes on increases in income, in-work costs, leisure time foregone, effects on employment rates (substitution effects), social costs associated with funding programmes through taxation, improved health outcomes, reductions in crime rates, multiplier effects on the economy, and the value of increased economic output. Based on the evidence, recommendations are given on whether and how to include these parameters in the DWP SCBA.

Starting with the increase in income, the value that individuals place on each additional pound they receive or lose is higher for people with low incomes relative to people with higher incomes. To account for this in employment programmes, monetary gains for programme participants should be weighed higher than the costs for taxpayers who fund the programmes (welfare weight). The weight for programme participants is set at 2.5 (relative to the average taxpayer).^[8] This weight is applied to any increases or losses in income incurred by participants as a result of the employment programme.

As for in-work costs, when someone moves into employment from unemployment/inactivity there are some unavoidable costs they may incur. These costs reduce the real gains from working. They include travel to and from work, and possible childcare costs for lone parents or families with second earners. Average childcare costs and average transportation costs should be subtracted from the income received from employment.

There is strong evidence of a positive causal effect of employment on individual health outcomes. Robust quantitative evidence comes from studies that look into the effect of employment on medical service usage rates. This evidence is used to estimate the National Health Service (NHS) savings per additional person employed. NHS savings are predicted to be larger for Employment and Support Allowance (ESA) participants who find work, as they incur greater initial public health costs. These resource savings should be included in the cost-benefit framework. It is estimated that one person moving from unemployment to employment incurs £508 less in NHS costs per year (non-ESA programmes). This figure rises to £1,016 for a person with disability who moves from unemployment to employment (ESA programmes).

The latest version of the DWP Model allows for the inclusion of a Quality Adjusted Life Years (QALYs) element. QALYs are a measure of disease burden, including both the quality and the quantity of life lived. They are used in economic evaluations to assess an intervention, where one QALY equates to one year of perfect health. The size of this effect is not discussed in Fujiwara (2010), but Public Health England sets it at 0.0675 QALY gain per person moving from unemployment to employment, when assuming the benefit is sustained for one year.^[9]

The framework also points to benefits and costs incurred from foregone leisure, multiplier effects for the wider economy, and increased economic output. These parameters are difficult to measure and, in some cases, may be offset by other hidden benefits or costs. Therefore, the DWP SCBA model recommends not including these parameters in the cost-benefit analysis.

Finally, the DWP SCBA framework recommends that some parameters be used for sensitivity checks. These include social costs associated with funding programmes through taxation and effects on employment rates (substitution effect). For the social cost of taxation, it is estimated that the net fiscal benefit from additional jobs should be multiplied by 0.2 to get the social cost of public finance. However, the evidence on this is weak and relies on strong assumptions. Therefore, it is recommended that it is not used in the main analysis.

As for the substitution effect, it is estimated to be 0.2 for supply-side programmes (like Fair Start Scotland) and 0.4 for demand-side programmes. The substitution effect also relies on weak evidence, and the recommendation is for it not to be used in the main analysis. There is no built-in measure in the DWP SCBA to account for a counterfactual. That is, the deadweight arising from the people who would have found a job without participating in the employment programme. According to DWP however, the 0.2 substitution effect is sometimes used to account for the counterfactual in the absence of a control group.

The DWP SCBA framework excludes a number of potentially significant costs and benefits due to a lack of robust evidence. These measures include the additional leisure time which participants forego, the non-pecuniary benefits associated with additional time in unsubsidised employment, and the economic multiplier effect which may result from the programme. The programmes outlined above also do not account for the reduction in crime that may result from movement to employment and do not include the costs of hiring and training incurred by employers (Fujiwara 2010).

Model choices and perspectives

The DWP SCBA allows for the analysis to be considered from different perspectives. These include participants, public finance, employers, and the wider society. This analysis will mainly consider the perspective of society. This is because it shows the richest picture of the impact of the programme as it takes into account wider determinants and incorporates the perspective of various stakeholders, including both taxpayers and programme participants. The public finance perspective will also be considered, given the implication that this has on the value for money of the service. The participants' perspective will also be briefly discussed to isolate the impact of this service on those who sought it. There is not enough information to capture the cost-benefit analysis from employers' perspective, so it will be excluded.

The DWP SCBA model takes values at the individual level, which can then be aggregated to represent their groups. The approach is, therefore, to create a number of groups of Fair Start Scotland participant types, which will be considered separately. The groupings capture the range of different Universal Credit treatments based on characteristics such as age, disability status, and number of children. Details on group choices, data decisions, and summary statistics are available in Annex A and Annex B. For participants in later cohorts, forecasts were used to provide estimates of any remaining employment outcomes and associated costs beyond the period for which data is held. This allows a comprehensive estimate of the full costs and benefits associated for the three cohorts. It should be noted that aggregate results rely very little on these forecasts (which represent only a small proportion of overall job outcomes) and any inaccuracies in forecasts will increase or decrease both costs and benefits (with job outcomes causing both costs and benefits).

With regards to parameter choices, the model includes distributional analysis to capture the higher welfare achieved from the distribution of public finance. The social cost of the exchequer is excluded because of the weak evidence on its size and impact. The substitution effect is not included for similar reasons.

The model also includes benefits that come from the improvement of the health of participants upon finding employment. This happens through two channels, the first is savings to the NHS and the second is improvement in the quality of life because of movement from unemployment to employment, measured by QALYs.

The counterfactual will be set at 20% as recommended by the DWP SCBA. This implies that 20% of the benefits would have still been achieved even in the absence of the programme. This is one of the central assumptions that greatly impacts the results and will therefore be varied for the sensitivity checks to account for a more pessimistic scenario and a more optimistic scenario.

The scale of benefits will depend on how long the benefit of the programme lasts. For the case of Fair Start Scotland, the persistence effect translates to the length of time in which participants have a job and earn a wage. Because participants are only observed for a certain period, it is not possible to tell for certain how long the benefits last for. Therefore, some estimation and assumptions need to be made. These are detailed in the next chapter. Given the significance of this measure and the big impact it has on the results, it will be varied for the sensitivity checks.

Inclusive growth and other well-being measures

In the Scottish Government's definition, inclusive growth means growth that combines increased prosperity with greater equality, creates opportunities for all, and distributes the benefits of increased prosperity fairly. The aims and design of Fair Start Scotland inherently lead to some aspects of inclusive growth at the Scotland-wide level. By increasing total economic activity of low-income groups, Fair Start Scotland by definition achieves inclusive growth.

However, considering other aspects of equality, this analysis will look at the different ways in which the programme may have benefited certain groups over others, based on location, disability status and gender. Therefore, to understand the extent to which Fair Start Scotland led to inclusive growth, measures of inclusivity can be captured by disaggregating the analysis. The results will show performance of the policy under different dimensions of inclusive growth.

One of the key differences between Fair Start Scotland and other employment programmes is the focus on wider wellbeing measures and impacts as it is built on the Scottish Government's key values for public services of dignity and respect, fairness and equality, and continuous improvement. Cost-benefit analysis allows for the inclusion of some aspects of wellbeing, namely the positive impact of having a job on participants' QALYs. However, there are other impacts to participating in Fair Start Scotland that are not captured by the cost-benefit analysis, both positive and negative. These will be discussed as part of the analysis.