Post-school funding body landscape simplification: analytical approach summary

3. Short-list appraisal

3.1 Approach

This Project is ultimately about structural change. The benefits relate to creating conditions to improve the system and therefore are not easily quantifiable or monetisable. This means that a cost-benefit analysis (CBA) is not suitable for appraisal.

In such circumstances, the alternative approach recommended by the Green Book is to use cost-effectiveness analysis (CEA). This approach requires an effectiveness measure which was derived from a Multi-Criteria Decision Analysis (MCDA). A cost analysis was conducted alongside the MCDA and together these formed the CEA.

3.1.1 Multi-Criteria Decision Analysis

MCDA is a Green Book approved method that assesses the performance against set criteria, taking into account the different viewpoints and perspectives of key stakeholders. It is widely used across the public sector in appraisals that do not rely predominantly on monetary valuations and facilitates broader engagement in the process. MCDA is a well-established technique that helps decision-makers make choices between alternative options when these are required to achieve multiple specific objectives. It is particularly effective when there is a mix of qualitative and quantitative criteria that are not directly comparable against each other and when the perspectives of multiple stakeholders may need to be considered, as is the case here.

There are two key steps to conducting MCDA

a. scoring of options against criteria, and

b. weighting of the criteria.

Scoring of the options against criteria

The scoring of the options took place in a workshop, where key Scottish Government and external stakeholders ranked each option against 10 pre-determined criteria spread across 5 criteria sets. These criteria were developed from the business needs set out in the Strategic Case of the OBC, and with input from policy leads across LLS, economists and other analysts in Scottish Government with experience of applying MCDA approaches.

Participants consisted of representatives from the three funding bodies in scope for the project, as well as other sector representatives and Scottish Government policy officials. Participants were invited based on their knowledge and expertise in a range of different elements of the funding body landscape, both from a delivery and customer perspective, and were invited to share their views and experience, rather than their organisations’ position, in discussion of the options and criteria.

In the workshop, participants were asked to consider each criterion in turn and to rank the three options based on their ability to meet it, relative to the other 2 options (including Business as Usual). If an option ranked 1^st, then it was considered the best option for meeting that criterion. If an option ranked 3^rd, then it was considered the worst option for meeting that criterion. Options could also be ranked equally if the participant was not able to make a distinction between two or more options for a specific criterion. Rankings were submitted on Microsoft Forms, alongside reasonings for the submitted rankings.

The rankings were then summed up at individual participant level and ranked again at an individual criterion and criteria set levels, with lowest total score within each criterion and criteria set ranking 1^st and highest total score 3^rd, to arrive at the overall result set out in Section 3.4.3 of the OBC.

Weighting of the criteria

For the purposes of the workshop, each criteria set and each criterion within them was initially weighted equally to allow these to be considered equally important.

Sensitivity analysis was then conducted to see if different criteria weightings changed the results. This was done using Monte Carlo analysis, which involves a large number of combinations of weightings being simulated to determine the probability of the main result changing under different scenarios.

An additional layer to this analysis was added by removing a random set of individual responses from each simulation to account for any bias from groupings of individuals.

Other sensitivity analysis

Other sensitivity tests undertaken included:

• Simple removal of specific organisational groupings from the results to see if results change.
• Adopting an alternative method in which the rankings are summed to arrive at the final results and seeing how the results change. Further detail on what this alternative method looks like can be found in Section 3.4.5 of the OBC.

3.1.2 Economic appraisal of costs

The economic appraisal of costs used figures derived from the Financial Case of the OBC. These costs were derived using financial returns and accounts of each of the respective public bodies in scope, as well as by considering costs from a number of comparable reform programmes from Scotland and across the UK, to arrive at reasonable estimates related to transition. A fuller explanation of the assumptions taken can be found in the Financial Case of the OBC.

The economic appraisal followed the Green Book methodology and principles, with the annual nominal costs for the 9-year appraisal period derived from the Financial Case converted into “real” base year prices – in this case 2025/26, the first year of intervention – discounted and summed to give a Present Value of Costs (PVC) over a chosen appraisal period.

• Real base year prices: Converting costs into “real” base year prices means that the effects of general inflation were removed.
• Appraisal period: Based on the HMT Green Book guidance, costs and benefits should be calculated over the lifetime of the intervention or asset and that, for many interventions, a time horizon of 10 years is suitable. SAAS, SFC and SDS provided costs for a 10-year period from 2024-25. Given that the first year of the intervention is 2025-26, a 9-year appraisal period was used instead.
• Discounting: a technique used to compare costs (and benefits where monetised figures are available) occurring over different periods of time on a consistent basis. This is based on the concept of time preference – the idea that people generally prefer to receive goods and services now rather than later. This also applies to spending, with people preferring to spend later rather than now, and therefore the spend now has a higher value than spend in the future. To use this technique, the Green Book specifies a social discount rate, which is an annual percentage reduction which is then applied to values in each year going forward that progressively reduces the current value of future values.

The results of this economic appraisal were presented in total PVC for each option as well as by cost category (staff, non-staff and transitional costs). However, as a considerable proportion of costs for Options 2 and 3 would, similarly, be incurred in Option 1, the additional costs expected to be incurred by Options 2 and 3 relative to Option 1 were also shown.

Simple sensitivity analysis was also conducted to see how the additional costs relative to Option 1 change if

• A 10% increase to year-on-year staff running costs is applied for those staff in scope of transferring to a new organisation from 2026/27;
• A 100% increase is applied to transition costs over 2025/26 and 2026/27; or
• Both of the above scenarios occur at the same time.

3.1.3 Cost-effectiveness analysis

Cost-effectiveness analysis (CEA) is a form of economic analysis that compares the relative costs and outcomes of different options with a specific “effectiveness measure” representing the outcomes. In this OBC, the overall scores of the MCDA formed the effectiveness measure (or outcome), as they give an indication of the expected outcomes from this project. This allows us to conduct CEA and to compare the results from the MCDA and cost appraisal more comprehensively in line with HMT Green Book Guidance.

Cost-effectiveness analysis presents a ratio of costs to effectiveness – i.e., a unit cost measure. In this case, the total PVC figures, resulting from the economic appraisal of the costs, were divided by the MCDA score (i.e. sum of criteria level ranks) for each option to arrive at the effectiveness ratio, with the lowest ratio being the most “cost-effective”.

Sensitivity analysis was conducted in the form of simple “tipping point” analysis – i.e. how much do the inputs, in this case the present value of costs and the MCDA scores, need to change in order for the overall CEA result to change. This was done in two ways:

• Changing the weighting of the 5 MCDA Criteria Sets to see which weightings scenarios lead to different results. For example, assigning Criteria Set 5 a weighting of 80% and 5% for other Criteria Sets.
• Increasing the total present value of costs to see at what cost level the results change.

3.2 Quality assurance

The methodology for the short-list appraisal was designed by EAS Economists, with input from relevant experts in the Scottish Government Economist and Operational Research Groups. The methodology was quality assured by Senior Economists as well as Head of the Economist Profession.

The main analysis was conducted by EAS Economists with the support of other Economists from Scottish Government.

The model for the Monte Carlo sensitivity analysis was supported by Scottish Government Operational Researchers who worked together with EAS Economists to set appropriate parameters for this. The model went through a quality assurance process based on the HMT Aqua Book, before final outputs were shared.

The results of the short-list appraisal were reviewed by Senior Economists and quality assured and signed off by the EAS Deputy Director.