LHEES: guidance on strategy level socio-economic assessments

Strategy level Multi-Criteria Analysis methodology

What is Multi-Criteria Analysis?

A key component of a strategy is to set out an achievable route map which will deliver the final vision, simultaneously incorporating multiple objectives. In the context of a Local Heat and Energy Efficiency Strategy (LHEES), this will include a portfolio of projects which jointly deliver against targets set for a 15 or 20 year timeline.

Strategic socio-economic assessments, including those used for LHEES, therefore need to balance a set of projects and initiatives, some of which are expected to be delivered in the shorter term while others will be delivered over the longer term. The projects to be delivered over the longer term will have more uncertainty, both in terms of their likelihood of delivery and in terms of the probable benefits achieved and costs involved.

Multi-Criteria Analysis (MCA) is a socio-economic decision-making tool which establishes preferences based on an explicit set of pre-defined objectives. All impacts are assessed, whether they can be monetised or not, through the development of a scoring system. This translates into a final value which can be used to rank and identify the most attractive intervention(s).

MCA allows a more formal approach to be taken to ensure that all impacts are recognised and taken into account by decision-makers. MCA can therefore assist with the identification of a single preferred option, or can be applied in order to create a short-list of potential options that can be appraised using additional techniques.

The added benefit associated with implementing MCA includes that interventions of a larger scale that serve to meet multiple objectives can be appropriately appraised using MCA methodologies. With that in mind, MCA has been identified as the most appropriate methodology for strategy-level socio-economic assessments, such as for LHEES.

Establishing a baseline

Identifying a set of projects and initiatives to be delivered over a 15-20 year horizon is challenging enough in its own right, before considering the necessity of identifying an alternate set of projects which can form the baseline. Without an identified baseline, however, it will be very difficult to estimate the incremental impact of the prioritised projects and initiatives as compared with what would otherwise happen.

A baseline of “do nothing” has been identified by the Scottish Government as the most realistic to use in the context of LHEES. This means that the proposed interventions will be assessed against a scenario whereby by the local authority takes no action other than what was already planned. As will be seen later, this makes the scoring of each indicator much simpler. Interventions and scenarios will still be assessed on whether they comply with current and planned legislation, regulations and policy, with this being one of the criteria against which they will be assessed against.

Conducting a Multi-Criteria Analysis

In contrast to a discrete project, an area-wide strategy must balance the contribution from multiple projects towards a set of pre-defined objectives. Projects within LHEES should not be considered in isolation, as in any given project there may be over delivery against certain objectives and under delivery against other objectives, while there is also potential for these anomalies to even out when considered in aggregate across a portfolio of multiple projects.

Additionally, a long-term strategic approach needs to be open to the inclusion of additional projects or initiatives as the strategy develops and progresses through time, including in response to changing circumstances. Under the expected design of LHEES, this fits well with the likelihood that regular reviews of the original LHEES will be required, for example every five years.

The process of conducting an MCA can be broken down into the following five key steps (Figure 2).

Figure 2: Five key steps towards conducting Multi-Criteria Analysis, adapted from Mateo (2012) ^[2]

Defining the problem and generating criteria

The first step towards conducting an MCA should involve an assessment of the overall problem or objective(s) that is/are being addressed through the socio-economic assessment process. In this instance, the overarching objectives relate to the proposed development of LHEES across Scotland, focused on removing poor energy efficiency as a driver of fuel poverty and decarbonising heat across local authority areas.

The identification of the relevant criteria that will collectively work towards the achievement of the goals of LHEES therefore forms a key part of this methodology. The criteria which have been selected against which LHEES will be assessed in this MCA are shown in Table 1 below.

Once suitable criteria have been developed and confirmed, the next step is for weightings to be applied to enable the analysis to account for the relative importance of each of the criteria. The weightings which have been agreed with Scottish Government are also shown in Table 1.

Table 1: The chosen criteria and weightings that will be used as part of this MCA methodology

Criteria	Weighting
Carbon emissions	.3
Fuel poverty	.3
Financial	.08
Local economic	.08
Local environmental	.08
Social	.08
Resilience	.08
Total	1

The criteria and weightings outlined in Table 1 have been agreed with Scottish Government, and reflect the overarching objectives of LHEES. The weightings outlined in this table, i.e. criteria level weightings, should remain fixed throughout the MCA process. Local authorities may, however, amend the indicator-level weightings (Figure 4), if local circumstances permit alterations. Any changes should be justified and discussed with Scottish Government prior to this methodology being implemented.

For simplicity, it is recommended that the sum of weightings are consistently normalised to a value of 1. This normalisation allows for easy comparison of weightings across the criteria (as they can be read as percentages summing to 100% and ensures that a consistent base is used when adding or removing criteria over time). Without normalising to 1, the addition or removal of criteria will distort the total value and make it difficult in future to compare directly across previous versions of your LHEES MCA.

As part of the LHEES review process, criteria and associated indicators may be changed after 5 years to reflect current circumstances across each local authority. It is also important to note that as this methodology is rolled out across local authorities, feedback will be taken on board and relevant changes made, where appropriate.

Impacts and Indicators

For each criterion in Table 1 there are likely to be numerous potential impacts to assess. In order to assess the scale of an impact each impact needs to be assigned an indicator. Interventions/projects will then be assessed against all indicators individually, and a score of 1 – 5 applied according to how it performs across each of these. For example, for the criterion ‘Local Environmental’, one key impact to consider is air quality:

Criterion = Environmental

Impact = Air quality

Indicator = Change in volume of pollutants

Score = Transfers to a score between 1-5

Table 3, later in this section, provides a comprehensive list of the impacts and indicators that should / could be considered, sitting under the seven criteria listed in Table 1. At the start of the process, a stakeholder group from your local authority area should be identified and convened to feed into this process, from defining impacts, to identifying the indicators and applying the scoring. It should not be done in isolation by a single individual.

Scoring Criteria

The development of a scoring system is vital to ensure that all impacts can be appropriately translated into a final value, which in turn can then be used to rank and identify the most beneficial interventions and aggregated project portfolios which will together optimise the benefits to society for the costs invested.

In this methodology each indicator should be given a score between 1 and 5, to no more than one decimal place. The broad definitions for each score are shown in Table 2.

Table 2: The scoring system for the MCA

Score	Description
0 to <3	The proposed intervention has a detrimental impact on the indicator compared to the baseline.
3	The proposed intervention has no impact on the indicator compared to the baseline.
>3 to 5	The proposed intervention has a positive impact on the indicator compared to the baseline.

The scoring system is influenced by the choice of baseline. The importance of a baseline
is discussed in the relevant section above. It has been determined that the baseline for assessing LHEES should be a ‘do nothing’ scenario.

A ‘do nothing’ scenario will therefore represent a 3 in the scoring system. Alternative interventions/project should therefore be appraised in relation to this baseline value of 3, where values above 3 indicate a net benefit to society in comparison to the baseline ‘do nothing’ scenario. Attributing a score below 3 where there is considered to be a net cost to society, as stated in Table 3. The greater the score above 3, the greater the expected benefits; conversely, the lower the score below 3, the greater the expected negative impact/costs to society.

Developing a scoring system

A 1-5 scoring system must be adopted and adhered throughout. It is, however, at the discretion of the local authority to define their own scoring system within this. This should be established and documented at the start of the process, and involve the input and agreement of the relevant stakeholders.

Feedback from the pilot LHEES programme suggests that this approach may present some political challenges and is likely to be dependent on the composition of the stakeholder group which is convened and asked to feed in to the ranking exercises.

Generic advice is provided below on how to develop a scoring system, with specific considerations for each indicator provided in Table 3.

Where indicators can be quantified

Where the indicators of an impact can be quantified, it is possible to develop a scoring scheme for that impact where each score is assigned in direct proportion to a specific value (or range of values) for that indicator.

To use carbon emissions as an example to illustrate this, the technical assessments of the interventions/projects being included in the MCA are already likely to have been assessed for the potential lifetime carbon savings they could deliver, with this calculation included in an earlier project identification study, opportunities assessment, or feasibility study. This quantification allows the range of carbon savings which have been quantified across all the interventions to be aligned to the 1-5 scoring scale, assigning a score (to one decimal point) proportionate to each carbon saving value (or range of values), as shown in the illustrative example in Figure 3 (noting, as always, that a score below 3 represents an impact which is worse than the ‘do nothing’ baseline, i.e. the intervention causes an increase in carbon emissions).

Figure 3: Applying scoring proportioned to carbon emissions saving values

Score	Lifetime carbon emissions savings (tCO2e)
…	…
2.7	-200 to -299
2.8	-100 to -199
2.9	-1 to -99
3	0
3.1	1 to 99
3.2	100 to 199
3.3	200 to 299
…	…

It is at your discretion as to how you apportion the scoring; for example, should you consider that the carbon savings offered by all your potential intervention are relatively modest you might decide that none of them warrant the application of a maximum score of a 5. This would be for you to decide, with the endorsement of the stakeholders you convene, provided the approach taken can be justified, is documented as a matter of record and is applied consistently in the assessment of all the interventions.

Where indicators cannot be quantified

Where the indicator is not something which can be readily quantified (and/or data are not available or reliable to enable quantification) discussion and consideration is needed to decide for each intervention whether there will be a net benefit or a net cost for that indicator, based as always on the baseline score of 3 for a ‘do nothing’ scenario.

Given that in these cases no quantification data is available for developing an accurate scoring scale for the indicators, it is important to have all key stakeholders inputting and discussing views on the relative impact of each intervention against the indicator in question. Stakeholder discussions should decide, of course, not only whether the score for each intervention should be above or below the baseline level of three, they also need to reach as objective a conclusion as possible for how far above or below the baseline score each intervention should be rated, i.e. to discuss and agree the justification for scoring project X as a 4 on a given indicator, while project Y scores at 3.5 and project Z gets 4.5 for that same indicator. It is particularly important for the justification for each score to be debated, agreed and recorded, for the avoidance of future doubt or dispute.

With this approach having no basis in objective and quantified data, it is inevitably broader in its interpretation and application; for this reason it is sensible when scoring in this way only to score at intervals of 0.5 (i.e. stick to using 1, 1.5, 2, 2.5, 3, 3.5, etc.). Attempting to go any more granular than 0.5 intervals would be inappropriate; it would suggest an unrealistic level of accuracy to make such finer grain distinctions and would be liable to tie up time unnecessarily in debating scoring to that level.

Rank Reversal

Another factor which can influence the validity of MCA results is often referred to as ‘Rank Reversal’. This can be defined as when the inclusion of additional criterion and indicators can reverse the preference ranking for those alternatives being appraised^[3].

This is considered a large inconsistency of an MCA approach, and practitioners therefore need to be aware that the inclusion of additional criteria or indicators may alter associated weightings and results from the MCA analysis. This effect would clearly be seen when adding an additional indicator to the criterion of carbon emissions (Table 3 below), which would subsequently alter the indicator-level weightings (Figure 4) from 30% to 15%.

Wherever possible, practitioners should therefore avoid adding criteria and indicators to MCA frameworks once initial appraisals have been conducted and rankings obtained. This will serve to reduce associated uncertainties of MCA approaches, thus improving the robustness and applicability of model results.

Table 3: Criteria, Impacts, Indicators and Scoring considerations

Criteria	Impact	Indicator	Scoring considerations
Carbon emissions	Carbon emissions	GHG Emissions (tonnes CO2e)	The annualised carbon emissions of an intervention may be identified during a technical feasibility study to allow you to develop a robust scoring system. Ensure that all intervention lifetimes for carbon savings are consistent (40 years recommended) and that the same decarbonisation scenario has been used across interventions. BEIS publish decarbonisation scenarios that should be used for consistency.
Fuel poverty	Fuel poverty	Risk of fuel poverty within an area	An appraisal of the overall fuel poverty risk of the area. How many fuel poor households are present? What are the drivers of fuel poverty in the area? Consider whether intervention is likely to have an impact on the drivers and levels of fuel poverty.
		Change in unit cost of heat (£ per kw)	The unit cost of heat of the proposed intervention may have been identified during technical feasibility, allowing the % change from the BAU scenario to be identified. This will allow a scoring scale to be developed according to the range of percentage changes. If not, it may be possible to determine which fuel sources
		Change in average EPC rating	What affect will an intervention have on average EPC ratings across the local area? This metric will be strongly correlated with fuel bill savings.
Financial	Project costs	Capital costs	Data should be available from the financial feasibility to allow you to develop a robust scoring scale. Note: if there are capital costs these would be scored as a net cost against the ‘do nothing’ baseline (i.e. score <3). It is important to assess capital costs regardless of who is providing the finance or funding for the intervention.
		Operation & Maintenance costs	Be sure to consider the lifetime costs of an intervention across 40 years. Ensure that across all interventions there is consistency with regards to the application of inflation.
		Fuel costs (£)	Total fuel cost savings should be quantified in the techno-economic feasibility study for an intervention. BEIS price projections to be applied- Annex M. If local fuel prices are available these can be used with the growth trend of the BEIS price projections applied to the market prices without VAT & CCL. Baseline data to be calculated. Where data isn’t available, consideration should be given to whether an intervention is likely to increase or decrease fuel bills (consideration given to fuel type and volume against the BAU scenario).
Local economic impacts	Jobs supported	Number of jobs supported	Consider whether there will be direct/indirect local jobs supported as a result of the intervention. (N.B. It is important to consider the duration of any jobs created).
	Skills	Skills supported and developed	Consider whether an intervention will result in localised workforces having greater access to skills development. It is important to distinguish between local and regional skills.
	Regeneration	Regeneration level	Consider what levels of local regeneration can be expected as a result of an intervention being implemented
Local environmental impacts	Air quality	Volume of pollutants (tonnes of nitrous oxide, particulate matter, sulphur oxide and ammonia)	It should be possible to quantify the annual volume of pollutants of each intervention through the technical feasibility. A robust scoring scale can then be developed. If this is not available, a discussion around the different fuel types across interventions and their relative impact on air quality assessed.
	Noise	Change in noise level (decibel)	Assessments against this indicator should consider changes in noise both during the construction and operation phases of interventions. Ideally, this will be modelled and quantified across the lifetime of the project. If this is not possible, consideration needs to be given to both phases when assigning a score.
	Built environment & local heritage	Quality of built environment	Consider whether an intervention will influence the quality of the surrounding built environment, and what impact this will have on local heritage.
	Biodiversity & ecosystem services	Proportion of green field sites / woodland/ wild habitat	This indicator takes into account the affect a given intervention will have on increasing/decreasing the proportion of surrounding green field sites and natural habitats.
Social	Health and wellbeing	Improved thermal comfort	Consider the influence an intervention will have on altering levels of thermal comfort. Improved insulation may reduce heat losses at the building level. The same amount of fuel may be consumed but returning a great level of heat as less heat is lost. Alternatively, the unit cost of heat could reduce allowing the same amount of energy spend for a greater volume of heat. To avoid double counting, if thermal comfort is driven by a reduction in the unit cost of heat, do not score this indicator as it has already been considered but still provide a narrative.
		Time available to work	Consider whether an intervention will alter the time available to work for local residents, taking into consideration the key construction phases of an intervention. This may be possible to model over the lifetime of the project. In most cases it is more likely to be an impact only during the construction/installation phase (if at all).
	Community	Proportion of recreational community space	Consider whether available community space will be altered as a result of an intervention being implemented.
	Consumer acceptance	Acceptance of proposed intervention	Are residents likely to oppose an intervention and could this prevent implementation? Or will it readily receive strong endorsement and acceptance among local residents? Consider why and whether this should affect its ranking.
Resilience	Local energy security	Reduction in imported fuel - national vs international sourcing of fuel	What affect will an intervention have on reducing the need for imported fuel types across an area, alongside the impact on local network capacities? It may be possible to quantify this over the lifetime of the interventions. Consideration should be given to national versus international fuel sources, and the volatility of different commodities.
	Alignment with current regulation	Meets current legislation / regulatory requirements	Does the intervention meet current legislation and regulatory requirements? Assessing the level to which each intervention exceeds legislation (including potential ‘future proofing’) will influence the scoring approach.
	Energy demand	Reduction in demand	From the technical viability assessment, it should be possible to quantify the reduction in demand against the baseline scenario, and therefore develop a robust scoring approach. Alternatively, consider whether relatively interventions would lead to a reduction in fuel consumption.

Time Horizon

In order for interventions to be assessed fairly the time horizon over which costs and benefits are assessed needs to be standardised. For example, when assessing the lifetime carbon saving potential of a range of insulation projects, the lifetime used needs to be the same across all. It is recommended that a 40-year time horizon is used.

Base year

For monetised impacts assessed, adopting an appropriate base year for discounting, from which the costs and benefits of alternative interventions will be compared, is important. The chosen base year is 2018.

Discounting

An important point to note at this stage is that all future costs associated with an intervention will need to be appropriately adjusted, to reflect the ‘social time preference’. This reflects the fact that a unit of currency will be worth more at present in comparison to any point in the future.

To deal with the differential timing of costs, a social discount rate is often applied. This is applied to calculate what is effectively a present value for future costs (i.e. allows for costs and benefits to be compared across varying time periods).

To ensure Green Book compliance, a social discount rate of 3.5% in real terms should be adopted for the first 30 years, reduced to 3.0% after 30 years^[4]. Table 4 details how a present value of £1,000 declines over a 10-year period with a discount rate of 3.5%.

Practitioners should note that taking into account the social time preference should be considered a separate action to accounting for inflation and should also be treated in isolation to financial discounting, which reflects the opportunity cost of capital.

Table 4: The decline of present values over a 10-year period with a discount rate of 3.5% (HM Treasury’s Green Book, 2018)

Year	0	1	2	3	4	5	6	7	8	9	10
Value	£1,000	£966	£934	£902	£871	£842	£814	£786	£759	£734	£709

Establish an appropriate evaluation matrix

An appropriate evaluation matrix needs to be set up in which all the MCA scoring is calculated and recorded. It is recommended at this stage that practitioners work towards producing a simple, effective spreadsheet model which clearly shows the criteria being assessed, their indicators, and the subsequent weightings, scores and totals.

As shown in the illustrative example in Figure 4, the score for each indicator is multiplied by its weighting. The weighted score for each indicator is then summed to give a total for each criterion and these summed up to give a total of each potential intervention. The total value of a grouping of interventions can also be calculated to provide the overall score for a strategy portfolio (and compared with other possible groupings of interventions).

Setting up a model in this format will contribute towards improving understanding and visibility of how each individual intervention, and the subsequent portfolio, has performed in the socio-economic assessment.

Figure 4: Illustrative example of an Evaluation Matrix for the MCA analysis

Ranking the alternatives

The final step is to rank alternatives according to their total score in order to identify the most appropriate interventions and the portfolio of projects to take forward, and also to allow comparison between different portfolio options.

Over time, as projects are updated, replaced or added to the existing project portfolio, the scores attributed to any specified indicator may change, altering the total value originally calculated through the MCA. This could, for example, be the case if technological advancements mean an improvement in the net carbon emissions but no improvements to air quality or impacts on fuel poverty alleviation.

Outputs of Multi-Criteria Analysis

Outputs from MCA assessments will help to communicate the portfolio of interventions that will form part of LHEES strategic delivery programmes across Local Authorities. A useful tool to develop when communicating MCA results is an Appraisal Summary Table (AST), which can be utilised to clearly highlight the scoring outputs from MCA across selected interventions (Table 5).

Table 5: Example of an Appraisal Summary Table (AST)

Criterion	Score	Weighting	Building Fabric Option A relative to baseline	Option A assessment of indicators	Building Fabric Option B relative to baseline	Option B assessment of indicators
Carbon emissions	1-5	0.3	(Score X weighting) - Baseline	Justification	(Score X weighting) - Baseline	Justification
Fuel Poverty	1-5	0.3
Financial	1-5	0.08
Local Economic impacts	1-5	0.08
Local environmental impacts	1-5	0.08
Social	1-5	0.08
Resilience	1-5	0.08
Total score	/	/

Detail regarding the methodologies adopted throughout should also be communicated. Table 1, Table 3, Figure 4 and Table 5 clearly show the criteria, indicators and weightings applied during the MCA process. These figures should therefore be used to communicate and justify any decisions that are made regarding the adoption/rejection of specific strategic-level interventions on the basis of socio-economic assessment.

Evaluation of MCA Uncertainty

The uncertainty associated with MCA approaches adopted throughout will need to be accounted for and recognised, forming a key part of finalised methodologies and communications.

Weighting and criterion uncertainties

The first level of uncertainty that needs to be considered is the subjectivity associated with assigning weights to each criterion. To account for such subjectivity, a sensitivity analysis can be completed that seeks to validate the methodologies adopted throughout assessments. Changing model assumptions (i.e. the weighting associated with individual criterion) can determine how the selection of alternative interventions may vary when the importance/weighting of criteria is changed during the selection process.

The scoring of interventions using MCA methodologies may also be a largely subjective approach, in part due to individual and project-level uncertainty. The scoring process itself is subjective in nature, and if this is accompanied by inadequate levels of information regarding an intervention, large uncertainties can be created throughout the decision making process.

An approach towards overcoming high levels of subjectivity is to make MCA as transparent as possible. Practitioners should note that MCA should be conducted by an agreed panel, rather than an individual. This will help to reduce the self-assessment bias associated with MCA methodologies, and will also allow for more robust justifications to be provided when presenting/discussing MCA results.

The Carbon Trust is an independent company with a mission to accelerate the move to a sustainable, low-carbon economy. The Carbon Trust:

• advises businesses, governments and the public sector on opportunities in a sustainable, low-carbon world;
• measures and certifies the environmental footprint of organisations, products and services;
• helps develop and deploy low-carbon technologies and solutions, from energy efficiency to renewable power.

www.carbontrust.com

+44 (0) 20 7170 7000

Lead Authors:
Chantalle Thomson
Chantalle.Thomson@carbontrust.com

Rob Franklin
Rob.Franklin@carbontrust.com

Contributing Author:
Dr Tanja Groth

Reviewer:
Paul Wedgwood
Paul.Wedgwood@carbontrust.com

Energy & Climate Change Directorate,
The Scottish Government