Infrastructure investment plan 2021-2022 to 2025-2026 - draft: consultation analysis

5. Assessing Impact of Proposed Infrastructure

Context

5.1 The Independent Commission recommended a new assessment framework is developed, in advance of the next IIP, to inform decisions about future infrastructure investment so that it best achieves desired outcomes.

5.2 There are inherent challenges in comparing the potential benefits of different types of infrastructure. It is not easy to compare investment in a school, hospital, or new digital public service, for example, because they may all deliver positive outcomes but not necessarily using comparable evidence or over the same timeframe.

5.3 In looking to develop a new approach, the Scottish Government has proposed that this is likely to take the form of a suite or dashboard of indicators to allow for a range of factors to be taken into account in any assessment, balancing potential trade-offs. This approach would be consistent with the National Performance Framework.

Question 3a

Do you agree that a dashboard of indicators is the best approach to enable informed decisions to be taken about the long-term trade-offs and choices in our infrastructure investments? Please provide the reasons for your response.

5.4 Question 3a was framed in the Consultation Document as an open question.

5.5 Where possible, we have clustered feedback to Question 3a to identify, at a high level, the extent of support (or otherwise) for the Scottish Government proposal to use a dashboard of indicators to enable informed decisions to be taken about the long-term trade-offs and choices in infrastructure investments. There is an element of repetition of points raised at Questions 3a, 3b and 3c. We have sought to reflect/combine key themes at the most appropriate question.

5.6 There is wide-ranging support for a dashboard of indicators approach to be adopted to enable informed decisions to be taken about the long-term trade-offs and choices in infrastructure investments. A majority of consultation respondents support the proposed approach (circa 70%). In the main, it is considered to be a "sensible", "pragmatic" and "sound" approach to take. Dashboards are also considered to be "a well-established practice in many industrial sectors and have proved useful for data visualisation".

5.7 Support for the adoption of a dashboard of indicators is further reflected in the variety of comments made that highlight the following themes:

• A robust framework that reflects a range of factors would ensure a holistic assessment of infrastructure investment, impact and contribution towards outcomes. A dashboard of indicators would ensure "clear alignment" with Government policies and priorities (e.g. addressing inequalities).
• It would be essential to have a "clear", "transparent", and "consistent" approach or set of parameters against which to inform decision making. It is further noted that decisions taken/reached should be guided by the available evidence base, that the process enables informed consideration of all of the issues (e.g. trade-offs and choices to be made), and that the appraisal and decision-making process is open to audit and scrutiny.
• A formalised assessment tool to underpin and help inform balanced decisions to be made is considered key, especially when it comes to considering the trade-offs that will require to be made between natural infrastructure and built environment projects which lead to a wide and diverse range of physical, economic and social impacts.
• The dashboard of indicators should provide sufficient flexibility both in terms of application across a wide range of infrastructure types, and being agile, responsive and adaptive to changes in the external environment over the delivery period for the Draft IIP (e.g. revisit and refocus objectives, respond to opportunities, identify gaps, tackle immediate and impending difficulties). Responding to COVID-19 was commonly mentioned in this context.
• That such an approach could facilitate greater consideration of synergies with other infrastructure projects and could be a useful method of communicating the various complex interactions associated with infrastructure investment.

5.8 Further, very strong support across all respondent groups is expressed for the proposal that the Impact Assessment and Prioritisation – Indicative Dashboard is underpinned by the Scottish Government National Performance Framework, Scotland's Centre for Regional Inclusive Growth Dashboard, and the United Nation (UN) Sustainable Development Goals (SDG). An "outcomes-focussed", "common-thread", and joined-up approach is therefore welcomed.

5.9 Alignment with existing frameworks is generally considered to be a "sensible" starting point and approach to measuring "the multifaceted impacts of infrastructure investment across Scotland". Taken together, these frameworks are sufficiently broad, and could encourage closer alignment across different policy areas.

5.10 There are, however, a variety of comments that ask for more detail or greater clarity, and/or provide suggestions for how a dashboard approach could be improved. These are considered in turn below.

5.11 First, there is acknowledgement of the relatively early stage of the process, and that more development work would be required to finalise (and for more detail to be provided on) the Common Investment Hierarchy in the first instance before outcomes and suite of indicators could be finalised. It is suggested that there is a requirement for further research to develop the appraisal framework and decision-making process, dashboard metrics and measures. In this regard, form is considered to follow function. Aligned to this is a request for additional guidance on how the dashboard of indicators would be applied in practice and how the indicators would be used to appraise infrastructure projects of different types.

5.12 The importance of having a clear appraisal methodology is noted as crucial to ensure: a shared understanding of how it would "inform decisions where projects have different impacts on different parts of the dashboard and how will it inform the trade-offs in infrastructure investment decisions"; and that infrastructure investment is considered "holistically". Related points include support for:

• The "weighting" of the indicators. The most posed question is whether all evaluation criteria would have equal value under all circumstances and the issue of regional differences. It is recognised that relative and appropriate weighting would be important to: assess trade-offs; reflect regional differences; and recognise differing levels of inequality/need, etc.
• There is recognition that a key challenge will be creating a robust appraisal process and scoring system that allows different kinds of infrastructure projects (and the benefits they create) to be compared, and which does not skew investment towards urban areas (i.e. delivers an equitable share of infrastructure investment for rural Scotland to maintain its natural assets and deliver inclusive economic growth). A point raised is that there has been under investment in infrastructure in remote areas and those areas outwith the commercial hubs, and that the new approach should not exacerbate this.
• Continued collaboration between the Scottish Government and key infrastructure sector bodies to agree the Key Performance Indicators (KPIs) and metrics (e.g. a "modelling sub-group is suggested" to "bring together a range of thinking…and ensure that the resulting model is challenging but usable and measurable by all parties to deliver the common goals of the plan". A co-design approach is considered crucial to share learning, insights, experience.

5.13 Second, there are comments that note key points for consideration and/or provide suggestions on how the proposed dashboard could be further developed. The most common themes identified are as follows:

• That the dashboard must sufficiently reflect and align with the Draft IIP vision (i.e. to support and enable an inclusive net zero emissions economy), Common Investment Hierarchy and strategic priorities, and should not be seen as a "box-ticking" exercise (i.e. must be embedded across all decision-making processes). It is suggested that the dashboard would need to sit alongside an appraisal methodology that identifies how the assessment of infrastructure projects is calculated and weighted against the indicators.
• Aligned to this are points around the importance of the framework meeting the Infrastructure Commission for Scotland's recommendation for "outcome-led, cross-infrastructure prioritisation". This would help ensure infrastructure initiatives are assessed and considered in an integrated and joined-up way. Support is expressed for a whole systems approach to the planning and delivery of infrastructure (i.e. considering infrastructure as a system rather than siloed sectors).
• That the dashboard would also need to ensure alignment with wider key policies to fully reflect the need to deliver an inclusive net zero carbon economy, and as noted above, to consider infrastructure and the use of it as a holistic system. The following strategies, plans and reviews are all mentioned - NPF4, Wellbeing Economy Framework, National Transport Strategy, Strategic Transport Projects Review 2, Climate Change Plan, Cleaner Air for Scotland 2 Strategy (currently in draft format), National Walking Strategy, Regional Spatial Strategies, Inward Investment Plan, Infrastructure Capital Plan, Green Recovery Plans).
• That a logic model or theory of change approach to ensuring a clear link between inputs, activities, outputs, outcomes and impacts could be a useful way to capture short, medium and long-term outcomes and impacts, and to make the dashboard more user-friendly.
• There is strong support for the dashboard: to use quantitative and qualitative measures and metrics (e.g. a diverse suite of metrics over and above "traditional" economic growth metrics such as Gross Domestic Product (GDP) to measure environmental sustainability, wellbeing, social justice, reduced inequality); to be flexible to adapt/evolve over time (e.g. as the relevance or robustness of data sources is explored); and to be capable of accommodating and making use of the range of information that may be available (e.g. locally-available information that can be used to identify opportunities for investment in natural infrastructure).
• Accessibility is important - in terms of how the dashboard would be shared and communicated with the general public, and to aid understanding of the how infrastructure investment benefits individuals and communities (e.g. a "traffic light system" is suggested).
• That it would be helpful if the Final IIP could include some worked up examples and case studies.

5.14 Thirdly, there are a few comments that make more specific points about the dashboard, including perceived gaps in indicators and/or a request for additional indicators in the Draft IIP diagram. The comments related to, for example: connectivity, education, creating inclusive accessible places, reducing inequalities,

5.15 A handful of consultation respondents (e.g. Local Government, Natural Environment and Climate Change) make explicit reference in their response that "no" they do not agree with the proposal to use a dashboard of indicators to enable informed decisions to be taken about the long-term trade-offs and choices in infrastructure investments.

5.16 The points raised are typically similar to those outlined above, with the most common points that:

• It is unclear whether the dashboard in its current form takes cognisence of regional variations in impact and importance of infrastructure (i.e. placed-based approaches).
• There is the potential for overlap across indicators, and it is not clear how these will be measured.
• It will be important that the process does not place less value or importance on natural infrastructure projects due to issues in measuring the impacts of such projects in more traditional economic terms.
• A concern raised is that a dashboard of indicators could allow projects to "score well" that satisfy some, but not all the indicators. Here, a suggestion is that a better approach may be to specify essential criteria (i.e. the achievement of net-zero, addressing the biodiversity crisis) and desirable criteria, and this could be reflected in the framework to inform decision-making.

Question 3b

What outcomes (and/or indicators) do you think should be included in developing a common assessment framework for prioritising infrastructure investment? In your response you may wish to consider how any of the suggested factors might:

• Link to the three themes of the Infrastructure Investment Plan (enabling net zero emissions and environmental sustainability; driving inclusive economic growth; and building resilient and sustainable places).
• Help address inequality, including for protected characteristic groups, and socioeconomic disadvantage.

5.17 Question 3b was framed in the Consultation Document as an open question. Where possible, we have clustered feedback to Question 3b to identify common themes or points. In addition to points raised regarding support for, and the benefits of, an outcomes-focussed approach and framework to be adopted (and alignment to the UN SDGs and National Performance Framework), the main themes identified through consultation responses are as follows.

5.18 There is broad support for the three proposed themes of the common assessment framework (i.e. environmental sustainability, sustainable places and inclusive growth). These are considered appropriate as overarching areas of focus for the framework, and for outcomes and indicators to be selected that reflect these themes.

5.19 There is wide acknowledgement of the inherent challenges that are likely to be encountered in the design of such a framework given difficulties in comparing different types of infrastructure projects. More specifically, there is broad reference to the assessment of natural infrastructure projects.

5.20 It is noted that these types of projects often present challenges in measurement in economic terms/value or quantified in monetary terms (e.g. social and environmental benefits such as wellbeing and community cohesion), or projects that will deliver economic benefits further down the line (i.e. not a quick economic return). For balanced decisions to be made, it is considered important that the multiple benefits of natural infrastructure proposals are understood and made more explicit within the framework.

5.21 It is further acknowledged that measures of success for an "inclusive net zero carbon economy" from infrastructure investment would need to be established, and for the new evidence base to go beyond traditional economic measures.

5.22 Here, it is noted that "a series of metrics or measures that can be used to assess the infrastructure impact on the delivery of net zero and inclusive economic growth outcomes is still in its relative infancy", and that collective understanding in this area is "under-developed" and has not yet "matured".

5.23 This relates to wider points regarding the importance placed by consultation respondents on the need for the indicator framework to include a broad mix of indicators that can be measured in a quantitative or in a qualitative sense.

"The proposed fifteen topics seem to cover the whole range of factors that need to be considered. Some can be quantified, including greenhouse gas emissions and renewable energy production associated with a project, while others, such as flood risk management, are difficult to quantify due to chance events. Difficulty of quantification is, however, certainly not a reason to exclude such indicators. A qualitative description at least should ensure that the factor is considered".

Chartered Institute of Ecology and Environmental Management

5.24 There is also broad acknowledgement that "data quality, relevance and availability is critical" in terms of supporting a consistent approach to comparing investment propositions and to inform decision-making. This is considered key to the success and reliability of this approach. Related points are around the importance of:

• Ensuring indicators are robust, SMART (i.e. specific, measurable, attainable, relevant and time-bound), manageable and meaningful.
• Establishing a clear baseline position to enable progress to be routinely and regularly updated and tracked.
• Outcomes/indicators included in the common assessment framework should reflect a spatial dimension (and the importance of the Place Principle being applied across all infrastructure investment).
• Capturing and monitoring data at a local/regional as well as at a national level (i.e. the level at which data is available, extent of disaggregation).

5.25 There are a wider set of comments which highlight that any framework for prioritising infrastructure investment would need to take cognisance of the following issues, factors or considerations:

• Any prioritisation framework should not exacerbate existing regional inequalities in terms of infrastructure investment decisions (and be capable of addressing the different types of challenges that are experienced in urban and rural Scotland).
• Various suggestions are made on additional investment assessment criteria that could be considered. This includes criteria such as: cost (e.g. to help assess the balance of expenditure on projects); whole life costs; cost/benefit analysis; evidence of need and deliverability; safety and risk of delivery.
• Equality and environmental impact assessments.
• Community wealth building and the wellbeing economy.
• An indication of what constitutes a "critical fail" (i.e. scoring of proposed investments against these indicators should include a "critical fail" scoring, which, if met, would preclude investments that would have negative impacts in terms of greenhouse gas emissions).

5.26 Finally, while the outcomes and indicators included in the dashboard are in the main welcomed and considered to be comprehensive, many consultation responses provide quite detailed and specific suggestions for indicators that could link to one or more of the three themes of the Draft IIP and/or could be used to measure the extent to which infrastructure investment decisions. These are wide-ranging. The Scottish Government will review all of the suggestions separately as it finalises its approach to prioritising investment in infrastructure.

Question 3c

Are there existing tools or methodologies you are aware of which you think the Scottish Government could draw on or adopt in developing its framework? You may wish to draw on examples from other countries in your response.

5.27 A majority of consultation respondents make specific reference to existing tools or methodologies that the Scottish Government could draw on or adopt in developing its framework. A wide variety of suggestions are put forward, including a mix of third party as well as consultation respondents' own organisational tools, datasets, or methodologies. These are considered in turn below.

5.28 Firstly, there is no universal or unanimous view provided on this, rather consultation respondents provide a long list of different existing tools or methodologies that may be of use to the Scottish Government (50+).

5.29 A large proportion appear to have a particular focus on natural infrastructure and assets (circa half). This reflects an earlier point that natural infrastructure could be better described, illustrated and understood within the Draft IIP, including the multiple benefits that it can generate.

5.30 Annex C provides further detail. Most respondents provide accompanying website links for the Scottish Government to find out more information and/or to access specific documents and resources.

5.31 Third party suggestions regarding existing tools or methodologies are grouped under headings of international, UK Scotland and regional. Each sub-group has been ordered in line with frequency of response (note: many, however, are individual reference points).

5.32 The feedback points to a number of existing outcome frameworks, including those at a Scotland level and those that are infrastructure specific that could be used and further built upon. Albeit there is recognition that the evidence base is perhaps less well developed in Scotland for natural infrastructure.

5.33 As such, there is clear support among consultation respondents for building on existing approaches in Scotland and best practice from elsewhere.

5.34 Further, there is a clear and strong willingness and openness among consultation respondents to share thinking, information and approaches, and as noted earlier, for a collaborative approach to be undertaken between the Scottish Government and key stakeholders to further develop the framework.