Digital appraisal manual for Scotland: guidance

The appraisal process

Pre-appraisal

It is important to prepare before going straight to the heart of the appraisal process. Undertaking the pre-appraisal steps of setting out the rationale for intervention and SMART (specific, measurable, attainable, relevant and time-limited) objectives will help guide the appraisal process and put it on a more solid footing.

Rationale

The first step in preparing for the appraisal process is to set out the rationale for government intervention. This may be an identified or perceived problem, or opportunity, regarding digital connectivity in Scotland. Market failures should be identified in the socio-economic dimension of the appraisal process. Background on the policy area should also be set out here for wider context – what is the strategic policy context and how does it fit with wider public policy objectives? What was the process in identifying the perceived problem or opportunity? The rationale should explain how the desired outcomes will be produced by the recommended delivery options. Identifying the rationale is a vital step in defining what is to be appraised.

Examples of rationale:

• to provide a new digital service (e.g. 5G) in a rural area with a sparse population where there is resultantly no commercial interest
• to increase the quantity or improve the quality of a service (e.g. 4G infill, upgrade of broadband provision) to meet Scotland’s evolving technology needs)
• to comply with regulatory changes

This initial stage should be appropriately informed by research and consultation with experts and stakeholders. A stakeholder identification exercise would be helpful that is proportionate to the complexity of the intervention. Consulting specialists and stakeholders throughout the appraisal process ensures good governance.

It is important that external, relevant issues are considered when identifying problems and opportunities to provide wider context. The process of change that will result from the proposed intervention may be influenced by issues summarised by PESTLE:

• political
• economic
• sociological
• technological
• legal
• environmental

At this stage it is good practice to set out all related interventions/strategies/policy programmes whether past, present or future. Are they linked in any way? Can we refer to any past evaluations? 

A clear quantitative understanding of “business as usual” (BAU) is also essential to understanding the current situation, and for identifying and planning the changes that may be required. It is recommended to identify and undertake a thorough analysis of the available data as an evidence base to justify the proposed intervention. The evidence base can consist of quantitative and/or qualitative data and intelligence to justify the rationale. The STAG technical database is regularly updated with ‘business as usual’ updates and this is proposed for the DAMS technical database that OCEA is developing to support this manual.

Objective setting

Strategic objectives should directly guide the appraisal (objective-led) by defining the scope and priorities, as well as measuring the outcomes of the proposed intervention. The objectives must be expressed in terms of outcomes and not service outputs where possible.

The principle of being objective-led, rather than solution-led allows the appraisal of options against objectives, recognised criteria and to establish policy directives, and provides a robust evidence base for decision makers. It should be made clear how the objectives of the intervention align with our core values/purpose and the updated digital strategy for Scotland. It is good practice to link specific strategic policy objectives to our high level mission and visions.

Setting clear objectives is essential.

SMART objectives

It is imperative that objectives are set with SMART principles in mind. A SMART objective will be:

• specific - it will say in precise terms what is sought
• measurable - means will exist to establish to stakeholders' satisfaction whether or not the objective has been achieved
• attainable - there is general agreement that the objective set can be reached
• relevant - the objective is a sensible indicator or proxy for the change which is sought
• timed - the objective will be associated with an agreed future point by which it will have been met

SMART objectives will drive the rest of the appraisal process across all dimensions set out. They will also inform the key performance indicators (KPIs) in the monitoring stage.

SMART objectives can be challenging to set – they demand insight, careful consideration and impose greater accountability. There is, however, an importance attached to making the necessary effort in arriving at SMART objectives as:

• SMART objectives provide an essential focus on the outcomes sought for the study, and, if properly set, will facilitate the satisfactory resolution of any conflicting priorities
• SMART objectives provide an opportunity for recognition of achievement, and as discussed further in the post appraisal section of this guidance, indicators must be developed based on the objectives set
• the articulation of SMART objectives may foster shared enthusiasm for the achievement of such objectives

At pre-appraisal, SMART objectives may be articulated in general terms, indicating the desired direction of change. It is recommended that this is sufficient for the purposes of qualitative initial appraisal.

It is important that SMART objectives are finalised in advance of post appraisal, with the intention of developing meaningful indicators for detailed quantitative appraisal and subsequent monitoring and evaluation purposes.

DAMS criteria

The DAMS criteria provide a framework to ensure all impacts are considered, whether positive or negative, and users should not begin the process of formulating digital objectives by considering only national objectives. These criteria focus the attention on issues specific to the digital context that should be taken into account when setting objectives. The related impacts are likely to be qualitative, but can be set out quantitatively where available. When identified, the extent of impact should be acknowledged and, if applicable, mitigation measures set out. The examples given under each criterion are not exhaustive, and it is the practitioner carrying out the appraisal or evaluation’s responsibility to ensure all impacts of a project are considered, both positive and negative

Environment and climate change

The factors considered under this criterion could vary significantly depending on the nature of the project. Where infrastructure is concerned, the impacts on geographic area (including biodiversity, historic environment and landscape) should be considered, however this may not be relevant for other projects (e.g. supporting businesses to develop digital skills). The following sub-criteria when creating the proposal’s SMART objectives.

• negative impact(s) on the environment that need to be abated, such as increased CO2 emissions or biodiversity damage
• positive impact(s) on the environment, such as increased digital technology use that in turn reduces physical travel, paper waste etc.

Resilience

The resilience criterion concerns whether the intervention is expected to provide the capacity to support a group or institution in the face of change or adversity. Resilience is a common feature of digital projects and has been especially important in Scotland’s response to the COVID-19 pandemic (e.g. businesses selling online as physical stores forced to close, increase in online communications).

• improve resilience of businesses with poor broadband/mobile connectivity
• allow for continuity of activity when employees work from home
• more consistent contact with emergency services, public services etc.
• support connectivity-enabled innovation

Economy and wellbeing

It’s essential that economy and wellbeing are considered in any intervention. This criterion concerns a wide range of potential impacts, for example:

•  concern to consider as it helps understand the net value to society of the intervention. Socio-economic factors should be of equal importance when considering the fiscal economic impacts on the objectives of the digital intervention. These impacts, particularly wider economic benefits, should be considered if they are proportionate to the scale of the proposed intervention
• economic efficiency – expected monetary and monetised gains to users and government (e.g. from time savings)
• impacts on the Scottish economy (macro-economic) such as output, employment, productivity or opening up future opportunities
• socio-economic factors such as wellbeing, health, education, equity and community
• inclusive growth and wellbeing economy

Integration and future proofing

Digital connectivity and skills affects every facet of modern life, and as such, it is essential to consider the extent to which an investment integrates the integration criterion, covering three different sub-criteria:

• policy integration – both other digital policies and wider government initiatives
• integration with other digital infrastructure – for example, whether there is reliance on pre-existing infrastructure or the project is expected to be supplementary
• whether the intervention require users to have a certain level of digital skills to meet objectives, or whether there are financial barriers in place in order for people to benefit from the intervention
• future proofing – importance of the project in the context of an increasingly digital society

Accessibility and inclusion

Some interventions aim to achieve benefits for society which may not be robustly monetisable, but nonetheless align with our objectives or core values:

• any expected changes to community accessibility e.g. online public services
• comparative accessibility - distributional impacts by people group or location
• impact on disparities between different groups, e.g. urban vs rural, areas of deprivation
• reference to existing commitments and strategies regarding accessibility in isolated or rural areas, or marginalised groups

Appraisal

Appraisal is the process of assessing the costs, benefits and risks of alternative ways to meet government objectives. It is a key tool in aiding decision makers to understand the potential effects, trade-offs and overall impact of options by providing an objective evidence base for decision making. The appraisal undertaken should be proportionate to the costs and expected impact to both the public sector and the public from a proposed intervention.

The appraisal of digital investments will take into account overall social welfare efficiency and not just economic market efficiency. The appraisal of public value is based on welfare economics and concerns all significant costs and benefits that affect the welfare and wellbeing of the entire Scottish population. As a result of this, factors may be unquantifiable and/or less robustly monetisable, such as wellbeing or inclusion.

Option generation

Following the pre-appraisal steps where the rationale and objectives are confirmed, the first stage of the appraisal process is to develop options that can achieve the desired outcomes and meet the set objectives. This step prevents the business case being retrofitted to fit a predetermined proposal, but rather to consider a range of approaches to eliminate bias.

The number of options required for the longlist and shortlist depends on the proposal. It may be the case that there is a limited number of viable options to be considered in the appraisal, which should be acknowledged.

Longlist

The longlist of options provides a process for narrowing down potential choices to confirm a viable set of options for shortlist analysis. The reason for generating multiple potential interventions is to avoid decision makers thinking too narrowly or being led by a preconceived solution. The ‘no intervention’ or business as usual position should also be considered as an option and will act as a counterfactual benchmark on which to compare the proposals.

The basis for selection or rejection should be based on such factors as feasibility, affordability, public acceptability, policy directives, climate change targets etc. Alternative options should be reviewed while taking into account the SMART objectives, value for money (VfM) and DAMS criteria. Digital interventions with expected long term costs and benefits may be influenced by external structural changes (PESTLE) which need to be considered. The longlist appraisal must also consider place-based, equalities and distributional effects.

The HM Treasury Green Book states that for a proposal to have a chance of being successful, it should meet the following five critical success factors (CSFs):

• strategic fit and business needs
• potential value for money
• supplier capacity and capability
• potential affordability
• potential achievability

Shortlist

Shortlist appraisal is where the expected costs and benefits of the proposed interventions are estimated, including the cost of risks and risk management. This stage is where the trade-off between the options is considered. It is important that shortlisted options are viable and are backed by evidence in their ability to meet the SMART objectives set out.

Where there is a clear difference in the social costs and benefits between alternative shortlisted options, social cost benefit analysis (CBA) is used. Where there is no measurable social difference between options, then social cost-effectiveness analysis (CEA) is appropriate.

1. Social cost benefit analysis (CBA) or

2. Social cost-effectiveness analysis (CEA)

Identification of the favoured, final option is based on the analysis of all potential options in regards to balance of costs, benefits, risks and unmonetisable factors, thus optimising value for money. Selection of the preferred option should consider a workshop that involves relevant experts and stakeholders.