Scottish climate action hubs: independent evaluation

Methodology

Methodological approach

A theory-based approach was used for the evaluation. This approach intended to understand how and why the hubs’ activities worked (or did not work) towards achieving the overall vision of the programme. The research focussed on understanding the causal pathways between the processes and activities of the hubs and how that led to specific outcomes.

At the outset of the evaluation, an evaluation framework was developed specifically for the pathfinder hub assessment. This framework included a set of research themes, indicators, and data collection methods. The evaluation made use of a mixed methods approach, combining qualitative and quantitative data. Qualitative data was collected through an online survey, semi-structured interviews and focus group discussions, Quantitative data was derived primarily from progress reports provided by the hubs and the Scottish Government.

The research process was designed to include all stakeholders who were willing to participate in the research. Online options were made available for all engagements to accommodate stakeholders who were not able to attend in-person events. Engagements were also open for anyone to attend. This approach promoted a more comprehensive understanding of different perspectives.

Evaluation framework

It was necessary to develop a bespoke evaluation framework for the pathfinder hub evaluation as the monitoring evaluation and learning (MEL) framework for the broader hubs programme was not yet finalised. The evaluation framework outlined a process for assessing the broader impact of the hubs while taking into account the key research questions posed by the Scottish Government. To achieve this, the evaluation framework was structured as a Theory of Change (TOC) with an impact statement, outcomes, outputs, inputs and indicators. The full TOC model is provided in Annex 1: TOC Model.

Impact

For the purpose of the evaluation, ‘impact’ as a concept was defined as the long-term effects produced by activities, directly or indirectly. The impact of the pathfinder Hubs, as defined through the research process, was described as:

The hubs are contributing to and adding value in the shift towards low-carbon and resilient communities.

Outcomes

For the purpose of this assessment, ‘outcome’ as a concept was defined as the achieved or likely short to medium-term effect of the outputs of activities delivered. In order for the above impact to be assessed, three outcomes were described as follows:

• The hubs are facilitating the necessary resources for climate response.
• The hubs are stimulating low-carbon and climate-resilient interventions.
• The hubs are enabling potential future climate response to take place.

Outputs

‘Output’ as a concept was defined as a tangible product or service produced as a result of programme or project activities. For the three outcomes to be assessed, a number of outputs were identified. These outputs are shown in figure 1 below.

Outcomes

1) The hubs are facilitating the necessary resources for climate response.

Outputs

1) Facilitating knowledge sharing

2) Building community awareness

3) Building relationship amongst stakeholders

Outcomes

2) The hubs are stimulating low-carbon and climate resilient interventions.

Outputs

4) Stimulating climate action

5) Supporting climate resilience

6) Supporting regional response to climate change

Outcomes

3) The hubs are enabling potential future climate response to take place.

Outputs

7) Supporting replication and scaling up

8) Promoting local ownership of climate response

9) Supporting policy input and decision making

10) Enabling access to climate finance

Figure 1: Pathfinder hub evaluation outcomes and outputs

It should be noted that the outputs were not exclusive to specific outcomes, and in many instances, outputs were linked to multiple outcomes. By way of example, the output “Enabling access to climate finance” was linked to both Outcome 3 and Outcome 1.

Inputs

Although the evaluation focussed primarily on the link between the output, outcomes and impacts, three key inputs were also identified at the outset of the evaluation. These inputs were:

• Funding from the Scottish Government for operational costs;
• A range of staff skills and expertise; and
• Physical office space for meetings and networking.

Indicators

A set of output and impact indicators were also developed for the evaluation. However, one of the primary limitations of the evaluation was the lack of adequate baseline data (see Section 0 Scope and limitations). This lack of baseline data was particularly relevant when developing indicators for the outputs. Without a reliable starting point, it was difficult to quantitatively assess impact through an increase or decrease in an output. To accommodate this limitation, qualitative indicators were developed to track the progress of the outputs. By way of example, for the output “Facilitating knowledge sharing”, the indicator that was developed was: “Evidence of increased knowledge sharing”, rather than an increased number of knowledge sharing activities. The full list of indicators is provided in Annex 2: Evaluation indicators of this report.

Data collection

There were four principal data collection activities undertaken during the evaluation: information review, focus groups, interviews and online survey.

Information review

Key documentation was reviewed to develop a comprehensive understanding of the climate action hubs programme. This review was the foundation for subsequent evaluation activities, ensuring an informed, data-driven approach to assessing the hubs' performance and impact.

Key documents included in the review were:

• Hub programme documentation and websites: The review commenced with an examination of the overall hub programme documents to better understand the strategic objectives, operational guidelines, and milestones for each hub.
• Pathfinder hub grant offer and contract documents: These documents described the agreed scope and outcomes of the two pathfinder hubs and provided a detailed description of the specific ambitions, scope, and planned activities of the pathfinder hubs at key stages of development.
• Annual, quarterly and interim progress reports: Progress reports submitted by the two pathfinder hubs to the Scottish Government on an annual quarterly and interim basis were reviewed. These documents provided insight into the milestones achieved, challenges faced, and adaptations made during the programme.

Focus groups

Two focus groups were held, one per pathfinder hub. The focus group meetings used a hybrid format, allowing participants to join in person or online. The focus groups were held in Aberdeen (focussing on the NESCAN hub on 13^th December 2023, with 18 participants attending) and in Inverness (focussing on the H&I hub on 14^th December 2023, with 18 participants attending)). Participants represented a broad cross-section of stakeholders, including board members, recipients of project grants, statutory agencies, local authorities, community members, SMEs and academics. The discussion was guided by the use of a semi-structured topic guide that had been agreed with the Project Steering Group (PSG).

Interviews

In-depth online interviews were held with 22 stakeholders (12 from the NESCAN hub, nine from H&I hub and one stakeholder for both hubs). These interviews provided qualitative insight into each hub’s impact and the overall hub programme to date.

The interviews were guided by a semi-structured topic guide that had been agreed with the PSG. Each interview lasted for up to one hour and covered all the key research questions of the evaluation framework. The research questions used in the interviews are listed in Annex 3: Interview research questions.

Online survey

A short online survey was developed and circulated to stakeholders and members of both hubs to gain insights from a broad range of respondents.

50 responses were received, 27 relating to the NESCAN hub and 23 relating to the H&I hub. The survey contained the key research questions of the evaluation framework. A copy of the online survey is included in Annex 4: Example online survey form.

Recommendations workshop

A final round of online stakeholder workshops took place with each pathfinder hub in February 2024. The purpose of these engagements was to present preliminary findings from the evaluation and to discuss recommendations for each hub going forward. 31 people attended the engagements (15 for the NESCAN hub and 16 for the H&I hub). As with the focus group discussions, participants represented a broad cross-section of hub stakeholders.

A list of stakeholders engaged per hub per consultation type is provided in Annex 5: Stakeholder Consultation Summary.

Thematic Analysis

Thematic analysis was carried out on the collected data. Four research themes were defined based on the primary research questions. These were:

1. Value-add

2. Shift to low-carbon and resilient communities

3. Resources

4. Sustainability

Within each primary theme, a number of topics or sub-themes were also defined (see Annex 6: Qualitative analysis themes). Interview and survey excerpts were coded using these themes and topics for each hub. Different types of responses were then categorised in each theme and topic.

Based on the findings of the evaluation, a set of recommendations was developed in consultation with stakeholders. The recommendations aimed to enhance the overall hubs programme's effectiveness and impact, focusing on immediate and long-term strategic improvements.

Scope and limitations

Scope

The primary intention of this study was to conduct a theory-based impact evaluation of the two pathfinder hubs. The evaluation focussed on understanding and measuring the changes that have occurred as a result of the hubs' activities in supporting communities' transition to a low-carbon resilient future. The evaluation aimed to assess the value added by the hubs in this transition.

Limitations

Several key limitations were acknowledged in the context of this research. Firstly, due to General Data Protection Regulation (GDPR) restrictions, the research team relied on the hub managers to provide contact details for all stakeholders. This reliance may have influenced the diversity and range of participants engaged in the study.

Furthermore, the dissemination of invitations to workshops and engagement sessions was also managed through the hubs. This approach may have affected the participation and representation of stakeholders.

To mitigate these risk, during interviews and engagements, respondents were asked to identify additional stakeholders that could be contacted as part of the research. Stakeholder targets were set for different types of engagements, and when these were not reached, the hub management was engaged to ensure the required numbers were achieved. For example, the H&I online survey was sent out twice by the hub manager as the number of responses to the first publication of the survey was not high enough.

An additional limitation of the assessment was the absence of an existing monitoring, evaluation, and learning (MEL) framework specific to the hubs' programme. The lack of an existing framework resulted in the research team developing an ad-hoc set of indicators based on the study's objectives and available data. A draft MEL framework for the hubs was developed in the final stage of this evaluation. The Theory of Change (TOC) for this MEL framework is provided in Annex 7: Draft Hub MEL Framework TOC.

Additionally, there was a lack of baseline data against which the progress and impact of the hubs could be easily measured. This limitation restricted the ability to quantitatively assess the impact of the hubs by monitoring changes over time or directly attributing outcomes to the hubs' interventions.

A final limitation of this research was that it was not possible (within the time and resource constraints) to conduct a full qualitative economic impact assessment to answer the question: “did the intervention represent the best possible use of available resources?”. Rather, this question was answered through a combination of documentation review and stakeholders feedback and perceptions.