Climate Justice Fund evaluation: final report

2 Evaluation Approach and Methodology

This section outlines the evaluation approach for the CJF evaluation. The evaluation design is based around the Organisation for Economic Cooperation and Development's Development Assistance Committee (OECD-DAC)^[7] criteria: relevance, coherence, effectiveness, efficiency, impact and sustainability. During the Inception Phase, evaluation Questions (EQs) at both the project and programme-level were developed against each of the OECD-DAC criteria. Each of the following questions were explored in relation to the climate justice ToC:

Relevance

• EQ1. To what extent are the CJF projects relevant to climate justice? How well focused on existing inequalities (such as wealth disparity and discrimination based upon gender, age, disability or indigenous status) were the design of the projects? [projects]
• EQ2. To what extent is the CJF approach aligned with climate justice pillars? [programme]

Coherence

• EQ3. How well has there been coherence with local programmes, grass roots efforts and national/local policy objectives? [projects]
• EQ4. How coherent are the projects as a combined portfolio of the CJF? [projects]
• EQ5. How coherent is the CJF with Scottish Government priorities and how they relate to climate justice? What are the synergies between Scottish Government climate programmes more broadly and how do they collaborate and overlap? [programme]

Efficiency

• EQ6. How efficient, for achieving climate justice, were the CJF partnerships and collaboration with national implementing partners, governments and stakeholders? [projects]
• EQ7. To what extent are the CJF governance and management arrangements consistent with achieving its strategic climate justice objectives? [programme]

Effectiveness

• EQ8. How well were the most affected people (vulnerable, women etc) targeted and given voice in CJF implementation (at the project and programme level)? [projects]
• EQ9. How do projects in the CJF portfolio as a whole incorporate learning? [projects]
• EQ10. How effective is the Scottish Government at leveraging lessons from the CJF to increase support and delivery of climate justice? [programme]

Impact

• EQ11. How have the CJF projects and programme as a whole contributed to climate justice outcomes? [projects]
• EQ12. Looking forward, what are the emerging strengths, weaknesses, constraints and opportunities in managing and implementing the CJF and can these inform a potential future programme phase? [programme]

Sustainability

• EQ13. To what extent did project implementing partners and/or beneficiaries assume ownership and responsibility for the project preparation, implementation, and sustainability? [projects]
• EQ14. To what extent has SG leveraged the CJF to strengthen climate justice collaboration, buy-in, replication and scale-up? [programme]

The evaluation draws on a mix of qualitative and quantitative data from a literature review, project and programme documentation and interviews with programme stakeholders (refer to Table 2 below for an outline of CJF stakeholders, Table 3 provides an overview of interviews against these stakeholder groups, while the full list of documents reviewed and stakeholders interviewed is provided in Annex 2). The evaluation uses principles from the Qualitative Impact Assessment Protocol (QuIP)^[8] and data synthesis (refer to Section 2.2 below) to respond to the evaluation questions and to develop a Theory of Change (ToC; refer to Section 3.2). The evaluation consisted of three phases, which are outlined below.

Table 2. Summary of CJF Stakeholders

Scottish-Based Stakeholders

Role in CJF: Scottish Government CJF managers, CJF programme/grant managers, non-CJF Scottish Government climate change projects

CJF project stakeholders

Role in CJF: Scottish-based grantees, partner country project implementers, beneficiaries and cooperatives

Non-CJF project stakeholders

Role in CJF: Partner country National, district and local government, government extension officers, village heads/traditional leaders, other NGOs and project implementers in CJF project areas

2.1 Evaluation Phases

Phase 1: Scoping: A five-week inception phase was used to mobilise the team, design and prepare for the evaluation. This phase included:

• The development of the evaluation questions;
• Selection of the evaluation methodology;
• A review of the overview CJF documents;
• A climate justice literature review (based on published climate justice literature, including academic, policy-focused and practitioner-led materials; refer to Annex 1);
• The development of an overarching climate justice ToC, which was further refined for CJF through this evaluation;
• The development of the evaluation tools (survey questions and data collection guides), which were developed around the OECD-DAC criteria focused on testing which aspects of climate justice (based on the climate justice ToC) were covered by the CJF through its portfolio of projects;
• Initial online/phone interviews with CCPM and CJIF programme/grant managers, as well as preparation for stakeholder meetings and fieldwork through calls with project implementing partners.

Phase 2: An Interim Document and Data Collection Phase: A five-week data collection phase was used to:

• Undertake Key Informant Interviews (KIIs) with Scottish stakeholders;
• Undertake KIIs for the country case studies in Malawi, Rwanda and Zambia case studies;
• Undertake an online survey of Round 1 and 2 project implementers using Survey Monkey;
• Provide a presentation of interim findings to ensure the evaluation aligned with Scottish Government expectations and needs.

Phase 3: A Final Analysis and Synthesis Phase: A two-week wrap-up phase was used to:

• Test and elaborate on the preliminary findings and the ToC with CJF implementation stakeholders through an online moderated discussion;
• Analyse and synthesise results from the literature review, country case studies, online surveys, Scottish KIIs and online moderated platform, in order to draw conclusions, highlight lessons and make recommendations for future phases of funding under the CJF.

2.2 Data Analysis Methods

Literature Review: A review of grey literature on climate justice was conducted as part of the inception phase (refer to Annex 1). In total, 21 documents were reviewed. The literature ranged from organisational policy briefs, project summaries, and NGO/foundation websites. Literature specific to climate justice is not expansive, with the majority published pre 2016 (i.e., more than five years old). The aim of the literature review was to, as far as possible, conduct a rapid review of relevant literature in order to help inform how to structure a global Climate Justice Pathways ToC. As the CJF does not have an existing overarching results framework or ToC, and given the focus on learning from the CJF to inform potential future phases of funding, this evaluation was tasked with building a specific ToC for the Scottish Government . Specifically, the objectives were to map what current funding and programmes are being implemented under the banner of 'climate justice'; who the key players are; clarify how Climate Justice is being defined by different actors and document commonalities between definitions; and identify any lessons learned from other programmes.

Document Review: More than 65 CJF programme and project documents were reviewed. The documents were assessed for evidence against the evaluation questions, as well as evidence of activities undertaken that align to the three pillars of climate justice. The document review was also used to refine key informant questions.

Theory of Change Development: The CJF does not have an overarching results framework or ToC (although the six Round 2 projects, 15 CJIF projects and the CCPM each have their own results frameworks). In addition, the literature review found a lack of consensus on the definition of climate justice internationally. As a result, the evaluators developed an overarching climate justice ToC, which was used to test the CJF portfolio to better understand how the CJF contributes to climate justice and whether there were aspects of climate justice that should be included in future phases of funding.

Key Informant Interviews: Semi-structured interviews were undertaken with CJF programme and project stakeholders and relevant non-project stakeholders (refer to Table 3). Separate guides were developed for grant managers, implementing partners, non-project stakeholders and beneficiaries. The questions were developed around the OECD-DAC criteria, with project specific questions related to climate justice for each intervention.

Table 3. CJF stakeholders and project engagement

Stakeholders	Count	Type	Comments
Evaluation Team	12 (7F)	n/a	The evaluation team of consisted of 12 national and international experts and field staff, it was made up of 7 Females (F) and 5 Males (M), and included 4 Malawians, 1 Rwandan and 1 Zambian)
Scottish-Based Stakeholders
Scottish Government CJF managers	4 (2F)	KIIs	1 Scottish Government CJF not available due to holidays
Grant Managers	4 (4F)	KIIs	1 CJIF, 3 CCPM
Non-CJF Scottish Government project	1 (1F)	KIIs	1 non-CJF not available due holidays
Round 1 Stakeholders
Round 1 project implementer	2	Online survey	Contact details provided by Scottish Government for 5 people across 4 projects, 1 project could not be contact (2 emails, both undeliverable), 1 out of office, no indication why other 2 projects did not respond
Round 2 Stakeholders
Round 2 project implementer	1	Online survey	Contact details provided by Scottish Government for 13 people across 6 projects (4 emails undeliverable). All projects had at least one email, 1 out of office, no indication why other projects did not respond
CCPM and CJIF Stakeholders			4 CJIF projects (all in Malawi) were not contactable, documents were reviewed but Scottish Government documentation does not provide details to complete analysis of climate justice elements
Deep-Dive: Malawi Project 1	12 (7F)	KIIs / QuIP	10 Beneficiaries (7F/3M; QuIP), 1 Project Officer (M), 1 Stakeholder (M)
Deep-Dive: Malawi Project 2	10 (4F)	KIIs / QuIP	6 Beneficiaries (3F/3M; QuIP), 1 Committee Chair (M), 1 Implementing Partner (M), 1 Project Officer (M), 1 Stakeholder (F)
Deep-Dive: Malawi Project 3	13 (9F)	KIIs / QuIP	9 Beneficiaries (7F/2M; QuIP), 1 Implementing Partner (M), 1 Project Officer (M), 2 Stakeholders (2F)
Deep-Dive: Malawi Project 4	13 (8F)	KIIs / QuIP	10 Beneficiaries (8F/2M; 1 Implementing Partner (M), 1 Project Officer (M), 1 Stakeholder (M)
Malawi Project 5	1	KIIs	1 Implementing Partner (M)
Malawi Project 6	1 (F)	KIIs	1 Implementing Partner (F)
Malawi Project 7	1	KIIs	1 Implementing Partner (M)
Malawi Project 8	1 (F)	KIIs	1 Implementing Partner (F)
Rwanda Project 1	7 (2F)	KIIs	3 Beneficiaries (1F/2M), 3 Implementing Partners (1F/2M), 1 Stakeholder (M)
Zambia Project 1	5 (1F)	KIIs	5 Implementing Partners (1F/4M)

Qualitative Impact Assessment Protocol: QuIP is a qualitative evaluation methodology that can be used to assess the types of impact that an intervention is producing, and to test the intervention's ToC. Principles from the QuIP method were used with project beneficiaries in the four deep dive Malawi projects. The approach is based on asking participants about the most significant changes that have taken place in different domains, or areas of their lives, over a specified period. The domains are selected to reflect the specific outcomes the intervention is intended to affect. Participants are also asked what they consider to be the main factors driving or enabling those changes. Participants are not directly asked about the intervention of interest, to avoid pro-project confirmation bias. The approach allows for the development of stories of change that describe which factors, including interventions, are linked to outcomes in each domain, and how. The QuIP method provides evidence of programme participants' own narratives of the causal mechanisms leading to change and is complementary to other approaches such as monitoring data, and qualitative and quantitative approaches.

Online Survey: An online survey based on the implementing partner interview guide was sent to all Round 1 and Round 2 project implementers, although, as noted in Table 2 above, only one Round 1 and two Round 2 project implementers responded despite three reminders. This is partially due to a number of emails no longer being valid but it is unclear why other projects did not respond (most likely due to the time since project close).

Online moderated platform: The online moderated platform was hosted on Live Minds (chosen due to its low bandwidth environment and alignment with GDPR). Respondents were invited to participate for approximately 20 minutes per day, over the three days, at a time of day that suited them (and their electricity/internet connection). The eight respondents (all CJF project implementers were invited to participate) included both grant managers, one Zambia project implementer and four respondents a Malawi based project. There was a high level of engagement on the platform around questions around climate justice, what it means to project implementers and the ToC, as well as project implementation, the role of partnerships, factors that contribute to project success and suggestions for improving future phases of CJF funding. For most questions, respondents could see and comment upon one another's responses, which respondents did, also commenting on how much they learnt from one another through the experience.

Data synthesis: The synthesis brings together the findings from across the literature and document reviews, and stakeholder engagement (through key informant interviews, online survey and online moderated platform). A systematic approach was used to compare, contrast and integrate the empirical evidence against the OECD-DAC criteria and synthesise the evaluation data to support synthesis of findings. Project documents were systematically reviewed against the evaluation questions, while question guides were developed to respond to different aspects of the OECD-DAC criteria. Results from the country case studies and online moderated platform were written up against these criteria and then cross-referenced with the Scottish-stakeholder key informant interviews and the Round 1 and 2 online survey to identify themes and establish the confidence in findings.

2.3 Evaluation Challenges

Almost all the CJF projects have now closed (some more than five years' ago), which meant that in many cases project staff no longer worked for the organisation that implemented the project and/or were not always available to discuss the implementation of CJF. In addition, this meant that recall of project activities and experiences may be incomplete. The evaluation used a combination of project documentation and interviews/surveys to attempt to mitigate this risk. However, the availability of project documentation for Round 1 was extremely limited and, as noted above, only one Round 1 and two Round 2 project implementers responded to the online survey, despite two follow up reminders. As such, project findings should be interpreted as referring to Round 3, except where Round 1 or Round 2 are specifically mentioned. In addition, with the exception of the four in-depth Malawi case studies, the evaluation relies on project's self-reporting and interviews with project staff, which means there is the potential for positive self-reporting bias in the findings (i.e. potential exaggeration of positive impacts and downplaying of negative impacts). However, this is less of an issue given the climate justice learning (rather than accountability) focus of the evaluation. The QuIP methodology is also designed to overcome this bias. Engagement could be increased in future CJF evaluations by undertaking evaluations in the final year of project implementation (while project staff are still in the field) and allowing for longer evaluation timeframes. For example, programme-level Monitoring and Evaluation (M&E) could run alongside programme implementation through either an independent evaluator or as part of the grant manager's role, which could include development of a baseline of the climate justice situation, as well as learning and communication products.

Covid restrictions meant that research covering the one Zambia and one Rwanda projects was desk-based. This was more complicated than usual as the strain of the lockdown restrictions in Rwanda and an upcoming election in Zambia meant that internet connection was inconsistent, causing delays in contacting stakeholders. There were also challenges with availability of Scottish stakeholders due to the summer holidays. This means that beneficiary feedback draws from four Malawi case study projects (using the QuIP methodology), as well as some phone interviews for the one Rwanda project. The evaluation took a flexible approach (conducting interviews later in the evaluation where necessary) to maximise the number of interviews possible despite these constraints.

The CJF uses a relatively open design approach, allowing projects to develop and adapt their approach in response to their work with communities. This aligns with a climate justice approach but means there is no one business case or programme results framework that could form the basis of the evaluation, although project results frameworks are in place for CCPM and each of the six Round 2 and 15 CJIF projects. The 12-week evaluation timeframe was therefore challenging, as the evaluation of the CJF programme and its 31 projects had to be fully designed and implemented within this period. The evaluation team prepared clear question guides and report templates to minimise this challenge, although depth of analysis of the different projects varies.