Climate Justice Fund evaluation: final report

3 Results

Key findings on how CJF and its portfolio of projects contribute to climate justice are presented below against each of the OECD-DAC criteria.

3.1 Definition of Climate Justice

Currently, there is no one 'official' definition of climate justice (refer to Annex 1). In the evaluation terms of reference, the Scottish Government defines climate justice as:

• 'An approach which recognises that it is those least responsible for the global climate emergency that are being affected first and most severely by it.
• An effective response must tackle existing inequalities such as wealth disparity and discrimination based upon gender, age, disability or indigenous status, as the impact of climate change can be made worse by these factors.'

The most commonly used definition found in the literature is from the Mary Robinson Foundation:^[9]

'Climate justice links human rights and development to achieve a human-centred approach, safeguarding the rights of the most vulnerable people and sharing the burdens and benefits of climate change and its impacts equitably and fairly. Climate justice is informed by science, responds to science and acknowledges the need for equitable stewardship of the world's resources.'

Drawing on the above definitions, the Scottish Government's climate justice policy, the evaluation literature review and the ToC (outlined in Section 3.2), the evaluator proposes the following climate justice definition:

Climate justice is a people-centred, human rights-based approach that aims to share the benefits of equitable global development and the burdens of climate change fairly, while building trust between developed and developing countries. Climate justice recognises that the poor and vulnerable are the first to be affected by climate change, exacerbating existing inequalities, and will suffer the most, despite having done little or nothing to cause the problem.

This Evaluation would determine that using this definition of climate justice would therefore mean that climate justice projects on the whole should include the following elements:

• ensuring a participatory, community-defined, needs-based approach to development (procedural justice);
• providing support for climate change resilience for the most affected areas and people in the Global South (distributive justice); and
• building understanding and capacity that enables local people to actively engage in decision-making and advocacy (transformative justice) to enable equitable, sustainable development in the face of climate change.

3.2 Theory of Change

A Theory of Change (ToC) is an outcomes-based approach that applies critical thinking to the design, implementation and evaluation of initiatives and programmes intended to support change in their contexts. Given the long-term nature of achieving the transformative aspects of climate justice, which requires global support at various levels, the evaluations ToC defines shorter-term outputs and medium-term outcomes, which provide intermediate measures that indicate the likelihood that CJF will achieve its long-term impact goals.

As part of the inception phase, a Climate Justice Pathways ToC was developed by the evaluator (Figure 2). In understanding climate justice, the evaluation drew on definitions of different types of justice to categorise potential interventions and impact pathways:

• Distributive Justice relates to equal access to and sharing of resources and benefits and is used in Climate Justice definitions to include both access to resources and benefits and equitable sharing of costs of responding to climate change;
• Procedural Justice relates to transparent, fair and equitable decision-making processes;
• Transformative Justice relates to structural inequities and focuses on mainstreaming understanding of Climate Justice issues, as well as building capacity.^[10]

The three climate change pillars incorporate both the approach to project implementation (procedural justice), as well as the types of interventions supported by a project (distributive and transformative justice). The three pillars are interlinked (hence the arrows between climate justice pillars). For example, participatory processes and needs assessments (procedural justice) supports the selection of interventions (distributive justice), while the success of these interventions is supported by strengthening local institutions (transformative justice) and engagement with local and national stakeholders, including government extension workers (procedural justice). Furthermore, by building communities capacity to make decisions around climate change and supporting community members to advocate for their community's needs and rights (transformative justice), projects can support both the sustainability of project outcomes, as well as leveraging additional support by communities for new interventions or scale-up, and replication by other actors.

Drawing on CJF programme documentation and the climate justice literature review, the three pillars of climate justice were built into the evaluation approach and question guides (particularly in terms of questions related to the CJF's relevance and impact in terms of climate justice). This Climate Justice Pathways ToC was then refined iteratively with Scottish Government and CJF stakeholders to provide a programme-level Climate Justice Interventions ToC (as shown in Figure 3 below). As part of the inception phase, a list of interventions under the three climate justice pillars were explored and elaborated. The Climate Justice Interventions TOC was used to evaluate the CJF for alignment with climate justice pillars (refer to Figure 4 in Section 3.5 below).

Both the climate justice pathways and interventions ToCs outline the separate but interlinked pathways to change along each of the three climate justice components: procedural justice, distributive justice and transformative justice. The ToC pathways show the progression from climate justice interventions through outputs and outcomes to achieve the overarching climate justice impact of: 'The benefits of sustainable global development are shared equitably through a people-centred, human rights-based approach that ensures the impacts of climate change do not disproportionately impact those who have done the least to contribute to it.'

The climate justice interventions ToC was shared with respondents in the online moderated platform for comment and to support its refinement for the CJF. Broadly, the ToC resonates and was popular with online moderated platform respondents who welcomed seeing the articulation of transformative, procedural and distributive justice within the ToC and the balance between global and local outcomes. Respondents agreed that climate justice projects should always be participatory (procedural justice) and that advocacy should be built in to future CJF projects (transformative justice). CJF stakeholders welcomed the potential development of an evaluation framework developed from the ToC with quantifiable indicators attached to the outcomes.

Specific CJF project stakeholder feedback on the ToC was used to further refine the Climate Justice Interventions TOC presented in Figure 3 below. In addition, the QuIP analysis presents beneficiary-defined ToCs for four Malawi projects.

While the ToCs provided a key tool by which the Evaluation could assess the Climate Justice Fund as a whole in delivering climate justice outcomes, care should be taken when interpreting individual project findings against this criteria. None of the CJF projects were originally designed or selected against these criteria.

The Climate Justice Interventions ToC (Figure 3) is proposed by the evaluator to support the development, monitoring and reporting of future phases of the CJF. The Climate Justice Interventions ToC provides the potential starting point for further development of a CJF results framework that would monitor and evaluate assumptions and changes along the intervention pathways.

3.3 Project Overview

3.3.1 Malawi

Malawi's overall climate is tropical, with temperatures in higher topographies relatively cool. The land is made up of a variety of flood plains, wetlands and forests within the Lower Shire Valley. These areas are particularly vulnerable to the effects of climate change. Drought and flood disasters are the most immediate impacts which directly affect more than 500,000 civilians.^[11] Malawi's economy is predominantly rainfed agriculturally based, which is experiencing decreasing yields of rainfed crops, a shift in the lean season and recurrent food shortages. Water resources are reducing in supply and quality; fisheries are experiencing reductions in productivity; and ecosystems are facing loss of biodiversity, reduced forest production and loss of ecosystem-dependent livelihoods. In addition, human health risks include increased diarrheal disease, expanding malaria areas to the highlands, and increased food insecurity. Changes in Malawi's hydrological cycle and changes in temperature may also negatively impact hydropower generation.^[12] This further exacerbates vulnerable people as the majority of domestic energy in Malawi is produced via hydropower. Gender inequalities are deep rooted in Malawi, contributing to sustained or worsening levels of poverty for women and girls, which in turn makes them more vulnerable to climate change related shocks^[13] by reducing their adaptive capacity.

Since 2012, CJF has supported a total of 20 projects in Malawi through three rounds of funding. The third round, implemented from 2017 to 2020, included Malawi Project 1(with a budget of £4.7 million), and innovation projects (total budget £1.5 million or with individual grants of approximately £100,000 in funding over two to three years), which supported eleven projects across all three regions of Malawi. Each of the innovation projects was selected on the criteria of trialling innovation and focused on one of three sectors: water, energy, or agriculture/food. Malawi Project 1 was selected to deliver a broader programme with longer time frame, and focused on all three sectors, by delivering layered interventions in targeted communities.

The evaluation looked in detail at four projects in Malawi summarised in Table 4. In addition, four other Malawi projects were evaluated through document review and phone interviews with implementing partners (also summarised in Table 4).

Table 4. Overview of Malawi projects evaluated

Deep-Dive: Malawi Project 1 (still underway at time of evaluation)

Summary of project: A consortium of local partners implementing a large range of resilience-focused interventions selected through a rigorous community participatory process, in several locations, including a student advocacy component.

Climate Justice aspects:

• Distributive Justice: The project builds community capacities for absorbing and adapting to climate change, with a focus on the needs of vulnerable groups.
• Procedural Justice: The project utilised participatory methods and works closely with local government and institutions.
• Transformative Justice: Both the community interventions and student component aim to build capacity for climate justice advocacy.

Deep-Dive: Malawi Project 2

Summary of project: The project rehabilitated an artesian well system to support irrigated farming, fish farms, and drinking water supply; this was complemented with support for additional activities including beekeeping.

Climate Justice aspects:

• Distributive Justice: The project built community capacities for absorbing and adapting to climate change through livelihoods support.
• Procedural and Transformative Justice: The project supported the development of local institutions to manage the artesian well system.

Deep-Dive: Malawi Project 3

Summary of project: The project worked with a community struggling with deforestation to pilot a clean cooking stove utilising crop waste.

Climate Justice aspects:

• Distributive Justice:The community received a prototype stove for use in its school.
• Procedural Justice: The community participated in the development of a technology aimed at meeting local needs.
• Transformative Justice: Supported systems and capacity for integrating communities into technology development.

Deep-Dive: Malawi Project 4 (still underway at time of evaluation)

Summary of project: The project aims to improve transparency around borehole quality by testing boreholes, build capacity for government borehole monitoring and management, and increase awareness of water rights through trainings and distribution of materials.

Climate Justice aspects:

• Distributive Justice: The long run objectives of the project are to improve water access in vulnerable communities.
• Procedural Justice: The project works closely with local government institutions, and supports communities in securing repairs.
• Transformative Justice: The project includes training and raising awareness of water rights among local stakeholders, and building institutional capacity.

Malawi Project 5

Summary of project: The project commenced in 2019 and ended in 2021. It was implemented as a partnership. The project primarily aimed to support female small holder farmers particularly, allowing them to grow crops all-year-round through greenhouse farming.

Climate Justice aspects:

• Distributive Justice: The community were supported to improve yields.
• Procedural Justice: The project trained extension officers and worked with cooperatives.
• Transformative Justice: n/a

Malawi Project 6

Summary of project: The project commenced in 2019 and ended in 2021. It was implemented in Chang'ombe and Sakata towns. The project aimed to help bakery cooperatives rely less on wood burning and transition to solar systems as a cleaner way to produce goods. Another intervention was encouraging tree planting as deforestation was a factor to climate change

Climate Justice aspects:

• Distributive Justice: Solar ovens were introduced that provide health benefits and reduce labour for women. Beneficiaries were also supported with diversifying their products.
• Procedural Justice: The project worked with other NGOs in the area, as well as undertaking focus group discussions to identify the communities and vulnerable peoples to support.
• Transformative Justice: n/a

Malawi Project 7

Summary of project: The project commenced in 2018 and ended in 2020 after two years of implementation. The project was started in October 2018 and was scheduled to end in June 30th 2020 but was given a 6 month no cost extension (due to the Corona Virus) up to December 2020. It was implemented in Sukumbizi tea cooperation in Mulanje, Phata sugar cooperation in Chikwawa, and KASFA rice cooperation in Karonga. The aim was to domesticate the AWS Standard 2.0 - Alliance for Water Stewardship.

Climate Justice aspects:

• Distributive Justice: The project influenced the water standard to make it more appropriate for smallholder farmers.
• Procedural Justice: n/a
• Transformative Justice: The project built understanding of cooperatives understanding of the legal framework for water, and supported representation in formal forum.

Malawi Project 8

Summary of project: The Sitolo Microgrid project begun in November 2017 and was scheduled to end in March 2020 but ended in July 2020 instead. The project empowered the communities to be able to use clean energy, understand its utility and benefits, and benefit from it as well through the provision of eco-friendly electrical systems.

Climate Justice aspects:

• Distributive Justice: The community was able to achieve cleaner energy (health benefits, reduced labour for women collecting wood).
• Procedural Justice: Cooperatives were brought together to exchange learnings.
• Transformative Justice: A lesson learning workshop was held with the Council who approved additional budget through its Local Development Fund and supported fundraising to scale activities.

* The majority of the innovation projects received approximately £100,000 in funding over two to three years (with the three 2019 grants having under two years), compared with a budget of £4.7m over four years for Malawi Project 1.

The Malawi CJF projects examined in detail in this evaluation all – at least partially – addressed all three pillars of climate justice: distributive, procedural, and transformative justice. However, the balance of focus differed across projects.

Within the pillar of distributive justice, interventions focused on delivering interventions to help support community resilience to shocks through absorptive and adaptive capacities, which enable communities most vulnerable to climate change impacts to withstand shocks and stressors better, and to adopt new approaches or livelihoods that are less vulnerable to climate change. For example, Malawi Project 1 delivered a wide range of interventions in communities, including trainings on improved and more resilient agricultural practices, watershed management projects, and livestock pass-on programmes; Malawi Project 2 supported resilient livelihoods such as fish farming and beekeeping. A number of projects employed methods of selecting project sites aimed at targeting the most vulnerable. For example, the Malawi Project 4 chose to work in an area where water access is a known challenge; the Malawi Project 3 worked with a community facing ongoing challenges with deforestation and lake drying.

Procedural justice was addressed through a wide range of approaches. With respect to participatory processes, some projects focused on working directly with communities, while others worked with government institutions, civic organizations, or local governance structures like Village Development Committees to ensure that the voices of those affected by climate change were included. For example, Malawi Project 1 used a rigorous participatory process to select interventions, and worked closely with a number of local government institutions. The Malawi Project 4 worked closely with government officials to build capacity for long-term management of rural water resources. In some cases, local voices were amplified by working with local organizations, or by embedding programs in existing multi-year projects. The Malawi Project 3 was an example of the latter, where the development of a clean cooking stove was implemented in partnership with a community that the implementing partners had a long-standing relationship with, partly in response to the community's advocacy for solutions to deforestations.

Transformational justice was also addressed through a variety of angles. In the context of projects that worked closely with communities, such as the Malawi Project 1 and the Malawi Project 2, mechanisms for addressing transformative justice included building the capacity of communities to advocate for themselves and engage in decision-making related to climate change. Some projects focused on achieving transformational justice at the level of various systems: the Malawi Project 3 developed a model for local integrating communities into the development of technologies to address climate change, while Malawi Project 4 seeks to increase transparency around borehole quality data, and to increase awareness and recognition of water rights in Malawi.

3.3.2 Rwanda

Rwanda is a landlocked country, with a moderate climate and relatively high rainfall. The two main seasons in Rwanda are a main rainy season from March to May, along with a shorter season from September to December. Overall, it is expected that increased temperatures and intensity of rainfall, with longer dry seasons will present different challenges to different regions: drought and desertification in the east; erosion and landslides in the mountainous region; and severe flooding in the central and northern region.^[14] It is expected that Rwanda's annual temperature will increase by as much as 2.3 degrees Celsius based on current projections, with the likely increase of extreme heat waves expected to last up to 22 days.^[15] Agriculture is one of the most important sectors in Rwanda, employing 80% of the population.^[2] rising temperatures are likely to reduce Rwanda's productivity of tea and coffee, two of its main export crops. This is due to heat stress and changes in agro-ecological zones.^[1] Households which rely upon rain fed agriculture are at risk of substantial post-harvest losses due to changes in rainfall intensity and frequency. Human health impacts are also likely, due to the stress on water resources that will likely increase incidents of extreme flood events, whilst expanding the area of vector borne disease and waterborne disease.^[2] Climate change may also impact on Rwanda's energy access. Currently 7 million people lack access to electricity.^[2] To overcome this, there has been an emphasis on the use of hydropower, however with the changes in seasons and increasing dry spells experienced, hydropower generation is expected to reduce.

This evaluation focused on the main Rwanda project funded by the Scottish Government under the Climate Justice Fund. The Rwanda Project was piloted in Kopakaki Dutegure and Buhanga Cooperative. The objectives of the Rwanda initiative were to assess the feasibility and financial sustainability of clean and green implementations in coffee cooperatives.

The project mainly covered two components of Clean and Green Technologies in Rwandan coffee cooperatives:

1. The installation and feasibility of Solar PV Units as a means of providing renewable electricity and extra capacity to cooperatives, and;

2. The installation and assessment of wastewater treatment facilities to cooperative Washing Stations that could treat waste effluence produced in the coffee cleaning process

In 2018 the project funded the installation of a Solar PV Unit in Buhanga Cooperative, Gisagara district in the Southern province. Buhanga was selected as a site to monitor and model the feasibility of Solar PV as a technology for use among other cooperatives.

The benefits provisioned to cooperatives from supporting wastewater treatment are enumerable and demonstrable. Though not all cooperatives are the same, and in Rwanda cooperative coffee washing stations (hereafter referred as CWS or washing station) and communities can vary distinctly by size, geography, topography, climate, harvest cycles, and levels of economic development.

The project set out to understand the different types of costs and benefits that wastewater treatment facilities can afford to cooperatives, paying close attention to the distinctions of how and if the different contexts affect these opportunities across Rwanda.

In 2018, Kopakaki Dutegure, a cooperative in the Western region of Karongi, received the first wastewater treatment site as part of the Rwanda Project. Water samples taken from Kopakaki treatment site demonstrated a significant reduction in water usage, as well as pollutants coming from the washing station. The water treatment facility also contributed to securing Rainforest Alliance certification, leading to both environmental and financial dividends. A Summary is shown next in Table 5.

Table 5. Overview of Rwanda projects evaluated

Project: Rwanda Project 1

Summary of project: The project primarily supported feasibility work and supporting a financially sustainable approach to greening production of coffee in two cooperatives in Buhanga and Kopakaki. Buhanga support started in 2018 with a solar PV installation. While In 2018, Kopakaki Dutegure, a cooperative in the Western region of Karongi, received the first wastewater treatment site as part of the project. Following a refinement and adaptation to the objectives of the project in late 2019, a second wastewater treatment system was installed at Buhanga in 2020

Climate Justice aspects:

Distributive Justice: Use of local labour rather than external actors who tend to be costly.

Adaptive approaches like use of local materials, like sand, stones, water, for construction purposes.

Procedural Justice: Conflict resolution between the cooperatives and the communities

Engagement of cooperatives

Partnership and participation of different stakeholders.

Transformative: n/a

The evaluation finds that the Rwanda Project has, to some degree, covered aspects of climate justice through distributive, and procedural, but not through transformative justice which is understandable given the focus on trialling innovations.

Distributive justice was only partially addressed by ensuring local labour services were used – and the use of local and environmentally friendly materials for construction. However the extent to which further distributive justice aspects, such as embedding project concepts within the local community is not clear.

Aspects of procedural justice were partially addressed through the inclusion of cooperatives and engagement of the local government with these cooperatives. In addition, a local firm was contracted to design and construct required infrastructure, ensuring local knowledge and skills were used as a key component of the project. The project implementer also noted that some aspects of procedural justice had been undertaken by its Rwanda programme more broadly prior to undertaking the CJF project.

3.3.3 Zambia

Zambia has three key climate regions: tropical savanna; warm semi-arid; and humid subtropical. Zambia experiences three distinct seasons: a hot and dry season (August to November); a wet season (November to April); and a cool and dry season (May to August).^[16] It is predicted that, overall, Zambia will experience an increase of extreme weather events and an overall increase in temperatures, with 'hot days' increasing by 15‑29% and 'hot nights' projected to increase by 26-54%.^[17] Although the overall rainfall is not projected to change based on the annual mean, the seasonality of rainfall may shift, with the proportion of rainfall from heavy events expected to increase significantly, further exposing the country to flash flooding. Zambia relies heavily on rain-fed agri-economy practices, with 9% of GDP generated through agriculture. Due to increased temperatures and changes in rainfall variability and intensity, Zambia is likely to see less predictable growing seasons, which will affect livelihoods and food security. Other aspects likely to affect those who depend on agriculture are: increased pests affecting crops and livestock; increased soil erosion; ruined crops due to waterlogging; and increases in crop failure. Periods of drought are likely to affect water resources and increase human and animal health impacts from extreme heat such as heat stress, with incidents of mortality in livestock also expected to increase.

The CJF has supported three climate justice projects in Zambia since 2017. This evaluation only considers Zambia Project 1, which commenced in November 2017 and ended in March 2021, three months behind schedule due to COVID-19 related restrictions. It was implemented as a tripartite partnership in 15 urban wards of Kabwe town (the capital of Central Province). The project primarily aimed to sensitize communities in sustainable waste management through demonstrations and community awareness programmes. The demonstrations consisted of biogas production and the production of charcoal briquettes. Table 6 shows a summary of the Zambia Energy Project that was evaluated.

Table 6. Overview of Zambia project evaluated

Project: Zambia Project 1

Summary of project: The project reduced dumpsite waste through repurposing and reusing the waste. Biodegradable waste such as kitchen leftovers and/or animal waste were converted to gas using a Biogas digester. Other waste such as groundnut shells, saw dust, twigs, and leaves were converted into smokeless charcoal briquettes, which were demonstrated to be a cheaper and more eco-friendly alternative to wood charcoal and firewood.

Climate Justice aspects:

Distributive Justice: Local initiatives around climate change adaptation.

Procedural Justice: Capacity, sustainability, embedding, ownership. Sharing of knowledge/ patents. Contributing to climate justice knowledge base. Engagement of communities in participatory processes. Working in partnership to share capacity/ benefits.

Transformative Justice: Capacity building at local, national and international levels to advocate for most affected peoples (gender, vulnerability).

The CJF Zambia Project examined in more detail within this evaluation has partially addressed all three pillars of climate justice to some degree, however to varying degrees within each.

Distributive justice was partially addressed through the project focusing on creating local initiatives around climate change adaptation. The project focused on community outreach initiatives whereby the communities' ability to adapt to climate change is said to have been enhanced. However, it is noted that the implementing partner interviewed views "climate justice" as safeguarding the environment by using eco-friendly solutions to replace harmful practices. This reflects the projects focus on innovation for which it was selected, rather than the human centered justice narrative of climate justice.

Procedural justice was partially addressed through three areas. The project ensured communities were mobilized with support from local authorities and community leaders. Community members were invited to participate in meetings and radio programmes as well as the operationalization of some of the project features. Waste collection groups were also established which was possible through the engagement of the Directorate of Public Health at the local Municipal Council in Kabwe. There has been, therefore, effort to embed project activities within local contexts beyond the direct project activities.

There has been some effort within the project to align to transformative justice, which crosses over slightly to the above two impact pathways, to embed capacity building in order to advocate for those most affected by climate change within the project region/scope (i.e., vulnerable groups). Community leaders were activated to serve as champions of climate justice in a bid to support equal access to the municipality's resources. The participation of district technocrats and political leaders reinforces the potential for climate resilience. The community was generally intrigued by alternative methods. The women would share the briquettes with neighbours and advocate for the product within their wards even without prompting from project teams.

3.4 Findings

The evaluation findings synthesise the qualitative and quantitative data from the document reviews and key informant interviews around a set of evaluation questions. The evaluation questions were designed to respond to the OECD-DAC good practice evaluation criteria (relevance, coherence, effectiveness, efficiency, impact and sustainability) at both the CJF project and programme-level.

3.4.1 Relevance

In terms of Relevance (are the interventions designed to deliver climate justice), the key findings are:

Evaluation Question 1:^[18] To what extent are the CJF projects relevant to climate justice? How well focused on existing inequalities (such as wealth disparity and discrimination based upon gender, age, disability or indigenous status) were the design of the projects?

CJF projects tended to target the most vulnerable areas but not always the most vulnerable peoples (Finding 1), while targeting of beneficiaries under the climate justice innovation projects tended to focus more on 'technical aspects' rather than taking a participatory approach to identifying local needs. Nevertheless, many CJF projects had a strong understanding of, and focus on, climate justice, although some projects had more of a standard development focus (Finding 2). In interpreting the results, it should also be noted that communities in Malawi, Rwanda and Zambia are vulnerable to a large number of shocks, and CJF projects addressed many of them in a variety of ways (Finding 3).

Evaluation Question 2: To what extent is the CJF approach aligned with climate justice pillars (distributive, procedural and transformative justice)?

The CJF approach is well aligned with climate justice pillars, although it could go further (Finding 4), in particular, by increasing focus on procedural justice (Finding 5).

Finding 1. CJF projects tended to target the most vulnerable areas but not always the most vulnerable people

Although projects could go further in incorporating targeting to women and the most vulnerable, the overall approach taken is good. In general, projects focused on targeting particularly vulnerable implementation areas, although not necessarily the most disadvantaged (including women, youth, those with disabilities etc. who could be targeted through the selection of appropriate interventions). Nevertheless, the areas are vulnerable to climate change and most people in these areas live below the poverty line. For example, the Zambia Project was relevant to climate justice as it targeted poor communities with limited means of coping with climate change. A more focused targeting approach could have been used, however, in order to place emphasis on women, elderly and disabled people. An unforeseen result, however, was that women were the main beneficiaries as they tended to collect and sell higher quantities of raw waste materials.

Targeting of beneficiaries under the climate justice innovation tended to focus more on 'technical aspects' rather than taking a participatory approach to identifying local needs. Many innovation-focused projects focused on the technical aspects of their interventions rather than working with communities to identify their needs. For example, the Rwanda project worked directly with identified cooperatives and not individual cooperative members or households. Therefore, the project was mostly relevant to the cooperatives with environmental and climate change challenges, thus the benefits/impact of the project was more directed at cooperatives than individual members. On the other hand, many projects (such as those in Zambia) used an open participation approach without targeting, which allowed anyone to join based on interest. By using both targeting of interventions towards vulnerable people (in design and participation) and an open approach that enables participation by the most interested community members, projects can maximise equity benefits.

Finding 2. Many CJF projects had a strong understanding of, and focus on, climate justice, although some projects had more of a standard development focus

Implementing staff at all levels of all projects were able to articulate how they viewed climate justice, and how their work contributes to it in different ways. In many cases, staff had a nuanced understanding of how local issues (such as deforestation and population pressures) interact with the effects of global climate change to put additional stress on vulnerable populations. Many expressed the view that communities need to both receive help, reflecting distributive justice, and be active participants in finding solutions, reflecting procedural and transformative justice. However, in some instances, understanding of climate justice could be broadened. For example, in Zambia, project staff understood climate justice as safeguarding the environment and using eco-friendly solutions, which supports community adaptation but overlooks aspects of equity and advocacy (procedural and transformative justice).

This keen understanding of climate justice pillars was reflected in a very strong focus on climate justice in project design for Malawi Project 1 and a strong focus for many other projects. Most projects had elements focusing on each of the three pillars of climate justice and a strong focus on vulnerable populations. However, at least some of the projects appear to be more traditional development projects that had been 'tagged' as climate justice to attract funding. On the other hand, some projects that were designed as climate change adaptation or mitigation projects, without a strong focus on climate justice went on to deal with climate justice issues in their implementation, particularly where they focused on procedural justice by working with the underserved (who are often otherwise overlooked in traditional development and climate change adaptation/mitigation projects). In addition, the variation in how well projects addressed the three pillars of climate justice has implications for programme impact and sustainability (i.e. procedural justice supports better targeting of local needs and is more likely to lead to local buy-in and ownership in the long-run, as is the case for the Malawi Project 1 and a Malawi Project 3). It will be important for future phases of CJF funding to clearly articulate expectations for climate justice projects to ensure projects fully align with the three pillars of climate justice proposed by this evaluation and this is built in from the project design phase. For example, the proposed, revised climate justice definition and ToC could be used as a basis for developing a programme-level M&E framework, which could be used when designing future invitations to tender and made available through a CJF website. In addition, more could be done to ensure projects focus on issues of gender and vulnerability.

Finding 3. Communities in Malawi, Rwanda and Zambia are vulnerable to a large number of shocks, and CJF projects addressed many of them in a variety of ways

Community members from the Malawi case study projects mentioned a large number of climate shocks that affect communities, and in particular, the most vulnerable members, including flooding, droughts and dry spells, high winds, lake drying, shorter wet seasons, and increasing pest pressures (which can be weather related). These shocks affect communities by reducing crop productivity, damaging homes, and harming livelihoods like fishing. The focus of the CJF projects on energy, food/agriculture, and water, is well aligned with addressing these shocks. Energy projects, such as the Malawi Project 3, seek to address deforestation, which can exacerbate flooding and wind damage. Water and agriculture projects address drought and shocks affecting crop productivity. Some projects, however, implemented activities outside the scope of these three sectors, but which still had relevance to climate shocks in Malawi. For example, Malawi Project 2 introduced beekeeping as an alternative livelihood which could help households adapt to reduced or less reliable income from farming.

Finding 4. The CJF approach is well aligned with climate justice pillars, although it could go further

The evaluation analysed CJF projects to identify in which ways the CJF has contributed to climate justice to-date (refer to Figure 4). The evaluation found that - through the various interventions under its portfolio of projects – the CJF programme covers almost all aspects of climate justice, though not all aspects of climate justice are covered by all individual CJF projects. This finding supported the adaptation of the Climate Justice Interventions ToC into a programme-level ToC, which is proposed by the evaluator as the ToC for future rounds of funding.

Across the CJF portfolio, elements of the different projects seek to address all aspects of climate justice, however, no one project includes all intervention types. Some notable examples of projects with strong interventions across all three climate justice pillars are Malawi Project 1 and the Zambia Project, which incorporate all three pillars of climate justice across their work. As can be seen from Figure 4, in general, the CJF contributes strongly towards building climate justice capacity at multiple levels (procedural justice), facilitating meaningful participation in design and implementation (procedural justice), targeting climate change projects and interventions to the most affected peoples and areas (distributive justice) and empowering communities to engage in decision-making around climate change (transformative justice).

The CJF could go further by taking a systematic approach to ensuring all aspects of climate justice are integrated from the project design stage. In particular, innovative projects (especially mitigation projects) tended to be weaker on procedural and transformative justice and were more likely to be largely traditional development projects with elements of climate justice added on to meet CJF selection criteria. Round 2 and climate justice innovation project selection criteria only covered distributive justice elements (apart from the need for a local partner and an aim to improve governance), while insufficient documentation was available to assess the selection criteria for Round 1 and Malawi Project 1. Here, it should be noted that this broad definition of climate justice was only developed as part of this evaluation and was not the working definition utilised within Scottish Government at the time of project procurement. Project alignment with climate justice could be greatly improved by building in more specific project, selection, M&E and reporting requirements that cover all aspects of climate justice as defined by this evaluation.

The evaluator proposes that future CJF projects should align to the Climate Justice Interventions ToC and where possible, incorporate activities that contribute to all three pillars of climate justice. Although this contribution could (and should) vary depending on the type of intervention, requiring projects to demonstrate how they are ensuring participatory and inclusive project design and implementation processes (procedural justice), building capacity around climate change and climate justice (procedural justice), targeting projects to the most affected areas and people (distributive justice) and empowering communities to engage in climate change decision-making (transformative justice), as well as advocacy (transformative justice). In addition, there is an opportunity to strengthen the sharing of learnings across projects and climate justice stakeholders (procedural justice) and local ownership of projects (procedural justice), as well as ensuring all projects contribute to climate resilience of local communities (distributive justice). This is discussed in further detail below. In addition, for the CJF to have a transformational impact on climate justice more broadly, there is a need to support replication and scale-up of successful climate justice project approaches, as well as sharing lessons with stakeholders more broadly (as part of the grant managers role and by supporting local people to advocate for themselves through national and international processes). These lessons are discussed in more detail below.

Respondents in the online moderated platform were asked what they would like to see in future CJF funding rounds, including the impacts they think such programmes should seek to achieve. Below is a summary of the criteria suggested:

• Sharing knowledge and learning
  • Sharing lessons from project delivery to improve other projects (e.g. 'our involvement the community in this pilot project was not expansive enough in terms of impacting the community at large').
  • Regular Grant holder/Contract Holder meetings;
  • Sharing lessons nationally to raise awareness of project successes ('unless this message and practice goes across the country, the accomplishments are negligible and main goal unfulfilled').
• Importance of co-design at inception phase once the grant or tender has been awarded. ('[Malawi Project 1] benefitted hugely from having an inception period to co-design the programme with the community and it's not an opportunity I've seen available in other funding models so I feel strongly it should be kept in the CJF going forward' -online moderated platform respondent). A two-phased approach was suggested for the commercial tender model i.e. following an inception phase, a specification would be agreed based on the programme co-designed with communities. This is driven by the reflection that previous rounds 'struggled to adhere' to initial contracts as 'the original specification was so broad.'
• Localisation and partnership is valued and is seen as a critical success factor
• Advocacy should be included as it is 'important to support systemic change and fairness'
• Smaller organisations should be funded
• Alignment - projects should align with country's Nationally Determined Contributions
• Preservation of eco-systems (e.g. forests & natural habitats)
• Promotion of alternative energy sources for factories for example solar power to reduce use of hydro powered electricity.

Finding 5. Procedural justice is an important element of climate justice

Participants in the online moderated platform were very clear that participation with local communities is inherent to climate justice and key to a successful climate justice project. There were some challenges with regular engagement due to Covid-19. However, grant holders involved in the online moderated platform were confident that their participatory work was done to a high standard. This is supported by the QuIP analysis of the Malawi Project 1 and the Malawi Project 3 but also indicates a correlation between projects engagement with communities and stakeholders and their willingness to become involved in the evaluation. Learnings can therefore be drawn from this finding but it should not be assumed that all CJF projects have had similar experiences.

Some reflections on 'what works' when adopting a participatory approach with communities include:

• Approaching challenges with an objective of understanding how community members viewed the problem
• Regular meetings with community members
• Recruitment and identification of community members through trusted partners, volunteers and/or radio
• Participatory Vulnerability and Capacity Assessment methodology – in some communities this had already been prepared and the CJF project complemented existing plans
• Particular strategies to ensure the meaningful participation of women, for example 'female only forums'.

Learning points about participatory approaches include:

• Some confusion over the term 'youth.' In some cultures, once someone is married or has had a child they are no longer considered youth. This meant some people did not self-identify as 'youth.'
• Codesign was new for some partners who were used to more formulated programmes. For example, at District level and they asked the project to 'come back when you have designed the programme' but once the approach was explained, they were receptive.
• It takes times to build relationships and trust with the district the communities so having a long inception phase is important for successful codesign.

3.4.2 Coherence

When it comes to Coherence (how well the interventions and projects align with each other and Scottish Government policy and programmes), the evaluation found:

Evaluation Question 3: How well has there been coherence with local programmes, grass roots efforts and national/local policy objectives?

CJF projects tended to have stronger internal coherence than external coherence, although some projects were highly coherent with local efforts, especially when they worked closely with local stakeholders (Finding 6). Complementarities across the climate justice typologies are also important (Finding 7). In addition, there may be more opportunities for CJF projects to complement each other and leverage learning from other projects (Finding 8).

Evaluation Question 4: How coherent are the projects as a combined portfolio of the CJF?

The CJF portfolio of projects are fairly heterogeneous, although coherence is achieved through their focus on (mostly) climate change adaptation projects (Finding 9).

Evaluation Question 5: How coherent is the CJF with Scottish Government priorities and how they relate to CJ? What are the synergies between Scottish Government climate programmes more broadly and how do they collaborate and overlap?

The CJF is both well aligned with Scottish Government policies and has influenced these policies to better align with climate justice (Finding 10). Nevertheless, an opportunity exists to increase collaboration, buy-in, replication and scale-up of climate justice projects (Finding 25 below).

Finding 6. CJF projects tended to have stronger internal coherence than external coherence, although some projects were highly coherent with local efforts, especially when they worked closely with local stakeholders

In general, the CJF projects showed stronger internal coherence and weaker external coherence, although some projects were strong in both. For example, the Malawi Project 1 and Project 4 both had strong internal and external coherence, including learning from its own previous projects, working with NGOs and government stakeholders. The Malawi Project 2 had strong internal but weaker external coherence. In Rwanda, the Rwanda Project lacked external coherence as it had not engaged with national or local authorities except for formalities or where it required their inputs (e.g. it did not engage with the Rwanda Cooperatives Agency) and could have strengthened its internal coherence by undertaking more detailed preparatory assessments with the selected co-operatives prior to commencing work.

Nevertheless, among the Malawi projects examined in-depth, there were a number of examples of ways that projects appeared to address local efforts and priorities. For example, the Malawi Project 4 worked with the Water Resources Authority to support their efforts to improve water resources management in Malawi, including supporting their advertising campaign aimed at encouraging borehole drillers to follow required registration procedures; 40 drillers since came forward to register. This project also worked closely with local government for developing procedures to address poor quality boreholes, which can be a politically sensitive issue since powerful politicians or organisations may sponsor poor quality boreholes.

Finding 7. Complementarities across the climate justice typologies are important

Projects that addressed all three climate justice typologies were more effective at achieving climate justice, had greater impact and are more sustainable than those that focused only on one or two aspects of climate justice. Addressing multiple constraints and issues appears to do a lot more for a community than doing a single thing. For example, where transformative issues are addressed through advocacy training or building systems for integrating communities into innovative research and development but immediate distributive justice issues like absorptive and adaptive capacity are not addressed, it will likely be hard for communities to see significant change that is relevant to them (such as the Malawi Project 4). Likewise, distributive projects that take a more top-down approach (lacking in procedural justice) tended to experience more delays and less impact and provided benefits that were potentially less relevant to communities (such as two of the Malawi water projects and the Rwanda Project). Whereas projects such as Malawi Project 1, Malawi Project 2 and the Zambia Project, which took a more holistic approach tended to achieve stronger results, with local ownership indicating the potential for long-term sustainability of interventions.

Finding 8. There may be more opportunities for CJF projects to complement each other and leverage learning from other projects

Among the projects examined, there were limited mentions of working with other CJF projects. One exception was Malawi Project 4, which used national water point mapping done by the Scottish Government Hydro Nation work under CJF to identify areas with large numbers of non-functional boreholes. Sequencing projects that generate data, and then building on them to support further innovation, advocacy or interventions, could help maximise the impact of the CJF. Projects engaged in similar activities, including livelihood interventions or youth advocacy efforts, could also potentially learn from sharing lessons across ongoing projects, while lessons on building knowledge, local decision-making and advocacy capacity could be beneficial to all CJF stakeholders. In some instances, projects were asked to collaborate but competition between actors (that compete for funding) meant this engagement was unsuccessful. In other instances, projects learnt from other actors outside of CJF (e.g. Solar Grid learnt from South African experience), although the opportunity to share learnings across projects and CJF stakeholders was not formally supported by the programme. An opportunity exists for future phases of CJF to provide better support for cross-learning platforms and organic collaboration.

Finding 9. The CJF portfolio of projects are fairly heterogeneous, although coherence is achieved through their focus on climate change adaptation projects

The CJF projects focus on climate change adaptation, in particular water (Rounds 1, 2 and 3), as well as more recently resilience, agriculture and energy (including some almost purely mitigation projects). This provides some coherence to the portfolio, however there is significant heterogeneity within the portfolio, in terms of focus, approach taken, and the size of funding (Round 2 projects received £400,000 to £700,000 in funding, Malawi Project 1 receives £4.7m, while the CJF innovation projects received £100,000). The diversity of projects makes it more difficult to generalise findings from this evaluation to the programme as a whole.

Finding 10. The CJF is both well aligned with Scottish Government policies and has influenced these policies to better align with climate justice

The CJF is well aligned with Scottish Government policies and its influence can be seen in the reference to climate justice across Scottish Government policies. The Scottish Government sets out its ambitious response to the climate crisis in its Programme for Government (December 2020), stating that the CJF will continue to support communities in partner countries to become more resilient to climate change. The fund has been described as a 'world first,' and links in with Scottish Government's desire to seek a 'just transition' for a low carbon/net zero future, ensuring a just and sustainable world for current and future generations. This aligns with key themes within Scottish Government's work on climate change domestically, where there is a heavy focus on 'just transition' and 'justice' – a cross-cutting theme throughout various directorates as well as the green recovery and 'well-being', which are a focus of climate change policies. The CJF also aligns with the Scottish Government's policy on international development, which has key focus countries of: Malawi, Zambia, Rwanda and Pakistan. A new focus is that of tackling gender inequality in these partner countries.

3.4.3 Efficiency

In terms of Efficiency (how well CJF resources were allocated towards achieving climate justice), key findings include:

Evaluation Question 6: How efficient, for achieving climate justice, were the CJF partnerships and collaboration with national implementing partners, governments and stakeholders?

It would be beneficial for CJF projects to collaborate more with national stakeholders, where this was done, it appears to have improved project results (Finding 11).

Evaluation Question 7: To what extent are the CJF governance and management arrangements consistent with achieving its strategic climate justice objectives?

Flexibility in the funding mechanism was lauded for allowing adaptation and learning (Finding 12). However, delays in funding or approvals - partially due to the chain of CJF project implementers - were cited as a challenge among some projects (Finding 13). In addition, more innovative projects may require more time and adaptation to get things right (Finding 14).

Finding 11. It would be beneficial for CJF projects to collaborate more with national stakeholders; where this was done, it appears to have improved project results

As noted under Finding 6, many CJF projects did not have sufficient external coherence and engagement with national stakeholders could be improved upon in future phases of the programme. However, in the few examples where external engagement did occur, it appears to have improved project results. For Malawi Project 1 while the programme was credited with directly supporting interventions, many respondents also mentioned support from government and local community structures, and often described coordination or support for government extension or local community committees coming through Malawi Project 1; while for the Malawi Project 3, relatively few examples of engagement were cited, but other projects by the project implementer, were viewed as positively contributing to increased engagement.

Finding 12. Flexibility in the funding mechanism was lauded for allowing adaptation and learning

On the one hand, several projects mentioned that flexibility in CJF funding and budgeting contributed to project success. For projects such as Malawi Project 1, which employed a participatory approach to selecting interventions, flexibility allowed work plans and budgeting to adapt to the activities selected through that process. Flexibility also enabled projects to adapt to challenges such as COVID-19 in their implementation.

Respondents in the online moderated platform also noted that they had a positive experience applying to the CJF. Respondents have positive feedback on the application process. The CJF was especially welcome due to the following:

• For some projects, the application process mobilised partners to work together and align and provided an opportunity to resource and implement a localisation approach (i.e. implementing a project with local partners), while for others, they had a project / partnership ready (informed by previous work) and CJF criteria was a good 'fit'
• One applicant had observed projects in other regions and saw this as an opportunity to adapt for their area
• The focus on innovation allowed projects to take a risk and try something new, which was welcomed
• Respondents welcomed the ethos of a human-centred, holistic approach of the CJF and there was a general feeling of excitement due to the recognition of climate injustices ('the shift to focusing on climate justice (not just climate change) was an important one').

Finding 13. Delays in funding or approvals - partially due to the chain of CJF project implementers - were cited as a challenge among some projects

On the other hand, many projects mentioned delays in receiving funds, or in receiving approvals for things such as extensions, as contributing to delays in project work and results. Getting clarity on reporting systems was also mentioned as a challenge, indicating the benefit of providing additional guidance and clarity to project implementers on the Scottish Government funding models and arrangements. As a result of these issues, projects have had to adapt their work plans and find ways to deliver activities within shorter periods. Some partners, particularly local organisations acting as sub-contractors to the lead organisation, also mentioned that their organisations had to finance activities and then get reimbursement, which can be a challenge for small, local organisations. These delays and challenges are likely to be a result of a combination of a long funding chain (Scottish Government to the grant manager to the Scottish partner to the implementing partner, potentially to a local partner), which caused delays in disbursements and potentially introduced costs in the transfer of funds, combined with short reporting cycles (many local implementing partners were required to deliver and report on a quarterly cycle, which meant a two-month delay in receiving funds could result in one-third of the time available for delivery) and budget cycles that did not allow automatic rolling over of funds between financial or programme years.

However, the covid pandemic was the biggest cause of delays (particularly due to lockdowns preventing staff from accessing project sites and working with beneficiaries in person), as well as challenges with flooding and the resultant reduced road access. Nevertheless, uncertainty over timing of activities would constrain national implementing partners' ability to plan for and work around factors that could be anticipated to cause delays (rainy season, election periods etc.). In some cases, projects noted the positive impact of working with local suppliers, which reduced costs, improved certainty of supply and protected them from exchange rate risks (for example, the Rwanda Project Buhanga Cooperative).

Finding 14. More innovative projects may require more time and adaptation to get things right

Malawi Project 1 was notable in that project participants credited the project with creating significant positive change in the community (such as, less vulnerability to various shocks, more income, more food security and having a wealth buffer against shocks). The programme included some innovations, including the methodology of its participatory approach to selecting interventions; however, the interventions themselves were not necessarily innovative—they were interventions that communities were familiar with and were confident worked.

In contrast, the Malawi Project 2 took a more innovative approach, introducing new technology in the form of an artesian well system that was aimed at leveraging the water source for drinking water, fish farming, and irrigation. Unfortunately, the system has required some adaptations and was not functional, with effects available for demonstration, at the time of the evaluation team's site visit.

Similarly, the cook stove prototype designed under the Malawi Project 3, despite undergoing improvements with input from the community, still appeared to face some challenges, including not heating quickly enough. The experience of these projects emphasise that projects that involve innovation may require several iterations of experimentation, evaluation, and adaptation in order to achieve an effective technology in a challenging context. In addition, successful innovations would benefit from further support for scale-up and roll out (i.e. the Zambia Project) and/or support to link to markets (i.e. the Malawi Project 3). These projects had considerably smaller timeframes for delivery, with three of those announced in 2019 having an implementation period of only 16 months.

Respondents in the online moderated platform were invited to state the optimum time duration of a CJF and expectations varied from 18 months to four to five years, indicating the different needs of different types of projects. Respondents were also asked to comment on the extent to which innovation needs longer time frames to implement. There were mixed views on this with some saying that innovation can be done relatively quickly (e.g. the stove piloted in Malawi took less than 12 months) and others describing it as an 'iterative process' often in the context of changing conditions and circumstances.

3.4.4 Effectiveness

Key findings on Effectiveness (alignment of interventions with its objectives) are outlined below:

Evaluation Question 8: How well were the most affected people (vulnerable, women etc) targeted and given voice in CJF implementation (at the project and programme level)?

CJF projects tended to target the most vulnerable areas but not always the most vulnerable peoples (Finding 1), targeting of beneficiaries under innovation projects tended to focus more on 'technical aspects' rather than taking a participatory approach to identifying local needs. Effectiveness varied by project, and was influenced by project design, context, and timeline (Finding 15). Understanding of climate justice concepts at the local level was often limited and may be relatively broad compared with international definitions (Finding 16). In addition, the focus of CJF projects means they may require longer time horizons to deliver results for communities (Finding 17).

Evaluation Question 9: How do projects in the CJF portfolio as a whole incorporate learning?

Project partnerships contributed to knowledge sharing, while collaboration with project stakeholders strengthened climate justice (Finding 18). There was some evidence of replication and scaling as a result of CJF projects (Finding 19).

Evaluation Question 10: How effective is the Scottish Government at leveraging lessons from the CJF to increase support and delivery of CJ?

An opportunity exists to increase collaboration, buy-in, replication and scale-up of climate justice projects (Finding 25 below).

Finding 15. Effectiveness varied by project, and was influenced by project design, context, and timeline

Through the QuIP analysis, climate justice was measured across eight domains:

• Distributive justice:
  • Adaptive capacity
  • Absorptive capacity
• Procedural justice:
  • Equity (equality and most vulnerable people's resilience capacity)
  • Inclusion (participation of most vulnerable)
  • Engagement (engagement of national and international stakeholders on issues that matter to the community)
• Transformative justice:
  • Community action (capacity of community to address its challenges related to resilience)
  • Community advocacy (capacity of community to advocate for itself with government or other national stakeholders
  • Knowledge (understanding of the impact of climate change on communities).

The four Malawi projects that the evaluation looked at in detail had very different designs, and different levels of effectiveness against this evaluation's QuIP criteria. Malawi Project 1 had by far the largest budget, longest implementation period and included a large range of activities targeting all three pillars of climate justice. It could therefore invest heavily in engagement with communities, and the results of the QuIP suggest that it was successful at improving climate justice in a number of domains. Of the four Malawi deep dive projects, Malawi Project 1 was the only one that fully achieved all aspects of climate justice. For the three climate justice innovation projects, results were mixed, with only some improvement achieved across some of the QuIP domains (most achievements were in transformative justice, with mixed results for distributive and procedural justice) and no improvements noted in some areas for some projects. For example, the equity situation under the Malawi Project 4, which sought to support community advocacy for water accountability, was 'unchanged/worsened' due to increasing climate shocks (with potential positive impacts from the project delayed due to covid-19) during the 22-month grant, while under the Malawi Project 2, reduced support from other NGOs engaged in the area and CJF project activities only reaching a limited number of people had seen inequity increase over the project period, although participation of women in community activities, decision-making and leadership was reported to have improved. Whereas, under the Malawi Project 3, adaptive capacity was determined by the evaluation to be 'unchanged/worsened' due to continuing external deforestation pressures, despite improvements resulting from the 16-month project. However, care should be taken in interpreting these results, as Malawi Project 1, a four-year programme, compared to the much smaller £100,000 total, two to three-year innovation projects. Additionally the QuIP methodology meant we were assessing projects against criteria that might not have been part of the original design of the project.

The smaller projects, on the other hand, were much smaller, with more limited scope. Despite this, some of them appear to have been effective at achieving some elements of climate justice. For example, the Malawi Project 4 was credited by government stakeholders with improving government capacity, and it achieved national media attention, raising awareness of water rights and the problem of poor-quality boreholes. The Malawi Project 3 contributed to strong local partnerships and a model for incorporating communities into technology research that has the potential to make technology development more inclusive and more effective. However, both projects were limited in that direct benefits to communities – largely distributive justice outcomes – may require years to be realised and the 'scaling up' and replication of the innovative approaches and technologies trialled through the original grants.

Finally, while the effectiveness of the Malawi Project 2 was limited by technical challenges with the installed water systems (which were not working at the time of the evaluation site visit), the evaluation views the project as having good potential to transform livelihoods, particularly as components of the project not dependent on the water system, such as beekeeping, have been effective for those participating in them.

The QuIP results found:

• Malawi Project 1: project took a coordinated approach with government partners and beneficiaries credit the project with improving equity and inclusion for a variety of vulnerable groups, and with improving community advocacy. Positive changes in the area of increased knowledge of the impact of climate change in community are mostly focused on community awareness, but some increased awareness among duty-bearers is also mentioned. The QuIP results for Malawi Project 1 were striking in that nearly every respondent viewed the situation as improving over the past four years, in nearly every domain. Respondents cited a large range of positive changes across multiple domains, and all but one respondent at least partially attributed these changes to interventions by the local the Malawi Project 1 Implementing Partner.
• Malawi Project 2: Overall, the QuIP results indicate that project beneficiaries attribute some positive changes in the area of absorptive and adaptive capacity, inclusion, engagement, and community capacity to interventions by the Implementing Partner. However, these changes appear to be limited, both in their scope and in the number of community members benefitting from them. Livelihood interventions not dependent on the system, e.g. inclusion of women and the poor in activities, and increased engagement with stakeholders are the elements of the Project most often linked with positive changes. Assessments of overall change for the community during the project period by the respondents across the domains were highly mixed. The domains most likely to have reported positive overall changes were inclusion, community capacity for action and advocacy, and knowledge of climate change impact of the community.
• Malawi Project 3: Overall, the QuIP results indicate that project participants attribute some positive changes to the project and activities implemented by project partners. One respondent specifically mentioned the biogas stove prototype as contributing to less tree cutting, and training by one of the implementing partners (as part of a separate project) were attributed with increasing inclusion in the community. However, especially in the areas of adaptive and absorptive capacity, the positive impact of the prototype stove which was designed and trialled during a 16-month grant appear to be overshadowed by urgent and substantial challenges with worsening climate conditions and deforestation. Overall, the respondents viewed most of the domains as either unchanged or improved. One exception was adaptive capacity, where some respondents viewed the domain as having worsened over the past four years. This was largely driven by perceptions of worsening weather shocks and greater difficulty obtaining fuelwood due to deforestation. In the domains of inclusion, engagement, community capacity to take action on climate change, and knowledge of community challenges due to climate change, respondents were roughly split half and half between seeing the situation as unchanged and improved.
• The Malawi Project 4: The project had not yet entered the community engagement phase of its work at the time of the evaluation, as its original duration of 16 months had been extended to 22 months to conclude by 30 September 2021 due to extraordinary issues (primarily the Covid-19 pandemic) causing delays in implementation, which is reflected in the results. The project had done some small activities providing direct support in some communities, such as helping communities obtain locks for their water pumps or relocate latrines to avoid water contamination, but the bulk of its work with communities was delayed due to covid-19. In general, respondents reported few changes in their community over the past four years in the domains of climate justice included in the survey. Where changes were noted, negative changes generally reflected increasing climate shocks, while positive changes generally reflected government extension trainings. Respondents noted that their community has very little engagement with stakeholders. They noted that NGOs have not worked in the community for quite some time, and government engagement and support is inconsistent. At the end of the interview, when asked specifically about the implementing partner's work in the community, some of the respondents were aware of it, but did not see it as contributing significantly to change in the community yet. These observations suggest that this community could benefit from the support of a project, supporting the community in advocating for itself to receive projects and support, including support beyond better borehole management.

In terms of the broader CJF portfolio, effectiveness also varied across the other CJF and Round 2 projects^[19] with the focus on innovation sometimes detracting from project's effectiveness in achieving climate justice aspects.As noted under Finding 12, there are differences between innovative versus core climate change projects that effect the most appropriate design, implementation and M&E approach. Of the deep dive projects, two innovative projects were less effective in achieving all aspects of climate justice as defined by this evaluation, while more traditional development type interventions (which have been tried and tested in the local context and are often better understood and trusted by local participants) were more effective.

The covid pandemic impacted effectiveness, with projects forced to pivot due to delays and challenges reaching project participants. While other non-implementation challenges, also influenced effectiveness in some instance (such as staff turnover in the Rwanda Project and problems sourcing raw waste materials for the biodigester in the Zambia project). Many of these challenges could not have been foreseen through risk management processes, although early engagement with communities through a more participative design process in Zambia should have identified the challenges with sources waste materials. However some could not, 2019 innovation projects had a small implementation period of only 16 months to trial their innovations.

Finding 16. Understanding of climate justice concepts at the local level was often limited and may be relatively broad compared with international definitions

Participants in CJF projects, particularly the Malawi Project 1(which delivered a wide range of interventions), described a large range of positive outcomes as a result of CJF activities. In many cases, the outcomes were in line with conventional resilience project outcomes, such as reducing vulnerability to flooding by engaging in watershed management projects, or reducing vulnerability to dry spells through irrigated farming or alternative livelihoods. However, positive impacts were sometimes less expected. For example, where solar kiosks had been introduced through a innovation project, community members described children being able to get haircuts for school, or the community being able to watch football together. For solar PV, beneficiaries reported improved safety and security (due to lightning at night), increased productivity from ability to work (and study) after dark, as well as benefits from being able to charge mobile phones. Such outcomes may represent aspects of life that are highly relevant to welfare in communities vulnerable to climate change, but may not be directly connected to climate shocks. In the case of the Rwanda Project, support with wastewater facilities has had the unintended benefit of reducing the existing conflict between the coffee cooperative and local community because of pollution, as well as reducing competition for water.

Participants in the online moderated platform reported that there is not always a strong local understanding of some climate justice concepts amongst their communities. One respondent shared some examples of things they have done to improve local understanding including working with university students to develop radio jingles and theatre workshops with community members. Respondents were also keen to stress the role of advocacy work as part of climate justice, highlighting the need for systemic change and promotion of fairness as key parts of climate justice.

Finding 17. The focus of CJF projects means they may require longer time horizons to deliver results for communities

A focus on procedural and transformational pillars of climate justice may require longer time horizons to deliver results for communities, as do innovation projects. Malawi Project 1 was notable in that project participants credited the project with creating significant positive change in the community. The project included some innovations, including the methodology of its participatory approach to selecting interventions; however, the interventions themselves were not necessarily innovative—they were interventions that communities were familiar with and were confident worked.

Projects such as Malawi Project 4 and Malawi Project 3, which had a greater focus on achieving procedural and transformational justice through changes to systems for rural water management and inclusive models for energy technology development, showed positive signs of impact within those systems. Government stakeholders credited Malawi Project 4 with increasing knowledge of water rights and capacity among officials, and Malawi Project 3 demonstrated that communities can be effective partners in applied technology research. However, as relatively short-term innovation pilot grants, these projects had limited impacts on community members themselves within the timeframe of this evaluation; the majority of community members for both interventions reported little overall change in domains such as adaptive and absorptive capacity, equity and inclusion, engagement, and community capacity for action and advocacy. This reflects the design and circumstances of both projects –Malawi Project 3 was designed to be a technology pilot project, while Malawi Project 4 includes a community engagement component that has not yet been implemented due to covid-19. However, these results also underscore how systems change may take considerable time. For example, the sequencing of Malawi Project 4 was likely done well, with engagement with government laying the groundwork for more effective advocacy from the community. However the benefits for the community will, as a result, be achieved with a lag.

It should also be noted that strength of evidence with regard to impact is weaker for the projects whose strategies focus on longer term impacts, as many of those impacts have not yet occurred at the time of this evaluation and speaks to the need for longer time horizons to deliver results for communities

Finding 18. Project partnerships contributed to knowledge sharing, while collaboration with project stakeholders strengthened climate justice

Respondents in the online moderated platform reported positive relationships and interest from stakeholders at the district authority level. CJF project stakeholders reported a number of benefits from this relationship, including:^[20]

• Improved knowledge of delivery partners on the extent and use of natural resources
• A high level of appreciation (amongst district authorities) for the CJF-funded projects in terms of its technologies and approach
• Helped to empower community members to analyse their vulnerabilities and take action
• Helped identify areas where waste needed to be recycled
• Changed actions by district authorities. For example, a mapping study revealed a negative impact of mining activities. The mining activities were consequently suspended by district authorities until a clear plan to rehabilitate mining sites is submitted and approved. The repair of water taps was also integrated into the sector plan following advocacy from the same committee which presented critical issues faced by their village.

There were no reported barriers to engaging with district authority stakeholders. Respondents reported setting up meetings and regular communication, although some field trips were unable to go ahead due to Covid-19. Webinars and field visits were reported as being successful mechanisms for engagement.

CJF project stakeholders also reported positive relationships with NGOs and national governments, although there were few tangible examples of the impact of these relationships.

Finding 19. There was some evidence of replication and scaling as a result of CJF projects

In some instance, project successes enabled project implementers or communities to leverage additional funding to replicate project activities, while in others, project benefits spilt over to nearby communities (both directly and through unintended impacts). Rwanda Project wastewater treatment at Kopakaki was so inspiring in its positive impact on water availability and community relations, that it was used to leverage additional funding which was used to provide wastewater treatment to a further six other cooperatives who also benefited from the CJF supported technology introduction. Previously, water and air pollution due to wastewater released from the washing stations and contaminating rivers in the neighbouring the coffee washing station, frequently created conflicts within the surrounding community and resulted in payment of fines due to environmental degradation. The wastewater treatment plant, helped by improving the relationship of the cooperative with the surrounding community. Farmers gardens are no longer affected by the wastewater, improving food security and contributing to the conservation of an entire natural eco-system.

In regard to Solar PV and wastewater treatment facility supply chains, the research commissioned through the Rwanda project highlighted questions over the sustainability of the solar product supply chains, with panels sourced from China. Most cooperatives cannot afford to adopt green or clean infrastructural improvements without grant funding. Donor attitudes toward certain 'green' innovations are fleeting, and therefore innovative blended financial approaches are required. Wastewater treatment facilities appeared to be more economically beneficially than solar PV units for cooperatives, leading to quicker environmental and economic returns (e.g., through certifications). The current economic landscape for Rwandan cooperatives creates a bottleneck to green development for smaller cooperatives.^[21]

3.4.5 Impact

In relation to climate justice Impact (the achievement of climate justice outcomes), the evaluation found:

Evaluation Question 11: How have the CJF projects and programme as a whole contributed to climate justice outcomes?

Distributive, procedural, and transformative justice were often complementary in projects, and projects that focused on all three approaches were highly successful (Finding 20). However, many communities in Malawi are facing overwhelming challenges. Impact needs to be very significant for local communities to perceive a positive change in the face of worsening shocks (Finding 21).

Evaluation Question 12: Looking forward, what are the emerging strengths, weaknesses, constraints and opportunities in managing and implementing the CJF and can these inform a potential future programme phase?

CJF has catalysed a shift towards climate justice (Finding 22). Nevertheless, an opportunity exists to increase collaboration, buy-in, replication and scale-up of climate justice projects (Finding 25 below). In addition, engagement with private sector firms was limited and likely to be challenging for grants-based projects to achieve (Finding 23).

Finding 20. Distributive, procedural, and transformative justice were often complementary in projects, and projects that focused on all three approaches were highly successful

Malawi Project 1 was notable in that community participants consistently credited the project with contributing to significant positive changes across multiple domains over the timeframe of the programme, including adaptive and absorptive capacity, equity and inclusion, stakeholder engagement, and community capacity for action and advocacy. Malawi Project 1 had a strong focus on all three pillars of climate justice and demonstrated how focusing on the three pillars together can create multiplicative effects. First, the strong focus on procedural justice through participatory approaches to programme design appears to have resulted in both increased engagement of vulnerable community members, with programme participants specifically crediting the participatory process with contributing to inclusion and community capacity to understand and address climate change, as well as with the project delivering interventions and activities that serve the community, particularly the most vulnerable.

Activities that focus on delivering interventions to communities can also create opportunities for broader engagement. Participants in both Malawi Project 1 and Malawi Project 2 cited increased contact with government staff like extension workers and with NGOs visiting the community as a result to implementation of the programmes; this contact was described as presenting opportunities for engagement.

Projects that complement distributive justice interventions focused on meeting immediate needs with interventions focused on building local capacity for governance and advocacy can also be highly successful at delivering transformative justice impacts. For example, one participant in Malawi Project 1 described how local committees in their community now had the capacity to collect data, produce reports, and use them in advocacy with government as a result of support from the Malawi Project 1's Implementing Partner, the local Malawi Project 1's implementing partner. Working with local partners to deliver such interventions can also serve as a means of building local organisation capacity, enabling them to go on to access additional resources.

CJF project stakeholders also supported the concept of three climate justice typologies as underpinning future climate justice projects. Participants in the online moderated platform strongly felt that all three pillars – procedural, transformative and distributive – need to be present for a climate justice project to be successful. And all respondents felt familiar with the language and terminology used to describe these justice categories, even for those for whom it was new. Respondents also recognised the complementarity of the three typologies, articulating a strong desire for inclusion, participation and cohesion. An additional typology of justice was suggested by one respondent: 'cohesive - unity amongst everyone involved.'

Finding 21. Many communities in Malawi are facing overwhelming challenges. Impact needs to be very significant for local communities to perceive a positive change in the face of worsening shocks

This means to be able to detect project success: 1) projects have to be designed to achieve a lot of immediate impact in communities or 2) evaluation and metrics have to be designed well to measure longer-term, systemic changes that there is a strong reason to believe will lead to transformative change.

Finding 22. CJF has catalysed a shift towards climate justice

Some online moderated platform respondents were familiar with climate justice before the project. For others it was a new concept: 'I had never considered […] what climate change is all about. Contributors, practices that cause it and the ecosystems affected, until this project' (online moderated platform respondent).

Respondents reported that since the CJF-funded project, they have done more climate justice work. For some, this has been an evolution of the funded project (e.g. 'I'm involved in a study to see what sort of cooking energy is being used and reasons of such a choice in middle class families in Malawi'). For others, this includes reframing previous work around climate change and focusing on the justice element. Some respondents also credit the CJF with accelerating and/or catalysing a shift (from climate change to deeper consideration of climate 'justice') that was already happening.

Scottish-based respondents noted a shift in narrative from climate change to climate justice, catalysed by the CJF. They note that Round 1 focused on adaptation to the impact of climate change but subsequent rounds focused on climate literacy, advocacy and community participation. For example, one Round 1 project implementer noted (through the online survey) 'Our project didn't connect the work being done to address the impact of climate change with advocacy in country, this would have amplified the voices of the community members and could have been key to more discussions around climate change and climate justice in-country and internationally. Linking advocacy in-country with international advocacy would deepen the 'justice' component of the CJF projects, and ensure projects were taking a multi-pronged approach to address a complex challenge.'

Finding 23. Engagement with private sector firms was limited and likely to be challenging for grants-based projects to achieve

Private sector engagement was limited in all of the Malawi projects examined in detail in this evaluation. The Malawi Project 4 worked towards building private sector awareness of borehole drilling regulations, and Malawi Project 1 supported communities in engaging with local businesses that were causing pollution in the community. The Malawi Project 3 had some engagement with the private sector regarding the cook stove prototype, but covid-19 restrictions limited meetings with prospective partners. Nevertheless, several projects worked closely with the private sector, for example the Rwanda Project is a private sector project, as was one of the Malawi agriculture projects.

3.4.6 Sustainability

In terms of CJF Sustainability (the extent to which climate justice outcomes are expected to endure), the evaluation found:

Evaluation Question 13: To what extent did project implementing partners and/or beneficiaries assume ownership and responsibility for the project preparation, implementation, and sustainability?

Projects have been effective at achieving buy-in from communities and government partners, particularly when communities see benefits (Finding 24).

Evaluation Question 14: To what extent has Scottish Government leveraged the CJF to strengthen CJ collaboration, buy-in, replication and scale-up?

CJF has catalysed a shift towards climate justice (Finding 22 above), although an opportunity exists to increase collaboration, buy-in, replication and scale-up of climate justice projects (Finding 25).

Finding 24. Projects have been effective at achieving buy-in from communities and government partners, particularly when communities see benefits

Where projects have produced positive impacts for communities, especially around livelihoods and incomes (distributive justice), respondents have been highly optimistic about sustaining those activities. This was particularly marked among Malawi Project 1 participants, but the Malawi Project 2 participants also mentioned they plan to carry on fish farming activities started under the project. Establishing local management committees and involving local government officials such as extension officers (procedural justice), are two mechanisms commonly mentioned as contributing to support for continuing activities after project support ends.

Another mechanism for achieving sustainability is providing direct capacity building for government or other local institutions. For example, the Malawi Project 4 worked with local officials to build capacity for borehole evaluation and management, and awareness of water rights.

Strength of evidence for this finding is necessarily limited by the timing of this evaluation, as the projects examined were either still underway or recently completed, so sustainability was inferred from buy-in and participant expectations rather than directly observed.

The use of local partners helped projects establish community buy-in, which was also enhanced by financial incentive for buy-in. Improving access to water and water rights is important to communities and enhances buy‑in. On the other hand, external stakeholder engagement and buy-in was weaker across the projects. In addition, some projects seek stakeholder buy-in at start-up but fail to maintain these relationships and collaboration, which is an important aspect of procedural justice and supports long-term sustainability.

Participants in the online moderated platform agreed that partnership working is both an important critical success factor for CJF and a beneficial outcome.

Finding 25. An opportunity exists to increase collaboration, buy-in, replication and scale-up of climate justice projects

The CJF is a 'first of its kind', government-led programme focused on achieving climate justice. However, the programme does not have a learning and communications component to support the dissemination of lessons and leveraging of additional support for climate justice. Many projects have developed learning reports and produced papers for international conferences. The climate justice innovation projects, in particular, have a learning focus and are required to report on learnings from the innovative approaches they have trialled, but there is no systematic process for sharing these lessons. Sharing of lessons from the CJF has occurred on an adhoc basis (e.g. the grant manager sharing reports between grantees) but there is no CJF platform for sharing results or lessons, nor a CJF website.

Many CJF stakeholders referred to the benefit of not only sharing project outputs but also creating a community of practice around climate justice through semi-regular learning events. These could be organised and hosted through the grant managers and would provide an opportunity to learn about both innovative technical approaches to respond to climate change (distributive justice), but also approaches and lessons on working collaboratively with communities and broader stakeholders (including NGOs and government stakeholders) in project design and implementation (procedural justice), as well as building local capacity and institutional structures for decision-making and advocacy around climate change and climate justice.

The CJF had begun engaging with project stakeholders prior to covid-19, however these activities have been put on hold as the programme and projects to pivot activities in response to the multiple challenges presented by a global pandemic. Nevertheless, the event was well received and stakeholders who attended it referred to it as an example of the type of engagement that would be welcomed by CJF implementers.