Climate Justice Fund evaluation: summary report

Executive Summary

Introduction

The Climate Justice Fund Intervention (CJF) was established by the Scottish Government in 2012 to help tackle the effects of climate change in Malawi, Zambia and Rwanda. Since its inception it has supported 31 climate justice projects and programmes through three separate calls-for-proposals: Round 1 (2012-2015), Round 2 (2014-2016) and Round 3 (2016-2021), which includes the larger Climate Challenge Programme Malawi (CCPM) and the Climate Justice Innovation Fund (CJIF) grants programme.

NIRAS-LTS was contracted by the Scottish Government to undertake this endline evaluation of the first three rounds of CJF funding to support learning and inform future phases of work. With the CJIF having concluded in March 2021, and the CCPM due to finish in September 2021, this evaluation provides a timely opportunity to take stock of what has worked, why and for whom. The objective of this evaluation is to 'assess the effectiveness of the CJF in delivering climate justice objectives and appraise the programme's achievements to-date. The findings will help inform the next phase of the CJF so that it remains influential.' The evaluation design is based around the Organisation for Economic Cooperation and Development's Development Assistance Criteria (OECD-DAC)^[1]: relevance, coherence, effectiveness, efficiency, impact and sustainability. Fourteen evaluation questions, covering both the project and programme-level, were developed against each of these criteria. The evaluation draws on a mix of qualitative and quantitative data from a literature review, project and programme documentation and interviews with programme stakeholders, drawing on principles from the Qualitative Impact Assessment Protocol (QuIP)^[2] and data synthesis, to respond to the evaluation questions and to develop a climate justice Theory of Change (ToC).

Climate Justice Definition and Theory of Change

During the inception phase, a Climate Justice Pathways Theory of Change (ToC) (Figure 1) was developed with the Scottish Government through an interactive workshop on the 17 June 2021. This established three interlinked pathways to change in line with the three distinct pillars of climate justice:

• Distributive Justice relates to equal access to, and sharing of resources and benefits. In Climate Justice definitions this includes both access to resources and benefits and equitable sharing of the costs of responding to climate change;
• Procedural Justice relates to transparent, fair and equitable decision-making processes;
• Transformative Justice relates to structural inequities and focuses on mainstreaming understanding of climate justice issues, as well as building capacity.^[3]

Taking this climate justice pathways ToC model as a starting point, a ToC specifically designed for the evaluation of the CJF programme (which we refer to as the Climate Justice Interventions Theory of Change), was developed iteratively with the Scottish Government and CJF stakeholders as part of the evaluation (Figure 2 below). This was used to a) evaluate the interventions supported under previous phases of the CJF and can possibly b) inform the development of future interventions.

The three interlinked climate justice pillars incorporate both the approach to project implementation (procedural justice), as well as the types of interventions supported by a project (distributive and transformative justice). As part of the Inception Phase, a list of potential interventions that can potentially support the achievement of the outputs and outcomes under the three climate justice pillars were explored and elaborated. For example, participatory processes and needs assessments (procedural justice) support the selection of interventions (distributive justice), the success of these interventions is supported by strengthening local institutions (transformative justice) and engagement with stakeholders, including government extension workers who support farmers in through advice and capacity building services (procedural justice). Furthermore, by building communities' capacity to make decisions around climate change and supporting community members to advocate for their community's needs and rights (transformative justice), projects can support both the sustainability of project outcomes, as well as leveraging additional support by communities for new interventions or scale-up, and replication by other actors.

The evaluation analysed CJF projects to identify the ways in which CJF has contributed to climate justice to date.

Findings

The evaluation findings synthesise the qualitative and quantitative data from the document reviews and key informant interviews around a set of evaluation questions. The evaluation questions were designed to respond to the OECD-DAC good practice evaluation criteria (relevance, coherence, effectiveness, efficiency, impact and sustainability) at both the CJF project and programme-level.

In terms of Relevance (are the interventions designed to deliver climate justice), the key findings are:

• Evaluation Question 1:^[4] To what extent are the CJF projects relevant to climate justice? How well focused on existing inequalities (such as wealth disparity and discrimination based upon gender, age, disability or indigenous status) were the design of the projects?

CJF projects tended to target the most vulnerable areas but not always the most vulnerable peoples. Targeting of beneficiaries under the innovation projects tended to focus more on 'technical aspects' rather than taking a participatory approach to identifying local needs. Nevertheless, many CJF projects had a strong understanding of, and focus on, climate justice, although some projects had more of a standard development focus. In interpreting the results, it should also be noted that communities in Malawi, Rwanda and Zambia are vulnerable to a large number of shocks (including flooding, droughts and dry spells), and CJF projects addressed these shocks in a variety of ways. For example, many projects improved access to water for drinking and/or irrigation, while others supported crop diversification to improve resilience, or clean energy solutions to improve productivity and reduce reliance on decreasing forest-resources.

• Evaluation Question 2: To what extent is the CJF approach aligned with climate justice pillars (distributive, procedural and transformative justice)?

The CJF approach is well aligned with the climate justice pillars, although it could go further by increasing focus on procedural justice.

When it comes to Coherence (how well the interventions and projects align with each other and Scottish Government policy and programmes), the evaluation found:

• Evaluation Question 3: How well has there been coherence with local programmes, grass roots efforts and national/local policy objectives?

CJF projects tended to be better at learning from their own experience and aligning projects with local needs (stronger internal coherence) than aligning project interventions with national policies and engaging with government stakeholders (weaker external coherence). Some projects were highly coherent with local efforts, especially when they worked closely with local stakeholders. In addition, there may be more opportunities for CJF projects to complement each other and leverage learning from other projects.

• Evaluation Question 4: How coherent are the projects as a combined portfolio of the CJF?

The CJF portfolio of projects are fairly heterogeneous, although coherence is achieved through their focus on (mostly) climate change adaptation projects.

• Evaluation Question 5: How coherent is the CJF with Scottish Government priorities and how they relate to CJ? What are the synergies between Scottish Government climate programmes more broadly and how do they collaborate and overlap?

The CJF is both well aligned with Scottish Government policies and has influenced these policies to better align with climate justice. Nevertheless, an opportunity exists to increase collaboration, buy-in, replication and scale-up of climate justice projects.

In terms of Efficiency (how well CJF resources were allocated towards achieving climate justice), key findings include:

• Evaluation Question 6: How efficient, for achieving climate justice, were the CJF partnerships and collaboration with national implementing partners, governments and stakeholders?

It would be beneficial for CJF projects to collaborate more with national stakeholders, where this was done, it appears to have improved project results. For example, by building capacity and coordinating with government agricultural extension workers to improve the level of ongoing support available to local communities.

Evaluation Question 7: To what extent are the CJF governance and management arrangements consistent with achieving its strategic climate justice objectives?

Flexibility in the funding mechanism was lauded for allowing adaptation and learning. However, delays in funding or approvals - partially due to the chain of CJF project implementers - were cited as a challenge among some projects. In addition, more innovative projects may require more time and adaptation.

Key findings on Effectiveness (alignment of interventions with its objectives) are outlined below:

• Evaluation Question 8: How well were the most affected people (vulnerable, women etc) targeted and given voice in CJF implementation (at the project and programme level)?

CJF projects tended to target the most vulnerable areas but not always the most vulnerable peoples, targeting of beneficiaries under the innovation projects tended to focus more on 'technical aspects' rather than taking a participatory approach to identifying local needs. Effectiveness varied by project, and was influenced by project design, context, and timeline. Understanding of climate justice concepts at the local level was often limited and may be relatively broad compared with international definitions. In addition, the focus of CJF projects means they may require longer time horizons to deliver results for communities.

• Evaluation Question 9: How do projects in the CJF portfolio as a whole incorporate learning?

Project partnerships contributed to knowledge sharing, while collaboration with project stakeholders strengthened climate justice. There was some evidence of replication and scaling as a result of CJF projects.

• Evaluation Question 10: How effective is the Scottish Government at leveraging lessons from the CJF to increase support and delivery of CJ?

An opportunity exists to increase collaboration, buy-in, replication and scale-up of climate justice projects. For example, building in systematic, cross-programme learning and communications component to future CJF phases to ensure lessons are shared from this innovative fund.

In relation to climate justice Impact (the achievement of climate justice outcomes), the evaluation found:

• Evaluation Question 11: How have the CJF projects and programme as a whole contributed to climate justice outcomes?

Although results vary across projects, CJF has achieved climate justice impact on-the-ground. Distributive, procedural, and transformative justice were often complementary in projects, and projects that focused on all three approaches were highly successful. However, many communities in Malawi are facing overwhelming challenges. Impact needs to be very significant for local communities to perceive a positive change in the face of worsening shocks.

• Evaluation Question 12:Looking forward, what are the emerging strengths, weaknesses, constraints and opportunities in managing and implementing the CJF and can these inform a potential future programme phase?

CJF has catalysed a shift towards climate justice. Nevertheless, an opportunity exists to increase collaboration, buy-in, replication and scale-up of climate justice projects. In addition, engagement with private sector firms was limited and likely to be challenging for grants-based projects to achieve.

In terms of CJF Sustainability (the extent to which climate justice outcomes are expected to endure), the evaluation found:

• Evaluation Question 13: To what extent did project implementing partners and/or beneficiaries assume ownership and responsibility for the project preparation, implementation, and sustainability?

Projects have been effective at achieving buy-in from communities and government partners, particularly when communities see benefits. For example, where projects have produced positive impacts for communities, especially around livelihoods and incomes (distributive justice), respondents were highly optimistic about sustaining those activities.

• Evaluation Question 14: To what extent has Scottish Government leveraged the CJF to strengthen CJ collaboration, buy-in, replication and scale-up?

CJF has catalysed a shift towards climate justice, although an opportunity exists to increase collaboration, buy-in, replication and scale-up of climate justice projects (refer to Evaluation Question 10).

Conclusions

Malawi, Rwanda and Zambia are amongst the countries contributing least to climate change but amongst those most vulnerable to climate change impacts. A climate justice approach is therefore highly relevant because it allows sustainable development for these countries, not only through direct funding support for climate change adaptation projects transferred from more developed countries but also through strengthening local capacities to respond to climate change and advocate for their own needs.

Drawing on the Scottish Government's climate justice policy and the evaluation's climate justice literature review, the evaluator proposes the following climate justice definition for future phases of the CJF: 'Climate justice is a people-centred, human rights-based approach that aims to share the benefits of equitable global development and the burdens of climate change fairly, while building trust between developed and developing countries. Climate justice recognises that the poor and vulnerable are the first to be affected by climate change, exacerbating existing inequalities, and will suffer the most, despite having done little or nothing to cause the problem.'

This definition of climate justice means that the evaluation looked for projects to include the following elements (recognising that this definition is being applied retrospectively, and was not the basis of the original grant awards):

• ensure a participatory, community-defined, needs-based approach to development (procedural justice);
• provide support for climate change resilience for the most affected areas and people in the Global South (distributive justice); and
• build understanding and capacity that enables local people to actively engage in decision-making and advocacy (transformative justice) to enable equitable, sustainable development in the face of climate change.

Key learnings from the evaluation include:

• It is important that the CJF clearly articulates its objectives, definition of climate justice, expectations and processes to project grantees. This supports them in delivering on the CJF objectives;
• All three pillars of climate justice are important for achieving CJF objectives and should be integrated from the design phase (through project selection, monitoring and reporting criteria):
• Procedural Justice:Engaging with both external stakeholders (such as national and district government) and local stakeholders (including women and the vulnerable) as part of a participatory and needs-based approach to identifying who to target and ensuring locally appropriate solutions;
• Distributive Justice:Ensuring support for climate change resilience (such as, livelihood development, access to water and energy etc.) is built into all projects (especially for innovation and mitigation focused projects) to ensure participants benefit from project activities;
• Transformative Justice: Active participation in finding solutions and advocating for their needs, empowering communities and ensuring sustainability of results;
• The time required for such an approach needs to be built into project timelines.

The evaluation finds that the CJF has been effective at delivering on its climate justice objectives, particularly in terms of mainstreaming climate justice within the Scottish Government and building knowledge and experience around climate justice through the CJF-supported projects. Although results vary across projects, CJF has achieved climate justice impact on-the-ground, including increased adaptive and absorptive capacity of vulnerable communities and, in some instances, improving equity. This was particularly the case for projects that addressed all three pillars of climate justice, engaged with local and national stakeholders and ensured the projects responded to local needs. Nevertheless, opportunities exist to build on this experience and share lessons to a broader audience beyond CJF direct stakeholders.

The CJF, through its portfolio of projects, covers all three pillars of climate justice: distributive, procedural and transformative. However, despite an increasing focus on broader aspects of climate justice, CJF project selection criteria have largely focused on distributive aspects of climate justice. Projects that have clearly addressed all three pillars of climate justice tend to be more effective, have greater impact and achieve more sustainable results. However, some aspects of climate justice (such as, participatory project design, developing capacity for advocacy and improving equity) may require longer implementation timelines than standard development projects. The procedural and transformative justice pillars are also important for addressing systemic climate justice issues. However, focusing on these aspects in short-term projects creates a risk that communities will participate in these initiatives and contribute to long term outcomes, but see few immediate returns. This could be addressed by ensuring projects cover all aspects of climate justice in their design and implementation and/or by clustering projects that focus on different aspects of climate justice within the same landscape.

Recommendations

The following recommendations are targeted at the Scottish Government CJF management team to support future CJF phases:

• Recommendation 1. It is recommended that CJF develops a set of project selection and monitoring criteria that ensure projects incorporate all aspects of climate justice (including procedural and transformative) in their design and implementation.
• Recommendation 2. It is recommended that CJF facilitates the sharing of lessons and learning from across its portfolio to support a community of practice that improves climate justice impacts both from across its portfolio and beyond.
• Recommendation 3. It is recommended that the CJF invests in larger programmes and/or supports smaller projects to provide clusters of interventions to communities.
• Recommendation 4. It is recommended that the CJF builds upon the flexibility of its approach that supports participatory processes in project design and implementation and complements these with more flexible project design and reporting cycles.
• Recommendation 5. It is recommended that Scottish Government leverages CJF learnings to support the integration of climate justice by other donors and programmes.