International Development - national indicator development: research report

2. Review of International Measurement Approaches

2.1. Introduction

2.1.1. The starting point for the assignment was to explore the range of policy areas that underpin Scotland's contribution of development support to other nations. This involved initial discussions with Scottish Government international Development policy team on:

• Which Scottish policy areas make a difference to Scotland's commitment to development?
• How such domestic policies and practice influence international development?
• What performance measures might be in scope to best represent this process at a national level?

2.1.2. Scottish Government analysts had already undertaken a preliminary assessment of the Center for Global Development's (CGD) Contribution to Development Index (CDI). The team reviewed both CDI literature and literature relating to the wider policy coherence for development in order to consider:

• Which policy areas make a difference to Scotland's commitment to development?
• How have these been measured in the international literature?
• What lessons can Scotland learn from this experience and how well does it apply to the current devolved policy position?
• What are potential indicators for Scotland and their data sources?

Setting key design parameters - what should the indicator components cover?

2.1.3. The discussions with Scottish Government identified five key policy areas that currently make or are expected to make in future, a distinct Scottish contribution to the development support to other nations. These are:

• Adoption and implementation of Equalities legislation into policy practice across protected rights groups on legal rights and anti-discrimination regulation alongside rights of access to services and support.
• Climate change including policy action to deliver net-zero carbon Scotland, knowhow on renewable technology and just transition support to alleviate the potential downside on the economy and society of implementing net zero policy.
• Health in terms of health services and the determinants of health (e.g. water management and sanitation, etc.). Scottish NHS and other support to other nations through staff volunteering, technical assistance and partnerships, hosting of medical staff from other nations to develop skills and medical practice.
• Education through providing access to students from other nations to come and study at Scottish Universities and colleges. This may also include academic partnerships with further and higher education sectors in other nations based on the strong reputation of Scotland's academic and educational administrative expertise
• Justice to include Police Scotland's support to other nations through training and technical advice and potentially other legal support and training for other nations' judiciary etc.

2.1.4. While discussions considered a broader range of policies and the potential inclusion of possible policy areas that may inhibit the contribution to development, it was agreed that the literature for measuring policy impacts would prioritise these five policy areas. However, the review of existing methods would also consider the inclusion of others that may be suggested in the literature - for example, policy action to deliver inclusive growth, where these conform to the design criteria (primarily a devolved policy responsibility).

2.1.5. The review of existing literature covered two areas:

• Scottish Government has already had initial discussions with the Center for Global Development (CGD) relating to their Commitment to Development Index (CDI) which is a primary indicator of the scale and quality of a country's contribution to international development and an obvious starting point for this review.

The literature relating to the measurement of the impact of policy coherence on international development.

2.1.6. Both these performance measurement frameworks have been reviewed by this research. The review considered how different indicators might reflect that Scotland's contribution to other nations may arise through:

• Indicator definitions that are relevant for Scotland by being both:
  • a measure of a devolved policy responsibility and
  • reflective of the contribution Scotland makes to the development of other countries
• Accessibility of data sources that can be updated efficiently on a regular basis.

2.1.7. The next section reviews the relevance of the CDI and its constituent parts in providing relevant indicators for the above. The subsequent section similarly reviews the broader policy coherence for development (PCD) literature for indicator components that may better fit with Scotland's devolved policy interests.

2.2. Review of the Commitment to Development Index (CDI)

2.2.1. It should be stressed at the outset that this review exclusively assesses the extent to which the long-standing benchmark Commitment to Development Index (or a subset of indicators) might be adopted as an indicator of Scotland's contribution to international development.

2.2.2. The CDI is a composite index constructed from seven components covering a range of policy areas - Aid, Finance, Technology, Environment, Trade, Security and Migration. Each component is made up of sub-components and the sub-components are in turn a composite of individual indicators. A full list of CDI variables is included in Annex B.

2.2.3. Prima facie, the CDI offers a number of features that would suit the NPF indicator:

• It covers a broad range of policies in place in the donor country with the aim of assessing their contribution to development (and so would be a measure of the whole of Scotland;
• It is a long-standing international methodology, established in 2003 and has been updated regularly since with some revisions to the methodology meaning that not all years are directly comparable;
• It is based on publicly available international data sources.

2.2.4. While it is clearly possible to track the change in the CDI over time, the real strength of the CDI measure is that it applies a consistent methodology to the international development activities of a number of countries. This allows to assess their contributions comparatively.

2.2.5. However, there are a number of areas where the CDI would be less suitable for NPF in Scotland:

• The majority of measures cover policy areas that are currently reserved to the UK government and so the index mostly measures the situation for the UK and not Scotland. Devolved policy areas are often contained in categories where reserved powers dominate and it is not possible to subdivide the weighting but even when all these areas are included the weighting of devolved policies is 38%.
• The evidence base on which the weights for different policy areas are based has often been challenged in the literature and is in part based on the judgement of researchers at CGD. The question for this assignment is not whether they are technically justified but whether they fit with Scotland's priorities for international development.^[6]
• The raw values of CDI's variables are on different scales. This makes standardising the values necessary to enable comparison across variables, and calculations of performance and ranking need to be standardised before they are combined. This is vital to ensure the stability of the index and that highly variable measures do not dominate. However, it adds a layer of complexity and means the index is much less transparent.
• Of the five policy areas we recommend to explore for indicator components for Scotland's contribution to international development, CDI indicators provide coverage of just the Climate Change theme (see Annex B for more details).

2.2.6. Subsequent discussions with the CGD team on the key features of the CDI highlighted that the reasons behind many of these 'gaps' in the index related to:

• A lack of comparable data across the G20 countries that the CDI covers
• The challenge of selecting a single quantitative indicator or even a basket of indicators that would adequately represent some of these policy domains for each country.

Overall conclusions on relevance of CDI to Scotland

2.2.7. It is clear that while the CDI provides an important benchmark of individual country's commitment to development, this does not apply when the country in question does not have full policy discretion. More significantly, the CDI measures are specifically designed to be 'near market' indicators of the key factors that influence the effectiveness of international development spend. While this may be relevant to Scotland in future, the current policy focus is on the beyond aid agenda of policy coherence. CDI has limited coverage of these policy areas that are central to Scotland's contribution to the development of other nations.

2.2.8. This does not mean that individual measures within CDI have no relevance for Scotland and where possible we have included them in the suggested measures as they are based on available data sources. The following indicators were identified as being potentially relevant:

• Greenhouse gas emissions
• Value of agricultural subsidies
• Value of arms sales
• Assessment of the integration policies for migrants (MIPEX)
• Scotland's share of asylum seekers
• Students from ODA-Receiving Countries on Higher Education courses in Scotland/ Total Tertiary Students

2.2.9. It is also apparent that the calculation of an index from a range of variables raises a number of questions that may not suit the NPF:

• Firstly, that changes in the value of a variable need to be standardised if they are to be combined so that significant swings in one indicator do not dominate other variables that are may change more gradually. This can be a complex process that means final index values are less transparent.
• Secondly, the individual variables need to be combined in some way. CGD recognise that this is dependent on expertise and judgement and gives rise to more challenge than the choice of individual variables.

2.3. Review of Policy Coherence Measures

Introduction

2.3.1. The concept of Policy Coherence for Development (PCD) originally emerged in the early 1990s from the realisation that non-aid policies of donors affect developing countries and should not distract but rather be supportive of international development goals. The PCD concept initially emphasised the responsibility of developed countries to consider the effect on developing countries when formulating domestic policies across different sectors (trade, finance, migration, security, technology, science).

2.3.2. The literature on Policy Coherence for Development (PCD) and, more recently, Policy Coherence for Sustainable Development (PCSD) is extensive with the earliest references back to 2005 being developed by the OECD and European Commission among other institutions. PCD is an approach and policy tool for integrating the economic, social, environmental and governance dimensions of sustainable development at all stages of domestic and international policy making. It is the aim of PCD to make foreign relations to be as ecologically, economically and socially coherent as possible and thereby to make international co-operation for international development more effective.^[7] Full details are included in Annex C.

2.3.3. PCD has been discussed in relation to the concept of Global Public Goods (GPGs). GPGs are in principle available to everyone and each country has an interest in contributing to their promotion. Examples might include a fair, robust and market- orientated trading system for goods and services or climate stability. A range of policy areas have been included in the PCD literature, however, the European Commission in collaboration with the Member States, identifies five broad PCD priority areas: (i) trade and finance, (ii) climate change, (iii) food security, (iv) migration and (v) security. Others have also included health, education and equalities.

Review of PCD/PCSD measurement frameworks

2.3.4. There are a number of PCD/PCSD frameworks. In general these focus on the policy procedures in place to ensure policy coherence for development. However, the vast majority do not specify performance variables or indicators that will assess a country's progress and impact in implementing PCD/PCSD. We found two studies that specified indicators that may be of interest: King and Matthews (2012)^[8] and Knoll (2014).^[9]

2.3.5. King and Matthews (2012) identify the challenges that can arise by a lack of precision in measure definition, for example measures have no defined scale and cannot be quantified. These criticisms are repeated by ECDPM (2015) who find that across the EU Member States reviewed 'different logical frameworks mix up objectives, targets, actions and indicators.'^[10]

2.3.6. King and Matthews (2012) propose 53 indicators across eight policy domains: Trade, Agriculture, Fisheries, Migration, Environment, Finance and Enterprise, Security, and Development Aid that reflect the components used in the CDI. In the Scottish context, most of these policy areas are reserved to the UK Government. In addition, while these indicators provide a basket of measures across these policy areas, many were drawn from specific studies rather than official statistics and so may not be repeatable or updated on a regular basis.

2.3.7. Knoll (2014) reviews 20 different policy domains through four different dimensions: Environment, Economic, Social and Political and grouped them together in five components based on their similarities and to provide a categorisation that was more accessible to decision-makers.^[11]

Table 2.1: Dimensions and policy domains in Knoll (2014)

Dimensions: Economic Component

Policy domains:

• Fiscal
• Financial

Dimensions: Social Component

Policy domains:

• Education
• Health
• Social Protection
• Equality
• Employment
• Science & technology

Dimensions: Global Component

Policy domains:

• Peace & security
• Co-operation
• Justice & human rights
• Human mobility & migration

Dimensions: Environmental Component

Policy domains:

• Energy
• Biodiversity
• Fisheries
• Rural & agricultural development

Dimensions: Production Component

Policy domains:

• Industry
• Infrastructure & transport
• Tourism
• Urban Planning

Source: Knoll (2014) Reported in PCDI Report Chapter 4 p129.^[12]

2.3.8. For the Policy Coherence for Development Index (PCDI), indicators were selected for each element of the matrix based on data from 234 countries and an initial set of 201 indicators. The removal of missing data reduced the dataset to 133 countries and 133 variables. These were then further reviewed using factor analysis to produce a list of 49 variables for 133 countries. These were organised into 31 indicators that promote policy coherence and 18 indicators that are contrary to sustainable development processes. The PCDI is an index and in a similar method to CDI, standardises the degree of change across variables and weights their influence before combining them for an overall score. Further detail on the PCDI's variable selection process can be found in Annex C.

2.3.9. A number of the 49 indicators can be mapped to Scotland's five key policy areas that were identified as relevant for Scotland's Contribution of development support to other nations: The full mapping is provided in Annex C but in summary the mapped variables comprise:

• Climate change
• Equality
• Education
• Determinants of health
• Justice

2.3.10. The PCDI framework suggsets that rather than present standalone policy areas, establishing a logical framework for these actions makes them more accessible to a wider audience and provides a clear association between policy action in Scotland and an improved contribution to the development of other nations. Hhow this can be done for Scotland is addressed in the next section.