Publication - Research and analysis

International Development - national indicator development: research report

Published: 6 November 2020
From: Director-General Strategy and External Affairs
Directorate: External Affairs Directorate
Topic: International
ISBN: 9781800040908

The report outlines the research commissioned for the development of the indicator ‘Contribution of development support to other nations’ that forms part of the National Outcome ‘We are open, connected and make a positive contribution internationally’ in the refreshed National Performance Framework

Supporting documents

Annex C: Review of Policy Coherence for Development Indicators

Background

PCD is an approach and policy tool for integrating the economic, social, environmental and governance dimensions of sustainable development at all stages of domestic and international policy making. It is the aim of Policy Coherence for Development to make foreign relations to be as ecologically, economically and socially coherent as possible and thereby to make international co-operation for international development more effective.^[36] The literature on Policy Coherence for Development (PCD), and more recently Policy Coherence for Sustainable Development (PCSD), is extensive with the earliest references back to 2005 being developed by the OECD and European Commission among other institutions.

PCD has been discussed in relation to the concept of Global Public Goods (GPGs). GPGs are in principle available to everyone and each country has an interest in contributing to their promotion. Examples might include a fair, robust and market- orientated trading system for goods and services or climate stability.

PCSD stems from the 2030 Agenda and the Addis Ababa Action Agenda. In addition, in the case of PCSD, given that the SDGs are universal, policy makers have to secure broader policy coherence by pursuing multiple goals globally. The OECD has defined PCSD as an approach and policy tool relevant to all countries, to be used at the domestic and international levels of policy making.^[37]

The European Commission in collaboration with the Member States, identifies five broad PCD priority areas: (i) trade and finance, (ii) climate change, (iii) food security, (iv) migration and (v) security.

Measuring the impact of PCD/ PCSD

There are a number of PCD/ PCSD frameworks in the literature but in general these focus on the policy procedures in place to ensure policy coherence for development but that the vast majority do not (yet) specify performance measures or indicators that will assess a country's progress and impact in implementing PCD/PCSD. An issue throughout is what has been set down in policy intent and what has been implemented to date.

In part this reflects the different mechanisms that countries have defined to drive their PCD process. These include:

Formal parliamentary committees or formal responsibility to consider PCD/PCSD in policy development
Annual reviews of PCD led by civil society organisations who report to government on progress and areas for improvement
Ex-post reviews of evaluations of international development initiatives to consider how other national policies have supported or hindered their effectiveness

A recent review of the use of PCD indicators by selected EU member states highlights that those countries who have a process do not necessarily then have official performance indicators that track PCD/PCSD progress.

	PCD Mechanism	'Official' cross-government PCD indicators
Belgium	Yes	Not yet
Denmark	Yes	Yes
Finland	Yes	Not yet
Germany	Yes	Not yet
Ireland	Yes	Not yet
Luxembourg	Yes	Not yet
Netherlands	Yes	Yes
Sweden	Yes	Yes

Source: ECDPM Discussion paper 171, Use of PCD indicators by a selection of EU Member States, Jan 2015

A number of reviews have highlighted that there is a need to strengthen and make explicit the causal logic chain between PCD/PCSD.^[38] An evaluation of the impact of PCD activity across member states similarly concluded that more work was required to establish a causal link between PCD/PCSD reviews of policies and the consequent impacts that might then be measured by indicators.^[39] The overall conclusion of the evaluation was that while changes in policies could be evidenced, it was not yet possible to ascribe an impact of these changes in quantitate terms on the international development process.

Determining expectation for impact is clearly a crucial stage in the process for defining relevant and appropriate indicators that this may need to be revisited with Scottish Government colleagues.

This image illustrates examples of chains of causality in the area of trade and finance in Sweden based on the ECDPM Discussion paper 171 — Example of a logic chain linking PCD/ PCSD to indicators: Sweden

Source: ECDPM Discussion paper 171, Use of PCD indicators by a selection of EU Member States, Jan 2015

In their seminal paper, King and Matthews (2012) also identified the challenges that can arise by a lack of precision in measure definition. This , for example, can be at the level of using concepts as indicators - in the Swedish example above, the indicators are not indicators as they have no defined scale and could not be quantified as they stand. These criticisms are repeated by ECDPM (2015) who found that across the member states reviewed "different logical frameworks mix up objectives, targets, actions and indicators."^[40]

King and Matthews (2012) proposed that indicators be categorised to avoid this confusion:

Outcome indicators: these focus on outcomes such as socio-economic variables (e.g. income per capita, student enrolment rates, etc.).
Policy outputs: that capture the changes in policies designed to be more coherent with development. E.g. the level of tuition fees for students from developing countries, food tariffs for imports from developing countries, etc.
Policy inputs: that should be used where it may be difficult to summarise the output of a policy into a single indicator, e.g. the proportion of funding that supports the primary objectives of the developing country.
Policy stance indicators: relate to treaty or protocol agreements e.g. the signing of international agreements on financial transparency, etc.

They also proposed a set of criteria for the selection of PCD indicators:

Transparency: Can a layperson understand what is happening? Does the index hide or reveal facts?
Policy relevance: Does the indicator/index relate to important societal debates?
Analytical soundness: Does the indicator measure the problem, or rather something else?
Responsiveness: Does a politician have any chance to improve the indicator/index?
Time horizon: How quickly can results be expected? Non-ambiguity of "welfare message": Does everybody agree that "more is better", or vice versa?
Accountability: Does the indicator/index point at those who should be held responsible?
Robustness/ independence of assumptions: Could the value of the indicator change drastically by fumbling with some assumptions?
Measurability, data availability: Will we see comparable figures in the next ten years?

Implications for selecting indicators for Scotland's NPF

The criteria set out above set a relatively high bar for the current practice on measuring PCD/PCSD using indicators. While we have found a wide range of reports on PCD/PCSD frameworks and procedures, only two provide a coherent approach to setting out baskets of indicators that could be used to track progress. These are set out below.

King and Matthews indicators for PCD in Ireland

Using their own design criteria, King and Matthews set out a set of indicators that they recommended to the Irish Government as a method to track their PCD progress. These reflect the key areas of Irish Government's interests and in many ways are similar to the components used in CGD CDI.

Trade Policy Indicators

T.1.1 Average Tariffs on Manufacturing Imports, 2010.
T.1.2 Share of Duty-Free Imports, 2009.
T.1.3 Trade Restrictiveness Indicators for Manufactured Goods, 2009.
T.1.4 Trends in Import Growth Rates, 2007-2009.
T.2.1 EU and Irish Trade Preference Utilisation, 2009.
T.3.1 ODA Expenditure on Trade Policies & Regulations, % of 2008 GDP.

Agriculture policy Indicators

A.1.1 Average Tariff on Agricultural Imports, 2010.
A.1.2 National Levels of Market Price Support, 2009.
A.1.3 Trade Restrictiveness Indices for Agricultural Goods, 2009.
A.1.4 Growth in Agricultural Imports from Developing Countries, 2007-2009.
A.2.1 Trade-distorting Support, 2007.
A.3.1 Agricultural ODA Expenditure, 2008.

Fisheries Policy Indicators

F.1.1 Ireland's Participation in International Agreements on Fisheries Protection, 2010.
F.1.2 DAC Country Compliance Scores for FAO (UN) Code of Conduct for Responsible Fisheries, 2006.
F.2.1 Average MFN and Applied Tariffs on Fish and Fish Products, 2008.
F.3.1 Government Financial Transfers to Fisheries Sector, as a % of the Total Landed Value, 2007.
F.4.1 Ireland's Industrial Pelagic Fishing Possibilities in Morocco, 2007-2011.
F.4.2 FAO (UN) Code of Conduct for Responsible Fisheries, Compliance Scores for FPA Countries, 2006.
F.4.3 Marine Protected Areas, % of Country's Exclusive Economic Zone, 2010.
F.4.4 Ireland's Contribution towards Fisheries Capacity Building in Developing Countries, 2008.

Migration Indicators

M.1.1 Non-DAC Inflow as a Percent of Total Population, 2008.
M.1.2 Number of Residents in Ireland from Different Regions of the World, 2006.
M.1.3 Country of Origin of African Migrants into Ireland, 2006.
M.2.1 Support for Remittances to Developing Countries, 2010.
M.3.1 Total UNHCR Population of Concern + Applications/ Billion USD of GDP, 2010.
M.4.1 Ratio of Tuition Fees for non-DAC students to DAC students and Irish Students, 2004.
M.4.2 Proportion of non-DAC (to total) students in tertiary education, 2007.

Environment Indicators

E.1.1 Environmental Protection ODA (Commitment), 2008.
E.2.1 Average Annual Growth Rate of GHG Emissions/PPP GDP, 1997-2007.
E.2.2 Performance in Meeting Kyoto Protocol Targets, 2008.
E.2.3 ODA Expenditure on Climate Change, 2008 (Second Rio Marker).
E.2.4 ODA Expenditure on Desertification, 2008 (Third Rio Marker).
E.3.1 ODA Expenditure on Biodiversity (Disbursement), 2008 (First Rio Marker).
E.3.2 Adoption of Convention of Biological Diversity and Related Protocol, 2010.
E.4.1 MFN Tariffs on Bioethanol, 2010.
E.4.2 Subsidies for Liquid Biofuels (Ethanol and Biodiesel), Most Recent Year.

Finance and Enterprise Policy Indicators

FE.1.1 ODA Expenditure on Debt Relief, 2007- 2008.
FE.2.1 Existence of Double Taxation Agreements with Irish Aid Priority Countries, 2010.
FE.3.1 Level of foreign bribery enforcement in OECD Convention Countries, 2011.
FE.4.1 Restrictions on the Flow of Technology to Developing Countries, 2010.

Security Policy Indicators

S.1.1 Peacekeeping Contribution, UN-run Operations, Progressively Weighted to the Present, 1993-2009.
S.1.2 Peacekeeping Contribution, Non UN-run Operations, Progressively Weighted to the Present, 1993-2009.
S.1.3 Expenditure on Security System Management and Reform, 2008.
S.1.4 Participation in Four Essential Security International Treaty and Related Policies, 2010.
S.2.1 Exports of Major Conventional Weapons, 2008.

Development Aid Indicators

DA.1.1 Level of Overseas Aid (ODA), 2010.
DA.2.1 Irish Aid Partner Country GNI per capita, 2008.
DA.2.2 Governance Quality, Kaufman and Kraay Government Effectiveness Scores, 2009.
DA.2.3 Corruption Levels, Kaufman and Kraay Control of Corruption Scores, 2010.
DA.2.4 Economic Management Quality, 2010.
DA.2.5 Strength of Social Inclusion Policies, 2008.
DA.3.1 % of Aid Flows Disbursed for Government Sector, 2007.
DA.3.2 ODA Expenditure Lost to Tied Aid, 2009.

Many of the indicator categories cover policy areas that remain reserved matters for Scotland and as noted above, these categories closely reflect the CDI components and so do not directly address a number of the policy themes we are seeking to capture.

PCDI (Policy Coherence for Development Index)

The Policy Coherence for Development Index (PCDI)^[41] is a tool designed to measure, evaluate and compare countries' commitment to sustainable, fair and equitable human development. The concept of Policy Coherence for Development (PCD) originally emerged in the early 1990s from the realisation that non-aid policies of donors affect developing countries and should not distract but rather be supportive of international development goals. The PCD concept initially emphasised the responsibility of developed countries to consider the effect on developing countries when formulating domestic policies across different sectors (trade, finance, migration, security, technology, science).

As the concept evolved, the PCDI has been developed to go beyond a 'do no harm' approach, also with a requirement to seek synergies between development co-operation and other policies as well as to correct existing incoherencies.

The PCDI analyses both the policies that make a positive contribution to a country's sustainable development and those that hinder it, not only within that country but also in third countries or on the planet as a whole. The PCDI is divided up into five components: economic, social, global, environmental and production.

Knoll (2014) reviewed 20 different policy domains through four different dimensions: environment, economic, social and political and grouped them together in five components based on their similarities and to provide a categorisation that was more accessible to decision-makers.

Economic Component

Fiscal
Financial

Social Component

Education
Health
Social Protection
Equality
Employment
Science & technology

Global Component

Peace & security
Co-operation
Justice & human rights
Human mobility & migration

Environmental Component

Energy
Biodiversity
Fisheries
Rural & agricultural development

Production Component

Industry
Infrastructure & transport
Tourism
Urban Planning

Source PCDI Report Chapter 4 p129.

Indicators were selected for each element of the matrix based on data from 234 countries and an initial set of 201 indicators. The removal of missing data reduced the dataset to 133 countries and 133 variables. These were then further reviewed using factor analysis to produce a list of 49 variables for 133 countries with six indicators in the economic component, nineteen in the social, ten in the global, eight in the environmental and six in the production component. These were organised into 31 indicators that promote policy coherence (such as inequality reduction, public spending on social protection and ratification of universal justice treaties) and 18 indicators that are contrary to sustainable development processes (such as school dropout rates, military spending and ecological footprint). The PCDI is an index and in a similar method to CDI, standardises the degree of change across variables and weights their influence before combining them for an overall score.

Of the 49 variables, 18 reflected indicators contrary to sustainable development processes, whereas the other 31 reflected indicators that favoured them.

The preparation of the variables involved the following actions:^[42]

Grouping of countries: the countries were grouped into 6 groups:
Group 1: OECD countries, accession countries and countries with enhanced cooperation;
Group 2: South
Group 3: Latin America
Group 4: Europe and Central Asia
Group 5: Sub-Saharan Africa
Group 6: Middle -East and North Africa
Exclusion of variables with high missing values (>40% and some with >30%) following the priority of each variable and the number of rem each policy.
Grouping of categorical variables (1/0) into a scale variable
Elimination of variables with high correlations among them variables are related that two or more of them are quantifying the same information, therefore they may reduce the reliability of the index. This may induce a double count in the aggregation step, reducing the reliable the use of statistical methods necessary
An analysis of outliers was carried out for each variable with a Boxplot analysis. To perform this analysis, all the variables were reviewed and the outliers that appeared were replaced by another value based on statistical criteria (e.g. the highest non-outlier variable, the median value, etc.) and logical interpretation criteria.
A Min-Max normalisation was applied to normalise the variables to follow a range between 0 and 1 (or between 0 and 100)
The classification of variables into those which support a country's development and those that hinder a country's development.

The result of this process produced the set of variables outlined in table C.2:

Table C.2: Overview of PCDI variables

Dimension: Economic component