International Development - national indicator development: research report

Summary

Background

1. The objective of this assignment is to develop a methodology for the calculation of the national indicator Contribution of development support to other nations that forms part of Scotland's National Performance Framework (NPF). In addition, the indicator should be:

• A measure for Scotland, not just the Scottish Government that will be durable and capture the full range of activities.
• Mainly focused on those policy areas where Scotland has devolved responsibility in particular: equalities, climate change, health, education and justice.
• Based on transparent and international data sources that can be measured and updated in a timely fashion, so that Scotland's progress can be assessed.
• Be able to track Scotland's performance over time and not be dependent on comparative measures.

2. A key part of the project is to review existing approaches to measuring the contribution to international development. The Scottish Government had already engaged with the Center for Global Development (CDG) on the Commitment to Development Index (CDI). Cambridge Policy Consultants (CPC) reviewed the wider literature on the impact of Policy Coherence for (Sustainable) Development (PCD/PCSD) on international development.

3. Discussions with the Scottish Government identified five policy areas that currently make or are expected to make a distinct Scottish contribution to the development support to other nations:

• Adoption and implementation of Equalities legislation into policy practice;
• Climate Change including policy action to deliver net-zero carbon Scotland, knowhow on renewable technology and just transition support;
• Health in terms of health services and the determinants of health;
• Education through providing access to students from other nations and academic partnerships with further and higher education sectors in other nations;
• Justice including training and technical advice and other legal support.

Review of approaches to measuring the wider impact of international development support

Commitment to Development Index

4. The long-standing benchmark CDI was reviewed for its suitability as the NPF indicator. The CDI is a composite index constructed from seven components covering a range of policy areas - Aid, Finance, Technology, Environment, Trade, Security and Migration. Key findings were:

• The majority of CDI measures cover policy areas that are currently reserved to the UK government and would therefore not reflect Scottish policy choices.
• Of the five policy areas selected to explore indicators for Scotland's contribution to international development, CDI indicators provide coverage of just the Climate Change theme.
• The methodology of CDI is based on cross-country comparisons and for the purpose of the NPF indicator would not provide a consistent measure of progress over time.^[1]

5. Nevertheless, there are a number of individual indicators used in the CDI that are relevant to developing an NPF indicator for Scotland covering access to higher education, trade, agricultural subsidies, low carbon economy and arms sales.

Policy Coherence for Development Index

6. The concept of Policy Coherence for Development (PCD) originally emerged in the early 1990s from the realisation that non-aid policies of donors affect developing countries and should not distract but rather be supportive of international development goals. A number of measurement frameworks have been proposed with the principal being PCDI. The PCDI is an index that adopts a similar method to CDI, standardises the degree of change across variables and weights their influence before combining them for an overall score.

7. PCDI uses 49 indicators that have been analysed with data for 133 countries and have been organised into 31 indicators that promote policy coherence and 18 indicators that are contrary to sustainable development processes. Key considerations are:

• Although a number of the measures are closer to the policy domains, some are based on ordinal values or policy assessments and so either cannot be quantified or would require significant effort to update each year.
• More so than CDI, PCDI is reliant on cross-country comparisons in benchmarking and aggregating indicators into an overall index. This would impact on the robustness of the NPF indicator over time.
• The weighting of different indicators is not transparent between different policy areas and so unlikely to fit with Scotland's policy approach.
• As with CDI, even if the overall structure of PCDI does not fit with the design criteria for NPF indicator, a number of individual variables may be relevant to developing an NPF indicator for Scotland.

8. Therefore, while neither the CDI nor the PCDI approach is directly replicable for the NPF indicator, the review identified a number of variables for consideration as indicator components.

Rationale and variable selection for the NPF indicator

9. The NPF is intended to provide a benchmark of progress on National Outcomes for Scotland. We recommend that a the NPF indicator adopts existing policy concepts wherever possible and that it builds on the Beyond Aid agenda in Scotland. As this is still under development we have considered how policy coherence sits with the existing strategic objectives set for Scotland's International Framework.

10. This NPF indicator aims to draw together:

• Scotland's reputation for developing durable and positive partnerships;
• Scotland's international best practice in environmental, equalities and rule of law standards;
• The policy areas where Scotland is pre-eminent, i.e. health, education, environment, justice, equalities in a manner that links how these relate to policy coherence and more effective development support to other nations.

11. It will be important that in future there is explicit reference to Scotland's emerging Beyond Aid agenda but it also the existing International Framework^[2] objectives. Table 1 links Scotland's ambitions we identified with a rationale and the International Framework objectives:

Table 1: Ambitions and rationale

Scottish Values:^[3] We are good global citizens

Rationale: Our economic, educational, cultural and heritage strengths are globally recognised, supporting our positive international reputation

International Framework Strategic Objectives: Relationships & Partnerships: Strengthen our external relationships, roles and networks

Scottish Values:^[3] We avoid harm to the development of other nations

Rationale: In order to provide credible support to other nations we need to ensure that we follow international best practice standards

International Framework Strategic Objectives: Reputation and Attractiveness: Build our reputation and international attractiveness

Scottish Values:^[3] We support development in other nations

Rationale: Promote knowledge exchange through sharing the experience and expertise of our public, private and community sectors

International Framework Strategic Objectives: Global Outlook: Enhance our global outlook to set the domestic conditions for success

12. The process to populate this framework with specific measurable indicators involves two stages:

• Conceptual: an assessment of which measures represent improved policy coherence for each element of the framework. This process adopted two stages itself:
  • Firstly, adoption of CDI or PCDI indicators for the relevant devolved policy area.
  • Secondly, the use of indicators suggested in the wider literature and stakeholder discussions to cover those policy areas that are not covered by CDI/PCDI frameworks
• Practical: What are the practical steps necessary to populate these measures with robust data, updated on a regular basis at reasonable cost.

Table 2: What Scottish values identified mean for contribution of development support to other nations

Scottish Values	What this means for contribution of development support to other nations	Linked National Outcome (NPF)
We are good global citizens	We work with partners to build capacity and engage in dialogue on development and human rights	Education, Economy & Human Rights
	We support migrant and asylum seeker populations coming to Scotland	Human Rights
	We welcome students from developing countries to our educational institutions	Children and Young People
We avoid harm to the development of other nations	We avoid contributing to climate change and environmental damage internationally	Environment
	We trade and invest fairly	Economy
We support development in other nations	We promote equality and human rights	Human Rights
	We promote knowledge exchange and share the experience and expertise of our public, private and community sectors	Education
	We work to improve health outcomes	Health
	We advocate trade to support development	Economy
	We support fairness under the law	Justice

13. This process produced a list of potential indicator components that were reviewed through internal discussions with Scottish Government including a workshop with policy officers and a workshop with external stakeholders.

14. The process excluded measures from the long list on the grounds that it is not possible to access robust data on a regular basis. In a minority of cases a single indicator could not be identified that would adequately represent the measure and reliably represent progress for Scotland.

Table 3: Proposed indicator components against identified Scottish values

Scottish Values	What this means for contribution to development of other nations	Proposed Indicator
We are good global citizens	We work with partners to build capacity and engage in dialogue on development and human rights	Scotland's connectedness to ODA recipient countries[4]
	We support migrant and asylum seeker populations coming to Scotland	Asylum seekers settled in Scotland per 100,000 population
	We welcome students from developing countries to our educational institutions	HE Students from DAC Least Developed Countries / Total Non-EU Students
We avoid harm to the development of other nations	We avoid contributing to climate change and environmental damage internationally	% of total waste treated in Scotland Value of the Low Carbon and Renewable Energy Economy (LCREE) in Scotland
	We trade and invest fairly	Value of Agricultural subsidies
We support development in other nations	We promote equality and human rights	% of international development funding devoted to vulnerable groups
	We promote knowledge exchange and share the experience and expertise of our public, private and community sectors	Value of R&D contracts in HEIs in partnership with ODA-eligible nations
	We value trade to support development	Value of goods imports from least developed ODA countries

Calculating a composite indicator for the NPF

15. The ultimate objective of this assignment has been to identify a number of relevant indicators that together provide a practical measure of Scotland's contribution to the development of other nations. For the purpose of NPF, this measure needs to be a single composite index. The final stage of this project was to aggregate the indicator components into a single indicator, using international best practice.

16. The combination of indicator components raises several questions:

• How can each measure be standardised so that they are on an equivalent basis? This is done through expressing the indicator in proportionate terms or with reference to the range of values (for example with reference to minimum and maximum values measured across all comparator countries).
• How to deal with any missing values?
• Should the different elements of indicators and sub-components be weighted differently? Both CDI and PCDI employ a degree of expert judgement in applying weights. Thus, while the weight values are apparent, the reasoning behind them is not.

17. The design criteria ruled out the use of comparators, and it follows that in combining the indicators for Scotland:

• There are no missing values in the selected indicator components as the availability of regular updates is one of the selection criteria.
• The most straightforward approach is that the NPF indicator measures year-on-year change in the underlying indicators as a measure of progress.
• At this stage, there is no objective basis to applying weights to the values of the individual indicators. So, the indicators are combined without weights.
• We use a geometric mean to combine the indicator components into the NPF indicator. This is the approach adopted by UNDP's Human Development Index (HDI) as mathematically it reduces the impact of significant variation in one of the indicator components dominating the score of the composite indicator.

Populating the NPF indicator with data

18. Annual data has been sourced for the proposed indicator components for the period 2014-2017. Indices have been calculated for each indicator component with a baseline value (=100) in 2017. The index value for the other years is then set as a proportion of this baseline value. For all but one indicator component, a larger value represents a larger positive impact on policy coherence. For one indicator component, the value of agricultural subsidies, an inverse of the index is used so that a larger value (that is considered to have a detrimental impact on policy coherence) is transposed into a lower index.

19. These indicator components are then combined for each year into an overall composite indicator. The values are the geometric mean of the indicator components. Table 4 below lists the index value for each year since 2014.

Table 4: NPF indicator scores 2014-2016 against baseline year 2017

	2014	2015	2016	2017
Indicator score for Scotland's Contribution Development support to other nations	75.2	84.4	77.5	100.0

20. On the basis that the 2017 indicator is set to the baseline of 100 this means observed changes in the index should be reported accordingly:

Table 5: Threshold for change

Value of Index in 2018	Measure of change
greater than 105	Improving index value
95 to 105	Maintaining index value
Less than 95	Worsening index value

Potential changes in the NPF indicator in future

21. The literature on how policy coherence leads to sustainable development is broad but often imprecise. Further research and evaluations could offer a new source of potential indicators to be included in the indicator in the future. This implies that the Scottish Government should keep this NPF indicator under review and seek to update and amend the individual indicator components periodically as necessary. There would appear to be a number of areas where this may be worthwhile in future:

• Include more policy areas if and when these are devolved to Scotland.
• Seek to include wider measures of partnership working and the contributions of Scottish expertise that is not captured in simple input measures. While they have proved intractable to a low-cost data collation process, new data sources may become available in the future that could address this gap.
• A more specific gap that may be filled by Scottish Government itself is the reporting on partnership activity associated with the International Framework Strategy. At present, no data is collected on the spend on partnerships at this level but this may also change in future.