International Development - national indicator development: research report

3. Establishing an Indicator for Scotland's Contribution of Development Support to Other Nations

3.1. An approach for Scotland's Contribution of the development support to other nations

3.1.1. The next stage of the project was to develop an approach that places the five broad policy areas into the context of how Scottish public life (not just Scottish Government policy, but also the public, partnerships and institutional practice) will contribute support to the development of other nations.

3.1.2. This stage is essential in order to clearly establish a logic chain between Scotland's wider ambitions and areas of activity that will contribute to the Beyond Aid agenda in Scotland. This would also provide a mechanism to consider which variables are most appropriate to represent these contributions in the National Performance Framework and summarise what might otherwise be a very wide range of concepts into a more manageable set. Ultimatively, all variables are combined in a composite indicator.

3.1.3. While the NPF is intended to provide a benchmark of progress for Scotland, we think that it is important the framework adopt existing policy concepts wherever possible. We have considered how policy coherence sits with the existing strategic objectives set for Scotland's International Framework.

3.1.4. The framework aims to draw together:

• Scotland's reputation for developing durable and positive partnerships;
• Scotland's international best practice in environmental, equalities and rule of law standards;
• The principal policy areas where Scotland is pre-eminent - health, education, environment, justice, equalities but in a manner that links how these relate to better policy coherence and ultimately more effective support to other nations;
• Other areas where policy coherence is seen as vital in order to provide a more effective contribution to the development of other nations and for which Scotland has devolved responsibility. The manner in which both CDI and PCDI map domestic policy coherence to development support and the literature from the OECD and EU were particularly important here.^[13]

3.1.5. The process to populate this framework with specific measurable indicators involves two stages:

• Conceptual: which measures best represent improved policy coherence for each element of the framework. This process adopted two stages itself:
• Firstly, adoption of CDI or PCDI indicators for the relevant devolved policy area.
• Secondly, the use of indicators suggested in the wider literature and stakeholder discussions to cover those policy areas that are not covered by CDI/PCDI frameworks
• Practical: what are the practical steps necessary to populate these measures with robust data, updated on a regular basis at reasonable cost.

3.1.6. The remainder of this section considers the issues arising from the conceptual stage of the process, while the practical investigation of potential variables and data sources is detailed in the next section.

3.1.7. The decision process behind the development of the NPF indicator is reproduced in Annex D. It also includes the account of the decisions taken to refine the indicator, based on comments from the project steering group, discussions with stakeholders supporting Scotland's international development activities and input from the Scottish Government International Development policy team. The draft framework is set out in the table below, the rationale for the inclusion of these measures is in Annex D.

Table 3.1: Initial Framework for Scotland's Contribution of development support to other nations

Scotland's Ambitions: We are good global citizens

What this means:

• We work with partners to build capacity and engage in dialogue on development and human rights
• We support migrant and asylum seeker populations coming to Scotland
• We welcome students from developing countries to our educational institutions
• We support other nations in humanitarian emergencies

Scotland's Ambitions: We avoid harm to the development of other nations

What this means:

• We avoid contributing to climate change and environmental damage internationally
• We support social enterprise, investment, innovation internationally
• We trade and invest fairly

Scotland's Ambitions: We support development in other nations

What this means:

• We work to reduce poverty
• We promote equality and human rights
• We promote knowledge exchange and share the experience and expertise of our public, private and community sectors
• We work to improve health outcomes
• We advocate trade to support development
• We support fairness under the law

3.1.8. There are two areas that have so far proved challenging to capture a simple all encompassing Scotland measure:

• An indicator of the range and quality of partnership working in Scotland in support of development in other nations. Partnerships are easy to describe but difficult to define. Measures of the performance are inherently multi-dimensional and difficult to capture in one or two indicators. The pathway We work with partners to build capacity and engage in dialogue on development and human rights was included specifically to capture such partnership activity, especially in relation to Scotland's International Framework.
• An indicator of the value of expertise provided by Scotland across the different policy areas. Capturing the scale and quality of the contribution arising from the provision of expertise is similarly challenging in a measure that should cover all of Scotland's contribution. Simple measures of the time spent by experts supporting development do not reflect their quality and insight and the diversity of these attributes hinders an aggregate measure of their quality.

3.1.9. The first draft framework was discussed in two stakeholder workshops involving Scottish Government policy teams from the International Development policy team and external stakeholders. There was general agreement on the structure of the framework but it was suggested that the framework should also:

• Consider including humanitarian support provided by Scotland;
• Explore what indicators may best reflect the expert input provided by Scotland into the development process;
• Look to include the scale of civil engagement,
• Capture partnership working between national, local government and agencies and civil society organisations;
• Include other potential negative factors such as Scotland's arms production for exports.

3.1.10. These suggestions were taken into account by adding humanitarian support to Scotland's ambitions and additional indicator components. The draft indicator had 25 proposed components, following these internal and external discussions. At this stage it is important to recognise that these measures were conceptual and the process for developing indicators (a defined measure with a specific data source) was the next stage of the process. This is outlined in the next section.

Table 3.2: Initial approach for Scotland's contribution to the development of other nations

Scottish Values	Pathways - What this means for contribution to development of other nations	Rationale for inclusion
We are good global citizens	We work with partners to build capacity and engage in dialogue on development and human rights	Partnership working has a long tradition in Scotland that combines with fair values to engage partners across a wide range of issues[14]
	We support migrant and asylum seeker populations coming to Scotland	Scotland has taken a different approach to welcoming migrant workers and asylum seekers to help create a more diverse and innovative society[15]
	We welcome students from developing countries to our educational institutions	Attraction of a diverse student body is important to HEIs to sit alongside involvement in international development projects and help build soft power in a wider range of alumni[16]
	We support other nations in humanitarian emergencies	Scottish people recognize and are concerned by the plight of people suffering from emergency situations[17]
We avoid harm to the development of other nations	We avoid contributing to climate change and environmental damage internationally	It is vital that Scotland can demonstrate that it has already taken difficult decisions on low-carbon growth and ensuring that Scotland can minimize waste exports to other countries in order to encourage greater investment in low-carbon development[18] Scotland has particular expertise in the development and deployment of renewable power[19]
	We trade and invest fairly	Ensuring fair and open trade with ODA countries by minimizing the impact of agricultural subsidies[20]
	We support social enterprise, investment, innovation internationally	Fair Trade, investment and social enterprise is an important part of Scottish economic development[21]
We support development in other nations	We work to reduce poverty	Poverty reduction is a key policy priority in Scotland [22]
	We promote equality and human rights	Promoting equality, diversity and inclusion are cornerstones of Scottish policies across the public, private and community sectors[23]
	We promote knowledge exchange and share the experience and expertise of our public, private and community sectors	Promoting the sharing and co-production of knowledge through enduring research partnerships is a cornerstone of HEIs approach to working with ODA countries[24]
	We work to improve health outcomes	Promoting the sharing and co-production of knowledge through enduring health partnerships[25]
	We value trade to support development	Open trade with ODA countries drives faster economic development[17]
	We support fairness under the law	Fairness and access to Justice through better community policing has established Scotland as international best practice[26]

3.1.11. For completeness, the final framework is set out in Table 3.3 below. Ten pathways are retained in the final framework. Eleven indicators are proposed although one is retained as a desired measure for inclusion in the future.

Table 3.3: Final Framework for Scotland's contribution to the development of other nations

Scottish Values	What this means for contribution to development of other nations	Linked National Outcome (NPF)
We are good global citizens	We work with partners to build capacity and engage in dialogue on development and human rights	Education, Economy & Human Rights
	We support migrant and asylum seeker populations coming to Scotland	Human Rights
	We welcome students from developing countries to our educational institutions	Children and Young People
We avoid harm to the development of other nations	We avoid contributing to climate change and environmental damage internationally	Environment
	We trade and invest fairly	Economy
We support development in other nations	We promote equality and human rights	Human Rights
	We promote knowledge exchange and share the experience and expertise of our public, private and community sectors	Education
	We work to improve health outcomes	Health
	We value trade to support development	Economy
	We support fairness under the law	Justice

3.1.12. The changes in the scope of the indicator between the draft and final versions have been driven by the need to define indicator components with robust and accessible datasets. This is discussed in more detail below.

3.2. Populating the NPF indicator with components

3.2.1. A practical stage followed the conceptual stage, described above, for populating the indicator with components.

3.2.2. Inevitably, the boundaries between the conceptual issues outlined above and the practicalities of measurement blur - what makes for a good measure of policy coherence does not necessarily mean that robust data is readily available at reasonable cost. We also need to consider the design criteria set out for the development of the NPF indicator, in particular:

• A measure for Scotland, not just the Scottish Government.
• Focused on those policy areas where Scotland has devolved responsibility.
• Based on transparent and international data sources that can be updated in a timely fashion and with minimal cost, so that Scotland's progress can be assessed on a regular basis.
• Provide an indicator for Scotland that can be compared over time and is not dependent on comparison with other countries.
• Be a cardinal indicator where changes in value relate as directly as possible to changes in policy coherence.^[27] There is limited value in selecting an indicator that is not dynamic or is already at a high level and has limited capacity to change.
• Wherever possible not replicate an indicator already part of the NPF.

3.2.3. These criteria indeed set a relatively high bar for the inclusion of indicators. Further discussions were undertaken with Scottish Government analysts on recommended data sources and indicator definitions. These are set out in Annex D, table D.2. Wherever possible we adopted indicator components similar in nature to those used by the CDI or the PCDI.

3.2.4. In almost all cases, any proposed measures that were subsequently excluded were on the grounds of mostly (i) no robust data source available or (ii) potential data is available but only at significant cost to produce or access. This process led to the exclusion of 15 proposed indicator components:

• For twelve it was not possible to access robust data; on a regular basis at reasonable cost
• For three because a single indicator could not be identified that would adequately represent the measure and reliably represent progress for Scotland.

3.2.5. These decisions are set out in Table D.2 in Annex D.The final set of proposed indicators are included in Table 3.4 with the current indicator values and sources of data used to calculate these.

Table 3.4: Final Proposed Indicator component set for the NPF indicator

Scottish Values	What this means for contribution to development of other nations	Proposed Indicator	Source	Value
We are good global citizens	We work with partners to build capacity and engage in dialogue on development and human rights	Scotland's connectedness to ODA recipient countries	Data not yet available[28]	n/a
	We support migrant and asylum seeker populations coming to Scotland	Asylum seekers settled in Scotland per 100,000 population	National Statistics regional data on Asylum support	3,916 in Jun 2018. 72.01 per 100,000 population.
	We welcome students from developing countries to our educational institutions	HE Students from DAC Least Developed Countries / Total Non-EU Students	HEFC data of students from Low income ODA countries studying on HE courses in Scotland as a % of all non-EU students	2.7% in 2018
We avoid harm to the development of other nations	We avoid contributing to climate change and environmental damage internationally	% of total waste treated in Scotland (SEPA) Value of the Low Carbon and Renewable Energy Economy (LCREE) in Scotland	SEPA report for 2016 % of waste treated domestically. BEIS report on low carbon economy	85% in 2016 £11.1bn in 2017
	We trade and invest fairly	Value of Agricultural subsidies	Total Income from Farming estimates for Scotland 2016-18, Jan 2019	£502m in 2018
We support development in other nations	We promote equality and human rights	% of international development funding devoted to vulnerable groups	SG data on annual spent on IDF projects: Percentage of total IDF spending that the sum of projects that report or should report on protected characteristics represent.	38.9% in 2018-19
	We promote knowledge exchange and share the experience and expertise of our public, private and community sectors	Value of R&D contracts in HEIs in partnership with ODA-eligible nations	UKRI Gateway to Research value of research projects that include partner from low income ODA country as % of total	5.46% in 2017
	We advocate trade to support development	Value of goods imports from low income ODA countries	Low income ODA imports of total goods imports Ex-EU in 2018.	5% in 2018

3.3. Calculating a composite indicator for the NPF

3.3.1. The objective of this assignment has been to identify a number of relevant indicator components that together provide a practical measure of Scotland's contribution to the development support to other nations. For the purposes of the National Performance Framework (NPF), this measure needs to be a single composite index. So, the final stage of this process is to determine how this should best achieved using international best practice.

3.3.2. Three international indices - the Campaign for Global Development Contribution to Development Index (CDI) and the Policy Coherence for Development Index (PCDI) (both reviewed in section 2) and the United Nation's Human Development Index (HDI) used by the United Nations Development Programme^[29] have developed methodologies for translating their individual indicators into a single index value.

3.3.3. The details of this process are included in Annex D. Each index adopts a slightly different approach, but there are a number of common stages:

• Techniques for addressing any missing values and data outliers;
• Standardised performance measures on a common scale to combine values to reflect the degree of change;
• Combine these values using weights to reflect relative importance of a component to international development.

3.3.4. The range of data collated on different countries by each of the indices is crucial to their approach to each stage:

• Missing values are often substituted using an average/benchmark value from other countries. Such averages are also used to identify and adjust any data outliers that may represent inaccurate or unreliable data;
• Both CDI and PCDI and to a lesser extent HDI use the minimum and maximum measured values for specific indicators to benchmark each country's actual value.
• PCDI uses the range of data from countries to statistically weight the importance of different indicators for policy coherence. However, as with CDI the final weighting is set by expert judgement.

3.3.5. For the NPF index, where there is only data available for Scotland, this approach has a number of implications:

• There is no need for a missing value approach as the selected indicator components were chosen because robust, regularly updated data sources are available.
• It would be possible to create benchmark values in order to normalise the individual variables, for example, using EU28 values to compare with Scotland. However, this would contravene a key design principle that the NPF indicator should not be reliant on comparisons with other countries^[30] and most importantly, it would limit the amount of data available further.

3.3.6. This suggests that the most straightforward approach is that the NPF indicator takes a year-on-year change in the combined indicator components as a measure of progress. The baseline year of this NPF indicator will have a value of 100. Subsequent scores will move in line with percentage change on this baseline value - for example, a 4% increase would provide a value of 104 and a 4% decline 96%.

3.3.7. Finally, this leaves the question of whether the individual pathways in the framework should be given different weight in calculating the overall contribution to the development of other nations. Both CDI and PCDI have adopted expert judgement in providing differential weights to their elements. These have been criticised in the literature as being opaque and difficult to interpret. We recommend that no weighting be applied to the different pathways in the framework at this stage, simply because we can find no robust evidence to suggest that any one element is more important than another. Should such evidence become available in the future then this recommendation should be revisited.

3.3.8. The combination of indicators should be undertaken using a geometric mean (rather than an arithmetic mean, for example). This is method adopted by the UNDP's HDI as it does not allow significant changes in one indicator to dominate the index as a whole and so indicators with very different scales can be combined. The geometric mean of n indicators is defined as:

ⁿ√(I₁ x I₂ x …..x I_n)