Impact assessment in governments: literature review

1. Introduction

Scottish policies are currently subject to:

1. Equality Impact Assessment (EqIA), assessing impacts on people with 'protected characteristics'

2. Fairer Scotland Duty assessment, aiming to reduce socio-economic inequality

3. Island Communities Impact Assessment, assessing unequal impacts on island communities

4. Child Rights and Wellbeing Impact Assessment, assessing impacts on children's human rights and wellbeing

5. Strategic Environmental Assessment (SEA), assessing impacts on the environment

6. Business and Regulatory Impact Assessment, assessing costs, benefits and risks

7. Data Protection Impact Assessment for projects involving personal data and privacy

This is in addition to Habitats Regulations Assessment, which has very different legal requirements and testing criteria. Each form of assessment applies to different actions; has a different focus; and has different underlying legislation.

1.1 Aims

This report presents the findings of a literature review of the impact assessment systems of five countries, to inform a review of Scotland's impact assessment system. The review aims to:

• provide the Scottish Government with timely, robust information on approaches to assessing policy impacts in other countries;
• critically appraise the effectiveness, efficiency and transparency of these approaches and any particular strengths or problems;
• consider how individual assessments operate within a wider system of impact assessments;
• identify gaps in the literature; and
• suggest how learning from this review might be applied to enhance the system of impact assessments in the Scottish Government.

The study assumes that the reader is familiar with the aims and main steps of the various types of assessment discussed.

1.2 Methodology

Countries analysed: Five countries were analysed in this literature review: Ireland, Netherlands, New Zealand, Sweden and Wales. They were chosen because they represent a range of different approaches to impact assessment; and have broadly similar areas and populations. They each also exemplify good or interesting practice in one or more forms of assessment:

• Ireland has wide-ranging regulatory impact assessment (RIA) requirements, and a supportive institutional context;
• The Netherlands has traditionally been a strong SEA champion, but this has been subject to 'streamlining' over the years. It has a well-developed system of gender impact assessment, and a robust system of RIA;
• New Zealand's impact assessments are so well-integrated into its policy-making processes that they are difficult to distinguish as separate entities, and it has a new requirement for climate impact assessment;
• Sweden is exemplary in its promotion of gender equality, although it seems less strong in terms of other forms of impact assessment; and
• Wales has a strong equality impact assessment process, and its wellbeing assessments are a model of integrated, future-looking assessments.

Appendix A summarises the impact assessment legislation and guidance available for each country and type of impact assessment.

In all, more than 1000 items of literature were reviewed, and more than 110 (plus web-links to legislation and guidance) are included here.

Search terms used: The literature review involved searching for the terms 'equality impact assessment', 'health impact assessment', 'impact assessment', 'policy appraisal', 'policy assessment', 'poverty proofing', 'regulatory impact assessment', 'rural proofing', and 'strategic environmental assessment' AND the country name, using the search engines Academic Search Complete, Science Direct, Google Scholar and Google. Where this information suggested that additional search terms were needed, these were used. Other literature suggested by this first trawl was also analysed. Where appropriate, other countries' impact assessment systems – environmental justice for the United States, equality impact assessment and rural proofing for Northern Ireland – were also reviewed. In the case of some authors (e.g. Nykvist, Radaelli) who wrote particularly informative articles more than ten years ago, a search for more recent literature by those authors was also carried out.

Policy level assessment: The literature review focused on assessments at the policy level (rather than plan/programme or project) level, as this is the Scottish Government's area of concern. For the academic literature in particular, this meant that many articles were rejected from further analysis, because they focused on other levels.

Timeframe: Literature dating pre-2000 was generally excluded, except where it added something significant. Even then, much of the available literature is more than ten years old. If literature dating pre-2010 was excluded, this would eliminate more than one-quarter of the references.

Topics for analysis: The topics for analysis in this report were suggested after an initial literature review, and were discussed and fine-tuned with the steering group. The subsequent chapters review these topics in turn.

Report structure: The original research aims and objectives were discussed with the project steering group at a meeting of 22 September 2020, and a draft structure for this report was agreed with the team on 21 October 2020. Although the structure differs slightly from that originally envisaged by the Invitation to Tender, it now more clearly focuses on issues of interest to the Scottish Government, including a longer discussion of integration of impact assessments, and of preconditions for effective impact assessment.

Peer review: A draft of this report was peer reviewed. Comments were taken on board and the report has been changed in response.

1.3 Quality of information

This literature review considers five types of impact assessment (environmental, equality, health, regulatory and rural) in five countries, plus other good practice where available: more than 25 combinations of country and assessment type. We have only been able to find limited information about some of these forms of assessment. In some cases the information is more than ten years old, but we have been unable to find more recent information to determine whether the original information is still valid.

For New Zealand, where the literature on impact assessment is limited except on health impact assessment, we contacted an expert who suggested further reading. However the timing and resources available for this research precluded other similar 'in person' requests.

Appendix B summarises the information sources that supports this report, focusing on 'critiques': literature that analyses the effectiveness of an impact assessment system, rather than setting out requirements, guidelines or suggested good practice.

The main data gaps identified in this research were information on:

• rural proofing
• how conflicts between the findings of different types of assessment can be dealt with
• best practice for any of the assessment types
• the process of setting up integrated assessment systems
• timing of actual assessments (as opposed to preferred/ideal timing)
• how impact assessment findings are used in policy-making

Further interviews of impact assessment practitioners or government officials in the countries analysed here would help to explore the issues raised in this report, and update the information available from this literature review.

1.4 Structure of this report

Section 2 summarises the legislation and guidance on impact assessment for each country; and whether/how the impact assessments are integrated.

Section 3 discusses whether impact assessments are carried out as expected: whether they are carried out at all, their timing, and whether they are more than a formality.

Section 4 discusses the effectiveness of the impact assessments in terms of whether they lead to changes to policies, increase public participation or improve knowledge and learning, and data available about the costs and benefits of impact assessments.

Section 5 considers integration of impact assessments, and how tensions between different forms of assessments are handled in practice.

Section 6 discusses preconditions for effective assessment, and lessons for future assessments.