Impact assessment in governments: literature review

Key messages from this review

This literature review was commissioned by the Scottish Government to inform their approach to impact assessment. We reviewed the literature regarding five types of policy level impact assessments (environment, equity, health, regulatory, rural) in five countries (Ireland, Netherlands, New Zealand, Sweden and Wales). These countries were most likely to require regulatory impact assessment, and least likely to require rural proofing.

More than 1000 potentially useful documents were identified using search engines. Of these, more than 110 plus legislation and guidance informed this report. Much of the literature is somewhat dated; relies on a limited number of case studies; and is carried out by academics who may be testing a hypothesis rather than presenting a balanced view. As such, the findings of this research need to be taken with caution.

What types of assessments are carried out? Scotland has more different types of impact assessment than any other countries studied. New Zealand has climate impact assessment and Wales has wellbeing of future generations assessment, neither of which is carried out in Scotland. Several countries have integrated impact assessments (e.g. Ireland's RIA, Wales' wellbeing assessment).

What assessment systems are particularly interesting? Welsh wellbeing assessment is interesting because it covers a wide range of impacts, is clearly future-looking, and seems to have strong government support. US 'environmental justice' assessment brings together environmental, health and equality dimensions, and seems effective at leading to changes. These assessment systems apply at the programme or plan level, rather than at policy level.

Are assessments actually carried out? Legally-required assessments are generally carried out, but based on evidence from this review many seem to be a formality, carried out late and/or with little influence on policy-making. However, gaps in the literature were identified relating to the timing of actual assessments and how their findings are used in policy-making.

How effective are the assessments that are carried out? In terms of:

• changes to policies – assessment effectiveness is mixed/limited
• public participation – this is very important for transparency and policy improvement. In practice public engagement is limited, but stakeholder engagement is more common.
• knowledge and learning – there is often learning by policy-makers, with consequent long-term organisational change
• costs v. benefits – not enough information exists to be able to come to a conclusion

In particular, even where an impact assessment does not lead to changes in a policy, it can have benefits in terms of improved transparency and accountability of decision-making, increased awareness of the public, and increased trust between stakeholders.

Is integration of impact assessments advisable? Integration of impact assessments – for instance bringing together environmental, social and economic impact assessment into a 'sustainability assessment' - may promote a more holistic approach to assessment, but care needs to be taken in terms of which elements get the most emphasis. Integration is not just a matter of new legal requirements and guidance: it involves issues of data availability, the number of indicators to use, terminology and frames of reference, build-up of expertise, and intersectoral cooperation. The level of integration depends on issues like what minimum standards or thresholds must be achieved and what trade-offs are permitted.

There is also the 'detail paradox', which states that the power of each objective diminishes with the addition of other objectives: in other words, the more detailed the assessment is, the less significance, on average, is attached to each detail.

What are preconditions for effective assessment? In rough order of importance:

• High-level commitment and supportive organisations
• Policy-makers' willingness to learn and change in response to the assessment findings
• Legal requirement for the impact assessment to be carried out
• Oversight and quality review of the assessments
• Fitting the assessment to the decision in terms of timing, types of alternatives considered, recommendations etc.
• Involvement of the public/stakeholders
• Starting the impact assessment early in the policy-making process
• Adequate funding
• Adequate data and expertise
• Collaboration and information sharing between assessors and government departments
• Follow up to check whether the policy incorporated the assessment recommendations, whether the assessment adequately identified impacts, and how the assessment process can be improved