Fireworks legislation and impacts: international evidence review

Appendix B: Methods

To conduct this review, a systematic process of search and assessment was followed, involving four broad stages:

1. Evidence search

2. Application of inclusion and exclusion criteria for assessing relevance

3. Quality assessment of studies

4. Synthesis of the body of evidence

The details of stages 1-3 are described below. The synthesis of evidence is presented in Section 4.

6.2.1. Search

In the first instance, the search for studies was carried out by the Scottish Government Library Service using KandE. KandE is an online search engine which covers a range of high quality databases, which are detailed below.

Table 3: List of databases searched

This search was informed by a range of key words and phrases, including 'fireworks' combined with:

• Sale
• Licencing
• Impact
• Use / misuse
• Anti-social behaviour
• Crime
• Noise
• Air quality / environmental
• Animal welfare
• Regulations / Legislation

To ensure the evidence identified was up-to-date and relevant, the specified time coverage for the search was from 2009-2019. The geographical coverage included the UK, Europe, North America, Canada, Australia and New Zealand. These countries were selected because they are comparable to Scotland in terms of culture, attitudes to health and safety and air quality.

A series of broader searches were then conducted using Google and Google Scholar, as a sweep of studies that may not have been found in the initial search. In addition, a snowballing technique was employed whereby the references of studies were reviewed for additional evidence.

6.2.2. Inclusion and exclusion criteria

Using the initial search results, the relevance of the studies was assessed. The table below provides a summary of the inclusion and exclusion criteria applied to the selection of the studies.

Table 4: Inclusion and exclusion criteria

Applying these criteria led to an evidence base comprising a wide range of sources, including academic journal articles, government reports, surveys, case studies, laboratory experiments, evaluations, evidence reviews, interviews and books.

25 of these studies were based in the UK; 14 in the US; 14 in European countries including Northern Ireland, Hungary, Italy, Malta, Netherlands, Norway, Poland, Portugal and Switzerland; 6 in Australia and New Zealand; 4 internationally and 3 from Asian countries including China, India and Japan. For one study identified, the country of origin was unclear.

6.2.3. Quality assessment

Each of the studies identified was then quality assessed. This involved identifying the key characteristics of the studies and their limitations, which are summarised in Appendix A.

The body of evidence identified in this report consists of 67 studies, many of which used high quality methods. In particular, 28 used quantitative methods such as surveys or analysis of injuries data, including 5 which used nationally representative data. There were also 10 studies based on case studies and/or qualitative methods such as interviews, providing a more in-depth insight into fireworks use.

However, as well as the limitations highlighted in Appendix A, the evidence base suffered from other shortcomings. In particular, there was a distinct lack of evidence based in Scotland or even the wider UK, and it is unclear how findings will apply to the Scottish context. For example, environmental impacts are found to be influenced by a range of factors that vary from one country to the next and so findings from other countries may not apply to Scotland. Further, there was a lack of literature on several themes present in the consultation and omnibus survey, including underage sales and anti-social behaviour.