Evaluation of Police and Fire Reform: Year 1: Annex 1: Evidence Review

2. Introduction and context

KEY POINTS

This Annex forms part of a four year evaluation commissioned by the Scottish Government to examine whether the aims of Police and fire reform have been met and consider what lessons might be learnt for any future public service reforms.

The evaluation began in February 2015, and is being delivered by the Scottish Institute for Policing Research ( SIPR), ScotCen and What Works Scotland ( WWS).

This annex provides a high level summary of the key themes emerging from the 'evidence review' element in year one, taking into account publicly available evidence produced to the end of November 2015.

The evidence review has two components. The first is a process of mapping and scoping the evidence landscape, creating a searchable evidence database. The second component is a narrative summary of these individual sources. It is the second element which is presented in this annex.

Overall, the evidence review aims to describe and assess the evidence base that exists in relation to the reforms; and to summarise emerging substantive claims and conclusions.

2.1 Purpose and nature of the evaluation

The Police and Fire Reform (Scotland) Act was passed in summer 2012, bringing a major programme of reform to both services in Scotland with the creation of single Police and Fire services. The new organisations became operational on 1 April 2013. The three aims of reform identified in the policy memorandum accompanying the act were to:

• Protect and improve local services despite financial cuts, by stopping duplication of support services eight times over and not cutting front line services;
• Create more equal access to specialist support and national capacity like murder capacity teams, Firearms teams and flood Rescue where and when they are needed;
• Strengthen the connection between the services and the communities they serve, providing an opportunity for more local councillors to be involved in shaping local services and for better integration with community planning (Scottish Parliament 2012).

This four-year project was commissioned by the Scottish Government to evaluate whether these aims have been met, and what lessons might be learnt for any future public service reforms. Specifically, the evaluation is seeking to:

• Evaluate if the three policy aims listed above have been met;
• Learn the lessons from the implementation of reform to inform the process of future public service reform;
• Evaluate the wider impact of the reform on the Justice and public service system.

The evaluation began in February 2015, and is being delivered by the Scottish Institute for Policing Research ( SIPR), ScotCen and What Works Scotland ( WWS). The evaluation has six main components (summarised in figure one).

Figure 1: Components of the evaluation

Task	Activities
Scoping phase	Introducing the evaluation to key stakeholders and organisations; identify key data/reports produced by relevant organisations; and identify current and future research and evidence plans of the organisations to help inform evidence review.
Evidence review	Gather information that can tell us more about the current 'picture' of Police and Fire Reform in Scotland. We will refresh this each year to keep the review up-to-date.
Key informant interviews	Identify key stakeholders in Scotland, and invite them to speak with us about their experiences of Police and Fire Reform focussing on how far they feel aims of reform have or haven't been met, and what the key lessons to learn from this might be.
Geographical and thematic case studies	Work in four areas across Scotland to understand local level experiences of, success factors for and barriers to Police and Fire Reform. Identify two key themes about which we will undertake more in depth 'case study' work.
Events and workshops	Run two international workshops to compare what is being learnt in Scotland with other countries that are also experiencing reform of their Police and/or Fire Services. Run workshops to help 'validate' our findings.
Report writing	This work will result in reports which will be submitted to the Scottish Government

2.2 Broader context of the evaluation

Following the passing of the Police and Fire Reform (Scotland) Act in 2012, the resulting reforms to the organisation, governance and delivery of Police and Fire and Rescue services are some of the largest and most complex changes to the public sector in Scotland for a generation.

In terms of policing, the Act brought together the 8 regional Police forces, the Scottish Police Services Authority and the Scottish Crime and Drug Enforcement Agency into two new national bodies: Police Scotland and the Scottish Police Authority. 'Local policing' became a statutory requirement at the level of the 32 council areas with local councils responsible for establishing local scrutiny arrangements. The Act also sets out a normative vision for policing in the form of a set of 'principles' focussed on community well-being, partnership working and harm reduction. In relation to Fire and Rescue services, the Act established the Scottish Fire and Rescue Service ( SFRS), replacing the eight former Fire and Rescue services and the Scottish Fire Services College. The Act also makes new arrangements for local engagement and partnership working, including a new statutory role in the Local Senior Officer and the development of local Fire and Rescue plans linked to community planning. As with policing, the purpose of SFRS has been articulated in terms of working in partnership on prevention, protection and response, and improving the safety and well-being of people in Scotland.

Both sets of reforms are set within a context of decreasing budgets and involve making significant financial savings in relatively short timescales. Police reform is forecast to save more than £1.1 billion over the 15 years to 2026 while SFRS is expected to save £328 million by 2027/28. Both Police Scotland and SFRS are also seeing important shifts in demand for their services. Although there has been a long term reduction in volume crime in Scotland (with recorded crime at a 41 year low), there are other areas where demand is increasing. These include the increased reporting of sexual offences, growing threats in relation to cyber crime and terrorism, and the impacts of broader demographic, environmental and policy changes that have consequences for policing. In relation to the Fire and Rescue service, there has been a long-term reduction in the number of Fires and Fire-related casualties and there are now on-going reviews of how SFRS delivers services in the future. These include exploring opportunities to be involved in a broader community safety role, mapping the distribution of resources in relation to risk, examining the future configuration of the retained duty system, and volunteer Firefighters who provide emergency response in many of Scotland's rural and remote communities.

A key difference between the services in the period since they were established has been the level of political and media scrutiny they have experienced. While the new SFRS has received relatively little political or media attention, Police Scotland has been the subject of intense interest in relation to the new arrangements for Police governance and its approach to delivering policing in local communities. In terms of governance, there were initial differences between the Chief Constable and the Chair of the SPA regarding the exact remit of SPA's responsibilities. In addition, the Scottish Parliament's Justice Committee established a Sub-Committee on Policing in March 2013 which has provided its own scrutiny of the implementation process. In terms of the approach to policing, the first eighteen months has seen significant attention given to a number of decisions regarding the use of tactics such as stop and search and the standing authority for trained Firearms officers to carry their weapons in public while on patrol. Scrutiny of these decisions has come in the form of inquiries established by SPA, HMICS, and debate within the Scottish Parliament's Sub-Committee on Policing.

A final observation on the context of these reforms is that Scotland is not alone in undertaking major structural reform of policing. Over the last 10 years, Denmark, Finland, Norway, the Netherlands and Sweden have all embarked on significant administrative re-organisation of their Police forces, typically involving the merging of Police districts to create more centralized strucutres. The Netherlands in particular has followed a very similar journey to Scotland by establishing a national Police force in January 2013 through the merger of 25 regional forces. Most of these countries are also currently engaged in evaluations of Police reform, creating significant opportunities for international learning and knolwedge exchange.

2.3 Purpose of this annex

This annex provides a high level summary of the key themes emerging from the 'evidence review' element of the work at the end of year one of the evaluation. It takes into account publicly available evidence produced to the end of November 2015. In part, this annex should be used to help further identify evidence held and published internally or externally which might fill the gaps identified and/or provide a fuller account of the evidence presented within each Aim.

2.4 Approach to the evidence review

The purpose of the year one evidence review is to draw together in one document what is currently known about how the aims of Police and Fire Reform are being addressed and the extent to which these aims have been met. In the first year it focuses on bringing together the various publicly available reports identified through the scoping phase of the research, presenting the key themes, findings and gaps to help inform subsequent stages of the evaluation. Specifically, the purposes of the evidence review are:

• To map the existing evidence landscape in relation to Police and Fire reform;
• To assess the quality of existing evidence;
• To begin to draw tentative conclusions about key substantive questions (relating to the aims of reform);
• To highlight areas in which the evidence base needs to be supplemented or improved (either as part of the evaluation itself or through feedback to relevant agencies involved in the production of existing evidence.

The evidence review has two main components. The first is a process of scoping and mapping the evidence landscape - identifying, assessing and summarising individual data sources. The output from this exercise has been captured in a directory of evidence - a searchable database containing a separate record for each document or data source included in the review. The second component is a narrative summary and analysis - a reading across rather than within - of these individual sources. It is the second element which is presented in this annex.

The 'evidence' included in the year one of the review consists largely of published reports by public bodies associated with the implementation of the reforms or with their oversight up to the end of November 2015. We draw a distinction here between data/information and evidence and regard the latter as involving a degree of analytical processing. As such, our review does not include all available information about Police and Fire reform, but those existing attempts to collate, synthesise, analyse or understand what a particular dataset may be telling us.

2.5 Reflecting on evidence quality

It is important to recognise that, as with any evidene review, we must be aware of evidence quality and to be clear about what it is possible to claim on the basis of the evidence available. In defining and assessing the quality of evidence we recognise that, as Nutley et al (2013) conclude, 'there is no simple answer to the question of what counts as good evidence. It depends on what we want to know, for what purposes, and in what contexts we envisage that evidence being used'. We recognise too that there will be conflicting views about the merits of different forms of evidence. In order to engage with these issues, we have identified a number of critiera which we consider when reviewing evidence and at the beginning of sections 3 - 5 we provide a brief overview of the quality of the evidence landscape to aid interpretation of our conclusion. These are as follows:

• Methodological adequacy. This includes consideration of how the data on which the source is based has been collected; how far it is possible or appropriate to generalise from these data to wider populations or contexts; and/or what potential sources of bias there may be and to what extent these have been controlled for. Stronger evidence might include, for example, the Scottish Social Attitudes Survey or Scottish Crime and Justic Survey which are statistcially robust and methodologically rigorous (although also has limitations, for example the extent to which it can tell us about very local level attitudes and perceptions). Less strong evidence might include that which draws on a smaller or partial sample size and/or has limited generalisability.
• Accessibility. This includes consideration of (for example) how far the data and conclusions are publicly accessible; how easily navigated and appropriately presented they might be and the extent to which they draw on or make reference to data sources that are unavailable or unpublished.
• Relevance. This includes consideration of how far individual sources address the aims of reform; whether they do so directly or indirectly and the extent to which they shed light on key issues such as geographic variability or change over time?
• Independence. This includes consideration of the source of the evidence; the nature of that body's involvement in the reform process; and how far there might be any reason to think that the evidence might be incomplete or presented in a particular light. Clearly, all evidence is produced for particular purposes from particular pespectives. However, examples of evidence which might be considered more independent include (for example) the work of Audit Scotland. Evidence can also be produced by organisations which might represent a particular community (for example COSLA) or be written from a particular perspective (for example the Scottish Labour Party report on police reform, known as the Pearson Review and led by Graeme Pearson MSP, Labour). It is important to be aware of the role that such different perspectives may play when coming to a view on degree of independence of evidence.
• Analytical adequacy. This includes consideration of how far the source seeks to draw conclusions from the evidence presented; the extent to which these appear to be appropriately grounded and how far the source draws on either an implicit or explicit 'theory of change' - i.e. an account of how mechanisms might operate in specific contexts to produce particular outcomes?

2.6 Structure of the annex

This annex first summarises the evidence strengths and weaknesses as they relate to the three aims of reform ( sections 3- 5). It then explores in more detail a small selection of evidence sources in their entirety ( section 6). Finally, implications for subsequent stages of the evaluation are identified in section 7.