Evaluation of Police and Fire Reform: Year 1: Annex 1: Evidence Review

1. Executive summary

Introduction and context

• This Annex to the Year 1 report is part of a four year evaluation commissioned by the Scottish Government to examine whether the aims of Police and Fire reform have been met and consider what lessons might be learnt for any future public service reforms.
• The evaluation began in February 2015, and is being delivered by the Scottish Institute for Policing Research ( SIPR), ScotCen and What Works Scotland ( WWS).
• This annex provides a high level summary of the key themes emerging from the 'evidence review' element in year one of the evaluation. It takes into account publicly available evidence produced to the end of November 2015. This annex, therefore, provides a full account of the evidence outlined briefly in the separate 'Summary' document.
• The evidence review has two components. The first is a process of mapping and scoping the evidence landscape, creating a searchable evidence database. The second component is a narrative summary of these individual sources. It is the second element which is presented in this annex.
• Overall, the year one evidence review aims to describe and assess the evidence base that exists in relation to the reforms, and to summarise emerging substantive claims and conclusions.

Achieving the aims of reform: an overview of the evidence

• For each of the three aims of reform, evidence has been brought together to provide a comprehensive picture of progress in terms of developments in policy and practice.
• With reforms as complex as those which Police and the Fire and Rescue service are currently engaged in there will always be evidence gaps and challenges around disentangling the effects of specific initiatives on particular outcomes.
• This overview of the evidence recognises these complexities and challenges but also suggests how these might be addressed going forwards.
• It is also clear that there is much evidence which falls outside the immediate area of the aims of reform, but nevertheless relates to issues and factors that are indirectly connected to them. Such evidence includes that which relates to workforce; estate or procurement for example. Indeed, both Police Scotland and SFRS have undertaken extensive evidence gathering in these areas. Evidence reviews in subsequent years may take a more thematic approach. Further work to be undertaken in subsequent annual reviews includes extending the remit to include non-published evidence.

Aim 1: To protect and improve local services despite financial cuts, by stopping duplication of support services eight times over and not cutting front line services

• Many of the changes associated with reform potentially impact on issues linked to the protection and improvement of services.
• This means that the evidence base is particularly diverse, and can be broken down into four 'types': (1) benefits realisation reporting; (2) performance reports, annual, strategic and inspectorate reports; (3) public perceptions data and (4) reports on strategic incidents and issues.
• Significant evidence has been generated regarding the ways in which both services are reducing the duplication of activities but there are also differences in terms of the availability of data on local perceptions of the respective services.
• National level performance statistics tend to show a fairly consistent trajectory of maintainence/improvement. However local level variations - and variations within particular typologies of incident/activity - are important to consider in the extent to which (for example) services are perceived and experienced to have been maintained and/or improved.
• Key evidence 'gaps' which the evaluation will help to address include a more detailed assessment of perception of front line service outcomes at a local level, and greater clarity regarding cause and effect (e.g. the extent to which an intervention was responsible for a change - or no change - with regard to front line services).

Aim 2: To create more equal access to specialist support and national capacity - like murder investigation teams, firearms teams or flood rescue - where and when they are needed

• The evidence base regarding this aim is concentrated in less diverse documents than is the case for Aims (1) or (3).
• Police Scotland's Post Implementation Benefits Review work concludes that the 'Operational Benefit' relating to this aim has been met. HMICS Local Policing + reports also identify that there is increased ease of access to national resources.
• In the case of Fire and Rescue, work by HMFSI and SFRS has established a detailed knowledge of geographical variations in equipment, capacity and skills and progress is reported to be made in addressing these regional variations.
• In relation to Policing, the evidence base is most extensive around process-based and transactional issues relating to the functioning of the new arrangements. For Fire and Rescue, there is extensive evidence of detailed consideration of variations in baseline resources and risk profiles in preparation for a more strategic approach to distributing specialist resources.
• For both services, the evidence is more limited in relation to outputs and outcomes; causal connections and inter-dependencies when evidencing progress toward this aim are not always clear, and other 'voices' within the services and communities about the impacts and implications of activities undertaken in relation to this aim need to be heard.
• Subsequent stages of this evaluation may focus on some of these evidence gaps - in particular, around perceptions of how far this aim has been achieved (and experiences of this), and understanding causal connections between service reconfiguration and specific outcomes.

Aim 3: To strengthen the connection between services and communities, by creating a new formal relationship with each of the 32 local authorities, involving many more local councillors and better integrating with community planning partnerships

• Evidence in relation to this aim is largely derived from descriptions of the establishment - and to a lesser extent, functioning - of local scrutiny and engagement arrangements and production of Local Police Plans, Multi Member Ward Police Plans and Local Fire and Rescue Plans.
• There is also some evidence regarding the nature of and approaches to community engagement taken by the services.
• Overall, evidence in relation to this aim is disparate and the causal links between the activities of reform and intended outcomes are not fully developed.
• There is also a lack of specificity around some key themes. For example, diverse 'communities' are often referenced in evidencing progress towards this aim but it would be helpful to have further evidence of how the services are strengthening their connections with different types of community and - crucially - the experiences and impacts of this. There is also a lack of evidence from specific communities about their interactions and relationships with the Police and Fire and Rescue services post reform.
• Overall, as is the case for aims 1 and 2, there is a need here to shift the evidence base from documenting the establishment, functioning and outputs to evidencing and understanding outcomes and impacts.

Complexities of the evidence base

• Evidence sources can give us important information about how far the aims of reform might be supported or hindered. Often this evidence may span a number of aims of reform and a range of wider issues rather than neatly addressing one single aim.
• In our review of evidence we have therefore ensured we have not limited the study just to those documents which engage explicitly with the three aims of reform and we have also recognised the importance of considering individual pieces of evidence in their entirety and assessing their strengths and limitations.
• We present a small number of 'evidence in focus' concise case studies, which summarise the aims, content, strengths and limitations of a range of evidence types across the two services.

Year one evidence review conclusions and next steps

• In terms of the overall publicly available evidence landscape, within each aim there are variations in the scope, depth and quality of evidence.
• There are similar strengths in the evidence base across the three aims. There is good documentation of the existence of new governance structures, processes and frameworks in order to deliver the aims of reform, and extensive quantitative performance data and transactional indicators.
• The gaps in the evidence base are also similar, relating to a need to move beyond descriptions of the 'establishment' and 'functioning' of new arrangements, to documenting and understanding local 'outcomes' and 'impacts', and doing so from multiple perspectives.
• Key challenges also include the development of plausible causal chains between activities and the aims of reform, awareness of any unintended and unanticipated consequences and assessing broader impacts on the justice system and wider public sector.
• It is also clear that there is much evidence which falls outside the immediate area of the aims of reform, but nevertheless examines factors that may be critical to achieving the aims of reform. Such evidence includes that which relates to topics including workforce; estate, procurement and Information Communication Technologies ( ICT).
• Overall, at this stage and on the basis of publicly available information, there is plausible and credible evidence of progress being made to achieving the three main long-term aims of reform. However the evidence base tends to indicate 'integration' rather than 'transformation' of service delivery, is process rather than outcome focused, oriented toward 'producer' rather than 'consumer' perspectives, and stronger on national rather than local information. Further, it is important that commitments to develop the evidence base in relation to the three aims are maintained, as there is still scope to enhance the evidence base in ways that will allow a clearer and more robust assessment of claims regarding the achievement of the aims of reform.

• The picture is complex and the evidence presented in this Year 1 report will need to be supplemented by other perspectives (to be addressed in the next stages of the evaluation) before a more comprehensive assessment can be made.