Evaluation of Police and Fire Reform: Year 1: Annex 1: Evidence Review

7. Conclusions from year one of the evidence review, and next steps

KEY POINTS

In terms of the overall publicly available evidence landscape, within each aim there are variations in the scope, depth and quality of evidence.

There are similar strengths in the evidence base across the Aims: there is good documentation of the existence of new governance structures, processes and frameworks in order to deliver the aims of reform, and extensive quantitative performance data and transactional indicators.

The gaps in the evidence base are also similar, relating to a need to move beyond descriptions of 'establishment' and 'functioning' to documenting and understanding local 'outcomes' and 'impacts', and doing so from multiple perspectives.

Key challenges also include the development of plausible causal chains between activities and the aims of reform, awareness of any unintended and unanticipated consequences and assessing broader impacts on the Justice system and wider public sector.

It is also clear that there is much evidence which falls outside the immediate area of the aims of reform, but nevertheless examines factors that may be critical to achieving the aims of reform. Such evidence includes that which relates to workforce; estate or procurement for example.

Overall, at this stage and on the basis of publicly available information, there is plausible and credible evidence of progress being made to achieving the three main long term aims of reform. However the evidence base tends to indicate 'integration' rather than 'transformation' of service delivery, is process rather than outcome focused, oriented toward 'producer' rather than 'consumer' perspectives, and stronger on national rather than local information. Further, it is important that commitments to develop the evidence base in relation to the three aims are maintained, as there is still scope to enhance the evidence base in ways that will allow a clearer and more robust assessment of claims regarding the achievement of the aims of reform.

The picture is complex and the evidence presented in this Year 1 report will need to be supplemented by other perspectives (to be addressed in the next stages of the evaluation) before a more comprehensive assessment can be made.

7.1 Conclusions

It is clear that there is evidence which supports claims that each of the three aims of reform is partially realised. In terms of the overall evidence landscape, within each aim there are variations in the scope, depth and quality of evidence.

It is also clear that there are similar evidence strengths across the three aims. Evidence regarding the existence of governance structures, processes and frameworks in order to deliver the aims of reform is generally in place. Similarly, with regard to Aims 1 and 2 there is quantitative transactional evidence of performance maintenance/improvement. It is clear that there has been much work undertaken in order to allow the forces to move forward with reform, for example baselining and risk profiling areas.

There are also similar evidence gaps across the three aims. These tend to relate to the need to move beyond evidencing 'establishment' and 'functioning' to understanding the 'outcomes' and 'impacts' of efforts to achieve reform. In so doing, understand the consequences (intended and unintended) of reform, and different experiences/perceptions from different perspectives. Further, there appears a general lack of data regarding attitudes and perceptions toward the efficacy (or otherwise) of reform.

There are challenges around the nature of the available evidence base. In the context of the broader aims of this evaluation, there has been limited work to date that maps the activities beneath all three aims of reform onto the activities of the wider Justice system, or seeks to understand in a systematic way the wider impacts of reform on partnership activities across community planning structures from the perspectives of multiple stakeholders. There are also challenges in establishing causal relationships between the indicators employed and the outcomes beneath which they are collated in order to evidence. This can influence the confidence with which claims might be made on the basis of the evidence available. We also note that much evaluation and ongoing monitoring is planned from both Police and Fire and Rescue Services.

It is important to be clear about the remit of the aims, and therefore the nature of the evidence required to demonstrate progress (or otherwise) toward their delivery. This is to some extent linked to the nature of evidence available. The aims are very specific, for example with Aim 1 stating that it seeks 'to protect and improve local services despite financial cuts, by stopping duplication of support services eight times over and not cutting front line services'. This raises questions regarding the evidence used. For example, are we only interested in evidence directly related to interventions that have protected and improved local services as a result of i) the prevention of duplication of support services and ii) not cutting front line services' or are we interested in interventions that may have protected and improved services through means other than preventing duplication of support services and not cutting front line services? We would argue that drawing causal links between interventions and outcomes is particularly challenging and therefore it is important to take account of the broader context within which indicators sit, and the factors which may influence them.

It is also clear that there is much evidence which falls outside the immediate area of the aims of reform, but nevertheless demonstrates fundamentally important factors in achieving the aims of reform. Whilst this evidence does not fall neatly within any of the three aims it is clearly relevant, and subsequent evidence reviews will develop in such a way as to take this evidence into account.

Overall, at this stage and on the basis of publicly available information, there is plausible and credible evidence of progress being made towards achieving the three main long-term aims of reform. However the evidence base tends to indicate 'integration' rather than 'transformation' of service delivery, is process rather than outcome focused, oriented toward 'producer' rather than 'consumer' perspectives, and stronger on national rather than local information. Further, it is important that commitments to develop the evidence base in relation to the three aims are maintained, as there is still scope to enhance the evidence base in ways that will allow a clearer and more robust assessment of claims regarding the achievement of the aims of reform.

The picture is complex and the evidence presented in this Year 1 report will need to be supplemented by other perspectives (to be addressed in the next stages of the evaluation) before a more comprehensive assessment can be made.

7.2 Next steps for the evidence review and evaluation

In part, this annex should be used to help further identify evidence held and published internally or externally which might fill the gaps identified and/or provide a fuller account of the evidence presented within each Aim. The geographical case studies and thematic case studies will be informed by the evidence gaps identified in year one's evidence review, and the themes emerging from the key informant interviews.

Future evidence reviews may consider a broader range of evidence, outwith the immediate aims of reform, recognising the ways in which the achievement of the aims of reform are contingent on other factors (such as workforce, for example). There is also further work to be undertaken in subsequent annual reviews in extending the remit to include non-published evidence.

There are a range of further elements to this evaluation (also summarised in figure 1):

• Key informant interviews (years one and three);
• Geographical case studies (years one and three);
• Thematic case studies (year three);
• International comparative workshops (years two and four);
• Annual evidence reviews.

The report of which this annex is part includes a summary of the themes emerging from the key informant interviews. These interviews with key informants operating at a national level are crucial to gaining additional insights into 'how' and 'why' the aims of reform have (or have not) been met. The four year nature of the evaluation will allow this evidence review to be used as a baseline against which progress toward the aims of reform - from multiple perspectives and at local and national level - might continue to be assessed.