Evaluation of police and fire reform year 4: international perspectives

5 Measuring impact of police reform 

Key findings: Measuring impact

• Police forces in each of the international case study locations found it challenging to attribute any improvements in police performance to reform due to the indicators used to measure performance not relating to reform. To address this, the New Zealand police force has reviewed, reduced and redrafted their performance indicators to better measure the impact of reform.
• All case study police forces think it will take more time to see the full impact of the benefits of reform than was originally anticipated given the complexity of the changes required
• Securing support for building in evaluation to police processes across all levels of seniority was a challenge associated with measuring the impact of reform, highlighted in both New Zealand and Norway.

Police forces found it challenging to evidence systematically (rather than anecdotally) the positive impact of partnership working.

This chapter explores both how the impact of reform is being measured in the four international case studies and common challenges experienced in evidencing the outcomes of reform.

In Norway and the Netherlands, evaluations of the overall impact of police reform are being carried out. In both countries, a government directorate or state committee is responsible for carrying out the evaluation. In addition, external research is carried out by universities and academic departments into specific elements of reform.

In New Zealand and Manchester, some elements of reform have been evaluated but neither police force has undertaken an overall evaluation of the impacts of reform. 

Evaluating the impact of reform on staff and workplace culture was discussed by interviewees from Norway and New Zealand. Both police forces ran surveys of internal staff to explore the experience of reform from the point of view of their staff members.

5.1 What challenges and opportunities are there in measuring impact?

5.1.1 Using existing performance indicators

Interviewees from all four international case studies mentioned the limitations associated with using their routine data, which they collect as part of their monitoring processes on police performance, for evaluating the impact of reform. One particular challenge identified is the ability to attribute any improvements in police performance to reform.

Traditional forms of monitoring police performance, such as crime rates and detection rates, were seen as a barrier to effectively measuring the impact of new ways of working and as running counter to the ethos of reform and innovation.

“Because traditionally what we see in the police…we’re kind of guilty of this is an emphasis on just bean counting, if you like, so how many burglaries have we had? How many have we detected? And actually when you study that, you see the kind of perverse incentives that creates within the system. It means that people just chase the numbers or just chase the easy detections because all they’re being measured on is those easy to count things.” (Interviewee: Manchester)

5.1.2 A review of performance indicators is needed 

There was a view in Manchester, the Netherlands and New Zealand that it is important to review the types of indicators they are using to measure performance. Interviewees discussed the need to introduce a new type of monitoring framework that would include new indicators which are better suited to assessing the impact of reform.

In New Zealand the police force has already reviewed their indicators and have drastically reduced and simplified their outcome measures to reflect the aims of reform. They have reduced around 2,000 performance indicators down to four outcome measures related to the main aims of the police service. This has also led to more autonomy at the local level in how to achieve these overall outcomes. 

5.1.3 Timescales

Another challenge which was identified in evaluating the impact of reform were the timescales over which the impact of reform can be identified. An overall evaluation of reform in the Netherlands police force concluded that the five-year period since the start of the reform was too short a time to fully understand the impact of the reform and recommended that a further evaluation should be commissioned. This reflects the more general observation that reform is a journey rather than an event and therefore it will take time to evidence its different impacts and implications.

5.1.4 Making effective use of evaluation findings

Securing support for building in evaluation to police processes across all levels of seniority was a challenge associated with measuring the impact of reform, highlighted in both New Zealand and Norway. 

“The top leadership in the Police sort of brush it to the side and…er... “It doesn’t have much to do with what we're doing” and so on. They don’t take it much seriously actually!” (Interviewee: Norway)

When discussing evaluating the outcomes of reform, particularly those relating to partnership working or prevention, interviewees focused on small scale evaluations of individual projects. An example of this was a cost-benefit analysis carried out by Greater Manchester Police Force of their place-based partnership working.

“And we do…we have got um…a cost benefit analysis tool that we developed so for each of the new ways of working that we do have locally, we’ve got a tool that we can run that through which sort of says what was demand before, how many agencies was it touching before, and what was the cost of that? So we know for example, the integrated neighbourhood working that we did…, they will say that for every £1 invested they save £3.23 I think it was, that sort of return on investment, and that’s not new money, that’s just using what we’ve currently got.” (Interviewee: Manchester)

There were examples of small-scale evaluations of partnership working initiatives given from Norway about reducing alcohol-related road traffic accidents and responding to people who present in mental health crisis in public places. 

One of the main challenges of carrying out small scale evaluations of projects was described by one interviewee from Manchester, who highlighted the challenge of understanding wider impacts of local initiatives and transferring learning from small projects and applying these more generally to policing.

“So we really only measure the easy stuff. How do we go and measure that on an individual level, at the most local level? What does that tell us about what we need to do to improve and how can we aggregate those individual measures up to show the impact to the wider community?” (Interviewee: Manchester)

5.1.5 Evaluating partnership working

Another challenge associated with measuring impact through formal evaluation, which was identified in Manchester, was evidencing the positive impact of partnership working. 

“Yeah. I mean, you know...so some o' the challenges in policing, we seem really focused on sort of evidence-based policing...You know, you can't really do a sort of significant evaluation on some o' this stuff. It's quite low-level. Some of it's about the relationship that the police deal with local groups as opposed to the activity as well, so some of it's quite intangible.” (Interviewee: Manchester)

Evaluation of partnership working was also viewed as a challenge in Scotland, the thematic case study of partnership, innovation and prevention,^[19] which formed part of this evaluation programme, found little evidence of systematic, independent evaluation of partnership activity in Scotland, although recent programmes of research supported by SIPR have begun to address this^[20]. However, there is a wider UK and international research literature^[21] that has provided some independent evaluation of partnership working involving the police which suggests a significant shift in police attitudes towards partnership working overtime. Among the emerging findings from this research is evidence that:

• The police increasingly see the advantages of partnership working because it allows for a more effective and pragmatic method of addressing social problems, encouraging longer term preventative work rather than short-term reactive approaches;
• Partnership working encourages strong inter-professional and inter-personal relationships with individuals from other organisations, building trust between police and other agencies to help address local problems; and
• While the police may often dominate many partnerships in terms of the resources they are able to mobilise, they increasingly engage in negotiation and compromise with other agencies as they learn to work together to adopt a problem-solving approach.