Evaluation of Police and Fire Reform: Year 1: Annex 1: Evidence Review

6. Broadening and deepening the evidence review

KEY POINTS

Evidence sources can give us important information about how far the aims of reform have been achieved. Often this evidence may span a number of aims of reform and address a range of wider issues rather than being focused on specific areas of reform activity.

In our review of evidence we have therefore ensured we have not limited the study just to those documents which engage explicitly with the three aims of reform and we have also recognised the importance of considering individual pieces of evidence in their entirety and assessing their strengths and limitations.

We therefore present a small number of 'evidence in focus' case studies, which summarise the aims, content, strengths and limitations of a range of evidence types across the two services.

Future evidence reviews, as well as the thematic case study phase of the evaluation, will develop this approach further.

6.1 Overview

The main focus of this report has been on aligning existing evidence with the aims of reform in order to provide an assessment of the extent to which the aims have been achieved. However, the evidence base relating to the impacts and implications of reform extends beyond the specific aims to include issues which are still important to understanding the nature, process and impact of reform. It is therefore important to broaden the evidence review in ways which engage strategically with topics that are important to the reform process. In addition the available evidence also needs to be assessed critically in terms of strengths and limitations. This means it is important to deepen the analysis of evidence section in order to come to a robust assessment of its contribution to the analysis of reform. This is done here via a set of 'evidence in focus' concise case studies, which summarise the aims, content, strengths and limitations of a range of evidence types across the two services.

6.2 Broadening the evidence review: the example of workforce surveys

It is clear that there is much evidence which does not directly address the three aims of reform but which is still of fundamental importance to the reform process. One area of strategic importance to both Police and fire and rescue services concerns the attitudes and perceptions of the workforce. SFRS undertook a 'cultural audit' in 2014/15 which provided important insights into the current and ideal 'cultural profiles' of the organisation. It helped distinguish between different organisational cultures in different part of the country and between different roles and ranks within SFRS.

Within Policing, there have also been major - published - pieces of research to better understand the impacts and implications of reform on the workforce. The Association of Scottish Police Superintendents ( ASPS) has commissioned two 'member's resilience surveys'. The 2015 survey finds that with regard to 'issues relating to most recent changes':

• 26% more concerned about career progression than a year ago;
• 39% more worried about personal finances than a year ago;
• 77% said demands of job have increased in the last year;
• 46% said they enjoy their work less than a year ago;
• 23% concerned pension changes mean they must work longer.

Whilst 'particular concerns' are identified as:

• Increased Work Demands;
• Decrease in Pension Provision;
• Reduced Work Resources;
• Personal or Family Security;
• Reduced Career Options ( ASPS 2015).

Many of these concerns were underlined by a broader staff survey commissioned by the Scottish Police Authority and Police Scotland and published in 2015. Focusing on the experience of officers and staff (and with a response rate of 50.4%), key findings included that: 56% of respondents understood the need for change; 46% supported the need for change; 70% of respondents wanted more opportunity to influence decisions made; 12% of respondents felt they had appropriate information on what Police Scotland plans mean for them and 33% of all respondents indicated an intention to leave (with a large proportion citing pension changes as a factor adversely affecting their commitment) (Axiom 2015).

6.3 Deepening the evidence review: assessing strengths and weaknesses of evidence

Although this report has focused on the ways in which individual documents can inform understanding of specific aims of reform, it has not provided a deeper assessment of the specific strengths and weakness of individual pieces of evidence. The following section uses 4 short 'evidence in focus' case studies to provide a more holistic assessment of specific pieces of evidence.

6.3.1 Evidence in focus: evidence from the voluntary sector

Stonewall Scotland: Your services your say (Stonewall Scotland 2014)

Who has produced the evidence and why?

This report has been produced by Stonewall Scotland on the basis of research commissioned to YouGov. It explores the experiences of LGBT people in relation to public services in Scotland. It focuses on health and social care, housing, Policing, family life, post-16 education and training, and the local community.

How has the evidence been gathered?

The report states that the total sample size was 1,043 lesbian, gay, bisexual and trans adults from across Scotland. The survey was conducted using an online interview administered to members of the YouGovPlc GB panel of 350,000+ individuals who have agreed to take part in surveys. Additional open recruitment through Stonewall Scotland was used to achieve the full sample. Fieldwork was undertaken between March and November 2013. The figures were weighted and are representative of Scottish adults by region and age. The resulting data was analysed and presented by Stonewall Scotland.

What can it tell us?

• Many respondents are not confident in Police Scotland's ability to tackle hate crime in their area;
• Respondents also reported lacking the confidence to report hate crime to the Police, and would feel uncomfortable being open about their sexual orientation or gender identity when interacting with them as a victim or suspect of crime;
• More than two in five (42 per cent) of respondents lacked confidence in Police Scotland's ability to address homophobic and transphobic hate crime in their area;
• More than a third (36 per cent) of respondents would not feel confident reporting a hate crime directly to the Police;
• More than a third (36 per cent) of respondents would feel uncomfortable being open about their sexual orientation or gender identity with the Police if they were a victim of any crime.

What appears to work well?

This research appears to help address evidence gaps by engaging with a specific community to understand their perceptions of, and experiences of, Police Scotland. It contextualises Aim 3 of reform by engaging more broadly with the theme of 'community links' beyond democratically elected stakeholders. It is a relatively large scale and statistically robust piece of work.

What might be improved?

The data is not able to tell us about regional variations in these experiences. Nor is it able to specifically tell us about perceptions of reform per se, or allow us to follow changes in attitude/experience over time to give us a picture of the situation pre- or post-reform.

6.3.2 Evidence in focus: evidence from HMFSI

HMFSI: Inspection of East Renfrewshire ( HMFSI 2015b)

Who has produced the evidence and why?

The local inspection evidence is gathered and assessed by HMFSI and is informed by the commitments set out in Fire and Rescue Framework for Scotland 2013 that connections should be strengthened between SFRS and local communities. The process of local inspection is also required to provide the necessary assurance around the satisfactory delivery of local services including whether specialist national resources can be accessed locally, there are good relations with partner organisations, and that there is cooperation with other scrutiny bodies to provide 'place-based' scrutiny of service provision.

How has the evidence been gathered?

A team of 5 led by HM Chief Inspector for the Fire and Rescue Service, visited service delivery locations and spoke to managers and uniformed and non-uniformed staff. The team also viewed premises and equipment and sampled local records to assess how business is being conducted. The gathering and analysis of evidence cross-references the SFRS's written plans and is structured around the Fire and Rescue Framework, focusing on:

• The Local Fire and Rescue Plan and Single Outcome Agreement;
• Improved service outcomes and protecting frontline services;
• More equal access to specialist resources and national capacity;
• Strengthened connection between SFRS and communities.

What can it tell us?

The inspection provides key statistics for the East Renfrewshire area, including numbers of incidents attended and whether these relate to fires, false alarms and road traffic collisions. Key findings to emerge from the local inspection include:

• The SFRS is delivering satisfactory 'response' and 'prevention and protection' functions in an area with low levels of operational activity;
• There are good relationships with the local authority (although the Local Area Liaison Officer has to deal with 3 local authorities) and partners but strategic relationships are weak;
• There is a good level of community relations activity and fire safety enforcement is effective;
• There is scope for Station Managers to be drawn into Scrutiny Committee activity;
• There are specialist resources available in neighbouring areas.

What appears to work well?

The local area inspections provide important insights into local service delivery and act as a very useful mechanism for flagging up issues that might require resolution at a national level. In particular, the inspections provide a 'voice' for frontline staff to highlight matters that concern them and a sense of the way in which national SFRS policies and practices and are impacting locally.

What might be improved?

There is scope for greater transparency around the methodology used in the local areas inspection programme (for example, in terms of numbers of people spoken and their roles/profiles) and also for including other perspectives and viewpoints (for example, from partner agencies, the local authority and local community). A greater focus on gathering evidence of the outcomes of activity and placing this in a comparative perspective so that the performance of one local area can be compared with that of areas with similar demand and risk profiles would also be helpful.

6.3.3 Evidence in focus: evidence from HMICS

HMICS: Local Area+ Inspection of Edinburgh City Division ( HMICS 2015a)

Who has produced the evidence and why?

This evidence has been produced as a result of an inspection by HMICS of Edinburgh City Police division, in the first half of 2015. It forms part of its Local Policing+ inspection programme which aims to assess the state, effectiveness and efficiency of local Policing, under the terms of the Police and Fire Reform (Scotland) Act 2012. Inspections are based on the HMICS framework which considers six overarching themes: Outcomes, Leadership and governance, Planning and process, People, Resources and Partnerships. During the inspection of Edinburgh Division HMICS examined in greater detail the division's approach to partnership working. This provided an opportunity for HMICS to assess the impact of change both locally and nationally in this area of Policing. Local Police custody facilities were also subject to inspection.

How has the evidence been gathered?

The inspection involved a team of eight led by the Assistant Inspector of Constabulary. It gathered evidence from a range of sources including surveys of stakeholders and councillors involved in local scrutiny of Policing; a review of data, strategies, policies and procedures; observation of community council meetings and of divisional processes and meetings, including those done in partnership with other agencies; and over 75 interviews and focus groups involving over 160 officers, staff, partners and stakeholders along with observations of 15 meetings and briefings.

What can it tell us?

The inspection report collates statistics for Edinburgh City such as number of recorded crimes and offences, detection rates, levels of public confidence and satisfaction, and numbers of public complaints. It summarises performance against the local Policing objectives for the division. Some of the key findings from the inspection include:

• Crimes per 10,000 of population in Edinburgh is the highest in Scotland;
• Edinburgh has the lowest detection rates of all divisions in Scotland;
• The division has made positive progress in six out of 15 local Policing plan objectives;
• The division has a good approach to consultation using a broad range of methods to contribute to the identification of national and local priorities;
• The division has supported the development of effective local scrutiny arrangements by raising awareness of wider Policing issues;
• Morale amongst officers and staff is mixed. There are concerns about the impact that increasing demand alongside reducing officer numbers is having on their ability to provide an effective service;
• The division has a good approach to absence management with some of the lowest sickness rates in Scotland;
• Strategic partnerships are good and there is a shared vision for community safety and wellbeing in Edinburgh.

What appears to work well?

The focus of the local inspection on the six themes provides a useful framework for presenting a rounded picture of local service delivery. The inspection offers a view from frontline officers and staff that highlights matters that concern them. It provides a good sense of the way in which national policies and practices are implemented locally and their effect. Specific 'case studies' presented in the inspection report provide a helpful focus on specific issues identified by the inspection.

What might be improved?

There is scope for greater specifics and detail about the methods used in the inspection (for example, how many specific focus groups with what number and kinds of officer, what number of interviews) and more explicit links between findings and the specific evidence for them. There might also be benefit in making greater use of evidence drawn from, for example, local partners, the local authority and local community to broaden what can appear a somewhat 'Police-focused' view.

6.3.4 Evidence in focus: evidence focusing on workforce issues

Axiom: Report for SPA/Police Scotland Opinion Survey 2015. September 2015 (Axiom 2015)

Who has produced the evidence and why?

This report has been produced by Axiom Consultancy - a market research organisation, based in Glasgow, with experience in the field of employee research - and appears to have been commissioned jointly by Police Scotland and the SPA. Roughly two years on from the introduction of the national force, the main aim of the survey was to provide a snapshot of officer and staff engagement. It covered perceptions of a range of issues including information and communications; line management; training and development; overall wellbeing; inclusion and equality; organisational change; organisational purpose and objectives; and individual commitment/intention to leave.

How has the evidence been gathered?

The survey sought the views of all Police officers, support staff and SPA employees across Scotland. It had an extensive development phase involving communication planning and phased design and testing of the questionnaire and mode of delivery. Employees who had no access to computers or who were absent for work (e.g. on maternity or sick leave) were sent a paper version of the questionnaire. All others were invited to complete the questionnaire online. The survey was 'live' from 18 May 2015 for approximately one month. The total achieved sample size was 11,796, representing an overall response rate of 50%. The authors of the report note that this is the highest response rate achieved in employee engagement surveys among similar large public sector organisations in the UK in recent years. It should be noted, however, that this still leaves significant scope for non-response bias in the achieved sample (see below). There is no indication that the achieved sample has been weighted to match the profile of the workforce as a whole, despite some clear variation in response rates.

What can it tell us?

The results of the survey - which were the focus on considerable political and media interest - are summarised as 'positive messages' and 'issues for improvement' (rather than 'negative messages'):

• Key positive messages included high levels of respondents reporting strong and positive connections with their job, their team and their line manager;
• Positive feedback was more common among those in national functions and who had joined the organisation in the previous two years;
• 'Less positive' views were reported to be more common among those in regional roles and/or without regular access to senior management;
• Issues that were highlighted as impacting particularly on staff engagement were information and communication, feedback, training and development, wellbeing and overall commitment (33% of all respondents indicated an intention to leave);
• The survey also captured a high degree of scepticism about whether senior managers would take action in relation to the results of the survey.

Overall, the results suggest that - two years after unification - there are some areas of key strength for the organisation but that these need to be set against (and are in danger of being undermined by) significant areas of workforce dissatisfaction.

What appears to work well?

The survey marks a serious engagement with issues of employee engagement and morale, and indicates a relatively widespread appetite among staff to have their voices heard. It highlights areas where immediate action is needed (and some of the strengths that may be built on to do that) and puts in place an important benchmark against which progress can be judged. The report itself provides summary findings in relation to all the topic areas covered by the survey and is generally accessible and clearly presented.

What might be improved?

The implications of a response rate of 51% are not discussed. In particular, the variations in response across different staff groups need to be considered. For example, the response rate was only 39% in the division/department with the largest overall workforce - namely, local Policing. Given the finding that those in local roles were less positive in their views, it is likely that overall levels of dissatisfaction would have been higher had the sample been weighted to account for differential non-response. The methods of analysis deployed are straightforward, using univariate and bivariate approaches. The absence of any multivariate analyses means that it is difficult to know what are the key drivers of positive or negative attitudes. The report refers to some 325,000 words of 'free text' responses, which appear to have been subsequently coded and incorporated into the quantitative analysis. It seems likely that there would be much in these responses that might also be worth examining qualitatively. Although response by region (East/West/North/national) is shown, there is no subsequent analysis of variation by geography or legacy force area. Overall, given the size of the dataset, there would appear to be significant scope for further analysis.