The Crerar Review: the report of the independent review of regulation, audit, inspection and complaints handling of public services in Scotland

06 PROPORTIONALITY OF EXTERNAL SCRUTINY

6.1 Applying the principle of proportionality to external scrutiny in a more strategic way will have significant implications. This chapter draws out the main issues about proportionality and the effectiveness, efficiency and coordination of existing scrutiny arrangements.

Growth of scrutiny

6.2 External scrutiny has grown in recent years in three ways:

• Eleven completely new bodies have been created since 1999 to perform scrutiny functions not previously carried out by another body;
• A number of new bodies have been created to carry out scrutiny previously performed by other organisations and in many cases these have been given additional functions;
• Some existing bodies have been given new or expanded scrutiny functions.

6.3 The most significant changes include:

• The audit of Best Value in local authorities by Audit Scotland on behalf of the Accounts Commission;
• The creation of Communities Scotland in 2001 and a new role in regulating and inspecting local authority housing services;
• The establishment of the Care Commission in 2002 as a national regulator of all care services in Scotland (approximately 15,000 providers in both the public and private sector ranging from child-minders to care homes for older people) and all independent health care. The Care Commission brought together the regulatory functions of 32 Local Authorities and 15 Health Boards;
• The establishment of NHS Quality Improvement Scotland in 2003, bringing together the functions of the Clinical Standards Board for Scotland, Health Technology Board for Scotland and Scottish Health Advisory Service;
• The establishment of a new Inspectorate of Prosecutions, covering Crown Office and Procurator Fiscal Service in 2004;
• A new system of inspections of criminal justice social work;
• The development of a multi-agency inspection regime for children's services, starting with child protection, led by Her Majesty's Inspectorate of Education;
• The launch of the Social Work Inspection Agency in 2005 to inspect services at a local authority level;
• The development of multi-agency inspection regimes of social work services for older people, people with learning disabilities, people who misuse substances and people with mental health problems; and
• The establishment of the Office of Scottish Charity Regulator in 2006.

6.4 Much of the growth reflects changes in scrutiny focus in response to more integrated models of service provision and is regarded as an important step-change in ensuring that scrutiny responds to new priorities. However it has not replaced existing external scrutiny of professions such as social work or police, or of single services such as prisons or schools. It is our view that continued growth of scrutiny is not sustainable, and there needs to be a more proportionate approach to how it is applied in future. As we noted in chapter 2, by 2008/09 baseline information should be available from new scrutiny regimes, such the Social Work Inspection Agency ( SWIA) social work services inspections, HMIE's INEA and INEA2 and child protection and children's services inspections, NHS Board Accountability Reviews, and Best Value audits of local government, which should enable appropriate decisions regarding the level of scrutiny that is required and what can be reduced.

Compliance burden

6.5 All stakeholders, including service providers, believe that there is a need for external scrutiny. However, many providers have complained of external scrutiny becoming a burden. The point was made particularly strongly by local government, but health boards, registered social landlords and care establishments all felt that the volume of performance reporting and external scrutiny had become disproportionate to the benefit delivered and to the risk it was set up to overcome, and was distracting resources from front-line delivery. Issues raised included:

• Visits by more than one scrutiny body at the same time and particularly the impact on senior management;
• Requests for the same or similar information in different formats;
• Lack of co-ordination between scrutiny bodies and lack of information sharing;
• Partner organisations being scrutinised separately, requiring them to divert resource to;sectoral scrutiny priorities and away from the partnership;
• Within the health sector, NHS Boards referred to overlap and duplication between NHS;Quality Improvement Scotland ( QIS) and Audit Scotland;
• Within the RSL/care provider sector, providers referred to overlap and duplication between local authorities, Communities Scotland and the Care Commission;
• Within local government, concerns were expressed about overlap between external scrutiny requirements and Scottish Government information requirements, for example compliance requirements of ring-fenced funding; and
• There was also concern about potential for overlap and duplication where the scrutiny activity concerned assessment of corporate structures or processes, for example between Best Value audits and HMIE, and/or SWIA inspections.

6.6 We noted two very specific examples of duplication that exist in the care sector which may be the result of separate sets of statutory duties being introduced:

• Many independent care providers are contracted by local authorities to deliver care services. Any organisation that provides any of the care services set out in the Regulation of Care (Scotland) Act 2001 must register with the Care Commission. An organisation providing more than one of the services (which many do) must register each service separately;
• Organisations providing care services are also subject to annual inspection by the Care Commission. Those same organisations are also required to demonstrate compliance with local authority contracts.

6.7 The information in diagram 1 illustrates the crowded nature of the scrutiny landscape for the local government sector.

Diagram 1: -The range of scrutiny bodies which impact on a local authority

Note: -Local authorities are also covered by the inspections of both Fire and Police Services, as carried out by the respective Inspectorates. However, they are not included in the diagram because the inspection activity focuses primarily on the bodies delivering the service e.g. police forces, fire stations etc.
Source: Scrutiny Review Stage 1 mapping survey

6.8 One local authority listed 15 organisations that regulated or inspected its services, with the largest overlap in inspection being in the children's services and education areas. The local authority concluded that many of the processes looked similar, but that there was little clarity about how these are brought together. They saw a need for one framework for integrated working, one self-evaluation tool and one inspection per authority.

6.9 Although scrutiny impacts on front-line service delivery, it has a particular impact on corporate management, who normally co-ordinate responses to scrutiny. This can divert senior management time from other priorities.

Other demands for information

6.10 External scrutiny is not the only source of demands for information. Service providers in Scotland are asked for information about their activity and performance from an increasing number of sources - diagram 2.

6.11 While each group or organisation may have legitimate and distinct reasons for seeking information, there is potential for collision between systems, resulting in multiple demands for information. This situation is fuelling concerns about the overall burden of compliance and reporting referred to by service providers.

Diagram 2: Information requests to service providers

Collaborative working

6.12 In its report on accountability and governance, the Parliament's Finance Committee noted the potential lack of co-ordination of work between scrutiny organisations operating in the same sphere, and recognised the need for better collaboration and sharing between them. The Committee made the following recommendations to help address this issue:

(a) Bodies with similar roles and responsibilities should be amalgamated wherever possible;
(b) The potential to pool the resources of existing bodies (such as sharing staff) should be considered wherever possible with a view to streamlining these organisations;
(c) Unnecessary direct remit overlaps should be dealt with by removing responsibility from one of the bodies involved and adjusting budgets accordingly; and
(d) Where bodies do not have direct overlaps but their roles inter-relate there must be an expectation from the Scottish Government that these organisations agree memoranda of understanding to minimise the possibility of a duplication of effort.

6.13 The Review endorses and supports the conclusions of the Finance Committee and these are reflected in our own recommendations.

6.14 External scrutiny organisations are aware of the potential for overlap and duplication and have developed collaborative working arrangements in response, such as:

• sharing information, coordinating visits etc;
• utilising common frameworks;
• placing reliance on each other's work;
• undertaking joint scrutiny activity; and
• sharing some performance and risk data.

6.15 The criticisms of this level of collaboration are that arrangements become overly complex due to the numbers of agencies involved, but at the same time the scope of coordination is limited and amount mainly to more effective scheduling of scrutiny work, with limited positive impact on providers.

Diagram 3: -Scrutiny bodies' reported formal information sharing arrangements

Diagram 4: -Scrutiny bodies' reported joint scrutiny activity between autumn 2001 and 2006

6.16 As service delivery has become more complex and as the lack of fit between delivery and the existing silo-based scrutiny organisations has become apparent, collaboration has become necessary. The emphasis on partnership working and multi-agency service delivery has resulted in a shift to scrutiny of services for client groups or cross-cutting services in some areas. Scrutiny organisations have taken on new areas of activity, which has required new forms of collaboration. Both the Joint Inspection of Children's Services and the multi-agency inspection programmes, being led by HMIE and SWIA respectively, involve a range of scrutiny organisations sharing information and undertaking joint inspection.

6.17 Such collaborative working arrangements can be complex, involving a wide range of stakeholders. An overview of the types of relationships and collaborative working arrangements that currently exist are reflected in diagrams 3 and 4.

6.18 In some cases, legislation is a constraint on joint working because it limits flexibility, for example, by specifying the nature and frequency of inspections, or limit information sharing. In some instances, joint working arrangements have been driven and enabled by legislation. For example, the Regulation of Care (Scotland) Act 2001 requires the Care Commission to work with HMIE on inspections of particular services. The Joint Inspection of Children's Services and Inspection of Social Work Services (Scotland) Act 2006, both requires and facilitates joint working between HMIE and a range of other external scrutiny organisations on inspections of children's services, and between SWIA and others on service inspections.

6.19 Strategic groups have been established to coordinate multi-agency inspection work. Examples are set out in the box below. The size and make-up of these groups are indicative of the range of interests that need to be considered. It illustrates the level of resources absorbed by collaborative working.

Multi-agency Inspection of Community Care Services

There is a Strategic Group for the body of work covering Community Care Services. The Group has around 23 members, including representatives from the scrutiny organisations and the Scottish Government.

There is also a Knowledge Management Group, with around 13 members.

For the inspection of Learning Disabilities Services in Ayrshire ( i.e. focusing on NHS Ayrshire & Arran and the three bordering local authorities), the inspection team consisted of around 29 individuals. The model for this inspection was commissioned externally.

Inspection of Services for Children

HMIE has established a lead group for the inspection of Services for Children. The group meets to discuss key developments with the inspection programme and processes. It has 22 members, representing seven organisations, six of which are scrutiny organisations.

In addition, there is the Services for Children Operational Sub-Group, with 17 members from the same seven organisations.

6.20 Collaboration has been widely regarded as a positive development because it is an attempt to align the scrutiny activity with emerging priorities. However, collaborative activities are additional to existing service-based or organisational scrutiny and have added to the overall volume of scrutiny activity, raising questions about the need to rationalise and to take a more strategic approach to how scrutiny resources should be prioritised.

Costs and benefits

Costs

6.21 Audit Scotland carried out an analysis of the costs of external scrutiny for the Review. This chapter summarises the main findings, the full paper ⁶ is in Annex D.

6.22 Costs can be split between direct costs (those costs incurred by those carrying out the scrutiny activity) and indirect or compliance costs (those costs incurred by those subject to scrutiny e.g. service providers such as local authorities or health boards).

6.23 The growth of scrutiny has led to an increase in direct costs. Direct scrutiny costs in Scotland have increased by approximately 55%, from around £60 million in 2002/03 to around £92 million in 2005/06. The total managed expenditure of the Scottish Government has increased by around 22% over the same period. The direct cost of external scrutiny in Scotland is around 0.18% of overall public sector expenditure i.e. for every £100 spent on public services, 18p is spent on scrutiny. This compares with around 0.20% in England.

6.24 Audit Scotland notes that the scale and cost of external scrutiny cannot be estimated with precision. Despite its attempts to gather information about the costs of the various external scrutiny activities, it identified a range of difficulties. Its work has highlighted the need for greater consistency and accuracy of external scrutiny cost data in order for a comprehensive understanding of the cost (including compliance costs - see below) and performance of external scrutiny in Scotland to be achieved. It noted that the key issues that need to be addressed include:

• The adoption of common scrutiny costing methodologies;
• More widespread use of regulatory compliance assessment models; and
• More accurate data capture on scrutiny compliance costs within public bodies themselves.

6.25 In 2005/06, three external scrutiny organisations - the Care Commission, Audit Scotland and HMIE - accounted for around 80% of the direct cost of external scrutiny. Each organisation's direct cost had also risen significantly since 2002/03 - by 35%, 20% and 95% respectively. Other external scrutiny organisations also experienced significant cost increases (details can be found in Audit Scotland's paper).

6.26 Audit Scotland attributes the bulk of the increase in scrutiny costs to a small number of key developments, such as the introduction of the Best Value audit regime, child protection inspections by HMIE and inspections of services for people with learning disabilities and services for older people by SWIA.

6.27 Service providers incur compliance costs when preparing for, participating in and responding to the outcomes of, scrutiny activity. However it is difficult to estimate these accurately. For example, the work commissioned by COSLA, SOLACE and the Improvement Service ⁷ asked councils about resources used in dealing with external scrutiny. All councils said that there were significant costs, but had difficulties in assessing these because:

• Councils do not use time recording systems;
• All information was based on retrospective assessment;
• Length of time that had elapsed since the scrutiny exercises reduced accuracy;
• Many officers continue to undertake operational tasks at the same time as responding to an external scrutiny exercise.

6.28 A survey of health boards indicated that only two of the regional health boards felt they would be able to estimate compliance costs, with most others suggesting it would not be possible without great difficulty and significant resource input.

Measuring Benefits

6.29 The costs of external scrutiny have to be balanced against the benefits, but the benefits and impact of external scrutiny are difficult to assess. One recent independent source of information about impact is the Review of Best Value Audit commissioned by the The Accounts Commission ⁸ which concluded that Best Value audits have had a significant positive impact on councils. Service providers across all sectors also identified positive benefits. However, most had difficulty in attributing benefit directly to scrutiny. In most cases, service providers across all sectors felt that the added value of external scrutiny did not justify the cost of compliance.

6.30 It is acknowledged that there is a significant gap in the assessment of impact and that the impact of external scrutiny is poorly understood, a point made by the Auditor General and acknowledged in the literature. The Auditor General argues that a more sophisticated understanding of the various impacts of inspection, regulation and public audit is required, in order to enable the collective impact of scrutiny to be maximised and unintended consequences to be removed. The Finance Committee's Inquiry into Accountability and Governance also noted that insufficient information on costs was made available when the Parliament's scrutiny commissioners had been set up. This reflects the need for more rigour in the way costs and benefits are measured, and the need for an improved system of impact assessment.

Scrutiny as a response to risk

6.31 A major part of the argument for change of the current external scrutiny arrangements hinges on arguments about the need for more proportionate scrutiny. Proportionality is strongly linked to risk, but it is the most difficult of the principles discussed in chapter 4, because of the complexity of the issues (for example, the relative weightings of the various risks facing service providers) and in defining appropriate responses.

6.32 External scrutiny is generally focused on areas where there are concerns about the safety of the public and service users, or on services that are considered vital to society, or where public funds are spent. For example, the Care Commission's work focuses on services predominantly delivered to vulnerable groups i.e. those who receive care. HMIE's work focuses on the education system. Audit Scotland's work focuses on financial probity and value for money. Ministers and the Parliament put in place processes that minimise the risk of the services failing or of money being used for the wrong purposes.

6.33 We have referred previously to the growth in scrutiny, particularly in recent years. Some of the recent external scrutiny activity has been introduced in response to a small number of service failures, where there was concern that the problems may be widespread and external scrutiny could uncover risks of service failures, thereby preventing things going wrong. Examples include inspections of children's services and inspections of specific aspects of social work services, such as learning disabilities services. When announcing the plans for the new inspections of children's services, the Minister was clear that no guarantees could be offered about safety, but there was clearly an expectation that the inspections would in some way reduce the risk of service failure.

6.34 We accept that the existence of external scrutiny is one factor that may influence service providers to minimise risks to services, but we are concerned that expectations about how far scrutiny can address perceived risks may be too high. External scrutiny cannot eradicate risk and it should not be used to manage risk - that is the business and responsibility of those who manage delivery of public services. It can assist in assessing and mitigating risk by testing the mechanisms by which organisations assess their own risk management processes and performance management systems and by reporting areas of deficiency.

6.35 We cannot ignore the bigger picture. Ministers and the Parliament must take responsibility for deciding what is a tolerable level of risk for particular services. Only then is it possible to consider what role external scrutiny might play in assisting service providers to manage their risks, and how else risks might best be mitigated. Only after this has been done is it possible to consider what should govern the judgements and decisions to apply external scrutiny - as opposed to other performance management functions/sanctions - as a mechanism to control and manage such risks.

6.36 The Better Regulation Commission's report, "Risk, Responsibility and Regulation - Whose Risk is it Anyway" ⁹ is a helpful starting point to consider the issue of risk and the use of external scrutiny. Diagram 5, adapted from a diagram of the "Regulatory Response to Risk" in that report, provides a helpful illustration of how external scrutiny is traditionally applied to address risks to service provision and the consequences this can have.

Diagram 5: The external scrutiny response to risk

Adapted from Better Regulation Commission (2006)

6.37 This approach can lead to improvement but might also deliver some undesirable outcomes, such as unnecessary increases in regulation, or poorer and less responsive services and potentially more risk of things going wrong if public service professionals are responding to externally imposed requirements rather than using and developing their professional judgement.

6.38 While all of the existing external scrutiny organisations and regimes were put in place to help address legitimate risks at the point in time at which they were introduced, we believe there are two fundamental issues that have not been considered consistently:

First, whether the application or scale of the activity was proportionate to the overall risk to public service provision. For example, in the case of vulnerable groups, we might expect a high level of risk, but does this necessitate a cyclical inspection process spanning all aspects of the service and/or all geographical areas?

Second, whether each new activity was considered against all existing external scrutiny activity already affecting the service or organisation i.e. where does the requirement fit in the overall list of priorities?

6.39 We believe that it is important to be realistic about the extent to which scrutiny reduces risk and contend that a major problem in the current system is that there is no consistent method by which to decide whether external scrutiny is an appropriate response to a particular risk or what the "right" amount of external scrutiny is in a particular set of circumstances. We will describe how this can be addressed later in the report.

Chapter summary and recommendations

Growth

• External scrutiny has grown in recent years, with 11 new bodies created since 1999 and several additional functions being given to existing bodies. Some growth has resulted from new regimes, such as Best Value, and some growth has been the result of additional multi-agency scrutiny activity. This level of growth of scrutiny is not sustainable without reduction in scrutiny elsewhere, and a more proportionate approach to the introduction of scrutiny is needed in future.

Burden

• Providers agree external scrutiny is needed but believe that the volume of scrutiny has become disproportionate to the benefit delivered and to the risk it was set up to overcome. It is distracting resources from front-line delivery.
• Lack of co-ordination and information-sharing between scrutiny organisations creates duplication and overlap for providers.
• The impact of external scrutiny is a particular problem in the local government sector. Compliance ties up extensive resources.
• External scrutiny has a particular impact on corporate management, who normally coordinate responses to scrutiny. This can divert senior management time from other priorities scrutiny impacts on front-line service delivery.

Collaborative working

• Some scrutiny organisations have developed collaborative working arrangements in response to potential overlap and duplication, but with limited positive impact on providers so far.
• To reflect partnership working in service delivery, some scrutiny is now applied to client groups by a collection of scrutiny organisations working together. This alignment has been welcomed, but these collaborative activities are additional to service-based scrutiny, and have increased the overall burden of scrutiny.
• Legislation can be a constraint on joint working because it limits the flexibility of scrutiny organisations.
• We agree with the Finance Committee recommendations to rationalise scrutiny arrangements and will refer to them in our recommendations later in the report.

Costs and benefits

• Growth of scrutiny has led to an increase in costs. The costs of running external scrutiny organisations increased from £60 million in 2002/03 to £92 million in 2005/06. This 55% growth seems disproportionate to the 22% growth in the Scottish Government's total managed expenditure over the same period. For every £100 spent on public services, 18p is spent on scrutiny.
• There is a need for more rigour in the way costs and benefits are measured, and the need for an improved system of impact assessment.

Risk

• In a proportionate framework, risk should be central to determining whether external scrutiny should be used, and, if it is, what its nature, scope and duration should be.
• External scrutiny cannot eradicate risk and it should not be used to manage risk - that is the responsibility of the providers who manage services.
• Using external scrutiny to address risk to services can lead to improvement. However, without first considering if it is a proportionate response to risk and without considering how any new scrutiny sits with existing scrutiny, new scrutiny will likely also deliver undesirable outcomes such as increased regulation or poorer and less responsive services.
• There is no consistent method by which to decide whether external scrutiny is an appropriate response to a particular risk or what the "right" amount of external scrutiny is in a particular set of circumstances.