Review of Scottish public sector procurement in construction

9. Data as an enabler of reform

9.1 Overview

9.1.1 Strategic policy decisions cannot be taken properly, nor direction set, without appropriate collation and analysis of public sector construction spend. Public sector construction spending is also an important economic lever, and so it is key that government knows how, where, when and on what this money is being spent to be able to measure and evaluate its full impact.

9.1.2 We note in chapter 5 the steps that have been taken to improve visibility of future construction activity for industry in the shape of the Infrastructure Investment Plan and our recommendations for improvements to pipeline information as the current approach to data collection is inconsistent and does not promote shared strategic thinking or decision-making on the use of that construction spend to achieve the greatest impact and benefit for Scotland.

9.1.3 There is also a lack of comprehensive collated data for public sector construction spend in Scotland. This means that any assessment of how efficiently and effectively that spend as a whole is being invested is extremely difficult to make.

9.1.4 The use of existing and improved data feeds into the delivery of many of our recommendations throughout the report.

9.1.5 In outlining our views on data, we are aware that careful consideration should be given to the usefulness and comparability of any data to be gathered and collated as well as any commercial sensitivities. Its purpose and value must be clear both at a disaggregate and aggregate level. Our vision is that the power of good quality data should be used as an enabler of reform both at strategic and local delivery levels.

9.2 Management information

9.2.1 A recent evidence review undertaken by the Scottish Government as part of the wider Procurement Reform Bill process concluded that:

"Drawing on the evidence from across the EU and the information available from current systems, there is a need to gather more data in a consistent and comparable manner across the public sector and the need to mandate the collection of that data. Progress has been made in the creation of different systems but more could be done to strengthen those systems to allow for the more systematic collection of reliable data, which would allow for a more detailed picture of procurement reform progress, the procurement landscape and the performance of organisations engaged in it." ^[58]

9.2.2 This holds true for the data we believe should be consistently gathered and analysed by the public sector in their management of construction projects and programmes. Such information would also provide evidence to support and measure policy development and implementation.

9.2.3 Our stakeholder consultations and evidence gathering as part of this review have highlighted areas within the public sector where good management information is currently being monitored and used to inform future procurement but it has also shown the varying stages that public sector partners are starting from. The following areas are the component parts which we believe all public sector partners should address when looking at data collection and usage.

9.3 Baseline data

9.3.1 Baseline data provides a point of reference to measure both existing and future performance and can be used as a basis for setting benchmarks and metrics. Establishing a baseline of current levels of performance should also enable the realistic appraisal of procurement approaches currently being used.

9.3.2 An important element in building that baseline is looking back and learning from experience. Audit Scotland's January 2011 "Management of the Scottish Government's capital investment programme" report recommended that the Scottish Government should develop standard criteria for inclusion in post project evaluations and ensure that they are completed for every major capital project and lessons learned are shared across all relevant public bodies. This was followed by their recent finding relating to local authorities which was that only 40 per cent of the projects they audited were delivered with the initial cost estimate ^[59] .

9.3.3 We see the need to establish a baseline position of performance in publicly funded construction projects and believe that such baseline data should include:

• How projects/programmes delivered against planned and agreed programme and cost.
• The contractual arrangements driving that performance - for example, was it a fixed price contract, target cost, cost reimbursable, with or without quantities, single stage, partnering, lump sum, cost plus? Did the contract type factor in the outcome?
• What quality standards were aspired to and then achieved?
• What was delivered against community benefit and sustainability targets?
• Identification of common major contractors.

9.3.4 In building such a baseline, data should be collected at key stages from inception to project completion to allow a full evaluation to be undertaken.

9.3.5 This would set the baseline for procuring organisations then to measure and challenge themselves against their own and, where relevant, others' performance and to challenge their approach to and expectations of future procurements. It would also allow the identification of opportunities to adopt an efficient and strategic approach to supplier relationship management across the public sector, building on the ideas set out in the UK Government's Construction Strategy, such as the use of strategic alignment agreements.

9.3.6 Consistency in the way this data is requested and recorded needs to recognise and be sensitive to sectoral nuances and contracting organisations' varying data collection starting points.

9.4 Benchmarking, Metrics and Key Performance Indicators

9.4.1 The performance of projects, programmes and indeed public bodies in relation to one another is facilitated by benchmarking, which is best achieved by common datasets and directly comparable indicators. Benchmarking, metrics and KPIs are currently used by many public sector organisations to assist their programme and project management processes but their scope, consistency and robustness varies - metrics being a numeric measure only whereas benchmarks and KPIs can be both numeric or qualitative. The objective of this data is to ensure value for money in the delivery of the specific asset but also to inform the most effective ways in which to make future investments. Once a baseline has been established, we see real value in benchmark performance measures being developed and used to inform future project evaluation and performance management. We believe that these should be developed on a sectoral basis first, with a view to collating and comparing more generically, where appropriate.

9.4.2 Such data would also identify opportunities for further information analysis and exchange to understand what might be driving benchmark statistics in both their and other procuring authorities and has the potential to be an efficiency driver by strengthening the public sector's ability to understand and challenge industry costs in an intelligent way.

9.4.3 This should eventually be extended to include ongoing data collection post-project delivery to allow fuller life-cycle costing considerations to be explored and used in future decision making whether on strategic capital and revenue budget requirements or in assessing individual project value for money.

9.4.4 This enhanced benchmarking data can then be used as a tool in business planning and new contract awards. The Chartered Institute of Public Finance and Accountancy recognises the importance of benchmarks:

"benchmarking offers a catalyst for driving efficiency, identifying new solutions that offer cost reductions and raising the performance of an organisation to the standards achieved by the best" ^[60] .

9.4.5 One such example of using data to inform future procurement comes from the Scottish Futures Trust. At a regional level data is gathered by the hubCos and SFT itself gathers data in respect of the Non Profit Distributing (" NPD") programme which it runs. SFT has developed a set of metrics for the Schools for the Future programme and in non-schools projects it has developed a method for establishing an "intelligent benchmark" which is then used to set an affordability cap for new projects. The extensive work by SFT, National Services Scotland and some other authorities provides scope for the development of a consistent and transparent basis to build better performance management and delivery through the development of benchmarks, metrics and KPIs.

9.4.6 Other data is currently collected by public sector clients, but it is not always readily accessible, nor is it always maintained regularly; in a common format; or for a common purpose. An example of this is Scottish Government Housing Supply Division ( HSD) which used to gather comprehensive data to be used in developing subsidy benchmarks and informing other policy decisions. In recent years, streamlining of the grant regime has meant that the detail of the data it has been able to draw on has been diminished. HSD still process Tender Returns for analysis and use in the Scottish Social Housing Tender Price Index but there is scope for some of that information to be used more widely in assessing value for money and shaping future affordable housing delivery as well as assessing what other data could be gathered to complement investment decisions. Scottish Government guidance notes that "all or a proportion of projects will be subject to post completion scheme review". ^[61] Reinstatement, rationalisation and reinvigoration of these processes will help HSD in assessing the effectiveness of the procurement decisions they and their partners are taking and informing future procurements. This recommendation for HSD could similarly be applied to other areas of the public sector.

9.4.7 It is strongly recommended that monitoring and benchmarking data is strengthened by all public bodies to help increase market intelligence and better inform future policy and programme management. While we accept that some categories of construction expenditure are specialised, there may also be common elements which could usefully be compared to produce benchmark information and help develop KPIs. The potential to extend this to benchmarking against private sector performance should also be considered.

9.4.8 We believe that the conduit for sharing benchmark information across the public sector should be the enhanced construction procurement policy function within the Scottish Government, and that, where relevant and non-commercial, that benchmark information is published where possible. Initial work should focus on the SFT approach with a view to formulating a standard public sector approach.

9.4.9 Good quality data should be used to identify what success looks like in the procurement of projects/programmes and the changes required to reflect good practice. Better data should also help in identifying areas where continuing to procure in isolation does not make sense.

9.5 Performance Management and Continuous Improvement

9.5.1 Monitoring performance through the use of benchmarks and KPIs will assist clients better to understand, manage and analyse that performance and the factors influencing it.

9.5.2 That learning can then be used across public sector construction to improve practice, out-turn and outcomes and reap full value from public investment and the holy grail of becoming the often quoted "intelligent client" who is actively involved throughout the project in managing risks with the contractor should then become more achievable.

9.5.3 As a result, target setting and KPIs should be more informed and improved data fed straight back in to business planning stages for new projects to improve the reasonableness of assumptions and expectations when it comes to budget setting (cost), quality and timescales. Linkages can also be made across other areas of performance e.g. design, community benefits delivery, payment performance, whole life costing.

9.5.4 Transparency and accountability should also be strengthened as a more robust set of measures and challenges will be applied to construction investment decisions. By applying these principles not only at business plan, project appraisal and completion but also during the construction phase as outlined in section 6.9 on project assurance, outcomes can be improved in a more responsive way.

9.5.5 There is some work to be done in ensuring that these approaches are wholeheartedly adopted across the public sector as Audit Scotland's findings as part of their report on capital investment by local authorities bear out:

"…just over half of the 63 completed projects in our sample have been evaluated to assess whether they have delivered the intended benefits." ^[62]

9.5.6 This builds on Audit Scotland's recommendation that:

"…public bodies should ensure that they carry out post-project evaluations within six months of project completion to determine whether projects have delivered, or are on course to deliver, the initial benefits intended. Evaluations should consider performance against cost, time and quality targets." ^[63]

9.5.7 The principle of learning lessons from past experience, however, is already being embraced by some following the 2011 Audit Scotland report statement on the need to develop standard Learning Lessons criteria and ensure that evaluation is carried out on all projects and shared across public bodies.

9.5.8 The Scottish Government has adopted a central strategic role in facilitating, promoting and sharing lessons learned and has mandated the use of Learning Lessons for major investment projects. A pilot project with delivery bodies which have more mature Learning Lessons procedures in place, including Transport Scotland, the Scottish Funding Council and the health sector is currently underway. This feedback will be supplemented with assessments from a separate exercise, involving Gateway Review and Key Stage Reviews with key lessons learned being published in early 2014.

9.5.9 We recognise at various points within this report the importance of sharing lessons learned and see real value in actions in this regard taking cognisance of the work already underway through the Learning Lessons approach to ensure that continuous improvement is achieved.

9.5.10 Recommendations:

Action should be taken to ensure robust systems are in place to track all spending on construction by public authorities such that a complete analysis of annual public sector construction spend in Scotland can be easily available.

Sectoral records of project outturn costs, including what they were estimated to cost at business plan and contract award stages and actual cost on completion, should be developed and maintained so as to provide meaningful benchmark figures for the public sector in Scotland. These records should also record timescales and quality measures to enable a true assessment of performance delivery to be made.

Guidance should be developed on robust management information requirements and should cover baseline data, benchmarks, metrics and KPIs.

Project evaluation should be promoted and should build on the Learning Lessons Approach.