Procurement Efficiencies: Monitoring and Evaluation of Devanha Phase 2

10 Framework Agreement

10.1 The 'Framework Agreement' agreed by Devanha has set out a number of objectives. This led to the development of KPIs (Key Performance Indicators) to assess the overall performance of the programme delivery. These KPIs were used to evaluate key areas which were considered important to the success of the initiative. Some of the KPIs set out in the original agreement were not retained due to difficulties in measurement (eliminate the adversarial approach; enhancement of service provider's reputation). Several KPIs were used to establish the Incentivisation Model which was based in the principle of Gain-share/Pain-share. These KPIs are: Zero health and safety incidents; Greater predictability of out-turn cost and programme; and Right first time with zero defects. They were measured and used to establish the financial pain or gain shared by the contractor and client.

Key Performance Indicators

10.2 Zero health and safety incidents: Scheme reports show eight 'reportable accidents' across the programme of works on site. The overall record is very good for Devanha, and demonstrates a high standard of working conditions with the framework contractors.

10.3 Predictability of programme: This was forecast at two points, scheme approval and at practical completion. At scheme approval there seems to be far less predictability than at practical completion. This is as expected, due to more complete information on site conditions and the exact nature of construction programme at practical completion stage. Once on site schemes appear to have been managed well in terms of completion times. Data is available for 29 schemes of which 11 schemes had reached completion ahead of the planned date, a further 11 had completed on time. Only 7 schemes had completed later than contractually agreed. Generally the programme of works has had a very good record of programme predictability. The Section 75 schemes, where the Framework Agreement was not applicable, appear to have a similar predictability, so there is no evidence to suggest that the framework agreement is the cause of the good performance on programme.

10.4 Defects: The incidence of defects has been highly variable. This appears to have been as a result of the different interpretation of defects made by clerks of works. In some cases individual marks on walls were recorded individually, while the same scale of defect may be recorded as a single defect on another scheme. The scheme reports show some schemes with 8 defects and others with over 100. The inconsistency has been a source of frustration for some of the stakeholders, with a direct influence on the pain share apportioned to contractors.

Framework Objectives

10.5 Other objectives included in the framework, but not used within the within incentivisation model for Devanha are:

• Team working and consideration of others;
• Eliminate adversarial approach;
• Employer and tenant satisfaction;
• Enhancement of Service Provider's reputation;
• Improvements in quality, productivity and value for money;
• Improvements in environmental performance.

10.6 Two of the KPIs are incorporated in the framework agreement: Team working and consideration of others; and Eliminate adversarial approach. A diagnostic tool was used by the procurement consultant to capture many aspects of the team working and collaborative approach that are being fostered through Devanha. Areas which ranked highly were: Collaboration and team work is a focus for senior management; Trust between 'Managers' and 'people'; Good communication supporting collaboration; and Collaboration leading to reduced programme times'. Devanha members recognised the importance of teamworking, however, evidence from workshops and interviews suggests that 'teams' for the delivery of individual schemes had been dependent on the individual personalities involved.

10.7 All of the individual RSL partners in Devanha Phase 2 have tenant satisfaction surveys that are conducted with new tenants. It was the intention of Devanha to adopt a programme-wide approach to evaluation tenant satisfaction. This has not been established and it is not possible to evaluate if there has been any improvement in tenant satisfaction between pre-Devanha schemes and the housing that has been procured through the framework. The RSLs are satisfied that the housing is of a high standard, and generally the quality has improved. The improved thermal performance of the Devanha schemes should also deliver lower fuel costs to tenants.

10.8 Devanha Phase 2 has taken place during a sharp UK wide economic downturn in the construction industry. Members of Devanha believe that the programme activity has protected the individual consultants and contractors to a certain degree. The continuity of work through the programme has enabled the companies to reduce or avoid redundancies but it is not possible to quantify the effect that Devanha may have had on the economic well-being of the companies involved.

10.9 The legacy of Devanha for the consultants and contractors involved is varied. Some recognise the valuable experience gained by using new procurement approaches, and the introduction of open-book costing, lean construction methods, and innovative technologies. However, we have no evidence on whether the service providers' reputations have been enhanced.

10.10 Improvements in quality and environmental performance are evident in most of the schemes delivered by Devanha. The detail of this is dealt with in the previous chapter. These objectives were always in competition with the objective of reducing costs. That Devanha developed a range of schemes with consistently high quality and reduced environmental impact is a very positive outcome.