Procurement Efficiencies: Monitoring and Evaluation of Devanha Phase 2

8 Procurement Consultants

8.1 Devanha Phase 2 funding was dependent on the use of an independent procurement consultant to provide training, and embed collaborative working linked to the use of NEC contracts and partnering. The Evaluation investigated the impact of the use of a procurement consultant through two questionnaire surveys, interviews and workshops. The various strands of research highlight tensions and dissatisfaction but also some positive results of involvement of the procurement consultant.

8.2 A questionnaire was sent out twice to the Devanha CEOs, other board members, staff, Development Directors and Project Managers to evaluate the effectiveness of a procurement consultant in the implementation of the Devanha programme. The first questionnaire was completed towards the beginning (summer 2008) and the second at the end of the programme (March 2012). There were 11 responses from a possible 17 to the 2008 questionnaire and 10 from a possible 16 to the 2012 questionnaire.

8.3 Table 8.1 shows the responses to questions which relate to issues which could be broadly defined as technical training.

Table 8.1 Technical Training - Contract

Devanha influencing understanding of NEC Contracts	1	2	3	4	5	CWC influencing understanding of NEC Contracts
2008	0	0	4	3	4	2008
2012	0	2	3	3	2	2012

8.4 In both years the majority of respondents saw the influence in understanding contracts as being primarily from the procurement consultants. This would be consistent with the consultants fulfilling their role. Interestingly, this was less pronounced in 2012 indicating that Devanha had increased in its confidence in relation to the contract. As shown in table 8.2, the procurement consultants were also seen as having led on understanding of open book cost management. This should, however, be seen in the context of relatively poor embedding of this technical training at the right levels, as discussed elsewhere in this report.

Table 8.2 Technical Training - Open Book

Devanha leading on understanding of Open book cost management	1	2	3	4	5	CWC leading on understanding of Open book cost management
2008	0	0	4	4	3	2008
2012	0	1	1	5	3	2012

8.5 Changing culture and work practices from the traditional to a more collaborative approach is an essential element of the modernised procurement agenda. The second major area explored in the questionnaire was the consultant influence on development of a collaborative team.

Table 8.3 Developing the Collaborative Team - Communication

Devanha influencing Improved Communication	1	2	3	4	5	CWC influencing Improved Communication
2008	1	4	4	5	1	2008
2012	1	5	3	1	0	2012

8.6 Views were fairly mixed on who had most influence on improved communication. This would seem appropriate and again the move towards Devanha having more influence later in the programme is consistent with Devanha taking ownership of the issue. However, this is not to say that communication was good. We have seen elsewhere in this report that communication with external stakeholders was a problem and that dissemination of Devanha practices throughout member and partner organisations did not always occur.

8.7 Across both surveys, respondents tended to see customer focus, continuous improvement and creative problem solving as being more influenced by Devanha. Once again this would be positive and consistent with Devanha taking responsibility for implementation of the programme, focus on improving quality and overcoming difficulties as they arose.

8.8 The questionnaire explored the question of leadership and empowerment. Table 8.4 shows that these issues were mostly seen as being influenced by Devanha.

Table 8.4 Leadership and Empowerment

Devanha influencing Leadership and Empowerment	1	2	3	4	5	CWC influencing Leadership and Empowerment
2008	0	5	5	1	0	2008
2012	1	7	2	0	0	2012

8.9 Leadership problems have limited Devanha's successes over the whole programme. The consultants recognised this and in interview suggest that they offered to take a more active role than their original brief required. The consultant proposed to take on more of a role in implementation of structure change below the Board level where the consultant would act as a 'change manager' on an interim basis. This, they believed, could have provided interim leadership which Devanha could take over through time. The proposal was not accepted by Devanha.

8.10 Nonetheless, the procurement consultant had a role in facilitating change within Devanha and the surveys explored satisfaction with how well the consultant had fulfilled this role. A majority of respondents to the 2012 procurement questionnaire were dissatisfied with consultant performance in facilitating workshops, communication and changing working practices within Devanha. The procurement consultant criticised the level of competency achieved in collaborative working and believed that there had been little change in the client or the supply chain, with an essentially traditional approach remaining.

8.11 Devanha members and partners believed they were already better informed of the issues pertinent to Devanha programme and the Procurement Consultant did not offer any significant input beyond organising the events. Several consultants and contractors arranged their own in-house training on topics in what they saw as a more relevant manner. This related particularly to 'Open-book costing'. Some members were concerned that procurement consultant fees had been out of proportion to the service offered, and that value for money had not been achieved.

8.12 The procurement consultants believed that Devanha benefited from their involvement but that 'inertia' within Devanha was a limiting factor. They also noted that more could have been achieved at the workshops if the right people had attended; they had advised that there were too many senior managers and not enough people delivering the projects. This meant learning did not permeate down to the appropriate people in the organisations.

8.13 Some positive changes were associated with the use of a Procurement Consultant although the perception of these positive changes generally diminished from the beginning of the programme to the end. 'Knowledge Sharing' is consistently seen as a positive change arising from the consultant's input. Positive views on 'Collaboration across RSLs'; 'Collaboration with Stakeholders and Supply Chain'; and 'Identification of Goals' were all evident but have all diminished in the view of respondents over the duration of Devanha. There were benefits from the use of the procurement consultant in relation to greater general awareness of collaborative practice and of the technical details of NEC contracts; new ways of working; introduction of open book; and provision of external cost validation.

8.14 The procurement consultant's diagnostic analysis^[4] showed that there was improvement in Devanha performance in collaborative working but they believed that more could have been achieved if there had been more 'drive' on the part of Devanha. Research workshops held with Core Group members in 2010 confirmed greater awareness and use of aspects of the NEC contract although there were still significant deficiencies (as reported in discussion of the process elsewhere in this report). In particular, there was enthusiasm for the benefits brought by adoption of the Gateway process in development of projects.

Delivery

8.15 A Likert Scale was used in the surveys to indicate the extent of involvement in delivery of the programme. A score of 1 meant that delivery was solely carried out by Devanha and 5 meant it was solely carried out by the consultant. Table 8.5 shows results from 2012 for three key areas of delivery where Devanha would be expected to lead but with support from the consultant.

Table 8.5 Involvement in Delivery (2012 survey)

	1	2	3	4	5
Devanha leading on Open book cost management	0	4	3	3	0	CWC leading on Open book cost management
Devanha leading on Performance improvement and measurement	1	3	4	2	0	CWC leading on Performance improvement and measurement
Devanha leading on Scheme mobilisation	4	3	3	0	0	CWC leading on Scheme mobilisation
Devanha leading on Modern methods of construction	5	3	2	0	0	CWC leading on Modern methods of construction

8.16 The pattern of answers shown in table 8.5 supports the view that the procurement consults were acting in a supportive role while Devanha took the lead in delivery in all areas. Given the unfamiliarity of the open book process it is reasonable that the consultants were seen to have a greater role than in other areas.

8.17 Performance improvement and measurement was an area where the consultant had a role. In practice there was a large difference between Devanha's expectations and what was achieved in the area of measurement. Through a number of workshops and interviews Devanha, its members and partners expressed dissatisfaction over the quality of data which was available for monitoring cost, scheme progress and performance.

8.18 The consultant had offered to produce a web based data management system which would simplify scheme data collection and allow the board and project managers to monitor and manage schemes efficiently. However, although this system was set up, it was never used and was seen by Devanha as not fit for purpose. The research team were able to witness the Programme Director taking large amounts of data from a variety of sources to try to populate the web based tool without any practical output and then producing alternative spreadsheets which were, instead, used to inform the Board and assist it in programme monitoring and control.

8.19 The consultant believed that the failings in this respect were not a function of the web based tool but occurred because Devanha did not use it as it was intended. A lot of data was held on the platform but Devanha did not devolve responsibility for maintaining it to the Project Managers as they should have. It was therefore too onerous for the Programme Director to maintain. A consultant representative stated that he tried to push Devanha to devolve this function but with no success.

8.20 Availability of data to assist in performance management and incentivisation was also problematic. Devanha and its partners were unable to produce data which would inform understanding of performance against KPIs early enough in projects for this to incentivise contractor performance as had been intended. Devanha and its partners viewed this as having been a failing in the system devised by the procurement consultants. The procurement consultants also saw this as a problem but believed the KPIs were set up appropriately but were not used properly in project monitoring status reports. Rather, they were completed by Clerks of Works at the end and therefore no use in incentivisation.

8.21 It is clear that the content and timing of data available for the Board to manage the programme was always a problem. KPI data was also available at the wrong time to incentivise improved performance. There are conflicting views over the causes of these problems and they were a source of tension and dissatisfaction between Devanha and its procurement consultants. That these difficulties were never resolved is another example of lack of strong leadership within Devanha. Devanha took responsibility for creative problem solving but a managing director would have had responsibility for resolving the problem and less than adequate fixes would not have been allowed to persist throughout the programme.

Conclusion

8.22 The use of a Procurement Consultant was considered by many Devanha members to be necessary for the introduction of different ways of working. This was particularly important for the financial and contractual framework established by the use of NEC and 'Open-book costing'. There was clearly some important assistance given in the use of NEC.

8.23 Effort expended in creating a web based tool was wasted as this was unusable by Devanha. Neither Devanha nor its project managers believed that the system contained data in a form which was compatible with their needs; data which was collected had to be manipulated to fit the requirements of the tool. The commissioning of such a tool must include ensuring that it aligns with the data available to and required by the client and the systems in place within the user organisations. This did not occur in Devanha.

8.24 Many participants in Devanha did not believe that the use of the Procurement Consultant was aligned to the needs of the Devanha members. Some consultees saw geographic separation as a cause of poor communication between the procurement consultants and Devanha.

8.25 The procurement consultant believed that Devanha and its partners had not learned the necessary competences of collaborative working but put this down to leadership issues and Devanha not being committed to the necessary improvements.

8.26 Given that there were a variety of areas of frustration which were not satisfactorily resolved, it could also reasonably be argued that a stronger, more focused leadership from Devanha could have addressed these issues in consultation with the procurement consultant and created more satisfactory outcomes.