Home Energy Efficiency Programmes Scotland: annual review 2016-2017

Part 4: Objective 3 - Housing Stock

The third stated objective of Warmer Homes Scotland is to improve Scotland’s housing stock and one of the key guiding principles is that the impacts of the scheme should be sustainable and long-term. On this basis, the scheme aims to ensure that any measures installed in properties are appropriate and will deliver benefits for current and future inhabitants.

Informed by last year’s Annual Review, this review sought to address the following questions to determine whether Warmer Homes Scotland is meeting this objective:

• Are appropriate measures for individual properties being recommended through the survey process?
• Why are some measures specified by the scheme not being recommended to or taken up customers?
• What are the most common challenges or concerns encountered by scheme surveyors when making recommendations?
• Are measures recommended being packaged together effectively to deliver the greatest benefit to the customer, within the funding available where possible?
• Should additional measures be included within the scheme to provide more tailored support to on-the-ground circumstances?
• Is the quality of installations delivered through the scheme being upheld over time and across regions in Scotland?

The most and least frequently installed measure during 2016/17 can be found at Table 5 .

From this, we can see that the two most frequently installed measures were smoke alarms and carbon monoxide detectors both of which are mandatory measures under the scheme. However, if you add all the different types available under the scheme, the most common type of measures installed were heating systems followed by boilers.

Of those measures installed during 2016/17, the least frequently installed were Solar Thermal, Hybrid Wall Insulation, Solar Pv and Biomass Boilers and there were 13 measures available under the scheme that were not installed in any properties. This likely reflects the make-up of the domestic property archetypes in Scotland and the relatively new technology these measures offer, although cost may also play a part. The reasons why some measures were not installed or recommended was examined more closely at the surveyors’ workshop held in October 2017. The outcomes of this workshop are discussed in more detail in Part 4 Section i. Survey of this review.

i. Survey

The initial survey undertaken by Warmworks is thorough and takes account of the property type and location to establish the most appropriate suite of measures that are tailored to the individual property. The survey is carried out using rdSAP software and relies on the skills and knowledge of the surveyor to determine the most appropriate suite of measures for each property, including recommending measures specified by the Warmer Homes Scotland contract that are not included in the rdSAP software.

Pennington Choices audits a 5% sample of surveys on a monthly basis, assessing them against set criteria. The results of the pre-installation audits are summarised at Table 6. This shows that a substantial majority of Warmer Homes Scotland surveys pass the audit requirements. This follows a similar trend to 2015/16, suggesting that surveys continue to be carried out in line with the contractual requirements and surveyors are recommending appropriate measures.

Surveys that failed the audit process increased from 0.2% in 2015/16, to 1.5% in 2016/17 and surveys being given a Pass Advisory at audit increased from 7.3% in 2015/16 to 11.5% in 2016/17. Given the scale of increase in surveys being conducted by Warmworks surveyors between the two review periods, this percentage seems proportionate and within a tolerable threshold to cause no concern regarding the appropriateness and robustness of the survey process.

Having said this, it is interesting to note that a direct comparison between only those months covered by the review process in 2015/16, i.e. October to March, and the same period in 2016/17, shows no increase in the number of “failed” surveys audited (1). The majority of surveys that failed the audit process in 2016/17 were carried out during the Spring/Summer period. However, without more data available on the individual contexts and circumstances of each survey during this period it is not possible to make any valid inferences as to whether this pattern can be reasonably explained by any underlying factor(s) but it may be possible to do so in future years when additional data sets become available.

Table 6: Results of Pennington audit of Warmer Homes Scotland surveys 2016/17

Month	Total Number of Audit	Pass		Pass: Advisory	Pass: Remedial	Fail
		No.	% of total completed
April	35	35	100	0	0	0
May	36	36	100	0	0	0
June	30	29	96.5	1	0	0
July	21	19	90.5	2	0	0
August	28	28	100	0	0	0
September	25	25	100	0	0	0
October	33	31	94	2	0	0
November	30	29	90	1	0	0
December	35	34	96.5	1	0	0
January	19	19	100	0	0	0
February	43	41	95	2	0	0
March	28	28	100	0	0	0
Total	363	354	97.5	9	0	0

Figure 12: Overall comparison of Pennington Choices audit inspections Year 1 versus Year 2 of the Warmer Homes Scotland contract

In response to Recommendation 8 in the 2015/16 Annual Review, to further investigate “why some measures specified by the scheme have not been offered to customers”, in October 2017 an interactive workshop with a sample group of 6 Warmworks surveyors was organised and chaired by the Scottish Government. The summary report of the surveyors workshop can be found at Appendix 2 to this review.

General consensus from the workshop suggested the main reasons why certain available measures (identified in the Pennington Choices Measure Specification Audit Report that is attached to the 2015/16 Annual Review) are not recommended, or only recommended on a small number of occasions, are:

• The perception by the surveyors that most of these measures are only suitable for new build properties.
• Customer attitudes and perceptions of certain measures (particularly new, innovative and unfamiliar technologies such as air source heat pumps) mean that a longer-term process of education around these measures is required.
• The higher costs associated with the installation of some of these measures, which in turn means a higher customer contribution level is required, deters many customers from accepting certain or initial recommendations made during the survey. Surveyors consider this in order to revise the overall recommendations package made, to focus on the measures that offer the customer the most benefit within the constraints of the grant levels.

Additional comments provided by the surveyors included:

• The current list of measures available under the scheme could be improved by adding the option to recommend the “installation of ordinary heavy curtains”, in place of the current option, “flexible thermal linings”. The former is seen as a more practical and low cost solution, and is less intrusive during install for the customer. It was suggested that local charities might be able to collaborate with Warmworks to deliver the measure.
• In general the grant level available through the scheme is appropriate, but in the case of rural properties - often stone built with “hard to treat” wall types - it is often considered too low to enable surveyors to offer the customer a fully comprehensive (holistic) package of complementary recommendations. This leads to surveyors having to make trade-offs in their decision of what to recommend, between achieving the greatest energy efficiency improvement for the property versus ensuring an appropriate level of cost (customer contribution) given the personal circumstances of the customer.

The following recommendations have been made after analysing the feedback received at the workshop:

Recommendation 5 - Given some of the measures are deemed to be more suitable for new builds and that it was suggested that the design of some of the measures, for example flexible thermal linings and water efficient taps, could be redesigned to increase their suitability, the existing list and design of the measures available under the scheme should be reviewed and updated to reflect the on-the-ground experience and insights of surveyors and their understanding of the suitability of the measures when taking the circumstances of customers, sensitivities around property type and need for any associated educational activities due to installation of unfamiliar technology.

Recommendation 6 - Consideration should be given to building on current training provision for surveyors and establishing a formal programme of annual refresher training for them to ensure any gaps are identified and addressed. Due to their pivotal role in the customer journey, this should include engagement with Scottish Government officials on the contract requirements relevant to the survey process. As part of this engagement, Scottish Government should provide Warmworks with details of the scenarios where some of the more underutilised measures were deemed suitable for use during the schemes development. All current surveyor training and manuals should be updated to reflect the outcome of this review.

Recommendation 7- Agree an appropriately robust process to enable suitable measures to be suggested, appraised and tested for future inclusion in the scheme by either Warmworks or the Scottish Government.

Recommendation 8 - To identify any gaps or areas for improvement in the customer journey, a review of the handling, follow up, recording and action taken on any comments provided by surveyors in completed household surveys should be conducted.

ii. Installation & Inspection

Warmworks subcontract all Warmer Homes Scotland installations to a supply chain of 30 companies (correct over the period of the review) operating across Scotland, including Everwarm that can undertake up to 50% of the installation work and is contracted to undertake work when other installers are unable to. All installers must have, or be working towards, PAS 2030 certification and must meet the other measure-specific standards set out in the contract, e.g. MCS, Oftec, Gas Safe. A Warmworks inspector inspects all works carried out under the Warmer Homes Scotland contract within 5 days of the works being completed.

Given the nature of the work being undertaken and the Warmer Homes Scotland customer base, it is vitally important that installations are carried out in line with the technical, health and safety, and customer care standards required by the contract.

To help monitor the quality of installation work being carried out under the Warmer Homes Scotland contract, quality assurance ( QA) checks are carried out by Pennington Choices Ltd on 2 work-in-progress ( WIP) jobs per month and a random 20% sample (with no fewer than 65 jobs per month) of completed installations through on-site and desk-based audits. Table 7 shows a summary of their overall findings (including survey, WIP, and post-installation) for 2016/17.

Table 7: Results of Pennington Choices on-site inspections of a 10% sample of Warmer Homes Scotland jobs

Inspection Outcome	Overall Result in 2016/17
Pass	79%
Pass Advisory	11%
Remedial	9%
Fail	2%
At Risk	0

Figure 13: Comparison of the results of the QA audits in 2015/16 and 2016/17

The result of the comparison of the outcome of the QA audits carried out by Pennington Choices during both 2015/16 and 2016/17 show that the total number of installations that passed inspection during the audit process has increased by 12%. This improvement to performance was delivered despite the increase in numbers of monthly installations during this period. This increase indicates that existing quality assurance measures are being applied consistently and effectively, remain robust and indicates an increase in the quality of installations being carried out under Warmer Home Scotland. The number of jobs marked as a “fail” at inspection has decreased by 2%, and the number “at risk” remains constant.

In response to Recommendation 9 in the 2015/16 annual review, a comparison of remedial work between 2015/16 and 2016/17 has been undertaken and the results are shown at Table 8. The analysis shows that the number of WIP jobs requiring remedial work, including those jobs recorded at audit as a “pass advisory”, “pass remedial” or “fail”, fell from 49% in 2015/16 to 46% in 2016/17, and the number of Post-Installation jobs requiring remedial work fell from 51% in 2015/16 to 29% in 2016/17. The relatively large decrease in the percentage of Post-Installation jobs requiring remedial work may imply that while sub-contractors continue to make some errors during the installation process ( WIP), the quality assurance protocols that Warmworks’ have in place at the end of the installation work are effective. The continuing improvement in quality assurance evidenced here should help increase confidence amongst Warmer Homes Scotland customers that the work carried out in their properties is of a high standard; and over the longer term, this trend in quality improvement may translate into greater economic efficiency in delivery of the programme and workload benefits for Warmworks sub-contractors.

Table 8: Number of installations requiring remedial work identified by the QA process in 2016/17

Inspection Category	Review Year	Total No. of Inspections	Total No. Requiring Remedial Work*	% Requiring Remedial Work
WIP*	2015/16	69	34	49
	2016/17	24	11	46
Post-Installation	2015/16	492	253	51
	2016/17	791	229	29

*This includes all jobs audited at post-installation phase that have been recorded as “pass advisory”, “pass remedial” or “fail”.

However, the extent to which action is taken, monitored and reported on accurately by Warmworks and its sub-contractors, in response to remedial work identified by Pennington Choices has not been assessed within the parameters of this review.

Recommendation 9 - An evaluation of the corresponding action taken, monitoring and audit process followed by Warmworks and its sub-contractors in response to remedial work identified by Pennington Choices should be undertaken. The outcome of this evaluation and any resulting actions that are made should be reviewed in the 2017/18 Annual Review.

iii. Right First Time KPI Adjusted Target Level

The 2015/16 review considered the impact that operational delivery and the behaviours that drive subcontractors had on the contractual Key Performance Indicators ( KPIs), particularly Right First Time ( RFT). During this evaluation, it was identified that, while the inspection process was pushing up standards, it was also impeding Warmworks’ ability to meet the RFT KPI target of 98% in areas with low customer volumes. Due to a statistical anomaly, the target in these areas effectively became a 100% target. This is because Warmworks is paid service provider fees by the Scottish Government to deliver Warmer Homes Scotland. The amount it is paid for each installation is dependent on the regional performance targets achieved and this statistical anomaly was identified as creating an artificial target of 100% in regions where the number of installs was low.

A 3 month trial period to adjust the RFT KPI was implemented as a result of this finding. The adjusted RFT KPI allowed for one failed job in areas with low volumes. In this context, low volume is defined as under 25 jobs per month. The trial was reviewed in December 2016 and it was agreed to extend the trial by a further 3 months to the end of March 2017 as the numbers installed in the initial 3 month period did not allow the trial amendment to be sufficiently tested.