Domestic and non-domestic energy performance certificates: review

3. Research Methodology

The Scottish Government published an analysis of responses to the three public consultations, and provided a generalised summary of the comments. This analysis did not examine in detail specific critical comments on the effectiveness of EPCs in Scotland, nor did it explore how these issues could be addressed going forward.

The research methodology employed to complete this review of domestic and non-domestic EPCs comprises of several specific tasks: a literature review, thematic analysis, workshops and sensitivity analysis of various issues.

3.1 Literature Review

Rather than the generalised summaries of the respective three consultation exercises that the Scottish Government published in November 2017, the initial focus of this project was the actual individual public responses received via the three separate public consultation exercises set out in section 3.1.1 below. These responses are all published on-line.

The objective of this desk review was therefore to identify, collate and categorise specific concerns about the SAP, RdSAP, and SBEM / NCM methodologies arising from the three separate consultation exercises, to allow a more detailed analysis of the nature of the concerns, and possible actions going forward. This thematic analysis would be used to inform the workshop discussion topics and subsequent identification of possible actions.

3.1.1 Data Sources

The primary sources of data to inform this report were the published responses^[29] to three separate public consultations contributing to the development of Energy Efficient Scotland (previously SEEP) and other related policy matters:

1. Scotland’s Energy Efficiency Programme (SEEP)^[30] - opened 24 January 2017 and closed 30 May 2017

• Scottish Government analysis of the responses received to the consultation on Scotland's Energy Efficiency Programme (SEEP) - published November 14, 2017^[31]
• 98 Published Responses^[32]

2. Heat and Energy Efficiency Strategies, and Regulation of District Heating (LHEES)^[33] - opened 24 January 2017 and closed 18 April 2017

• Scottish Government analysis of Responses to the Consultation on Heat and Energy Efficiency Strategies, and Regulation of District Heating – published November 14, 2017^[34]
• 84 Published Responses^[35]

3. Energy Efficiency and Condition Standards in Private Rented Housing (PRS)^[36] - opened 7 April 2017 and closed 30 June 2017

• Scottish Government analysis of responses to the public consultation on energy efficiency and condition standards in private rented housing – published November 14, 2017^[37]
• 161 Published Responses^[38]

3.2 Thematic analysis of the public consultation responses

Text searches were carried out on 343 individual responses to the three public consultations using 17 different search terms (see Appendix A.1 for the full list of the initial search terms used). These search terms were identified through an iterative process involving reading the initial consultation documents, the consultation questions, the individual published responses, and the concerns of this review. This search exercise was not just concerned with the consultation document questions specifically addressing SAP, RdSAP, SBEM or EPC issues, but drew from across the responses, to inform on the breadth of EPC-related issues.

Of the 343 published responses, 101 were categorised as having ‘no relevant comment’ with regard to SAP, RdSAP, SBEM or EPC issues. The remaining 242 responses generated 1066 comments that were identified as being of concern to this review, and then extracted and collated within a spreadsheet. Each comment was identified by its response number^[39] and the consultation it was extracted from.

Subsequently, the 1066 contributions were reviewed three times by the same person: first to assign the three metatags to each comment; then, to allocate the response to one of 18 broad themes along with a descriptor; and finally, to ascribe a keyword to provide an overall summary of the response content. This approach allowed the individual responses subsequently to be collated into common themes and topics for further discussion and analysis in the workshops and in this report.

A graphic representation of this analysis process is provided in Figure 3.2.1 below.

Figure 3.2.1 - Process diagram describing the method for analysing and categorising comments relating to the use of EPCs in buildings

3.2.1 Metatags

All 1,066 contributions were categorised according to three different metatags (see Figure 3.2.2).

Figure 3.2.2 – Three metatags used to categorise responses

• Building Sector (domestic / non-domestic / both): that is, the response pertained specifically to domestic buildings or non-domestic buildings, or both (i.e. the response concerned EPCs or assessment of the energy performance of buildings but did not explicitly reference either SBEM, or SAP or RdSAP so, in principle, it could be applied equally to either building type).

• EPC Comment (critical / neutral / positive): that is, the response was explicitly ‘critical’ (i.e. negative in tone) of SAP, RdSAP, SBEM or EPCs; ‘neutral’ if it did not explicitly mention SAP, RdSAP, SBEM, or EPCs within its content; or ‘positive’ if it explicitly mentioned SAP, RdSAP, SBEM or EPCs in a positive way.
• Issue Classification (calculation / assessment / reporting / database): that is, the response was assigned to one of the four stages of the EPC process (as set out in section 2.1):
  • calculation, that is, the response was concerned with either the BREDEM methodology, or the SAP, RdSAP or SBEM calculation methodologies used to define the energy models in theory, or with how these methodologies are represented within the calculation models and software via algorithms or data requirements.
  • assessment, that is, the response was concerned with the assessor, the data collected by the assessor, the conventions governing the collection of data by the assessor, the inputting of the data into the software by the assessor, or the quality assurance procedures overseeing the assessor.
  • reporting, that is, the response was concerned with the physical output of the EPC process: the certificate or the attached advice report and recommendations, or the process used to produce the recommendations
  • database, that is, the response was concerned with the storage of the EPC-related data on the national register, access to the data, or retrieval of the data from the register.

3.2.2 Broad themes

Each of these 1,066 contributions was then assigned and described within one or more of 18 broad themes (see Figure 3.2.3 below). Additional descriptors were applied within each of these broad themes to enable more detailed analysis of the broad themes (see Appendix A.2). A response could be assigned to any number of these 18 broad themes.

Figure 3.2.3 – 18 broad themes used to categorise responses

3.2.3 Keywords

Finally, each contribution was described by one of 35 keywords (see Figure 3.2.4 below), to allow further filtering of the contributions as necessary to inform the thematic and topic analysis. Unlike the broad themes, a response was described by a single keyword / phrase. This approach resulted in some responses being split into several parts to adequately describe their content.

Figure 3.2.4 – 35 keywords assigned to responses

3.3 Workshops

Rather than limit this research project to a desk analysis of the responses, the project sought to open the discussion up to a wider public forum, to canvass opinion on the EPC issues and possible actions going forward.

The thematic analysis informed the identification and content of the topics for presentation at four workshops held around Scotland during February and March 2018. These events were day-long open events designed with the intention to be pro-active in canvassing discussion and opinions on the topics, and allowing feedback to be captured from each participant. Presentations on the broad themes distilled from the analysis were introduced with evidence provided by the research team, followed by a facilitated discussion with the participants. Participants were asked to record any comments and/or their views of this evidence within structured workbooks to allow the research team to determine if this evidence had any bearing on consolidating or changing opinions on the effectiveness of the EPC system in Scotland.

The discussion at the workshops, and the feedback captured in the workbooks was used in the topical analysis presented in Section 6, and in the identification of possible actions going forward.

3.4 Supplementary Topic Notes: Additional Modelling and Sensitivity Analysis

As part of this project, the research team produced six supplementary topic notes on specific EPC-related concerns with regard to domestic dwellings that came out of the responses to the three public consultations. These topic notes included more detailed discussion on issues relating to:

• assessing traditional buildings;
• assessing rooms in the roof;
• using default and full window measurement data within the RdSAP program;
• comparing the impact of RdSAP ventilation defaults within a full SAP program;
• assessing the impact of changing the thermal mass parameter within RdSAP; and,
• comparing the differences between using a full SAP program and RdSAP with regard to assessing community heating.

The underlying intention of these supplementary topic notes and sensitivity analysis was to produce empirical data to inform the workshop discussions rather than rely on anecdotal evidence and discourse alone. These topic notes are published in a separate addendum to this report.

3.5 Topic Analysis and Identification of Possible Actions

The thematic analysis, workshop presentations, workshop feedback, supplementary topic notes and sensitivity analysis were distilled down to topical discussions on the SAP, RdSAP, SBEM and EPC-related issues and, most importantly, to identify possible actions to enhance the EPC-related processes going forward.

In identifying possible actions, they have been categorised (and colour-coded in this report) using the issue classification breakdown set out in section 3.2.2 above – that is, calculation, assessment, reporting and database–related actions. These categories not only reflect the four stages of the EPC process, but how a potential action is categorised has implications for how it can be dealt with, and by whom.

Figure 3.5.1 – 4 categories for possible EPC-related actions

Calculation actions would need changes to the underlying BREDEM / NCM methodology and / or the calculation models that are used to calculate the energy performance of a building, and its SAP score (i.e. SAP or RdSAP) or its Building Energy Performance Rating (i.e. SBEM). For example, these changes may include adding something that is not included currently within BREDEM, or SAP, RdSAP and SBEM/NCM, or modifying the equations, algorithms, or the default reference data used within the models.

These actions are currently within the remit of, and lead by, two UK government departments, BEIS and MHCLG. The Scottish Government has an input into the discussions, but so too does the wider industry. Major changes in the underlying methodologies do not occur quickly. The development of ‘SAP 10’ (i.e. the replacement for SAP 2012) provides a case in point. Proposed changes went out to consultation in November 2016, with a closing date of end of January 2017. The government response to the consultation was published in November 2017, with the formal SAP 10 document only published in July 2018^[40], and it may be another year or more before it is fully implemented in practice^[41]. Yet, even before the preparation of the consultation, there was more than a year of work in the preparation of a draft SAP 2016^[42] and a number of technical discussion documents^[43]. Changes to the underlying methodologies are likely to have a time frame of at least 3 years.

As the underlying methodologies are empirically-based, changes in SAP, RdSAP and SBEM are based on new data. If a change to the underlying methodology also requires data collection to inform or validate the process, then that will increase the time required to make a change.

Equally, simply making changes to the calculation software involves time for the accreditation organisations to make the changes, and then for the software to be tested and approved.

Assessment actions would need changes to the conventions, quality assurance procedures, or training that inform the data collection and survey of a building. Changes to conventions are currently within the remit of SAP, RdSAP and SBEM working groups, and any proposed changes signed off by BEIS or MHCLG respectively. Again, the Scottish Government has an input into these working groups, as does the wider industry. Simply changing a convention (that requires no change to the software) is likely to take up to a year to implement. Setting new training standards or qualifications would take time to develop, agree, and then implement. Once they were agreed, training programmes would need to be revised and rolled out. Invoking new quality assurance procedures, which falls within the remit of the Scottish Government, is likely to take at least 2 years to agree with the accreditation organisations and to roll out.

Reporting actions would incur changes to the format of the EPC or the associated advice report and / or recommendations, or the underlying protocols that determine which recommendations appear or do not appear on the EPC, their order, and the technical specification of the recommendation that is modelled (i.e. Appendix T of the SAP methodology).

The format of the EPC or its associated advice report and the recommendations are within the remit of the Scottish Government. Changing the format would impact on the accreditation organisations and software providers where it would require software changes. Changing the basis used to produce the recommendations (i.e. Appendix T), either to produce an alternative format for the recommendations, or to develop a completely different approach such as the least cost improvements to meet a minimum defined standard, would require changes to the methodology and to the software. Simply adding or revising text within the existing EPC format would take from a few months to a year to agree, approve and to implement. A more fundamental change to the calculation and presentation of recommendations would take more than a year to implement.

Database actions would need changes to the way we store, access and or retrieve the EPC-related information from the national register. Again, changes to the database arrangements are within the Scottish Government remit, but the actual database is managed by EST Scotland who are appointed as ‘keeper of the register’. Changes to the database arrangements would require to be programmed and implemented as part of an overall development plan for the register.

All possible actions identified across the four categories above were also assessed against an indicative framework with regard to the rationale for the change, the time frame needed to implement the change, the cost of making the change, the impact on the rating, and the magnitude of the building stock affected (see Figure 3.5.1).

Figure 3.5.1 – Indicative framework to assess rationale, time frame, cost, impact and magnitude of possible changes to EPC-related possible actions