Domestic and non-domestic energy performance certificates: review

4. Thematic analysis

Within the spreadsheet holding the contributions extracted from the three consultation responses, each contribution was identified by its data source (i.e. LHEES, SEEP or PRS), assigned values for each of the three metatags (i.e. building sector, the nature of the EPC comment, and the issue classification), allocated to one or more broad themes, and allocated a keyword. This purpose of this activity was to allow the various individual contributions to be collated into common themes and topics, effectively, to take the diverse array of individual contributions and build a new narrative to inform discussion in the workshops and the topical analysis carried out in this report.

4.1 Frequency analysis of responses

The PRS consultation was the largest source of contributions on SAP, RdSAP, SBEM and EPC-related issues considered in this review: the total number of individual responses with comments; the total number of comments; the mean number of comments identified within a response; and the least number of responses where no relevant comment was identified. At the other end of the spectrum was the LHEES consultation, with the lowest number of individual responses, the fewest number of comments, the lowest mean number of comments, and the highest number of responses where no relevant comment was identified (see Table 4.1.1). The SEEP and PRS consultations included specific questions with regard to SAP, RdSAP, SBEM and / or EPCs; LHEES did not.

Table 4.1.1: Public consultations and SAP, RdSAP, SBEM and EPC-related contributions

Data Source	Total number of individual responses (n=343)	Number of individual responses where ‘no relevant comment’ identified (n=101)	Number of individual responses where relevant comments identified (n=242)	Total number of comments identified (n=1066)	Mean number of comments per response where relevant comment identified
LHEES	84	57	27	51	1.89
SEEP	98	36	62	226	3.64
PRS	161	8	153	789	5.16

Contributions focussed specifically on domestic building-related issues accounted for almost half of all of the contributions (i.e. 49.7%), while non-domestic building specific comments accounted for under 3%; contributions attributable to both domestic and non-domestic building accounted for the rest (i.e. 47.4%) (see Table 4.1.2). Overwhelmingly, the domestic building-related issues came from the PRS consultation (i.e. 87.5%), which is not surprising as this consultation was concerned explicitly about setting standards in private rented dwellings; only 4 of the 789 PRS contributions (i.e. 0.5%) were concerned with non-domestic buildings. The SEEP consultation specifically included mention of SBEM within its consultation questions, so it is not surprising that the largest number of non-domestic building specific comments were identified within this consultation (i.e. 64.5%), but the non-domestic building specific comments accounted for only a small part of all the SEEP comments included here (i.e. 8.8%).

Table 4.1.2: Building sector focus of public consultations responses on SAP, RdSAP, SBEM and EPC-related contributions

Building Sector	Count (n=1066)	LHEES (n=51)	SEEP (n=226)	PRS (n=789)
Non-domestic	31 (2.9%)	7 (row 22.5%) / (column 13.7%)	20 (row 64.5%) / (column 8.8%)	4 (row 12.9%) / (column 0.5%)
Domestic	530 (49.7%)	16 (row 3.2%) / (column 31.3%)	50 (row 9.4%) / (column 22.1%)	464 (row 87.5%) / (column 58.8%)
Both	505 (47.4%)	28 (row 5.5%) / (column 54.9%)	156 (row 30.9%) / (column 69.0%)	321 (row 63.6%) / (column 13.7%)

In terms of the tone of the EPC-related comments, the overwhelming majority were deemed to be neutral overall (i.e.74.2%), and within each of the three consultations (i.e. 98% of LHEES comments, 62.8% of SEEP comments, and 75.9% of PRS comments (see Table 4.1.3). Critical comments accounted for less than 20% of the total (i.e. 18.9%) and positive comments, 6.8%. While the PRS consultation accounted for the largest number of the critical comments, critical comments accounted for a larger percentage of the SEEP responses (i.e. 29.2%) than the other two consultations (17.0% of the PRS comments and just 2% of the LHEES comments).

Table 4.1.3: Tone of EPC comment of public consultations responses on SAP, RdSAP, SBEM and EPC-related contributions

EPC Comment	Count (n=1066)	LHEES (n=51)	SEEP (n=226)	PRS (n=789)
Critical	202 (18.9%)	1 (row 0.5%) / (column 2.0%)	67 (row 33.2%) / (column 29.6%)	134 (row 66.3%) / (column 17.0%)
Neutral	791 (74.2%)	50 (row 6.3%) / (column 98.0%)	142 (row 18.0%) / (column 62.8%)	599 (row 75.7%) / (column 75.9%)
Positive	73 (6.8%)	0 (row 0%) / (column 0%)	17 (row 23.3%) / (column 7.5%)	56 (row 76.7%) / (column 7.1%)

Across the three consultations, the responses were roughly evenly spread three ways across the calculation, assessment or reporting issue classification categories; database category only accounted for 2.3% of the overall responses (see Table 4.1.4). Differences in the concerns are seen in the individual consultation documents: calculation-related responses accounted for almost half of the LHEES and SEEP responses, but only a quarter of the PRS responses. Within the PRS responses, the assessment-related responses accounted for 40% and the reporting-related responses accounted for a third. While database-related responses were low overall, they accounted for a quarter of the LHEES responses.

Table 4.1.4: Initial issue classification of public consultations responses on SAP, RdSAP, SBEM and EPC-related contributions

Issue Classification	Count (n=1066)	LHEES (n=51)	SEEP (n=226)	PRS (n=789)
Calculation	319 (29.9%)	24 (row 7.5%) / (column 47.1%)	109 (row 34.2%) / (column 48.2%)	186 (row 58.3%) / (column 23.6%)
Assessment	402 (37.7%)	11 (row 2.7%) / (column 21.6%)	63 (row 15.7%) / (column 27.9%)	328 (row 81.6%) / (column 41.6%)
Reporting	321 (30.1%)	3 (row 0.9%) / (column 5.9%)	50 (row 15.6%) / (column 22.1%)	268 (row 83.5%) / (column 34.0%)
Database	24 (2.3%)	13 (row 54.2%) / (column 25.5%)	4 (row 16.7%) / (column 1.8%)	7 (row 29.2%) / (column 0.9%)

4.2 Broad Themes

The individual responses were allocated to, and further described within one or more of 18 broad themes (see Appendix A.2 for the 18 broad themes and the descriptors). A frequency analysis of the broad themes is set out in Table 4.2.1. The frequency analysis of the broad themes is broken down by the data source in Table 4.2.2, and by the issue classification in Table 4.2.3.

Table 4.2.1 - Frequency analysis of broad themes applied to contributions extracted from the responses to the three public consultations

The broad themes of methodology and modelling-related issues dominated the responses across all three public consultations. Beyond that, occupancy, database, and data-storage issues were of greater concern to the LHEES respondents than to the SEEP and PRS respondents. In contrast, assessor, reporting and improvement related issues were of greater concern to the PRS respondents than to the LHEES and SEEP respondents.

The broad themes of methodology and modelling-related issues dominated the responses across all the issue classification categories. Maybe not surprisingly, assessor related issues were the most frequently identified broad theme within the assessment classification, while reporting and improvement related issues were heavily identified within the reporting classification. Overall, the database classification accounted for significantly fewer issues than did the calculation, assessment and reporting classifications.

Table 4.2.2 - Frequency analysis of Broad Themes by public consultation data source

Broad Themes	LHEES	SEEP	PRS
Methodology (n=794)	48	191	555
Modelling (n=725)	44	170	511
Report (n=299)	3	46	250
Improvements (n=289)	5	37	247
Assessor (n=286)	2	31	253
Occupancy (n=128)	20	50	58
Built Form (n=106)	0	10	96
Conventions (n=88)	0	17	71
Location (n=65)	1	12	52
Heating (n=64)	9	23	32
Fabric (n=56)	0	47	9
Data Collection (n=52)	24	16	12
Data Storage & Retrieval (n=33)	17	4	12
Energy Supply (n=29)	0	14	15
Climate (n=26)	6	6	14
Condition (n=19)	1	4	14
Age of Dwelling (n=12)	0	4	8
Ventilation (n=11)	2	1	8

Table 4.2.3 - Frequency analysis of Broad Themes by Issue Classification

Broad Themes	Calculation	Assessment	Reporting	Database
Methodology (n=794)	316	153	311	14
Modelling (n=725)	314	139	260	12
Report (n=299)	39	76	184	0
Improvements (n=289)	37	75	177	0
Assessor (n=286)	0	278	8	0
Occupancy (n=128)	65	21	34	8
Built Form (n=106)	33	39	34	0
Conventions (n=88)	16	58	14	0
Location (n=65)	26	17	21	1
Heating (n=64)	42	11	9	2
Fabric (n=56)	22	25	9	0
Data Collection (n=52)	18	8	6	20
Data Storage & Retrieval (n=33)	5	2	4	22
Energy Supply (n=29)	23	3	3	0
Climate (n=26)	19	2	0	5
Condition (n=19)	1	18	0	0
Age of Dwelling (n=12)	3	7	2	0
Ventilation (n=11)	3	8	0	0

A pivot table analysis of the broad themes by data source, building sector, and issue classification is set out in Appendix A.4)

Additional descriptors were applied within each of these broad themes to allow more detailed topic analysis and identification of specific issues to occur for the workshops. These will be explored more fully in the next section.

4.3 Keywords

Finally, each contribution was described by one of 35 keywords (see Appendix A.3), to allow further filtering of the contributions, as necessary to inform the more detailed topic analysis and identification of specific issues to occur for the workshops. A frequency analysis of the broad themes is set out in Table 4.3.1. It can be seen that three keywords (i.e. surveyor skills, recommendations, and performance gap) each account for more than 10% of the response contributions. Two more (i.e. metric and minimum standards) account for between 5 and 10% of the response contributions. In total, these five keywords account for 60.9% of all the responses.

Table 4.3.1 - Frequency analysis of keyword applied to contributions extracted from the responses to the three public consultations

The frequency analysis of the broad themes is broken down by the data source in Table 4.3.2, and by the issue classification in Table 4.3.3.

The overall rank ordering of the keywords is influenced heavily by the PRS responses: the five most common keywords overall are also the five most common within the PRS responses (see Table 4.3.2). With SEEP, the most common three keywords were performance gap, benchmarking and decarbonisation; while amongst the LHEES responses, the most common keywords were real data, district heating and database.

Table 4.3.2 - Frequency analysis of Keywords by public consultation data source

Keyword (n=1066)	LHEES (n=51)	SEEP (n=226)	PRS (n=789)
surveyor skills (n=208)	1	12	195
recommendations (n=178)	2	17	159
performance gap (n=139)	4	67	68
metric (n=70)	1	13	56
minimum standards (n=54)		2	52
awareness of SAP (n=41)		13	28
decarbonisation (n=32)	2	18	12
traditional buildings (n=31)		2	29
new technologies (n=26)		8	18
QA procedures (n=24)		5	19
benchmarking (n=23)		20	3
database (n=22)	10	3	9
quality assurance (n=21)		7	14
district heating (n=19)	10	3	6
real data (n=19)	13	4	2
conventions (n=17)		1	16
integrity (n=17)		7	10
flawed (n=16)		2	14
review and update (n=16)		3	13
alternative model (n=15)	2	1	12
independence (n=12)	1	2	9
electric heating (n=11)	1	1	9
reporting (n=9)	1	3	5
ventilation (n=9)	2		7
accountability (n=6)			6
fuel poverty (n=6)		2	4
thermal mass (n=5)		2	3
administration (n=4)			4
affordable warmth (n=4)		3	1
thermal imaging (n=4)	1	2	1
embodied energy (n=2)		2
hard to treat (n=2)		1	1
windows (n=2)			2
improvements (n=1)			1
room in roof (n=1)			1

When disaggregated by the issue classification categories, a different keyword dominates each issue classification category: with calculation issues, the keyword ‘performance gap’; with assessment issues, the keyword ‘surveyor skills’; with reporting issues, the keyword ‘recommendations’; and with database issues, the keyword ‘database’ (see Table 4.3.3). Two keywords appear across all four issue classifications: benchmarking and district heating; although not amongst the most common keywords in any category, these two demonstrate a breadth to their nature.

Table 4.3.3 - Frequency analysis of Keywords by Issue Classification category

Keyword (n=1066)	Calculation (n=319)	Assessment (n=402)	Reporting (n=321)	Database (n=24)
surveyor skills (n=208)		202	6
recommendations (n=178)	23	26	129
performance gap (n=139)	100	21	18
metric (n=70)	36	5	29
minimum standards (n=54)	1	13	40
awareness of SAP (n=41)	4	2	35
decarbonisation (n=32)	26	1	5
traditional buildings (n=31)	22	9
new technologies (n=26)	19	2	5
QA procedures (n=24)		24
benchmarking (n=23)	6	7	9	1
database (n=22)			2	20
quality assurance (n=21)	1	20
district heating (n=19)	11	1	4	3
real data (n=19)	15	3	1
conventions (n=17)	1	13	3
integrity (n=17)		16	1
flawed (n=16)	8	2	6
review and update (n=16)	11		5
alternative model (n=15)	11	1	3
independence (n=12)		12
electric heating (n=11)	9	2
reporting (n=9)	1		8
ventilation (n=9)	3	6
accountability (n=6)		5	1
fuel poverty (n=6)	2		4
thermal mass (n=5)	5
administration (n=4)			4
affordable warmth (n=4)	1		3
thermal imaging (n=4)		4
embodied energy (n=2)	2
hard to treat (n=2)	1	1
windows (n=2)		2
improvements (n=1)		1
room in roof (n=1)		1

Key
cells in Bold identify the keyword most common within the Issue Classification category

In Table 4.3.4, the 35 keywords are cross-referenced against the 18 broad themes. What emerges is that five of the keywords fall within at least two-thirds of the broad themes identified in this review: surveyor skills, recommendations, performance gap, metric, and traditional buildings. These are not only amongst the most common contributions identified with the three public consultations but also demonstrate that these issues are not just narrowly focussed. Another nine of the keywords fall with between half and two-thirds of the broad themes: minimum standards, decarbonisation, new technologies, benchmarking, database, district heating, alternative models, electric heating and ventilation.

Table 4.3.4 – Frequency analysis of Keywords cross-referenced by Broad Themes

4.4 Non-domestic Buildings

The contributions from the three public consultations were categorised as pertaining specifically to either domestic buildings or non-domestic buildings, or both (i.e. they were concerned with EPCs or the energy performance of buildings but did not explicitly reference either SBEM, or SAP or RdSAP, so could in principle be applied to equally to both types of building).

4.4.1 Thematic Analysis

When the frequency of the broad themes identified in the contributions categorised as pertaining to ‘non-domestic’ and ‘both’ domestic and non-domestic buildings are sorted by the source of the contribution, what emerges is the dominance of the contributions from the PRS consultation (see Table 4.4.1): 976 of the 1568 responses (i.e. 62%) came from the PRS consultation. The same dominance of the PRS contributions is seen when the frequency of the assigned keywords is sorted by the source of the contribution (see Table 4.4.2): 290 of the 501 contributions (i.e. 58%) came from the PRS consultation.

Table 4.4.1: Frequency of Broad theme contributions by Consultation (non-domestic only and both sectors)

Broad Theme	LHEES		SEEP		PRS		Total
	Both	Non-dom	Both	Non-dom	Both	Non-dom
Methodology	26	6	129	18	259	4	442
Modelling	22	6	121	14	247	3	413
Report	1	0	29	5	100	1	136
Improvements	3	0	28	0	99	1	131
Assessor	0	1	23	0	72	0	96
Occupancy	10	0	34	1	28	1	74
Built Form	0	0	2	2	39	0	43
Convention	0	0	8	0	33	0	41
Heating	7	0	18	0	15	0	40
Data Collection	10	4	12	3	7	0	36
Location	0	0	4	3	17	0	24
Fabric	0	0	4	1	17	0	22
Climate	5	0	4	1	11	0	21
Data Storage & Retrieval	5	3	3	0	5	0	16
Energy Supply	0	0	8	0	5	0	13
Condition	1	0	3	0	8	0	12
Age	0	0	2	1	1	0	4
Ventilation	0	0	1	0	3	0	4
Total	90	20	433	49	966	10	1568

Table 4.4.2: Frequency of Keywords by Consultation (non-domestic only and both sectors)

Keyword	LHEES		SEEP		PRS		Total
	Both	Non-dom	Both	Non-dom	Both	Non-dom
recommendations	2	0	13	2	78	1	96
performance gap	3	0	55	3	31	1	93
quality assurance	0	0	17	0	48	0	65
Metric	1	0	8	0	35	0	44
benchmarking	0	0	8	12	1	2	23
decarbonisation	1	0	13	0	9	0	23
traditional buildings	0	0	2	0	19	0	21
awareness of SAP	0	0	8	0	12	0	20
real data	8	4	2	1	2	0	17
flawed	0	0	2	0	13	0	15
review & update	0	0	2	1	10	0	13
district heating	8	0	3	0	1	0	12
minimum standards	0	0	2	0	6	0	8
database	2	0	2	0	4	0	8
alternative model	0	2	0	0	4	0	6
surveyor skills	0	1	5	0	0	0	6
electric heating	1	0	1	0	4	0	6
new technologies	0	0	5	1	0	0	6
reporting	1	0	0	0	2	0	3
thermal imaging	1	0	2	0	0	0	3
administration	0	0	0	0	3	0	3
ventilation	0	0	0	0	2	0	2
thermal mass	0	0	2	0	0	0	2
embodied energy	0	0	2	0	0	0	2
hard to treat	0	0	0	0	1	0	1
room in roof	0	0	0	0	1	0	1
affordable warmth	0	0	1	0	0	0	1
fuel poverty	0	0	1	0	0	0	1
Total	28	7	156	20	286	4	501

Non-domestic specific issues with regard to SBEM and EPCs were primarily raised via the responses to the SEEP and LHEES consultations, and were at a significantly lower level than the comments regarding domestic buildings. This in part is not surprising: the LHEES consultation was primarily focussed on local heat networks, and while the SEEP consultation sought discussion on both domestic and non-domestic buildings, there were no specific questions about non-domestic buildings so most of the contributions were categorised as being of concern to ‘both’ domestic and non-domestic buildings. The responses relating to the PRS consultation (by far, the most numerous overall) were primarily concerned with domestic building issues. Where contributions were identified as pertaining to ‘both’ the domestic and non-domestic buildings (e.g. concerned with EPCs or the energy performance of buildings but did not explicitly reference either SBEM or SAP or RdSAP), the decision was taken to include these within the discussions within the rest of this report where appropriate.

Only 31 contributions were categorised as pertaining specifically to the non-domestic sector, and represented a very small proportion (2.9%) of the overall 1066 contributions. These 31 non-domestic specific building contributions fell across 13 of the 18 broad themes.^[44] Most of the non-domestic contributions were concerned with either methodological or modelling issues, or both. The overall frequency of the broad theme results are summarised by the public consultation they came from in Table 4.4.3 below.

Table 4.4.3: Frequency of Broad theme contribution by Consultation (non-domestic only)

Broad theme	LHEES	SEEP	PRS	Total
Methodology	6	18	4	28
Modelling	6	14	3	23
Data Collection	4	3	0	7
Report	0	5	1	6
Data Storage & Retrieval	3	0	0	3
Location	0	3	0	3
Built Form	0	2	0	2
Occupancy	0	1	1	2
Age	0	1	0	1
Fabric	0	1	0	1
Improvements	0	0	1	1
Assessor	1	0	0	1
Climate	0	1	0	1
Total	20	49	10	79

4.5 Comment on the Thematic Analysis

Through the frequency analysis, the use of the pivot tables, and the application of the broad theme descriptors, and keywords, the thematic analysis allowed 1066 diverse individual comments to be collated in a manner to highlight common themes and concerns, and to gauge the strength of these concerns through the frequency they were expressed in the public consultations. Some of this frequency reflected specific questions included within the consultation exercises.

What emerges is overlapping broad theme and keyword groupings touching a large number of the responses to the public consultations:

• SAP, RdSAP and SBEM metric-related issues: performance gap, metric, methodology and modelling
• Assessor-related issues: assessor, surveyor skills, quality assurance procedures, quality assurance, integrity, and independence
• EPC reporting and recommendation-related issues: reporting, recommendations, minimum standards, improvements, methodology, and modelling
• Traditional Buildings-related issues: traditional buildings, methodology, modelling, built form, and fabric

A separate analysis was carried out on the non-domestic building-related contributions. The small number of contributions specifically identified as pertaining to non-domestic buildings where overwhelmed by the concerns for domestic dwellings arising from the responses to the PRS consultation.

At a lower level of response, keywords of a more technical nature that touch a number of broad themes also emerge:

• decarbonisation
• new technologies
• benchmarking
• database
• district heating
• ventilation
• electric heating

This ‘bottom-up’ approach was used to identify topic areas and technical issues to be explored in more detail for further consideration through presentations, discussion and deliberation at the workshops. The shaping of the workshop topics and issues is discussed in the next section.