Scottish House Condition Survey: 2021 Key Findings
Figures from the 2021 survey, including updated fuel poverty rates, energy efficiency ratings and data on external disrepair.
6 External+ Data Quality
Key Points
- Due to Covid-19 restrictions the 2021 Scottish Household Survey (SHS) was undertaken using a push to telephone/video approach and the 2021 Scottish House Condition Survey (SHCS), which is part of the SHS, was undertaken using an external+ approach.
- Households were invited to participate in the 2021 SHS by advanced letter. When households did not opt-in, interviewers were unable to knock on doors to encourage participation, but were able to attempt to make contact by telephone if the household was matched to a telephone number. Interviews were conducted via telephone/video. They are usually conducted by in-home face-to-face interviewing.
- If households then agreed to participate in the follow-up 2021 SHCS, a qualified surveyor would visit the dwelling to carry out an external-only inspection. Data on internal aspects of the dwelling required for energy modelling were obtained from the householder by telephone or (in a small number of cases[17]) from an Energy Performance Certificate.
- Due to the change in approach for the 2021 SHCS, the results are not directly comparable with the National Statistics from previous waves of the survey.
- As such, we are publishing the key findings as Experimental Statistics representing a snapshot of the key attributes, energy efficiency and condition of the housing stock and fuel poverty levels in 2021. The results for 2021 should not be compared with those for previous or future years.
- The enforced changes to the 2021 SHS have resulted in the profile of respondents changing and issues with representativeness, which also impact the 2021 SHCS. Similarly, the changes to 2021 SHCS have resulted in mode effects, particularly where surveyors have had to rely on householders providing information by telephone, e.g., on the extent of low-energy lighting or presence of secondary heating systems.
- Calibration totals for household tenure and deprivation were added to the usual SHCS calibration model. These resolved some but not all of the issues with the representativeness of the sample.
- We have found families, low-income households, and households with pre-payment meters to be underrepresented after calibration weighting. Those who own outright are overrepresented.
- Key estimates from the SHCS are impacted by the representativeness of the sample and the mode effects. It is likely that the rates of fuel poverty, urgent disrepair to critical elements and households meeting (but not exceeding) the bedroom standard are being underestimated. It is likely that energy efficiency ratings and the percentage of dwellings exceeding the bedroom standard (by 2 or more bedrooms) are being over estimated.
- We do not propose adopting the external+ approach for future waves of the SHCS as it is not possible to collect all the data through external only inspections. If this approach were to be used again in the future, then consideration would have to be given as to how to achieve a more representative sample and how to address the mode effects due to relying on householders providing information on internal aspects of the dwelling rather than qualified surveyors obtaining this information through inspections.
6.1 External+ Approach
The Scottish House Condition Survey (SHCS) usually involves a visual inspection of the inside and outside of the property. However, due to Covid-19 restrictions the 2021 SHCS was carried out by an external-only inspection, supplemented with alternative sources of data, e.g., from the Energy Performance Certificate (EPC), and the householder providing information to surveyors via telephone.
The external+ approach was designed to provide as reliable as possible estimates of key statistics, including on fuel poverty, energy efficiency and external repairs, while maintaining no contact with the household. No data was collected on internal aspects such as room repairs and aspects of housing standards.
The 2021 external+ SHCS questionnaire was like that for previous years. However, only those questions within the red boxes were asked as part of the external+ survey.
The physical (SHCS) fieldwork took place in COVID protection levels 0, 1 and 2 only. Households in level 3+ areas were still invited to participate in the external+ physical survey, but the appointment was banked and carried out only once the area had returned to a lower protection level.
No summary surveys (dwelling descriptions and abbreviated dwelling descriptions) were collected.
6.2 Potential sources of bias compared to previous waves
The 2021 social survey adopted the push to telephone/video approach used in 2020. This approach is known to have introduced bias in the achieved sample compared to previous waves. This was documented in the Scottish Household Survey 2020: methodology and impact of change in mode report. One example of this is an over-representation of owner-occupied households and an under-representation of households in the rented sectors (prior to calibration weighting).
As the physical survey is a subsample of the social survey, any bias in the composition of the achieved sample for the social survey will inevitably impact on the composition of the achieved sample for the physical survey in a similar way.
Other biases may have been introduced due to the change in the mode of collection. This is unlikely to have impacted on the external aspects of the physical survey, e.g., external repairs. However, it is likely to have impacted on aspects of the physical survey which would have normally been collected as part of the internal inspection, e.g., the extent of low-energy lighting or presence of a secondary heating system, where surveyors would have been reliant on the householder providing this information via telephone or data from EPCs.
It is worth noting that not all households have a valid EPC and coverage varies by household and dwelling characteristics. For example, newer dwellings are more likely to have a valid EPC than older dwellings and households in the rented sector are more likely to have one than those that are owner occupied. As EPCs are valid for ten years the data can be up to ten years out of date, in the event of any energy efficiency improvements subsequently made to the dwelling.
We have no way of identifying when information from an EPC was used by surveyors. However, anecdotal evidence from surveyors suggests that this was rarely the case. Therefore, any bias due to the change of mode is likely to be mainly due to surveyors relying on householders self-reporting information via telephone that they would usually collect as part of the internal inspection of the dwelling.
6.3 Sampling
Usually around half of the households selected to participate in the social survey are asked to agree to a follow-up visit for the physical survey. Like the social survey, all assumptions underpinning the sampling approach had to be revised for the 2021 physical survey.
Pre-pandemic, sample targets were set using estimates of the conversion rate from the social survey to the physical survey by local authority. However, the 2020 push to telephone/video social survey (the approach that was also adopted in 2021) suggested that owner occupiers were more likely to take part than households in the private and social rented sectors. Therefore, rather than allocating an address to the physical survey before the social survey had been completed as had been done previously, sampling was done within the social survey interview after the household tenure had been established. The social survey interview script routed a certain proportion of households into the physical survey based on tenure and local authority, with the goal being to achieve a more representative sample of owner occupiers and renters and ensure the sample contained enough renters to allow disaggregation of key statistics by tenure.
Table 6.1 shows the 2021 SHS sample by local authority. It should be noted that the sample was considerably larger than previous years, reflecting the lower response rate associated with the push to telephone/video approach used for the 2021 social survey. A total of 9,952 social interviews were achieved, giving a response rate of 11%. Furthermore, the sample was drawn in two halves. Telephone matching was used for the first half of the sample but not the second. Telephone matching was found to improve the response rate (25% compared to 10%), hence the fact the first half of the sample was smaller than the second, but increase the non-response bias in the achieved sample. For further details see the section on survey response in the SHS 2021 methodology and fieldwork outcomes report.
As shown in Table 6.2, in the first half of the 2021 social survey all renters were routed into the SHCS and asked to agree to a surveyor visit. For owner occupiers, the sampling fraction differed by local authority, ranging from 34.1% in Fife to 48.8% in Scottish Borders. The sample fractions were revised for the second half of the 2021 sample based on updated projections of the likely number of respondents by local authority and tenure from the first half of the sample. For owner occupiers this ranged from 10.4% in West Lothian to 48.8% in South Ayrshire. For renters this ranged from 1.9% in Renfrewshire to 96.9% in East Lothian. The large variation in the sample fractions by local authority for the second half of the sample is because the number of owner occupiers and renters that would be interviewed from the first half of the sample had to be forecast. In some local authorities considerably more or less social survey interviews with those who rent were achieved than forecast.
6.4 Weighting
Weighting for the SHCS is done in two stages: a selection weighting stage to address the unequal selection probabilities followed by calibration weighting to correct for non-response bias.
6.4.1 Selection Weighting
Selection weights are inverse to the probability of being selected to participate in a survey. The SHCS sample is usually stratified by local authority, with smaller local authorities and those with historically lower response rates having higher sampling rates. The usual SHCS selection probability is the number of households in the sample divided by the NRS household estimate.
However, this approach had to be adapted for the 2021 external+ SHCS. This is because the selection probability depended on the local authority and household tenure, the latter obtained as part of the social survey interview.
To determine the SHCS selection probabilities we therefore needed to know the composition of the 2021 SHS sample by local authority and tenure and then compare this to estimates of the housing stock by local authority and tenure.
The tenure of households in the SHS sample is unknown prior to completion of the social survey interview. This information is collected as part of the social survey interview, and we do not have household-level administrative data on tenure. Therefore, this was estimated using estimates of the proportion of households by tenure for each local authority from the 2019 Scottish Survey Core Questions (SSCQ).
To estimate the SHCS ‘subsample’, the proportions in Table 6.2 were applied to our estimate of the SHS sample by local authority and tenure. This effectively provides an estimate of the number of households by local authority and tenure that would have been asked to participate in the physical survey had all households in the SHS sample participated, which we need to know to calculate the SHCS selection probabilities.
Finally, estimates of the housing stock by local authority and tenure were produced by apportioning NRS’ 2021 household estimates by local authority using estimates of the proportion of households by tenure for each local authority from the 2019 SSCQ.
Table 6.3 shows the estimated selection probabilities for the 2021 external+ SHCS by local authority and tenure. For owner occupiers these ranged from 0.5% in City of Edinburgh and Highland to 7.1% in Na h-Eileanan Siar. For renters these ranged from 0.9% in Renfrewshire to 14.3% in Na h-Eileanan Siar.
Though the SHS sample was drawn in two halves, with telephone matching used for the first half of the sample but not the second, results for 2021 are being presented by combining the two halves of the sample. Therefore, the selection probabilities have been averaged across the two halves of the sample.
6.4.2 Calibration Weighting
Calibration weighting corrects for non-response bias in surveys by weighting the achieved sample so that it is consistent with known external totals.
For the 2021 external+ SHCS, two calibration models were considered. These are summarised in Table 6.4. These models use the same approach to selection weighting described above. Model 1 uses the ‘usual’ SHCS calibration model and model 2 adds two additional calibration characteristics to this model, namely household tenure and Scottish Index of Multiple Deprivation (SIMD).
As described in section 6.5, calibration model 2 was found to resolve some of the issues with the representativeness of the sample better than calibration model 1. Therefore, the results from the 2021 SHCS presented in this report are weighted using calibration model 2.
The usual calibration weighting for the SHCS adjusts the selection weights so that the weighted achieved sample is consistent with
- The number of households in each local authority
- Dwelling age at Scotland level
- Dwelling type at Scotland level
- Urban-rural classification at Scotland level
NRS’ household estimates are the source for the first of these and the others are sourced from the SHCS sample itself (the dwelling descriptions and abbreviated dwelling descriptions).
For almost all households in the SHCS sample, a dwelling description or abbreviated dwelling description is undertaken, even when a social survey interview or physical survey was not completed. These capture data on the dwelling age and type. Having weighted the sample to account for unequal selection probabilities, these then provide estimates of the composition of the housing stock by dwelling age and type from a large sample with minimal non-response bias and are used to set calibration totals for dwelling age and type. A similar process is used to set calibration totals for urban-rural classification, which can be determined from the address information in the sample file. For further details see the SHS 2019 methodology and fieldwork outcomes report.
However, to minimise surveyor travel, dwelling descriptions and abbreviated dwelling descriptions were not undertaken as part of the 2021 external+ SHCS. This means that the calibration totals for dwelling age and type need to be rolled forward from the 2019 SHCS.
Figure 6.1: Percentage of dwellings by age, 2012 to 2019
Data Source: Table DQ4 in 'External+ Data Quality' tables.
Figure 6.2: Percentage of dwellings by type, 2012 to 2019
Data Source: Table DQ5 in 'External+ Data Quality' tables.
Figure 6.1 shows how the composition of the housing stock has changed by dwelling age from 2012 to 2019. Figure 6.2 shows the same but for dwelling type. The composition of the housing stock by dwelling type has been stable between 2012 and 2019. However, there has been a five-percentage point increase in the proportion of post 2002 dwellings from 7% in 2012 to 12% in 2019, reflecting new build housing.
Though new house building was impacted by the coronavirus pandemic, National Statistics for 2020-21 show that 14,798 new build houses were completed in Scotland in 2020-21. Therefore, it is reasonable to expect that the proportion of post 2002 dwellings in the housing stock will have continued to increase between 2019 and 2021. As the calibration totals for dwelling age are being rolled forward from the 2019 SHCS, the results from the 2021 external+ SHCS will underestimate the proportion of post 2002 dwellings. Though not ideal, given the lack of alternative data we have judged this to be satisfactory for making the results from the 2021 external+ SHCS as representative of the housing stock as possible. But users should be mindful of this when interpreting results.
Figure 6.3: Percentage of dwellings by tenure
Data Source: Table DQ6 in 'External+ Data Quality' tables.
Figure 6.3 shows estimates of the percentage of dwellings by tenure from various sources. The Scottish Government’s statistics on housing stock by tenure are the most robust estimates.
The unweighted estimates from the 2021 external+ SHCS are closer to the stock by tenure estimates than the unweighted estimates from the 2021 SHS. (Recall that the push to telephone/video approach used for the 2021 SHS is known to result in owner occupiers being overrepresented relative to renters in the achieved sample.) However, this is mainly due to the approach to sampling that was taken for the 2021 external+ SHCS which meant that renters were more likely to be asked to participate than owner occupiers. Having accounted for the different selection probabilities, weighted estimates from the 2021 external+ SHCS using the ‘usual’ calibration model (model 1) are then closer to the unweighted estimates from the 2021 SHS and further from the stock by tenure estimates. Therefore, we considered a second calibration model which included household tenure.
For consistency with the approach to weighting taken for the social survey and due to the under-representation of households from the most deprived areas in the achieved sample for the 2021 external+ SHCS, we also included SIMD in this calibration model.
Again, for consistency with the approach to weighting taken for the social survey, the calibration totals for household tenure were derived by applying estimates from the 2019 SSCQ to NRS household estimates for 2021.
6.5 Impact on Data Quality
The impact of the changes to the 2021 SHCS on data quality are driven by the enforced changes to the social survey (SHS) which have impacted on the profile and representativeness of the achieved sample, and the enforced changes to the physical survey (SHCS) which have introduced mode effects. Key estimates from the SHCS, e.g., on fuel poverty and energy efficiency, have likely been impacted by a combination of both.
In this section we present estimates from the 2021 SHCS based on calibration models 1 and 2 and compare these with estimates from the 2019 SHCS. These comparisons are provided only to illustrate the impact the enforced changes to the 2021 social and physical surveys have had on data quality. As noted previously, the results from the 2021 SHCS are not directly comparable with the National Statistics from previous waves of the survey.
The estimates from the 2021 SHCS based on calibration model 2, which includes calibration totals for household tenure and deprivation, are found to resolve some of the issues with the representativeness of the sample and are generally more in line (but still not comparable) with those from the 2019 survey. Therefore, all results from the 2021 SHCS presented elsewhere in this report are weighted based on calibration model 2.
6.5.1 Profile and representativeness of the sample due to changes to the social survey
Figure 6.4 shows the composition of the weighted achieved SHCS sample by household tenure, household type and annual household income for 2019 and 2021. Estimates for 2021 are provided for calibration models 1 and 2.
It has been established that owner occupiers were overrepresented relative to renters in the achieved sample for the 2021 social survey. Figure 6.4 shows that if household tenure is not included in the calibration (model 1) then owner occupiers are also overrepresented relative to renters in the achieved sample for the 2021 SHCS. This is despite the 2021 SHCS over sampling renters - which then requires renters to be weighted down at the selection weighting stage.
Including totals for the number of owner occupied and rented households in the calibration (model 2) enforces a representative balance between these in the sample. However, within these tenures there are still some large differences compared to 2019. For example, those that own outright (38%) have increased by three percentage points compared to 2019 (35%).
Family households were underrepresented in the achieved sample for the 2021 SHS and older and other households were overrepresented. Figure 6.4 shows that this impacts on the SHCS subsample. Including household tenure and SIMD in the calibration (model 2) helps somewhat in addressing this, with the estimate for older households (35%) similar to that from 2019 (33%). However, families (18%) remain underrepresented compared to 2019 (24%) and other households (48%) remain overrepresented compared to 2019 (44%).
Similarly higher income households were overrepresented in the achieved sample for the 2021 SHS and Figure 6.4 shows that this is reflected in the 2021 SHCS. Again, this is addressed somewhat but not completely by including household tenure and SIMD in the calibration (model 2). For example, this brings the proportion of households with an income of £45,000 or more per year to 23% (compared to 24% for model 1) which is closer to the corresponding estimate for 2019 (21%). However, higher income households remain overrepresented and low-income households remain underrepresented compared to 2019.
Figure 6.4: Selected household characteristics, 2019 and 2021 using calibration models 1 and 2
Data Source: Table DQ7 in 'External+ Data Quality' tables.
Figure 6.5 shows the composition of the achieved sample for the 2021 SHCS by selected dwelling characteristics for 2019 and 2021. Estimates for 2021 are provided for calibration models 1 and 2.
Dwellings in the most deprived areas were underrepresented in the 2021 SHS. Figure 6.5 shows that if SIMD is not included in the calibration (model 1) then dwellings in the 15% most deprived areas are also underrepresented in the 2021 SHCS. This is resolved by including SIMD in the calibration (model 2).
Dwellings with prepayment meters and the distribution of dwellings by Council Tax band are also better represented relative to the 2019 baseline by including household tenure and SIMD in the calibration (model 2). Though even under model 2, the estimate of dwellings with a prepayment meter in 2021 (14%) represents a decrease compared with 2019 (17%).
This is unlikely to represent a genuine change and is likely due to the over representation of higher income households in the 2021 SHS and SHCS. These households are less likely to have prepayment meters. The Department for Energy Security and Net Zero publish quarterly estimates of the regional variation in gas and electricity customer numbers by payment type. These estimates show that the proportion of customers in Scotland with prepayment standard electricity, economy 7 electricity and gas meters in 2021 is similar to 2019.
Figure 6.5: Selected dwelling characteristics, 2019 and 2021 using calibration models 1 and 2
Data Source: Table DQ8 in 'External+ Data Quality' tables.
6.5.2 Mode effects due to changes to the physical survey
Figure 6.6 shows the proportion of dwellings for which all fixed lighting is low-energy lighting, the proportion of dwellings with a secondary heating system and the distribution of dwellings by total internal floor area for 2019 and 2021. Estimates for 2021 are provided for calibration models 1 and 2. It is apparent that neither calibration model brings the estimates noticeably more into line with the 2019 baselines.
The proportion of dwellings for which all fixed lighting is low-energy lighting has increased by 17 percentage points to 41% in 2021 compared to 2019 (for calibration models 1 and 2). This is unlikely to represent a genuine change and is likely due to the mode of collection used for the 2021 external+ SHCS. This information was required to undertake the energy modelling and had to be collected. However, this is not something that surveyors would have been able to observe for themselves as they were not able to enter dwellings. In most cases surveyors would have been reliant on the householder providing this information via telephone. It is likely that householders have tended to overestimate the proportion of fixed lighting that is low-energy. BRE who undertook the energy modelling for the 2021 external+ SHCS reported a decrease in modelled energy consumption and costs associated with lighting, in part due to the increase in the number of dwellings for which all fixed lighting is low-energy lighting. This then impacts on energy efficiency ratings (and will also impact on estimates of fuel poverty annual running costs).
BRE have advised that an average-sized dwelling (total internal floor area of 89 meters squared) going from 0% to 100% low-energy (fixed) lighting would save around £50 per year which could add around 1 to 2.5 points to the SAP score, dependent on the initial SAP score.
A similar issue was reported in relation to a decrease in the proportion of dwellings with a secondary heating system (3% of dwellings in 2019 compared to 2% of dwellings in 2021) and a subsequent decrease in modelled energy consumption and costs associated with secondary space heating. Again, surveyors would have been reliant on the householder providing this information via telephone and it is likely that householders have tended to under report the presence of secondary heating systems.
Mode effects are also likely to have impacted total internal floor area calculations, which are a key component of the energy modelling. BRE reported that there were a few external+ surveys for which dwelling measurements were missing and had to be obtained from other sources, e.g., Google Maps and Rightmove. This is likely because surveyors had to record all dwelling measurements externally and this may have made it difficult to obtain measurements for non-standard dwellings.
Figure 6.6: Selected dwelling characteristics, 2019 and 2021 using calibration models 1 and 2
Data Source: Table DQ8 in 'External+ Data Quality' tables.
6.5.3 Impact on Key Statistics
Figure 6.7 shows selected key statistics from the SHCS for 2019 and 2021. Estimates for 2021 are provided for calibration models 1 and 2.
For calibration models 1 and 2 there is a 7-percentage point increase in the proportion of dwellings with an EPC rating of band C or above to 52% in 2021 compared to 45% in 2019. This corresponds to an increase of around 190,000 households. This is unlikely to represent the true scale of any improvement in the energy efficiency of dwellings between 2019 and 2021. This increase is most likely due to the mode effects associated with the 2021 external+ SHCS (particularly around the questions on the proportion of fixed low-energy lighting and secondary heating systems) and the subsequent impact on the outputs from the energy modelling (i.e., energy efficiency ratings) as well as the issues with the representativeness of the sample.
It should be noted that of the 1.02 million dwellings rated EPC band D in the 2019 SHCS, around 120,000 (5% of all dwellings) were one SAP point short of being rated EPC band C. This illustrates that any mode effects which result in even a small increase in energy efficiency ratings could have a big impact of the distribution of dwellings by EPC band.
In addition, it is likely that the non-response bias in the 2021 SHS and SHCS samples will have contributed. However, the effects of this are likely to work in opposite directions. For example, social renters and families are underrepresented and they tend to live in the most energy efficient dwellings. Higher income households are overrepresented and they tend to live in the most energy efficient dwellings.
Similarly, for both calibration models there are decreases in the proportion of households in fuel poverty and extreme fuel poverty compared to 2019, though the estimates from calibration model 2 are more in line with the 2019 baseline. The fuel poverty and extreme fuel poverty estimates for 2021 based on calibration model 2 represent decreases of 5 and 3 percentage points compared to 2019, respectively[18].
Fuel poverty has three main drivers: high energy prices, low income, and poor energy efficiency. We have established that higher income households have been overrepresented in social survey and that the mode effects due to the changes to the physical survey are increasing the modelled energy efficiency ratings for dwellings. We would expect these to drive fuel poverty rates down. According to the domestic energy price indices published by the Department for Energy Security and Net Zero, there was a modest decrease of 0.1% in the price of domestic fuels between 2019 and 2021. Within this there was a 9.8% decrease in the price of gas and a 6.8% increase in the price of electricity. Overall, we would expect energy prices to have had a neutral effect on fuel poverty rates between 2019 and 2021.
So clearly the decreases in the fuel poverty and extreme fuel poverty rates are not likely and are most likely due the over representation of higher income households in the social survey and the mode effects due to the changes to the physical survey that are driving the increases in energy efficiency ratings.
Figure 6.7 shows that there has been a 3-percentage point decrease in dwellings with urgent disrepair to critical elements in 2021 (calibration model 2) compared to 2019. This is unlikely to be a result of mode effects due to the changes to the physical survey. The urgency of disrepair is only assessed for external and common elements which would normally be assessed by surveyors from outside the dwelling. While a genuine improvement cannot be ruled out, it is more likely this is due to issues with the representativeness of the sample.
Figure 6.7 shows that there has been a decrease in the percentage of dwellings that meet (but do not exceed) the minimum requirements of the bedroom standard from 29% in 2019 to 25% in 2021 under calibration model 2 and 22% in 2021 under calibration model 1. Furthermore, under calibration models 1 and 2 there have been 5 and 3 percentage point increases, respectively, in the proportion of dwellings that exceed the bedroom standard by two or more bedrooms. It is most likely that these changes are due to the representativeness of the social survey. For example, we know that higher income households which are overrepresented in the social survey are more likely to live in accommodation which exceeds the bedroom standard by more than two bedrooms.
Figure 6.7: Selected key statistics, 2019 and 2021 using calibration models 1 and 2
Data Source: Table DQ9 in 'External+ Data Quality' tables.
Footnotes
[17]There is no systematic way to determine when data from Energy Performance Certificates was used. This is based on anecdotal feedback from regional managers and surveyors.
[18] It should be noted that the fuel poverty estimates for 2019 and 2021 presented in this section are not directly comparable. As explained in the fuel poverty section of this report, we have incorporated methodological changes into our fuel poverty estimates for 2021 which are not reflected in the estimates for 2019. Had fuel poverty estimates for 2021 been produced on the same basis as the estimates for 2019, under calibration model 2 the fuel poverty rate would have been 21% and the extreme fuel poverty rate would have been 10%. (Under calibration model 1 the fuel poverty rate would have been 19% and the extreme fuel poverty rate would have been 9%.) Clearly the methodological changes incorporated into our fuel poverty estimates for 2021 do not alone account for the differences between 2019 and 2021.
Contact
Email: shcs@gov.scot
There is a problem
Thanks for your feedback