Energy Efficient Scotland Transition Programme Survey Evaluation

Methods

Data Collection and Analysis

Time and resource constraints resulted in a survey with eight local authorities participating in the Transition Programme, selected by the University of Edinburgh research team. The selection aimed to represent the variety of pilots implemented across Scotland, including the range of: engagement strategies (e.g. drop-in centre; mail outs; telephone advice line); delivery partners (e.g. Changeworks; Energy Agency; Scarf), and locations (e.g. urban, rural), as highlighted in Table 1.

The survey was designed by the University of Edinburgh research team in collaboration with local authorities. It comprised three sections, with closed, multiple response, and open-ended questions (a copy of the survey is in Appendix 2).

The surveys investigated energy efficiency engagement in two sectors –domestic (i.e. the household survey) and non-domestic (this survey was particularly targeted at small to medium enterprises). Questions were designed to establish the extent to which the engagement strategies affected domestic and non-domestic sector engagement with, and investment in, energy efficiency measures, and whether some strategies were more effective than others in stimulating action. The domestic survey was also designed to investigate any significant variations or correlations in answers based on age group, tenure type and income band.

The non-domestic survey response rate was very low (as reflected in comments on research limitations), and hence this report is based on household respondents only.

In total, the survey had 23 questions investigating four themes:

• Demographic data and property characteristics
• Householders' knowledge of energy efficiency measures;
• Householder engagement with energy efficiency within the property and sources of finance used before and after engagement with the programme;
• Opinions on local and national government engagement in domestic energy efficiency improvements at present and in the future.

Local authorities and/ or their delivery partners distributed the surveys to households that had been engaged at least once and had consented to taking part in the research. This includes householders who had approached, or been approached by, a local authority as part of engagement strategy.

Local authorities and/ or their delivery partners distributed the surveys by email and/or post, depending on participant contact preferences. The mode and time of survey distribution therefore varied. Participants were asked to respond either online, via Qualtrics (a specialist survey software), or by returning a completed paper survey, using a pre-paid envelope. For the latter, data were entered manually into Qualtrics by the University of Edinburgh research team. A cover letter was included with the survey, and reminder emails and/or postcards were sent to each household. Incentives were provided to encourage participation (e.g. via a prize draw to win a £50 or £100 shopping voucher); these were determined by the local authority and delivery partner teams.

Data were anonymised for privacy and confidentiality. The key descriptive statistics and themes were analysed, and triangulated with data from previous Energy Efficient Scotland pilots, for example, interviews with local authority representatives conducted in Phase 1, Phase 2, and the Transition Programme. The survey data are presented and visualised using Excel.

In most cases, the data are presented for the local authority as a whole and then analysed according to engagement strategy. In some cases, data are compared across the eight local authorities. For consistency, the presentation of local authorities remains largely the same throughout the report – the sequence has been generated randomly (Table 1). The order for engagement strategy is sorted from largest to smallest number of responses to examine the effectiveness of the strategy. As with all research there were some limitations; these are described next.

Research Limitations

This evaluation focuses on understanding households' experiences and perceptions of the engagement strategies trialled in the Transition Programme. It does not explore the views or experiences of the local authorities or delivery partners running the engagement activities, nor does it assess the costs or relative value for money of different engagement activities.

The eight local authorities involved in this survey evaluation are not a representative sample of all the local authorities, which engaged in the Transition Programme, due to time and resource constraints. However, this selection does try to capture the variety of pilots implemented across Scotland.

Sending surveys to all households in local authorities was out-with the scope of this report in terms of time, cost and staff resources; therefore, a sampling technique was used. The surveys were restricted to householders that had already engaged with the local authority and had consented to take part in research. This sampling technique excluded households which had not engaged at all, or had chosen not to be contacted. Caution is used in interpreting the results, as this is not a representative sample of all local authority areas participating in the Transition Programme (as detailed in Appendix 1). In addition, due to time and resource constraints, there was no matched control group of households not engaged.

Although every effort was used to encourage participation, for example, by providing financial incentives and survey completion reminders, the response rate across the non-domestic and domestic sector varied. The response rate for the non-domestic sector was too low for quantitative analysis (7%) and responses are not included in the report. The response rate for the domestic survey was 14% (n=490) and therefore provided a reasonable dataset for quantitative analysis and descriptive statistics. This can aid understanding of household responses to engagement, but we have avoided over-generalising from the results, because of relatively low response rates.

The pilots in each area were tailored to local priorities; therefore, the survey was not comparing the same strategy across local authorities. There were for example different delivery partners conducting various forms of engagement in different areas, as represented in Table 2. For instance, both Fife (Changeworks) and East Ayrshire (Energy Agency) held events as part of their strategies. Different delivery partners and event content, in different places and times, make them distinct. Hence, caution is needed when making comparisons between local authorities.

In addition, engagement strategies were not used in isolation, but often in combination. Therefore, indications of the relative effectiveness of different methods should be treated with caution.

Every effort was made to ensure the survey was clear, concise and flexible, however some results were ambiguous. Some 'data cleaning' was conducted (consistent with social survey methodologies), which involved removing duplicates and errors and re-formatting data for analysis.

The survey was designed for respondents to provide multiple answers to some questions; this allowed greater flexibility than 'single-option' answers. For questions which enabled multiple selections, total responses will not equal 100 per cent. Furthermore, percentages reported can add up to more or less than 100 per cent due to rounding. These instances have been noted throughout the report.