HES Homecare pilot: evaluation

4. Results

4.1. Delivering the service

Establishing partnerships

A critical part of the Energycarers' role was to establish partnerships with community and healthcare organisations in the region, with a view to those organisations then being able to identify and refer people to the HES Homecare service. These organisations included NHS, social services, and local housing associations. The relationship between the Health & Social Care providers in Moray took time to develop; the HES Homecare team reported that this delayed referrals. Well connected individuals can be important for establishing and maintaining partnerships:

"Early on in the process we met somebody fantastic in Dumfries and Galloway who is very well integrated with lots of different organisations in the health and social care sphere was able to facilitate access for us to communication channels for them…. She was able to put HES Homecare information on the front page of their internal staff intranet…. She was able to offer slots in newsletters. And - this is quite crucial - she was able to offer us a speaking slot at a health and social care locality celebration in the first few months of the project, and then again towards the end of the first year"

By including the HES Homecare pilot on the front page of the NHS internal staff intranet, in newsletters, and at events, this individual helped to provide a constant reminder of the service, which resulted in a good number of referrals.

The following elements were found to be most useful for communications:

• Elevator pitch - to explain what the project offers briefly.
• Image/s for use in advertorial, referral tools, presentations.
• Project summary newsletter content and image.
• Direct mail letter from third party endorsing the project (e.g. Council, Care and Repair).
• Email from third party endorsing the project (e.g. Council, Care and Repair).
• Text addressed to potential referrers.
• Text addressed to potential service users.
• Text addressed to relatives/carers of potential service users.
• Map of area/s covered, showing towns and villages included.
• Postcode list.
• Poster for display in GP practices, community venues, advice agencies.

Despite this, the majority of referrals received still came through Home Energy Scotland's existing database of clients, for example, those who had previously contacted the service, but not been eligible for or able to pursue support at that time. At interview, the HES Homecare team noted that the limited timeframe of the pilot that was then extended may have led to dwindling numbers of referrals later in the pilot delivery:

"we were still telling them in March 2018 that it's running until March 2018. So I'm not sure whether we ran out of momentum because of, having said that, whether there might be people thinking, 'Oh, that scheme doesn't exist anymore.' I'm not sure that we've been able to get back to all of them to let them know, 'we're still here, you can still make referrals.'"

Thus, it could be difficult to maintain partnerships with the variety of organisations that could potentially help in making referrals. At interview, the HES Homecare team reported that it could be challenging to maintain these partnerships whilst also managing customer journeys. In particular, initial contact with partners could yield a high number of referrals, but these would dwindle over time. The HES Homecare team reminded partners about the service, but regular, repeated contact was difficult to maintain and this could be time-consuming work:

"the challenging part of trying to keep up with all your different partners with all the different casework going on while then trying to go and speak to the partners again for a second time…. there's only been a few different people that there's been a continuous stream of speaking to them"

The interviews highlighted that it was helpful for the Energycarers to be maintaining partnerships because they could provide case study details to demonstrate the value of the service. However, it could also be difficult for Energycarers to juggle these different responsibilities.

Everyday work of delivery

The HES Homecare scheme was intended to take a 'person-centred' approach, with each case receiving a specific focus. This process was summarised by a member of the HES Homecare team at interview:

"Normally when offered the home visit they would then accept that and say, "OK, come down and see me then." I would then on the first visit speak to them about their concerns and what their feelings about the house [are], you know, because… I mean, I'm fairly qualified to walk into a house and see that there's a boiler on the wall… But these people actually live there and have probably lived there for a decade or so. So to actually have a conversation with them about the house is always useful. And then they tell you what they would see as the problems and then you can try and see what they see as the problems, maybe something you'd noted as being something that you could help out with and then you can put them on a journey that resolves both their problems and maybe something they've observed as well."

Thus, technical knowledge sat alongside social understandings in the delivery of HES Homecare's 'person-centred approach'. This could mean that client interactions could be time intense. For example, Case Study 3 details that 10 visits were made to the household. Case Study 5 provides an indication of the challenges of navigating different routes to support for clients, particularly where only partial funding is made available for the remedial works required for the property. This meant that the Energycarer role needed to be filled by individuals with some technical expertise in terms of retrofitting and energy consumption, and a personable approach to understanding individual's experiences. Factoring in the time spent with different clients, the Energy Saving Trust have estimated that each Energycarer could manage approximately 50 cases per year using this model, depending on the needs of clients.

4.2. Changes to internal temperature

Tinytag data from the three properties that were analysed did not show any significant difference in internal temperature over the period studied. Table 6 shows the results from the internal temperature analysis; it is important to note that that sample sizes that were compared are very small due to the limited amount of data available, therefore any impact derived from these results should be used with caution, as the data does not capture the full breadth of heating behaviour from the occupants nor the annual seasonal changes in temperature and therefore heating demand. Property A had a new gas boiler installed, Property B had draught proofing vents installed on its external walls and Property C has new electric storage heaters installed.

Table 6: Internal temperature analysis results.

			Internal Temperature Analysis Results
			Sample Size	T.Test	Temperature			Standard Deviation
Project	Measure	Property ID	Weeks Before and After Measure	95% Significance	Before Installation (°C)	After Installation (°C)	Change (%)	Change (%)
Homecare	Gas Boiler	A	7	No	12.8	12.7	1%	78%
Homecare	Draughtproofing	B	3	No	13.9	14.0	-1%	-66%
Homecare	Electric Storage Heaters	C	6	No	19.4	19.3	1%	133%

For these three households, the T.Test on the internal temperature before and after the measure was found not to be significant, this is because the duration of monitoring is over such a short amount of time. The percentage change indicates that there was only a 1% difference which can be assumed to be within the margins of error. The standard deviation in internal temperature (otherwise known as the change in variability of internal temperature) shows a very large change both increasing in variability in properties A and C; and a large reduction in variability in property B. However, given the sample size is very small, is over different parts of the same heating season, and does not include a whole heating season before and after installation of the measure, the change in variation is more likely to be due to normal variation in behaviour which has been over represented due to the short monitoring duration.

However, where collected beyond these three properties, the Tinytag data has been helpful for building a picture of what happens in particular properties following the provision of advice and technical intervention. Although not statistically significant or comparative of replacement heating systems, this information can help to understand the impacts of the HES Homecare service on an individual basis. For example, the data included in Case Study 1 indicates a change in heating pattern following Energycarer advice to use the programmer and alter the temperature when the residents were away from the home. Tinytag data was also used to supplement Case Study 2. In this case, the occupant did not have a working central heating system and they were not able to heat four rooms of their property. The temperature data indicates that between February and April the living room temperature rarely exceeded 10C and that the lowest temperature was around 2C. These extremely low internal temperatures were corroborated by the Energycarers at interview. They reported their own thermal discomfort at being in those spaces (for example, noting that it was "Absolutely Baltic" in one resident's home); residents sleeping in the main living space because it was the only room heated; being dressed in many layers; and using duvets in living spaces to keep warm. The Energycarers also noted that, in some cases, they were able to notice a physical difference in the temperature of the property.

During interview, the HES Homecare team reported that some Tinytags are still in place and they have permissions to collect data for another winter. Consequently, there may be some longer-term data available for analysis in due course.

4.3. Changes in comfort and occupant satisfaction

The following analysis focuses on the data collected through the social surveys to assess changes in comfort and occupant satisfaction through the HES Homecare trial. This analysis seeks to gauge (a) the extent to which significant change had occurred between Time 1 and Time 2, that is, pre-and post-intervention; and (b) whether changes were greater among the experimental compared with the control group.

Thermal comfort during the winter months

Figure 1 illustrates the change in respondents perceived thermal comfort during the winter months at Time 1 and Time 2 of the survey. Generally, it shows that those receiving both the HES Homecare and standard CLO services perceive themselves to be cool or much too cool in their homes. Figure 1 also indicates some shift towards feeling warmer, with some respondents answering that they are 'comfortably warm' at Time 2. For the participants in Dumfries & Galloway between the Time 1 and Time 2 surveys, four showed no change in overall thermal comfort level, two improved substantially and one showed a minor improvement. Of those in the Moray area, three showed no change, two showed a minor improvement and one showed a deterioration between Time 1 and Time 2. Amongst those receiving the standard CLO service, two showed no change, two showed a deterioration and one showed a minor improvement. With such small numbers and mixed results, it is not possible from this data to identify whether the HES Homecare service yields a greater improvement in thermal comfort than the standard CLO service. Indeed, half of respondents indicated no change in thermal comfort had taken place, this includes those in the HES Homecare trial groups.

Figure 1: Change in thermal comfort during winter months between Time 1 (blue bars) and Time 2 (orange bars) survey. Case 1-14 (solid bars) received the HES Homecare service; cases 15-18 (hatched bars) received the standard CLO service.

Questions asking about behavioural change, for example: 'Thinking back to last winter, how frequently did you do the following to prevent yourself being too cold at home?', showed some modest changes in peoples' actions. For example, in the Dumfries and Galloway group, some respondents answered that they use extra clothing and outdoor clothing less at Time 2 than Time 1, suggesting higher levels of thermal comfort. The Moray group also showed modest improvements (for example, instead of wearing additional clothes indoors 'very often' at Time 1 they did this 'often' at Time 2). However, in both cases there was no overall change and no statistical evidence to support these being patterns across the sample.

Home satisfaction and house conditions

Figure 2 presents a summary of responses about general satisfaction with the home. It shows some shift towards greater levels of satisfaction for recipients of both the HES Homecare and standard CLO services. For the question: On the whole, how satisfied or dissatisfied are you with your home at the moment?, the group in Dumfries and Galloway had a mean score of +0.86 (approximating to 'fairly satisfied') at Time 1, and +1.43 at Time 2 (between 'fairly' and 'very satisfied'). This is an increase of +0.57 through the course of the trial. Surveys from Moray householders showed low levels of home satisfaction at Time 1 with an average of -1.5 (between 'fairly' and 'very' dissatisfied), which increased to +0.33 at Time 2 (between 'no opinion' and 'fairly satisfied'). This is an increase of +1.8 in the mean score over time. For those receiving the standard CLO services, the scores cluster around 'no opinion' (0) to 'fairly satisfied' (+1) at both Time 1 (mean of +0.8) and Time 2 (mean of +1.0), and little evidence of change over time. However, two of these respondents are not same people at Time 1 and Time 2, making it difficult to draw firm comparisons.

Figure 2: Change in general satisfaction with home between Time 1 (blue bars) and Time 2 (orange bars) survey. Case 1-14 (solid bars) received the HES Homecare service; cases 15-18 (hatched bars) received the standard CLO service.

With respect to house conditions, few survey respondents recorded instances of damp walls, mould, draughts or condensation. Where these conditions were reported, some respondents noted a marginal improvement in levels of draught (reporting that 'more than half' of rooms were draughty at Time 1, and 'less than half' were draughty at Time 2).

Health and wellbeing

Figure 3 summaries respondents' perceptions of their health and wellbeing before and after any interventions from the HES Homecare and CLO services, for the 18 households that completed both T1 and T2 surveys. For the HES Homecare recipients in Dumfries and Galloway, the Time 1 mean score was -0.86 (marginally less than 'fair'), this rose modestly to +1.0 ('fair') at Time 2, with three respondents indicating no change; two recorded a slight deterioration, and two a modest improvement between T1 and T2. No-one scores their health and well-being 'excellent' and only one person 'very good'. For those receiving the HES Homecare service in Moray, at Time 1, the mean score was -1.33 ('Fair' to 'Poor'), improving to -0.83 at Time 2 (just about 'Fair'). Four respondents record 'no change' over time, and two a slight improvement in health and wellbeing (e.g. 'poor' to 'fair'). For the recipients of the standard CLO service, two cases have been discounted on the grounds that respondents were different members of household at T1 and T2, and hence responses are not valid. There is no change over the time period in the remaining three cases (recorded as Poor, Fair and Very Good).

Figure 3: Change in perceived health and wellbeing between Time 1 (blue bars) and Time 2 (orange bars) survey. Case 1-14 (solid bars) received the HES Homecare service; cases 15-18 (hatched bars) received the standard CLO service.

Together, these results suggest that, while there have been modest reported improvements over time with regard to thermal comfort, behavioural change, house conditions, home satisfaction and health and well-being, there is insufficient variation, notably between the experimental groups and the control group, to conclude that the intervention has made a significant difference. There are not major differences between the 'experimental' groups (those receiving the HES Homecare service in Dumfries & Galloway and Moray) and the 'control' group (those receiving the standard CLO service in the Highlands/Orkney). Further, the results presented are not statistically significant and they are from a small sample which makes it inappropriate to draw firm conclusions.

4.4. Wider impacts of the service

The people in receipt of the HES Homecare service were often acutely vulnerable. Many of the recipients were older people, they were also people with health concerns that affected their mobility or ability to work. For example, the health concerns listed in the attached case studies include: a heart condition (Case Study 2), cancer (Case Study 5), and chronic obstructive pulmonary disease (COPD) (Case Study 6). In both Case Studies 3 and 6, one of the residents passed away during the course of receiving the HES Homecare service. For this vulnerable client group, HES Homecare offered benefits beyond energy saving alone.

Through HES Homecare, the residents received additional support through the processes of applying for funds and receiving interventions in the home. One of the HES Homecare team noted that "the freedom to spend time with the householder, to establish that relationship of trust and to support them every step of the way" was one of the main benefits of the scheme. For example, one of the residents in Case Study 3 struggled to keep up with the standard services available via telephone support. The repeated visits (10 in total) were particularly important for guiding this person through the installation of a gas central heating system, especially when their partner passed away. This type of service was also highlighted as particularly important for a resident with chronic fatigue who made progress in receiving additional benefits with the help of regular reminders and Energycarer support. Having a consistent and personable level of support was identified by the HES Homecare team as a critical aspect of this service. For example, one of the team members highlighted that a client "was dreading" the visit because they were concerned about how they would be treated. They noted that "the last thing" that these vulnerable residents would want was an "Energycarer sitting there with a clipboard". The Energycarer role is thus important for providing "one point of contact, people feel really supported. They know who they're speaking to. Who's going to be chasing things up for them".

In addition, the support included energy saving advice such as how to set heating programmers, which was reported by the Energycarers to have been heeded by clients. Although not all of the measures identified through the HES Homecare service (repairing broken windows, repairing roof leaks) will have a significant impact on energy consumption, they could be important for health and the occupant's quality of life. These cases highlight the importance of remedial works for these residents, but also the challenges that can arise in funding these works. In Case Study 5, the visits undertaken as part of the HES Homecare service provided an opportunity to identify myriad problems in the home that would not necessarily have been apparent through phone intervention. These experiences led one member of the HES Homecare team to suggest that existing schemes like Warmer Homes Scotland might be re-considered to support with the treatment of mould and damp, for example.

In relation to this, another wider benefit of the HES Homecare scheme was its ability to identify people who are not supported through other means. For example, some of the recipients of the service were privately renting from family members or friends. These individuals are exempt from landlord registration requirements, so the properties in their care may "slip through the gaps" for receiving home improvements. Case Study 2 provides an example of how people in this situation can subsequently be discounted for receiving additional support. Case Study 4 details the extent to which an Energycarer can help to encourage retrofitting activity in a privately rented home. Unfortunately, in this case the negotiations were eventually unsuccessful, due to additional costs which were not anticipated early in the process. However, the personal contact of the Energycarer was useful for liaising between landlord and tenant, and may be a model to carry forward in some way for wider engagement with the private rented sector.