TEC programme data review and evaluation: options study

6.0 Conclusions

The TEC programme has achieved much in the three years of work that have been considered as part of this evaluation. Although building on previous work, especially in telecare and HMHM, the programme includes a sophisticated set of complementary and well-developed technologies that are beginning to change the way in which health and social care are being delivered in Scotland.

The purpose of this evaluation was to draw together and synthesise the existing research produced under the auspices of the TEC programme. From the outset, we adopted a flexible approach to respond to any unexpected findings. During the research our analysis identified implementation as an area that merited further exploration and an additional module of work was introduced. For some of the workstreams, evaluations have served their purpose of developing an evidence base to promote adoption, and the next phase of work will focus more on the barriers and enablers to implementation and the issues with achieving change at scale. Due to the large network effects of the interventions, achieving change at scale is important because the value of the technologies increases with the number of users.

In this final section, we reflect on the research questions we presented at the outset and what we can conclude for each.

1. What is the range of the existing evidence and data on the TEC programme? What is the quality of that evidence and where are the key gaps for both range and quality?

The TEC programme has produced a sizeable number of good quality evaluations, which are varied in terms of methodologies, geographies, stakeholders included, and issues considered. There are some clear patterns. The largest and most detailed studies exist within the workstreams that are at the furthest stage of implementation ( i.e. HMHM and telecare). The focus in both video conferencing and digital platforms has been on developing, testing and deploying of technology and infrastructure to support those workstreams, and there are consequently fewer, smaller-scale evaluations.

Although the evidence base is strong for many outcomes, there are some gaps. Studies have tended to focus on one stakeholder group and there are gaps relating to groups like carers, or some conditions. Other gaps that have been identified are outcomes for the workforce, quality of life impacts for some groups, potential negative impacts such as health inequality and implementation. A series of workstream-specific recommendations have also been made.

Almost all of the evidence generated relates to the workstreams, rather than the programme as a whole. However, our study has tentatively explored whether there were programmatic impacts, above and beyond the individual workstreams. We concluded, albeit based on small-scale evidence, that it is unlikely that the interventions could exist without the programme support. Indeed, a first step in technology-enabled care implementation is the development of a structured, funded programme of work.

2. What do we know about the barriers and enablers to effective implementation of TEC?

There is a considerable amount of information on barriers and enablers to implementation although more in relation to HMHM and telecare than the other workstreams. We also found despite diversity and complexity, that there was much similarity across the workstreams, suggesting that this is an issue that could be successfully tackled programmatically.

Across all areas, the most commonly identified barrier was staff resistance. Other noteworthy barriers include digital skills, infrastructure, lack of specialist skills, lack of interest from senior managers and a lack of interoperability. A further barrier that emerged from the workshop discussion was the lack of appropriate data. Staff require distinct kinds of data for different audiences and spoke of the value of data on effectiveness. However, concerns were also raised about the quality of monitoring data and the weakness in some existing data collection systems was raised as an additional barrier.

3. Once TEC is implemented, what do we know about how sustainable its use is over time?

For telecare, data is available on sustainability from the 2006 development programme. However, this was more limited in other areas. This is to be expected from a relatively new intervention and programme, where longitudinal data are limited. It is an area that we recommend the programme focus on for future research, and we have identified sustainability of use and sustainability of outcome as two discreet evaluation considerations.

4. What is the state of knowledge on cost effectiveness? How is cost-effectiveness taken into consideration when scaling-up an approach – what is the standard of evidence used? How is cost effectiveness compared against clinical effectiveness and personal outcomes?

There is a reasonable amount of evidence that TEC interventions are (or have the potential) to be cost effective. It is clear that telecare and HMHM can lead to reductions in demand for hospital beds and care homes/care at home, which have direct economic values. However, as discussed earlier, the value of these is small at the margin and the unit costs of bed days may overstate the actual savings generated. Nonetheless, the potential value has been demonstrated and should be realised at scale. In terms of methodological rigour, the studies are broadly of good quality. However, they are narrow in scope and tend not to compare intervention costs with personal or clinical effectiveness. A series of recommendations to improve the consistency of studies has been made.

5. What are the longer-term monitoring and evaluation requirements to support the on-going development and delivery of TEC? Are there standardised metrics that can be recommended for some grantees to report against that would support the development of the evidence base?

The report has developed a measurement framework and set of evaluation principles based on the RRRP approach. This is combined with a set of workstream specific recommendations and recommendations for implementation and economic research. Increasingly, the programme should seek to incorporate economic and implementation considerations into evaluation. There is also scope for standardisation across all areas of research. This could be achieved by the development of bespoke guidance tools that could be hosted on a microsite and managed internally by staff with specialist evaluation skills.

Finally, the programme should seek to build on the successful approaches it has already adopted, not least an emphasis on the importance of evaluation, which not only make the policy case for the social value of TEC but also supports frontline staff – a key constituency in scaling up – to promote the benefits locally.

As the programme moves into the next phase, there is a clear opportunity to use the latest thinking on TEC evaluation and implementation to improve take-up in Scotland and solidify international leadership in this area.