TEC programme data review and evaluation: summary report

Findings

• There is emerging evidence for the value and impact of the TEC programme. Stakeholder engagement with implementers of technology enabled care found that the TEC programme is highly valued and that it is unlikely the technology development, deployment and adoption could have happened independently. This conclusion is supported by the secondary literature on implementing technology enabled care which highlights the need for dedicated funding, personnel and expertise to drive a programme of work such as this, especially in light of widespread barriers to adoption.
• Our data review found a strong commitment to measurement and evaluation and its role in supporting continuous improvement of the TEC programme. Consistent with international evidence, there is solid evidence for the two workstreams ( HMHM & Telecare) that are furthest in the implementation cycle and emergent evidence for the workstreams where technologies are either in development (digital platforms) or in the early stages of implementation ( VC).
• There are challenges inherent in evaluating technology enabled care interventions which mean that standard methods of conducting health evaluation, such as randomised controlled trials, are often not appropriate. We recommend the adoption of the Rapid, Relevant, Research Process ( RRRP) as a framework for future measurement and evaluation activities. The RRRP framework addresses evaluation challenges such as the need for flexibility, speed, adaptability, attention to contextual factors and the evolving nature of the interventions. A set of principles have been developed that should inform evaluations for the programme.
• There is variability in the robustness of evaluation studies, including economic evaluations. To ensure best practice is mainstreamed and to avoid duplication of effort we recommend the development of guidance and resources to support measurement and evaluation as well as the development of in-house expertise.
• Our work on implementation found that the TEC programme is already applying many established implementation strategies that are consistent with international best practice. Further support on the use of IS frameworks to inform technology-enabled care delivery could assist with mainstreaming of technologies by addressing common barriers to adoption.

Logic models and data review

The TEC programme has a strong commitment to evaluation and this is reflected in the number of evaluation studies undertaken. As a group, these studies point to a solid evidence base, especially for two of the workstreams; telecare and HMHM. In other areas, the evidence base is still ‘emergent’. However, this is to be expected, given that workstreams are at different stages of implementation and many of the outcomes are expected to take place beyond the life of the programme. In addition, some were building on existing evidence, from Scotland and elsewhere, which negated the need for fresh evaluation.

The purpose of developing the logic models was to systematically compare workstream outcomes with the existing evidence produced by the programme. A total of 95 documents were submitted to Just Economics for review. Of these 54 contained evaluation data, and these were synthesised. This section provides a summary of the findings from this work by describing each programme and its evidence base. We begin with the overarching programme.

TEC Programme findings

At the heart of the TEC programme’s strategic vision is catalysing the transformation of health and care services such that technology becomes integrated as ‘business as usual’, thereby allowing more delivery of services at home and in community settings. As well as providing a ‘home’ for each of the workstreams, it aims to add value i.e. to be ‘more than the sum of its parts’. Stakeholder engagement carried out as part of this commission with implementers of technology enabled care interventions found that the programme is highly valued and that it is unlikely the technology development, deployment and adoption could have happened independently.

The programme operates at two levels. At a national level, the programe seeks to create the conditions to support scale up and adoptions. This includes building the infrastructure, tools and approaches, and tackling the existing technological, social, organisational and cultural barriers. At a local level, it seeks to grow TEC activities and initiatives through dedicated funding to local organisations implementing or trialling TEC initiatives.

There have already been considerable successes for the programme, with an additional 66,000 clients benefiting from technology-enabled care (to February 2018), sharing of learning across settings, and progress on infrastructural issues, such as the analogue to digital switchover ^[1] . Going forward, the TEC Programme Board has identified the following four areas in its Strategic Priorities for technology enabled care 2018-21 in support of the new Digital Health and Care Strategy for Scotland:

Transforming local systems – supporting exemplars that are seeking to transform local health and social care systems using digital technology to shift local systems upstream to prevention, self-management and greater independent living

Developing approaches once for Scotland – developing approaches that have been shown to be effective, supporting scaling up across Scotland and addressing barriers that require national level action

Preparing for the future – identifying and testing new approaches that offer the potential to achieve change at scale

Building capabilities and supporting improvement – championing, supporting, gathering and promoting the evidence of what works, to develop the culture and skills that recognise and use digital TEC including through developing business cases, supporting strategic planning and delivery.

These strategic priorities aim to increase the number of people using fit-for-purpose technologies and for those technologies to either a) free up capacity within the system without any risk to care quality or b) improve health and social care outcomes (or, indeed, both simultaneously). Ultimately, the goal is to enable citizens to enjoy longer, happier and healthier lives in their own homes, thereby reducing demand on current services and freeing up capacity.

Achieving these strategic priorities occurs via several pathways. These include the use of the Attend Anywhere video conferencing system for remote clinical and care consultations and decision support, improved condition control and self-management via remote monitoring and digital platforms, more technology-enabled care options being used in care homes and better access to specialist services, especially for those with long term conditions and those experiencing health inequalities. Wider social benefits have been identified for interventions at scale. These include improving the viability of vulnerable services and communities, reducing CO2 emissions and economic benefits from the improved productivity of a healthier workforce. Ultimately, however, the goal is to enable citizens to enjoy longer, happier and healthier lives in their own homes.

A signal that these technologies are being exploited to their full potential is that technology-enabled care options become the default within decision-making in health and care teams. This is a key objective for some of the workstreams and the overall programme. This underlines the importance of the promotion and advocacy work and the continuous improvement through evaluation.

There are also some challenges. An objective of the TEC programme is to reduce health inequalities and there are various means by which the technologies can play a role within this. However, some of the technologies also require minimum skills and infrastructure to be fully exploited. Given that those who experience health inequalities are also more likely to be digitally excluded there is a risk that inequalitites will be exacerbated, as the more digitally included are better placed to access the benefits. A second challenge for technology enabled care is that many of the benefits for the health and social care system are only realised once a critical mass of users is reached. That is, whilst technology-enabled care users can personally benefit, the wider societal benefits can only be achieved at scale. Demonstrating ‘cashable’ savings is difficult in the short-term and the marginal savings from – for example bed days saved - is likely to grossly undervalue the potential economic impact of the programme.

As discussed, most of the evaluations to date have focused on the workstreams and the efficacy of the technologies they are developing and promoting, rather than the programme itself.

Workstream findings

This section describes each of the workstreams in turns and summarises the evidence base for each. We begin with telecare.

Telecare

Telecare refers to continuous, automatic and remote monitoring of users by means of sensors to enable them to continue living in their own home by minimising risks such as a fall or gas and flood detection. A key element of the programme in Scotland is an effective triage system, which ensures that emergencies are prioritised. Telecare is one of the most widely used forms of technology enabled care in Scotland. According to the most recent data, there are nearly 130,000 local authority-provided telecare systems currently in operation with an estimated additional 50,000 people being suported via housing and care provider organisation ^[2] . Among those aged 75 and older, 20% are using a telecare service.

There is a relatively strong evidence base for telecare. This reflects its advanced stage of implementation relative to other technology-enabled services. A range of outcomes were evidenced for clients in the existing evaluations, including:

• enhanced dignity, independence and quality of life in clients
• increased confidence of vulnerable clients to be more active
• increased health and well-being in carers
• reduced unplanned hospital admissions and prevention or delay of admission to care homes

One of the key challenges for telecare is variability in provision across Scotland. There are a range of providers, often using different equipment, and charging regimes vary. Although a policy, rather than programmatic issue, further research may be required to identify ways to tackle service variability. Other evaluation priorities for this workstream include exploring sustainability and why some users drop out over time.

Video Conferencing ( VC)

The initial focus of the video conferencing workstream was extending traditional VC from NHS settings to community health and social care settings. This had some initial success, but was delayed for technical reasons. In late 2016, the integration of VC into health and social care has been faciliated primarily through the bespoke Attend Anywhere system. This allows health and care staff to offer video calls as part of their day-to-day operations. There has been considerable success with rollout. Attend Anywhere is functional in 13 out of 14 Health Boards. However, given its very recent adoption, the evidence base is limited at this stage.

Evidence from evaluations of other VC systems suggest there are likely to be considerable benefits, particularly where VC facilitates specialist input to remote areas. Outcomes attributed to VC in these studies include better pharmaceutical management, improved access to specialist services, reduced hospital admissions and length of stay, time and travel savings for staff and clients, and family and clients having greater confidence in the care they are receiving. Areas for future development include a focus on the use of Attend Anywhere for wider public services, and impacts on the workforce and carers/families.

Digital Platforms

This workstream is concerned with supporting the development of an integrated digital platform that enables people to access their health and care records, engage with the health service and manage their own healthcare. This is an ambitious goal which has led to an independent programme of work. The role of the TEC programme has been to support development work for the National Health and Social Care portal, including commissioning a business case and testing components of the portal. The TEC Programme also supported two key citizen-facing programmes, Living it UP and ALISS, under this workstream, as well as some local work.

Given that this workstream is largely concerned with technology that is still to be developed, there is not an existing evidence base as such. However, the Living It Up platform evaluation points to some of the benefits of online platforms for clients with a chronic condition, including a greater sense of control and self-management, reduced social isolation and access to information on their condition and ways to manage it. This area of work could benefit from a greater emphasis on outcomes evaluation from platforms such as ALISS.

Home and Mobile Health Monitoring ( HMHM)

HMHM is the use of digital remote monitoring technology to enable patients outside of hospital to receive, record and relay clinically relevant information about their health and wellbeing. At the start of the programme, there were 485 users of HMHM. This had increased to 10,780 by December 2017, with 12 Health and Social Care Partnerships undertaking work in this area.

This workstream had the largest number of existing evaluations, reflecting the extent to which the technologies underpinning HMHM are becoming embedded in clinical practice (compared to video conferencing and digital platforms) and the specific focus on developing an evidence base. The main outcomes for patients centre around a feeling of reassurance from the increased connectedness that HMHM offers. There was both quantitative and qualitative evidence for clinically-focussed outcomes around health and some evidence of reduced health visits and hospital admissions. Challenges centre mainly around the use of the technology and overcoming the initial trepidation of staff and patients around its use. Future evaluations would be enhanced by including a greater emphasis on quality of life impacts and the experiences of carers, as well as seeking to evidence the role these technologies can play in prevention.

Economic evaluation data review

There were ten studies that contained economic data. All of those studies found a positive return on investment in technology enabled care. The two areas which have seen the greatest levels of funding and are furthest along with implementation - HMHM and telecare – have the strongest economic evidence base. It is our assessment that, across all workstreams, the studies underestimate the value of technology-enabled care as most placed an economic value on a narrow set of outcomes and considered value creation only for public bodies (i.e. non-economic outcomes were not valued in the main).

There was considerable methodological variation across the studies. Our main recommendation relates to greater standardisation of approaches across studies, thus enabling greater comparability and ensuring best practice in economic evaluation.

Implementation Research

A key finding from the data review is that, although many of the technologies have established an evidence base, the pace of implementation can be slow. This has also been noted in the wider literature on technology-enabled care (e.g. Glasgow et al. 2013; Ossebaard and van Gemert-Pijnen 2016). Implementation Science ( IS) has become an increasingly popular way to bridge the gap between science and practice and is an approach that could be usefully applied to the Scottish TEC programme to assist with mainstreaming of technologies, and the wider implementation of digital health and care as a whole. The literature points to several ways in which implementation is challenging in the context of technology-enabled care, including the rapidly evolving nature of the technologies, the importance of context and the need for multi-stakeholder buy-in.

The main findings from the primary and secondary research on implementation in this study are as follows:

• The TEC programme is already employing many implementation strategies that are consistent with international best practice
• The barriers and enablers identified in the primary research are consistent with those in the wider IS and technology-enabled care literature
• Staff resistance is the most frequently mentioned barriers, and more could be done to understand the determinants of staff resistance
• Further support on the use of IS frameworks to inform technology-enabled care delivery could assist with mainstreaming of technologies

Some enablers highighted by the research were:

• A good initial user experience of the technology
• Technology that works well, is personalised with simple protocols
• Staff and management buy-in with appropriate training and ongoing support and specialist skills
• Organisational readiness to embark on digital transformation

Further research to support implementation has already been identified as a key priority for the programme.

Measurement Framework

The difficulty of assessing technology-enabled care against the principles and requirements of traditional evaluation methods, such as Randomised Controlled Trials ( RCTs) was a theme that emerged throughout this study. It is clear that the standard methods of conducting evaluation, especially in health, are often not appropriate in this field.

Our high-level review of evaluation frameworks revealed a number of alternatives, the most promising of which is the Glasgow et al. (2016) Rapid, Relevant, Research Process ( RRRP). The RRRP framework addresses evaluation challenges such as the need for flexibility, speed, adaptability, attention to contextual factors and the evolving nature of the interventions.

Due to its close relevance to technology-enabled care, we propose that RRRP is adopted as an overarching framework for future evaluation for all digital health and care implementations. The adoption of a broad framework such as this should ensure a more consistent and strategic approach to evaluation and that best practice elements such as stakeholder engagement are being used. We have developed the following nine principles to guide measurement and evaluation for the TEC programme. In addition, we make several recommendations to improve measurement and evaluation set out in the next section.

Proposed evaluation principles

Principle 1: Be strategic: evaluations should add value and be cost-effective

Evaluation resources should be carefully deployed to ensure that they are addressing gaps. Where evidence already exists, even if it has been collected outside the TEC programme, evaluation should not seek to repeat this.

Principle 2: Plan and Scope: each evaluation should be carefully planned and scoped

Once an area of evaluation is chosen, it will require careful planning to ensure a good research design and appropriate scope are chosen. Well-planned evaluations can still encounter problems, but good planning minimises the risk of this.

Principle 3: Measure what matters

There are three aspects to this principle: (a) measure outcomes; (b) measure things relevant to people; and (c) ensure that indirect effects/externalities are captured. Co-producing research with your stakeholders is a good way to ensure the things that matter are being measured.

Principle 4: Methodological Plurality: the most appropriate methodology/approach should be chosen from a range of options

There are many different evaluation methods and approaches. No single approach or method is appropriate to all situations, nor is any intrinsically better than another: they all have strengths and weaknesses and work more or less well in different contexts. Employing a range of approaches across the programme gives richness to the data and should also enable flexibility ( Principle 6).

Principle 5: Timeliness: evaluation findings should be available in a timely fashion

As discussed, there is typically a considerable time lag involved when using experimental research designs such as RCTs. In the context of technology-enabled care, where the technologies under study are rapidly evolving, this delay may mean the findings are all but obsolete by the time of publication. When planning an evaluation, the likely time lag for any methodology should be considered alongside the time scale within which the results are needed.

Principle 6: Flexibility: there should be a focus on evaluation methods, which are iterative, adaptive and flexible

As mentioned above, the rapid evolution of technology in the field of TEC has implications for evaluation. As well as providing rapid results, the evaluation method needs to support the process of continuous learning by providing feedback loops.

Principle 7: Context matters: it should be central, focused on and reported

A key difference between the RRRP and the traditional approach to evaluation is the importance of context. Evaluation plans should demonstrate how they will ensure that context is considered, and evaluation findings should, wherever appropriate, include an understanding of how the context may have influenced the results.

Principle 8: Involve stakeholders and clients/citizens throughout

The involvement of stakeholders at the planning stage ensures that the evaluation measures things that are most important to those directly experiencing the change and multi-stakeholder approaches are now commonplace in many types of evaluation.

Principle 9: Use technology: data collection and analysis should be automated where possible

Data collection and analysis can be very time consuming, as well as susceptible to human error and bias. Using technology to support measurement reduces the risks of human error (e.g. from data being inputted incorrectly) and can be less resource intensive.