TEC programme data review and evaluation: options study

1.0 Background and approach

This section describes the background to the Technology Enabled Care ( TEC) Programme and to this study. It sets out the research questions, methodological approach taken and provides definitions of terms used in this report.

1.1 About the TEC programme

Launched in late 2014, the Scottish Government's TEC Programme is a circa £9 million a year Scotland-wide programme designed to significantly increase citizen choice and control in health, well-being and care services. Due to the start-up time required by many of the projects, the first phase of the programme will conclude in September 2018. Building on this, a second phase of work focusing on scaling up and adoption will commence from April 2018. The new Digital Health and Care Strategy for Scotland, published in April 2018, will build on achievements to date and set out future developments and priorities.

The TEC Programme was set up to respond to the need for service transformation in the context of rising demand for health and social care. By providing central funding to a dedicated programme, the aim is to drive technology enabled care deployment at the local level, alongside national infrastructure and support work and to place Scotland at the forefront of innovative approaches to technology enabled care

Scotland's leadership in this area has been recognised by the award of '4-star Reference Site' status by the European Commission's Innovation Partnership for Active & Healthy Ageing. This award, which is the highest standard under the system, was in recognition of its work on technology enabled care in an integrated care setting. Only 8 out of 74 sites were awarded 4-star status. Progress with TEC in Scotland has also been acknowledged in the recent Health and Sports Committee report (Health and Sports Committee, 2018)

The Programme has been structured over five interrelated workstreams:

Expanding Home and Mobile Health Monitoring ( HMHM)

Expanding the use of video conferencing ( VC)

Digital platforms

Further expanding and embedding the use of telecare

Improvement and support

Although most of the activities that take place under these workstreams are distinct, they are also intended to be mutually reinforcing. The aim is for the whole to be 'greater than the sum of its parts', thereby furthering the adoption of technology-enabled care over and above the discrete activities. This recognition of the cross-cutting element of the programme led to the development of the fifth workstream, which is focused on enablers and barriers to implementation.

1.2 About this study

A key component of the TEC programme is a focus on service transformation and continuous improvement supported by measurement and evaluation. Over its life, many evaluative activities have taken place. These have primarily sought to demonstrate the effectiveness of technologies that sit within individual workstreams. They take the form of one-off evaluations, improvement reports and ongoing monitoring, and forecast reports to assess the impact of scaling up, particularly on cost-effectiveness.

The purpose of this study is to synthesise the findings of these evaluative activities and inform the priorities for future evaluations by systematically identifying gaps in evidence. Specifically, the aims are to:

• Develop a clear understanding of the expected outcomes of the TEC investment at programme and workstream levels for different groups of people over the short-, medium- and long-term and how these are expected to be achieved.
• Establish the nature, quality and relevance of existing evidence and identify key information gaps.
• Detail a range of options for Scottish Government, in association with the TEC Board to consider for a further phase of evaluation, explaining the extent to which each option would be robust and cost-effective in understanding the impacts of technology-enabled care and establishing longer term monitoring.

A further 11 additional sub-aims were also identified, which have informed the development of the research questions in the next section.

1.3 Research questions

The research will address the following five research questions:

• What is the range of the existing evidence and data on the TEC programme? What is the quality of that evidence and where are the key gaps for both range and quality?
• What do we know about the barriers and enablers to effective implementation of TEC?
• Once TEC is implemented, what do we know about how sustainable its use is over time?
• What is the state of knowledge on cost effectiveness? How is cost-effectiveness taken into consideration when scaling-up an approach – what is the standard of evidence used? How is cost effectiveness compared against clinical effectiveness and personal outcomes?
• What are the longer-term monitoring and evaluation requirements to support the on-going development and delivery of TEC? Are there standardised metrics that can be recommended for some grantees to report against that would support the development of the evidence base?

1.4 Study methodology

The study was carried out in seven stages between September 2017 and March 2018.

Stage 1: Develop logic models for workstreams and overall programme
Building on existing work in this area, a series of logic models were developed. This was conducted over the course of two workshops with workstream and programme staff. The logic models were subsequently refined by the researchers with reference to programme documents and secondary literature. Final versions were reviewed by workstream and programme staff. The logic models were developed to provide a means of systematically assessing the strength of the evidence base in each area of work against an agreed outcomes framework.

Stage 2: Initial classification of evidence
Evaluation reports were shared with Just Economics for inclusion in the review. Documents were categorised in a spreadsheet depending on whether they contained qualitative, quantitative or economic data (or a combination of these). Some contained non-evaluation reports ( e.g. evaluation frameworks). These were reviewed for information but were not included in the data synthesis. Appendix 1 sets out the results of the classification.

Stage 3: Data review
Qualitative and quantitative data were reviewed separately. The quantitative reports were inputted into an Excel spreadsheet, which extracted data on geography, beneficiary, subject, sample size, population size, study type and outcomes. This process allowed us to efficiently summarise the quantitative evidence base, including key findings and gaps.

For the qualitative data, NVivo was used to analyse the findings. The analysis began by separating out the data by aims ( e.g. outcomes, implementation and stakeholder type). For each sub-group, further categorisations, such as geography, age group, or beneficiary, were identified. These were then cross-tabulated with a set of variables (nodes) such as outcomes, satisfaction, barriers/enablers and sustainability. These nodes were developed through coding of the raw data. Further sub-nodes were also identified where appropriate. Appendix 2 sets out the results of the data review by workstream.

Stage 4: Comparing data and evidence with logic model
In this stage, the evidence base was mapped onto the logic models developed in Stage 1. Matrices were constructed to indicate which elements of the logic model were supported by data, the type of data (qualitative/quantitative/mixed) and providing an assessment of its quality. The evidence matrices can be found in Appendix 3.

Stage 5: Economic data review
All TEC evaluations containing economic data were identified during Stage 2. This resulted in 11 evaluation studies across the four workstreams, which were systematically analysed with results entered in an Excel spreadsheet. For each study, data was extracted on findings, method, sensitivity analysis, discounting, benefit period, and input costs. This enabled us to make comparisons across studies and recommendations for standardising future economic evaluations.

Stage 6: Implementation research
One of the key findings of the data review is that, even for technologies where there is a strong evidence base, mainstreaming can be slow. As a result, it was decided to focus Stage 6 on implementation with an emphasis on barriers, enablers and strategies.

Stage 6 comprised three strands:

1) Identification of barriers and enablers to implementation in TEC evaluation studies (conducted as part of Stage 3).
2) Literature review on Implementation Science ( IS), including its use in TEC to understand how IS had developed in TEC and what lessons could be learned.
3) Two workshops with staff involved in frontline delivery of technology-enabled care across Scotland and the TEC programme (see Appendix 4)

Stage 7: Measurement framework
The final research task was the development of a measurement framework and options for future evaluation. This was informed by:

1) A high-level literature review of measurement and evaluation approaches to technology-enabled care
2) The review of existing evaluations in Stages 2-3
3) Direct engagement with TEC staff at the logic model and implementation workshops in Stages 1 and 6

1.5 Definitions

For the purposes of the Scottish Government-funded programme and this study, Technology Enabled Care ( TEC) is defined as "where outcomes for individuals in home or community settings are improved through the application of technology as an integral part of quality, cost-effective care and support". There is an explicit emphasis on home and community settings and on quality. The focus is on individuals rather than, more narrowly, patients. This reflects the integrated context for health and social care in Scotland.

The definition encompasses a vast range of applications from the provision of online information to sophisticated monitoring devices and complex remote interventions. There is also a wide range of related terms including digital health, eHealth, mHealth, telehealth, telemedicine, assisted living, home automation and smart homes, and these terms are often used interchangeably. This creates some difficulty when comparing technologies as the same technologies can sometimes be described using different terms. Whilst the term TEC is usefully broad, this also means it covers a wide range of applications that differ from each other in terms of their intended process and outcomes.

A final note on language relates to the beneficiaries of the TEC programme. Under some workstreams, such as telecare, that are in the social care sector, beneficiaries are known as service users. Under health-oriented strands, such as HMHM, they are called patients. For services like the health and care portal, citizens may be the most appropriate term. To overcome this, we have used the term 'clients' for all direct beneficiaries or users unless otherwise specified.

1.6 Report structure

The remainder of the report is structured as follows:

• Section 2 presents the findings from the logic model development and data synthesis. We describe the narrative, present the logic models and set out the evidence base for the programme, and for each workstream.
• Section 3 present the findings of the economic evaluation review
• Section 4 discusses the findings of the implementation research
• Section 5 presents the measurement framework, evaluation options and concludes the report.