Type 2 Diabetes - framework for prevention, early detection and intervention: evaluation

2. Evaluation approach

The aim of the evaluation was to offer the Scottish Government and its delivery partners an in-depth understanding of the Framework's implementation in the first year. In particular, the work aimed to:

• assess whether the Framework was delivered as intended
• identify effective strategies for reaching target groups
• identify potential barriers to implementation as well as enablers of success.

An additional aim was included as a result of the COVID-19 pandemic:

• identify the impacts to implementation caused by COVID-19 and the steps taken to adapt services.

The researchers were also invited to consider the detailed questions developed by the EAG (see appendix 1). The approach to undertaking this evaluation is described in this chapter and summarised in the diagram below.

Figure 2.1 Methodology

Phase 1. Set Up Period

• Inception meeting
• Initial contact with early adopter areas
• Desk based activity
• Production of local evaluation and sample plans
• Design of research tools
• PBPP application and Health Board permissions

Phase 2. Fieldwork

• Interviews with programme delivery leads
• Interviews and survey with primary, secondary, community care staff and weight management providers
• Interviews with local stakeholders
• Interviews with patients

Phase 3: Analysis and Reporting

• Updates for and meetings with Scottish Government
• Analysis of all data
• Internal team analysis meeting
• Draft final report
• Final report

2.1 Phase 1: Set-up period

The first phase involved initial contact, meetings and discussions with local leads in each early adopter area to gather details of the plans for the implementation of the Framework and progress so far.

To prepare for the design of the evaluation plans, background documentation was reviewed to provide details of progress and plans in each of the three areas. The evaluation and respondent sampling plans were agreed with each area setting out the type and number of interviewees that would be engaged in the evaluation. Research tools, such as surveys and interview schedules for various stakeholders were produced alongside privacy notices.

Before the fieldwork could begin, approval from the Public Benefit and Privacy Panel (PBPP) had to be obtained. An application was submitted in December 2019 with final approval only received in late November 2020. PBPP approval came with stipulations on the methodology in terms of recruitment of participants, the questions that were to be asked and the data that could be collected. Following subsequent approval from local NHS board research and information governance teams, fieldwork commenced in January 2021.

Impact of the COVID-19 pandemic on the evaluation

As a result of COVID-19, fieldwork for non-essential research was paused. In addition,there were significant personnel changes in each of the early adopter areas as staff were re-deployed and responsibilities for certain roles changed, with some key staff moving to other organisations. Often, it was not possible to interview these people before they left their role.

The current study was not designed to be a comprehensive service audit or service mapping exercise. The information gathered about the services delivered or planned within a given adopter area was dependent on the role and knowledge of the individual/s interviewed for that area. Some staff were new in post when interviewed and so were not necessarily able to provide comprehensive information about activities delivered. Therefore, while the data gathered provides a sound overview of changes brought about by the Framework, it it is not intended to provide full details of all weight management services and programmes in all adopter areas.

2.2 Phase 2: Fieldwork

The fieldwork included interviews with patients; health care staff (in primary, secondary and community settings) and weight management providers; programme delivery leads and other local stakeholders who supported delivery. To improve access to staff working on the frontline, who may have found it difficult to make time for an interview, an online survey was also included. This was aimed at primary care staff and weight management providers involved in the delivery of weight management programmes.

A total of 83 individuals contributed their views to the evaluation.

Table 2.1: Evaluation participants

Area	Patients	Health care staff & weight management providers (interviews)	Health care staff & weight management providers (survey)	Programme delivery leads	Local stakeholders
Ayrshire & Arran	7	14	12	2	1
East Region	3	12	6	4	1
Tayside	6	4	4	3	4
Total	16	30	22	9	6

As a result of COVID-19 restrictions, all interviews took place remotely via telephone or Microsoft Teams.

Programme delivery leads

Nine programme delivery leads across the three early adopter areas were interviewed. These included programme managers/leads, project managers/leads and clinical leads. Discussions covered key areas about delivery and implementation of the Framework.

Healthcare staff and weight management providers

30 staff working in primary, secondary or community care or weight management providers were inteviewed. In line with the PBPP requirements, local leads identified the interviewees and sent an invitation to take part along with information about the evaluation. Through these interviews, issues were explored around the implementation of the Framework in the early adopter area, the changes and activity that were taking place, perceptions and experiences of implementation, lessons learned and the impact of COVID-19.

Engaging health professionals is always a challenge in evaluations like this because their clinical responsibilities and time pressures limit their opportunities to take part in interviews. This situation was exacerbated by the additional pressures caused by COVID-19 thus an online survey was introduced covering the same topics as in interviews. Local leads promoted the survey through direct emails to staff, social media and newsletters and 22 responses were received.

Local stakeholders

Six local stakeholders took part in interviews as part of the evaluation. These interviewees included representatives of organisations that supported delivery such as health and social care partnerships (HSCPs), universities and third sector bodies.

Patients

There were 16 interviews with patients who were at risk of or had been diagnosed with type 2 diabetes. In line with the PBPP requirements, the local lead approached potential interviewees to request their participation and provided information about the evaluation and their role so that they could give informed consent to take part. Individuals who were willing to participate either contacted Blake Stevenson directly, or early adopter site staff sent their contact details securely, with their permission, to Blake Stevenson. Due to the PBPP stipulations, no personal data, such as demographic characteristics, was captured from patients.

These discussions covered patients' views and experiences of the support and advice that they had received with type 2 diabetes and any education or weight management programmes that they had taken part in.

The number of patients interviewed was lower than intended. This was partly a result of COVID-19 pressures and delays resulting in lower numbers taking part in programmes, which limited the pool of potential participants. It was also partly due to the process of recruitment as a result of information governance requirements which relied on patients making contact with the team.

It is important to note, this was a qualitative evaluation and qualitative research is not intended to be representative. The patients raised valid issues but these cannot be taken as representative or an indication of how widespread. Those that did engage with this research were likely to be more engaged in the programmes and hence overall were more likely to be more positive about experiences.

2.3 Phase 3: Analysis and reporting

The qualitative data gathered from the interviews and survey were analysed in line with the research questions and any other emerging themes.

This involved a process of coding responses to identify key and recurring themes as well as any differences in viewpoints among different groups of participants or areas. The online survey responses were coded and analysed in the same way as for the interviews and incorporated alongside the interview data. The research team held a team meeting to explore the themes and issues emerging and design the report outline for discussion and agreement with the Scottish Government team. There was also a process for fact-checking the information in the draft report.

The findings are presented thematically and aligned to the three main aims of the evaluation and include an overview of the journey of each early adopter area.

Quotes from respondents have been included to illustrate key points, attributed to the type of respondent and area where this does not risk confidentiality.