Universal Health Visiting Pathway evaluation - phase 1: main report - primary research with health visitors and parents and case note review

The Evaluation of Health Visiting in Scotland

This section covers the methodological approach and methods employed in the evaluation process. The methods have also been published in the British Medical Journal Open¹¹. The study adopted realist evaluation¹² principles, employing both qualitative and quantitative data in a mixed-methods approach, to address the aims of the evaluation.

Previous Research

In April 2013, implementation of the Universal Health Visiting Pathway was piloted in NHS Ayrshire and Arran. A process evaluation of the service¹³ was undertaken in 2015 to provide initial learning and guidance to support the national roll out of the UHVP.

In Spring 2016, an evaluability assessment¹⁴ was commissioned by Scottish Government. The aim was to assess the strengths and weaknesses of various options for evaluating the Universal Health Visiting Pathway and to provide recommendations on the methodology for the evaluation. The evaluability assessment was a collaborative, process including key stakeholders from policy, practice and research. The final recommendation of the evaluability group was a process evaluation and an outcomes evaluation based on natural experiment methodology.

Aims of the evaluation

Since every family will take about five years to go through the health visiting pathway (pre-birth to school entry or when child is 5 years), it is important to ensure the full cycle of the health visiting pathway can be evaluated. Phase 1 of the evaluation commenced in 2018 and provides baseline outcomes data and early learning about the health visiting pathway. Phase 2 of the evaluation will provide evidence regarding the medium to long term impact of the health visiting pathway and identify further areas for improvement.

The key aims of the evaluation are:

1) to examine what elements of the UHVP are being implemented in which areas, when and how.

2) to determine the extent to which the UHVP is implemented and delivered across Scotland and assess any associated impacts over the longer term.

3) to identify and explain to what extent recommendations to fill gaps in the UHVP are delivered and their impacts on services, staff and children and families.

Realist evaluation is well suited to understand the complexities of healthcare programmes and can provide useful insights into how programmes work, whilst placing emphasis on identifying and explaining factors that can be improved in order to enhance the programme^15,16,17. The UHVP can be described as complex intervention and it is important to understand and explain how it works. Realist evaluation is concerned with understanding how contexts interact with underlying mechanisms of programmes to produce health and wellbeing outcomes for children and families. The implementation of the UHVP was not uniform across Health Boards and therefore the evaluation aims to contextualise findings based on the stage of roll-out across or between Health Boards.

Structure of the evaluation

In order to address the aims of the evaluation, the evaluation began with a comprehensive review of the existing logic model as described below.

Following this, four main research approaches or evaluation components were identified to assess outcomes. This included qualitative research with parents and health visitors, a case note review, surveys with parents and health visitors and routine data analysis (findings from the routine data analysis will be reported separately).

As stated above, the evaluation has been designed across two phases:

• Phase 1 – provides baseline outcomes data and early learning in regard to the process evaluation.
• Phase 2 – will provide evidence in regard to the outcomes that health visiting is contributing towards and provide further information for the development of the processes health visitors use.

Logic Model Review

A logic model supports evaluation by setting out the relationships and assumptions, between what a programme will do and what changes it expects to deliver. A logic model can be particularly valuable in drawing out gaps between the ingredients of a programme, the underlying assumptions, and the anticipated outcomes. As part of the evaluation and the realist approach it was important to understand the assumptions underpinning the UHVP from the perspectives of stakeholders involved in its design and implementation, as well as those with an interest in early years policies and programmes aimed at improving outcomes for children and families.

A range of stakeholders were identified and invited to take part in two separate workshops. Stakeholders included Scottish Government policy teams, researchers and professionals, senior nursing, and health visiting staff from Health Boards, third sector organisations, academic experts in Health Visiting and the members of the Evaluation Team.

The logic model review part of the evaluation was designed to revise the previous logic model produced in 2016 as part of the Evaluability Assessment. This review, which was carried out in October 2018, was necessary because it was agreed that a revised model was needed to reflect current practices and to incorporate new learning that might have ensued from the ongoing implementation of the pathway. The revised logic model was expected to drive the rest of the evaluation. It was carried out across two workshops and guided by co-production principles to enable a structured, participatory approach where participants were actively engaged to contribute.

Workshop one

As part of the activities for workshop one, stakeholders were divided into five groups and were asked to review the existing logic model and discuss whether the inputs and activities reflected current or expected practice in line with the pathway. Stakeholders were asked to review and discuss all of the outcomes within the logic model. Following the group discussions, all groups reconvened, and feedback was gathered and discussed by the wider group. Several changes were made to a number of outcomes. New outcomes were also introduced to ensure that the logic model captured appropriate outcomes for children and families as well as the health visitor workforce. Most importantly, stakeholders agreed that the logic model should carefully consider priorities relevant to the current role of health visitors and should be aligned with outcomes, which are realistic and achievable.

Following the first workshop, the evaluation team gathered all of the feedback and developed a revised first draft of the logic model. The draft logic model was then circulated to workshop participants (stakeholders) in advance of the second workshop.

Workshop two

During workshop two, stakeholders were asked to briefly review the draft logic model and provide any final comments. These final comments were used to refine the logic model further. The final logic model is shown in Appendix 2. The items in the logic model have been numbered for ease of reference. Overall, the final logic model sets out the short, medium and longer-term outcomes of the UHVP. As part of workshop two, stakeholders also outlined relevant data sources required to measure these outcomes.

Study setting and population

This is a national evaluation and covers all 14 Health Board areas in Scotland. However, some elements were carried out in five Health Boards or case study areas in order to provide a more in-depth understanding of the UHVP. A robust process informed the rationale for selecting Health Boards as case study areas. It included a self-completion questionnaire sent to all Directors of Nursing in each of the 14 Health Boards to inquire about the stage of the UHVP implementation. Using the information received from the self-completion questionnaires alongside the geographical profile of each area, population data and the Scottish Index of Multiple Deprivation, the following case study areas were selected:

• NHS Ayrshire and Arran
• NHS Borders
• NHS Grampian
• NHS Lothian
• NHS Tayside

Table 1.1 shows the evaluation components and the Health Board areas they cover.

Table 1.1 evaluation components and Health Board areas of coverage

Health Board Areas

• All Health Boards

  Study Components

  • Survey of health visitors
  • Survey of parents
  • Routine secondary data analysis
• Five Health Boards (case study areas)
  • Qualitative interviews and focus groups with health visitors
  • Qualitative interviews with parents
  • Case note review

Interviews and Focus Groups

Health visitors from each of the case study areas were invited to take part in an interview or focus group. Parents were also invited to take part in interviews. Around 90% of the interviews were face-to-face with about 10% conducted by telephone. Interviews and focus groups lasted between 30 minutes and an hour and a half. Interviews were conducted between May 2019 and January 2020.

Recruitment and data collection

Health visitors – Interviews and Focus Groups

Health visitors from each case study area were sent information about the evaluation and invited to participate in either a focus group or individual semi-structured interview. The topic guide (Appendix 3) explored adherence to implementation and delivery of UHVP, barriers and facilitators to delivery, perceived impact of UHVP on outcomes for children and families, training and support structures, as well as engagement with other professionals.

Parents – Interviews

Parents were recruited through their health visitors, who informed them about the study and offered them a study information pack. The recruitment strategy was continually reviewed to ensure parents with a range of characteristics (e.g. age, first time parents) were included in the sample. The information pack included details of how to contact the researchers for those interested in participating in the study. Potential participants were contacted by a member of the evaluation team to arrange a suitable venue and time for interview. The topic guide (Appendix 4) included questions around experiences of parents as well as their perceived impact on how the service has influenced their families’ health and well-being. Each participant received a high street store vouchers worth £20 in return for their time.

Information about the number of parents and health visitors who took part in the study is provided in Table 1.2.

Table 1.2: Number of participants that took part in the qualitative research

Participants	Health Board 1	Health Board 2	Health Board 3	Health Board 4	Health Board 5
Health visitors – Interviews	10	10	10	10	10
Health visitors – Focus Groups	10	5	5	6	7
Parents – Interviews	7	12	12	12	12
Total (138)	27	27	27	28	29

Analysis of interview and focus group data

All interviews and focus groups were audio recorded and transcribed verbatim. Transcribed data were coded and analysed by thematic analysis, using QSR NVivo V.11. Thematic analysis was used particularly because of its suitability in exploring qualitative data of this nature, as it is possible to examine both within-case and cross-case themes.

Case note review

A data collection tool was tested, and data was gathered from health visitor case notes between October and the end of December 2019, ensuring that no identifiable personal information was attached to any of the data.

Case note sample

The sampling criteria was finalised in discussion with Clinical Managers within the Boards.

Inclusion criteria

• 15 case notes from each area, evenly spread across Health and Social Care Partnerships (HSCPs) (HSCPs are the organisations formed as part of the integration of services provided by Health Boards and Councils in Scotland. Each partnership is jointly run by the NHS and local authority).
• Case notes of children that were at least 13 months old
• Randomly selected – but with Health Plan Indicator (HPI) of Core and Additional levels proportionate to the geographical area.

Exclusion criteria

• Transferred in from another area at any point
• Participation in Family Nurse Partnership (FNP) at any point (Families that have a family nurse receive FNP from pregnancy until their child is aged two. Families participating in FNP receive a more intensive and more regular service than is outlined in UHVP and their experiences are not likely to be typical of those receiving the UHVP - they were therefore excluded from this study).

Table 1.3. Case note review sample

Health Board (HB)	HB1	HB2	HB3	HB4	HB5
Total number of case records used	15	13	15	15	15
Number of Core case records	10	8	10	11	9
Number of Additional case records	5	5	5	4	6
Youngest age born during week commencing	Week 39 In 2018	Week 30 In 2018	Week 39 in 2018	Week 50 in 2018	Week 20 in 2019
Oldest age born during week commencing	Week 10 in 2015	Week 21 In 2016	Week 49 in 2016	Week 51 in 2017	Week 27 in 2015

Case Note Data collection

Although the main data collection tool was used consistently, the case notes were a mixture of electronic and paper records. Electronic case notes were supported using different platforms across the Boards. This variance across the case notes created some challenges in terms of consistently being able to locate similar information from within the records – as the same information was recorded differently across systems.

Case Note Data analysis

The anonymous data was transferred from the data collection tool onto an MS excel spreadsheet. Every case was assigned an identification code and there were no personal identifiable data linked to any identification code. Information relating to each case was kept together to ensure a level of understanding around health visitor contacts. Analysis of cases from each Health Board area were reported separately.

Health Visitors’ Survey

The questionnaire (Appendix 5) for the health visitors’ survey was designed by the evaluation team with advice from the Scottish Government and the Evaluation Advisory Group. It was also reviewed by a small number of health visitors, whose feedback was incorporated before it was converted to a script that could be completed online.

The survey was open from late November 2019 to early January 2020. Executive Directors of Nursing and Directors of Public Health in each Health Board were asked to nominate a staff member to forward an email invitation from the evaluation team to all health visitors and family nurses (and their managers) in their area and encourage them to take part in the survey. In total 554 health visitors and 37 family nurses took part in the workforce survey about their experiences and views of the UHVP.

The data were weighted to match the profile of health visitors across Scotland in terms of Health Board and the proportion of part-time and full-time staff.

Parents’ Survey

The impact of the COVID-19 pandemic

The parents’ survey was both delayed and redesigned due to the COVID-19 pandemic. As NHS Scotland was placed on an emergency footing in March 2020, it was agreed that it was no longer appropriate to pursue the original plan of asking Health Boards to undertake the sampling of parents on behalf of the evaluation team. Various options for alternative ways of approaching the Phase 1 parent survey were discussed. It was acknowledged that all the alternative options would result in a much reduced sample size (from the original target of 6,000 online responses) and greatly reduced subgroup analysis, but it was also agreed that delaying the parent survey further was not desirable.

Of the options identified, it was agreed that conducting a survey using re-contact details for families with children under five who had taken part in existing Scottish Government surveys was the preferred route. The revised design involved:

• Approaching parents of children aged under five years old who consented to be recontacted in either the Scottish Household Survey or Scottish Health Survey in 2018 or 2019 (sample details provided by Scottish Government to the evaluation team)
• Inviting all of these parents to take part in the survey online initially, either by email invitation (where emails were available), or letter
• Boosting the online survey with telephone interviews.

The questionnaire for parents

The questionnaire aimed at parents was piloted with a small number of parents, whose feedback was incorporated before it was converted to a script that could be completed online and by Computer-Assisted-Telephone-Interviewing (CATI) (Appendix 6). As the fieldwork took place during the COVID-19 pandemic, and after Scotland had been in lockdown, the questionnaire was amended, prior to fieldwork, to take account of the potential impact of the lockdown on health visiting services. A small number of questions on engagement with wider health services and on parents’ mental wellbeing were also added to the questionnaire in order to provide the Scottish Government with data on views and experiences of health services among parents of young children since the beginning of the COVID-19 pandemic. However, the parent survey mainly focused around key UHVP outcomes as experienced in the early roll-out / pre-roll out stage of UHVP in relation to key target outcomes.

Survey data collection

The survey was open from 12 August to 17 September 2020. All parents in the sample (2,317) were invited to take part in the survey online via an email or letter (if email addresses were unavailable). The invite requested that the parent who had had most contact with health visitors complete the survey. After the online survey had been live for a week, telephone fieldwork began and a proportion of those who had not yet taken part online were invited to take part in a telephone interview. Quotas were set on area deprivation (as measured by Scottish Index of Multiple Deprivation (SIMD) quintiles), with the telephone interviewing targeting (where possible given the available sample) those in the SIMD quintiles where response to the online survey was lowest. In total, 550 parents took part in the survey, with 250 taking part online and 300 taking part by telephone (24% of the total issued sample).

For parents with more than one child aged five or under, at the start of the survey, one of their children was selected randomly to be the focus of the survey, and parents were asked subsequently to answer about their experiences of health visiting only in relation to this child.

While the vast majority of families with children under five are seen by a health visitor, a small minority, where the parents are aged under 20 years, are seen by a family nurse from pregnancy until the child is two years old. Only one respondent in the survey reported seeing a family nurse so, for purposes of simplicity, the term health visitor is used throughout to refer to all respondents.

Survey analysis

The data from the parents’ survey is unweighted. There is no definitive source of data on the profile of parents or carers of children under school-age in Scotland. However, analysis of the Scottish Household survey of the profile of highest income householders living in a household with a child aged 5 or under (the vast majority of whom will be their parents or carers) suggests that older parents are likely to be slightly over-represented in the sample (parents aged 35 years and above accounted for 62% of the UHVP parent sample, compared with 51% of highest income householders in the 2019 Scottish Household Survey). The sample is also somewhat skewed towards parents of children aged 3 or older, and has slightly fewer parents in the most deprived quintile of areas (as measured by SIMD) compared with other quintiles. Significant differences between these groups of parents are noted in the relevant sections of this report.

As the number of respondents was insufficient to allow analysis of differences between individual Boards, Health Boards were grouped into Health Board Regions for analysis as follows:

• East (Fife, Lothian and Borders)
• North (Western Isles, Grampian, Highland, Orkney, Shetland, Tayside)
• West (Ayrshire & Arran, Dumfries and Galloway, Forth Valley, Greater Glasgow & Clyde and Lanarkshire).

Survey sample

Table 1.4 below, shows key demographics for the achieved sample as a whole, and for each mode of data collection.

Table 1.4 Parent survey: sample profile

Parent’s age (at time of survey)	All	Online	Telephone
16-29	14%	13%	15%
30-34	24%	22%	26%
35+	62%	65%	59%
Number of other children in the household
None	33%	36%	30%
1 or more other children	67%	64%	70%
Age of selected child (for whom they were asked to answer the survey questions)
Under 1	6%	6%	6%
1 year-old	14%	17%	11%
2 year-old	18%	20%	17%
3 year-old	24%	23%	25%
4 year-old	24%	20%	27%
5 year-old	13%	14%	13%
Deprivation (SIMD) quintile (all who gave a postcode = 540)
1 (most deprived)	16%	16%	15%
2	20%	17%	22%
3	21%	18%	23%
4	22%	26%	19%
5 (least deprived)	22%	23%	21%
Urban-rural (all who gave a postcode = 540)
Urban	75%	75%	75%
Rural	25%	25%	25%
Household income (all those who gave an income = 479)
£15,599 or less	8%	7%	8%
£15,600-£25,999	15%	17%	13%
£26,000-£36,399	19%	18%	19%
£36,400-£51,999	22%	25%	20%
£52,000+	37%	32%	40%

It should be noted that the sample of parents is a small sample and the results should be interpreted with caution.

Routine Data Analysis

The methods adopted for the routine data analysis elements of the evaluation will be published within each of the relevant routine data reports.

Ethics approval

The evaluation received approval from the School of Health in Social Science Research Ethics Committee, University of Edinburgh, the case note review received approval from the Public Benefit and Privacy Panel for Health and Social Care in Scotland and the survey received approval from the Scottish Government Statistics Public Benefit and Privacy Panel.

Reports

The following four reports will be produced as part of the Phase 1 evaluation:

• Phase 1 Report – Primary Research with Health Visitors and Parents and Case Note Review (this report)
• Phase 1 Report – Routine Data Analysis – Workforce (published)
• Phase 1 Report – Routine Data Analysis – Outcomes (due to publish 2022)
• Phase 1 Report – Routine Data Analysis – Implementation and Delivery (due to publish 2022)

Impact of COVID-19

The first phase of the evaluation was almost complete prior to the COVID-19 pandemic, which resulted in a national lockdown in March 2020. The fieldwork in relation to the case note review, qualitative focus groups and interviews with health visitors and parents had all been completed prior to March 2020. The planning for the parents’ survey was still in process in February and March 2020. The onset of the pandemic meant that the parents’ survey methodology had to be substantially changed.

After redesigning the methodology for the parents’ survey, an online and telephone survey of parents took place between 12 August to 17 September 2020. Due to the timing of the questionnaire, questions about the on-going COVID-19 pandemic were included in the survey alongside the original questions relating to health visiting services more generally. The results of this are reported in section 8 of this report.

The last year (2020) has been a period of great uncertainty across NHS Scotland, including health visiting, which continued throughout but was delivered in a completely new and untested way. The health visiting service had to immediately pivot to a digital or telephone-based service, with home-visits being restricted to a small number of priority areas such as the early days and weeks following birth and support for the most vulnerable families.

The full evaluation of the UHVP will be conducted in two phases, this report focuses on Phase 1. While Phase 1 of the evaluation does not fully capture the changes to the service during COVID-19, it will set the scene for some of the anticipated findings from Phase 2, which will explore the impact of the pandemic more fully.