Universal Health Visiting Pathway evaluation: phase 1 report - routine data analysis - baseline outcomes

Methodology

Study design

In this outcome evaluation, routine clinical and administrative data have been used to examine 18 outcome measures agreed with the UHVP Research Advisory Group.⁴ The outcomes were agreed to enable investigation of the impact that the Universal Health Visiting Pathway has had on children's development and health. Over the course of Phases 1 and 2 of the evaluation, the results of two cohorts of children will be compared:

• those children who did not receive any part of the UHVP pathway (the fully unexposed group) and
• those children who were born on or after the date on which the UHVP was fully implemented in the health board in which the child was born (the fully exposed group).

The study design used to conduct this evaluation is known as an interrupted time-series (ITS) design, and is described in greater detail in the 'UHVP Outcome Evaluation Statistical Analysis Plan'.⁵ Briefly, ITS compares the rate of a particular outcome before and after implementation of the UHVP, using a counterfactual after the implementation of the UHVP. The counterfactual is an extrapolation of the trend in the fully unexposed group, continued after the intervention has been implemented, to demonstrate what might be expected to happen, had the intervention not occurred. The trend in the fully exposed group can then be compared with the counterfactual.⁶

In this current report, which forms part of the first phase of the evaluation, baseline information only is provided on each of the 18 outcome measures derived from the fully unexposed cohort of children. The aim of this report is to explore trends in outcome measures, prior to exposure to the pathway, which will be built into data models in the second phase of the evaluation. Alongside this, the quality of the data relating to the 18 outcomes has been assessed.

Outcome measures

After detailed discussion with the UHVP Research Advisory Group, wider members of the UHVP evaluation team, Information Services Division (ISD) (now Public Health Scotland) and Scottish Government (SG) analysts, the 18 outcome measures to be included in the UHVP outcome evaluation were agreed in February 2020. A full description of the process is included in the report 'Outcome measures to be included in the outcome evaluation'.⁴ The outcome measures are listed in Table 1.

Study population

As noted above, in this report our analyses are restricted to the pre-intervention (fully unexposed) cohort. These children were born between 1 January 2011 and 31 March 2016, and all were offered:

• the health visitor first visit,
• the 6-8 week child health review, and
• the 27-30 month child health review, which has been delivered across Scotland from April 2013.

No children are included who were partially exposed to the UHVP (i.e. children who received a 13-15 month review or selected additional contacts in infancy, but were born before the UHVP programme was fully implemented in the health board in which they live). To include partially exposed children in the analysis would potentially dilute the observed effect of the intervention, since these children would not have received the benefit of the intervention in its entirety. Thus for some health boards there is a group of fully unexposed or partially exposed children, born between 1 April 2016 and the day before the UHVP was implemented in that health board, who do not contribute data to the analysis.

In three health boards, some pathway infancy contacts were offered to families before the implementation date for the UHVP (see Table 2). As can be observed, the number of children born before 1 April 2016 who were partially exposed to the UHVP in these health boards is very small (n = 4,256), i.e. 1.4% of the number of live births between 1 January 2011 and 31 March 2016 (n = 297,337). These children have been excluded from the analysis in this report for Outcomes 1-7.

It was not possible to exclude children who might have been partially exposed to the UHVP for the child protection outcomes (Outcome groups 8-9). Where the number of children with a child protection episode was very small (i.e. less than 5) in a council area, the exact data were not released by the Scottish Government, in order to avoid the identification of children. This meant that when it came to grouping these cases by health board, a health board such as Tayside, which comprises three council areas, all of which had data suppressed due to low numbers, could in reality contain 0 to 12 children with a child protection episode. For this reason it was not possible to exclude these partially exposed children from the health board outcome data.

For clarity, inclusion and exclusion criteria for the fully unexposed cohort are specified below.

Inclusion criteria

All babies who were born in Scotland between 1 January 2011 and 31 March 2016 inclusively, and who have not been exposed to any of the additional visits that were newly introduced in the UHVP (i.e. the 13-15 month and 4-5 year visits) are included in the fully unexposed group.

Exclusion criteria

The following groups of babies born in health boards where elements of the UHVP were implemented before April 2016 have been excluded (see Table 2):

• Babies born in Orkney, from 1 October 2015 to 31 March 2016 (i.e. the last two quarters);
• Babies born in Shetland, from 1 April 2015 to 31 March 2016 (the last four quarters);
• Babies born in Tayside, from 1 April 2015 to 31 March 2016 (the last four quarters).

Data sources

In Phase 1, analyses of the fully unexposed children have been undertaken solely using aggregate data. This means that unlinked numerator and denominator data were requested from the numerous data sources (see Table 3) (i.e. count data for (i) sequential quarters of the number of live births, and – separately – (ii) the number of children who have the outcome of interest).

There are a number of limitations associated with this approach. For example, children who moved into an area can subsequently appear in the numerator (but are excluded from the denominator); children who died or moved out of an area are not removed from the denominator; and children are assumed to remain in their birth board of residence until outcomes data are collected (i.e. up to their third birthday). However, it is probable that these numerator/denominator mismatches balance out, and the estimates generated using an unlinked approach are likely to be valid in their findings.

The number of live births in specified quarters of the year were taken from the National Records of Scotland (NRS) live birth registrations and were based on the date of birth, not the date of registration of the birth. However, for Outcome 2 (immunisation coverage), the number of live births was taken from the Scottish Immunisation Recall System (SIRS) as longitudinal data at individual child level were required to construct this outcome measure.

Data for the outcome measures were drawn from several sources, including the Child Health Surveillance Programme (pre-school) (CHSP-PS) May 2020 (Public Health Scotland); the Scottish Immunisation Recall System; Management Information and Dental Accounting (MIDAS), PHS; the Scottish Morbidity Record (SMR01), ISD; and the Scottish Exchange of Data (ScotXed); see Table 3 for specific details.

In order to try to align the live birth data with the child health data for the 27-30 month review, the postcode at birth was used to derive the NHS health board of residence, wherever possible. However, if this postcode was missing, then the postcode on the CHI in the quarter following birth was used instead.

Analysis

Datasets

Data have been provided in sequential three-month units (i.e. quarters). A birth cohort is defined as the number of live births within a specified quarter. The fully unexposed group consists of 21 birth cohorts, and includes all live births in every quarter between 1 January 2011 and 31 March 2016.

Outcome groups 1-7 (14 outcomes in all) were analysed using data for the whole of Scotland, and data were also stratified according to Scottish Index of Multiple Deprivation (SIMD) quintiles, based on the child's home postcode. A set of supplementary data tables for each outcome are provided for transparency; in these, data are displayed for (i) the whole of Scotland, (ii) stratified by SIMD quintiles, and (iii) stratified by individual NHS health boards of child's residence.

It was not possible to examine Outcome groups 8 and 9 by level of area deprivation, since data stratified by SIMD quintile were not available for child safety and protection interventions, due to the potential risk of identification.

Scotland-level data

As was described earlier (also see Table 2), children born in Orkney in the final two quarters of the unexposed time period, and children born in Shetland and Tayside in the final four quarters of that period, may have been exposed to some elements of the UHVP; therefore these children were removed from the Scotland datafile for Outcome groups 1-7.

However, as explained above, data for Outcome groups 8 and 9, which relate to Child Protection and Looked after children, were supplied by councils rather than health boards. Only a very small number of children have positive outcomes for these outcome measures. If fewer than five children have been recorded as being on the Child Protection Reigister (CPR) or having Looked After Child (LAC) status in any council area, the actual data value is suppressed, due to the potential risk of disclosure and to help maintain patient confidentiality. For the boards where there may have been children partially exposed to the UHVP (Angus, Dundee, and Perth (that make up NHS Tayside) and NHS Shetland and NHS Orkney), the data for the potentially partially exposed children have been included in this part of the analysis, since it is not possible to ascertain the precise number of children who had a positive outcome. It is not anticipated that inclusion in this cohort will impact on the overall findings, due to the overall small sample size.

SIMD data

SIMD data stratified by health board were not available. Therefore, when examining SIMD graphs, a small number of children in the final year of follow-up (from April 2015 to March 2016) may have been exposed to some elements of the UHVP: that is, approximately 4,256 of the 55,258 children born during this period (7.7%) may have been partially exposed to the UHVP.

At a Scotland level, these data were stratified by SIMD into six groups: the SIMD quintiles, ranging from SIMD 1 (most deprived) to SIMD 5 (least deprived), and a group 'SIMD unknown'. The 'SIMD unknown' group consisted of children whose postcode was missing and also children whose postcodes do not map to an SIMD quintile (for example, new housing developments). For outcomes recorded on a child health review form, it is possible that the child's postcode was missing from the form and thus the child's SIMD quintile is unknown. Unfortunately it is not possible to give a precise estimate of the number of children affected, since the data on live births have been accessed from a different source (NRS) to the data relating to the relevant child health review (CHSP-PS).

For Outcome 2 (Complete coverage of universal primary and end infancy immunisations by second birthday), longitudinal data have been provided and therefore it is possible to identify the number of children who have not been included in the deprivation analysis and the percentage of SIMD-unknown children who have received full immunisation by their second birthday.

For the other six outcome groups for which SIMD data are available, it is not possible to estimate the number of children with 'SIMD unknown' status. The dataset for each outcome group consists of the merging of unlinked data from two sources: a datafile containing NRS live births is merged with data requested from the CHSP-PS or MIDAS or SMR01 into a single datafile. For these six outcome groups, the number of children with unknown SIMD has been calculated as a percentage of the relevant data for the whole of Scotland. Taking developmental-concerns outcomes as an example, the following percentages have been calculated for Outcome 5a:

Percentage of 27-30 month reviews where child SIMD is unknown = (Number of 27-30 month reviews where child SIMD is unknown / Total number of 27-30 month reviews reported for Scotland) x 100%

Percentage with complete data for developmental domains, where child SIMD is unknown = (Number of SIMD-unknown children with complete data for developmental domains / Total number of children with complete data for developmental domains for Scotland) x 100%

Percentage without a developmental concern, but some domains incomplete or missing, where child SIMD is unknown = (Number of SIMD-unknown children without a developmental concern, but some domains incomplete or missing / Total number of children without a developmental concern, but some domains incomplete or missing, recorded for Scotland) x 100%

Percentage with any developmental concern (Outcome 5a), where child SIMD is unknown = (Number of SIMD-unknown children with any developmental concern / Total number of children with any developmental concern, recorded for Scotland) x 100%

Detail of the outcome measures used is included in Appendix 1. It was not possible to obtain data stratified by SIMD group for the two child protection outcome groups (Outcome group 8: child placed on the CPR by 3rd birthday, and Outcome 9: Looked after child status by 3rd birthday), due to the small numbers of children with a child protection or looked after child episode and the possibility of identification.