Family Nurse Partnership evaluation: methods and process

6. Discussion

6.1. Establishing the evaluation framework 

We have established the framework for evaluating FNP using data linkage and routine Scottish data. The required governance approvals were obtained and eDRIS successfully received and mapped FNP Clients to the SMR02 with little loss of records. The majority of women excluded were due to no CHI number or delivery record. The quality of the fields from the SMR02 used to identify the potential control sample was found to be high with few missing values. Controls were successfully identified using the SMR02 dataset. Both FNP Clients and Controls were linked to health data and descriptive data for both groups were summarised. Education and social care data (e.g. women ever in care or need) were not received to examine at baseline. Therefore, the impact of these data and possible imbalances will be examined once the remainder of the data is received. Any such imbalances will be adjusted for in the analysis. eDRIS also successfully identified a potential control sample of eligible women who would meet criteria for FNP but were not enrolled to receive the FNP programme. Approval was not gained from Public Benefit and Privacy Panel to use this group. Even though these women were not FNP Clients and would not be included in the evaluation, it was vital to understand how they might differ from the women enrolled in FNP. As such, it is important to take ethnicity into account during analysis since the proportion of white women in this group were comparable to the Controls.  

We were unable to match FNP Clients to Controls using propensity score matching methods due to the proportion of missing data. For propensity scores to have been generated and used for matching, outcome data would have had to have been sought beforehand. This was outside of the scope of this study. Instead, the cohort of eligible Controls identified in the one-year pre and post recruitment and within interval periods where recruitment ceased will be used. This still maximizes the study cohort of mothers and children (8,221 and 8,270 respectively) and will provide results that will be more generalisable and result in higher statistical power. Any imbalances in important confounders will be adjusted for by modelling. 

6.2. Scope of evaluation outcomes

The assessment of effectiveness in the evaluation is limited to outcomes available from routinely collected data. The outcomes for this study have been selected by matching routinely collected administrative data to the Scottish FNP logic model based on the underlying programme theory. FNP has three main aims in Scotland: 1) to improve pregnancy outcomes; 2) to improve child health and development; and 3) to improve parents’ economic self-sufficiency. The breadth of these aims is intended to capture the complexity of the intervention. Translated into outcomes within the logic model, these aims were matched to the available routine data. This resulted in data matches or proxy data matches for around 50% of the outcomes detailed within the model. 

Therefore, the included outcomes have been selected on the basis that they are outcomes FNP aims to influence and for which there is routine data, rather than a set of specific outcomes where research indicates the most significant contribution. Where there is a clear hypothesis for why FNP will contribute to a specific outcome this has been described. The remaining outcomes will be regarded as exploratory outcomes, pre-specified but not hypothesised. In addition, special consideration will be given to outcomes where bias may have been incorporated in their reporting. An example of this is child health assessments up to 24 months, which are carried out by different health professionals depending upon which group (FNP clients or Controls) women are in. 

There are no primary outcomes selected for this evaluation. This approach is consistent with some previous trials of FNP (for example, the original Elmira trial) and with other evaluations of home visiting programmes (20), where impact is expected to arise across multiple domains and over multiple time-points. A consequence of this approach is that the large number of statistical tests performed will increase the risk of finding a significant result by chance (false-positive error). Given a 0.05 alpha there is an 83% (1-0.95³⁴) chance that at least one of these tests are statistically significant by chance when the conclusion is not true in the population. Recently, James Heckman and colleagues re-assessed the findings of the Memphis trial of NFP using a ‘step-down’ approach to address the challenge of multiple significance testing (21). Although in Heckman’s re-analysis fewer treatment effects survived corrections for multiple hypothesis testing they observed strong effects for boys, sustained until age 12. For individual studies, other correction methods such as Bonferroni have been suggested but are likely to prove overly conservative and risk the possibility of Type II error (i.e. incorrectly concluding no effect when one does actually exist). 

The numbers of available FNP Clients are in the region reported in the Evaluability Assessment (estimated around 3000 births in FNP cohorts between 2010 and 2015) but the number of Controls are lower than expected (7). One reason for this might be that Controls were defined as ‘First births to mothers aged <20 at conception’ where as our cohort of Controls were further restricted to gestational age and previous births. They also included projected numbers for 2013/14-2014/15, based on an assumption of a 6% annual fall in the number of such births. Nevertheless we have an adequate ratio of FNP Clients to Controls for the evaluation.  

As currently, we haven’t performed any matching to EAS, we do not know how well it will be done or who will be matched. As already noted, the matching of Education data to the Population Spine presents a 50% chance of matching error for twins. For this evaluation, this means that outcomes from Education and Health on an individual level may not relate to the correct twin. However, on a population level, this should not impact on the overall results as both would have recieved FNP or not. 

6.3. Legal & Ethical considerations

This study has been considered a service evaluation as it is limited to using data obtained as part of usual care. This was confirmed by the South East Scotland Research Ethics Service. The principal legal consideration is of unintentional identification of individuals. Such a risk could be increased through the combination of clinical and socio-demographic attributes from a single or multiple datasets[41].This risk is managed through the de-identification of sensitive and personally identifiable data items before matching and before being made available for analysis as well as the disclosure controls placed on all outputs from the safe haven. There are various governance and contractual requirements placed on this study, many of which were identified and required by the PBPP and/or EAS panel before final approval. All researchers have completed the information governance training required by eDRIS to evidence their “approved researcher” status. Thirty-three data sharing and/or data processing agreements were set up between HBs, NSS, NRS, EAS and SG to allow the transfer, processing and storage of data for this study. The study has a steering committee to provide independent oversight of the study, this was set up by the study team and their remit is to provide independent advice on the study including ensuring retaining scientific robustness.