Family Nurse Partnership evaluation: methods and process

4. Analysis

4.1. Power calculation 

The previous ‘Evaluability Assessment’ estimated around 3000 births in FNP cohorts between 2010 and 2015 and around 6000 in the Controls (7). This large sample size would permit very precise estimation of overall intervention effects for a primary or co-primary outcomes. However as, there is no pre-specified prioritization of the 34 short- or medium-term outcomes for the main report, no power calculation is necesary.

Section 4.2 will cover analyses relating to the first three objectives and will be reported here in this document. 

4.2. Identification of FNP Clients and Controls

4.2.1. FNP Clients 

The number of FNP Clients identified and received from FNP SIS and matched to the SMR02 dataset by eDRIS will be reported using a flow chart. Additional checks will be made on the eligibility of FNP Clients. The SMR02 fields will be compared against the fields recorded in the FNP SIS for the FNP Clients (Table 7). This will allow us to measure the robustness of these fields used to identify the Controls.

Table 7: Maternal variables required from FNP and SMR02 datasets assessing eligibility criteria 

4.2.1. Controls

After eDRIS have identified all eligible Controls based on the FNP eligibility criteria (Table 2), and recruiting periods (Table 1), several checks will be made on the data similar to those described for FNP Clients. Based on the SMR02 fields, we will check that Controls are eligible for FNP. 

A well-conducted RCT would provide a precise, unbiased estimate of the effectiveness of FNP due to the process of treatment allocation via randomisation. In a natural experiment, to enable an unbiased comparison of cases and controls, measured risk factors associated with outcomes (known as covariates) should be sufficiently similar (and thus balanced) for both exposure groups (i.e. FNP Clients and Controls). 

As the potential Controls would have been eligible for enrolment on FNP during a period of active recruitment, they are already a more homogenous comparison population. It is likely, therefore, that the controls will be sufficiently similar to FNP Clients. However a possible threat to an unbiased comparison is the enrolment of mothers into FNP on criteria other than area, age, and parity (for example, an additional subjective judgement of clinical need). If mothers who were approached but not enrolled to participate in FNP differ from FNP clients in a manner that is associated with variation in outcomes, comparisons with all eligible Controls may under- or overestimate any effect of FNP. 

One way to address this is to match the population on further maternal characteristics (such as smoking, co-morbidities, etc.) as it may provide a more valid estimate of effect because only women with similar observed characteristics are included, thus the results are more comparable. The disadvantages of this matching would be that not all cases would be matched to controls which would risk the exclusion of cases and reduce the sample size. It also only gives a solution to short-term outcomes where the cohort is maximized; matching would provide balance based on the whole cohort but as the sample reduces over time due to loss to follow up, the characteristics of the sample will also change and risk of imbalance is again present. 

One of the approaches to matching of controls considered was propensity score matching (PSM) where the ability to examine, quantify and balance the recorded characteristics between the exposed and non-exposed groups can be easily implemented and a large number of measureable covariates can be adjusted for. Key to this method is that the propensity score (the predicted probability of enrolment obtained via regressing FNP enrolment on all available covariates) can be generated without sight of outcome, eliminating any possible bias in selecting the best match that provides the most favourable result (akin to randomisation and assessing outcome measures in a trial). However, PSM is sensitive to missing data and to be able to perform matching on imputed datasets, outcome data on all eligible Controls are required to obtain a pooled intervention effect over imputed datasets (12). 

Therefore, given the possibility of a homogeneous comparison group and considerations such as the feasibility of gaining approval for the additional data required for PSM, a pragmatic approach was adopted to use all available controls. Using all controls, results will be more generalizable, and will result in higher power. 

4.2.2. Descriptive analysis: FNP Clients and Controls

Measurable pre-recruitment/at booking maternal demographics and socioeconomic covariates associated with the FNP enrolment and outcomes were decided in advance (Table 8). Covariates should be variables that are not affected by exposure and measured before recruitment into FNP. The maternal characteristics will be described in the FNP Clients and all Controls using summary statistics (e.g. N (%), mean (standard deviation (SD))).  In addition key summary characteristics of women offered but not enrolled in FNP or not offered FNP in an FNP recruiting period will be supplied by eDRIS using a subset of characteristics from Table 8 (deprivation quintile, ethnic group, age at LMP, gestation at antenatal booking, BMI at booking, history of smoking, drug use, previous pregnancies).

Table 8: Maternal baseline characteristics at (or before) date of antenatal booking/enrolment

¹ Appendix 3 for BNF codes used to define dispensing of medications

4.2.3. Main analyses

With no primary outcome, equal importance will be given to each short and medium term outcome. All comparative analyses will be pre-specified and conducted on an intention to treat (ITT) basis. ITT in this study means that the analysis will include everyone who started the programme, according to their original ‘allocation’. This means that the intervention group will be all women enrolled in FNP regardless of the treatment (intervention) they actually received.

All outcomes will be described by group (FNP Clients or Controls) and using summary statistics such as the number per group (percentage of all group), mean (alongside standard deviation) or median (alongside 25^th to 75^th centiles). All analyses will compare outcomes between women receiving FNP and those in the control group. Multilevel regression models will be used to allow for potential clustering of outcomes within NHS HBs and with the FNP teams/cohort. This means the analysis will take into account that possibility that similar outcomes occur because women live within the same area or receive support within the same team/cohort. 

Depending on the outcome, the effect of FNP (the difference for an outcome between FNP Clients and Controls) will be estimated and presented alongside a 95% confidence interval (CI) and p-value. This will show the certainty of the effect estimates. A wide confidence interval means that the difference between Cases and Controls could be much higher or lower than the difference estimated. In contrast, a small confidence interval provides a much more precise estimate of the real difference. A full technical description of the statistical analysis plan is in Appendix 4. 

Before sight of any outcome data, a detailed statistical analysis plan (SAP) will be written and signed off by the co-lead for the project. The reporting and presentation of results will be in accordance with the GUILD, STROBE, RECORD and TREND guidelines to ensure the comprehensive reporting of this evaluation (13–16). The statistical packages SPSS and Stata will be used for all analyses (17,18). We will adhere to the NSS Statistical Disclosure Control protocol (11,19).