Scotland's Baby Box: evaluability assessment

Chapter 2: The Evaluability Assessment process

Evaluability Assessment ( EA) is a systematic, collaborative approach to the planning of an evaluation project. It involves engaging stakeholders, clarifying intervention goals, developing a theory of change (logic model) and deciding whether and how a useful evaluation could be carried out at a reasonable cost.

This EA was conducted on behalf of the Scottish Government by the Evaluability Assessment Collaborative ( EAC). The EAC is a consortium of researchers from the Medical Research Council/ Chief Scientist Office Social and Public Health Sciences Unit, University of Glasgow and the Scottish Collaboration for Public Health Research and Policy, University of Edinburgh. The EA comprised three workshops, held between January and March 2018.

A wide range of stakeholders were identified and invited to take part in the workshops. They included staff from the Scottish Government, Midwifery, third sector and charitable organisations, and academic experts in safe sleeping and infant mortality. The full list of the workshop participants is provided in Appendix 1.

Workshop 1 (24 January 2018)

The first workshop began with an introduction to the EA process by the EAC. This was followed by a brief policy overview of the Baby Box by Scottish Government policy leads, which set out the rationale and purpose of the Baby Box initiative. The EAC then presented the logic model, which had been developed as part of the pilot evaluation of the Baby Box. Stakeholders agreed that it would be more useful to review and revise the existing logic model, rather than develop a new one, to reflect current operations and learning.

Stakeholders were divided into three sub-groups and were asked to discuss whether all the important inputs, activities and outcomes were captured in the logic model and to suggest other relevant items. The findings from each sub-group were then considered by the main group. The main points that emerged are outlined below.

Short term outcomes

Stakeholders felt that training and education of the workforce to ensure that they understand the contents, purpose and use of the Box was a crucial short term outcome. Stakeholders also agreed that the initial outcome of "engagement with maternity services" should be changed to "attempt to engage with wider services," to reflect the fact that the Baby Box may initiate engagement with services other than maternity services.

Stakeholders argued that seeking support was an important step in its own right, and should be measured whether or not support was obtained. Stakeholders also felt that the original long term outcome of "cultural shift in attitudes to equality – we all started life sleeping in a Box", should be measured across the short and medium term, and reworded as "shared understanding of a society that values and supports all children and families".

Figure 1 shows the revised logic model, including the short term outcomes that were finally included.

Figure 1. Scotland's Baby Box – Logic model

Assumptions, synergies and risks:
Assumptions: Box adds to (rather than duplicating) parent's own resources (material and knowledge/information); Baby Boxes operating in wider package of maternity/family benefits – maternity grants, BSG, increased ELC provision.
Synergies: Operation of a wider package of maternity/family benefits – maternity grants, BSG, increased ELC provision.
Risks: Change in retailer behaviour (possible unintended consequences of universal provision of new born essential); media responses to Box (could undercut perceived usefulness/value and/or create stigma around accepting if not on board).
* Risk and Positive behaviours: breastfeeding, sleeping practice, health seeking behaviour and play/attachment.
+ The focus of the evaluation will be on the short and medium term outcomes, but the longterm outcomes should be passively monitored (as agreed upon by stakeholders at the first workshop on 24/1/2018).

Medium term outcomes

A key point raised during discussion was that the medium term outcomes should include reduction of specific risk behaviours as well as increasing positive behaviours as depicted by the original logic model. The medium term outcomes that stakeholders agreed to include in the revised logic model are shown in Figure 1.

Long term outcomes

The original logic model listed a number of longer term outcomes including lower infant mortality rates and reduced inequalities in infant mortality, improved parent child attachment and reduced inequalities in early learning outcomes. Participants noted that there were many other initiatives, such as the Universal Health Visiting Pathway and Best Start Grants, which might affect such outcomes.

It was also acknowledged that some relevant outcomes (such as serious adverse events) might be rare, or change only gradually following the introduction of the Baby Box. Following discussion, it was agreed that it would be difficult to attribute change in rare or longer term outcomes to the Baby Box and that the evaluation should focus on short to medium term outcomes, which could be more directly attributable to the Box.

It was also agreed that the focus of the evaluation should be on the Baby Box as a mechanism for engagement and education rather than as providing a safe sleeping space for infants. However, it was recognised that longer term child health, wellbeing and development outcomes should be monitored to identify possible impacts on health inequalities or other unintended consequences.

The final long term outcomes that stakeholders agreed should be monitored are outlined in Figure 1.

Workshop 2 (7 February 2018)

The main purpose of the second workshop was to identify potential primary and secondary data sources that could be used to measure short, medium and longterm outcomes.

Stakeholders considered the information collected during routine postnatal assessments in relation to the outcomes within the revised logic model. It was agreed that Health Visitors' assessment forms and other routinely collected data could be used to measure a range of short and medium term outcomes, but that additional primary data would be required to capture outcomes such as practitioners' and parents' understandings of the Baby Box and sustained engagement with services. In terms of longer term outcomes, such as reduced inequalities in infant mortality, it was recognised that National Records of Scotland ( NRS) data could be used, supplemented by hospital admissions data to monitor other long term outcomes.

Workshop 3 (16 March 2018)

Workshop 3 involved identification and discussion of potential evaluation options, and assessment of their strengths and weaknesses.

It was noted that nationwide implementation of the Baby Box scheme ruled out a randomised trial of the scheme as a whole, though variations in provision could in principle still be trialled. Options considered included qualitative research with parents and practitioners, possibly supplemented by a quantitative survey, analysis of levels and trends in outcomes using routinely collected information to identify changes associated with the introduction of the Baby Box, and a prospective study of births in Scotland and a control area elsewhere in the UK, to identify differences in sleeping practices and other risk behaviours associated with receipt of the Box.

A number of constraints on evaluation design were noted. Only 25% of parents registered for a Baby Box provided consent to be contacted for research. This is not a problem for identifying a qualitative sample, but does create problems in using the database for survey sampling. There were also uncertainties about the suitability of the data recorded on the Child Health Systems Programme for use in an outcomes evaluation, for example where there are currently unreported elements being considered.

Evaluation options are set out and appraised in Chapter 3.