Scotland's Baby Box: evaluability assessment

Executive summary

Background

This report presents the Evaluability Assessment ( EA) of Scotland's Baby Box scheme. The Baby Box scheme was piloted in Clackmannanshire and Orkney, Scotland from January to March 2017. It was then fully rolled out across Scotland in August 2017. Baby Boxes are new to Scotland and there is little available evaluation evidence to date on their impact from other countries. The Scottish Government is committed to evaluating the impact of the Baby Box initiative in Scotland and requested an EA to seek the advice of independent experts in identifying priorities for, and approaches to, evaluation.

The Evaluability Assessment process

Evaluability Assessment is a systematic, collaborative approach to the planning of an evaluation that involves engaging stakeholders, clarifying intervention goals, developing a theory of change or a logic model and deciding whether and how a useful evaluation could be carried out at a reasonable cost. The EA was conducted over the course of three workshops, between January and March 2018. Stakeholders who participated in the workshops were from the Midwifery profession, Scottish Government, third sector and charitable organisations, and academic researchers. The EA was conducted by the Evaluability Assessment Collaborative ( EAC) on behalf of the Scottish Government.

Evaluation options

Stakeholders agreed that the evaluation should include both outcome and process evaluation elements, to identify the effect of the Baby Box and to explore the processes and mechanisms by which effects were achieved. It was agreed that the evaluation should focus on short to medium term outcomes, which could be more readily attributed to the Baby Box. It was recognised that the Baby Box may have effects on important rare or longer term child health, wellbeing and development outcomes, but these were likely also to be impacted by a wide range of other policies and external factors, such as the Universal Health Visiting Pathway and Best Start Grant, making it problematic to identify and separate the impact of the Baby Box, as is the case with evaluation of other specific initiatives. It was agreed that rare or longer term outcomes should be monitored to identify possible effects on health and wellbeing or unintended consequences that might indicate a need for further research.

It was agreed that a process evaluation should be carried out. This might use qualitative interviews with practitioners and Baby Box recipients to examine implementation, mechanisms of impact, and contextual factors. This could be supplemented by case note reviews and a quantitative survey of recipients.

Evaluation of outcomes could potentially use data collected through the Child Health Systems Programme ( CHSP) Pre-School system of assessments carried out by Health Visitors. Assessments are carried out at specified intervals following the birth of a child, and collect data on a range of relevant outcomes, including infant feeding, sleeping position, development and referrals to other services. The system is intended to ensure consistent recording of outcomes, but its primary purpose is to support the delivery of services. Therefore, validation work would be required to assess its suitability for evaluation uses.

Alternatively, primary data could be collected from a prospective cohort of births in Scotland and a comparison area (or areas) in England or Wales. This would allow much greater flexibility in deciding what data could be collected but would involve substantial methodological challenges and could be considerably more expensive than the other options.

In the absence of previous evaluations of Baby Box interventions delivered on a whole population basis, and the difficulties identified with the options for evaluating impacts, a phased approach to commissioning evaluation may be advisable. A substantial initial phase of feasibility testing and development work would be required prior to final decisions about the design of a substantive process and outcome evaluation.