Scotland's Baby Box: evaluability assessment

Chapter 3: Evaluation options

This chapter describes options for process and outcome evaluation identified during the workshops. Below each evaluation option, a table illustrates which outcomes would be addressed ( tables 1- 4). Pros and cons for each option are summarised in Table 5.

Option 0: No additional evaluation

Management information systems collect data on Baby Box registrations, including some information on applicants' demographic and geographical characteristics. No information is routinely collected through the registration process on outcomes or the mechanisms by which they are achieved. Without additional data gathering and analysis, evaluation would be limited to aspects of the registration and delivery process, such as analysis of regional or demographic variation in take up rates.

Option 1: Process evaluation

Stakeholders agreed that any approach adopted to evaluate the Baby Box should include a process evaluation in order to understand how the policy is implemented and how it may lead to outcomes.

Process evaluation recognises that intervention (in this case Scotland's Baby Box) outcomes depend on the interaction between the intervention and its context, and aims to understand the functioning of an intervention by examining implementation, mechanisms of impact, and contextual factors such as the characteristics of the target population and other services available to recipients of the intervention (Moore et al., 2015).

Understanding the role of context is particularly important for the evaluation of the Baby Box because other programmes and services, which also aim to improve health and wellbeing outcomes for children and families in Scotland, such as the Universal Health Visiting Pathway and Best Start Grant, are being introduced concurrently. Including a process evaluation alongside an outcomes evaluation can both enhance understanding of how outcomes are achieved, and contribute to improving design and implementation of the Baby Box.

A process evaluation can employ a range of data and methods, including qualitative interviews, focus groups, case note review, and quantitative surveys. Qualitative approaches use semi or unstructured interviews or focus groups, and samples that are designed to include participants with a range of relevant characteristics, rather than to be strictly representative of the whole population. Quantitative surveys use structured methods of data gathering, and representative samples.

1.1 Qualitative interviews and focus groups with parents and practitioners

Interviews and focus groups with parents and practitioners could be used to explore a range of short and medium term outcomes. For example, interviews and focus groups with parents could explore whether parents' understanding of what is important for their baby had been influenced by receiving the Baby Box, and whether they had consciously changed their plans or spending decisions in relation to new-born essentials.

They could also explore whether parents' awareness and understanding of risk and positive behaviours, such as breastfeeding and sleeping practices, had been influenced by receiving a Baby Box, whether it had encouraged them to engage with other services, and what kind of message(s) they thought the Baby Box projected as a feature of Government policy towards families with new babies.

Qualitative research of this kind will provide a deeper understanding of the context and processes behind particular impacts of the Baby Box but it will not provide generalisable information or give an indication of the prevalence of particular impacts.

Parents could be purposively sampled from those providing consent to be contacted for research. Although only 25% of parents registered agree to be contacted, the numbers should be sufficient to stratify the sample to ensure inclusion of a range of NHS Boards, different levels of the urban/rural classification and levels of deprivation. Further stratifying by the age of the baby would allow changes over time in sleeping practices, perceptions of the Box, and understanding of risk to be explored.

Interviews or focus groups with practitioners could be used to explore their understandings of the purpose of the Baby Box, how they perceived its influence on their interaction with parents, whether they felt able and confident to discuss the contents and purpose of the Baby Box with parents, and whether they felt it provided an opportunity to engage parents with other services. Sampling of practitioners should be stratified to match the sampling of parents so as to allow triangulation of the results of the parent and practitioner interviews.

Table 1 outlines the outcomes identified in the logic model that could be addressed by this option.

Table 1. Outcomes, example questions and methods for option 1.1

Short term outcomes	Examples of questions	Possible data sources
Reduced expenditure on new-born essentials	How much is the Baby Box saving families, especially low-income families, on new-born essentials?	Parent interviews or focus groups.
Parents understand and use Box and contents	Has the Baby Box changed parents' perceptions about what is essential for a baby, including safety, positive interactions?	Parent interviews or focus groups.
Increased parents' and workforce understanding of risk and positive behaviours	Has the Baby Box improved parents' understanding of risk and positive behaviours, such as breastfeeding and sleeping practices? Has the Baby Box improved practitioners' understanding of risk and positive behaviours?	Parent interviews or focus groups. Midwife/Health visitor focus groups or interviews.
Workforce understands contents and purpose of box	Are practitioners confident, skilled and competent to discuss contents and purpose of Box with parents?	Midwife/Health visitor focus groups or interviews.
Attempt to engage with wider services	Have referrals to universal and wider services changed since the Baby Box? Is the Baby Box offering new opportunities for practitioners to engage with parents stemming from the information pack provided in the Box? Is the Baby Box offering new opportunities to identify families who are unlikely to engage with services?	Practitioner interviews or focus groups with practitioners. Case note review.
Medium term outcomes	Examples of questions	Possible data sources
Shared understanding of a society that values and supports all children	Does the Baby Box change people's perceptions about universal benefits? Has uptake of the Baby Box changed over time; is this a reflection of a cultural change?	Parent interviews or focus groups. Parent interviews or focus groups. Case note review.
Increased positive behaviors and reduced risk behaviors	Are parents increasingly engaging in positive behaviours and reducing risk behaviours?	Parent interviews or focus groups.
Sustained engagement with wider services	Is there greater access to and uptake of services, where information is provided, to improve health and wellbeing for parents?	Health Visitor interviews or focus groups. Case note review. Practitioners in other services interviews or focus groups.

1.2 Quantitative surveys of parents and practitioners

(a) A survey of parents sampled from the registration database (of those that have consented to further contact for research) could be conducted to examine short and medium term outcomes, such as expenditure on new-born essentials, the extent to which particular items in the Box are used and the prevalence of risk or protective behaviours. This will provide quantitative information that is generalisable to the wider population. As noted, the proportion of parents who currently consent to research is about 25%. As those who consent are unlikely to be representative of all parents registered for the Baby Box, it will be necessary to stratify the sample to ensure that some groups (for example more affluent or better educated parents) are not over-represented. Disproportionate sampling and any additional imbalance in response could be corrected by applying appropriate weights during analysis, with some loss of statistical efficiency.

(b) Scottish Government officials mentioned that formal training on the Baby Box for health professionals could be considered in the future. A baseline survey could be conducted prior to the introduction of a formal training programme. Once the training has been implemented, existing reporting mechanisms could be used to monitor participation and a further survey carried out with practitioners who have received the training to ascertain whether their knowledge, skills and competencies to discuss the purpose, contents and use of the Box with parents had improved.

Table 2 outlines the outcomes identified in the logic model that could be addressed by this option.

Table 2. Outcomes, example questions and methods for option 1.2

Short term outcomes	Examples of questions	Possible data sources
Reduced expenditure on new-born essentials	How much is the Baby Box saving families, especially low-income families on new-born essentials?	Surveys of parents.
Increased parents' and workforce understanding of risk and positive behaviours	Has the Baby Box improved parents' understanding of risk and positive behaviours, such as breastfeeding and sleeping practices? Has the Baby Box improved practitioners understanding of risk and positive behaviours?	Surveys of parents. Midwives'/Health visitors' self-reported knowledge, skills and competencies –surveys.
Workforce understands contents and purpose of Box	Are practitioners confident, skilled and competent to discuss contents and purpose of Box with parents?	Midwives'/Health Visitors' self-reported knowledge, skills and competencies – surveys.
Attempt to engage with wider services	Have referrals to universal and wider services changed since the Baby Box? Is the Baby Box offering new opportunities for practitioners to engage with parents? Is the Baby Box offering new opportunities to identify families who are unlikely to engage with services?	Midwives'/Health Visitors' self-reported knowledge, skills and competencies- surveys.
Medium term outcomes	Examples of questions	Possible data sources
Shared understanding of a society that values and supports all children	Does the Baby Box change peoples' perceptions about universal benefits? Has uptake of the Baby Box changed over time; is this a reflection of a cultural change?	Surveys with parents.
Reduced inequalities in access to new-born essentials	Is there improved and more equitable access to new-born essentials?	Survey of parents (comparing across SES groups).
Increased positive behaviors and reduced risk behaviors	Are parents increasingly engaging in positive behaviours and reducing risk behaviours?	Surveys of parents.
Sustained engagement with wider services	Is there greater access to and uptake of services to improve health and wellbeing for parents?	Surveys of parents.

Option 2: Outcome evaluation using routinely collected data

The Baby Box is expected to achieve a wide range of outcomes, over a range of timescales, making evaluation of its impact using routinely collected data challenging. The Baby Box has already been implemented on a universal basis across the whole of Scotland, and so an experimental trial of the scheme as a whole is no longer a feasible or ethical option. We focus in this section on approaches that can be applied retrospectively using routinely collected data. Under option 3 below we consider a research design that involves prospective data gathering but in the context of an observational rather than experimental study design.

As noted above, stakeholders agreed that the focus for any outcome evaluation should be on short or medium term outcomes, given the difficulty of attributing longer term outcomes to the Baby Box in the context of other interventions for babies and parents such as the Best Start Grant and the Universal Health Visiting Pathway. It was also agreed that some rare or longer term outcomes related to child and maternal health, including sudden unexpected death in infancy, should be monitored only (rather than being part of the evaluation) to check that there were no adverse changes in levels, trends or socio-economic patterning of these outcomes that might require further scrutiny. Option 2 therefore considers how routinely collected data could be used to measure change in short/medium term outcomes, such as sleeping practices, development outcomes, referrals to other services, etc.

One possible approach to measuring the impact of the Baby Box would be to use interrupted time series methods to analyse changes in the level and trend of routinely monitored child health outcomes before and after the introduction of the Baby Box. Information on the health of all babies and young children in Scotland is collected via the Child Health Systems Programme ( CHSP) Pre-School system of assessments. The present system has largely been in place since 2013, though the forms currently in use were introduced in 2016, with further changes in April 2017 including the introduction of additional assessments at 13-15 months and 4-5 years as part of the Universal Health Visitor Pathway. Health Visitors complete standardised forms at the first post-partum visit around 10 days following a birth, 6-8 weeks post birth, and when the child is 13-15 months, 27-30 months and 4-5 years.

The assessments collect a range of information relevant to the evaluation of the Baby Box. At the first post-partum visit, information is collected on smokers in the household and infant feeding (at birth, hospital discharge and current method). The 6-8 week review collects information on feeding (breast, bottle or both); parental concerns (feeding, appearance, behaviour, hearing, eyes, sleeping, movement, illness, crying, weight gain, and other); development (gross motor, hearing and communication, vision and social awareness); physical measures; diagnoses and concerns related to the child's health, development and wellbeing; sleeping position (prone, supine and side); and referrals to other services. Subsequent assessments collect information on feeding (13-15 month only); development; physical measures; diagnoses/concerns related to the child's health, development and wellbeing; and referrals to other services.

CHSP Pre-School system of assessments information is a potentially valuable resource for evaluating the effect of the Baby Box on infant feeding, sleeping position, development, and parents' engagement with services but there are significant limitations that need to be taken into account. Implementation is the responsibility of NHS Boards, and practice varies within and between Boards. Although national guidance for the 27-30 month assessment was issued in April 2017 recommending that all Boards use the same development assessment tool (the Ages and Stages Questionnaire, version 3), practice before then differed widely ( ISD 2018a, ISD n.d.). ISD note that 'in some NHS Boards a phased implementation occurred after the system was adopted. Therefore, caution should be taken when interpreting data around the implementation period. In addition, many data items are not mandatory and recording practices vary between NHS Boards and individual health professionals. Therefore, not all data items on the review forms can be used for analysis ( ISD 2018b).

While some of the information, particularly the items used in national reporting such as breastfeeding, is believed to be of high quality ( ISD 2017), other items (including sleeping position) that are not routinely analysed have not been validated, and validation work would be needed to ascertain whether they could be used for evaluation purposes.

The child development data is potentially useful but there may be a need to stratify analyses according to the tools used, or to focus on a subset of Boards using particular development assessment tools over a period of years. Implementation of the 13-15 month assessment overlapped with implementation of the Baby Box, so only Boards that introduced the additional assessment rapidly could provide data both pre and post implementation of the Baby Box, and the length of the pre-implementation series would be limited.

If a consistent monthly series of data could be obtained from the first post-partum visit and 6-8 week assessments for relevant outcomes such as feeding, sleeping position and development, interrupted time series methods could be used to identify changes in the levels and/or trends in those outcomes associated with the introduction of the Baby Box. This method would control for pre-intervention trends in the outcomes of interest, but not for the effects of other events that occur around the same time as the Baby Box was implemented, such as other changes in ante- or post-natal care.

Interrupted time series approaches are well-suited to evaluating policies that are implemented at a specific point in time, such as the Baby Box. The analyses could use aggregate data ( e.g. monthly counts) rather than individual data, and would not require linkage of Baby Box registrations with the CHSP Pre-School system of assessments data. However, it would be necessary to use individual level data for analyses stratified by parental characteristics such as age and deprivation scores. Without linkage to application data, the analyses would provide 'intention to treat' ( ITT) estimates, identifying the effect of the Baby Box on all those eligible, rather than the effect on recipients.

As the take-up of the Baby Box is high (83% in June 2018) but not complete, the ITT and 'on treatment' analyses may differ. Similar methods could eventually be applied to the 27-30 month assessment data but attributing changes in outcomes at that stage to the Baby Box would be problematic for the reasons noted above, and a post-intervention series of data would not be available until late 2019.

Table 3 outlines the outcomes identified in the logic model that could be addressed by this option.

Table 3. Outcomes, example questions and methods for option 2

Short term outcomes	Examples of questions	Possible data sources
Attempt to engage with wider services	Have referrals to universal and wider services changed since the Baby Box? Is the Baby Box offering new opportunities for practitioners to engage with parents? Is the Baby Box offering new opportunities to identify families who are unlikely to engage with services?	ISD data – Health Visitors' records of first visit, 6-8 weeks visit. Data from other services if available.
Medium term outcomes	Examples of questions	Possible data sources
Increased positive behaviors and reduced risk behaviors	Are parents increasingly engaging in positive behaviours and reducing risk behaviours?	ISD data – Health Visitors' records.
Sustained engagement with wider services	Is there greater access to and uptake of services to improve health and wellbeing for parents?	ISD data – Health Visitors' records.
Long term outcomes	Examples of questions	Possible data sources
Improved maternal/child health and wellbeing outcomes	Does exposure to second hand smoke change after the introduction of Baby Box? To be monitored over time to track changes.	ISD data – Health Visitors' records. ISD data – hospital admissions.
Reduced inequalities in maternal/child health and wellbeing outcomes	Is there a reduction in inequalities in maternal/child health after introduction of the Baby Box? To be monitored over time to track changes in health inequalities.	ISD data – Health Visitors' records.
Reduced inequalities and improvement in early years development outcomes	Is there a change in inequalities and improvement in early years development outcomes after the introduction of the Baby Box? To be monitored over time to track changes in health inequalities.	ISD data – Health Visitors' records.
Reduced inequalities in infant mortality	Is there a change in inequalities in infant mortality after the introduction of the Baby Box? To be monitored over time to track changes in in infant mortality.	National Records of Scotland data.

Option 3: Birth Cohort Study

An alternative to the use of routinely collected data that was suggested at the third workshop was to collect primary data from a prospective cohort of births in Scotland and a comparison area (or areas) in England or Wales. This would allow a much finer-grained assessment of sleeping practices and other key outcome domains, for example using diaries kept by participants. It would provide information about how practices differed between recipients of the Baby Box and non-recipients. Note however that such differences could not be interpreted as effects of the Baby Box because there would be no information on how practices had changed following its introduction, and the effects of the Baby Box would be confounded by other differences between Scotland and the comparison area(s) in services provided.

If the sample were stratified, for example by deprivation scores based on participants' postcodes, a cohort study could also provide insights into socio-economic variation in sleeping practices and other key behaviours among recipients and non-recipients. As with the overall impact of the Baby Box, it would not be possible to estimate impacts of the Box on inequalities because information about changes in the socio-economic patterning of outcomes following the introduction of the Box would be lacking.

A cohort study of this kind would be a very substantial undertaking. The advantages in terms of flexibility to determine what data should be gathered need to be weighed against the costs and risks involved in attempting to recruit participants during pregnancy or very shortly after birth, and collecting substantial amounts of data in the early stages of parenthood. Extensive development, consideration of ethics, feasibility and piloting work would be needed to: determine the best way to engage maternity services to manage the identification and recruitment of parents; approach parents (or prospective parents) to take part in the study; estimate required sample sizes; and identify (or develop) valid and reliable instruments for gathering data. It would be substantially more costly and time-consuming than a partly or wholly retrospective study relying on routinely collected data but may be worth considering if other monitoring or research raised concerns about possible adverse effects associated with the Baby Box.

Table 4 outlines the outcomes identified in the logic model that could be addressed by this option.

Table 4. Outcomes, example questions and methods for option 3

Medium term outcomes	Examples of questions	Possible data sources
Reduced inequalities in access to new-born essentials	What are the differences in access to new born essentials in the Baby Box and non-Baby Box area?	Cohort survey of parents (comparing across SES groups).
Increased positive behaviors and reduced risk behaviors	Are parents increasingly engaging in positive behaviours and reducing risk behaviours?	Cohort survey of parents (including sleep and feeding diaries).

Table 5 summarises the evaluation options.

Table 5. Summary of evaluation options

Option	Description	Comments
Option 0	No additional evaluation	Management information systems will provide some data on inputs and outputs, such as volume of applications, take-up rates, etc., but no information on outcomes or the mechanisms by which they are achieved.
Option 1.1 – Process - qualitative	Qualitative process evaluation based on interviews and/or focus groups with purposive sample of practitioners and recipients; recipient sample to be drawn from those providing consent to contact for research in their application	Will provide information on process of delivery, mechanisms of impact and contextual factors ( e.g. receipt of other interventions), and insights into the way the Baby Box is perceived and used by practitioners and recipients. For example, short term outcomes 'has the Baby Box improved parents' understanding of risk and positive behaviours'. Will not provide quantified estimates of outputs or short/medium term outcomes, or any information about longer term outcomes.
Option 1.2 – Process - quantitative	(a) Survey (by phone or face-to-face) of recipients sampled from applicant database (b) Surveys of practitioners	Would provide quantitative estimates of short/medium term outcomes, such as expenditures on new-born essentials, levels of understanding of risk/protective behaviours, etc., but would not allow identification of impact due to lack of control/counterfactual. Parents who consent to research (25% of total) are unlikely to be representative of all applicants; an opt-in procedure prior to interview may accentuate this bias. It could be corrected with post stratification weights, with some loss of statistical efficiency. Surveys of practitioners before and after training could be used to assess and improve the effectiveness of training. Can be hard to achieve good response rates.
Option 2 – Outcomes evaluation using retrospective routine data - quantitative	Quantitative evaluation of routinely collected data from CHPS Pre-School data	Would provide quantitative estimates of the impact of the Baby Box on some short/medium term outcomes, such as levels and trends of sleeping practices, development outcomes, play/attachment, referrals to other services, etc., using interrupted time series to control for pre-intervention trends. Consistent series of data only likely to be available for outcomes recorded at first visit or 6-8 weeks. Highly dependent on quality/coverage of routinely collected data, which is unknown for many outcomes. Would identify the effect of the Box on all those eligible, rather than on recipients.
Option 3 – Outcomes evaluation using prospective cohort	Prospective cohort of births in Scotland and England/Wales	Could provide detailed information, specific to the requirements of the evaluation, on a range of short/medium term outcomes, such as sleeping practices and other risk/protective behaviours, and quantitative estimates of differences in these behaviours between Baby Box recipients and non-recipients. Would not provide direct estimates of the impact of the Baby Box. Methodologically challenging, and would require extensive development, feasibility and piloting work. It would also need to go through a medical ethical review process.