Baby Box: evaluation

1. Introduction and methods

This report presents the main findings from an evaluation of Scotland's Baby Box scheme, conducted by Ipsos MORI Scotland on behalf of the Scottish Government between June 2019 and Autumn 2020.

Scotland's Baby Box Scheme

Scotland's Baby Box, based on a similar long-established scheme in Finland, is a universal welcome gift for all new babies in Scotland, providing families with a range of essential items for their first 6 months. The box includes items for the baby (for example, clothes, books, digital thermometers, and comforters) and items for the mother (including nursing pads, maternity towels, and information leaflets).^[1] These items are delivered in a sturdy cardboard box, which comes with a mattress, protector, fitted sheet and blanket and can be used as a safe sleeping space during the early months of a baby's life.^[2] Midwives invite pregnant women to register for the scheme during their 20-24-week antenatal appointment, and the box is delivered between 32 and 36 weeks of pregnancy.

The scheme was piloted in early 2017, in Forth Valley and Orkney, and then rolled out across Scotland from August 2017. Between August 2017 and the end of 2020, around 164,000 registrations for baby boxes were received from families in Scotland.^[3]

The Baby Box scheme aims to contribute to improving child outcomes and to tackling deprivation and inequality, while recognising that it is only one of a range of Scottish Government policies that aim to give children the best possible start in life. The contents of the baby box are designed to inform and support positive parenting behaviours, through both information (about safe sleeping and maternal mental health) and provision of specific items, like books and baby wraps to support parent-child attachment, and nursing pads for breastfeeding. It was also anticipated that the box would act as a mechanism for encouraging parental engagement with services – including those parents who may otherwise be less engaged with services – in order to enable conversations about parenting and child health and wellbeing. The scheme aims to contribute to tackling inequality by ensuring that all families have access to key essential items for their baby's early weeks and months for free. By offering the box universally, to all expectant parents regardless of income or circumstance, the Scottish Government also intended the scheme would help foster a shared understanding of a society that values and supports all children.

Aims of the evaluation

The overall aim of the Baby Box evaluation was to assess what, if any, impact the scheme may be having on the expected short- and medium-term outcomes, as set out in the Evaluability Assessment conducted for the Baby Box scheme in 2018^[4]. These outcomes cover areas including: perceived impacts on child health and wellbeing; perceived financial impact on families; parental learning around risk and positive behaviours such as safe sleeping and breastfeeding; and reported parental engagement with services. In addition, the evaluation examines how the scheme is operating in practice, including levels of uptake, and processes around registration and delivery and workforce training. It also considers parents' and professionals' views of the box being offered universally, to all expectant parents in Scotland.

The more detailed outcomes and research questions addressed by this evaluation are outlined in the introductions to the relevant chapter(s).

Methodology

The evaluation was conducted by Ipsos MORI Scotland, between Summer 2019 and Autumn 2020. A mixed-method approach was adopted, including:

• Analysis of data on registration and uptake (using anonymised registration data provided by the baby box delivery contractor, APS)
• A mixed mode (online and telephone) survey of 2,236 parents
• An online survey of 870 health visitors, midwifes and family nurses
• In-depth qualitative interviews with 36 parents, 24 midwives, 20 health visitors and four family nurses across six case study Health Board areas.

Parent survey

A mixed mode (telephone and online) survey of parents was conducted in October-November 2019 to explore parents' experiences and views of the Baby Box scheme. The initial sample (5,543 parents) consisted of all parents with a due date between February and August 2019 who had registered for a baby box and agreed (by ticking a box on their registration form) that they could be contacted for research purposes (around 21% of parents who registered for a box agreed to this).

All parents who had provided an email address (94% of those in the initial sample) were invited by email to complete the survey online. 1,724 parents completed the survey online, and a further 512 parents were interviewed over the phone by Ipsos MORI telephone interviewers.

The telephone interviewing was intended to boost the response rate from those parents whose response was disproportionately lower online (when compared to the profile of all parents who had registered for a box during the sample period). Quotas were therefore set for the telephone interviewing to ensure the final sample was as representative as possible of all parents registering for a box between February and August 2019^[5], in terms of Health Board, area deprivation (as measured by Scottish Index of Multiple Deprivation quintiles), and the age of the mother. The data were also weighted to correct for any remaining differences in response on these measures. (See Appendix B for more detailed information on the profile of survey respondents).

Survey of midwives, health visitors and family nurses

Directors of Nursing in each Health Board were asked to forward an email invitation from Ipsos MORI to all community midwives, health visitors and family nurses (and their managers) in their area to invite them to take part in an online survey between late November 2019 and early January 2020.^[6] In total, 870 nurses responded to the survey – 279 midwives, and 591 health visitors and family nurses (as the number of family nurses was very small – 37 in total – their views are grouped together with those of health visitors for analysis in this report). The final data were weighted to match the profile of Health Visitors and Midwives across Scotland in terms of geographic area and the proportion of part-time and full-time staff.

Both the parents' and the professionals' questionnaires were designed by Ipsos MORI with advice from the Scottish Government and the Research Advisory Group. They were also tested with a small number of parents and with a small group of midwives and health visitors respectively, whose feedback was incorporated before the two questionnaires were finalised. See Appendix C for the full parent and professionals' questionnaires.

Qualitative interviews with parents

At the end of the parents' survey, respondents were asked if they would be willing to participate in a more detailed follow-up interview. Participants were recruited from across six Health Board areas^[7], to ensure diversity in terms of geography, rurality/urbanity and deprivation. Quotas were also set based on maternal age, annual household income, whether or not this was their first baby, and survey responses on whether they had used the baby box for sleeping, and whether they felt the box had saved them money (see Appendix B for the profile of participants).

36 in-depth interviews with parents were conducted in February and March 2020 by members of the Ipsos MORI Scotland research team. The majority of interviews (28) took place face-to-face (in participants' homes or another venue of their choosing). A smaller number were conducted by telephone, either because of parental preference or because it was necessitated by COVID-19 restrictions from 16^th March onwards. Two were conducted by email, at the parents' request.

In a majority of cases, the interview was with the mother (who is usually the parent who registers for the box). However, where participants lived with a partner, it was made clear that they were welcome to participate in the interview, and in six cases a father also contributed.

Qualitative interviews with midwives, health visitors and family nurses

The researchers also interviewed 24 midwives, 40 health visitors and 4 family nurses in more detail about their views of the Baby Box scheme. Interviewees were recruited from survey respondents who had agreed to be re-contacted, and through area Midwifery and Health Visiting leads. Participants were recruited from across the same six Health Board areas as parent interviewees (see above). Fieldwork began in March 2020 but was paused when the NHS was put on emergency footing as a result of the COVID-19 pandemic. The majority of interviews were conducted by phone in summer 2020.

For both the parental and workforce in-depth interviews, interviewers used a flexible topic guide to ensure that key issues were covered with each participant (see Appendix D). Interviews were audio-recorded and detailed notes were made after each interview, summarising views on key topics. The data was then systematically reviewed to identify the full range of views expressed. All parents received £30 as a thank you for their time.

Scope and limitations

As with any study, this evaluation is subject to a number of limitations.

First, it was not possible to conduct an experimental or quasi-experimental evaluation of the Baby Box scheme. The most reliable way of establishing the impact of any programme or intervention is to compare outcomes for participants with outcomes for similar individuals who did not go through the intervention (a 'control' group). However, the simultaneous roll-out of the Baby Box scheme across Scotland and the high level of take-up meant there was no appropriate control group available.^[8] Without a control group, it is not possible to definitively attribute any reported outcomes to the Baby Box scheme, rather than wider contextual factors.

A second (related) limitation is that the evaluation is based on self-report data from those who received the box (parents) or may have observed parents using it (health professionals). As such (and in the absence of a control group), although the evaluation aims to assess the potential impact of the scheme, it can only report on people's perceptions of the impacts of the Baby Box scheme, rather than on definitive outcomes. When dealing with perceived impacts there is inevitably room for error. There is a risk that impacts may be overstated or misattributed, given the difficulty of individuals accurately identifying and isolating the impact of a specific intervention from all other contextual factors that might influence their behaviour, views or experiences.

Third, in relation to the survey of parents specifically, the sample was drawn from those who consented to be contacted for research purposes when registering for their baby box (21% of all those who registered for a box in the period the sample was drawn from). As discussed above, the final achieved sample of parents was weighted to match the profile of all parents who registered for a box on key known characteristics (maternal age, area deprivation, and Health Board). However, while this means it is broadly representative of all parents registering for a box on these known characteristics, we cannot be certain that there are no other (unknown) differences in the profile of those who did and did not agree to be contacted for research which may have impacted on their views of the scheme.

Finally, the COVID-19 pandemic necessitated minor changes to the evaluation design and timescales. As described above, four in-depth interviews with parents had to be conducted by phone rather than face-to-face. The impact of this is likely to have been minimal. At the start of the pandemic, the qualitative fieldwork with health professionals had just begun and a small number of depth interviews had been undertaken. When the NHS was placed on an emergency footing in March 2020, fieldwork was paused and did not restart until July 2020. The meant that the majority of these took place at a time when health professionals would have been reflecting on the impact of the Baby Box scheme after a period of enormous upheaval – both in the NHS and people's lives more generally. While the pandemic-related changes did not feature heavily in the interviews, they may still have changed participants' perspectives on the impact of the Baby Box scheme.

Report conventions and structure

The findings in this report are organised thematically, so that findings from the different elements can be triangulated in answering the research questions. Each chapter begins with a boxed summary of the key findings and an outline of the research questions that chapter addresses. Boxed points for consideration in terms of the future development of Scotland's Baby Box scheme are included at relevant points within chapters. These are based on the researchers' reflections on the findings and are intended to highlight questions that the Scottish Government and its partners, particularly Health Boards, may wish to consider in relation to the future development of the Baby Box scheme.

Survey findings are always subject to a margin of error, which determines how confident we can be that any differences are likely to be a true reflection of differences in the population or may simply have occurred by chance. In this report, any differences highlighted between sub-groups of survey respondents (e.g. parents of different ages) are statistically significant at the 95% level, unless otherwise stated.

When interpreting findings from the in-depth interviews with parents and health professionals, it is important to remember that qualitative samples are designed to ensure that a range of different views and experiences are captured, rather than to estimate the prevalence of particular views or experiences. As such, quantifying language, such as 'all', 'most' or 'a few' is avoided as far as possible when discussing the qualitative findings in this report.

The remainder of this report is structured as follows:

• Chapter 2 covers operational processes and uptake of the scheme, including views of the registration and delivery process
• Chapter 3 provides an overview of use of the box and its contents, including perceptions of which items were most and least useful, and reasons why parents do or do not choose to use the box itself for sleeping
• Chapter 4 discusses the perceived impact of the scheme, including potential impacts on parental knowledge and behaviour, financial impacts, and impacts on engagement between families and their midwives or health visitors
• Chapter 5 outlines the views of midwives and health visitors/family nurses on the training and information they have received around the Baby Box scheme, and their views of its purpose and their role in relation to it
• Chapter 6 examines parents' and professionals' views on the universal availability of the scheme.