Perinatal and infant mental health programme: evaluability assessment

3 The Evaluation

3.1 Aim of the Evaluation

It was agreed that the aim of the Evaluation would be 'to find out if the mental health of women in the perinatal period and the mental health of infants, up to the age of three years, had improved as a result of changes in services and improved access to appropriate help and support'. The evaluation, therefore, has four main objectives:

1. To assess if women in the perinatal period are able to access appropriate mental health care services in a timely manner.

2. To assess if primary care givers are able to access appropriate mental health care services for their infants in a timely manner.

3. To assess if practitioners in contact with young families understand the importance of mental health in the perinatal period for both mothers and their babies and are able to appropriately identify, support and refer women and infants who may need more specialist intervention.

4. To assess if offering women and their families early intervention helps to prevent referral to more specialised mental health services.

In order to be able to respond to these objectives, the third evaluability workshop, prioritised several of the medium-term outcomes and stakeholders identified the questions for evaluation. The outcomes were then turned into the 'core' evaluation questions. The answers to the evaluation questions are the means by which the success or otherwise of the programme would be measured. Beneath each core question, several sub questions were then developed which would need to be answered to be able to fully answer the core question. The two outcomes that related to the relationship between infants and their primary care giver were considered together. Some of the sub-questions are the same for different core questions. However, the answers are likely to be slightly different depending on the context. This means that it will be possible to answer different core questions with the same sub-question.

3.2 Evaluation Questions

Core Question 1: Are women with perinatal mental health issues identified early and offered prompt appropriate care?

1.1 What understanding do professionals have about risks to mental health in perinatal period?

1.2 What understanding do women and their families about the risks to mental health in perinatal period?

1.3 Are all women regularly and appropriately assessed throughout the perinatal period? Including:

• Routinely at booking?
• Later bookers?
• Those with pre-existing mental health conditions?
• At risk groups?

1.4 Are appropriate screening tools used?

1.5 Are all professionals aware of where to refer women and are they aware of the range of appropriate interventions? Including those for women with

• Pre-existing mental health issues
• Mild to moderate mental health issues

1.6 Do all women have access to appropriate services throughout Scotland, at the most appropriate time? Including:

• Time from referral to access
• Do services respond timeously and appropriately?

1.7 Are all the staff fully and appropriately skilled and are services appropriately resources with multidisciplinary teams to meet demand?

Core Question 2: Are women at risk of perinatal mental health problems and suicide identified and appropriately supported?

2.1 What understanding do professionals have about risks to mental health in perinatal period?

2.2 What understanding do women and their families about the risks to mental health in perinatal period?

2.3 Are all women regularly and appropriately assessed throughout the perinatal period? Including:

• Routinely at booking?
• Later bookers?
• Those with pre-existing mental health conditions?
• At risk groups?

2.4 Are appropriate screening tools used?

2.5 Are all professionals aware of where to refer women and are they aware of the range of appropriate interventions? Including those for women with

• Pre-existing mental health issues
• Mild to moderate mental health issues

2.6 Do all women have access to appropriate services throughout Scotland, at the most appropriate time? Including:

• Time from referral to access
• Do services respond timeously and appropriately?

2.7 Are all the staff fully and appropriately skilled and are services appropriately resources with multidisciplinary teams to meet demand?

2.8 Do women feel able to be open about their thoughts for suicide and/or self-harm with relevant professionals?

2.9 What has been the impact of suicide awareness campaigns?

Core question 3: What barriers do women face in disclosing perinatal mental health difficulties?

3.1 What understanding do women and their families have about the risks to mental health in perinatal period?

3.2 What stops women talking about their mental health?

• Stigma?
• Certain groups?
• Professional behaviour

3.3 Are appropriate screening tools used?

3.4 Do all women have access to appropriate services throughout Scotland, at the most appropriate time?

• Time from referral to access

3.5 Do services respond timely and appropriately?

Core Question 4: Are infants at risk of mental health problems identified early and support offered to their care-giver?

4.1 What understanding do professionals have about risks to infant mental health?

4.2 What understanding do women and families have about risks to infant mental health?

4.3 Are women whose infant may be at risk of poor infant mental health identified in the antenatal and postnatal period?

4.4 How, by whom and at what time points are infants assessed before the age of 3 years?

4.5 Are infants at risk of poor mental health identified as soon as possible, irrespective of their social circumstances?

4.6 Is there an appropriate pathway in response to poor mental health (based on GIRFEC principles) and is this used appropriately?

4.7 Are infants at risk of poor mental health referred and offered appropriate and prompt support nationally?

4.8 What is the impact of support (i.e. services and/or intervention) in response to poor infant mental health on the long term outcomes for the child?

Core Question 5: Are primary care givers and their families supported to meet the needs of their child/children?

Core Question 6: Are primary care givers and their families supported to form and maintain a healthy relationship with their child/children?

5/6.1 What understanding, knowledge and skills do professionals have about healthy relationships between infant and care-giver?

• Including assessment

5/6.2 What understanding do women and their families have about what constitutes a healthy relationship with their infant?

5/6.3 What support is available for all parents to encourage a healthy relationship between them and their child?

• Timely?
• Who is providing it?

5/6.4 Are all professionals aware of what support is available both nationally and locally?

5/6.5 Do all women and significant care givers have access to appropriate support throughout Scotland, at the most appropriate time?

• Awareness?
• Enablers?
• Barriers?

3.3 Evaluation Approach

3.3.1 Theory Based Design

We strongly recommend that a theory based approach is taken to the evaluation. This relies on the development of a Theory of Change with clear pathways which demonstrate how a particular action should result in the desired change. As the current programme logic model (theory of change) only demonstrates the desired outcomes for the programme (the right hand side of the model), it is recommended that the rest of the model is completed when decisions about the proposed actions of the programme have been made around. This would include resource allocations, activities and outputs. The evaluation would then test whether the inputs actually produced the desired outcomes.

There are three main components that the evaluation needs to address:

• The structure and resources of the programme
• The process of implementation and how the resources are used
• The outcomes of the programme and the impact it has made on the intended beneficiaries

3.3.2 Evaluation Methodology

There are a variety of methods that can be used when evaluating an initiative like the Perinatal and Infant Mental Health programme. Randomised Control Trials (RCTs) are often held up as the 'gold standard' of evaluative research but, in practice, there are significant limitations to this approach in the real world. For example, it would be ethically difficult to divide families into groups where one group received the revised programme and one did not, particularly if the revised programme included interventions that had been shown to have positive benefits for women, their families and infants. Another option is a natural experimental approach where the natural variation between groups is used to compare results. When the planned roll out of a programme is staged, the results of the early intervention group can be compared to the group who have not yet received the programme. This is a useful approach as long as that there is no reason why one area received the intervention before another, which would introduce bias into the sample.

For this evaluation, the NHS Health Boards are aiming to roll out improved services as quickly as possible and some services are already being implemented. Therefore, it is considered that a 'before-and-after' approach might be most appropriate, with varying baselines depending on the stage of implementation of a particular programme. This approach entails looking at outcomes for women, their families and babies, as well as the services, before the programmes are put in place and, then, assessing them again after implementation. Whilst 'before-and-after' designs are very common, they have limitations. It is important to ensure that any changes observed in the evaluation can be, as far as possible, directly attributed to the programme of intervention. For instance, the evaluation might detect that stigma around mental health issues in the perinatal period have reduced, but it is possible that stigma in the population had reduced for other reasons.

Using a 'before-and-after' approach, it is very important that Phase 1 of the evaluation produces a baseline which can be used as a comparison for the later Phase 2. Some elements of the programme have started already, so a decision might have to be made about the most appropriate time point to establish a baseline or baselines. For some elements of the programme, retrospective data may be available, while for others the work might not have fully started so a baseline could be established relatively near the beginning. For others, a certain date to act as a baseline may have to be taken, preferably based on a change in the work stream or a change in the pace of implementation. Any baseline should cover the main indicators that will be used in the evaluation. It is strongly recommended, therefore, that a survey of women, a survey of professionals and a case note review is carried out as soon as possible. Phase 2 would be methodologically the same as Phase 1 and be carried out around 3 years after the baselines have been established. In order to monitor whether the programme is on track, some form of simple process evaluation could be undertaken on an annual basis to record what activities are happening.

3.3.1 Evaluation Governance

We recommend that an Evaluation Advisory group is set up to oversee the evaluation. This should comprise mental health professionals, healthcare workers who cover maternity and early years, Scottish Government, evaluation experts and third sector and academic institutions who have an interest in this area.

We recommend the evaluation is put out to competitive tender and that this is done using a two staged approach covering the 'before' and 'after' aspects of the evaluation.

3.4 Potential sources of information

The table in Appendix 2 shows the evaluation questions in tabular form. The final two columns show the potential data sources that could be used to answer these questions and the sources of these data. They can broadly be divided into the following categories:

3.4.1 Women and/or their infant

• Routine data from Public Health Scotland, Data and Intelligence directorate (formerly Information Service Division, National Services Scotland)^[4]
  • Information held within the maternity record
  • Admissions to hospital and Mother and Baby Units (MBUs) for mental health issues in the perinatal period
  • Information held within the Child Health Surveillance System (CHS)
  • Information held by the Family Nurse Partnership
• Current SMR02 data return collects diagnoses at hospital admission (during and/or at end of pregnancy and this can include ICD10 codes for mental health conditions). Currently, no specific capture by national data return of this at booking.
• National Records of Scotland (NRS) for suicide patterns in women in perinatal period
• In the case of stillbirth and neonatal death, maternal history is examined by 'Mothers and Babies: Reducing Risk through Audits and Confidential Enquiries across the UK' (MBRRACE-UK)
• The Maternity/Neonatal Hub (Mat/Neo Hub) is undertaking ongoing work with Perinatal Mental Health network (as part of Enhanced Mat Dataset for Scotland, EMADS) to standardise the questions asked at Booking, the tools used and the data recorded on mental health history and mental assessment at antenatal booking.
• National data only gathered on booking and delivery. Delivery data capture offers the opportunity to consider previous maternity journey and record problems identified. E.g. smoking, drugs, mental and physical health.
• Clinical notes held by Health Boards. These include patient/client records from midwives, health visitors, mental health workers (including those working in the community and those in clinical settings and those working in the private and third sector), paediatricians, and consultant psychiatrists.
• Information gathered directly from women and their families by survey or qualitative interviewing.

3.4.2 Work force

• Data from NHS Education for Scotland (NES) about the content of training course for staff as well as the numbers and the professional discipline of the staff trained.
• Information directly from practitioners using survey or qualitative interviews.

3.4.3 Programme implementation

• Information from Health Boards about their implementation of perinatal and infant mental health work plans.

3.4.4 Other sources of information

There are various other sources of data, such as additional analysis which Local Intelligence Support Team (LIST) analysts have been asked to do. In addition, the Nurse Directors have set up a short term working group to look at all nursing data as it is recognised that this area needs some attention. Depending when the evaluation is conducted more data, might become available through this route. Nationally, data about Health Visiting and Community Mental Health activity is limited.

3.5 Proposed Evaluation studies

In order to answer the evaluation questions, and using, where possible, existing data, it is proposed that the following studies are undertaken. If a 'before-and-after' design is used the majority of the studies need to be take place at baseline and again, in 3 to 5 years' time, to measure the impact of the programme. However, because services are constantly changing and developing, it also needs to be recognised that the studies may reflect slightly different time points in terms of the baseline they present. It is, therefore, recommended that the Evaluation is commissioned as soon as possible so that the maximum amount of consistency in the timing of baseline measures can be achieved.

3.5.1 Analysis of existing data

Analysis of existing data would help answer evaluation questions 1.3, 1.4, 1.6, 1.7, 2.3, 2.4, 2.6, 2.7, 2.9, 3.3, 3.4, 4.3, 4.4, 4.6, 4.7, 4.8, 5/6.6

Women

This could include

• Any references to mental health or risk factors held in the maternity record
• Information collected from Psychological Therapies Services
• Data from the Maternity Care Survey, re-analysed if necessary
• Analysis of the admissions data for perinatal mental health to secondary care.
• Information from NRS about deaths of women in the perinatal period

Infants

This could include

• Known factors about infant mental health or risk factors held in the Child Health Surveillance Programme (CHSP) and the Family Nurse Partnership (FNP) records.
• Analysis of data from Early Years providers about access to their services and approaches to infants who appear to have additional support needs for mental health issues.
• There are also various ad hoc studies which could be used or reanalysed to inform on infants' mental health. These include Growing Up in Scotland (GUS) and the Scottish Study of Early Learning and Childcare (SSELC). These would give quantitative data on very young children as a baseline and elements of them could be included in any bespoke data collection after the programme has been running several years.
• It would be most useful to be able to link data for this study e.g. the maternity record could be linked to the Health Visitor (HV) record which could be linked to SMR02 data and linked to any admissions data. It would also be useful to link prescribing data to the maternity record. However, this might prove difficult to achieve. GP data would also be useful although it is unlikely this would be made available. We would recommend holding discussions with the Local Intelligence Support Teams to see what data could be accessed at a local level.
• Other data on specific groups of children, such as those who are Looked after and Accommodated or on the Child Protection register, should be monitored to see if there is any change, although this might be unlikely in the time frame of the evaluation.

Workforce

• Collection of information from NHS Education for Scotland about training

3.5.2 Study of Clinical Notes (Clinical Note Review)

A clinical note review would help answer evaluation questions: 1.3, 1.4, 1.5, 1.6, 2.3, 2.4, 2.5, 2.6, 3.3, 3.4, 4.3, 4.5, 4.6, 4.7.

Women

Women and their families are likely to have many encounters with a wider range of health and social care professionals throughout the perinatal period. Each encounter is an opportunity for prevention of and early intervention for mental health issues in the perinatal period. Based on the perinatal mental health pathways developed by NHS England, the Managed Clinical Network (MCN) are developing similar pathways for the Scottish population spanning from pre-conception to urgent admission to a MBU. We suggest that it would be worthwhile to document a woman's various encounters including time points before a woman is admitted to a MBU.

A focussed study of clinical notes (Clinical Note Review) would enable the histories of a sample of women to be studied. These notes should cover midwife and Health Visitor records and also mental health worker records ranging from Community Psychiatric Nurses to consultants' clinics. If possible, it would be useful to link these notes i.e. a review is undertaken of a woman's pregnancy record, the Health Visitor record and any subsequent referral record and the notes of the agency referred into.

Infants

A similar approach can be taken for infants. The main source of data will be Health Visitor notes followed by any notes from referral agencies which may also include early years' practitioners as well as health staff. Through the Health Visitor record, it should be able to track any concerns around an infants' mental health and well-being particularly in the first three years.

3.5.3 Survey of women and their families

This survey would help answer evaluation questions: 1.2, 2.2, 2.9, 3.1, 3.2, 4.2, 4.5, 5/6.2, 5/6.5

A survey of women and their families should be undertaken along with qualitative interviews. The survey could cover what women and their families understand about mental health in the perinatal period and the mental health of their baby, whether women and/or their families are aware of the risk of suicide and self-harm, sources of help for women and their families as well as the barriers to accessing help both for themselves and their babies. It could also ask women about their understanding of what constitutes a healthy relationship with their baby. If desired, women could also be asked about their experience of existing services although this is not explicitly articulated in the evaluation questions. It may also be useful to target certain women e.g. those who have used services of various types.

In qualitative interviews, the experience of women with mental health issues would be explored more fully along with their experience of enablers and barriers to obtaining help. The bond/attachment between mother/primary care giver and baby could also be explored.

3.5.4 Survey of Practitioners

This study would help answer evaluation questions: 1.1, 1.4, 1.5, 1.7, 2.1, 2.4, 2.5, 2.8, 3.3, 4.1, 5/6.1, 5/6.4, 5/6.6.

A survey of a range of practitioners, including those in health and statutory services and the Third Sector, who are in contact with women and families in the perinatal period and/or with infants in the very early years should be undertaken along with qualitative interviews. This survey would cover practitioners' understanding of perinatal mental health both in women and infants and their knowledge of risk factors for poor mental health. The survey would cover the use of screening tools, awareness of appropriate interventions and referral pathways. It could also ask questions about the training they have received about perinatal mental health and/or infant mental health and how they have implemented that training. In addition, the perspectives of practitioners about what they believe helps or hinders women and their families accessing mental health services could be sought along with their views about the role of statutory, informal and third sector organisations in supporting the mental health of women and their families as well as infant mental health. The survey could take into account the different levels at which the practitioners intervene i.e. which of the five tiers they operate within (see Appendix 1). It would be useful also to ask practitioners to self-assess their own competencies and in this area and whether they need enhanced training.

3.5.5 Mapping of Existing Services

It would be useful to map, as far as possible, existing services which support women, their families and infants in the very early years.^[5] This could be a desk based exercise but should include the third sector as well as health and other statutory services.

3.5.6 Recording and monitoring the implementation of the Programme

The programme works to yearly work plans which aim to describe the process of implementing the recommendations set out in the Needs Assessment. We would suggest that, for every annual work plans, outputs are agreed and indicators for those outputs established in order to monitor whether the work plans are being implemented effectively. For example, if an increase in staffing is a key objective of a work plan, then an indicator would give the baseline position and, then, the expected increase in numbers for the year. Currently, the establishment of indicators has been done with some of the suggested outputs but not with others.

In addition, it might be useful at key points to have a 'taking stock' event or study. This could be done either through regular meetings with those responsible for the implementation of the programme, or a series of interviews with stakeholders conducted at key points in the roll out. This will give an idea of some of the challenges that have had to be overcome and some of the successes in terms of implementation. It addition, it will be useful information for the implementation of any adaptions the programme might need to make as the implementation is rolled out. Equally important, measures need to be taken of unintended consequences of the implementation of the programme. For instance, does raising awareness of perinatal mental health issues place more pressure on other parts of the system such as primary care? This is where hearing the experience of professionals and women and their families as the programme unfolds will be very important.

It is strongly suggested that the 'left hand side' of the logic model is now developed. Not only will this show clearly what actions and expected outputs are planned but would also demonstrate how the actions link to the outcomes. At this point, this could be a desk based exercise. It would also enable the indicators for the activities and outputs to be established and, hence, reported on as the programme progresses.

The suitability of training courses should also be measured as part of the monitoring of the implementation of the programme. Understanding the content of what frontline workers are being taught will form a necessary backdrop into evaluating the impact of that training on practice. Links should, therefore, be made with NES in order to obtain numbers accessing their courses and any assessments they have conducted as to the content and implementation of learning.

It should be stressed that monitoring the implementation of the work plans is not in itself a measure of the success of the programme. However, it will describe and assess the mechanism whereby any improvement in outcomes is achieved. The achievement of the short term outcomes as described in the programme logic model is a measure of the success of implementing the programme.

3.5.7 Literature review of women's mental health in the perinatal period and of infant mental health

A literature review about women's mental health in the perinatal period and infant mental health up to the age of three years would help ensure that the programme implementation is in line with best practice. It would also underpin the standards the evaluation would use to assess the programme. For instance, a review of the literature will help determine what appropriate pathways for infant mental health are and what would be considered appropriate interventions at what time points. Although cited last, in fact, this should be undertaken as soon as possible.