The Evaluation of the Family Nurse Partnership Programme in Scotland: Phase 1 Report – Intake and Early Pregnancy

Appendix B: Evaluation methods

1.1 The evaluation team

In October 2009, the Scottish Centre for Social Research with Jacki Gordon & Associates was commissioned to carry out an evaluation of FNP in Scotland.

At an early stage, it was agreed that the evaluation group would be comprised of the external evaluation team ( ScotCen with Jacki Gordon) and an internal monitoring team (represented by the NHS Lothian FNP Lead) which would be responsible for the collection and collation of the routine monitoring data. Jacki Gordon would take the lead role in terms of facilitating and developing the monitoring and evaluation framework which would inform the evaluation throughout, while ScotCen would be responsible for qualitative data collection with stakeholders, clients, family respondents and members of the FNP team and would also be responsible for reporting.

1.2 Aims and objectives of the evaluation

The overall aim of the evaluation is to assess the implementation of the programme in Edinburgh, and then to use the learning from the experience to assess whether the programme can be implemented in other areas of the country.

The specific objectives of the research are to:

1. Review existing evidence of the Family Nurse Partnership Programme

• identify evaluations of the programme in the USA and elsewhere
• identify different approaches to evaluation that have been used elsewhere with a view to replicating methods where appropriate
• take note of the evaluation carried out in England
• assess implications for programme implementation and evaluation development.

2. Work with the FNP programme to develop a monitoring and evaluation framework

• identify a minimum/essential dataset
• identify acceptable levels for delivery and outcomes of interest
• define an expected timeframe which would identify how well the programme is being delivered and its impact on participants.

3. Assess the implementation of the FNP programme using the monitoring and evaluation framework:

• establish a baseline
• identify the factors which support or inhibit the delivery of the programme.
• report family nurse experience of implementing the programme, clients' and programme management views and experiences of the programme
• Identify programme alerts
• Identify the impacts the programme has on associated community services and on the community nurse workforce in Edinburgh.
• Identify the cost/benefits of implementing the programme.
• Reflect on contribution of the programme to the wider policy context.

4. Identify the wider implications should the FNP programme be extended:

• Make recommendations for implementing the programme

• Identify refinements to the monitoring and evaluation framework for use elsewhere.

1.3 Overview of research methods

The evaluation research is addressing the objectives using a range of formative and summative approaches, including:

• A review of published and grey literature to inform the evaluation (on-going)
• The development of a monitoring and evaluation framework using a logic modeling approach based around Theories of Change
• The development of appropriate and informed instruments for routine monitoring and agreed indicators and measures
• Analysis of data collected for all clients using the measures developed for routine monitoring, to assess the implementation of the programme (Objective 3)
• Longitudinal qualitative research (case study approach) involving repeat in-depth matched interviews with a panel of up to 15 clients, a nominated significant other and the clients' family nurses, interviewed at key points during the client's pregnancy and the first two years of her child's life (See Chart 1)
• Key informant and team interviews at key points to inform, assess and reflect on the implementation of the programme

1.4 The recruitment of clients to the longitudinal panel

Clients were asked to participate in the research on a strictly opt-in basis. Family nurses were asked to give a leaflet explaining the evaluation to all of their clients. The leaflet described what would be involved and how the research would contribute to our understanding of FNP in Scotland. At that stage, all that was requested of clients was that they indicate a willingness to be contacted about the research and to consent to their contact details being passed to the research team. The information was entered onto a form which the FNP Supervisor reviewed and then passed to us.

The panel of 15 clients was recruited to the evaluation on the basis of their expected date of delivery ( EDD), their age at conception of the FNP baby ( LMP) and who was their family nurse. The aim was to interview clients at the rate of 5 per month over a period of three month, to ensure that there was a range of ages and that the clients of all family nurses were, as far as possible, equally represented. This was to ensure that, if possible, no one nurse would have to be interviewed about more than 3 clients.

The Supervisor was asked to review the names of clients who had consented and to remove anyone who she felt might be harmed in any way through participation in the evaluation.

Of the 148 initially recruited to FNP, 82 (55%) agreed to share their contact details with the research team. Two clients were excluded by the Supervisor and a further 25 were ineligible because they had already had their baby. We then contacted by letter and then by telephone 27 of the remaining 55 eligible clients who would provide the spread of characteristics we wanted to include and, of these, 15 were successfully interviewed. The interviews were conducted over a somewhat longer period than initially for a variety of reasons, but largely because of our need to spread the panel across all family nurses and have a range in relation to the clients' ages.

The research team gave a presentation to the family nurses which described the evaluation and what would be required of them personally. It was a agreed that interviews with the family nurses would, as far as possible, cover both their own experiences of the programme and their experiences in relation to the specific clients included in the panel. Access to the nurses would be via the FNP Supervisor who would co-ordinate interview times.

1.5 The content of interviews

The interviews with key stakeholder informants within Scottish Government, NHS Lothian, Edinburgh City Council and the Department of Health (England) focused on:

• understandings of the programme (and changes over time)
• perceptions of desirable and achievable outcomes
• views of the training and support
• personal experiences
• perceptions of the barriers and facilitators to implementation
• reflections on the success (or otherwise) of the project

The interviews with clients were informed by the monitoring and evaluation framework (see example below) and included:

• understanding of FNP
• recruitment process
• reasons for participation
• visiting schedule, content of contacts
• perceptions of usefulness and appropriateness of schedule and contacts
• use of antenatal services
• impact of FNP on health knowledge and behaviours
• preparedness for the birth
• use of community resources
• well-being
• self-efficacy
• relationship with family nurse and with FNP
• relationships with partner, family members and friends

Family nurses were only interviewed about specific clients if they client herself had given us signed consent to speak with her nurse about her. The interviews with clients' family nurses focused on the following generic/ general topics:

• background information about the family nurse
• perceptions of their FNP training
• assessments and record keeping
• supervision
• perceptions of the aims and value of the programme

In addition, the interviews asked family nurses to reflect on their experiences with specific clients who had consented to the researchers approaching their family nurse. This element of the interviews with family nurses addressed the topics included in the client interview, but from the family nurses' perspectives, including their perceptions of their influence on the client's knowledge and behaviour.

Chart 1: Schedule of repeat interviews with clients, family members and family nurses

Respondent	Pregnancy	Postpartum	12 months	21-24 months
Client	Interview 1	Interview 2	Interview 3	Interview 4
Family member		Interview 1		Interview 2
Family nurse	Interview 1	Interview 2		Interview 3

1.6 The analytical process

The analysis and integration of data collected from multiple sources was informed by linked stages, which comprised:

• development of the monitoring and evaluation framework which specified the data that was required in order to address the evaluation aims and objectives
• specification of monitoring data items for the evaluation
• identification of topics/issues for interviews with stakeholders, clients, nurses and family members
• development of topic guides
• development of thematic charts (themes and sub-themes) which summarise narrative qualitative data
• descriptive analysis
• interpretive analysis
• integration of quantitative and qualitative data

1.6.1 Analysis of qualitative data

The interviews with stakeholders, clients, family members/significant others, family nurses and other members of the FNP implementation team are digitally recorded and transcribed verbatim (that is, word for word) and in the vernacular (that is, including slang words and local expressions).

The first step in the analysis process required the research team to read through all the transcripts. We then developed a coding frame which would allow us to apply a descriptive label to what people had said about different topics. The coding frame was made up of a number of broad themes and sub-themes.

The next step entailed testing the coding framework. This was achieved by members of the research team each coding the same transcripts, comparing the coding and then revising the coding framework to ensure that all responses could be meaningfully encapsulated.

The third step involved summarising all the interview data under an appropriate descriptive sub-theme. In this way, we were able to divide each interview transcript into comparable sections and bring together what different respondents had said about - for example - how they found out about the programme. The data can be displayed in a matrix which shows the summaries for every respondent by theme and sub-theme. The example below uses data collected as part of the evaluation, but not from the same three respondents. The nurse interviews were coded and summarised in two ways: first, in relation to generic topics (such as training or supervision) and, second, in relation to the specific clients about whom they were interviewed. This a is a complex approach, but one which will ultimately allow us to explore narratives from a range of perspectives over time.

These thematic data are then further summarised into briefer descriptions and, from these, categories are developed which group responses within and across themes in meaningful ways which encapsulate the views and experiences all respondents.

The aim throughout is:

• to map the range and diversity of all responses within the interviews
• to capture that range, regardless of how many respondents gave particular responses and
• to explore patterns of responses - for example in relation to age or any other factor which helps us to explain and understand the data.

1.6.2 Reporting qualitative data

The samples were purposively selected and we cannot, therefore, express the data in a way that implies statistical representativeness. Our approach, however, ensures that all views and experiences are represented and contribute to grounded descriptions and interpretations of the data. In terms of reporting the interview data, that meant that we do not quantify responses. Nor could we say that one factor was more important than another as we simply do not know.

The task when reporting qualitative data is show the scope of views or experiences. There is rarely a single perspective which represents an entire group of respondents. It is also important to be aware that respondents' accounts reflected their perceptions of events or situations: they may or may not have been "factually" accurate or have accorded with other respondents' or stakeholders' perceptions or views, but they did nevertheless represent that person's reality and were therefore equally valid. While this may at times seem to be at odds with what is "known" about a situation or setting, the key issue is that it is these individual perceptions and meanings which help us to understand whether and how the programme is influencing clients and their families.

We use quotes throughout to illustrate our interpretation of the data, rather than as a tool to make a point about individual respondents. Finally, more information about our "Framework" approach to qualitative data management, analysis and reporting can be found in "Qualitative Research Practice" (Ritchie and Lewis, 2003) and while Woodfield et al (2001) discuss longitudinal qualitative research approaches. References are included in the main report.

In this first interim report, we have focused very much on giving as full a description possible of clients' and family nurses' views and experiences of the programme. Later reports will integrate client and nurse narratives over time.

1.7 Ethical and NHS approval

Application for ethical approval was submitted to the South-East Scotland Research Ethics Committee. The study was approved May 2010. Application was simultaneously sought for local NHS access and this was received June 2010.