Family Nurse Partnership in Scotland: revaluation report

3 Methodology

This section provides an overview of the Revaluation method. 

3.1 Revaluation Approach

Revaluation is a structured participative methodology that generates data qualitatively through capturing the stories told by clients and service providers at multiple levels within a complex system. 

Similar to applied social research methods such as participatory evaluation (Cousins and Earl 1992), participatory action research (Walter 2012; Bergold and Thomas 2012), stakeholder evaluation (Hengtsberger-Sims and McMillan 1991) and realist evaluation (Pawson and Tilley 1997; 2004), Revaluation views interventions such as FNP as active, changing, context-dependent and multi-layered. Revaluation starts from the understanding that complex systems and their value cannot be understood fully from any one place but require observation and analysis of the views and opinions of actors at multiple levels within the system.

The Revaluation approach to understanding value in a complex system was pioneered in 2015 by learning studio and AD Research and Analysis, working in partnership with NHS Improvement, Horizons’ team. The Centre for the Evaluation of Complexity Across the Nexus (CECAN)^[2] supported an academic fellowship and Revaluation has been described in their series of Policy & Practice Notes^[3]. Since 2015, Revaluation has been used in over 25 settings, aiming to help system leaders and activists understand the value of the transformations they are bringing about. A community of practice – the Revaluation Collective – began to exchange experiences about how to measure and understand value in complex systems in 2018.

This method has previously been used to explore the value of the Family Nurse Partnership across a national system through work commissioned by the Public Health Agency (PHA) in Northern Ireland (2017-18). 

3.2 Revaluation of FNP in Scotland

Given the relevance of the PHA Revaluation study to the Scottish context, the Scottish Government felt that it would be useful to undertake a similar study to explore and understand the value of FNP to clients, providers and other stakeholders in Scotland. 

The main aims of the Revaluation were to:

• Provide a detailed multi-level account of the structure, development, processes, networks and relationships that make up the Scottish FNP system and the complexities within it; 
• Understand and highlight the perceived value of FNP from the perspectives of those operating within the system, through stories generated by Family Nurses and other actors;
• Allow FNP nurses to reflect on their views of value of their work and develop a stronger, shared understanding of this value; and
• Identify key areas where learning from FNP has been shared and how this can be enhanced in the future.

3.3 Revaluation Process

Revaluation relies on storytelling to explore the experiences of those providing and receiving the FNP programme. To capture the different perspectives of actors at different levels, the Revaluation team subdivided the whole FNP system into five levels: nano; micro; meso; exo; and macro (see Table 1 for levels and methods of data collection), each describing a different level of focus. 

Nano stories were collected from 164 clients, of these 29 were deepened. At a Micro level, there were 108 Family Nurse Stories and at a Meso level 20 team stories. The nano and micro levels provide the most stories, as most people in the FNP system are within these levels (see Appendix 1 for example nano story collection template).

Table 1: Levels within the FNP System and Methods of Data Collection

Level	Who?	Focus of stories	Collected through
Nano	Client/mother, baby, client’s partner /baby’s father, and nurse, family members and other key stakeholders.	The value of FNP for FNP clients and their children.	Client Interviews (by Family Nurse) Case Note Review (by Family Nurse) Written Narratives (by Revaluation Team) Iteration of Narratives between Family Nurses and Revaluation Team Discussions with/Written Submissions from other case workers, other health care professionals, social care colleagues, friends and family members.
Micro	Family Nurses, Supervisors in their team and Data Managers	Focused on the value of FNP for them.	Focus group discussions with teams (by the revaluation team)  Written submissions by FN’s  Written Narratives (Individual)
Meso	The Supervisors, and other Staff in the 20 FNP teams, and their Family Advisory Boards	Focused on their view of the value of FNP within the local governance system.	Facilitated Team Discussion Iteration of Narratives (reflection on and clarification of emerging narratives to inform overall team narrative)  Written Narratives (Team level)
Exo	Regional Leads	Focused on programme governance and measuring success.	Focus Group and Interviews with Regional Leads
Macro	Scottish Government	Exploring what matters in FNP, programme governance and how FNP is monitored.	Focus group and Interviews with key stakeholders.

3.3.1 Storying

Revaluation relies on storytelling to explore the experiences of those providing and receiving the FNP programme. Participants were asked to share their stories of the positive value of FNP from their perspectives. The process of capturing these stories was facilitated and supported by the Revaluation team. Stories were elicited by different means dependent on the level within the system (see Table 1) but primarily focus group and interviews were used to create written narratives. This was subsequently shared with participants and amended based on their feedback. These processes were described by the evaluation team as iteration and cascading. 

Deepening

Once the stories were collected, the Revaluation team facilitated each local FNP team to choose a small selection of stories to translate from narrative form into a six-box grid format. This grid was used to structure and explore the different kinds of value contained within the stories. The Revaluation team used this information to create deepened stories (see Section 4.9 ) for examples.

The six-box grid is split into three rows:

• Calculate involves presenting quantifiable outputs and outcomes gleaned from the stories generated, and assigning indicative monetary value to these, outcomes (for example, in terms of potential savings made or costs incurred).
• Calibrate involves qualitative judgements about the relative merits of different actions and outcomes.
• Capacitate involves measuring the characteristics and capacity of the system, and the potential of FNP to increase its capacity in future.

The grid is also split into two columns:

• Visible refers to known data that is already observed and collected (and often measured) within the system.
• Invisible refers to knowable data that could be collected and measured if desired. For example, invisible data may include information known by frontline staff but not systematically collected.  

An effort was made to link the emerging perceived value of FNP with theoretical cost savings using the top right-hand box in the grid (calculate-invisible). For example, the costs saved as a result of the FNP programme averting greater service-use among clients. Whilst this indicated that FNP is perceived to reduce costs, it is not possible to establish or monetise savings from this data with any degree of certainty.

3.3.2 FNP Client Vulnerability 

In the Northern Ireland Revaluation study, a list of 43 vulnerability factors was constructed by the Revaluation team. This was based on vulnerabilities that emerged from FNP client stories and informed by relevant national frameworks. This list of potential vulnerabilities was used by family nurses in Northern Ireland to profile their current caseloads. Nurses rated their clients against these vulnerabilities based on their clinical knowledge of whether the client had currently or previously experienced this vulnerability at the time of enrolment onto FNP. 

In order to better understand the characteristics of the FNP caseload in Scotland, a similar exercise was undertaken in the Scottish Revaluation. This recorded data indicative of the characteristics of, and particular challenges faced by, the whole population of FNP clients in Scotland.

Data was collected by Family Nurses for all 2,083 FNP clients who were enrolled on the programme at the time of data collection. They were provided with a “vulnerabilities profiling tool” (see Appendix 2), which comprised of a checklist of 43 vulnerabilities for clients similar to those from Northern Ireland but adjusted for cultural relevance to Scotland. Family Nurses were asked to retrospectively complete this for each client, indicating all vulnerabilities that they perceived the client to have experienced at the time of enrolment onto FNP. Family Nurses responses were based on their clinical knowledge and consultation of case notes if required. The anonymised information was then processed by local data managers entered into an excel spreadsheet, the data was then collated centrally and analysed.

3.4 Methodological Limitations

3.4.1 Storying

Qualitative data such as that obtained in the storytelling component of this Revaluation enables us to understand whether an intervention such as FNP is acceptable and valued by those affected rather than prove a causal relationship between an intervention and outcomes. While individual accounts cannot be taken as representative of the experience of everyone involved, by identifying recurring themes across individual participants we can gain important insights into the issues that are relevant to a group as a whole and consider how these relate to the underlying theoretical basis of FNP  (see Section 4).

The pre-existing interpersonal relationship existing between the Family Nurse, their clients and other stakeholders increases the likelihood of open and honest conversations and richer narratives around value. However, stories will also reflect individual beliefs and potential biases. For example, it is important to acknowledge that the explicit focus of the current Revaluation on value is likely to have increased the focus for those taking part on the benefits of FNP. Consequently, they are less likely to have considered or raised issues and challenges associated with the delivery of the programme. 

3.4.2 Vulnerabilities

When interpreting the findings of the vulnerability analysis, it is important to bear in mind several limitations. First, due to the time difference between clients enrolment and Family Nurses completion of the tool, Family Nurses may not have been able to recall each client’s vulnerabilities precisely. In some cases Family Nurses were recalling clients who were enrolled almost three years previously. Additionally, it is likely that some clients would have experienced vulnerabilities that the Family Nurses were not aware of at their point of entry to the programme. Indeed, given the emphasis in FNP on building strong, trusting relationships during the course of the programme, it would be anticipated that more vulnerabilities will be disclosed as this relationship develops. And finally, many of the vulnerability categories were relatively loosely defined, and may have been interpreted differently by different Family Nurses. 

That said however, the findings from the vulnerabilities analysis are consistent with recent Scottish figures highlighting a strong correlation between deprivation and teenage pregnancy (Information Services Division, 2018). Deprivation, in turn is associated with increased exposure to risks and poorer health outcomes (NHS Health Scotland, 2018). While rates of teenage pregnancy have decreased in the past decade across all levels of deprivation, pregnancy rates among those living in the areas of highest deprivation remain five times higher than those in the least deprived areas (58.9 compared to 11.8 per 1,000). Those under 20 years old in the most deprived area are also much more likely to deliver their baby than terminate their pregnancy, with a rate 12 times that in the least deprived areas (40.2 compared to 3.3 per 1,000) 

This exercise piloted the Vulnerabilities tool and improvements and alterations will be considered before measuring vulnerabilities in the future. It is also recognised that factors that can make people more resilient were not measured, these will be considered going forward (see section 4.8).