Whole Family Wellbeing Funding (WFWF) Programme - year 2: process and impact evaluation - full report

8 Monitoring the impact of WFWF support

This section explores CSPP experiences of monitoring the impact of WFWF, using data in decision making, and monitoring whole systems change.

Key findings

• All CSPPs had systems and evaluations in place to monitor and assess their WFWF delivery performance, typically at an activity or service level. Many CSPPs described improvements to the volume and quality of delivery performance and outcome data since Year 1.
• It was less common for CSPPs to have established systems for data analysis and interpretation and change planning at a system level.
• The functionality and agility of data collection and monitoring systems was a barrier to CSPPs achieving some WFWF outcomes at the pace envisioned by Scottish Government. For example, CSPP partners were often using monitoring systems that did not easily align with others or information could only be added for the referred person, making the tracking of whole families difficult. Many CSPPs did have plans to establish or enhance systems for intelligence gathering to inform WFWF support.
• In Year 1 of WFWF delivery, CSPPs struggled with using data in strategic decision-making. This continued into Year 2, but more CSPPs were in the process of developing their analytical capacity to use data to inform service delivery. Good practice sharing at multi-agency CSPP events and meetings helped evidence-based decision making. To inform strategic decisions, in some CSPPs the events and meetings were structured and facilitated in ways that encouraged multi-agency practitioners to question and critically assess the practice or evidence shared.

Monitoring the impact of WFWF support

To inform transformative family support, CSPPs continued to establish systems for capturing regular feedback from partners, and children, young people, and families. This is discussed in the section above: Outcomes and contributing factors: Children, young people and families at the centre of service design.

All CSPPs had systems in place to monitor their WFWF delivery performance, typically at an activity or service level. Common monitoring data sources were the number and characteristics of families accessing support, alongside family and staff perceptions of outcomes. Many CSPPs described improvements to the volume and quality of delivery performance and outcome data since Year 1, and strategic leads attributed this to prioritising this in their WFWF activity and in working in collaboration with third sector partners.

“The strength of the data coming through is significant…the sheer numbers and the stories and looking at what else is needed to support families. It’s been looking at how families can become more self-managing, and a lot of it is qualitative data.”

Strategic Lead

Many CSPPs used evaluations to assess their impact of their WFWF support, either self-evaluation or externally commissioned evaluations. For example, South Ayrshire commissioned Horizons Research to assess their existing family support provision, to identify opportunities to focus WFWF allocation. Horizons Research found Belmont Family First’s work with schools delivered effective family support. In response, South Ayrshire CSPP planned to scale up this support to all school clusters. In Aberdeen, managers used an external evaluator to assist with data collection around how staff were finding WFWF delivery and impacts on staff satisfaction.

A small number of CSPPs had yet to establish indicators for assessing the extent of their achievement of the WFWF outcomes. This was because they were still in the early stages of implementing these plans. A strategic lead explained that they wanted to be proportionate with monitoring data collection, ensuring staff collected the right data to inform service planning.

“It is early days; we haven’t set out outcomes for measuring yet. It is important that staff are collecting the right data, and they are not overloaded by data collection.”

Strategic lead

It was less common for CSPPs to have established systems to analyse and interpret feedback and monitoring data from other support services, such as adult or education services. Though many CSPPs did have plans to establish or enhance systems for intelligence gathering to inform WFWF support. For example, the Spotlight below discusses how Fife has plans to create a performance framework that supports system monitoring, data collection and planning.

Figure 14 CSPP Spotlight: Fife

Lead Data Analyst role developing analytical capacity and data use across all strategic priorities of the CSPP.

Fife’s Children’s Services data group includes representation from across the CSPP. The group has been working for a number of years to improve the impact of data. This has included development of a core dataset to encourage routine monitoring data collection across Children’s Services, including WFWF activities. The indicators were informed by the Core Wellbeing Indicators, developed as part of the Children Young People and Families Outcomes Framework (Scottish Government, 2023), and reflect CSPP approaches and priorities in Fife. The core dataset was intended to complement a wider dataset consisting of SHANARRI indicators, a set of eight wellbeing outcomes used in Scotland's GIRFEC framework. This wider evidence base informs the CSPP’s Joint Strategic Needs Assessment (see Glossary in Annex 3). Throughout its ongoing work to improve local data, the data group identified a lack of confidence across the CSPP in gathering the right data, identifying appropriate indicators and using improvement approaches effectively in response to the data collected. In response, they created the Lead Data Analyst post, to provide the capacity needed to address these challenges. Prior to this post, Fife used staff from different partners to collect and interpret service data and feedback, and to populate pilot templates for draft indicators. This approach relied on existing limited capacity, and as a result, progress in improving data was slow and datasets were often out of date. The post was filled shortly before the writing of this evaluation report. “We felt we got to a point that we really needed the capacity in place to develop the [data and analytical] capacity and the process, to then be taken forward by partners...We knew that we needed this a while back, but through WFWF work, we’ve been able to resource it properly.” Strategic lead The Analyst’s responsibilities include establishing relationships with multi-agency partners’ data staff; understanding data availability, quality and how to access it; supporting the data group’s development of meaningful performance reports; and creating a description of Fife’s WFWF whole systems approach (to ensure data captured covers all activity). The Analyst will also provide capacity to improve the effective use of data, including Quality Improvement work and the CSPP’s developing use of the Supporting Families: A National Self-Assessment Toolkit for Change (see Glossary in Annex 3) and the WFWF logic model.

East Lothian used their WFWF allocation to employ a Community Research Officer to improve and increase their data collation and synthesis capacity, discussed in the Spotlight below.

Figure 15 CSPP Spotlight: East Lothian

East Lothian’s Community Research Officer aims to improve and increase system-wide analysis capacity. The Community Research Officer post required someone with data literacy, an awareness of the complexity of family support needs, and a passion for change and improvement. Their remit was to work across the CSPP to develop joint working related to monitoring data and collating feedback. This was reported to be in the early stages and no information was provided on how this data was analysed or used yet. “We’ve got a bespoke person who's absolutely focussed on the data and keeps us really clear.

Service manager

The use of data in strategic decisions

CSPPs were in the process of developing the analytical capacity to analyse and interpret monitoring and feedback data within strategic decision-making. This was a limitation identified in Year 1 of WFWF delivery that continued into Year 2.

Good practice sharing at multi-agency CSPP events and meetings helped evidence-based decision making. Case study CSPPs described how this commonly led to identifying skills gaps and prompted work to deliver guidance and training to staff.

"These [events] are really helpful to inform our strategic decision making about where we go next as a partnership, so we take time to do that and feed it into the key groups."

Strategic lead

In addition to sharing good practice, to inform strategic decisions, the events and meetings were structured and facilitated in ways that encouraged multi-agency practitioners to question and critically assess the practice or evidence shared.

“Our communities are telling us what is going to meet that need is X, Y and Z, and we have that discussion with all our key leads who are the decision makers and who are budget holders. Because WFWF is across health and education, it's about making sure that all our partners have an opportunity to express their own opinion. There is that open dialogue and people are not afraid to challenge each other respectfully and to say, actually, I don't know if that's the best spend or how about thinking about doing it this way, or is that better delivered by education because it's universal.”

Service manager

A barrier to CSPPs using evidence in strategic decisions was CSPP capacity and capabilities to analyse the volume of evidence, particularly when it was qualitative data. Staff skill with interpreting evidence, especially when this suggested different views or needs across services and families, was another limitation CSPPs faced.

“Children and families don't speak with a single voice... each of them will have their own lens, their own perspective...we must try and collate some of the key themes [and] try and provide...the biggest impact that we possibly can. That will always mean there are some needs we're not able to respond to."

Strategic lead

Monitoring whole systems change

The functionality and agility of monitoring systems for analysis and change planning was a barrier to CSPPs achieving some WFWF outcomes at the pace envisioned by Scottish Government. For example, a limiting factor of some whole systems change outcomes was related to the challenges with the sharing of data and analysis across organisations. Reasons for this included the lack of streamlined processes, such as misaligned management information systems, and not having data sharing agreements in place to enable sharing was one issue. The lack of agreement on how to share information across the CSPP, what information to share and how frequently this would be shared to ensure most value was a related issue.

Practitioners also discussed how, in some services, monitoring systems only allowed information to be added for the referred person, not additional family members, which limited their ability to track support for the whole family (see Section 4 above). This challenge was not unique to WFWF; as it impacts wider Children’s Services Planning (see Glossary in Annex 3).

South Lanarkshire are trying to use a combination of service delivery data, outcome data (such as educational attendance and exclusion), family and staff feedback, and staff confidence with whole systems approach, to monitor their whole systems approach. This has allowed them to gain a holistic picture of a family’s situation.