Whole Family Wellbeing Funding (WFWF) - year 1 process evaluation: final report

6 Monitoring WFWF delivery

This section discusses how CSPPs monitored their WFWF performance. Findings are based on evidence from initial plans and CSP annual report analysis, and qualitative interviews with case study strategic leads, local WFWF leads and frontline practitioners.

Key findings

• It is a positive sign that all CSPPs had identified indicators to monitor performance of the WFWF activity to some extent. All CSPPs were also aware and undertaking development work to understand the monitoring indicators they would use to evidence the outcomes of their WFWF activity. Data collection was less progressed for outcomes, compared to collecting data on delivery. This likely reflects the stage of WFWF implementation.
• CSPPs had varying levels of confidence in using collected data for strategic decision making. CSPPs with more advanced analytical capabilities had pre-existing data and performance groups to manage data collection and establish its use in their planning and strategy.
• Going forward they expressed ambition to use feedback from children, young people and families collected to shape their WFWF support offer in a more formal and structured way.

The initial plan guidance from Scottish Government instructed CSPPs to establish monitoring systems to assess their progress in achieving intended WFWF outcomes. This included to:

• Provide information through Children’s Services Plans (CSP)^[14] and CSP annual reports about the performance and impact of WFWF activity. This should be supported by information on how CSPPs are approaching the delivery of holistic family support through the WFWF.
• Develop local measures of success criteria, key milestones, and quality indicators, aligned with the logic model established for the funding.
• Provide supplementary information required by the of the WFWF evaluation, where possible as part of reporting on Children’s Services Plans.

Monitoring the impact of WFWF support

Collecting monitoring data

All CSPPs intended to use existing quantitative management information to monitor the performance and outcomes of WFWF activity (see Figure 13). The extent to which CSPPs reported that they had defined the required indicators and were collecting required evidence to monitor progress though varied considerably. This was primarily linked to diversity and progress with implementing their WFWF activities.

WFWF funding supported a workshop for staff and project leads to train them on how to monitor outcomes using the Children’s Services Data Group's framework, including specifying outcome measures and monitoring processes.

Figure 13 CSPP Spotlight: Fife

Using existing data service groups to select priority metrics for impact monitoring, then running workshops to train project leads in how to use them.

Fife used their pre-existing Children’s Services Data Group to select existing metrics to use as part of their performance and outcomes monitoring. They created a high-level dataset and scorecard, which incorporated wellbeing indicators and a modified framework derived from the ‘Supporting Families: A National Self-Assessment Toolkit for Change’ Framework. Initially, the dataset focused on activity measures, with plans to transition to outcome measurement as outcomes became evident.

CSP annual report analysis suggested all CSPPs had identified indicators to monitor performance of the WFWF activity to some extent, but the maturity of the data collection process varied among CSPPs. Some CSPPs used WFWF to refine their family support monitoring approaches. This included an example of a CSPP implementing new IT software to better track and measure children and families’ journeys across the family support system. CSPPs less mature in their approach to monitoring were collecting basic baseline data (e.g. numbers of beneficiaries), while some CSPPs were going further and capturing stories from children, young people and families and frontline practitioners.

The range of monitoring indicators used by CSPPs to evidence outcomes varied according to the diversity of their WFWF activities. Although CSP annual report analysis indicated that all CSPPs were mindful of the need to evidence outcomes, progress towards this varied considerably.

Some CSPPs had developed new templates for stakeholders to use to monitor outcomes. Several CSPPs had also begun reporting into new channels (e.g. the Council’s Social Work Services Committee or contributing to the Children Services Plan Joint CSP annual report).

The progress templates included in CSP annual reports (described in Section 2: Secondary data analysis) also revealed that many CSPPs planned to use established measures to demonstrate progress in health and wellbeing outcomes. This included the Scottish Government’s core wellbeing indicators (a list of potential indicators to measure wellbeing, that are part of Scotland’s children, young people and families outcomes framework; Scottish Government, 2023) or the Stirling Children’s Wellbeing Scale (a standardised scale to measure emotional and psychological wellbeing in children; Liddle and Carter, 2015). Whilst some CSPPs were planning to monitor the number of referrals onto crisis level support to assess whether the intervention that had been developed through WFWF was reducing referrals.

Most CSPPs primarily collected qualitative outcomes data informally, such as reviewing comments from service users during support sessions, rather than through formal data collection (e.g. depth interviews or structured case studies). In some cases, frontline practitioners collected qualitative data from parents after group WFWF support sessions to assess the effectiveness of the support. One CSPP described plans to gather qualitative data from parents who declined referral to better understand and address barriers to uptake.

CSPPs also utilised management data from a range of partner organisations to report on the performance and outcomes of their WFWF activity. For example, the CSP annual report data for one CSPP highlighted an increase in the number of care-leavers in education, training, and employment between 2021-2022 and 2022-2023, indicating a positive impact of their support for this group.

The use of data in strategic decisions

CSPPs had varying levels of confidence in using collected data for strategic decision making. CSPPs with more advanced analytical capabilities had pre-existing data and performance groups to manage data collection and establish its use in their planning and strategy. These groups were given oversight of all WFWF support, facilitating data interpretation across different services and informing support service decisions (see Figure 14).

Figure 14 CSPP Spotlight: Aberdeen City

Using monitoring data within strategic meetings to help cross-agency decision making.

Aberdeen City's Children Services Board (CSB) holds multi-agency meetings where partners, including children’s services, health, and education, share and monitor data. This practice enhances cross-partner comprehension of service impacts and challenges. Sharing data helps partners grasp how different support services interact and address issues such as service overload and its effects on other services. This understanding then informs strategic decisions aimed at fostering more collaborative and holistic family support.

Through consistent monitoring and reviewing data, CSPPs who were more established in terms of monitoring processes had become more agile, adapting their support offers in response to emerging evidence. For example, one CSPP reported that they had removed funding from community activity organisations where their data did not show evidence of impact on children, young people, and families in the local area.

Some strategic leads and local WFWF leads reflected that, although they collected a lot of data, they had been limited to informally using feedback from children, young people, and families to shape the design of their WFWF delivery. In other words, feedback collected was not systematically analysed and interpreted but used in an ad-hoc way to inform decision-making. Going forward, they expressed ambition to use feedback from children, young people, and families collected to shape their WFWF support offer in a more formal and systematic way.

Monitoring whole system change

In addition to monitoring family outcomes, CSPPs were asked to monitor system change across the partnership. Initially, creating monitoring approaches for system change was a key task for all CSPPs but progress towards establishing systems was varied. Some CSPPs were still in the planning stage and had not embedded systems for monitoring systemic change, whilst others had established approaches, but their use was inconsistent.

Approaches to monitoring system change varied by activity, from understanding the scope and complexity of their family support system (for example, the links between support, organisations, and any gaps), to defining which system they were attempting to change and setting questions to help them understand how change was happening, or will happen, within that system.

One CSPP reported revising their commissioning requirements for new WFWF activity to include enhanced monitoring data collection, so that they could use data across organisations to monitor the whole system. Following this revision, new services would need to report their impact against the CSPP’s devised ‘five pillars of WFWF’, which they were in the process of operationalising at the time of writing their CSP annual report. The CSPP lead believed this would help the CSPP to capture evidence of WFWF outcomes in new services.

CSPPs had lower confidence in monitoring intangible activities like collaboration between partners, service integration, and shared accountability. They engaged in ‘thinking work’, but there was no evidence of data being collected or analysed around these aspects of system change. A challenge for any system change initiatives is how to evidence contribution to the initiative, and CSPPs were uncertain about the expectations and how to set up systems to achieve this.

“The difficulty is how can you prove it is this money that made the difference because there are a whole load of other things going on at the same time…we can pull the stats [e.g. on level of collaboration between partner services], but can you say as a direct result of this funding stream this happened?”

Strategic Lead