Whole Family Wellbeing Funding (WFWF): year 1 - process evaluation - interim report

4. WFWF monitoring and outcomes

Introduction

This section presents an overview of CSPPs' plans for monitoring outcomes and describes what short-term outcomes they intend to achieve by the end of Year 1 of the WFWF.

Within the initial plan guidance, CSPPs were advised that arrangements must be put in place to ensure that monitoring and reporting contributes to evaluation of the funding and demonstrates progress against the outcomes sought. This included:

• Providing baseline information through Children's Services Plans and annual reports, supplemented by the inclusion of information on the CSPP's evolving strategic approaches to delivering holistic family support through use and spend of this funding.
• Developing local measures of success criteria, key milestones and quality indicators, aligned with the evaluation framework developed for the funding.
• Contributing supplementary information required by the evaluation framework developed for the funding, where possible as part of reporting on Children's Services Plans.

Monitoring performance

CSPPs understand the importance of evidencing their progress and outcomes for WFWF, beyond Year 1. Most CSPPs included plans for monitoring and/or evaluation activity in their initial plans beyond the evaluation delivered by IFF. This ranged from CSPPs using existing data to evidence outcomes to collecting baseline data. All CSPPs intend to use quantitative management information and statistics, and some are committed to capturing stories from children, young people, and families and practitioners, through qualitative evidence.

Due to the diverse range of activities CSPPs have planned or undertaken, specific indicators to collect evidence on WFWF support vary between CSPPs. Examples of indicators CSPPs plan to use for monitoring progress are health, social and wellbeing outcomes for children, young people, and families, and delivery performance of support and services.

Some CSPPs have undertaken activity, or plan to, to understand the complexity of their family support system (for example, the linkages between support and any gaps). There was also evidence of some CSPPs starting to define the system they are attempting to change and setting questions to help them understand how change is happening, or will happen, within that system. This activity is in recognition that monitoring performance goes beyond simply funded activities, and there is a need to evidence any system change.

Scoping how best to monitor performance and system change, including the exact metrics relevant for their WFWF activities and how to develop a monitoring system that looks across WFWF activities to include system change progress, continues to be a main activity for CSPPs. For some, their plans involve building their analysis capacity, either through hiring or freeing up capacity from analysts or data officers.

A few CSPPs have progressed with identifying their priority metrics to monitor their performance. Fife, for example, is leveraging its existing children's services data group to develop a dataset and scorecard based on the WFWF outcomes in the logic model, which a data officer will then update, analyse and report on quarterly.

Some CSPPs also planned on implementing new IT software using WFWF funding to better track and measure children and families' journeys across the system, for example, North Ayrshire, described in the Spotlight box below.

Spotlight on…

North Ayrshire were setting up a new and enhanced IT system called 'Eclipse', which would replace their existing recording system. This will allow them to track children and families' journeys across agencies, identifying if anyone re-appears in the system.

When Eclipse becomes embedded, it will provide them with richer data in the form of automatically-generated genograms on the system, which will contain details of extended family members. A genogram is a visual tool that shows a family tree and is used to give a pictorial representation of a family system. If, for example, service providers felt it was necessary to link in with a grandparent because they were identified as a protective factor and a strength, they will be able to do that more easily.

Overall, it is anticipated that Eclipse will provide agencies with more data on the work they are delivering and associated outcomes.

A CSPP is also revising their commissioning requirements for new support. New services will need to report their impact against the five pillars of WFWF, which they are in the process of operationalising. This ensures that the CSPP's ability to evidence WFWF outcomes is built into new services.

CSPPs identified some risks to their ability to effectively measure their progress towards achieving WFWF outcomes. A risk is that practitioners may not have the time and skills to record information needed, and to collect high-quality, qualitative evidence of children, young people, and families' experiences.

Another risk relates to the nature of WFWF as a systems-wide change initiative; it requires CSPPs to monitor many, different services and support and to be able to link them together to assess their combined impact.

CSPPs are data rich and in a good position to monitor outcomes like child-level social and health outcomes because this is required for annual reporting, and delivery of WFWF activity (e.g., what is delivered, to who and when) because this is required for service and financial monitoring. Some intended outcomes for WFWF activities are less tangible. CSPPs are less familiar with, and thus less confident, with how best to measure collaboration between partners, service integration and shared accountability across partners:

"Some of the successes are obviously going to be that much harder to actually tangibly demonstrate because it is about relationships. It is about how agencies work together. It's about how we support families in a more connected basis and some of that is harder to actually demonstrate, you know, if it's attendance levels, if it's academic achievements, then that's a much clearer process."

Strategic lead

Fife has ideas for how to capture evidence on outcomes related to partner relationships. The CSPP has plans for a workforce survey to regularly capture views on consultation, collaboration and partnership working.

CSPPs would benefit from more clarity on what monitoring performance is required of them from the Scottish Government, to ensure they are meeting expectations:

"There is no clear ask about what you're reporting on … no clear success criteria or dataset."

Manager

Since interviews were conducted with CSPPs, the Scottish Government has produced guidance and a template for reporting WFWF activity as part of the annual Children's Services Plan reporting cycle.

A common challenge for any system-change initiative is how to attribute observed changes to the initiative. WFWF is no different, and CSPPs wonder whether they are expected to attribute their performance to WFWF and if so, what the Scottish Government's expectations are on that and how best to set-up their systems to do that:

"The difficulty is how can you prove it is this money that made the difference because there are a whole load of other things going on at the same time … we can pull the stats, but can you say as a direct result of this funding stream this happened?"

Strategic lead

Intended outcomes by end of Year 1

Within the initial plans, CSPPs were asked to set out what early outcomes they anticipated achieving from WFWF Element 1 activities within the first year, and by June 2023, and specifically in relation to those outlined in the WFWF logic model. These outcomes were developed by the Scottish Government in collaboration with stakeholders, including the Family Support Advisory Group, and can be found in the WFWF logic model (Annex 1).

Table 3, overleaf, summarises WFWF short-term outcomes and the number of CSPPs who indicated in their initial plans they intended to deliver on each outcome. In summary, the most common intended outcomes were to embed holistic whole family support; (re)design whole family support; and improve children, young people, and families' access to support. The less common intended outcomes were to build transformational change; innovate; develop a holistic workforce approach; and invest locally in planning system change. The final evaluation report will discuss progress towards these outcomes.

CSPP strategic leads and managers also described intending to achieve several outcomes not included in the logic model by the end of Year 1. Half of the CSPPs planned to measure performance for the first year of WFWF delivery against the aim of getting people into post to deliver WFWF activities. This was not explicitly one of the original outcomes in the logic model, but strategic leads and managers noted that it was a required activity for delivering several of the related outcomes outlined in the logic model. For example, increased holistic whole family support service capacity – scaled and new services are integrated.

Spotlight on…

South Lanarkshire are seeking to hire 40 staff to deliver four Early Help Hubs, including family and parent support workers, service and team managers, Hub coordinators and a trauma practitioner. Recruitment is underway, with some posts filled by staff from the previous mental health care services.

At end of Year 1, they were confident the service would be populated and embedded in their Children's Services and across their CSPP governance structure and partnership approach.

When discussing children, young people, and families' outcomes, most WFWF managers focused on children, young people, and families' outcomes as individual-level outcomes for children and families, such as social and health outcomes. For example, one manager gave examples of early CSPP successes which included "One young person has not had an exclusion in the three months the pilot has been involved".

It was evident that the framing of short-term outcomes for children, young people, and families in the logic model may not be familiar to all CSPPs. They would benefit from being reminded that in Year 1, more realistic outcomes of engagement are expected, not individual-level social or health outcomes.