Whole Family Wellbeing Funding (WFWF) Programme - year 2: process and impact evaluation - full report

2 Evaluation approach

Aims and approach

In September 2023, the Scottish Government commissioned IFF Research to undertake a process and impact evaluation (see Glossary in Annex 3) of the WFWF Elements 1, 2 and 3 in Year 2. This built on IFF's process evaluation of the implementation of Elements 1 and 2 in Year 1 (2022 to 2023) (Scottish Government, 2024b.

The process evaluation assesses how the WFWF was designed and delivered. It explored questions such as: was the initiative delivered as intended, what worked well or less well (for whom and why), what could be improved, and what can be learnt for related future initiatives. The Year 2 evaluation also includes an impact evaluation. The impact evaluation aims to assess the difference (if any) the WFWF has made to the outcomes set out within the logic model. It responds to questions such as: to what extent has WFWF produced the intended outcomes appropriate at this stage of delivery, for whom and in what circumstances were these results seen, to what extent can the achieved outcomes be attributed to WFWF, and what could be improved to strengthen the difference the WFWF can make in the future.

The Year 2 evaluation aimed to:

• Understand how CSPPs continued to use their WFWF allocation;
• Understand and assess the extent of achievement of the outcomes related to the four core components of WFWF based on the availability and strength of evidence, and the factors contributing to these outcomes;
• Assess the extent to which Element 3 projects align with the long-term outcomes of WFWF logic model^[3], and their delivery progress; and
• Provide evidence to inform and enable Scottish Government and CSPPs to make future improvements to WFWF policy and practice.

To fulfil these aims, IFF researchers adopted multiple methods (described below) to gather data, analyse this and assess the evidence overall.

Figure 1 below summarises the evaluation approach and timescales.

Figure 1 Evaluation approach overview

Scoping and set-up: September 2023 to February 2024

1) Inception and set-up;

2) document review, including logic model;

3) young people panel session 1.

CSPP case studies: February to August 2024

1) Strategic groups;

2) manager groups;

3) frontline staff groups;

4) children, young people and parent interviews;

5) young people panel session 2.

Final analysis and outputs: August to December 2024

1) Secondary analysis;

2) contribution and findings workshop;

3) Final report;

4) children and young people output;

5) young people panel session 3

Scoping and set-up

The evaluation began with an initial scoping and set-up phase to review the existing WFWF logic model, review the strategic documents relating to Element 3 and develop an evaluation and contribution analysis framework (in Annex 1). This helped to govern evaluation direction and inform research tool development.

To help support key elements of the evaluation, a panel of six young people with lived experience of family support (aged between 13 and 17) from two CSPPs^[4] participated in three online sessions. The first session (February 2024) explored the best approaches to recruit and engage children, young people and families in the evaluation. The second session (May 2024) explored their perspective on early findings. The final session (October 2024) involved co-design of an infographic (published alongside this report) to share findings in an accessible way for children, young people and families.

Qualitative case studies

Five of the 30^[5] CSPPs were selected as case studies. CSPP case studies were chosen to provide coverage across: amounts of funding received; whether they participate as a Collaborative Partnership through Element 2; rurality; population; deprivation rank; the rate of children on the child protection register; the types of activities outlined in their initial plans (see Glossary in Annex 3); and where CSPPs were assessed to be in terms of in transforming their support when the programme began in autumn 2022 (see Figure 3 in section ‘Understanding how CSPPs continued to use their WFWF allocation’ below).

Each case study included qualitative discussions with strategic leads, local WFWF managers, frontline practitioners, and children, young people and families. Topics covered can be found in the evaluation framework in Annex 1. These discussions were carried out between February and August 2024. See Table 1 below for a summary of fieldwork, and Annex 5: Case Study selection for details of how case study CSPPs were selected, and their characteristics.

Table 1 Fieldwork targets and completed

Note: Fieldwork with frontline practitioners and children, young people, and families was completed with fewer participants than intended due to one CSPP feeling they had not sufficiently progressed their delivery to take part.

The analysis of case study data was an ongoing, iterative process, involving continuous review of the data and emerging themes against the research objectives throughout the evaluation. Interview data from the case studies was organised into an analysis framework based on the evaluation framework (Annex 1). The research team systematically summarised the evidence and then validated it through analysis sessions, where they discussed key insights and tested emerging theories. More detail on the qualitative analysis approach is provided in Annex 5.

Secondary data analysis

The Scottish Government developed a template for CSPPs to provide annual reports on their WFWF plans (see Annex 6 for the questions used by CSPPs to develop their WFWF initial plans). The template was included as part of the annual CSP report for 2022-23 and 2023-24. The template aimed to gather information about CSPPs’ Element 1 WFWF activities to date, funding spent, the key challenges and successes CSPPs had experienced in delivering their WFWF activity and evidence of the extent of achievement towards outcomes. By the deadline set for inclusion in the evaluation analysis, annual WFWF progress templates were returned by 22 of 30 CSPPs. Of the case study areas, Fife, and East Lothian submitted their annual reports in time for the analysis; while East Ayrshire, Aberdeen and South Lanarkshire did not.

Eleven reports from the twelve Element 3 projects were submitted, using a template created by the Scottish Government. One project had not started at the point these templates were due. This template aimed to gather information on achievement of project objectives, key successes, challenges in delivery, and lessons for the future. Additionally, two documents related to progress on Element 2 were provided.

Using an analysis framework designed by IFF Research, a Scottish Government researcher entered the information provided in these reports into this framework. IFF researchers then conducted a quality assurance review, analysed this data to draw out key themes aligned to each of the research questions, and triangulated the findings with the qualitative data collected from the five case study areas outlined above.

Theory based evaluation: contribution analysis

The evaluation used a contribution analysis method to achieve the evaluation aims (see Glossary in Annex 3). This method compares the existing WFWF logic model with the evidence collected to determine if the WFWF contributed to the observed outcomes. Contribution analysis is a rigorous approach for evaluating complex systems. The contribution analysis approach employed here utilised all evaluation evidence collected (i.e. qualitative case study data and analysis of 22 WFWF annual progress templates).

The goal of contribution analysis is to create an evidence-based narrative that a person unfamiliar with the WFWF Programme would accept as a plausible explanation of the contribution of the WFWF to achieving the outcomes in the logic model. Annex 5 has more detail on the contribution analysis approach taken for this evaluation.

An assessment of the level of achievement of each outcome formed the basis of the analysis. Once all of the evidence was assembled (case study qualitative data and WFWF reports from 22 CSPPs), IFF assessed the extent to which the outcomes in the WFWF logic model had been achieved. All evidence was assessed using a critical review process by the IFF evaluation team for each CSPP, against each outcome. This process involved reviewing and discussing all sources of data, the different types of activities CSPPs have implemented, and accounted for different stages in CSPPs’ family support journey, to reach a consensus on the assessment of each outcome, for each CSPP.

The evaluation team also assessed the strength of the evidence base for each outcome, for each CSPP. This was done to provide a judgement on the strength of the data available to evidence the extent to which outcomes were (or were not) achieved.

For assessment of the strength of evidence in each of the five case study areas, evidence was considered strong where two or more quality data sources provided consistent evidence that indicated whether (or not) an outcome had been achieved. Two data sources could include different groups interviewed across the case studies and/or the WFWF annual report analysis. The strength of the evidence base for the extent to which an outcome had (or had not) been achieved was rated lower when i) evidence of achievement was based on a single data source; ii) the evidence did not align with the expected theory in the logic model (i.e. how the inputs, activities and outputs were anticipated to lead to outcomes)^[6]; or iii) when evidence from multiple sources was contradictory.

For CSPPs where only a WFWF annual report was included in the analysis, the strength of the evidence base for each outcome (for each CSPP) was assessed, based on whether and how well the reports substantiated achievement. Strong evidence was where the extent of achievement for an outcome was evidenced using data or statistics. Evidence was considered weak where reports indicated some level of achievement without providing information to substantiate it in a strong way, e.g. anecdotal feedback.

The final stage of the assessment process involved a workshop where the evaluation team reviewed and discussed each outcome in terms of the extent of achievement and strength of evidence across the WFWF programme. This involved the evaluation team considering, discussing and weighing up the evidence to collectively agree an overall conclusion about achievement for each outcome. This process determined the overall, evidence-based assessment about whether each outcome was ‘achieved’, ‘partly achieved’, ‘not achieved’, or whether the assessment of achievement was ‘inconclusive’ for the programme as a whole (Table 2 below).

Where an outcome was considered to have been achieved through this overall assessment, this did not mean this was achieved across all CSPPs. Nor did it mean that activities associated with this outcome had stopped or can/should do so. Continued achievement of outcomes may depend on ongoing delivery. Additionally, in future years of the WFWF programme, the assessment of an outcome may change, due to changes to local delivery, contextual factors, and availability of evidence.

Table 2 Overall assessment categories

Evaluation considerations

Overall, this evaluation has provided a robust and thorough assessment of the effectiveness of WFWF implementation in relation to the Funding’s goals. It has also provided an understanding of its impact on both children, young people and families, and CSPPs (including leaders, managers and family services practitioners working on WFWF-funded support). The use of contribution analysis has enabled the evaluation team to draw conclusions about whether the Funding has contributed to the outcomes observed. In other words, it has created an evidence-based narrative that a person unfamiliar with the WFWF Programme would accept as a plausible explanation of the contribution of the WFWF to achieving the outcomes in the logic model.

There are a number of important factors to consider when reading the findings in this report:

1. Given that the broader activities of Children’s Services Planning (see Section 1) in local areas aim to progress many of the same outcomes as the WFWF programme, is it not possible for this evaluation to isolate the effects of WFWF activity from this context. It was also not always possible to isolate WFWF impacts from the ever-changing landscape beyond CSPP and Scottish Government control; for example, the ongoing post-pandemic recovery, and the cost-of-living crisis impacted family needs and services during the period of the evaluation. The fieldwork topic guides and analysis of qualitative data and annual reports aimed to identify the contribution of WFWF and considered the contribution of the wider context where possible. Where evidence is solely related to WFWF, this is clearly indicated within the report.

2. Readers should keep in mind that WFWF is an ever-evolving policy and the data collected reflects a point-in-time. A key change to note is that at the point of data collection, WFWF was due to end in March 2026. However, in September 2024, the Scottish Government committed to extending the timeframe for the WFWF beyond March 2026 (Scottish Government, 2024a). Readers should keep this in mind when this report talks about timeframes for funding and sustainability beyond the end of the funding timeframes.

3. This report provides readers with a sense of the overall journey of CSPPs in their second year of implementing WFWF. There is a substantial amount of work relating to holistic family support beyond WFWF that is not covered by this report, and which may be contributing to the observed outcomes. This includes the recent statutory review of children’s services plans covering the 2023 – 2026 planning cycle, which provides an overview of the current delivery context and effectiveness of the broader children’s services planning landscape of which holistic family support is one part (Scottish Government, 2025).

4. Qualitative evidence from the five case study areas is not intended to imply prevalence but rather to illustrate the range of CSPP experiences implementing WFWF and provide depth of understanding. Findings in this report also represent achievement at the point of data collection and provide illustrative examples that are not intended to be comprehensive. Therefore, it should be noted that the qualitative findings from this evaluation may not be generalisable beyond the particular case study areas explored in this research and the particular point and time it covers. The evaluation has triangulated the qualitative evidence with analysis of annual reports (where available) to build a more comprehensive picture.

5. CSPPs were all at different stages of delivering WFWF activities during the period of this evaluation, as expected. This meant the evaluation’s level of engagement with some audiences (in particular frontline practitioners, and children, young people, and families) varied. For example, in some CSPPs, frontline practitioners did not feel able to comment or provide observations on WFWF outcomes, because activities were in early stages. Data collection was adapted, in agreement with the Scottish Government, to align with the capacity and stage of delivery in one case study area.

6. Secondary data was limited by the availability and quality of data provided in Element 1 annual reports, which varied across CSPPs. Annual reports were returned by 22 (of the possible 32) local areas by the deadline for inclusion in evaluation analysis. This return rate for CSP initial plans meant that data, especially relating to the extent of achievement of WFWF early outcomes, was incomplete. Therefore, again, readers should not generalise conclusions across all CSPPs. The quality and completeness of information provided in the CSP annual reports also varied, and this limited the scope of analysis. Information variability included limited detail or explanation (e.g. activities delivered, consultation groups convened); and limited or unclear evidence of the extent of achievement of outcomes (e.g. presented narrative on outcomes expected to be achieved instead of actual achievement to date and/or did not provide supporting evidence for an outcome).

7. The quality and completeness of the Element 3 project reports also varied, and limited the scope of analysis, particularly as not all projects had begun to implement activities. Information variability included limited detail or explanation (e.g. activities delivered and how they linked together). As expected, given the early stages of delivery across all Element 3 projects, there was limited evidence on outcomes achieved to date.

8. Data available about Element 2 (including both qualitative and secondary) was limited. This was due to some of the activities with collaborative partnerships having not progressed as planned within some areas.^[9] Additionally, evidence from secondary sources was limited to the amount of detail provided by local areas within their WFWF templates. Where evidence for Element 2 (including the Learning into Action Network) has been collected, it is included where relevant in this report.

Structure of this report

The remainder of this report discusses the findings from the Year 2 evaluation. Section 3 explores how CSPPs have continued to allocate WFWF resources, highlighting the progress in their journey toward delivering transformative support and their engagement with the Scottish Government’s support. Sections 4 through 7 discuss the outcomes and key factors influencing each of the WFWF’s core components: improving availability and access; fostering leadership, workforce development, and cultural change; engaging children, young people, and families in service design; and implementing a whole systems approach. Section 8 discusses how CSPPs monitor and evaluate the performance of the WFWF. Finally, Section 9 concludes the report with a set of recommendations developed by IFF Research for both the Scottish Government and CSPPs to support future progress.