Whole Family Wellbeing Funding (WFWF) Programme - year 2: process and impact evaluation - full report

Annex 5: Research approach

In September 2023, the Scottish Government commissioned IFF Research to undertake a process and impact evaluation of the WFWF in Year 2, of Elements 1, 2 and 3. This built on IFF's process evaluation of the implementation of Elements 1 and 2 in Year 1, undertaken during 2022-2023.

Analysis of CSP annual reports

As part of the local authority obligations under the Children and Young People (Scotland) Act 2014, Children’s Services Planning Partnership were required to prepare a Children’s Services Plan for 2020-2023. The CSP annual reports are a summary of progress for that given year. In Year 1 of the evaluation, alongside the statutory CSP annual report, IFF and the Scottish Government worked together to develop a template specifically focussed on the WFWF. It was not mandatory to complete this. 22 CSPPs chose to submit a report in 2023-24. These were analysed by Scottish Government into a bespoke framework designed by IFF.

Secondary analysis was limited by the availability and quality of data provided by CSPPs. 22 CSP annual reports were provided to IFF for analysis. The information provided in the CSP annual reports was limited in some places which reduced the scope of analysis that could be undertaken. Common challenges included:

• Lack of detail or explanation, such as not providing sufficient description of activities e.g. description of programmes or groups convened.
• Lack of or unclear evidence of the extent of achievement towards outcomes. For example, failing to explain through what mechanisms an outcome had been achieved or providing robust data to substantiate achievement.

Case Study selection

Six CSPPs took part in the Year 1 evaluation. These were Glasgow City, East Ayrshire, North Ayrshire, Aberdeen City, Fife and South Lanarkshire. CSPP case studies were chosen to provide coverage across: funding amount, whether they participate as a Collaborative Partnership through Element 2 (East Ayrshire and Glasgow City both received Element 2 support), area type (predominantly urban or rural), population, deprivation rank, needs of the family population (defined by the rate of children on the child protection register), type of activities outlined in their initial plans, and holistic whole family support journey stage (whether CSPPs were at an early, moderate or advanced stage in transforming their support) (for more details, see Scottish Government, 2024b).

Due to budget constraints and the need to focus data collection to capture the depth required for contribution analysis, five CSPPs were asked to take part in the Year 2 evaluation. Fife, East Ayrshire, Aberdeen City and South Lanarkshire continued as case studies from Year 1. Through mutual agreement, Glasgow City and North Ayrshire did not continue as case studies. This decision was predominantly due to the changing nature of their WFWF activity. East Lothian was selected to take part as the final Year 2 case study, primarily because they were receiving Element 2 collaborative support.

Interviewed population by case study

The table below shows the numbers of interviews conducted with each audience within each case-study CSPP.

Children, young people and families participant profile

The table below shows the profile of children, young people and families who took part in qualitative interviews across the case-study CSPPs.

Qualitative data management and analysis

All discussions were recorded with consent, stored on IFF’s secure drive in a folder to which only designated team members had access, and written up thematically by the researcher using a bespoke analysis framework.

IFF’s qualitative analytical approach is informed by grounded theory and structured by the research questions but builds upwards from the views of participants. It is continuous (during and after fieldwork periods, and between phases) and iterative, moving between the data, research objectives and emerging themes.

The analysis framework was structured by key research questions and data entered into relevant cells including direct quotes and examples. It included columns for the researchers’ own interpretation and key conclusions. Data was then coded, looking for patterns by theme within and across interviews.

The analysis process consisted of two key elements. Firstly, a process-driven element using matrix mapping framework technique. Recordings of discussions coded and systematically summarised into an analytical framework organised by issue and theme. Secondly, an interpretative element focussed on identifying patterns within the data and undertaking sub-group analysis. Researcher analysis sessions, led by the director, during which the team came together to discuss and test emerging themes and insights, were conducted after each phase and used to support interpretation of the data.

All evidence sources were analysed in their own right; the analysis process then went on to compare and contrast the findings across evidence sources. During this, the quality of evidence was weighed up. Any inconsistencies between different data sources were explored and explained. Where there were competing findings by evidence source, stronger evidence was considered over evidence with gaps.

Additional detail: Contribution analysis

Due to the evaluation timings and the need for Scottish Government to redevelop the existing logic model, IFF and Scottish Government agreed to undertake a variation of contribution analysis that did not require an updated logic model. Contribution analysis was embedded throughout the research design. For example, the design of research tools, and synthesising of evidence, explicitly looked at how each of the WFWF four core components led to activities that were attributed to outcomes. In practice, this involved using the data collected from CSPPs through the evaluation to assess the extent of achievement of the outcomes in the WFWF logic model. The IFF evaluation team then traced backwards from those changes to see what has led to those changes and see whether that aligned with the theory in the WFWF logic model.

There were six key steps to contribution analysis that were undertaken as part of this Year 2 WFWF evaluation.

Step 1: Scoping. Development of narrative and hypothesis through inception meeting, Element 3 document review, and review of the logic model and logic model workshop with key Scottish Government stakeholders (November 2023).

Step 2: evaluation and contribution analysis framework. Involved the development of an evaluation and contribution framework to function as the evaluation plan for the rest of the evaluation (see Annex 1).

Step 3: data collection. Involved primary data collection from the five case study areas and triangulating this with secondary data.

Step 4: assessing the evidence and challenges to it. Part-way through data collection, a contribution workshop took place (June 2024), involving Scottish Government colleagues and the evaluation team. During this workshop, the strengths and weaknesses of the logic model were reviewed, considering the available evidence gathered to date and the relevance of other influencing factors.

Step 5: testing and revising. Involved undertaking remaining data collection to further unpack assumptions around how the funding influences outcomes, the main drivers of change and associated risks to enable the testing of key contribution claims or areas in the logic model that were weakest or had uncertainties. Moreover, remaining case study interviews aimed to fill any gaps in the evidence.

Step 6: synthesis and reporting. Key findings from across the data collection were brought together to analyse and address each evaluation question and to underpin the analysis and ‘testing’ of the logic model. All sources of evidence were used to explore each core component of the logic model, enabling the triangulation of findings, making sense of potentially contradicting evidence, and providing an evidence-based assessment (as described in Section 2 and 9) about whether each outcome was ‘achieved’, ‘partly achieved’, ‘not achieved’, or whether the assessment of achievement was ‘inconclusive’.