Attainment Scotland Fund 2022-2026 evaluation: analytical plan - year 2 2023 -2024

Appendix C: Impact Feasibility Study- Summary

Introduction

The ASF Evaluation Analytical Plan for Year 1 outlined the aim of taking a collaborative approach to assessing the feasibility of using quantitative and qualitative methods of assessing impact, and to develop an impact evaluation plan for Years 2 to 5 of the Evaluation.

A feasibility study was therefore undertaken in summer/autumn 2023 to carry out the assessment of different approaches and to recommend options to take forward. The study consisted of six phases with an iterative approach including identifying impact evaluation questions; mapping existing evidence and data gaps; seeking stakeholder views; and finally developing potential approaches and options for the impact evaluation.

The feasibility study set out to examine a wide range of potential evidence sources and methodologies. However, there was a particular emphasis placed at the outset on existing quantitative data, data linkage, and potential longitudinal approaches; and a qualitative focus on the views and experiences of children and young people, and those working directly with them.

Impact Evaluation Questions

From the full set of evaluation questions included in the ASF Evaluation Strategy, the impact feasibility study identified the following individual evaluation questions for inclusion in the impact evaluation:

• To what extent did the fund contribute to a closing of the attainment gap between the most and least socio-economically disadvantaged children and young people, in line with stretch aims?
• To what extent did the fund contribute to an education system which encourages, reflects and values the breadth of achievements that contribute to improved outcomes for children and young people?
• To what extent did schools, local authorities and other stakeholders, including pupils, families and communities feel there had been progress towards achieving outcomes?
• To what extent was there improvement in children and young people's readiness to learn through focusing on engagement, attendance, confidence and wellbeing?

Mapping evidence to evaluation questions

Evidence gaps and areas for further exploration were identified in the mapping phase. Based on the evidence/evaluation question mapping undertaken during the feasibility study, it was clear that there are evaluation questions that have a number of existing sources, and those where there is an evidence gap that needs to be addressed. These included:

A number of existing evidence sources/indicators provide data on progress towards closing the poverty-related attainment gap. Several of these are already used in the evaluation, but others – such as PISA and the Health and Wellbeing Census – can be further utilised and existing NIF measures can be further analysed for more granular insight.

There are evidence gaps around perception data on the extent to which the Fund is contributing towards closing the poverty related attainment gap, and the factors supporting this, as well as the extent to which the full breadth of achievement is supported by the Fund.

Existing evidence sources need to be utilised and new source developed to evaluate improvement in children and young people's readiness to learn.

Feedback from stakeholders

A key element of the approach to undertaking the impact feasibility study involved working with a sub-group of the ASF Evaluation Advisory Panel. This engagement underpinned the whole approach and was particularly valuable in eliciting stakeholder views, including:

• Utilise existing reporting and evidence where possible to avoid survey fatigue.
• Interrogate data to understand who is progressing and in what circumstance.
• Evaluate at the macro (programme) level and micro (school) level, using case studies to augment national picture analysis.
• Prioritise research with children and young people, including those who are 'hard to reach', those who may not be engaged in school, and those who are care experienced.
• Engage with the National Programmes who have established networks with children and young people.
• Ensure that 'story telling' is part of the evidence that is developed, in order to detail the 'why' and 'how' of change.
• Ensure that research with schools reaches past the headteacher and into the classroom.
• Focus the evaluation on improvement – and what is driving this – rather than purely focusing on the attainment gap.

Impact Evaluation Approach and Options

Based on the different phases of the feasibility study – including examining existing data/evidence, other UK evaluation approaches, and feedback from a range of stakeholders – a set of potential options for the impact evaluation have emerged. It is unlikely that all options can be addressed to the same degree in the impact evaluation, however they provide a guide to how activities and timelines should be designed.

These options are organised based around four levels which have beenidentified through the feasibility study as important for the impact evaluation. Figure 1 below outlines the four levels.

It will be important that the impact evaluation includes elements across each level, which will be brought together into a coherent whole to evaluate the extent of change, the contribution the Fund is making towards this change, and (if possible) why the change is occurring.

Mixed method approach

A mixed method approach will be required for the impact evaluation, combining quantitative measures with national and local level qualitative measures. This approach is recommended for a number of reasons, including:

• The impact evaluation questions included in this feasibility study will require both quantitative and qualitative evidence to address them;
• Learning from UK evaluations of similar programmes demonstrated the importance of mixed method approaches;
• The ASF Evaluation Advisory Panel, which provided their views and expertise for this feasibility study, stressed the importance of combining quantitative and qualitative evidence to understand the change that is being achieved and what is contributing towards it.

Next Steps

The next steps are to bring a range of options together into one coherent approach to the impact evaluation. This would take a mixed methods approach to gathering data across all four levels (core measures/wider evidence/primary data at macro/national and micro/local).

A cross-sectional approach will be developed which will involve undertaking primary research at different points in time to understand the impact of the Fund. This could seek to incorporate some longitudinal elements at repeated time series. This will include primary research at the national (macro) and local (micro) level with a wide range of stakeholders, including children and young people, practitioners and wider stakeholders. The primary research will be supported by analysis of core NIF measures and wider evidence, including evidence from documentary analysis, regular surveys and evidence from partners.