Attainment Scotland Fund - 2022 to 2026: evaluation strategy

Section 2

Towards a new evaluation approach

Scottish Government and Education Scotland have worked together to undertake a comprehensive review of the ASF Evaluation Strategy to ensure that the evaluation continues to assess progress towards closing the poverty related attainment gap within the context of the new Scottish Attainment Challenge Mission and offers insights into the operation of the refreshed programme.

The new evaluation approach is underpinned by the revised Logic Model, and builds on learning from the previous evaluation. It continues to address aspects considered within the first five years of the Attainment Scotland Fund evaluation, and identifies a number of further areas for evaluation focus.

Table 1: Summary of key changes in evaluation approach

1. A multi-year evaluation strategy, with shorter, more frequent publications scheduled;

2. An in-year rather than retrospective evaluation approach.

3. A broadened scope for the evaluation, to include the Care Experienced Children and Young People Fund, and interaction between National Programmes and SAC programme. A thematic evaluation approach to interventions will also be in scope;

4. Widen the range of stakeholders involved in the evaluation, including the establishment of an Evaluation Advisory Panel to embed stakeholder collaboration in the implementation of the ASF Evaluation Strategy.

The sections below provide an overview of key aspects of the revised approach.

Multi-year evaluation strategy

The evaluation cycle is moving from an annual to a multi-year approach, which aims to build an evidence base over time, offering the opportunity to explore different aspects of the programme over points in time. This approach offers the scope to be more innovative, adaptive and flexible, pursuing key themes as they arise.

Some aspects of the evaluation, such as the NIF measures, will still be reported on annually or on a more regular basis. Other aspects will be considered across the five year period, with interim and summative reporting built in to the multi-year approach.

Figure 1 below provides an outline model of the multi-year evaluation strategy for illustrative purposes. As this indicates, four strands of evaluation focus are proposed:

• Process evaluation to consider implementation of the refreshed Attainment Scotland Fund, including Strategic Equity Fund, Pupil Equity Fund and Care Experienced Children and Young People Fund;
• A thematic strand of evaluation which will respond to emerging system priorities and consider 'what works, for whom and in what circumstances'. An early thematic focus on engaging families and communities is planned, for example, with other themes, such as a focus on the cost of living crisis, emerging in response to system needs;
• Reporting on the National Improvement Framework (NIF) measures;
• Evaluation of the impact of ASF, through a combination of qualitative and quantitative evidence.

Detailed analytical plans will be developed on an annual basis.

Moving from retrospective to in-year evaluation approach

In order to provide more timely evidence in line with the intention to enable insight into the current year of the programme to support ongoing improvement, the approach will focus on evaluation in 'real time' rather than retrospective evaluation of the previous school year^[1]. This will be supported by shorter, more frequent publications in place of the previous retrospective annual reporting format.

Evaluation Scope

The scope of the initial evaluation focused on the Challenge Authorities Fund, Schools Programme Fund and Pupil Equity Fund. The revised scope will include the Strategic Equity Fund and Pupil Equity Fund, and will be broadened to include the Care Experienced Children and Young People (CECYP) Fund. How the National Programmes interact with the Scottish Attainment Challenge (such as partnership working) will also be in scope.

Whilst individual interventions^[2] for equity are outwith the scope of the evaluation, as it would not be feasible to undertake a rigorous evaluation of the large number of interventions, evaluation at the thematic level (for example, families and communities) is within scope.

Evaluation Advisory Panel

In order to ensure the views of external stakeholders are included, an Evaluation Advisory Panel comprised of membership across key stakeholder groups has been established to ensure external views, experience and expertise are embedded in the implementation and ongoing delivery of the ASF Evaluation Strategy. The Panel operates similarly to a Research Advisory Group throughout the duration of the Scottish Attainment Challenge Programme. The panel meets at key stages in the implementation of the Evaluation Strategy and operates in an Advisory capacity which supports Scottish Government Analysts and the internal working groups comprising Scottish Government and Education Scotland.

Guiding principles

A initial set of guiding principles to inform the development of the ASF Evaluation Strategy have been drawn together:

• Informed by emerging theory and practice, including relevant evidence on improving outcomes for children and young people.
• Emphasis on providing learning for the system and policy makers through regular reporting and dissemination to support utilisation of emerging evidence as it becomes available.
• Evidence closer to 'real time' to ensure learning supports system change/improvement through regular feedback (e.g. seeks to ensure early evidence on implementation of Strategic Equity Fund, the Framework for Recovery and Accelerating Progress, Stretch Aims etc).
• Emphasis on developing a body of evidence which continues to 'tell the story' of the development of the Scottish Attainment Challenge, ensuring there are points of continuity between initial Scottish Attainment Challenge programme and the refreshed programme.
• Ensures inclusion of wider stakeholder voices, including children and young people, families, practitioners, and communities as well as reflecting the increased emphasis on partnership working with youth sector.
• Responds to issues raised in relevant national audits and reviews, and takes into account the broader policy context such as The Promise, education reform and changes to the curriculum.
• Mixed methods approaches, drawing on wider expertise and ensuring a mix of in-house, collaborative and commissioned pieces.
• Focuses on the programmatic level, but seeks to contribute to the evidence base on 'what works/for whom/in what circumstances'.