Attainment Scotland Fund 2022-2026 evaluation: analytical plan - year 2 2023 -2024

Second annual analytical plan for the Attainment Scotland Fund (ASF) evaluation 2022 to 2026 covering the academic year 2023 to 2024. This is a companion document to the ASF evaluation strategy 2022 to 2026 published in November 2022.


Analytical Plan Year 2 (2023/24)

Purpose of the document

This document outlines the second annual evaluation plan for the Attainment Scotland Fund (ASF) 2022-26 and covers 2023-24, the second year of the refreshed Scottish Attainment Challenge. This is a companion document to the ASF Evaluation Strategy 2022-26, published in November 2022. The first Analytical Plan was published in February 2023. [1]

The document is intended to be accessible to all those with an interest in education in Scotland and beyond; including teachers, support workers, parents/carers, public bodies, and third sector organisations.

The ASF Evaluation

The ASF evaluation aims to provide learning about the overall implementation of the refreshed ASF and the extent to which progress has been made towards meeting intended outcomes articulated in the Scottish Attainment Challenge Logic Model in support of the refreshed programme mission.

The Evaluation Strategy for the Attainment Scotland Fund 2022-2026 [2] sets out the broad terms of the Scottish Government's approach to evaluating the ASF during this parliamentary term. The Strategy outlines four key strands of evaluation focus across the multi-year evaluation:

  • Process evaluation to consider implementation of the refreshed Attainment Scotland Fund, including Strategic Equity Fund, Pupil Equity Fund and Care Experienced Children and Young People Fund;
  • A thematic strand of evaluation which will respond to emerging system priorities and consider 'what works, for whom and in what circumstances'.
  • Reporting on the National Improvement Framework (NIF) measures;
  • Evaluation of the impact of ASF, through a combination of qualitative and quantitative measures.

This Analytical Plan

This second annual plan, continues a refreshed approach to evaluating progress in closing the poverty related attainment gap through the Attainment Scotland Fund. Priorities in the Analytical Plan include:

  • Continuing to evaluate the implementation of the refreshed Scottish Attainment Challenge through research with a wider set of stakeholders.
  • Quantitative reporting on core NIF Measures.
  • Implement an impact evaluation plan following publication of the Impact Evaluation Feasibility Study undertaken in Year 1.
  • Publish three thematic evaluation reports on: readiness to learn; families and communities; children and young person voice. Plans for year two will include a focus on learning and teaching outcomes from the logic model.

Progress of the ASF Evaluation in Year 1 (2022/23)

We have worked with the Evaluation Advisory Panel[3] throughout the first year of the new Evaluation Strategy taking a flexible, adaptive, and responsive approach to building an evidence base. In the first year we have seen some adaptations and changes to the Plan which are noted in Table 1.

The key research activities/outputs from Year 1 were:

  • A report on the implementation of the refreshed Scottish Attainment Challenge based on a survey of local authority attainment challenge leads.
  • Publication of a report that summarises evidence around improvement in attainment and health and wellbeing, and the gap between pupils from the most and least deprived areas.
  • Collaboration with National Programmes on engagement with Children and Young People on their views on readiness to learn.
  • Collaboration with Education Scotland on the theme of Families and Communities.
  • An impact evaluation feasibility study in collaboration with the Evaluation Advisory Panel.
Table 1: Progress of the ASF Evaluation in Year 1

Evaluation strand

Planned activities

Progress

Process

  • Review of existing evidence.
  • Survey of Local Authority Scottish Attainment Challenge Leads.
  • Interviews and focus groups with key stakeholders and partners.

Thematic

Thematic evaluation and development activity including:

  • Review of existing evidence on priority themes: readiness to learn; families and communities; engagement in decision-making.
  • Literature review on 'readiness to learn' to inform design of primary research in Year 2.
  • Scope, undertake and publish a small-scale study on engaging and supporting families and communities in the context of the Scottish Attainment Challenge.
  • Work with evaluation partners to explore 'what works and what could be improved, for whom and in what circumstances' in engaging children and young people in decision-making/voice in the context of the Scottish Attainment Challenge.

Working in collaboration with Education Scotland who have Families and Communities as the focus of their fourth Triannual Report[4]. Given that we have access to evidence collected from local authorities for Triannual Report 4, we made the decision to utlise this instead of commissioning case study research.

A thematic summary report and three separate companion reports (for each thematic area) will be published in February 2024. This will have particular benefits in terms of highlighting the interdepencies between the thematic areas and articulating the cross cutting themes.

National Improvement Framework (NIF) measures reporting

  • Undertake analysis on the available data for NIF measures related to the 2021/22 academic year including 11 original key measures plus two new measures identified in consultation exercise.
  • Output: Publish a standalone summary in April 2023.

The National Improvement Framework (NIF) measures reporting published June 2023

Impact

  • Work with stakeholders and partners in existing groups to develop scope/approach.
  • Undertake activities to assess existing and new approaches to measuring impact.
  • Output: Publish impact evaluation plan for Years 2 to 5.

Impact Feasibility Study completed

Evaluation priorities for 2023/24 school year

Our priorities for 2023/24 are to continue to undertake a programme of analytical activity in support of the new Evaluation Strategy, with a focus on taking forward recommendations from the Impact Feasbility Study, reporting on the core NIF measures, continuing aspects of the process evaluation and ongoing exploration of existing and newly identified thematic areas.

Process: The evaluation will continue to focus on the implementation of the refreshed Scottish Attainment Challenge and new funding arrangements, developing insights on issues such as funding, guidance, Stretch Aims, planning and outcomes. In year 2, the focus of research activities will be schools and a wider set of stakeholders, which widens the focus from the local authority level in Year 1.

Impact: An Impact Evaluation Feasibility Study was undertaken in collaboration with the Evaluation Advisory Panel in Year 1. The study provides an in-depth consideration of the most promising potential approaches to assessing impact across Years 2 - 5. A summary of the process and options considered are contained at Appendix C. In year two, we will progress the development and initial implementation of a plan for impact assessment, in line with impact feasibility study recommendations.

NIF Attainment and Health and Wellbeing Measures: In order to maintain trend data and comparability, the core NIF measures continue to form the basis of quantitative reporting in Spring 2024 on the attainment gap. We will explore the feasbility of an expanded analysis of the NIF Attainment and Health and Wellbeing Measures.

Thematic: Analytical activity in Year 1 sought to explore 'what works, for whom and in what circumstances' on a number of priority themes including readiness to learn, families and communities, and children and young peoples engagement in decision-making/voice. We will publish a summary report and three separate companion reports on the thematic areas in February 2024. Whilst we continue to explore the thematic focus going forward with the Evaluation Advisory Panel, plans for year two will include developing a focus on learning and teaching outcomes from the logic model, and a continued focus on readiness to learn and children and children and young peoples voice in decision making. It is proposed that families and communities thematic focus will be complete in year two with the publication of the thematic report.

Our specific areas of activity are:

Table 2: Evaluation Activities 2023/24

Evaluation strand

Objective

Activities

Process

To provide learning on what worked well/what could be improved in the process of implementing ASF overall and at the funding stream level (Strategic Equity Fund, Pupil Equity Fund and Care Experienced Children and Young People Fund).

  • Continue review of existing evidence.
  • Interviews and focus groups with key stakeholders and partners.
  • Publish the Year 2 Process Evaluation report, Summer 2024.

Thematic

To provide learning and increase the evidence base on 'what works and what could be improved, for whom, and in what circumstances' at the thematic level:

  • Readiness to learn
  • Families and communities
  • Engagement in decision-making/Voice (children and young people, families and communities)
  • Learning and teaching

Thematic evaluation and development activity including:

  • Publish a summary report on the three thematic areas with supporting in-depth reports
  • Identify areas for phase 2 of the thematic areas, readiness to learn and children and young people's voice in decision making.
  • Families and communities thematic strand will be complete in Year two with the publication of the Report in February 2024.
  • To explore learning and teaching outcomes as a new thematic focus.
  • Output: Publish Report February 2024 for readiness to learn, children's voice and families and communities.

National Improvement Framework (NIF) measures reporting

  • To assess the impact of the overall fund in improving outcomes in educational attainment and achievement and health and wellbeing and closing the attainment gap between the most and least socio-economically disadvantaged children and young people.
  • Undertake analysis on the available data for NIF measures related to the 2023/24 academic year including the core NIF Attainment and health and wellbeing measures.
  • Output: Publish report in Spring 2024

Impact

  • Work with stakeholders and partners in existing groups to develop scope/approach;
  • Undertake activities to assess existing and new approaches to measuring impact;
  • Development and initial implementation of plan for impact assessment, in line with impact feasibility study.

Section 2: Overview of the Analytical Plan

In order to address each of the four key evaluation strands over the duration of the Evaluation Strategy 2022-26, we have developed a high level overview of the analytical plan. This is outlined in Table 3 below. The detail of analytical activity during each year will be determined in line with the adaptive and flexible approach proposed in the Evaluation Strategy. The proposed activity for year 2, 2023/24 is set out in Appendix A.

Table 3: Analytical Plan - Years 1 to 5 Overview

Evaluation Strand

Year 1

Year 2

Year 3

Year 4

Year 5

Process

Implementation study

Implementation study

Embedded processes

-

Summative processes and implementation

Thematic

2.1 Readiness to Learn – scoping/ literature review

2.1 Readiness to Learn – Publication of Thematic Report

Continuation /identification of emerging themes and priorities for thematic focus

2.2 Families and Communities

2.2 Families and Communities – Publication of thematic Report

2.3 Voice

2.3 Voice Publication of Thematic Report

Continuation/identification of emerging themes and priorities for thematic focus

2.4 Learning and Teaching – Explore with EAP

Explore Learning and Teaching new thematic area

NIF Reporting

Annual reporting

Annual reporting

Annual reporting

Annual reporting

Annual reporting

Impact

Impact Feasibility Study

Development and initial implementation of plan for impact assessment, in line with impact feasibility study

Contact

Email: joanna.shedden@gov.scot

Back to top