Attainment Scotland Fund evaluation: third interim report - year 4

Chapter 1 Introduction and Methodology

Introduction

1.1 The Attainment Scotland Fund (ASF) evaluation began in 2015 and follows the duration of the Scottish Attainment Challenge (SAC). An evaluation of the first two years of the ASF was published in March 2018, with the Year 3 (2017/18) evaluation published in June 2019.

1.2 This chapter provides detail on the aims of the evaluation, the overall approach and the structure of the Year 4 evaluation report.

Background

1.3 The SAC was launched by the First Minister in February 2015 to help close the poverty-related attainment gap. It is underpinned by the National Improvement Framework, Curriculum for Excellence and Getting it Right for Every Child. Backed by the £750 million ASF over the course of this Parliament, it prioritises improvements in literacy, numeracy, health and wellbeing of those children adversely affected by poverty in Scotland's schools. Achieving excellence and equity in education are the key aims.

1.4 The SAC leads system change through a tripartite shared leadership of national government, local government and the executive improvement agency, Education Scotland. The core aims of the programme are to support and empower headteachers, schools, local authorities and their partners to develop their own approaches, reflecting their own local circumstances.

1.5 The SAC has the following main strands:

• Pupil Equity Funding (PEF) provides £120m directly to schools for headteachers to use at their discretion on initiatives that they consider will help close the poverty related attainment gap. Over 95% of schools in Scotland have been allocated funding for pupils in Primary 1 through to third year of secondary school, based on the estimated numbers of pupils registered for free school meals.
• The Challenge Authority and Schools Programmes provide additional resource to nine local authorities, and 73^[1] schools outwith those local authorities with the highest concentrations of deprivation. Each Challenge Authority/Schools Programme school receives funding and support to deliver improvement plans focused on literacy, numeracy and health and wellbeing to tackle the poverty related attainment gap.
• Care Experienced Children and Young People (CECYP) funding for targeted initiatives, activities, and resources, designed to improve the educational outcomes of this group.

1.6 These strands have developed over the period of the SAC. Challenge Authority and Schools Programme were the initial funding streams which commenced in 2015, followed by the introduction of PEF in 2017/18.

1.7 Finally, the CECYP grant was introduced in 2018/19. Additionally, the SAC supports a number of national programmes, including: staffing supply and capacity; professional learning and school leadership; investment in Regional Improvement Collaboratives; and a small number of third sector-led initiatives.

1.8 The Scottish Government's 2019/20 Programme for Government included a commitment to continue funding the SAC at current levels in 2021/22, reinforcing their sustained focus on closing the poverty-related attainment gap.

Methodology

1.9 The evaluation aims to provide learning about the overall implementation of the ASF and the extent to which the aims of the ASF have been met.

1.10 The evaluation has the following objectives:

• Assess the impact of the overall fund in improving attainment and health and wellbeing and reducing the difference between pupils from the most and least deprived areas;
• Assess the extent to which the further aims of the Fund have been met: promote capacity for data-based self-evaluation and improvement, and encourage collaboration between schools and local authorities;
• Provide learning and increase the Scottish evidence base of what does and does not work to improve attainment and health and wellbeing, especially of pupils from the most deprived areas;
• Provide learning on what did and did not work well in the process of implementing the Fund across participating Challenge Authorities and schools and which factors helped and hindered the fund achieving its outcomes.

1.11 It is recognised that there are limitations to the evaluation methodology, for example a focus on the specific 'interventions' being implemented by schools and local authorities and the outcomes of those interventions is out of scope. This is coupled with an approach to reporting that attempts to avoid burdening the education system with transaction by transaction analysis recognising the trust placed in the education profession to make the right judgements about how resources are used.

1.12 The current evaluation methodology aims to make the best use of existing data to support our understanding of factors that support improvement in closing the attainment gap. Whilst it remains difficult to assess the reasons behind any observed improvement in attainment or closing the poverty related attainment gap, and whether these changes have occurred as a direct result of the Fund, the evaluation continues to support our understanding of related factors and the contribution the Fund has made to the realisation of these factors.

1.13 The Evaluation Strategy of the Attainment Scotland Fund, published in 2018, sets out the intended approach to the evaluation, describing the research methods used between then and 2020 to address evaluation aims for Years 3 and 4 of the ASF^[2]. A revised set of evaluation questions were developed for Year 4 to more clearly align the evaluation to the SAC logic model. This articulates short and medium-term outcomes and the long-term outcomes for the SAC^[3]. Figure 1.2 sets out the long-term outcomes of the SAC.

Figure 1.2: Long-term outcomes for Scottish Attainment Challenge

1. Embedded and sustained practices related to addressing the impact of the poverty-related attainment gap

2. All children and young people are achieving the expected or excellent educational outcomes, regardless of their background

3. An education system which is aspirational, inclusive in practice and approaches for all including teachers, parents and carers, children and young people

4. Closing the attainment gap between the most and least disadvantaged children and young people

1.14 Evidence in relation to the ASF aims has been gathered from a range of sources and evaluated against a set of research questions. Following the publication of the Year 3 report, the logic model and research questions were revised to reflect the evolution of the SAC. The revised questions focus on the impact of approaches that schools and local authorities have implemented to address the poverty-related attainment gap, progressing from the initial focus on processes and inputs as the SAC programme has developed.

1.15 The sources used to inform progress in Year 4 of the ASF are set out below.

• Administrative data: Information gathered as part of the routine organisation of the ASF provides data primarily on the funding that local authorities and schools received from the different funding streams.

• Challenge Authority and Schools Programme progress reports: Challenge Authorities and Schools Programme Progress Reports in March and September 2019.

• Local Authority Survey 2019: Between July and September 2019, all 32 local authorities were invited to take part in an online survey. The survey explored local authority perspectives on ASF and was published in November 2019. This was the third wave of the survey. The first wave took place in 2016 with Challenge Authorities only and the second wave was undertaken in 2018 with all local authorities.

The third wave of the survey saw a move away from the 'mini survey' approach used previously to an expanded survey. This built on the findings of previous waves of the survey but in addition to considering existing themes (such as governance, funding, sustainability, PEF planning and implementation, and unintended consequences) also sought local authority perspectives of:

• developing approaches to closing the poverty-related attainment gap
• the extent to which data and evidence featured in decision making at the local level
• the extent to which the Fund increased collaboration
• factors supporting and mitigating progress towards closing the poverty-related attainment gap within local authorities

Views on planning and implementation of the CECYP Fund were also sought in the Local Authority Survey. The CECYP Fund is however outwith the scope of this evaluation report.

• Headteacher Survey 2019: In Autumn 2019 an online survey was distributed to headteachers of schools receiving Challenge Authority, Schools Programme and PEF funding. The purpose of the survey was to provide an insight into the experiences of headteachers in relation to ASF and to understand changes over time and across different respondent groups (funding stream; urban versus rural; primary versus secondary). This was the fourth year of the survey and 1,102 headteachers responded to the survey, representing a 47% response rate. The response rate increased by seven percentage points from 40% in 2018. The inclusion of all PEF-only schools for the first time in the 2019 survey resulted in approximately 1,000 more headteachers being invited to respond in 2019 than the previous year. The full report from the survey has been published online https://www.gov.scot/ISBN/978-1-83960-894-0

• Quantitative data on attainment and wellbeing: The report draws on measures published in the National Improvement Framework Interactive Evidence Report (NIFIER). Analysis focuses on patterns of attainment across Challenge Authorities non-Challenge Authorities and Scotland overall. The only change to reporting of attainment and wellbeing data for the Year 4 report is in relation to the Achievement of the Curriculum for Excellence Levels (ACEL) data. Previous reports did not include analysis of changes in ACEL data over time at local authority level as the data was still in development. However, with the publication of ACEL data for 2018/19 the 'Experimental Statistics' label has been removed and analysis of changes between 2016/17 and 2018/19 has been included.

Table 1.1: Data Sources for the ASF Evaluation

Data Source	Coverage	Years covered
		Year 1 (2015/16)	Year 2 (2016/17)	Year 3 (2017/18)	Year 4 (2018/19)
Administrative data (financial information)	All Challenge Authorities, Schools Programme local authorities and schools receiving PEF	√	√	√	√
Challenge Authority Progress Reports	All 9 Challenge Authorities	√	√	√	√
Local Authority Mini Survey	Year 1: (Challenge Authorities only) Year 3: Year 4:	√		√	√
Headteacher Survey	Years 1 and 2: (Challenge Authorities and Schools Programme) Year 3: (Challenge Authority, Schools Programme funding plus sample of PEF-only schools) Year 4: All schools in receipt of ASF funding (Challenge Authority, Schools Programme, PEF-only)	√	√	√	√
Quantitative data on attainment and wellbeing	Analysis of attainment measures set out in the 2020 National Improvement Framework.	√	√	√	√

Report Structure

1.16 This report focuses on ASF in Year 4 (2018/19) of the SAC. Findings highlighted in the report seek to show changes over the duration of the fund to date encompassing changes in Year 4 from Year 3 and Years 1 and 2.

1.17 Similar to the Year 3 report, this evaluation report considers Challenge Authority, Schools Programme and PEF funding streams and seeks to highlight any emerging differences across the three funding streams.

1.18 The report is structured around the inputs, activities, short- to medium-term outcomes and long-term impact of the ASF. This reflects the revised research questions for Year 4 which are set out in the chapter structure overview below.

Chapter 1 Introduction, Background and Methodology

Chapter 2 Inputs: Governance and Funding

• What did and did not work well in the national and local governance and support as part of the Fund?
• How much funding did local authorities and schools receive, to what extent did they consider it adequate, supplement it with other funding sources, and use it in accordance with the fund's requirements?

Chapter 3 Activities and Outputs: Approaches

• How did schools and local authorities identify, select and implement their approach for addressing the poverty-related attainment gap?
• To what extent did the selected approach aim to support pupils (and parents) from the most deprived backgrounds?

Chapter 4 Short and medium term outcomes: Perceptions of success, collaboration, use of data and evidence

• To what extent did schools and local authorities involved with the fund feel the intended outcomes of their approach had been achieved?
• To what extent did stakeholders understand, engage and further the programme aims, and why?
• To what extent has the fund encouraged collaboration, and why?
• To what extent did schools and authorities use data, analysis and evidence to drive improvements as part of the fund?

Chapter 5 Long-term outcomes: Contribution to improvement and reduction of the poverty-related attainment gap, sustainability, unintended consequences

• To what extent did the fund contribute to an improvement in attainment and health and wellbeing, and a reduction of the gap between pupils from the most and least deprived areas?
• To what extent can the focus on addressing the poverty-related attainment gap be sustained beyond the years of funding?
• Did the fund have any unintended consequences?

Chapter 6 Discussion and conclusions