Attainment Scotland Fund evaluation: fourth interim report - year 5

Chapter 1: Introduction and Methodology

Introduction

1.1 The Attainment Scotland Fund (ASF) evaluation began in 2015 and follows the duration of the Scottish Attainment Challenge (SAC). An evaluation of the first two years of the ASF was published in March 2018, with the Year 3 (2017/18) evaluation report published in June 2019 and the Year 4 (2018/19) report published in October 2020.

1.2 This chapter sets the Year 5 Evaluation Report (2019/20) in the context of the ongoing COVID-19 pandemic and provides detail on the aims of the evaluation, the overall approach and the structure of the Report.

Background

COVID-19 Context

1.3 This report covers 2019/20, a year which saw unprecedented change as a result of the COVID-19 pandemic. This academic year was in many ways a year of two parts – the period from June 2019 to March 2020, prior to the emergence of the COVID-19 virus, then from March 2020 to June 2020, the first lockdown period of school building closures and remote learning in Scotland.

1.4 The likelihood that school building closures will have an impact on the poverty-related attainment gap has been widely acknowledged. This makes it critical that the evaluation of the ASF should consider the impacts of this period on the operation and use of the fund. An Equity Audit was undertaken during the autumn of 2020 to help deepen understanding about the impact COVID-19 and resultant school building closures had on children from socio-economically disadvantaged backgrounds.

1.5 The findings of the Equity Audit highlight the importance of an ongoing, long-term and system wide focus on closing the poverty related attainment gap. They also highlight the scale and potential depth of the impacts of the pandemic, and recognise that the full extent of those may not become fully visible for some time.

SAC Context

1.6 The SAC was launched in February 2015 to help close the poverty-related attainment gap. It is underpinned by the National Improvement Framework, Curriculum for Excellence and Getting it Right for Every Child. Backed by the £750 million ASF over the course of this Parliament, it prioritises improvements in literacy, numeracy, health and wellbeing of those children adversely affected by poverty in Scotland’s schools. Achieving excellence and equity in education are the key aims.

1.7 The SAC leads system change through a tripartite shared leadership of national government, local government and the executive improvement agency, Education Scotland. The core aims of the programme are to support and empower headteachers, schools, local authorities and their partners to develop focused and innovative approaches to improving outcomes for learners, reflecting their own local circumstances.

1.8 The SAC has the following main strands:

• The Challenge Authority and Schools Programmes provide additional resource to nine local authorities, and a further 73 schools outwith those local authorities with the highest proportions of deprivation. Each Challenge Authority/Schools Programme school receives funding and support to deliver improvement plans focused on literacy, numeracy and health and wellbeing to tackle the poverty-related attainment gap. The nine 'Challenge Authorities' are Glasgow, Dundee, Inverclyde, West Dunbartonshire, North Ayrshire, Clackmannanshire, North Lanarkshire, East Ayrshire and Renfrewshire.
• Pupil Equity Funding (PEF) provides £120m each year, directly to schools for headteachers to use at their discretion on initiatives that they consider will help close the poverty-related attainment gap. Over 97% of schools in Scotland have been allocated funding for pupils in Primary 1 through to third year of secondary school, based on the estimated numbers of pupils registered for free school meals.
• Care Experienced Children and Young People (CECYP) funding for targeted initiatives, activities, and resources, designed to improve the educational outcomes of this group. This was introduced in 2018/19 with funding allocated to all local authorities based on the number of looked after children they have in their care.
• Additionally a number of national programmes have been supported for targeted work to raise attainment and improve equity, including: staffing supply and capacity; professional learning and school leadership; investment in Regional Improvement Collaboratives and a number of third sector organisations.

1.9 These strands have developed over the period of the SAC. Challenge Authority and Schools Programme were the initial funding streams which commenced in 2015, followed by the introduction of PEF in 2017/18 and the CECYP in 2018/19.

1.10 The Scottish Government’s 2019/20 Programme for Government included a commitment to continue funding the SAC at current levels for the duration of 2021/22, one year beyond the current parliamentary term, reinforcing the sustained focus on closing the poverty-related attainment gap. This evaluation report for Year 5 of ASF is published alongside an overarching report on progress towards closing the poverty-related attainment gap since the introduction of SAC, ‘Closing the Poverty-related Attainment Gap: A Report on Progress 2016 – 2021’

Methodology

1.11 The evaluation aims to provide learning about the overall implementation of the ASF and the extent to which the aims of the ASF have been met.

1.12 The evaluation has the following objectives:

• Assess the impact of the overall fund in improving attainment and health and wellbeing and reducing the difference between pupils from the most and least deprived areas;
• Assess the extent to which the further aims of the fund have been met: promote capacity for data-based self-evaluation and improvement, and encourage collaboration between schools and local authorities;
• Provide learning and increase the Scottish evidence base of what does and does not work to improve attainment and health and wellbeing, especially of pupils from the most deprived areas;
• Provide learning on what did and did not work well in the process of implementing the fund across participating Challenge Authorities and schools and which factors helped and hindered the fund achieving its outcomes.

1.13 Figure 1.2 sets out the long-term outcomes for the SAC as set out in the SAC Logic Model. A full refresh of the Logic Model and associated evaluation strategy was planned for Year 5, however, given the impacts of the COVID-19 pandemic, a process of adapting the SAC Logic Model to take account of COVID-19 was taken forward instead. This process was taken forward with Education Scotland and Scottish Government policy officials and analysts. The ASF Evaluation research questions were subsequently adapted in line with the SAC Logic Model, resulting in a sub-set of COVID-19 impact evaluation questions.

Figure 1.2: Long-term outcomes for Scottish Attainment Challenge

1. Embedded and sustained practices related to addressing the impact of the poverty-related attainment gap

2. All children and young people are achieving the expected or excellent educational outcomes, regardless of their background

3. An education system which is aspirational, inclusive in practice and approaches for all including teachers, parents and carers, children and young people

4. Closing the attainment gap between the most and least disadvantaged children and young people

1.14 It continues to be recognised that there are limitations to the evaluation methodology, for example the specific ‘interventions’ being implemented by schools and local authorities and the outcomes of those interventions is out of scope. There is also an approach to reporting that attempts to avoid burdening the education system with onerous reporting requirements, recognising the trust placed in the education sector to make considered judgements about how resources are used.

1.15 Whilst response rates to surveys and available data (Achievement of Curriculum for Excellence Levels) have been impacted by COVID-19, the current evaluation methodology continues to make the best use of existing data to inform our understanding of factors that support improvement in closing the attainment gap at the five year point of the programme.

1.16 Evidence in relation to the ASF aims has been gathered from a range of sources and evaluated against the research questions.

1.17 The sources used to inform progress in Year 5 of the ASF are set out below.

• Administrative data: Information gathered as part of the routine organisation of the ASF provides data primarily on the funding that local authorities and schools received from the different funding streams.

• Challenge Authority and Schools Programme progress reports: Challenge Authorities and Schools Programme Progress Reports in March and September 2020.

• Local Authority Survey 2020: Between December 2020 and January 2021, all 32 local authorities were invited to take part in an online survey about perspectives on the ASF. The survey response rate was lower than the 2019 survey (15 responses in 2020, compared to 28 responses in 2019), and it is likely that the survey response was impacted by the COVID-19 context. In addition, the 2019 response included all nine Challenge Authorities, whereas the 2020 response included five of the nine Challenge Authorities. The 2020 Local Authority Survey results should therefore be interpreted with some caution. For example, direct comparisons with 2019 findings, and the opportunity for considering responses by funding stream, are limited. Broad continuation/discontinuation of trends are however referenced in the text.

This was the fourth wave of the Local Authority survey. The first wave took place in 2016 with Challenge Authorities only and the second (2018) and third (2019) waves were undertaken with all local authorities.

The third wave introduced an expanded survey format. This built on the findings of previous waves of the survey but in addition to considering existing themes (such as governance, funding, sustainability, PEF planning and implementation) also sought local authority perspectives of:

• developing approaches to closing the poverty-related attainment gap;
• the extent to which data and evidence featured in decision making at the local level;
• the extent to which the fund increased collaboration;
• factors supporting and mitigating progress towards closing the poverty-related attainment gap within local authorities.
  • Quantitative data on attainment and wellbeing: The report draws on available data consistent with the National Improvement Framework measures of the attainment gap. Analysis focuses on patterns of attainment across Challenge Authorities, non-Challenge Authorities and Scotland overall. Data on attendance and exclusion is not available for 2019/20 as it is published on a biennial basis. Achievement of Curriculum for Excellence Levels (ACEL) data was not collected in 2019/20.

The fourth wave covering 2019/20 maintained the expanded survey format, but also considered COVID-19 impacts for all survey questions specifically focusing on the period of school building closures between March – June 2020.

Views on planning and implementation of the CECYP Fund were also sought in the Local Authority Survey. The CECYP Fund is however outwith the scope of this evaluation report.

• Headteacher Survey 2020: In January 2020 an online survey was distributed to headteachers of all Challenge Authority, Schools Programme schools and a representative sample of schools receiving PEF funding. The purpose of the survey was to provide an insight into the experiences of headteachers in relation to ASF and to understand changes over time and across different respondent groups (funding stream; urban versus rural; primary versus secondary). This was the fifth year of the survey and 420 headteachers responded to the survey, representing a 27% response rate. The response rate is reduced in 2019/20 which reflects the COVID-19 context, with a second lockdown and resultant school building closures. (Response rate in 2019 47%, 40% in 2018. The inclusion of all PEF-only schools for the first time in the 2019 survey resulted in approximately 1,000 more headteachers being invited to respond in 2019 than the previous year). The Headteacher Survey 2020 top-line findings were published alongside this report, with a full report scheduled for May 2021.

Table 1.1: Data Sources for the ASF Evaluation

Data Source	Coverage	Years covered
		Year 1 (2015/16)	Year 2 (2016/17)	Year 3 (2017/18)	Year 4 (2018/19)	Year 5 (2019/20)
Administrative data (financial information)	All Challenge Authorities, Schools Programme local authorities and schools receiving PEF	√	√	√	√	√
Challenge Authority and Schools Progress Reports	All 9 Challenge Authorities and all Schools Programme Progress Reports	√	√	√	√	√
Local Authority Survey	Year 1: Challenge Authorities only Years 3, 4 and 5: all local authorities surveyed	√		√	√	√
Headteacher Survey	Years 1 and 2: (Challenge Authorities and Schools Programme schools) Year 3: (Challenge Authority, Schools Programme funding plus sample of PEF-only schools) Year 4: All schools in receipt of ASF funding (Challenge Authority, Schools Programme, PEF-only) Year 5: All schools in receipt of ASF funding (Challenge Authority, Schools Programme, PEF-only)	√	√	√	√	√
Quantitative data on attainment and wellbeing	Analysis of attainment measures set out in the 2020 National Improvement Framework.	√	√	√	√	√(limited data available)

Report Structure

1.18 This report focuses on ASF in Year 5 (2019/20) of the SAC. Findings highlighted in the report seek to show changes over the duration of the fund to date encompassing changes in Year 5 from Years 1 to 4.

1.19 Similar to the Year 4 report, this evaluation report considers Challenge Authority, Schools Programme and PEF funding streams and seeks to highlight any emerging differences across the three funding streams.

1.20 The report is structured around the inputs, activities, short- to medium-term outcomes and long-term impact of the ASF. This reflects the revised research questions for Year 5 which are set out in the chapter structure overview below. The COVID-19 focus of the evaluation questions introduced for the Year 5 evaluation are also indicated below. These provided a key focus for the adaptation of survey instruments for the Year 5 evaluation.

Chapter 1 Introduction, Background and Methodology

Chapter 2 Inputs: Governance and Funding

• What did and did not work well in the national and local governance and support as part of the Fund?
• How much funding did local authorities and schools receive, to what extent did they consider it adequate, supplement it with other funding sources, and use it in accordance with the fund’s requirements?

COVID-19 Focus:

• What did and didn’t work well in terms of changes to national organisation and governance of the Fund as a result of COVID-19: focus on funding flexibilities, how funds were used differently and how they supported the programme aims.

Chapter 3 Activities and Outputs: Approaches

• How did schools and local authorities identify, select and implement their approach for addressing the poverty-related attainment gap?
• To what extent did the selected approach aim to support pupils (and parents) from the most deprived backgrounds?

COVID-19 Focus:

• In what ways were approaches modified or identified to support pupils during the period of school building closures?

Chapter 4 Short and medium term outcomes: Perceptions of success, collaboration, use of data and evidence

• To what extent did schools and local authorities involved with the fund feel the intended outcomes of their approach had been achieved?

• To what extent did stakeholders understand, engage and further the programme aims, and why?
• To what extent has the fund encouraged collaboration, and why?

• To what extent did schools and authorities use data, analysis and evidence to drive improvements as part of the fund?

COVID-19 Focus:

• How did COVID-19 impact on the outcomes that Local Authorities and schools were intending to progress? How did COVID-19 impact on collaboration? How did COVID-19 impact on use of data/analysis/evidence?

Chapter 5 Long-term outcomes: Contribution to improvement and reduction of the poverty-related attainment gap, sustainability

• To what extent did the fund contribute to an improvement in attainment and health and wellbeing, and a reduction of the gap between pupils from the most and least deprived areas?
• To what extent can the focus on addressing the poverty-related attainment gap be sustained beyond the years of funding?

COVID-19 Focus:

• Perceptions of impact of COVID-19 on existing patterns of numeracy and literacy attainment and health and wellbeing? focus on families moving into being affected by poverty. What does this mean for our understanding of the ‘target group’, its size/scale/make-up? To what extent did sustainability remain a factor or replaced by immediate COVID-19 concerns?

Chapter 6 Discussion and conclusions