Attainment Scotland Fund evaluation: fifth interim report - year 6

Chapter 1: Introduction and Methodology

Introduction

1.1 The Attainment Scotland Fund (ASF) evaluation began in 2015 and follows the duration of the Scottish Attainment Challenge (SAC). The evaluation focuses on the ASF which is the collective name for the funding strands that support the Scottish Attainment Challenge. An evaluation of the first two years of the ASF was published in March 2018, with the Year 3 (2017/18) evaluation report published in June 2019 the Year 4 (2018/19) report published in October 2020 and the year 5 (2019/20) published in June 2021.

1.2 This chapter sets the Year 6 Evaluation Report (2020/21) in the context of the ongoing COVID-19 pandemic, the recovery period for education and the SAC Refreshed mission and it provides detail on the aims of the evaluation, the overall approach and the structure of the Report.

Background

Context

1.3 This report covers the 2020/21 academic year, a year which saw a continued period of change as a result of the ongoing COVID-19 pandemic and the recovery period for education. This academic year involved a period of school building closures in January - March 2021.

1.4 The ongoing COVID-19 pandemic has continued to have an impact on the poverty-related attainment gap in 2020/21. This makes it critical that the evaluation of the ASF considers the impacts of this period on the operation and use of the fund. The Equity Audit published in January 2021 highlighted that the pandemic disruption and periods of school building closures had a disproportionate impact on children and young people impacted by poverty.

1.5 The findings reinforce the importance of focusing resources on an ongoing, long-term and system wide effort to address recovery and tackle the poverty-related attainment gap.

Scottish Attainment Challenge Context

1.6 The Scottish Attainment Challenge was launched in February 2015 to help tackle the poverty-related attainment gap. It is underpinned by the National Improvement Framework, Curriculum for Excellence and Getting it Right for Every Child. It prioritises improvements in literacy, numeracy, health and wellbeing of those children adversely affected by poverty.

1.7 Achieving excellence and equity in education are the key aims as set out in the National Improvement Framework. The Scottish Attainment Challenge leads system change through a tripartite shared leadership of national government, local government and the executive improvement agency, Education Scotland. The arrangement is designed to support and empower headteachers, schools, local authorities and their partners to develop focused and innovative approaches to improving outcomes for learners, reflecting their own local circumstances.

The current SAC has the following main strands:

• The Challenge Authority and Schools Programmes provide additional resource to nine local authorities, and a further 73 schools outwith those local authorities with the highest proportions of deprivation. The nine 'Challenge Authorities' are Glasgow, Dundee, Inverclyde, West Dunbartonshire, North Ayrshire, Clackmannanshire, North Lanarkshire, East Ayrshire and Renfrewshire.
• Pupil Equity Funding (PEF) provides £120m each year, directly to schools for headteachers to use at their discretion on initiatives that they consider will help close the poverty-related attainment gap. Over 97% of schools in Scotland have been allocated funding for pupils in Primary 1 through to third year of secondary school, based on the estimated numbers of pupils registered for free school meals.
• Care Experienced Children and Young People (CECYP) funding for targeted initiatives, activities, and resources, designed to improve the educational outcomes of this group. This was introduced in 2018/19 with funding allocated to all local authorities based on the number of looked after children they have in their care.
• Additionally a number of national programmes have been supported for targeted work to raise attainment and improve equity, including: staffing supply and capacity; professional learning and school leadership; investment in Regional Improvement Collaboratives and a number of third sector organisations.

1.8 These strands have developed over the period of the SAC. Challenge Authority and Schools Programme were the initial funding streams which commenced in 2015, followed by the introduction of PEF in 2017/18 and the CECYP in 2018/19.

Scottish Attainment Challenge Refreshed Mission

1.9 Within 2020/21 the refreshed Scottish Attainment Challenge mission was developed. In November 2021 the Cabinet Secretary set out plans for a refreshed mission, following a period of consultation with an Advisory Group involving key stakeholders. The refreshed mission is:

• To use education to improve outcomes for children and young people impacted by poverty, with a focus on tackling the poverty related attainment gap.

1.10 In refreshing the Scottish Attainment Challenge mission key evidence sources were drawn on including the Closing the Poverty Related Attainment Gap: Progress Report 2016-2021 and the Audit Scotland Report: Improving Outcomes for Young People Through Education (also March 2021). Considerations from these reports shaped the refreshed programme and The Framework for Recovery and Accelerating Progress which has been developed to support the next phase of Challenge. This sets out the roles and responsibilities across the education system in the context of the Scottish Attainment Challenge and introduces a requirement for local authorities to set a series of locally identified stretch aims for progress for all pupils and for closing the poverty related attainment gap. Changes to the Attainment Scotland Fund see all 32 local authorities receiving funding from the Attainment Scotland Fund - Strategic Equity Funding. This will be distributed equitably based on Children in Low Income Families. Pupil Equity Funding will continue with allocations confirmed over the Parliamentary term, providing clarity for headteachers to make long term plans for their approaches to tackling the poverty related attainment gap.

1.11 In February and March 2022 a full review of the Logic Model^[1] was undertaken in collaboration and consultation with key stakeholders. The development of a refreshed SAC Evaluation Strategy is currently in progress using the new Programme Logic Model.

Methodology

1.12 The evaluation aims to provide learning about the overall implementation of the ASF and the extent to which the aims of the ASF have been met.

1.13 The evaluation has the following objectives:

• Assess the impact of the overall fund in improving attainment and health and wellbeing and reducing the difference between pupils from the most and least socio-economically deprived areas;
• Assess the extent to which the further aims and outcomes of the fund have been met including: promote capacity for data-based self-evaluation and improvement, encourage collaboration between schools and local authorities; embedding of equity across the education system; engagement with and support for pupils and their families living in poverty to engage with learning;
• Provide learning and increase the Scottish evidence base of what does and does not work to improve attainment and health and wellbeing, especially of pupils from the most socio-economically deprived areas;
• Provide learning on what did and did not work well in the process of implementing the fund across participating Challenge Authorities and schools and which factors helped and hindered the fund achieving its outcomes. In Year 6, to provide further learning on PEF.
• Provide learning on which factors helped and hindered the fund achieving its outcomes, with a specific focus on the impact of COVID-19.

1.14 Figure 1.2 sets out the long-term outcomes for the SAC as set out in the SAC Logic Model for the current Programme. In 2020 the SAC Logic Model was adapted to take account of COVID-19. This process was taken forward with Education Scotland and Scottish Government policy officials and analysts.

1.15 The ASF Evaluation research questions were subsequently adapted in line with the SAC Logic Model, resulting in a sub-set of COVID-19 impact evaluation questions. A further review of the ASF evaluation research objective and evaluation questions was undertaken to guide the 2020/21 evaluation, in recognition of the ongoing impact of COVID-19 and the development of the programme. The following priority areas were identified for the Year 6 Evaluation:

• Families and communities;
• Continuing focus on COVID-19 response/recovery;
• Emphasis on data and evidence for improvement;
• Collaboration;
• The importance of a greater focus on PEF; and,
• Intersection with other policy areas such as tackling child poverty.

Figure 1.2: Long-term outcomes for Scottish Attainment Challenge

1. Embedded and sustained practices related to addressing the impact of the poverty-related attainment gap

2. All children and young people are achieving the expected or excellent educational outcomes, regardless of their background

3. An education system which is aspirational, inclusive in practice and approaches for all including teachers, parents and carers, children and young people

4. Closing the attainment gap between the most and least disadvantaged children and young people

1.16 Evidence in relation to the ASF aims has been gathered from a range of sources and evaluated against the research questions.

1.17 The sources used to inform progress in Year 6 of the ASF are set out below.

• Administrative data: Information gathered as part of the routine organisation of the ASF provides data primarily on the funding that local authorities and schools received from the different funding streams.
• Challenge Authority and Schools Programme progress reports: Challenge Authorities and Schools Programme Progress Reports in March and September 2021.
• Local Authority Survey 2021: The Local Authority Survey was launched in mid-November 2021 and closed on 20 December 2021. In total, 26 local authorities responded to the survey (8 Challenge Authority, 10 Schools Programme, 8 PEF-only), an increase in response from the LA survey 2020 which achieved 15 responses, although fewer than the 28 responses in 2019. Analysis of the survey results have informed the findings in this overall evaluation report.

This was the fifth wave of the Local Authority survey. The first wave took place in 2016 with Challenge Authorities only and the second (2018), third (2019) and fourth (2020) waves were undertaken with all local authorities.

The third wave introduced an expanded survey format. This built on the findings of previous waves of the survey but in addition to considering existing themes (such as governance, funding, sustainability, PEF planning and implementation) also sought local authority perspectives of:

• developing approaches to closing the poverty-related attainment gap;
• the extent to which data and evidence featured in decision making at the local level;
• the extent to which the fund increased collaboration;
• factors supporting and mitigating progress towards closing the poverty-related attainment gap within local authorities.

The fifth wave covering 2020/21 maintained the expanded survey format, but also considered COVID-19 impacts for all survey questions specifically focusing on the period of school building closures between January - March 2021.

Views on planning and implementation of the CECYP Fund were also sought in the Local Authority Survey. The CECYP Fund is however outwith the scope of this evaluation report and therefore is reported on separately.

• Headteacher Survey 2021: The sixth wave of the Headteacher Survey 2021 was undertaken in Autumn 2021 on an externally commissioned basis. All Challenge Authority, Schools Programme and PEF-only schools were invited to respond to the survey. The survey invited responses from headteachers related to the 2020/21 academic year including the period of school building closures from January to March 2021.
• The survey sampling approach was modified from the previous survey, with a return to a 100% sample approach. As a result, the achieved sample for the present survey was around 50% larger than the 2020 achieved sample.
• The survey achieved an overall response rate of 25% (597 responses in total), slightly below the response rate for 2020 (27%; 420 responses from a sample survey approach for PEF-only schools). Survey responses have been weighted by ASF stream and urban/rural location to adjust for response bias^[2].
• The low survey response achieved suggests the need for some caution in relation to the generalisability of survey findings to the wider population.
• Quantitative data on attainment and wellbeing: The report draws on available data consistent with the National Improvement Framework measures of the attainment gap. Analysis focuses on patterns of attainment across Challenge Authorities, non-Challenge Authorities and Scotland overall. The following should be noted:
  • Achievement of Curriculum for Excellence Levels (ACEL) - For 2020/21 data was collected for Primary school (P1, P4 and P7) pupils only. Secondary school and special school data was not collected due to other pressures on these schools including implementation of the SQA National Qualifications Alternative Certification Model which was used to award National 5s, Highers and Advanced Highers in 2021.
  • Health and Wellbeing - The impact of COVID-19 on data collection means there is no update available for 2020/21 for the Total Difficulties Score and the Mental Wellbeing score (SALSUS/WEMWBS). The current available date was reported on in the ASF Year 5 Report (2019/20).

1.18 It should be noted that that evaluation of specific 'interventions' being implemented by schools and local authorities and the outcomes of those interventions is out of scope of the evaluation.

1.19 Whilst response rates to surveys and available data (Achievement of Curriculum for Excellence Levels) have been impacted by COVID-19, the current evaluation methodology continues to make the best use of existing data to inform our understanding of factors that support improvement in closing the attainment gap at the six year stage of the current Programme.

Table 1.1: Data Sources for the ASF Evaluation

Data Source	Coverage	Years covered
		Year 1 (2015/16)	Year 2 (2016/17)	Year 3 (2017/18)	Year 4 (2018/19)	Year 5 (2019/20)	Year 6 (2020/21)
Administrative data (financial information)	All Challenge Authorities, Schools Programme local authorities and schools receiving PEF	✓	✓	✓	✓	✓	✓
Challenge Authority and Schools Progress Reports	All 9 Challenge Authorities and all Schools Programme Progress Reports	✓	✓	✓	✓	✓	✓
Local Authority Survey	Year 1: Challenge Authorities only Years 3, 4, 5, 6: all local authorities surveyed	✓		✓	✓	✓	✓
Headteacher Survey	Years 1 and 2: (Challenge Authorities and Schools Programme schools) Year 3: (Challenge Authority, Schools Programme funding plus sample of PEF-only schools) Year 4: All schools in receipt of ASF funding (Challenge Authority, Schools Programme, PEF-only) Year 5: All schools in receipt of ASF funding (Challenge Authority, Schools Programme, (50% sample of PEF-only) Year 6: All schools in receipt of ASF funding (Challenge Authority, Schools Programme, PEF-only)	✓	✓	✓	✓	✓	✓
Quantitative data on attainment and wellbeing	Analysis of attainment measures set out in the 2022 National Improvement Framework.	✓	✓	✓	✓	✓ (limited data available)	✓ (not all data available)

Report Structure

1.20 This report focuses on ASF in Year 6 (2020/21) of the SAC. Findings highlighted in the report seek to show changes over the duration of the fund to date encompassing changes in Year 6 from Years 1 to 5.

1.21 Similar to the Year 5 report, this evaluation report considers Challenge Authority, Schools Programme and PEF funding streams and seeks to highlight any emerging differences across the three funding streams.

1.22 The report is structured around the inputs, activities, short to medium-term outcomes and long-term impact of the ASF. This reflects the revisions made to the evaluation questions for Year 5 which have been subject to consideration and revised again for Year 6. The evaluation questions are set out in the chapter structure overview below (chapters 2-5 contain a summary of key findings box). The COVID-19 focus of the evaluation questions introduced for the Year 5 evaluation were also maintained for Year 6 and were reviewed to reflect current evaluation requirements. The Year 6 evaluation questions provided a key focus for the adaptation of survey instruments for the evaluation.

Chapter 1 Introduction, Background and Methodology

Chapter 2 Inputs: Governance and Funding

• What did and did not work well in the national and local governance and support as part of the Fund?
• How much funding did local authorities and schools receive, to what extent did they consider it adequate, supplement it with other funding sources, and use it in accordance with the fund's requirements?

COVID-19 Focus:

• What did and didn't work well in terms of changes to national organisation and governance of the Fund as a result of COVID-19: focus on funding flexibilities, how funds were used differently and how they supported the programme aims.

Chapter 3 Activities and Outputs: Approaches

• How did schools and local authorities identify, select and implement their approach for addressing the poverty-related attainment gap?
• To what extent did the selected approach aim to support pupils (and parents) from the most deprived backgrounds?

COVID-19 Focus:

• In what ways were approaches modified or identified to support pupils during the period of school building closures?

Chapter 4 Short and medium term outcomes: Perceptions of success, collaboration, use of data and evidence

• To what extent did schools and local authorities involved with the fund feel the intended outcomes of their approach had been achieved/ there had been progress towards achieving these outcomes?
• To what extent has the fund encouraged collaboration, and why?
• To what extent did schools and authorities use data, analysis and evidence to drive improvements as part of the fund?
• To what extent has the fund contributed to engagement with and support for families and communities?

COVID-19 Focus:

• How did COVID-19 impact on the outcomes that Local Authorities and schools were intending to progress? How did COVID-19 impact on collaboration? How did COVID-19 impact on use of data/analysis/evidence?

Chapter 5 Long-term outcomes: Contribution to improvement and reduction of the poverty-related attainment gap

• To what extent did the fund contribute to an improvement in attainment and health and wellbeing, and a reduction of the gap between pupils from the most and least deprived areas?
• To what extent did the fund contribute to equity becoming embedded at different levels of the education system?

COVID-19 Focus:

• Perceptions of impact of COVID-19 on existing patterns of numeracy and literacy attainment and health and wellbeing? Focus on families moving into being affected by poverty. What does this mean for our understanding of the 'target group', its size/scale/make-up? To what extent did sustainability remain a factor or replaced by immediate COVID-19 concerns?

Chapter 6 Discussion and conclusions