Attainment Scotland Fund evaluation: fourth interim report - year 5

Chapter 4 Short and medium term outcomes

4.1 This chapter considers the impact of the ASF in terms of making progress towards the short- and medium-term outcomes. The initial section considers local authority and school perceptions of success in meeting their agreed outcomes.

4.2 The second section explores in more detail two of the key aims of the ASF: the extent to which the fund encouraged collaboration, and the extent to which data and evidence were used to drive improvements as part of the fund.

Perceptions of Success

4.3 This section explores evidence collected in respect of schools’ and local authorities’ perceptions of success in meeting their short-term and medium term outcomes, to address the following evaluation question:

‘to what extent did schools and local authorities involved with the Fund feel the intended outcomes of their approach had been achieved?’

4.4 Evidence from Challenge Authority and Schools Programme progress reports, Headteacher Survey 2020 and Local Authority Survey 2020 is utilised to inform this section.

Embedding approaches

4.5 There was evidence of perceived success with regard to embedding approaches to achieving equity within schools. Findings from the Headteacher Survey 2020 showed the majority (84%) of headteacher respondents felt their approach was embedded within their school community to a great or moderate extent, very consistent with 2019 survey findings and showing broad consistency across key respondent groups in terms of funding stream, and urban/rural schools.

Understanding of challenges and barriers faced by pupils affected by poverty

4.6 Schools’ levels of understanding of the challenges and barriers faced by pupils affected by poverty were high, with nearly all (98 per cent) headteachers who responded to the Headteacher Survey 2020 indicating they understood the challenges and barriers faced by pupils affected by poverty, including 78 per cent who felt they understood this ‘to a great extent’. This was similar to 2019 Headteacher Survey results, and was consistent across ASF streams. There were however some geographical variations with those in small town and rural areas less likely to feel they understood the challenges. Those with lower proportions of pupils in receipt of FSM (and therefore lower PEF allocations) were also less likely to feel they understood the challenges.

Impact of COVID-19

4.7 Local authorities perceived a range of factors associated with COVID-19 to have impacted on progress towards closing the poverty-related attainment gap, as indicated through responses to the Local Authority Survey 2020.

4.8 It was recognised that there remained a great deal of uncertainty regarding the extent of the impact on progress. Several local authorities referenced internal data gathering exercises undertaken (such as an equity audit), which had sought to understand and quantify the impacts at the local authority level.

4.9 Reference was made to increasing numbers of families experiencing poverty as a result of the pandemic, and of the need to put in place appropriate responses. Longer-term plans were frequently paused in order to focus on short-term priorities, as described in Chapter 3, but all respondents described a continuing focus on closing the poverty-related attainment gap throughout the period of school building closures. Rather, the focus was on limiting and mitigating the impact of COVID-19 on pupils and families affected by socio-demographic disadvantage, whilst recognising increases in socio-economic disadvantage because of COVID-19. Local authorities variously described a range of actions in order to mitigate the impact of COVID-19, including:

• adjustments, adaptations, pausing of existing activity;
• altering governance and funding to respond to emerging challenges;
• creating recovery plans and longer-term planning;
• planning for future potential periods of school building closures, or for pupils self-isolating;
• considering the provision of specialised support to address the needs of those pupils not making expected progress;
• addressing digital connectivity challenges, particularly in rural areas;
• developing practice for remote learning.

4.10 Additional funding to support purchase of devices was welcomed, but delays in purchasing devices due to huge demand has been a factor, and access for younger primary pupils remains a concern.

4.11 Headteachers also viewed COVID-19 and school building closures to have impacted on their progress in closing the poverty-related attainment gap. The extent of perceived impact is explored further in Chapter 5.

Collaboration

4.12 This section considers the extent to which progress has been achieved towards the stated medium-term outcome on collaboration articulated in the SAC logic model:

‘Increased evidence of collaboration across the education system (between schools, local authorities, third sector, other delivery partners and professionals such as social work) to deliver and evaluate approaches aimed at closing the poverty-related attainment gap’.

4.13 It considers the extent to which ASF encouraged collaboration amongst those receiving funding through Challenge Authority, Schools Programme and/or PEF. It also considers where such collaborations were developing.

4.14 Evidence from previous years of the ASF evaluation has continuously highlighted the positive contribution of the fund to both the level and nature of collaboration.

4.15 As in the Year 4 report, evidence sources have been explored to consider the extent to which strong evidence of collaboration continued in Year 5.

To what extent did the fund encourage collaboration?

4.16 Overall, perceptions of the extent of collaborative working in schools was viewed by headteachers to have increased as a result of the fund. The majority of headteachers responding to the Headteacher Survey 2020 had seen an increase in collaborative working in their school up to March 2020 as a result of the ASF, with two-thirds (65%) indicating this and just over a third (36%) who had seen a large increase. This was broadly consistent with 2019 Headteacher Survey findings, with increased collaboration as a result of the fund a consistent evaluation finding since the introduction of ASF.

4.17 In addition, almost half (46%) of respondents to the Headteacher Survey 2020 indicated they had seen an increase in collaborative working during the school building closures between March and June 2020.

4.18 There was some variation in school experiences of collaborative working, with primary schools and schools in rural areas less likely to have seen an increase in collaborative working, both up to the period of school building closures and during the period of school building closures.

4.19 At the local authority level, strong perceptions of the extent of collaboration as a result of the fund continued to be evidenced from Local Authority Survey respondents. All respondents to the Local Authority Survey 2020 indicated collaborations had increased as a result of the fund, approximately half of whom indicated that this was to a great extent and half to some extent. This continues the trend previously identified in the Local Authority Survey 2019, where the majority indicated the fund had increased collaboration to a great or to some extent.

4.20 Given the lower number of local authority responses to the Local Authority Survey in 2020, it is not possible to provide analysis at the funding stream level. However, it is notable that collaborations were viewed as having increased to a great extent as a result of ASF by all five Challenge Authority respondents. Evidence of increasing collaboration also featured strongly in Challenge Authority progress reports, particularly in relation to health and wellbeing, and with academic institutions and specialist providers.

Locus of increased collaboration

4.21 Collaborations were described on a range of levels in Local Authority Survey 2020 responses, from regional and local authority-wide through to collaborations at school cluster and individual school level, as illustrated below:

• Enhanced collaboration across local authority services. On occasion, the structuring of local authority services was noted as enabling more effective collaborative working, such as through creation of joint education and children/young people’s services;
• Collaborative working with Attainment Advisers and SAC lead officers;
• The importance of partnerships with third sector agencies, particularly in terms of PEF;
• Specific collaborations emerging in terms of Care Experienced Children and Young People Fund. For example, a PEF-only local authority highlighted collaboration resulting from the Virtual School Head Teacher role in relation to care experienced children and young people;
• Opportunities for networking and collaboration for senior leaders within and beyond school clusters, including sharing good practice, participation in collegiate activities, professional development etc;
• The development of specific mechanisms (e.g. Equity Networks) and roles (such as ‘Challenge’ leaders) to support and facilitate collaboration;
• Sharing and pooling of resources amongst a school cluster or learning community.

4.22 Themes emerging from the Local Authority Survey 2020 in terms of local authority perspectives on collaboration with respect to the fund were broadly similar to those emerging from the 2019 survey:

• Funding as a driver of collaboration;
• Collaboration key to enabling sharing of practice, capacity building and improvement;
• Ongoing development of mechanisms to support collaboration and to enable more strategic and systematic collaboration at a number of levels and between different levels of the system;
• Benefits of collaboration continue to be identified e.g. supporting best use of resources; supporting a focus on improving outcomes for children and young people through working collaboratively across services.

4.23 A number of the above themes are illustrated in the following extracts from local authority survey responses:

‘Funding has allowed for models of greater transfer of skills, knowledge and innovation across the authority and beyond. Capacity has been built and establishments are more confident in succession planning. ASF has provided greater opportunities to integrate programmes having the greatest impact quickly. Innovation is distributed quickly as there are shared values and goals.’ (Challenge Authority respondent)

‘Collaboration within and across all sectors has increased extensively enabling a relentless focus on enhancing teaching and learning, while sharing and planning experiences across the BGE and beyond. This has been both as a direct result of planned programmes and often as an unforeseen gain.’ (Challenge Authority respondent)

‘Much of the ASF work in this authority has been based on effective partnership working and collaboration. Over time we have developed robust collaborative approaches to support working with a range of partners and this has helped to establish and embed several significant partnerships, and supported new and emerging partnerships.’ (Challenge Authority respondent)

‘As a result of the ASF collaboration has continued to be strengthened with a clear focus on raising attainment and closing the poverty-related attainment gap. While effective collaboration has always been a focus the ASF has helped to shape some of the ways in which we collaborate.’ (Schools Programme respondent)

Changes in collaborations over the previous year

4.24 The Local Authority Survey 2020 invited local authorities to provide an assessment of the extent to which collaborations had changed over the previous year. There was some evidence that collaborations had changed, with five of the 15 respondents stating these had changed significantly and eight indicating these had changed to some extent. For a further two, there were perceptions of limited change.

4.25 Examples of changes in collaborations over the period August 2019 to March 2020 mentioned by respondents include:

• Regional Improvement Collaborative partnership development – family schools groups sharing good practice to build greater understanding of the challenges faced by families living in poverty;
• Expansion of Excellence and Equity Literacy and Numeracy Groups to include wider range of multi-agency partners;
• Increased partnership working with Robert Owen Centre (University of Glasgow) leading to developments across schools in local authority.

4.26 In the March to June 2020 period, respondents highlighted aspects of collaboration associated with the closure of school buildings and transition to remote learning. A range of new collaborations with a wider range of partners, as well as strengthened collaborations with existing partners, were identified as local authorities sought quickly to respond to the demands and needs of responding to COVID-19, with several respondents pointing to the pace of this. Collaborations featured strongly in local authorities’ use of flexibilities (as discussed in Chapter 2).

4.27 Various mechanisms to support collaborative working during school building closures were described. For example, the creation of multi-disciplinary teams functioning in localities across a local authority area to foster collaboration and appropriate response across local authority and third sector partners to meet the needs of vulnerable families. Several respondents pointed to the shared goals of collaborative partners to respond appropriately to the pandemic as a strength, with trust arising from this shared approach.

Use of Data and Evidence

4.28 This section explores the extent to which schools and local authorities have used data, analysis and knowledge of what works to monitor and inform their activity related to closing the poverty-related attainment gap in order to address the following evaluation question:

‘To what extent did schools and authorities use data, analysis and evidence to drive improvements as part of the Fund?’

4.29 Evidence sources considered include Challenge Authority and Schools Programme progress reports, annual Headteacher Survey and Local Authority Survey findings.

4.30 In line with the positive trend demonstrated since the inception of the ASF, data and evidence continue to feature strongly to support ASF activity in Year 5. This was particularly evident in the Challenge Authority progress reports at the local authority level, and the Schools Programme progress reports at the school level, with many examples of the ways in which strengthened use of data and evidence was supporting decision-making, planning, and measurement of impact. Whilst the Challenge Authority progress reports overall detail progress in increasing expertise at data collection, analysis and informed use at the local authority level and school level, it is also clear that progress remains somewhat variable across different local authorities. For those on the journey to increased data collection and analysis capacity, there is a recognition of where progress is required and a clear intention to develop towards increasing data capacity at all levels.

Data and evidence supporting decision-making

4.31 At local authority level, evidence from the Local Authority Survey 2020 suggests that data and evidence relevant to the local context continued to feature strongly in local authority decision-making with regard to the ASF. The 2019 survey found that this was the case for the majority of local authorities, and for all of the Challenge Authorities. In 2020, the majority of local authorities who responded similarly found data and evidence had featured to a great extent in decision-making over the previous year (11 of 15 to a great extent, and two to some extent).

4.32 For respondents who indicated data use had strongly featured, a range of approaches utilised at different levels in their locality were evidenced, from local authority-wide through to cluster and school level. The application of various tools was described, including both nationally available tools (such as Insight) as well as bespoke tools created at the local authority level to support data analysis and its application to support decision-making. This was also clearly evidenced in Challenge Authority progress reports.

Exemplar: Using data to inform decision-making

A Schools Programme local authority described the process of using data to inform decision-making at school and local authority level

• schools in receipt of Schools Programme funding had been using their data to inform decision-making with regard to closing the poverty-related attainment gap in their context.
• schools with high levels of PEF funding were working collaboratively with the Attainment Advisors and the local authority SAC lead to use local data to identify needs.
• PEF clinics held across the local authority area for schools to participate in to make use of school level data and evidence to support schools to make decisions on how to allocate PEF most effectively to close the gap in their local context.

4.33 Evidence on schools’ use of data and evaluation in relation to ASF supported activity drawn from the Headteacher Survey 2020 continued to be positive. With respect to the use of data and evidence to inform development of their school’s approach, the large majority (84%) of headteacher respondents indicated this as ‘very good’ or ‘good’.

The role of data and evidence in driving improvements

4.34 The importance of using data and evidence to support decision-making to drive improvement at different levels of the education system was clear across evidence sources, as previously identified in the Year 4 report. It was apparent from Challenge Authority Progress Reports that there continues to be closer analysis across NIF measures and sub measures combined with local contextual data, which in turn highlights where targeted support is needed and strengthens understanding of where achievement and progress are taking place for children and young people affected by poverty. There was evidence of a clear understanding and in some places embedding that analysis and use of data to improve teaching and learning and in identifying the approaches and interventions that maximise the learner outcomes.

4.35 Linking collaboration around data use in driving improvements emerged as an important related aspect. For example, collaboration in terms of data sharing and dialogue around improvement was highlighted as follows:

‘Greater sharing of data across schools and more transparency in our approach, with schools being placed in comparator groupings to further develop dialogue about school improvement.’ (PEF-only local authority)

Improvements in data capability and capacity

4.36 Improved capability and capacity across the system to use data and evidence was also evident in Local Authority Survey 2020 responses. Several local authorities described specific approaches to increase data literacy, including data literacy training for teachers, the development of data networks, and local authority data officers working collaboratively with schools. Challenge Authority progress reports highlighted examples of increased capability and capacity, including upskilling at various levels, from classroom teachers through to the continuing development of data specialist posts in some instances alongside developing skills for school leaders assisting them in self-evaluation and the critical use of school data. There was also increasing evidence of the use of data dashboards to assist schools in tracking data and developing skills. Where bespoke data dashboards are not available data packs are often used to ensure that all practitioners have access to and understand their unique dataset and are able to more effectively direct interventions to improve children and young people’s learning, progress and achievement.

4.37 Skills in measuring the impact of school’s approaches was also positively indicated, with 82% of 2020 Headteacher Survey respondents positive about their ability to identify appropriate measures, and 76% positive about their use of evidence to measure impact. A similar proportion of respondents (78%) were positive about their ability to measure progress and impact of approaches.

4.38 Despite remaining a strong feature, it is notable that headteacher perceptions on some measures rating schools’ approaches to use of data and evidence had decreased in the 2020 Headteacher Survey from previous years. Particularly, perceptions of ability to use evidence to measure impact had decreased from 90% in the 2018 and 2019 surveys to 76% in the 2020 survey, a decrease of 14 percentage points. In addition, there was some variance in terms of funding stream on this measure, with schools receiving PEF-only and those with lower levels of PEF allocation less positive than other schools regarding use of evidence to measure impact.

4.39 School level evidence from Headteacher Survey 2020 indicates that just under two-thirds (63%) of schools felt that ASF support had helped to develop staff skills and knowledge in using data and evaluation, although there was significant variation with PEF-only schools, those in rural areas and those with lower PEF allocations indicating less positive perceptions of this.

Use of data and evidence during the period of school building closures

4.40 A number of aspects were highlighted in terms of the use of data and evidence during the March to June 2020 period of school building closures in the Local Authority Survey 2020, although there remained a strong focus on data and evidence to drive decision-making during this period.

4.41 Local authorities highlighted the different focus required during this period, for example the need to develop new sources of data and evidence to support the transition to remote learning such as engagement data and surveys of parents and pupils on their experiences of remote learning. Data and evidence from existing data sets were appropriately utilised to shape local responses to the pandemic, such as data sharing with third sector partners in order to put in place additional support for vulnerable pupils.

4.42 It was also recognised that the pandemic had impacted on the collection of some data sets.

The role of Attainment Advisors in promoting collaboration and use of data and evidence

4.43 A key feature emerging in particular from Challenge Authority and Schools Programme progress reports was the role of Attainment Advisors in promoting collaboration and use of data and evidence at a range of levels from working with local authority SAC leads down to working with individual schools.