Attainment Scotland Fund evaluation: third interim report - year 4

Chapter 4 Short and medium term outcomes

4.1 This chapter considers the impact of the ASF in terms of making progress towards the short- and medium-term outcomes. Following on from Chapter 3 which explored the development of approaches taken by schools and local authorities with regard to addressing the poverty-related attainment gap, and the extent to which approaches focused on supporting parents and pupils from the most deprived backgrounds, this chapter considers local authority and school perceptions of success in meeting their agreed outcomes.

4.2 This is followed by an exploration of two of the key further aims of the ASF, as outlined in the SAC logic model. This includes the extent to which the fund encouraged collaboration, and secondly the extent to which data and evidence were used to drive improvements as part of the fund.

Perceptions of Success

4.3 This section explores evidence collected in respect of the following evaluation question: 'to what extent did schools and local authorities involved with the Fund feel the intended outcomes of their approach had been achieved?'

4.4 Evidence from Challenge Authority progress reports show a shift away from earlier reported short- and medium-term outcomes focused on the professional developments of teachers or support staff. Challenge Authorities all reported on approaches and interventions being implemented around literacy, numeracy, and health and wellbeing. Evidence of the shift away from outcomes prioritised in previous years are the strategic approaches which were identified by a number of local authorities which they had specifically tailored to fit their local needs and circumstances. For example, one local authority cited an intention to have a system wide model of change and improvement which focused on building capacity at all levels and another local authority referred to a whole school approach to nurture across the sector.

4.5 Other priority areas that were identified in Challenge Authority Progress Reports under different work streams included:

• Leadership
• Capacity Building
• Families and communities (including parental engagement)
• Primary to secondary transition
• Early years
• Care experienced pupils
• Data analysis

4.6 Similar to the Challenge Authority progress reporting there was evidence in Schools Programme progress reports of continuation and consolidation of existing approaches and interventions and a general maturation and refinement of approaches. For example it was evident that in some schools new interventions were being introduced in response to need identified through previous or existing SAC funded activity and interventions. This demonstrates some evidence of responsiveness to emerging needs within the schools programme at the local level.

4.7 The extent of embedding of approaches to achieve equity in education within schools was explored in the Headteacher Survey 2019. A large majority of headteacher respondents (84%) were of the view that their approach to achieving equity in education was embedded in their school community with only 2% disagreeing. There was however some variation in views amongst respondent groups, with headteachers of PEF-only schools and those with lower PEF allocations being significantly less likely to feel the approach to achieving equity is embedded.

4.8 Follow-up qualitative feedback in the Headteacher Survey highlighted the extent to which achieving equity had been central to schools' approaches to closing the poverty-related attainment gap, with some schools suggesting that a focus on equity was already a key aspect of their approach. Around 9 in 10 (91%) of schools reported having seen an improvement in closing the poverty-related attainment gap and/or health and wellbeing as a result of ASF supported approaches. A minority (19%) have seen a lot of improvement to date, while 1 in 3 (37%) have seen or are expecting to see a lot of improvement. This is considered further in Chapter 6 (reported impacts).

Collaboration

4.9 One of the further aims of the Fund is the encouragement of collaboration between schools and local authorities, with a stated medium-term outcome articulated in the SAC logic model being:

'Increased evidence of collaboration across the education system (between schools, local authorities, third sector, other delivery partners and professionals such as social work) to deliver and evaluate approaches aimed at closing the poverty-related attainment gap'.

4.10 This section considers the extent to which the ASF encouraged collaboration amongst those receiving funding through Challenge Authority, Schools Programme and/or PEF. It also considers where such collaborations were developing.

4.11 Evaluation evidence reported on regarding the earlier years of the ASF highlighted the positive contribution of the Fund to both the level and nature of collaboration, with collaboration within and across schools, and with external partners commonly reported.

4.12 This section draws on a number of evidence sources to explore the extent to which these positive findings in relation to collaboration continued in Year 4. This includes the Local Authority and Headteachers Surveys for 2019, as well as Challenge Authority and Schools Programme progress reports.

4.13 Collaboration is explored in a number of different aspects, both within and across levels in the education system and between aspects of the education system and other organisations including partner agencies, universities and others.

To what extent did the fund encourage collaboration?

4.14 Overall, collaborative working in schools was viewed by headteachers to have increased as a result of the Fund. This has been a consistent evaluation finding since the introduction of ASF. The majority of headteachers had seen an increase in collaborative working in their school as a result of ASF. Approximately two thirds (64%) of respondents to the Headteacher Survey indicated this, including approximately a quarter (27%) of respondents who felt there had been a large increase in collaborative working in their school as a result of the fund.

4.15 There was significant variation across funding streams with regard to the extent to which ASF support has led to an increase in collaborative working in schools. For example, headteacher respondents to the 2019 survey receiving support via Challenge Authority and Schools Programmes were more likely to have seen an increase in collaborative working (77% and 78% respectively), whilst PEF-only schools were least likely to have seen such a change (58%). This reflects a continuing trend of variation by funding stream from previous years. Schools with a lower PEF allocation were also found to be significantly less likely to have seen an increase in collaborative working in their school.

4.16 Whilst an increase in collaborative working in schools has been a consistent finding since the introduction of ASF, there is noted variability over time, ranging from 71% in 2016 to 77% in 2017, then falling to 71% in 2018 and to 64% in 2019. However, the introduction of PEF in 2017/18 and subsequent inclusion of headteachers whose schools received PEF-only support in the survey may be an explanatory factor.

4.17 At the local authority level, perceptions of the extent of collaboration as a result of the Fund emerged strongly from responses to the Local Authority Survey 2019. Of 27 responses to a question about collaboration, 22 indicated that the fund had increased collaboration. Ten perceived the fund to have increased collaboration to a great extent, 12 to some extent and five to a limited extent. In terms of the nine Challenge Authorities, seven perceived the fund had increased collaboration to a great extent, with the remaining two indicating the fund had increased collaboration to some extent.

Locus of increased collaboration

4.18 In terms of perceptions of where collaboration had increased, schools noted increased collaborative working mostly with other schools in their local authority. Sixty-four percent of headteachers who responded to the Headteacher Survey 2019 indicated such an increase. This was particularly the case for Challenge Authority schools.

4.19 Headteachers who responded to the survey also reported increased collaborative working with families and communities (51%), with third sector organisations (34%) and with professionals in health, social work, and educational psychology (29%). There was less evidence of collaboration with other schools outwith the local authority, with only 17% indicating the existence of such collaborations. There was also little increase in collaboration between schools and universities, with only 6% of headteacher respondents to the Headteacher Survey 2019 indicating such an increase.

4.20 Schools Programme progress reports also provided evidence regarding the level and nature of collaboration:

• within schools
• between schools
• between primary and secondary schools
• within local authorities

4.21 The extent of partnerships, particularly with other professionals (e.g. Speech and Language Therapists) and with third sector organisations was also evident in Schools Programme progress reports. Partnerships with third sector organisations tended to be related to the contracting and delivery of specific interventions or projects, and were seen across a whole range of areas with many different partner organisations. Partnerships appeared particularly prevalent in relation to Health and Wellbeing-related activities and interventions.

4.22 Collaboration continued to feature strongly as a theme within Challenge Authority progress reports and provided some further detail on mechanisms to facilitate collaboration. For example, there was evidence of established networks within local authorities (such as networks linking headteachers to colleagues working at authority-wide SAC programme level), as well as specific collaborations (such as networks linking those with responsibilities for data and evidence).

4.23 Partnerships with universities, third sector organisations and other public sector professionals were also commonly reported within Challenge Authority progress reports. The importance of partnerships with third sector organisations to deliver specific interventions or projects as part of authority work-streams was highly evident in many reports.

4.24 In addition, collaborations beyond the local authority level were also highlighted in progress reports. For example, Schools Programme progress reports highlighted examples of wider collaborations, such as schools collaborating at the Regional Improvement Collaborative (RIC) level (e.g. good practice sharing at RIC level regarding Family Link Worker interventions). The role of RICs in facilitating increased collaboration with regard to ASF will be further explored as part of the forthcoming review of RICs.

4.25 A broad number of themes emerged from the Local Authority Survey 2019 in terms of local authority perspectives on collaboration with respect to the Fund:

• Funding viewed as a driver of collaboration;
• PEF viewed as a driver of collaboration in PEF-only authorities;
• Mechanisms developing to support more strategic and systematic collaboration such as Career Long Professional Learning (CLPL), leadership training, capacity building and local support networks;
• Collaboration enabling sharing of practice;
• Collaboration evident at different levels of the system and between different levels of the system, for example: within schools/between school clusters/within local authorities/between schools and wider partners such as third sector organisations and universities/between local authorities;
• Identified benefits of collaboration such as enabling best use of resources and enabling a focus on improving outcomes for children and young people through working collaboratively with other services.

4.26 An illustration of increased opportunities for collaboration at different levels and the focus on improving outcomes is offered through the following quote from a local authority respondent (Challenge Authority) to the 2019 survey:

'The fund has brought new opportunities for teams to work together, particularly in literacy and numeracy and [health and wellbeing]. Our cross-council working has also increased and we have been able to develop new relationships with other services to strengthen outcomes for young people.'

4.27 Despite many positives associated with increased collaboration as outlined above, local authority responses also made reference to potential concerns and barriers associated with increased collaboration. For example, collaboration was noted as time consuming, and also requires considerable staff availability.

Use of Data and Evidence

4.28 This section explores the extent to which schools and local authorities have used data, analysis and knowledge of what works to monitor and inform their activity related to closing the poverty-related attainment gap in order to address the following evaluation question:

'To what extent did schools and authorities use data, analysis and evidence to drive improvements as part of the Fund?'

4.29 A number of evidence sources were considered including Challenge Authority and Schools Programme progress reports, annual Headteacher Survey and Local Authority Survey findings.

4.30 Continuing the positive trend demonstrated since the inception of the ASF in the first interim report for Years 1 and 2, and the second interim report in Year 3, data and evidence are utilised increasingly in a variety of ways to support ASF activity.

Data and evidence supporting decision-making

4.31 At local authority level, evidence drawn from the Local Authority Survey 2019 suggests that data and evidence relevant to the local context featured in local authority decision-making with regard to the ASF to a great extent for the majority of local authorities, and for all of the Challenge Authorities.

4.32 Progress reports also provide evidence of local authority level use of data and evidence within Challenge Authorities. The continued and expanding use of data to support targeting, monitoring and evaluation of work-streams, initiatives and approaches was evident across Challenge Authority mid-year progress reports, pointing to the increased focus on data and evaluation to support decision-making and focus on improvement.

4.33 Within Schools Programme progress reports, the use of data to support targeting, monitoring and evaluation of ASF approaches and interventions was also clearly evident. Although the extent varied widely, reports described approaches involving extensive data analysis. Examples of approaches included:

• A school which had identified gaps through data analysis which had led to the development of focus for Schools Programme- and PEF-related activity within the school;
• A 'positive destinations' model within a school utilising Insight virtual comparator data to develop rigorous tracking and monitoring for pupils in SIMD 1 and 2 with a focus on early interventions to prevent negative destinations.

4.34 Evidence on schools' use of data and evaluation in relation to ASF supported activity drawn from the Year 4 Headteacher Survey explored use of evidence in evaluating approaches. At the school level, consistent with 2018 findings, 90% of Headteacher Survey respondents indicated they always used available evidence to measure the impact of interventions.

4.35 Evaluation plans were in place to measure the impact of ASF supported approaches in the great majority of schools, with 95% of Headteacher Survey respondents indicating the presence of a plan. There were a number of reasons for the absence of an evaluation plan provided by the remaining 5%, including referenced changes to schools' approach or indicators requiring the production of a new plan, changes in leadership or staffing constraints delaying production, or difficulty identifying success measures for approaches being implemented.

4.36 Increasingly rich data environments were suggested by evidence sources, with a range of mechanisms for using data including combining of data from different sources and different levels within the system. An increasing focus on, and availability of local data was also apparent. Feedback gathered from headteachers as part of the Year 4 survey indicated use of a range of data tools including BGE toolkit^[6] and Insight^[7], alongside evidence relating to participation rates, attendance, and progress through specific ASF programmes and interventions.

The role of data and evidence in driving improvements

4.37 The importance of using data and evidence to support decision-making at different levels of the education system, with a focus on driving improvement was clear across evidence sources.

4.38 For example, improvement-related benefits of enhanced data and evidence use were evident in responses to the Local Authority Survey. A number of themes were highlighted related to improvement in terms of data and evidence use:

• Targeting resources for improvement
• Driving change and improvement
• Identifying gaps and priorities
• Measuring impact
• Tailoring planning to local contexts

4.39 At the school level, increased use of data and evidence appeared part of a wider change of culture/approach within schools resulting from the introduction of ASF which had created opportunities for the use of data in terms of reflecting on existing practice, introducing changes based on evidence, and measuring the impact of changes.

Improvements in data capability

4.40 Improved capability across the system to use data and evidence was also evident. Challenge Authority progress reports highlighted many examples of such increased capability, including upskilling at various levels, from classroom teachers through to the continuing development of data specialist posts in some instances.

4.41 The Headteacher Survey 2019 explored the extent to which headteachers felt confident using data. Confidence in use of data and evidence to support and inform ASF approaches has continued to increase year-on-year, with 93% of headteachers responding to the survey indicating increased confidence using evidence to inform the development of approaches to ASF in 2019, a three point increase on 2018 findings and a nine point increase on 2017 findings.

4.42 The Headteacher Survey also explored the extent to which knowledge and skills in relation to use of data and evidence have increased as a result of the Fund. Reported improvements in skills and knowledge in utilising data and evidence for teaching, planning, evaluation and improvement at the school level continue, with two-thirds (66%) of respondents indicating improvements in the 2019 survey. This represents an increase on 2018 survey findings, when 60% of headteacher respondents indicated an increase in skills and knowledge regarding data and evidence utilisation. However, evidence of any longer term upwards trend in skills and knowledge is less apparent, given that 69% of headteachers responding to the 2017 survey indicated improved skills and knowledge.

4.43 In terms of variation by funding stream, headteacher respondents from Schools Programme schools were typically more positive than those in Challenge Authority or PEF-only schools in terms of increased confidence in using evidence and in terms of perceptions of increased skills and knowledge regarding data and evidence.

Issues and challenges in use of data and evidence

4.44 Despite the many positives outlined above related to the use of data and evidence, and clear evidence of improving capability and capacity, there was some evidence of potentially overwhelming volumes of data in some local authorities. Challenges related to the use of data and evidence also emerged through qualitative feedback gathered as part of the Year 4 survey. For example, schools pointed to challenges in terms of evaluating the impact of specific interventions, not least related to difficulties attributing impact to interventions.