Attainment Scotland Fund evaluation: interim report (year 3)

4. Short and Medium term Outcomes 

4.1. This chapter begins to consider the impact of the ASF. It starts by describing the short and medium term outcomes that local authorities and schools reported they were trying to achieve, and their perceptions around success in meeting these outcomes. 

4.2. The discussion of short to medium term outcomes is then followed by sections focusing in detail on the impact of the ASF in terms of:

• Collaboration
• Use of Data
• Unintended consequences  

Short and medium term outcomes of interventions

4.3. Evidence from the Challenge Authority reports indicated that interventions were trying to improve outcomes for children and young people, teachers/other staff and parents. Most of the reported short and medium term outcomes focussed on the professional development of teachers or support staff. Examples of outcomes are shown below. 

• Teachers/Other Staff: Improved confidence in identifying/implementing pedagogical approaches; increased teaching and learning skills; improved skills and confidence in relation to using data or improvement methodologies; improved leadership skills; increased motivation of staff to change and share practice; increased sharing of practice between schools and improved relationships with partners
• Children and Young People: increased attainment, attendance and exclusions; improved confidence, health and wellbeing; improved curricular transitions; improved quality of learning experience; improved relationships between pupils and teachers 
• Parents: Increased parental engagement; increased confidence in supporting their children’s learning; improved positive relationships within the family. 

4.4. Evidence on factors that local authorities thought contributed to whether interventions achieved their short and medium term outcomes was gathered from the Challenge Authority progress reports.  Factors raised included: 

• a focus on collaboration and partnership working;
• the ability to create an approach tailored to local context and circumstances;
• a strategic approach together with a focus on specific interventions;
• support and challenge provided by local authority and Attainment Advisors;
• increased leadership capacity; and,
• the embedding of evaluation and feedback loops to ensure that learning contributed to ongoing development. 

4.5. Evidence on factors that helped interventions to succeed in improving attainment more generally is discussed in Chapter 5. 

Collaboration

4.6. This section considers the extent to which the ASF encouraged collaboration amongst those receiving Challenge Authority, Schools Programme funding or PEF. Two key questions were considered:

• To what extent did the ASF encourage collaboration?
• What factors helped and hindered collaboration?

4.7. The evaluation of Years 1 and 2 highlighted the positive contribution of the ASF to both the level and nature of collaboration, with collaboration within and across schools and with external partners most commonly reported. This continued to be reflected in evidence gathered for the evaluation of Year 3.

4.8. Evidence has been gathered primarily from the headteacher survey, Challenge Authority progress reports and school case studies. 

4.9. Collaborative working in schools was viewed by headteachers to have increased as a result of the ASF. Specifically, 71% of headteachers felt there had been an increase in collaboration in their school as a result of the ASF. 

4.10. There were differences in the pattern of responses across funding streams. Headteachers involved in the Schools Programme were most likely to have seen an increase in collaborative working (98%) whilst those in receipt of PEF only were least likely to have seen an increase (66%). 

4.11. Some of the headteachers of schools in receipt of only PEF did report an increase in collaborative working but did not feel this was attributable to the ASF; 24% reported this compared to 15% of headteachers in Challenge Authorities and 2% of headteachers in the Schools Programme. 

Figure 4.1: Change in number of staff working collaboratively, headteacher survey 2018

4.12. Those in receipt of highest 25% of PEF allocations were also more likely than those receiving the lowest 25% of PEF allocations to indicate there had been a large increase in collaborative working (51% vs 19%). 

4.13. The most commonly reported reason in relation to why headteachers felt there has been an increase in collaboration was that there had been greater emphasis on collaboration from school leadership. Other factors identified in association with improved collaboration included a change of culture/ethos, increased staff resourcing and staff time, and collaboration being embedded as part of professional learning. 

4.14. In instances where headteachers did not identify improved collaboration as a result of the ASF, this was commonly due to perceptions of an existing well-established culture of collaboration. There was also some evidence of headteachers believing that the ASF had not had sufficient impact on staff time or opportunities within schools for collaborative working. 

4.15. Schools involved in the case studies highlighted the importance of collaborative working to the success of interventions. Most felt there had been an increase in sharing practice and collegiate working across the school, evidenced, for example, by an increase in the number of Professional Learning Communities or collaborative research groups. 

4.16. Collaboration continued to feature strongly in Year 3 Challenge Authority progress reports. In terms of collaboration with external partners, partnerships were evident with universities, the third sector and other professionals, such as social work and educational psychologists. 

4.17. A mapping exercise in April 2018 revealed that Challenge Authorities were working with universities to support their evaluation of the impact of the ASF. Challenge Authority progress reports also indicated specific engagement with the Robert Owen Centre (University of Glasgow). 

4.18. Also highlighted in Challenge Authority progress reports were improved opportunities for collaboration between schools. There was also some emerging evidence in September progress reports of improved linkages between primary and secondary schools.  

4.19. Local authorities responding to the online survey also discussed improved collaboration between schools as a positive consequence of the ASF. 

4.20. There was some evidence of the development of mechanisms to facilitate collaboration within Challenge Authorities. For example, one Challenge Authority had created a number of thematic networks (e.g., numeracy, early years, STEM) to support collaboration.   

4.21. In addition, whilst the evaluation found limited evidence of collaboration between local authorities, Regional Improvement Collaboratives were mentioned as an important mechanism for information sharing. It will be important to monitor the progress of this in future years of the evaluation. 

Use of Data and Evidence 

4.22. This section explores the extent to which schools and local authorities have used data, analysis and knowledge of what works to monitor and inform their improvement activity. 

4.23. Evidence was gathered in bi-annual Challenge Authority progress reports, the annual headteacher survey, local authority mini survey and school case studies. 

4.24. As evidenced during the first two years of funding, there was clear evidence of an ongoing commitment to the use of data for identifying, targeting and monitoring interventions in Year 3. For example, analysis of Challenge Authority progress reports showed that embedding use of data was viewed by local authorities as crucial to the success of their identified work-streams. 

4.25. A wide variety of data sources were cited by Challenge Authorities in their progress reports. These included:

• Achievement of Curriculum for Excellence Level (ACEL)
• New Group Reading Test (NGRT)
• Positive destinations
• Attendance 
• Exclusions

4.26. Most Challenge Authorities complemented this data with other evidence sources, including: local surveys (including pre- and post-implementation); qualitative focus groups; feedback forms; pupil assessments and attendance tracking. 

4.27. In addition, in April 2018, a mapping study revealed that 6 out of the 9 Challenge Authorities had commissioned an external evaluation to help measure progress and impact of the funding received. These evaluations were undertaken by universities and typically focussed on one of their planned work-streams.  

4.28. Training on evaluation and understanding data continued to be delivered to headteachers, teachers and/or practitioners in Challenge Authorities. 

4.29. Furthermore, there was some evidence amongst Challenge Authorities of the creation of bespoke tools for direct use by schools. For example, one Challenge Authority had created a monitoring and tracking database for schools to use to track progress on interventions. In another, a specific tool to support schools to measure the impact of interventions was being piloted across the local authority.  There was also evidence of the development of bespoke measures of deprivation to enable targeting of improvement activity by some local authorities. This raises the potential for authorities to learn from each other about these approaches as they develop, and to share emerging practice so that these can support other authorities in their own developments. 

4.30. As was found in Years 1 and 2, local authorities (both Challenge and non-Challenge) continued to value the role of Attainment Advisors in planning and evaluation. This included support with identifying appropriate outcomes and data analysis. 

4.31. Overall, at local authority level, there was clear recognition of the value of using data to drive improvements. There was also evidence that authorities were making efforts to gather a range of data. It remains important to consider the impact of this increased data use on educational decisions and practice. 

4.32. The headteacher survey and school case studies provided evidence of how data was being used at the school level. 

4.33. Year 3 of the headteacher survey highlighted positive features of data use at the school level. There were no significant differences in responses from different funding streams or changes from Year 2 of the survey.  

• 90% felt confident using data and evidence to inform the development of interventions; 

• 90% reported that they always use evidence to measure the impact of interventions;

• 92% reported having an evaluation plan in place to monitor the progress and impact of interventions.

4.34. Over half of headteachers (60%) also felt their skills and knowledge of how to use data for teaching, planning and improvement had improved through the ASF. Headteachers of schools receiving PEF only were less likely to report that their skills and knowledge had improved. 

Figure 4.2: Skills and knowledge on data use, headteacher survey

4.35. Headteachers of schools receiving the highest 25% of PEF allocations were also significantly more likely than those receiving the lowest 25% of PEF allocations to report that their skills and knowledge in relation to data use had improved through the ASF (79% vs 38%). 

4.36. Schools involved in the case studies were asked about how they had used data to support planning and monitor impact. There was some evidence of improved use of evidence to support targeted work. However, whilst data use emerged as a key feature of success at local authority level, it featured somewhat less strongly at school level. Schools were less likely to discuss data use as a key factor influencing the success of their interventions. 

Unintended Consequences 

4.37. Overall, the unintended consequences reported by headteachers and local authorities were consistent with those reported in Years 1 and 2 of the ASF. Evidence has been drawn from the online surveys for local authorities and headteachers. 

4.38. The most common unintended positive consequence reported in Year 3 was around improved collaboration. Local authorities responding to the online survey discussed an increase in schools working together and headteachers reported an increase in collaboration and improved partnership working. 

4.39. Other common unintended positive consequences reported included:

• Increased skill development 
• A change in culture/ethos and increased awareness of the impact of deprivation 
• Improvement in pupil and parent engagement 

4.40. In line with the findings from Years 1 and 2, stakeholders continued to report concern around the impact on staff workload and resources as well as a concern around a sense of division or exclusion between those benefitting and not benefitting from the ASF. 

4.41. Another unintended negative consequence reported by headteachers in Year 3 was the associated reporting requirements of the ASF and the pressure to demonstrate improvement.