Attainment Scotland Fund evaluation: headteacher survey report 2019

5. Impact

5.1. This section summarises views on the impact of ASF supported approaches to closing the poverty-related attainment gap. This includes views on the factors that contribute to or limit success, whether impacts are likely to be sustainable, and the extent to which ASF support has contributed to an increase in collaborative working.

Progress in closing the poverty-related attainment gap

5.2. Around 9 in 10 (91%) schools reported seeing an improvement in closing the poverty-related gap in attainment and/or health and wellbeing as a result of ASF supported approaches. This included 19% that had seen 'a lot' of improvement to date.

5.3. Nearly all schools (98%) expected to see improvement in closing the gap over the next five years. This included 34% who expected to see 'a lot' of improvement, and a total of 37% who had seen or were expecting to see a lot of improvement. Survey responses indicated that those who have seen some improvement to date were more likely to expect further improvement over the next five years.

5.4. There has been a 13-point increase since 2017 in the proportion of schools reporting an improvement in closing the gap; from 78% in 2017, to 88% in 2018 and 91% in 2019. There has been no statistically significant change in the proportion of schools expecting to see improvement.

5.5. The survey also showed some variation in views across funding streams, with Schools Programme respondents most likely to report an improvement in closing the gap. PEF-only schools, particularly those with a lower PEF allocation, were least likely to report an improvement.

Factors influencing progress

5.6. In addition to variation across respondent groups (such as funding stream, PEF allocation and urban/rural geography), survey analysis also considered correlation between perceived progress in closing the gap and other aspects of headteacher views and experiences. This considered a wide range of factors including schools' approach to closing the gap, headteachers' understanding and awareness in shaping that approach, embedding equity, use of evidence, improvement in evidence skills, collaborative working, and views on availability of support for PEF.

5.7. Analysis indicates that the headteachers most likely to have seen progress in closing the gap were those who had seen a change in culture or ethos (more collaborative working and/or embedding the approach to equity) or have improved their understanding of barriers faced by pupils and families. In particular, the following subgroups were statistically significantly more likely to have seen improvement as a result of ASF supported approaches:

Respondents most likely to have seen progress in closing the gap

• Have seen an increase in collaborative working
• Feel the approach to equity is embedded in the school community
• Feel they understand the challenges and barriers faced by pupils and parents affected by poverty
• Feel confident using evidence to inform development of their approach
• Feel they had sufficient support to develop their school plan for PEF
• Always use available evidence to measure the impact of approaches

5.8. The survey also asked for headteachers' views on the factors that contribute to or hinder the success of ASF supported approaches. Respondents were asked to identify the top three supporting factors and barriers from pre-defined lists.

5.9. Ability to implement approaches relevant to the school context, and teaching and staffing resources were the most commonly cited factors supporting schools' progress. More than half of respondents felt that these have contributed to their progress in closing the poverty-related attainment gap (58% and 52% respectively). Other key factors included higher quality learning and teaching, and use of evidence and the approach to evaluation.

5.10. Staff time and workload, and reduction in other services/resources were seen as the main factors limiting progress in closing the poverty-related attainment gap. Each were mentioned by 44% of respondents. Other commonly mentioned factors included staffing resources and recruitment, level of PEF received and staff absences.

5.11. Survey results indicated some variation across respondent groups in the barriers seen as limiting progress. For example, primary schools were more likely to mention staff absences as a barrier, while secondary schools were more likely to mention staffing resources and recruitment. Staff absences were also a key barrier for schools with a higher PEF allocation, while the level of PEF received was the second most commonly mentioned barrier for schools with a lower allocation.

5.12. Follow-up qualitative feedback was consistent with survey findings in relation to the factors seen as limiting schools' progress in closing the poverty-related gap.^[11]

5.13. In particular, follow-up participants referred to ongoing reduction in wider school resourcing as limiting the extent to which ASF support has enabled them to make 'additional' progress. Participants also suggested that schools are limited in the impact they can deliver for pupils, and that investment to close the poverty-related gap in schools must be supported by wider investment in communities and families. This was in terms of improving quality of life and experiences for pupils outside school, but also building more positive attitudes towards education and raising aspirations.

"We expect overall attainment figures to continue to change and have already seen lots of other positive impacts. But I am sceptical about 'sustainability as many real issues still remain within education in Scotland in terms of cutbacks in resources, staffing and workload. Communities and families need to be given funding and resources too - the issue will not be solved until schools, communities and government work together."

Sustainability

5.14. The survey included a series of questions to explore headteachers' views on the sustainability of their approach to closing the poverty-related attainment gap. This included whether headteachers felt any improvement achieved to date would be sustainable beyond the years of funding, and whether they expected the focus on closing the poverty-related attainment gap to be sustainable beyond funding.

5.15. Around 2 in 5 (41%) survey respondents expected that ASF supported improvement will be sustainable beyond the years of funding. This represented a 17-point reduction from the 2017 survey; from 58% in 2017, to 42% in 2018 and 41% in 2019. There has been a corresponding 17-point increase in the proportion who feel that impacts will not be sustainable.

5.16. Survey findings also suggested a correlation between views on sustainability and perceived improvement to date. Those who had seen improvement to date in the poverty-related attainment gap were statistically significantly more likely to expect improvements to be sustainable beyond funding.

5.17. In addition to sustainability of progress made to date, the survey also asked about sustainability of the focus on closing the poverty-related attainment gap. Around 2 in 5 expected the focus to be sustainable to a 'great' or 'moderate' extent (41%). A further 43% felt that the focus on closing the gap may be sustainable 'to some extent', and 15% felt that the focus would not be sustainable.

5.18. Survey results suggested that Schools Programme schools were more positive than others on the sustainability of the focus on closing the gap (as was the case for sustainability of progress to date). There was also considerable overlap between headteachers who felt that progress achieved to date would be sustainable, and those who felt that the focus on closing the poverty-related attainment gap within their school would be sustainable. A little more than half (53%) expected progress and/or the focus on closing the gap to be sustainable beyond funding.

5.19. The survey asked headteachers to expand on why they expected improvements and/or focus on equity to be sustainable, or why they do not expect these to be sustainable. Qualitative analysis of these responses has been undertaken to identify the key factors mentioned by respondents. These are summarised below.

5.20. This analysis indicated that staff training and development was, by some margin, the most common factor for schools who expect their improvement and/or focus on equity to be sustainable. More than half of those who expect their approach to be sustainable referred to staff training, development and capacity building. This included reference to schools having a specific focus on development of existing staff (including embedding practice) to ensure the sustainability of their approach, and to developing capacity for provision of training and development to ensure new staff can support their approach.

5.21. The importance of staff is also reflected in 'loss of staffing and skills' being by far the most common reason for schools who feel their approach will not be sustainable; loss of staff and skills was mentioned by a large majority of these schools. This included a view that schools would lose staff capacity in the absence of ASF support, and that this would have an inevitable negative impact on their ability to maintain their approach to closing the poverty-related attainment gap.

Why feel improvement/focus will be sustainable (n=556) ^[12]

Staff training, skills development and capacity building	57%
Embedded practice, pedagogy development	26%
Raising awareness and change of ethos/culture	16%
Ongoing access to resources	13%
Developed capacity to use data/evidence to inform approaches	12%
Collaboration within school, with partners and parents	10%
Longer-term health and wellbeing impacts	7%
Developed a collective, shared focus	4%
Why feel improvement/focus will not be sustainable (n=364)
Loss of staffing and skills	85%
Loss of initiatives and approaches/interventions	28%
Reduction in wider budgets/resourcing	14%
Loss of external support and access to services	8%
Pupils' health and wellbeing needs	6%

5.22. Follow-up qualitative feedback reflected these findings in highlighting staff capacity and resources as supporting the sustainability of improvement, and a key risk that improvement to date will not be sustainable.^[13]

5.23. Follow-up participants highlighted the positive impact of dedicated, often specialist, staff input in delivering effective approaches and achieving progress in closing the gap. Participants also referred to difficulties recruiting the required staff as having limited their progress to date, and noted that loss of staff time in the event of funding being withdrawn would lead to some loss of progress.

5.24. Qualitative feedback also highlighted the importance of the focus on equity, and a shared understanding of what equity means, as key to the sustainability of their approach. These participants referred to the positive impact of having developed a clear overall vision and master plan to guide their approach to closing the poverty-related gap, and noted that this vision will remain in place post-funding. Reference was also made to ASF support having enabled schools to try innovative approaches, and for those that prove most effective to be integrated into their ongoing approach.

"Having an overall vision/master plan is key for sustainability, avoiding the idea that one year will "fix" the issues that exist. Our plan is based on the central idea that we need to build the school's capacity, and this will take time to develop. We have used funding as a foundation to try innovative approaches, and integrated what works into the core business of the school."

Collaborative working

5.25. The majority of headteachers had seen an increase in collaborative working in their school as a result of ASF support. Nearly 2 in 3 (64%) indicated this, including around 1 in 4 (27%) who had seen a large increase in collaborative working. This represented a 13-point reduction since 2017 in the proportion of respondents who had seen some increase in collaborative working, although the 2017 result represented a peak following 2016 (71% in 2016, 77% in 2017, 71% in 2018).

5.26. Survey responses indicated statistically significant variation across funding streams in the extent to which ASF support has led to an increase in collaborative working. Challenge Authority and Schools Programme respondents were more likely to have seen an increase in collaborative working (77% and 78% respectively), while PEF-only schools were least likely to have seen such a change (58%). Schools with a lower PEF allocation were also statistically significantly less likely than others to have seen an increase in collaborative working.

5.27. Schools were most likely to have seen an increase in collaborative working with other schools in their local authority, particularly for Challenge Authority schools. Schools also reported increased collaborative working with families and communities, with third sector organisations (particularly Challenge Authority or Schools Programme schools) and with professionals.

5.28. Follow-up qualitative feedback highlighted factors that may influence the extent to which ASF has supported an increase in collaborative working within and across schools.^[15]

5.29. A shared vision for the school and set of common values were seen by some as crucial in facilitating greater collaboration. Follow-up participants referred to a unified understanding of the school's purpose, and shared commitment from staff to improve outcomes for young people, as key elements in ensuring collaborative working if effective in supporting these improved outcomes:

"It is only through establishing a shared vision of what our school community is for that we can work together to achieve positive outcomes for pupils. Staff understand their role in this process - every strategy employed within the school can be linked back to the vision."

5.30. This qualitative feedback also provided examples where schools felt that their collaborative working had developed in recent years to be a key strength, such that this was not necessarily seen as a key focus for further improvement:

"Collaboration is already working at a high level within our school. I would say that this is not an area which requires improvement, but we are constantly on the lookout for further opportunities to enhance our already strong collaborative approach."

Parental and family engagement

5.31. As noted earlier in this section, survey respondents saw parental and family engagement as a key element for progress in closing the poverty-related gap. Nearly 1 in 3 respondents noted parental engagement as a factor in their success to date, although nearly 1 in 4 also suggested that challenges in engaging parents had been a barrier to their progress.

5.32. Comments from these schools highlighted the importance of parents and families in supporting improvement for pupils affected by poverty. This was particularly in terms of supporting pupil attendance and engagement (parental engagement has formed a key part of the approach to supporting disengaged pupils for some schools) and improving pupil aspirations.

5.33. Schools noted that a clear commitment to, and recognition of the value of, parental engagement is required. This was particularly in the context of substantial time often being required to build relationships with families. Schools referred to some parents having negative associations with the school environment and cited examples of parental engagement having been built gradually, including through word-of-mouth. Several schools noted that, through ongoing engagement, parents have come to see themselves as part of the school community. Benefits were also reported around parents being more willing to volunteer and engage with school activities, and seeing the school as a source of help and support.

5.34. Comments from schools also highlighted a number of specific approaches to achieving and maintaining parental engagement. These included use of extra-curricular and physical/sports activities designed to engage parents' interests – and ensuring free access to activities for parents and pupils. Comments also referred to the importance of ensuring parents are able to engage with the school in a way they are comfortable with; ensuring parental engagement events are welcoming and informal (including details such as providing refreshments), and providing opportunities for individual engagement or feedback if this is preferred.

5.35. Follow-up qualitative feedback also included reference to the value of a clear message around the positive impact that parental engagement can have for pupils, and on what is (and is not) expected of parents and carers.^[16]

5.36. Follow-up participants raised concerns that parents can feel overwhelmed with messages around their role in their child's education, and may see 'parental engagement' as an additional expectation on their time. Some schools perceived a need for greater clarity, nationally and locally, on how parents can positively support their child's learning, without this being seen as an undue burden.

Unintended consequences

5.37. The survey asked headteachers about any unintended consequences of ASF support. Around 1 in 3 (33%) respondents had seen unintended positive consequences as a result of their receipt of ASF funding, and around 1 in 8 (13%) had seen unintended negative consequences. These findings were broadly consistent across key respondent groups, although Schools Programme respondents were more likely than others to have seen unintended consequences.

5.38. The survey provided an opportunity for respondents to expand on the unintended consequences they had seen as a result of ASF support. Responses are summarised below.

5.39. Headteachers were asked to highlight consequences which were not intended or anticipated by their own planning, rather than commenting on the wider policy intentions of the Attainment Scotland Fund. It is notable that a number of the consequences mentioned by respondents – such as improved collaboration and partnership working – are key policy intentions for the Fund.

5.40. Headteacher responses were also broadly consistent with unintended consequences reported in previous surveys. For example, the two most commonly mentioned positive (improved partnership working and staff training/skills development) and negative (additional workload and reporting requirements) consequences remained unchanged from the 2018 survey.

Unintended positive consequences (n=310) ^[17]

More and better collaboration and partnership working	25%
Training and skills development for staff	15%
More leadership opportunities for staff	13%
Improved engagement with parents/families	13%
Improved pupil engagement and attendance	12%
Change in school ethos/culture, a shared focus	9%
Better awareness and understanding of the impact of poverty on attainment/wellbeing, and what 'equity' means	7%
More and better use of data/evidence	6%

Unintended negative consequences (n=144)

Additional workload for leadership, management and administrative roles	26%
Reduction in wider school budgets and external supports, loss of free access to 3rd sector support	15%
Difficulties in staff recruitment and retention	11%
Potential for schools, pupils or parents to feel excluded, difficulty around what 'equity' means	8%
Reporting/evaluation requirements and timescales, pressure to demonstrate improvement	5%
Concerns around stigma for pupils/families	4%
A lack of autonomy in planning/implementation	2%