Attainment Scotland Fund evaluation: headteacher survey 2018

6. Impact and sustainability

6.1. This section sets out findings on views and experiences on the impact of funded interventions. This includes views on the factors that contribute to or hinder success, whether positive impacts are likely to be sustainable, the extent to which ASF support has contributed to an increase in collaborative working, and whether schools have seen any unintended positive or negative consequences of ASF support.

Impact of interventions

6.2. Nearly 9 in 10 (88%) schools have seen an improvement in closing the poverty-related gap in attainment and/or health and wellbeing as a result of ASF supported interventions. A minority of these (12%) have seen 'a lot' of improvement to date, although more than half (56%) of respondents are expecting to see a lot of improvement over the next five years as a result of ASF supported interventions (see figure below). Survey responses indicate that those who have seen some improvement to date are significantly more likely to expect further improvement over the next five years.

6.3. There has been a 10 point increase from the 2017 survey in the proportion of schools seeing an improvement in closing the gap. The survey also shows some variation in views across funding streams, with Schools Programme respondents most likely and PEF-only least likely to have seen an improvement. This difference may reflect that PEF-only schools did not have access to ASF support prior to 2017/18.

Perceived improvement in closing the poverty-related gap in attainment or health/wellbeing

6.4. More detailed survey analysis has been undertaken to assess the strength of correlation between perceived improvements in wellbeing/attainment, and a range of other factors and respondent groups. This indicates that the following groups are significantly more likely to have seen improvement as a result of ASF support interventions: those who feel confident using evidence to inform development of interventions; those who feel they had sufficient support available to develop their school plan; those who have an evaluation plan in place; and those who have seen an increase in collaborative working.

6.5. The survey also asked for headteachers' views on the factors that contribute to or hinder the success of ASF supported interventions. The factors most commonly raised by respondents are summarised below.

Contributing to success (n=245)

Teachers and staffing resource	70%[5]
Teacher training and skills development	18%
Resources and funding	18%
Family engagement to support pupils	17%
A shared set of objectives, collective focus	13%
Use of data and evidence to inform approaches	13%
Effective targeting of support	13%

Potential barriers to success (n=247)

Staffing, staff time and workload	51%
Resources and wider funding	24%
Family engagement	22%
Mental health, behavioural issues and support needs of pupils	14%
Wider community issues, pupils' experience at home and family wellbeing	14%
Organisational issues, reporting and paperwork requirements	12%
Difficulties ensuring effective targeting	10%
Teacher training and skills development	9%

Sustainability

6.6. Around 2 in 5 (42%) survey respondents expect that improvement as a result of ASF supported interventions will be sustainable beyond the years of funding. This represents a 16 point reduction from the 2017 survey, where 58% of respondents felt improvements would be sustainable. However, it should be noted that a substantial proportion (42%) of respondents were unsure of the sustainability of impacts, and only 16% were of the view that impacts would not be sustainable.

6.7. Survey findings also suggest a correlation between views on sustainability and perception of improvement to date; those who have seen improvement in attainment or wellbeing to date are significant more likely than others to expect improvements to be sustainable beyond funding.

6.8. The survey asked headteachers to expand on why they do (or do not) expect improvements to be sustainable. Responses give an indication of the factors influencing views here.

Why feel improvement will be sustainable (n=115)

Staffing training, skills development and capacity building	70%[6]
Embedded practice, pedagogy development	26%
Raising awareness and change of ethos/culture	20%
Ongoing access to resources	13%
Developed a collective, shared focus	10%
Developed capacity to use data/evidence to inform approaches	9%
Collaboration within school, with partners and parents	7%
Longer-term health and wellbeing impacts	6%

Why feel improvement will not be sustainable (n=47)

Loss of staffing, resources and skills	81%
Loss of initiatives and interventions	28%
Loss of external support and access to services	12%
Too early to tell	12%
Pupils' health and wellbeing needs	9%
Reduction in local funding/resourcing	7%

Collaborative working

6.9. The majority of headteachers have seen an increase in collaborative working in their school as a result of ASF support. Around 7 in 10 (71%) indicate this, including around 3 in 10 (29%) who have seen a large increase in collaborative working.

6.10. Survey responses indicate significant variation across funding streams in the extent to which ASF support has led to an increase in collaborative working. Schools Programme respondents are more likely than others to have seen an increase in collaborative working as a result of funding (98% have done so), while PEF-only schools are least likely to have seen such a change (66%).

Whether seen increase in collaborative working as result of Fund

6.11. The survey asked headteachers to expand on why they feel there has been an increase in collaborative working as a result of ASF support. These are summarised below.

Greater emphasis on collaboration from school leadership, empowerment of teaching staff	41%[7]
More staff resourcing and staff time	23%
Collaboration embedded as a key part of CPD	16%
Change of culture/ethos across the school	12%
More opportunities for collaboration and partnership	12%

N=227

6.12. A small number of respondents who feel the Fund has not led to greater collaboration within their school provided further comment on this. These respondents (around 2% of all survey respondents) most commonly referred to a culture of collaboration already being well-established prior to the school receiving ASF support. Some also referred to ASF support not having had sufficient impact on staff time or opportunities within their school for collaborative working.

Unintended consequences

6.13. In addition to views on the extent to which interventions had produced the anticipated impacts, the survey also asked headteachers about any unintended consequences of ASF support.

6.14. Nearly 2 in 5 (38%) respondents have seen unintended positive consequences as a result of their receipt of ASF funding, and nearly 1 in 5 (18%) have seen unintended negative consequences. These findings are broadly consistent across key respondent groups, although primary school respondents are more likely than secondary school respondents to have seen unintended negative consequences.

6.15. The survey asked headteachers to expand on the unintended consequences they had seen as a result of ASF support. Responses are summarised below.

Unintended positive consequences (n=108)

More and better collaboration and partnership working	28%[8]
Training and skills development for staff	21%
Improved pupil and parent engagement, attendance	19%
A change in school ethos/culture, a shared focus	16%
More leadership opportunities for staff	12%
Better awareness/understanding of the attainment and wellbeing gap, recognition of the value of targeting	7%
More and better use of data/evidence	7%

Unintended negative consequences (n=66)

Additional workload for leadership, management and administrative roles	39%
Reporting requirements, pressure to demonstrate improvement	20%
Reduction in other resourcing, loss of free access to 3rd sector support, perception that 'awash' with funding	9%
Potential for schools, pupils or parents to feel excluded where not the focus of ASF	9%
Recruitment difficulties	6%
Difficulty balancing ASF with other priorities	3%