Attainment Scotland Fund evaluation - Headteacher Survey 2021: technical report

1. Survey methodology and response

1.1. The Scottish Government commissioned the survey to include headteachers of schools in receipt of Challenge Authority, Schools Programme and/or Pupil Equity Funding. The overall aim was to gather information in order to build on learning from previous surveys to further improve operation of the ASF, and to maximise the impact of programmes supported by the Fund. This included the following specific objectives:

• Provide insight on the experience of headteachers benefiting through each of the ASF streams during the 2020/21 school year, and understanding the impact of COVID-19 on ASF-supported approaches;
• Consider whether (and how) experiences have varied across key groups;
• Build on longitudinal data to monitor changes over time; and
• Provide evidence of what is working and what is not working well to inform ongoing delivery of the ASF.

1.2. This was the sixth Headteacher Survey since 2016. The survey scope and content have evolved over this period, and for the present survey this involved some revisions to reflect changes in COVID-19 related government restrictions since the 2020 survey, and to reduce the number of ‘open’ free text questions that headteachers are asked to complete. The main themes explored through the survey were:

• Development of ASF-supported approaches including understanding of the challenges and barriers faced by pupils affected by poverty, capacity to select from the range of approaches that could be used to close the poverty-related attainment gap, and the extent to which achieving equity in education is embedded within school communities;
• Use of data and evaluation, including views on skills and capacity within the school to use evidence to develop approaches and measure their impact;
• Impact achieved in terms of closing the poverty-related attainment gap including how COVID-19 had affected progress to date, and views on the sustainability of progress and the focus on equity in education;
• The extent to which ASF support has contributed to an increase in collaborative working, within and between schools, and with other agencies;
• Views on processes around Pupil Equity Funding including developing a PEF plan, and the specific impact of PEF.

1.3. The survey sampling approach was modified from the previous survey, with a return to a 100% sample approach. On this basis, the survey was issued to headteachers of all Challenge Authority and Schools Programme schools, and all those in receipt of Pupil Equity Funding only. As a result, the survey sample for the present survey was around 50% larger than the 2020 survey sample.

1.4. The profile of schools across the three ASF streams and urban/rural geography is summarised below.

Note: ‘Unknown’ urban/rural location is recorded where school postcode information could not be matched to geocoding data.

1.5. Urban/rural stratification of the survey sample was based on the 6-fold Scottish Government urban/rural classification^[1]:

• Urban area: schools in areas classified as ‘1: large urban’ or ‘2: other urban’;
• Small town: schools in areas classified as ‘3: accessible small town’ or ‘4. remote small towns’; and
• Rural area: schools in areas classified as ‘5. accessible rural areas’ or ‘6. remote rural areas’.

1.6. Consistent with previous surveys, survey invites were issued directly to schools, supported by promotion via Education Scotland and local authorities. The survey was issued in October 2021 and the fieldwork period ran to mid-November 2021.

Survey response

1.7. A total of 597 responses were received by survey close, equivalent to an overall response rate of 25%. The level of survey response is similar to the 27% achieved by the 2020 survey, but remains substantially lower than response to the 2019 survey (47%). This is likely to reflect in part the continuing pressures on schools associated with the COVID-19 pandemic; the present survey was conducted in October and November 2021 with high COVID-19 case rates across Scotland.

1.8. In this context, we very much appreciate those schools able to participate. Moreover, the increased sample size used for the present survey means that the volume of responses is nearly 50% higher than the 2020 survey.

1.9. The table below summarises the profile of survey respondents, and compares this with all schools in receipt of ASF support. In terms of the profile of respondents, the largest groups are PEF-only schools, primary schools, and schools in urban areas. This differs somewhat from the profile of all ASF-supported schools on a number of indicators:

• PEF-only schools are over-represented and Challenge Authority schools under-represented due to a lower response rate for Challenge Authority schools. Survey weighting has been used to correct for this response bias.
• Schools with lower PEF allocations are under-represented, and those with higher allocations are over-represented.
• Schools in rural areas are under-represented, and schools in urban areas over-represented. Again, survey weighting has been used to correct for this.

Analysis and reporting

1.10. Survey responses have been weighted by ASF stream and urban/rural location to adjust for response bias^[3]; all results presented in the remainder of this report are weighted. Base numbers for each survey question vary due to question non-response – i.e. the results exclude non-respondents to the question unless stated otherwise.

1.11. The volume of survey responses permitted more detailed analysis across key respondent groups. Survey analysis has used hypothesis tests with a 5% significance level to identify significant differences across these groups, and from previous survey findings.^[4] These included:

• ASF stream;
• PEF allocation;
• School sector; and
• Urban/rural location.

1.12. Where variation across these groups is noted in the body of the survey report, this is based on a statistically significant difference between groups.

1.13. Survey data showed some inconsistency between responses and data on ASF support provided to schools - for example, 20 Challenge Authority respondents indicated that their school received only Pupil Equity Funding. Consistent with previous surveys, the categorisation of respondents used in our analysis has been based on Scottish Government records rather than self-reporting.

1.14. The survey included a number of questions giving headteachers the opportunity to respond in their own words. This feedback has been coded into broad themes, and the results presented in the survey report. This includes the percentage of respondents coded into each theme – note that these percentages are based on those answering the question, and respondents can be assigned to multiple themes. Presentation of written feedback also includes direct quotes - this material has been edited for brevity and to ensure anonymity.