Attainment Scotland Fund Evaluation – Headteacher Survey 2020: technical report

1. Survey methodology and response

1.1. The Scottish Government commissioned the survey to include headteachers of schools in receipt of Challenge Authority, Schools Programme and/or Pupil Equity Funding. The overall aim was to gather information in order to build on learning from previous surveys to further improve operation of the ASF, and to maximise the impact of programmes supported by the Fund. This included the following specific objectives:

• Provide insight on the experience of headteachers benefiting through each of the ASF streams during the 2019/20 school year, and understanding the impact of COVID-19 on ASF-supported approaches during March to June 2020;
• Consider whether (and how) experiences have varied across key groups;
• Build on longitudinal data to monitor changes over time; and
• Provide evidence of what is working and what is not working well to inform ongoing delivery of the ASF.

1.2. This was the fifth Headteacher Survey since 2016. The survey scope and content have evolved over this period, and for the present survey this involved streamlining of existing questions to minimise the burden on schools, and a small number of new questions exploring the impact of COVID-19 on work to close the attainment gap. The main themes explored through the survey were:

• Development of ASF-supported approaches including understanding of the challenges and barriers faced by pupils affected by poverty, capacity to select from the range of approaches that could be used to close the poverty-related attainment gap, and the extent to which achieving equity in education is embedded within school communities;
• Use of data and evaluation, including views on skills and capacity within the school to use evidence to develop approaches and measure their impact;
• Impact achieved in terms of closing the poverty-related attainment gap including how COVID-19 had affected progress to date, and views on the sustainability of progress and the focus on equity in education;
• The extent to which ASF support has contributed to an increase in collaborative working, within and between schools, and with other agencies;
• Views on processes around Pupil Equity Funding, including developing a PEF plan.

1.3. The survey sampling approach was modified from the 2019 survey to minimise the burden on the sector in the context of ongoing school building closures, while ensuring sufficient volume of responses to meet analysis requirements. On this basis, the survey was issued to headteachers of all Challenge Authority and Schools Programme schools, and 50% of those in receipt of Pupil Equity Funding only. The profile of schools across the three ASF streams and urban/rural geography is summarised below.

Survey population by ASF stream

Note: 'Unknown' urban/rural location is recorded where school postcode information could not be matched to geocoding data.

1.4. Urban/rural stratification of the survey sample was based on the 6-fold Scottish Government urban/rural classification^[1]:

• Urban area: schools in areas classified as '1: large urban' or '2: other urban';
• Small town: schools in areas classified as '3: accessible small town' or '4. remote small towns'; and
• Rural area: schools in areas classified as '5. accessible rural areas' or '6. remote rural areas'.

1.5. Consistent with previous surveys, survey invites were issued directly to schools, supported by promotion via Education Scotland and local authorities. The survey was issued in January 2021 and the fieldwork period extended to enable the broadest possible response, running to mid February 2021.

Survey response

1.6. A total of 420 responses were received by survey close, equivalent to an overall response rate of 27%. This represents a 20-point decrease since 2019, primarily due to a 30-point decrease in response from PEF-only schools (13% compared to 43% in 2019). As noted above, survey fieldwork took place in early 2021 during a period of school building closures and an associated increase in pressure on school resources. In this context, we very much appreciate those schools able to participate. Moreover, the volume of response is sufficient to produce robust results to inform the wider ASF evaluation (see 'Analysis and Reporting' later in this section for further detail).

Overview of survey response

1.7. The table below summarises the profile of survey respondents, and compares this with all schools in receipt of ASF support. In terms of the profile of respondents, the largest groups are Challenge Authority schools, primary schools, schools in urban areas, and schools with a middle or upper PEF allocation. This differs from the profile of all ASF-supported schools on a number of indicators, in part due to the change in sampling approach for the present survey (i.e. the reduced sample size for PEF-only schools):

• Challenge Authority schools are over-represented and PEF-only schools under-represented. This is due to the 50% sample taken of PEF-only schools (compared to the 100% sample of CA schools), compounded by a lower response rate from PEF-only schools. Survey weighting has been used to correct for this response bias.
• Schools with lower PEF allocations are under-represented, and those with higher allocations are over-represented. This is also in part due to the under-representation of PEF-only schools (which typically have lower PEF allocations) and over-representation of CA schools (which typically have higher allocations).
• Schools in rural areas are under-represented, and schools in urban areas over-represented. Again, survey weighting has been used to correct for this.

Profile of survey respondents

Analysis and reporting

1.8. Survey responses have been weighted by ASF stream and urban/rural location to adjust for response bias; all results presented in the remainder of this report are weighted. Base numbers for each survey question vary due to question non-response – i.e. they results exclude non-respondents to the question unless stated otherwise.

1.9. Survey analysis has used hypothesis tests with a 5% significance level to identify significant differences from previous survey findings, and across key respondent groups. These included:

• ASF stream;
• PEF allocation;
• School sector; and
• Urban/rural location.

1.10. Where variation across these groups is noted in the body of the survey report, this is based on a statistically significant difference between groups.

1.11. Survey data showed some inconsistency between responses and data on ASF support provided to schools - for example, 28 Challenge Authority respondents indicated that their school received only Pupil Equity Funding. The categorisation of respondents used in our analysis has been based on Scottish Government records rather than self-reporting.

1.12. The survey included a number of questions giving headteachers the opportunity to respond in their own words. This feedback has been coded into broad themes, and the results presented in the survey report. This includes the percentage of respondents coded into each theme – note that these percentages are based on those answering the question, and respondents can be assigned to multiple themes. Presentation of written feedback also includes direct quotes - this material has been edited for brevity and to ensure anonymity.