Attainment Scotland Fund evaluation: headteacher survey 2019 - technical report

1. Survey methodology and response

Scottish Government commissioned the survey to include headteachers of schools in receipt of Challenge Authority, Schools Programme and/or Pupil Equity Funding. The overall aim was to gather information in order to build on learning from previous surveys to further improve operation of the ASF, and to maximise the impact of programmes supported by the Fund. This included the following specific objectives:

• Provide insight on the experience of headteachers benefiting through each of the ASF streams, identifying any variation in experience or views across schools;
• Build on longitudinal data to monitor changes over time; and
• Provide evidence of what is working and what is not working well to inform ongoing delivery of the ASF.

Survey content was adapted from previous exercises to maintain longitudinal data. Content was streamlined in response to feedback during the 2018 survey, and a number of new questions added to reflect the focus of ongoing evaluation of ASF. These included new questions to explore:

• Understanding of the challenges and barriers faced by pupils affected by poverty;
• Awareness of the range of approaches that could be used to close the poverty-related attainment gap, and confidence in selecting the approach(es) that would be most effective;
• The extent to which achieving equity in education, specifically in relation to the poverty-related attainment gap, is embedded within school communities;
• The extent to which the approach taken by schools has changed over the previous year; and
• Views on the sustainability of the focus on closing the poverty-related attainment gap (as distinct from the sustainability of any improvement).

The survey was issued to all schools in receipt of ASF support via Challenge Authority, Schools Programme, and/or the Pupil Equity Fund. The profile of schools across the three ASF streams and urban/rural geography is summarised below.

Urban/rural stratification of the survey sample was based on the 6-fold Scottish Government urban/rural classification^[2]:

• Urban area: schools in areas classified as '1: large urban' or '2: other urban';
• Small town: schools in areas classified as '3: accessible small town' or '4. remote small towns'; and
• Rural area: schools in areas classified as '5. accessible rural areas' or '6. remote rural areas'.

Consistent with previous surveys, survey invites were issued direct to schools, supported by promotion via Education Scotland and local authorities. The survey was issued in September 2019 and the fieldwork period maximised to enable the broadest possible response, running for 8 weeks to early November 2019.

Survey response

A total of 1,102 responses were received by survey close, equivalent to an overall response rate of 47%. This represents a 7-point increase in survey response since 2018, primarily due to a 14-point increase in response from PEF-only schools (43% compared to 29% in 2018).

The table below summarises the profile of survey respondents, and compares this with all schools in receipt of ASF support. In terms of the profile of respondents, the largest groups are PEF-only schools, primary schools and schools in urban areas. This is broadly consistent with the profile of all schools in receipt of ASF support, although there are some areas of minor over and under-representation. The main points of note are:

• Challenge authority schools are over-represented (by 5 points) and PEF-only schools under-represented (by 6 points). This is due to the lower survey response rate for PEF-only schools. Survey weighting has been used to correct for this response bias.
• Schools in rural areas are under-represented (by 4 points). Again survey weighting has been used to correct for this.

Follow-up engagement

Additional written responses were requested from a small subset of survey respondents, to gather illustrative examples of how schools have experienced key themes around the development, implementation and impact of ASF supported approaches.

This follow-up engagement was focused around the broad themes noted below, with each participant asked to consider one of the six themes. Schools were selected to ensure a mix of primary and secondary sectors and urban/rural locations, and a mix of positive and negative experiences in relation to each of the themes. A total of 24 schools were invited to provide additional feedback, with 15 responses achieved by fieldwork close.

Analysis and reporting

Survey data showed some inconsistency between responses and data on ASF support provided to schools. For example, 47 Challenge Authority and 2 Schools Programme respondents indicated their school received only Pupil Equity Funding, suggesting some lack of clarity on funding streams. The categorisation of respondents used in our analysis has been based on Scottish Government records rather than self-reporting.

Survey responses have been weighted by ASF stream and urban/rural location to adjust for response bias. Results presented in the survey report are based on respondents to each question – i.e. they exclude non-respondents to individual questions unless stated otherwise.

Survey analysis is based on hypothesis tests with a 5% significance level to identify statistically significant differences from previous survey findings, and across key respondent groups. These included:

• ASF stream;
• PEF allocation;
• School sector; and
• Urban/rural location.

Where variation across these groups is noted in the body of the survey report, this is based on a statistically significant difference between groups.

Qualitative feedback gathered through the survey and follow-up engagement with headteachers is presented alongside survey results. This includes direct quotes, some of which were edited for brevity and to ensure anonymity.