Attainment Scotland Fund evaluation: headteacher survey report 2019

2. Methodology

2.1. This section summarises the survey fieldwork approach, and the level and profile of response.

Survey design and fieldwork

2.2. The survey content was adapted from previous exercises to maintain longitudinal data, streamlined in response to feedback during the 2018 survey, and new questions added to reflect the focus of ongoing evaluation of ASF. These included questions on:

• Understanding of the challenges and barriers faced by pupils affected by poverty;
• Awareness of the range of approaches that could be used to close the poverty-related attainment gap, and confidence in selecting the approach(es) that would be most effective;
• The extent to which achieving equity in education, specifically in relation to the poverty-related attainment gap, is embedded within school communities;
• The extent to which the approach taken by schools has changed over the previous year;
• Views on the sustainability of the focus on closing the poverty-related attainment gap; and
• Views on PEF timescales and additionality.

2.3. The survey was issued to headteachers of all schools in receipt of ASF support via Challenge Authority, Schools Programme, and/or Pupil Equity Fund. The profile of schools across the three ASF streams and urban/rural geography^[3] is summarised below.

2.4. Consistent with previous surveys, survey invites were issued directly to schools, supported by promotion via Education Scotland and local authorities. The survey was issued in September 2019 and the fieldwork period maximised to enable the broadest possible response, running for eight weeks to early November 2019. The survey response and approach to survey weighting is summarised over the following pages, with further detail provided in a Technical Report included in the Supporting Documents.

Survey response

2.5. A total of 1,102 responses were received by survey close, equivalent to an overall response rate of 47%. This represents a seven-point increase in survey response since 2018, primarily due to a 14-point increase in response from PEF-only schools (43% compared to 29% in 2018).

2.6. It should also be noted that the larger survey population in 2019 (including all PEF-only schools for the first time) means that the volume of responses is nearly twice that achieved by the 2018 survey; 1,102 compared to 553 responses. The larger volume of responses has enabled more detailed analysis to identify variation across key respondent groups.

2.7. The table below summarises the profile of survey respondents, and compares this with all schools in receipt of ASF support. In terms of the profile of respondents, the largest groups are PEF-only schools, primary schools and schools in urban areas. This is broadly consistent with the profile of all schools in receipt of ASF support, although there are some areas of minor over and under-representation. The main points of note are:

• Challenge authority schools are over-represented (by five points) and PEF-only schools under-represented (by six points). This is due to the lower survey response rate for PEF-only schools. Survey weighting has been used to correct for this response bias.
• Schools in rural areas are under-represented (by four points). Again survey weighting has been used to correct for this.

Follow-up engagement

2.8. Additional written responses were requested from a small subset of survey respondents, to gather examples of how schools have experienced key themes around the development, implementation and impact of ASF supported approaches.

2.9. This follow-up engagement was focused around the broad themes noted below, with each participant asked to consider one of the six themes. A purposive sampling^[5] approach was used, using survey responses to ensure a mix of positive and negative experiences across the six themes. Selection of schools also included primary and secondary sectors, and urban/rural locations.

2.10. A total of 24 schools were invited to provide additional feedback, with 15 responses achieved. This included a mix of positive and negative experiences for most themes, with the exception of collaboration where the only response was from a school with a negative experience (although positive comments on collaboration were provided by schools responding to other themes). As such, follow-up engagement provides a mix of illustrative examples of schools' positive and negative experiences across these themes.

Analysis and reporting

2.11. Survey data showed some inconsistency between responses and data on ASF support provided to schools. For example, 47 Challenge Authority and two Schools Programme respondents indicated their school received only Pupil Equity Funding. The categorisation of respondents used in our analysis has been based on Scottish Government records rather than self-reporting.

2.12. Survey responses have been weighted by ASF stream and urban/rural location to adjust for response bias; all results presented in the remainder of this report are weighted. Base numbers for each survey question vary due to question non-response – i.e. they results exclude non-respondents to the question unless stated otherwise.

2.13. Survey analysis has used hypothesis tests with a 5% significance level to identify significant differences from previous survey findings, and across key respondent groups. These included:

• ASF stream;
• PEF allocation;
• School sector; and
• Urban/rural location.

2.14. Where variation across these groups is noted in the body of the report, this is based on a statistically significant difference between groups.

2.15. Qualitative feedback gathered through the follow-up engagement with schools is presented alongside survey results under the relevant themes. This includes direct quotes drawn from written comments provided by survey respondents, and follow-up engagement. This direct feedback has been edited for brevity and to ensure anonymity.

2.16. A full list of survey questions, tabular survey results and follow-up pro forma are provided in an associated Technical Report included in the Supporting Documents for this report.