Attainment Scotland Fund evaluation: headteacher survey 2018

2. Methodology

2.1. This section summarises the survey fieldwork approach, and the level and profile of response.

Survey design and fieldwork

2.2. Survey content was adapted from previous exercises to maintain longitudinal data, and was adapted to ensure relevance to the experience of schools in receipt of Pupil Equity Funding only, included in the survey for the first time in 2018.

2.3. The survey sample comprised 100% of schools in receipt of Challenge Authority and Schools Programme funding, and a sample of schools in receipt of PEF-only stratified by urban/rural geography and the level of PEF allocation. The survey sample is summarised below

Survey sample structure

2.4. The survey was issued direct to schools by email in October 2018, supported by promotion via Education Scotland and local authorities. The fieldwork period was maximised to enable the broadest possible response, running for 9 weeks to early December 2018.

2.5. The survey response and approach to survey weighting is summarised over the following pages, with further detail provided in an associated Technical Report.

Survey response

2.6. A total of 553 responses were received by survey close, equivalent to an overall response rate of 40% and comparing with 52% in 2017. The lower overall response is due to a lower 29% response from 'PEF-only' schools included in the survey for the first time this year; the 47% response rate for Challenge Authority schools is similar to that achieved in 2017, and the 69% Schools Programme response is 10 points higher than 2017.

2.7. It should also be noted that the larger survey sample in 2018 means that the volume of responses is significantly larger than that achieved by previous surveys; 553 compare to 315 responses in 2017. The larger volume of responses has enabled more detailed analysis to identify variation in views and experiences across key respondent groups.

Survey response 2016 to 2018

2.8. The profile of survey respondents is summarised below. The largest groups are Challenge Authority schools, schools with a higher PEF allocation, primary schools and schools in urban areas. The lower response received from PEF-only schools has resulted in some under-representation of this group - survey weighting has been used to correct for this response bias.

Profile of survey respondents (n=553)

Note: Quartile 1 = lower PEF allocation, Quartile 4 = higher allocation.

Analysis and reporting

2.9. Survey data show some inconsistency between responses and data on ASF support provided to schools. For example, 84 Challenge Authority and 2 Schools Programme respondents indicated their school received only Pupil Equity Funding, suggesting some lack of clarity on funding streams. The final categorisation of respondents has sought to correct these inconsistencies.

2.10. Survey responses have been weighted by ASF stream and urban/rural location. This adjusts for response bias and to account for the over-sampling of Challenge Authority and Schools Programme funding streams in the original survey sample. Results presented in the remainder of this report are based on respondents to each question - i.e. they exclude non-respondents to individual questions unless we state otherwise.

2.11. Survey analysis has used 95% Confidence Interval tests to identify significant differences from previous survey findings, and across key respondent groups. These include ASF stream, PEF allocation, school sector and urban/rural location. Where variation across these groups is noted in the body of the report, this is based on a statistically significant difference between groups.

2.12. A full list of survey questions and tabular survey results are provided in an associated Technical Report.