Attainment Scotland Fund evaluation: headteacher survey 2018 - appendix

1. Survey methodology and response

1.1. Scottish Government commissioned the survey to include headteachers of schools in receipt of Challenge Authority, Schools Programme and/or Pupil Equity Funding. The overall aim is to build on learning from previous surveys to further improve operation of the ASF, and to maximise the impact of programmes supported by the Fund. This includes the following specific objectives:

• Provide insight on the experience of headteachers benefiting through each of the ASF streams, identifying any variation in experience or views across schools;
• Build on longitudinal data to monitor changes over time; and
• Provide evidence of what is working and what is not working well to inform ongoing delivery of the ASF.

1.2. Survey content was adapted from previous exercises to maintain longitudinal data, and was adapted to ensure relevance to the experience of schools in receipt of Pupil Equity Funding only, included in the survey for the first time in 2018. The survey sample comprised 100% of schools in receipt of Challenge Authority and Schools Programme funding, and a sample of schools in receipt of PEF-only stratified by urban/rural geography and the level of PEF allocation. The survey sample is summarised below.

Survey sample structure

1.3. Urban/rural stratification of the survey sample was based on the 6-fold Scottish Government urban/rural classification:

• Urban area: schools in areas classified as '1: large urban' or '2: other urban';
• Small town: schools in areas classified as '3: accessible small town' or '4. remote small towns'; and
• Rural area: schools in areas classified as '5. accessible rural areas' or '6. remote rural areas'.

1.4. PEF is allocated on basis of the number of pupils in receipt of Free School Meals. For the purposes of this research, different thresholds were calculated for Primary and Secondary schools. The table below displays the thresholds calculated for PEF. A primary school that received up to £7,200 is in the bottom 25% of PEF allocations. Whereas a primary school that received over £66,600 is in the group of those that receive the top 25% of PEF allocations. In secondary schools, those that received up to £33,600 are in the bottom 25% of PEF allocations and those that received above £102,000 are in the top 25% of PEF allocations.

PEF threshold by school type (2017/18)

1.5. The survey was issued direct to schools by email in October 2018, supported by promotion via Education Scotland and local authorities. The fieldwork period was maximised to enable the broadest possible response, running for 9 weeks to early December 2018.

Survey response

1.6. A total of 553 responses were received by survey close, equivalent to an overall response rate of 40% and comparing with 52% in 2017. The lower overall response is due to a lower 29% response from 'PEF-only' schools included in the survey for the first time this year; the 47% response rate for Challenge Authority schools is similar to that achieved in 2017, and the 69% Schools Programme response is 10 points higher than 2017.

Overview of survey response

1.7. The profile of survey respondents is summarised over the page. The largest groups are Challenge Authority schools, schools with a higher PEF allocation, primary schools and schools in urban areas. The lower response received from PEF-only schools has resulted in some under-representation of this group - survey weighting has been used to correct for this response bias.

Profile of survey respondents (n=553)

Note: Quartile 1 = lower PEF allocation, Quartile 4 = higher allocation.

Analysis and reporting

1.8. Survey data show some inconsistency between responses and data on ASF support provided to schools. For example, 84 Challenge Authority and 2 Schools Programme respondents indicated their school received only Pupil Equity Funding, suggesting some lack of clarity on funding streams. The final categorisation of respondents has sought to correct these inconsistencies.

1.9. Survey responses have been weighted against the overall profile of schools in receipt of ASF funding, based on ASF stream and urban/rural location. This has been designed to adjust for survey response bias, and to account for the over-sampling of Challenge Authority and Schools Programme funding streams in the original survey sample.

1.10. Survey analysis has used 95% Confidence Interval tests to identify significant differences from previous survey findings, and across key respondent groups. These include ASF stream, PEF allocation, school sector and urban/rural location. Where variation across these groups is noted in the main survey report, this is based on a statistically significant difference between groups.

1.11. The survey included several open ended questions where respondents were invited to answer in their own words. Key word analysis has been used to identify the main themes emerging through responses to these questions; for questions where sufficient number of written responses have been received, we provide tabular results of this analysis in the main report.