Attainment Scotland Fund evaluation: interim report (years 1 and 2)

2. Methodology

2.1. This chapter provides an overview of the evaluation strategy for the Attainment Scotland Fund. It describes the research questions driving the evaluation activities and gives a summary of the different data sources used.

Overview of Evaluation Design

2.2. The evaluation aims to provide learning about the overall implementation of the fund and the extent to which the aims of the Attainment Scotland Fund have been met. More widely, it will contribute to the Scottish evidence base around what works or does not work to improve attainment and close the attainment gap.

2.3. The evaluation adopts a mixed methods approach that combines a range of different data sources. The evaluation plan has been designed to respond flexibly to any further decisions and activities around the Scottish Attainment Challenge.

Evaluation Questions

2.4. The evaluation questions were developed in partnership with key stakeholders to address the objectives of this evaluation.

2.5. Table 2.1 portrays the 13 questions that informed the evaluation design.

Table 2.1: Evaluation Questions

1.	What did and didn't work well in the national and local governance and support as part of the fund?
2.	How much funding did LAs and schools receive, to what extent did they consider it adequate, supplement it with other funding sources, and use it in accordance with the fund’s requirements?
3.	To what extent do stakeholders understand, engage and further the programme aims, and why?
4.	What type of initiatives were organised and to what extent did they focus on literacy, numeracy, health and wellbeing or other topics?
5.	How were interventions targeted?
6.	To what extent did the interventions succeed in reaching the target groups?
7.	To what extent did the interventions achieve their short and medium term outcomes?
8.	To what extent did the different types of interventions succeed in improving attainment and health and wellbeing, and why?
9.	To what extent are interventions sustainable beyond the four years of the funding?
10.	To what extent did the fund contribute to an improvement in attainment and health and wellbeing, and a reduction of the gap between pupils from the most and least deprived areas?
11.	To what extent did schools and authorities use data, analysis and knowledge of what works to drive improvements as part of the fund?
12.	To what extent has the fund encouraged collaboration and why?
13.	Did the fund have any unintended consequences?

Methods

2.6. Multiple data sources fed into the evidence collected. These include:

• Quantitative data from attainment related measures
• Scottish Government administrative data
• Challenge Authority reports
• School reports
• Surveys (local authority survey and headteacher survey)
• Qualitative research study

2.7. A summary of each of the different data sources employed follows.

Attainment Related Measures

2.8. The Scottish Government currently gathers or has access to data on attainment of pupils using a variety of measures. For the purpose of this evaluation, data has been gathered for the set of key and sub measures as selected in the National Improvement Framework.

2.9. An overview of the measures used to monitor the long term outcomes of the Fund is provided in Table 2.2.

Table 2.2: Measures used to monitor ASF long-term outcomes

Measure	Pre ASF	Year 1 (2015/16)	Year 2 (2016/17)	Year 3 (2017/18)	Year 4 (2018/19)
New Group Reading Test ( NGRT)		✔	✔
SCQF Level 5 & 6	✔	✔	✔	✔	✔
Achievement of Curriculum Levels ( ACEL)		✔	✔	✔	✔
Attendance Rates	✔		✔		✔
Exclusion Rates	✔		✔		✔
Participation Measure			✔	✔	✔

2.10. In addition, the New Group Reading Test ( NGRT) assessed the reading attainment of a sample of pupils attending schools involved in the Scottish Attainment Challenge Programme during the first two years.

2.11. For the purposes of this evaluation, we have presented the data, where possible, at a national level and for each of the nine Challenge Authorities: Clackmannanshire, Dundee City, East Ayrshire, Glasgow City, Inverclyde, North Ayrshire, North Lanarkshire, Renfrewshire and West Dunbartonshire.

2.12. For each local authority, the data is also shown, again where possible, by pupils living in the most and least deprived areas. This has been defined using the Scottish Index of Multiple Deprivation ( SIMD). That is, we compare the outcomes of pupils living in the 20% most deprived areas ( SIMD quintile 1) to those living in the 20% least deprived areas ( SIMD quintile 5).

2.13. The current report provides findings relating to the first two years of the Attainment Scotland Fund. Data available for Year 1 (2015/16) and Year 2 (2016/17) is included in this report. Where possible, data prior to the implementation of the fund is also included to provide some context.

SG Administrative Data

2.14. Scottish Government policy officials shared information collected as part of the routine monitoring with the evaluation team.

2.15. This information was mainly used to provide background and contextual framing for the evaluation. The data also helped to answer research question two on the funding local authorities and schools received, discussed in more detail in Chapter 4.

Challenge Authority Reports

2.16. Throughout the implementation of the fund, Challenge Authorities were asked to provide written documentation of their planned activities and progress. Table 2.3 outlines the reports gathered from Challenge Authorities.

Table 2.3: Challenge Authority Documentation

	Report	Submitted by	Included in current report
Year 1 (2015/16)	Proposed initiatives for first year	7 Challenge Authorities	✔
	Mid-Year Reports		✔
	End of Year Reports		✔
Year 2 (2016/17)	Proposed initiatives for second year	9 Challenge Authorities	✔
	Mid-Year Progress Reports		✔
	End of Year Progress Reports		✔
Year 3 (2017/18)	Proposed initiatives for third year
	Mid-Year Progress Reports
	End of Year Progress Reports
Year 4 (2018/19)	Proposed initiatives for third year
	Mid-Year Progress Reports
	End of Year Progress Reports

2.17. Document analysis was undertaken on the contents of these reports, using the 13 evaluation questions as a priori codes for analysis. The analysis was carried out separately by two analysts within Scottish Government Learning Analysis before meeting to discuss interpretations. This report provides the findings from analysis of Year 1 and 2 progress reports and plans.

School Reports and Plans

2.18. Information was extracted from 2016/17 Evaluative Reports and 2017/18 proposals submitted by the 46 primary schools and 28 secondary schools included in the Schools Programme in May 2017. The interventions and approaches included within these reports represent only the work schools undertook using Attainment Scotland Fund money. This may not represent the totality of the work that schools carried out during 2016/17 that positively impacted on attainment.

Headteacher Survey

2.19. An annual online survey collected the views and experiences of all headteachers of schools involved in the Challenge Authorities or Schools Programme.

2.20. The first wave was in Autumn 2016 and was distributed to primary school headteachers in the 7 Challenge Authorities and 57 headteachers of those in the Schools Programme. Overall, 181 headteachers responded to the survey. This represented a response rate of 40%.

2.21. The second wave took place in Autumn 2017 and was distributed to headteachers of primary and secondary schools in the 9 Challenge Authorities and 74 headteachers of those in the Schools Programme. Overall, 315 headteachers responded to the survey, representing a 52% response rate.

Figure 2.1: Headteacher Survey (Years 1 & 2)

Local Authority Mini Survey

2.22. In April 2016, a short online survey was sent to project leads in seven Challenge Authorities receiving Attainment Scotland Funding at that time.

2.23. The purpose of this survey was to ascertain their experiences of the fund so far ( i.e. end of Year 1).

2.24. The survey involved six questions asking participants about their perspectives on what was working well; what could be improved; positive and negative consequences of the fund; and sustainability.

2.25. Responses were received from six of the seven Challenge Authorities and this data is included where appropriate in the current report.

Qualitative Research Study

2.26. Research Scotland was commissioned to conduct a qualitative research study exploring the confidential experiences of key stakeholders involved in the Attainment Scotland Fund, such as headteachers, teachers and other school staff, parents, Attainment Advisors and local authority Directors and project leads.

2.27. The research was externally commissioned to promote reliable findings and the overall evaluation's credibility. Findings from the qualitative research have been triangulated with data gathered in-house.

2.28. Semi-structured interview guides were developed with input from the Research Advisory Group, comprising of staff from the Scottish Government's Raising Attainment Policy Unit, the Learning Analysis Unit and Education Scotland. The guides were designed to explore topics related to the 13 evaluation themes. Fieldwork took place between May and September 2017.

2.29. The qualitative research informed the full range of evaluation themes. The findings have been used alongside the other data sources in this report to provide a comprehensive understanding of the implementation and impact of the fund. The full report from Research Scotland is included in Appendix A.