Attainment Scotland Fund (years 3 and 4): evaluation strategy

2. Methodology

Overview of Evaluation Strategy

2.1. The Scottish Government and Education Scotland have worked together to ensure the evaluation of the Attainment Scotland Fund is comprehensive yet at the same time does not impose additional bureaucracy and thus hinder the fund's intention to promote flexibility and freedom.

2.2. To this aim, the evaluation strategy for Years 3 and 4 continues to make best use of existing data including, routine monitoring, surveys that are already in place, and evidence from attainment related measures. Where there are clear gaps in evidence, new work is planned to address these.

2.3. The Attainment Scotland Fund evaluation began in 2015 and follows the duration of the initiative to 2020. An evaluation of the first two years of the fund (from 2015 to 2017) has been published and is available on: www.gov.scot/scotlandfund-interimreport-years-1-2

2.4. Evaluation of PEF sits within the wider Attainment Scotland Fund evaluation and research activities considering PEF began in Year 3.

Aims of Evaluation

2.5. The evaluation of the Attainment Scotland Fund aims to provide learning about the overall implementation of the fund and the extent to which the aims of the fund have been met.

2.6. The evaluation has the following objectives:

• Assess the impact of the overall fund in improving attainment and health and wellbeing and reducing the difference between pupils from the most and least deprived areas.
• Assess the extent to which the further aims of the fund have been met: promote capacity for data-based self-evaluation and improvement and, encourage collaboration between schools and local authorities.
• Provide learning and increase the Scottish evidence base of what does and does not work to improve attainment and health and wellbeing, especially of pupils from the most deprived areas.
• Provide learning on what did and did not work well in the process of implementing the fund across participating Challenge Authorities and schools and which factors helped and hindered the fund achieving its outcomes.

2.7. These objectives are underpinned by a set of research questions which cover a variety of themes, including, amongst others, governance, targeting, sustainability, use of data and collaboration. The questions are:

1. What did and didn't work well in the national and local governance and support as part of the fund?

2. How much funding did LAs and schools receive, to what extent did they consider it adequate, supplement it with other funding sources, and use it in accordance with the fund’s requirements?

3. To what extent do stakeholders understand, engage and further the programme aims, and why?

4. What type of initiatives were organised and to what extent did they focus on literacy, numeracy, health and wellbeing or other topics?

5. How were interventions targeted?

6. To what extent did the interventions succeed in reaching the target groups?

7. To what extent did the interventions achieve their short and medium term outcomes?

8. To what extent did the different types of interventions succeed in improving attainment and health and wellbeing, and why?

9. To what extent are interventions sustainable beyond the four years of the funding?

10. To what extent did the fund contribute to an improvement in attainment and health and wellbeing, and a reduction of the gap between pupils from the most and least deprived areas?

11. To what extent did schools and authorities use data, analysis and knowledge of what works to drive improvements as part of the fund?

12. To what extent has the fund encouraged collaboration and why?

13. Did the fund have any unintended consequences?

2.8. In addition, specific questions have been included to address the introduction of Pupil Equity Funding in Year 3 of the fund. The questions are:

PEF 1: Did the fund have any unintended consequences?

PEF 2: How did Pupil Equity Funding and Attainment Scotland Fund interact in authorities and schools that received both?

PEF 3: What have we learned about the approach of giving significant resources to headteachers, compared with giving them to authorities?

Methods

2.9. The evaluation of the Attainment Scotland Fund gathers data from a broad range of sources. The Scottish Attainment Challenge team work very closely with other stakeholders who are also involved in collecting data on the progress and implementation of the Attainment Scotland Fund. Scottish Government will continue to work together with these stakeholders to inform our decisions about the further working of the fund.

2.10. The evaluation of the Attainment Scotland Fund has gathered data from a range of sources. These methods will continue to gather data relevant for the evaluation of Years 3 and 4, an include:

• Challenge Authority Reports
• Surveys (local authorities and headteachers)
• Qualitative Research involving schools across Scotland
• School Reports and Plans
• Quantitative data from attainment related measures agreed in the 2018 National Improvement Framework and Improvement Plan

2.11. In addition, we will undertake the following research activities to fill gaps in our knowledge base regarding PEF:

• Local Authority mini-survey
• Case Studies of schools receiving PEF

2.12. All data sources are described in detail in the next section.

2.13. Figure 2.1 overleaf gives an overview of the evaluation timeline. It shows when evaluation activities relevant to each of the planned data sources will take place.

Figure 2.1: Evaluation Timeline

Challenge Authority Reports

2.14. Challenge Authorities are required to submit reports twice a year to the Scottish Government on their activities and progress with regard to the Attainment Scotland Fund.

2.15. These reports do not directly ask Challenge Authorities to report on their experience of PEF. However, as Challenge Authorities will have received both types of funding, any reference to PEF is used to inform the evaluation.

2.16. This data is used primarily to help us understand the interaction between PEF and the Attainment Scotland Fund in Challenge Authorities receiving both types of funding.

Headteacher Survey

2.17. An annual online survey collects the views and experiences of the Attainment Scotland Fund from headteachers of schools in Challenge Authorities and the Schools Programme.

2.18. The next wave of the survey in Autumn 2018 will be expanded to include headteachers of schools in receipt of PEF only, as well as headteachers from Challenge Authorities and the Schools Programme. This will help us ascertain the impact of PEF on its own.

2.19. Following the Autumn 2018 wave, we will review the need to continue this survey on an annual basis to schools and consider the sample required.

Qualitative Research

2.20. In Year 2 of the Attainment Scotland Fund, Research Scotland were commissioned to conduct a qualitative research study exploring the experiences of key stakeholders involved in SAC. This external research study allowed us to triangulate data and enhance the reliability of the our findings.

2.21. This research explored the extent to which the fund was accomplishing its short and medium-term outcomes and what was and was not working well in the implementation of the fund.

2.22. In Year 4 of the fund, we plan to repeat this research to understand how the programme has evolved over time and identify highlights and challenges in the implementation of the fund.

2.23. In repeating this qualitative research, it will also specifically attempt to identify lessons from those involved with PEF. We will therefore expand the sample to include schools and local authorities in receipt of PEF only.

School Reports and Plans

2.24. An analysis of school plans provides information about the range of interventions selected with the relative focus on Numeracy, Literacy and Health and Wellbeing.

Attainment Advisor Quarterly Reports

2.25. Attainment Advisors complete Quarterly Reports which consider the activities, outputs and reach of the programme to provide a professional view of progress in schools and local authorities towards short-, medium- and long-term outcomes.

Rapid Outcome Assessments

2.26. Rapid Outcome Assessments ( ROAs) are conducted collaboratively with those the Attainment Advisors support (involving local authority teams in Challenge Authorities and headteachers in Schools Programme authorities). Their primary purpose is for informing improvement in Education Scotland delivered support. These assessments identify the role of other policies, such as NIF and PEF and how these interact with SAC to achieve programme outcomes.

Challenge Authority Inspections

2.27. Education Scotland is leading a programme of local authority inspections within the nine Challenge Authorities. This programme of inspection will be completed by December 2018. Each inspection will focus on evaluating how well the education leadership, governance and management arrangements of the Scottish Attainment Challenge and Pupil Equity Fund are being used to improve learning, raise attainment and close the poverty related attainment gap.

Attainment Related Measures

2.28. The long term outcomes of the Attainment Scotland Fund are assessed using the measures agreed in the 2018 National Improvement Framework and Improvement Plan. The plan sets out a basket of key measures and sub measures to assess progress.

2.29. Measures used in the evaluation are consistent with the plan and are outlined in Table 2.1.

Table 2.1: Attainment Related Measures

Measure	Participants	Source
Key Measures
Literacy and Numeracy: The proportion of children achieving expected levels in literacy and numeracy	Primary (P1, P4 and P7 combined) Secondary (S3)	Achievement of Curriculum for Excellence Levels ( ACEL) – Scottish Government
Qualifications: The proportion receiving 1 or more award at Level 5, and those receiving 1 or more award at Level 6	School leavers	Scottish Credit Qualifications – Scottish Government
Participation: The proportion participating in education, training or employment	16-19 year olds	Participation Measure – Skills Development Scotland
Total Difficulties Score: The proportion of children who had a borderline or abnormal score	4-12 year olds 13 & 15 year olds	Scottish Health Survey SALSUS
Sub Measures
Literacy and Numeracy: The proportion of children achieving expected levels	Primary: P1 P4 P7	Achievement of Curriculum for Excellence Levels ( ACEL) – Scottish Government
Mental Wellbeing: Score on the Warwick-Edinburgh Mental Wellbeing Scale ( WEMWBS)	13 & 15 year olds (boys and girls separately)	SALSUS
Attendance Rate: The proportion of pupils in attendance	Primary Secondary	Scottish Government
Exclusion Rate: The total number of exclusions by total number of pupils (rate per 1000 pupils)	Primary Secondary	Scottish Government

2.30. Whilst we are unable to determine the unique contribution the Attainment Scotland Fund has made, we are able to use the outcomes of these measures to collectively assess the extent to which the aims of the Attainment Scotland Fund have been met.

Case Studies

2.31. In order to gain a more in-depth understanding of the experience of schools receiving PEF, we aim to take a case study approach to gather data from around 10 schools.

2.32. Headteachers responding to the survey in Autumn 2018 will be invited to volunteer to take part in a follow-up case study which will specifically explore the experience of PEF within their school.

2.33. The case studies will help us gain a better understanding of the impact of PEF and the experience of receiving money directly from Scottish Government.

Local Authority Mini Survey

2.34. Whilst the headteacher survey and the follow up case studies help us to understand the school perspective, the experience of local authorities with regard to PEF is not something we can readily ascertain from existing data sources.

2.35. We therefore plan to distribute a short online survey to all 32 Local Authorities following the first year of PEF ( i.e. 2018/19).

2.36. The purpose of this survey will be to explore the support provided to schools, including any issues around the local authority's role as employer.

2.37. Both the distribution and analysis of the survey responses will be managed internally by Scottish Government.