Attainment Scotland Fund 2022 to 2026 evaluation: analytical plan - year 1 (2022/23)

Section 3: Proposed activity for Year 1

This section introduces the analytical activity we propose for Year 1 (2022-23) for each of the evaluation strands.

1. Process Evaluation

We will undertake a process study in the first year of implementation of the refreshed Scottish Attainment Challenge, in order to provide learning on what worked well and what could be improved in the process of implementing the ASF overall, as well as a specific focus at the funding stream level in terms of the Strategic Equity Fund, Pupil Equity Fund and Care Experienced Children and Young People Fund^[2].

The Process Evaluation will focus on Evaluation Questions^[3] related to:

1. Governance

What worked well and what could be improved in the national and local governance and support with implementation of the refreshed Scottish Attainment Challenge?

2. Funding

What funding was allocated through the Scottish Attainment Challenge Refresh ASF to schools and local authorities, to what extent was it used within funds requirements and/or supplemented with other funding sources? What were stakeholders views on the implementation of the new funding structure introduced with the Scottish Attainment Challenge refresh?

3. Implementation

How did local authorities implement the Strategic Equity Fund alongside Pupil Equity Fund and Care Experienced Children and Young People Fund?

4. Approaches

How do the approaches for equity support pupils (and parents/carers) from the most socio-economically disadvantaged backgrounds?

5. Monitoring and evaluation

How are schools and local authorities monitoring, refining and evaluating their approaches to address the poverty-related attainment gap?

In Year 1, we will utilise a range of approaches to gain an understanding at national, regional and local level and across stakeholders of the processes and early implementation of the Attainment Scotland Fund.

We will:

• Undertake a review of existing data and evidence.
• Design and conduct a survey of Local Authority Scottish Attainment Challenge Leads.
• Undertake interviews and focus groups with key stakeholders and partners.
• Publish process study report.

2. Thematic

Whilst individual interventions are outwith the scope of the evaluation, the Evaluation Strategy^[4] has broadened in scope to include thematic evaluation. Discussions with internal and external groupings (ASF Evaluation Working Group and Evaluation Advisory Panel) have suggested a number of initial areas for thematic exploration, based on emerging priorities, new areas of focus and continued exploration. Emerging insights into impact will also be sought on these thematic areas.

Planned analytical activity on the three initial thematic areas proposed are outlined below.

2.1 Readiness to learn

As a result of extensive stakeholder consultation and collaboration, the refreshed Scottish Attainment Challenge Logic Model published in March 2022 included a new outcome focus related to 'readiness to learn', with outcomes proposed at both short- and medium-term:

Short-term outcome: a sustained focus on children and young people's readiness to learn through focusing on engagement and attendance, confidence and wellbeing.

Medium-term outcome: improvements in engagement, attendance, confidence and wellbeing of children and young people.

Discussions with internal and external groupings (EWG, EAP) have suggested that this should be a priority for thematic exploration, with a new Evaluation Question developed to support this.

In Year 1, we will commence an aspect of the evaluation to provide learning and increase the evidence base on 'what works and what could be improved, for whom, and in what circumstances' on the 'readiness to learn' theme, and to gather evidence related to:

Evaluation Question 13: To what extent was there improvement in children and young people's readiness to learn through focusing on engagement, attendance, confidence and wellbeing?

We will:

• Undertake a review of existing data (February - May 2023).
• Develop a detailed proposal for a review of the literature/evidence related to readiness to learn in the context of closing the poverty-related attainment gap.
• Undertake the review and publish a report on the findings.
• Prepare proposals for primary data gathering in Year 2 which utilise the review of existing data and literature/evidence review to develop our understanding of readiness to learn in the context of the Attainment Scotland Fund. The initial proposal is to explore children and young people's, parents'/carers' and practitioners' perceptions of 'readiness to learn' in the context of the Scottish Attainment Challenge and to undertake more detailed in-depth qualitative exploration of themes emerging to dive deeper into current understandings and practices and to develop insights into 'what works, for whom and in what circumstances?'

2.2 Families and communities

An aspect of thematic evaluation was developed in the later years of the initial ASF evaluation, with a focus on exploring engagement with and support for families and communities. This yielded important insights into support and engagement with families and communities in the context of the ASF, such as the important role of family learning and family link workers and associated teams, and the importance of effective links with wider partners both in public and third sector, as highlighted in the ASF Year 6 evaluation report.

In Year 1, we will continue to explore the theme of engagement with, and support for, families and communities in the context of the ASF. This aspect of thematic evaluation will seek to provide learning and increase the evidence base on 'what works and what could be improved, for whom, and in what circumstances' on the 'families and communities' theme, and to gather evidence related to:

Evaluation Question 8: To what extent has the fund embedded engagement with and support for families and communities?

We will:

• Undertake a review of existing data (February - May 2023)
• In parallel with the existing data review, develop a detailed proposal for a small scale research study on the theme of families and communities and explore potential avenues for undertaking the study (e.g. commissioned basis, partnership). The initial proposal is that this will utilise a case study approach, working with selected local authorities to explore in detail approaches to engaging and supporting families and communities (selected for approaches identified in thematic analysis of existing data outlined above and through wider intelligence gathering including through Attainment Advisers). Case studies may be at the local authority level, at lower levels of geography (school cluster/school/stage) or focused on particular groups.
• Undertake, report and publish findings.
• In discussion with evaluation partners and wider stakeholders, consider whether further research activities on the families and communities theme are suggested by the findings and develop appropriate proposals for Year 2.

2.3 Voice – children and young people's and families and communities engagement in decision-making

There is a recognised need for a strengthened focus on the engagement of children and young people and families and communities in the ASF evaluation. As a result, the revised Evaluation Strategy includes a new Evaluation Question to support evidence gathering on this theme and this has been identified as a priority thematic focus for Year 1.

In Year 1, we plan to progress analysis to provide learning and increase the evidence base on 'what works and what could be improved, for whom, and in what circumstances' around the 'engagement in decision making' theme in support of:

Evaluation Question 12: To what extent were children and young people and their families and communities engaged in decision-making, what was the impact of this engagement, and was there evidence of engagement becoming embedded in the learner journey?

In Year 1, we will utilise a range of approaches in support of this evaluation focus, including reviewing existing data and gathering primary data. A range of initiatives are in progress to strengthen children and young people's voice in the Scottish Attainment Challenge^[5]. These provide the opportunity for the development of partnership approaches to gathering evidence on this theme.

We will:

• Undertake a review of existing data (February - May 2023)
• Work with evaluation partners to explore 'what works and what could be improved, for whom and in what circumstances' in engaging children and young people in decision-making in the context of the Scottish Attainment Challenge.
• Explore and cost options for question inclusion in national surveys (e.g. Young People in Scotland survey, Ipsos Parents Omnibus survey).
• Publish an initial report of findings.
• Develop potential primary data gathering options for Year 2, for example, perspectives of national representatives and practitioners at regional and local level, and of parents/carers.

3. Reporting on National Improvement Framework (NIF) measures

A core objective of the evaluation is to assess the impact of the overall fund in improving outcomes in educational attainment and achievement and health and wellbeing and closing the poverty-related attainment gap between the most and least disadvantaged children and young people^[6]. This relates to Evaluation Question 15: To what extent did the fund contribute to a closing of the attainment gap between the most and least socio-economically disadvantaged children and young people, in line with the refreshed Scottish Attainment Challenge Mission?

In Year 1, and throughout the remainder of ASF Evaluation 2022 - 26, annual reporting on progress in closing the poverty-related attainment gap using NIF measures will continue to be undertaken, taking an accessible and proportionate approach to reporting. This will bring together previously published data related to the NIF measures, including the eleven original key measures and the two new measures introduced following the consultation exercise in 2022^[7]. This will support ongoing reporting of key trends initiated in the ASF annual evaluation reports undertaken between 2017 and 2022.

We will:

• Undertake analysis on NIF measures and prepare a report for publication in April 2023.
• Reporting will align with the existing programme of statistical releases, with consideration given to improving accessibility of reporting for a range of different audiences.

4. Impact feasibility study

All aspects of the evaluation – including the thematic strand and NIF reporting – will provide insight into the impact of the programme. However, the 2022/23 period also provides an opportunity for in-depth and collaborative consideration of the most promising potential approaches to assessing impact which will be put in place for Years 2 – 5. Whilst this does mean that evidence of and reporting on impact will be more limited in Year 1, it is anticipated that the development of a robust plan for gathering impact assessment will lead to a strengthened multi-year evaluation.

This will take the form of a collaborative feasibility study which will aim to combine what we know about tried and tested methods, with expert input on new approaches and data sources. This may involve both partnership work with stakeholders (for example, through the ASF Evaluation Advisory Panel) and commissioned work.

The feasibility study will examine a wide range of potential evidence sources and methodologies. However, there will be a particular quantitative focus on existing data, data linkage, and potential longitudinal approaches; and a qualitative focus on the views and experiences of those working directly with children and young people.

The output of the impact evaluation strand in Year 1 will be an impact evaluation plan for Years 2 to 5 of the evaluation, based on the recommendations developed in the impact feasibility study.

We will:

• Utilise existing groups – such as the ASF Evaluation Advisory Panel – or establish a short-life working group to define scope of work and collaborative approach.
• Assess existing and new approaches to measuring impact.
• Publish an impact evaluation plan for Years 2 to 5 of the evaluation.