Coronavirus (COVID-19): impact of school building closures - equity audit

Annex A: Methodology - Rapid evidence review

Phase 1

Introduction

Rapid evidence review methodology (also referred to as Rapid Evidence Assessment) was utilised to review a wide range of existing and current evidence. A rapid evidence review provides a more structured and rigorous search and quality assessment of the evidence than a literature review but is not as exhaustive as a systematic evidence review.

Rapid evidence reviews are most often employed where rapid access is needed to high quality evidence to inform policy, decisions and planning on emergent issues. Given the imperative to synthesise the large body of evidence available in relation to the impact of the COVID-19 pandemic on the educational experiences and attainment of children and young people affected by poverty, to be of utility to policy makers and the education sector, a rapid evidence review was considered the most appropriate methodology.

Process

Rapid evidence reviews employ a specific methodology that aims to identify the most relevant studies on a specific topic as comprehensively as possible. A key inquiry question (and sub questions – set out in the table further below) are agreed defining the search terms and methods for extracting evidence.

The selection of studies is based on explicit criteria. An element of quality appraisal, considering the methodological quality of the studies is also assessed. A synthesis of what the evidence indicates is produced usually in the form of a narrative report. The process of developing an evidence review generally follows the key stages set out below:

1. Develop inquiry question – What does this rapid evidence review answer?

2. Inclusion Criteria – Which studies will be taken into account?

3. Search Strategy – How were the studies sought?

4. Selection process – How were the studies selected?

5. Critical Appraisal – How was the quality of the studies determined?

6. Results – Definitions and thematic summaries

7. Synthesis of main findings

Equity Audit Stage 1 – Rapid Evidence Review - Methodology

Inquiry Questions

Key Inquiry question:

1. What does the existing research and wider evidence suggest has been the impact of the COVID-19 pandemic on the educational experiences and attainment of children and young people, and in particular those children and young people affected by poverty.

Sub questions:

1. What has been the impact of the COVID-19 pandemic on educational experiences and attainment – for all children and young people and in particular for children and young people affected by poverty?

2. What evidence is there from children and young people’s perspective/voice and in particular on the voice of children and young people whose educational experiences and attainment are affected by the poverty-related attainment gap, on the impacts of school building closures and move to remote learning?

3. What policy and practice responses have education systems developed during the COVID-19 pandemic specifically to address children and young people whose educational experiences and attainment are affected by the poverty-related attainment gap and what does the emerging research/evidence base suggest has been the impact of policy and practice responses?

4. What factors are emerging which are associated with mitigating the impact of COVID-19 on children and young whose educational experiences and attainment are affected by the poverty-related attainment gap?

Inclusion Criteria

Inclusion

1. Material published in English between 23/3/20 to 1/9/20 covering Scotland/devolved nations/UK-wide /Europe/International

2. Material focused on children and young people aged 5 up to age 18 in a school setting (early years is out of scope)

3. Material focused on educational experiences, outcomes with focus on poverty disadvantage, interaction of poverty disadvantage with ethnicity, ASN/SEN

4. Material included- surveys and other primary research, systematic review, think pieces, narrative review, articles

Exclusion

1. Material not published in English

2. Material related to higher or further education

3. Material with a clinical focus

4. Articles that are opinion pieces, editorials or letters

5. Studies that have a pre-COVID-19 focus on impact of school building closures (Out of scope however some material will be used as context setting)

Search Strategy

Three key methods were employed to gather relevant evidence:

1) Systematic searches conducted on our behalf by Scottish Government Library

2) Evidence gathered through stakeholder engagement exercise with:

• members of CERG, and
• with Research Strategy for Education in Scotland National Advisory Group (NAG) and Academic Reference Group (ARG)

3) Ad hoc evidence gathered by Equity Audit Phase 1 Short Life Working Group members through ongoing research engagement, website monitoring, newsfeeds and other updates.

Search Terms/keywords

Children/young people

Schools

Covid-19/pandemic/communicable diseases

Education 5 – 18 years

Learning loss

Attainment gap

Equity/deprivation/poverty/disadvantage/low income

School education

Teacher

Pupil recovery

Education recovery

Home learning

School (building) closures

Blended learning

Remote learning

Digital access

Digital exclusion

Parental/family involvement

Parental/family engagement

ASN/SEN

Pupil wellbeing/mental health

Teacher wellbeing/mental health

Community

Critical Appraisal

We focused on the quality and rigour of the evidence and in addition we identified the difference in source material between research studies and evidence collection. We also took into consideration of a significant number of media articles and think pieces. Editorials, articles promoting a specific agenda or viewpoint unsubstantiated by evidence were discounted. Assessing the broad scope of material available was crucial in gaining a clear picture of what the evidence is telling us. In light of this we considered other rapid evidence reviews and evidence collected by key stakeholders such as third sector organisations and charitable bodies.

Results, Definitions and thematic summaries

All ‘in scope’ relevant references identified through the literature searches were added to the spreadsheet. Initial thematic coding was developed to code material. Working group members created thematic summaries for the majority of themes.

Policy responses

• Policy approaches
• Global policy responses – generic
• Global policy responses – specific to disadvantage/poverty/deprivation

Socio-demographic

• Rurality
• Changing economic circumstances
• Communities

Practice responses

• Remote learning
• Blended learning
• Recovery – re-opening schools

Impacts

• ‘Learning loss’ and attainment
• Learning experiences

Voice (cross cutting theme)

• Child/pupil voice
• Parent voice
• Teacher voice

Resources

• Parental involvement/engagement
• Availability of resources in the home
• Digital divide
• Food poverty

Vulnerabilities

• Vulnerable young people
• Care-experienced children and young people
• Children and young people with caring responsibilities
• New vulnerabilities

ASN/SEN

Mental health and wellbeing

Synthesis of key findings

Thematic summaries were used develop a narrative report which highlighted key findings from the evidence, structured according to the inquiry questions.