Coronavirus (COVID-19) teacher and lecturer estimates - 2020 results: EQIA

Equality Impact Assessment

Title of policy

2020 Results using teacher and lecturer estimates

Summary of aims and desired outcomes of policy

This policy aims to reward pupils for their predicted achievements in national qualifications, ensuring confidence in the fairness and value of Scottish education.

Directorate: Division: Team

Learning Directorate: Curriculum, Qualifications and Gaelic: Senior Phase Unit

Coronavirus (COVID-19): 2020 Results – teacher and lecturer estimates

Executive Summary

The aim of the policy to award results in 2020 is to help achieve equity and fairness for all learners, and maintain faith in the Education system in the absence of being able to assess coursework or hold a full exam diet.  A full Equality Impact Assessment (EQIA) was undertaken to consider the potential impact on young people with protected characteristics.

This document is an assessment of the impact of using teacher and lecturer estimates for National 5, Highers and Advanced Highers for the 2019/20 school year on groups with protected characteristics, and collates the considerations of:

• An Equalities Impact Assessment (EQIA)
• A Fairer Scotland Duty Assessment (FSDA)

A Children’s Rights and Wellbeing Impact Assessment (CRWIA^[1]) has been considered separately.

In summary, our findings show that whilst there are differences between how awards were made in 2020 compared to previous years, which means direct comparisons should not be made, this analysis shows that for all levels of qualifications pass rates for all groups are higher in 2020 than in recent years. The data also shows that for all levels of qualifications the changes in pass rates are larger for those learners in more deprived areas than in less deprived areas. The same is true for those learners with Additional Support Needs and, for both National 5s and Highers, the increase in pass rates was higher for those learners assessed as, or declared as, disabled. Considering pass rates by ethnic group, the White – Scottish group saw the second highest increase at National 5 and the highest increase at Higher and Advanced Higher. At National 5 and Higher the smallest increases were observed amongst the Asian – Chinese group but at Advanced Higher the smallest increase was observed in the Asian – Pakistani group.

This document also sets out the additional actions the Scottish Government has taken after listening to the views of learners and other education stakeholders. This included commissioning of a rapid review of awarding in 2020, led by Professor Mark Priestley of Stirling University, and the widening of the OECD review of Curriculum for Excellence to consider the approach to assessment and qualifications in Scotland. 

Overall the EQIA process did not identify any indirect or direct, discrimination through the policy intention or design. However, the Scottish Government has listened to the findings of the independent Priestley review^[2] and is already taking measures to ensure fairness of making awards in 2021.  The Scottish Government’s response to the review and its recommendations can be found here^[3].

Background

As part of the Scottish Government’s response to the COVID-19 pandemic, the Deputy First Minister announced on 19 March 2020 that all local authority schools in Scotland would close from the end of the day on Friday 20 March 2020.  This announcement also included the cancellation of the examination diet in Spring 2020, due to the COVID-19 pandemic and the risk to public health should the diet go ahead.

There was no established process for delivering National 5, Highers and Advanced Higher qualification results, outside the normal assessment processes.  The Scottish Qualifications Authority (SQA) was asked by the Scottish Government to develop an alternative certification model to ensure that young people could receive awards this year. Until late March 2020, it was envisaged that it may have been possible for SQA to receive and mark coursework assessment components. However, public health advice at that point made this no longer feasible or safe. SQA developed a certification model^[4], which gathered teachers’ and lecturers’ estimates in the absence of any other information and involved moderation of these estimates across all centres to maintain standards. This approach to awarding would be supported by a free and substantial appeals process.

The estimates received by SQA in May showed an increase in attainment at grades A-C by 10.4 percentage points for National 5s, by 14 percentage points for Highers, and by 13.4 percentage points for Advanced Highers compared to results in 2019. These estimates, if awarded without moderation, would have represented a very significant increase in the pass rates across the board and a change, over a one year period, without precedent in the history of Scottish national examinations.

As a result of the SQA moderation process, around a quarter, or 134,000, of teacher and lecturer estimates of individual grades were adjusted, with just under 76,000 candidates having one or more of their grades lowered when compared to the teacher estimate. Around three quarters of these grade estimates were not adjusted during the moderation process.

The alternative certification model process resulted in an increase in the pass rate at National 5, Higher and Advanced Higher.  However, the system of moderation also meant some learners did not receive awards they felt they were capable of achieving – and that their teachers and lecturers believed they deserved.

Despite the headline improvements in the pass rate at National 5, Higher and Advanced Higher, and the fact that the pass rate amongst pupils in the most deprived areas increased by more than amongst those in the least deprived communities, the results left many young people feeling that their future had been determined by statistical modelling rather than their own capability and capacity.

In responding to those concerns the Deputy First Minister announced on 11 August that he was directing SQA to re-issue awards for those students who were downgraded, on the basis of original teacher or lecturer estimates.  In cases where moderation led to an increased grade, learners were not to lose that award.

This change in policy has led to a National 5 pass rate of 89.0%. This is 10.8 percentage points higher than 2019. The Higher pass rate was 89.3%, 14.5 percentage points higher than 2019; and the Advanced Higher pass rate was 93.1%, which is 13.7 percentage points higher than 2019.

This approach, using teacher and lecturer predicted grades, ensured that the majority of learners received the grades their teachers and lecturers estimated. This policy change aimed to maintain faith in the Education system and give young people, from all backgrounds, the confidence that their hard work will be fairly rewarded.

Support for Centres to estimate learner grades

During April and May 2020 SQA provided additional support for centre estimates, to assist teachers and lectures in delivering predicted awards based on their professional judgement. This support was delivered to support centre management and teachers/lecturers in making these decisions and school/college management in their quality assurance of the predicted grades.

The three measures of support were:

• Information for Centres — Producing Estimates Session 2019–20. Updated and more detailed guidance to support decision-making and submitting estimates;
• SQA Academy online course on estimates — Development of a bespoke online course to support centres, including equalities advice; and
• Provision of Centre Data on Historical Estimation Accuracy. Release of estimates and results information for the past three years to all schools and colleges.

Scope of this equality impact assessment

This document assesses the impact of using teacher and lecturer estimates to award 2020 results in place of results downgraded through the SQA alternative certification model.

The change to the awarding of results predominately affects young people, in a school environment, but there is a small number of learners that will be affected out with the school environment, either in a college setting or through distance / home learning.

This equality impact assessment has involved assessing the impact on candidates with protected characteristics to ensure that, as far as possible, learners with protected characteristics have not been disadvantaged by the procedure for making awards.

In developing this impact assessment, the Scottish Government is mindful of the three needs of the Public Sector Equality Duty (PSED) - eliminate unlawful discrimination, harassment and victimisation; advance equality of opportunity between people who share a protected characteristic and those who do not; and foster good relations between people who share a protected characteristic and those who do not. The Scottish Government recognises that while the measures may positively impact on one or more of the protected characteristics, the introduction of the measures may also have a disproportionate negative impact on one or more of the protected characteristics. Where any negative impacts have been identified, we have sought to mitigate/eliminate these. We are also mindful that the equality duty is not just about negating or mitigating negative impacts, as we also have a positive duty to promote equality.

This document combines the government's commitment to the following impact assessments:

Equalities Impact Assessment (EQIA)

In line with The Equality Act 2010, the nine protected characteristics being considered are:

• Age
• Disability
• Sex
• Gender reassignment
• Pregnancy & maternity
• Race
• Religion or belief
• Sexual orientation
• Marriage & civil partnership^[5]

Given the importance of assessing the impact on each of the protected characteristics, the Scottish Government has considered the effect of these measures against the needs of the general equality duty as set out in section 149 of the Equality Act 2010 to eliminate unlawful discrimination, harassment and victimisation, advance equality of opportunity between people who share a protected characteristic and those who do not, and foster good relations between people who share a protected characteristic and those who do not. The Scottish Government has also considered whether the measures could constitute direct and/or indirect discrimination.

Specifically, the EQIA considers impacts on equalities groups based on the three tests it is required to address:

• Does this policy eliminate discrimination for each of the nine protected characteristics? If not, is the discrimination justifiable? Can it be mitigated?
• Does this policy advance equality of opportunity between people who share a protected characteristic and those who do not?
• Does this policy foster good community relations between people who share a protected characteristic and those who do not?

Education centres and authorities have a duty not to discriminate against learners with protected characteristics including disabilities, sexual orientation and race. This duty includes the way education is provided, access to a benefit, facility or service, and exclusion. They must not treat disabled learners less favourably and must take reasonable steps to avoid putting these learners at a substantial disadvantage. 

Public sector organisations are also required, to collaborate with each other to take actions necessary to uphold rights and safeguard wellbeing of looked after children, young people and care leavers, as set out in the Children and Young People (Scotland) Act 2014 on Corporate Parenting. This means public sector organisations must think carefully about their organisation’s role and to listen to what looked after children and care leavers need so that no unnecessary disadvantages are experienced.

Fairer Scotland Duty Assessment (FSDA)

In line with The Equality Act (2010), and the Scottish Government's commitment to Fairer Scotland Duty Assessments since April 2018, this document will also consider how the policy has impacted on members of society experiencing socio-economic disadvantage.

Methodology

In assessing the equalities impact, a range of evidence was considered to supplement understanding gained through both SQA and Scottish Government equalities monitoring and engagement.

A desk based review of evidence was compiled by SQA for the Equality Impact Assessment^[6] published on the use of teacher estimates of grades in August 2020 and highlights evidence from a range of sources including Ofqual Research and Analysis Literature Review and Rules of the Game (Wyness 2017).  

This work is supplemented with additional analyses of teacher estimates and attainment data using the Scottish Government pupil census^[7]. This analysis covers learners across all publicly funded schools. No information on pupils at independent schools is held centrally. 

Engagement with young people was a key element of the evidence base for making this policy decision. Following the announcement of examination results, the Deputy First Minister had Zoom meetings with several young people who had written to him either privately and/or by open letter to discuss the SQA results. They included the SQA Where’s Our Say group, who discussed children’s rights issues. The Deputy First Minster has stated regularly that the views of young people about the downgrading of results strongly influenced his decision to revert all lowered grades to the original teacher estimates.  

This process was supported with stakeholder engagement with representatives from relevant organisations including education organisations and representative groups, helping to inform this EQIA.

Evidence on use of Teacher and Lecturer Estimates

Teachers and lecturers bring extensive knowledge and professional judgement to the process of estimating learners’ awards for National Qualifications, submitting estimates each year for almost all candidates in graded National Courses.  For 2020 awards SQA provided additional guidance to centres to support teachers and lecturers to make judgements based on robust evidence-based estimates for every candidate. In the process of estimating a learner’s final grade, conscious or unconscious bias, either positive or negative, may occur with respect to any of the protected characteristics. SQA undertook an analysis of teacher estimates over time and a literature review of teacher and lecturer estimates, as part of its Equality Impact Assessment on the alternative certification model – the key findings from these analyses are reflected below.

Evidence on some protected characteristics is limited due to small numbers of learners represented in research samples or to a more general lack of data availability for that grouping.

Analysis of teacher estimates over time

Analysis, conducted by SQA^[8], highlighted the difference between teacher estimated grades and awarded grades, based on 2019 data, with:

On average, only 45% of estimated grades matched the actual grades that were awarded.

SQA’s analysis also draws on studies from across the UK, for GCSEs, AS level and A level, about the accuracy of teacher estimates, evidencing similar trends to SQA’s data, with patterns showing similar levels of accuracy and over- and under-estimation and variations across subjects. SQA’s literature review picked up the differences in estimation by educational institutions as well: 

with further education colleges being least accurate and selective schools being most accurate. This may be accounted for in part by the variability of the cohorts and their corresponding attainment.

Analysis by Ofqual^[9] on the accuracy of teacher estimates presented similar findings as SQA:

subject has a small but unsystematic effect; sex and age have small effects that are inconsistent across subjects; centre type has a small effect that may be attributable to correlation between centre type and attainment. There are likely some effects on estimation accuracy of ethnicity (that is more over-estimation for some ethnic minority groups) and disadvantage (that is more overestimation for the more disadvantaged in general and less over-estimation for the higher attainers)

Mitigation by SQA for accuracy of estimates – All candidates potentially impacted - to support the validity of centre estimates SQA provided new information/instructions for centres outlining clearly the basis for estimates and suggested management quality assurance approaches; provision of historical data to inform estimators and management quality assurance, and online course with specific section on recognising and addressing conscious and unconscious bias. This was provided to centres in April and May 2020. Additionally all schools were required to have internal verification processes in relation to assessment decisions.

Potential sources and impact of bias 

The SQA equality impact assessment^[10], highlighted the potential that 

use of centre estimates, without any corroborating candidate assessment information, may present some risks to accurate and fair awarding for some candidates, including those who share protected characteristics or those who have contexts that present barriers to accessing qualifications (for example care-experienced young people). 

The literature review SQA conducted on teacher and lecturer estimates describes a range of potential unconscious biases may contribute to inaccurate estimates, with particular concern for potential under-estimation:

Research such as Rules of the Game (Wyness 2017) found that high-achieving disadvantaged students often have their grades under-estimated, with data indicating that black and minority ethnic students — including Gypsy Roma and Irish Traveller students — are more likely to be in these deprivation categories. Where race is combined with deprivation, research suggests that under-estimation may be more likely, although at this time the probability of this is unknown. 

The possibility of negative bias against those who share protected characteristics might result in under-awarding of grades. There may be a further negative impact on equality of opportunity where this prevents or delays progress to further planned education, employment or training. Whether bias results in either under- or over-estimating and in turn under- or over-awarding, there is a risk to good relations between those who share protected characteristics and those who do not where inequitable treatment is perceived or evidenced.

Ofqual’s review of research on bias in estimates found that: 

… subject has a small but unsystematic effect; gender and age have small effects which are inconsistent across subjects; centre type has a small effect which can be speculated to be attributable to the correlation between centre type and attainment and attainment-dependent prediction accuracy. There are likely some effects on prediction accuracy of ethnicity (that is, more over-prediction for some ethnic minority groups) and disadvantage (that is, more overprediction for the more disadvantaged in general, and less over-prediction for the more disadvantaged among high attainers) but those effects have not been properly estimated.’ (p16)

Mitigation by SQA for potential bias – All candidates potentially impacted – to support the validity of centre estimates SQA provided new information/instructions for centres outlining clearly the basis for estimates and suggested management quality assurance approaches; provision of historical data to inform estimators and management quality assurance, and online course with specific section on recognising and addressing conscious and unconscious bias. This was provided to centres in April and May 2020.  Additionally all schools were required to have internal verification processes in relation to assessment decisions.

Estimating for external learners

Learners who were home-schooled or faced some other disrupted attendance, such as Gypsy and Traveller children, potentially were at risk of centres being unable to provide an estimate of grades, leading to no certification.

Mitigation by SQA for external learners - SQA provided guidance to centres encouraging them to provide estimates for all candidates where they had the evidence that allowed them to do so.

Analysis of 2020 results 

The national results for 2020, based on teacher predicted grades, presented a National 5 pass rate of 89.0%, 10.8 percentage points higher than 2019. A Higher pass rate of 89.3%, 14.5 percentage points higher than 2019; and an Advanced Higher pass rate of 93.1%, which is 13.7 percentage points higher than 2019.

An analysis of these results broken down by pupil characteristics was undertaken.

The Scottish Government holds pupil characteristic information for pupils at publicly funded schools. SQA provided the Scottish Government with 2020 grades for National 5, Higher and Advanced Higher for these pupils for the purposes of this EQIA. The characteristic data was then matched to it by the Scottish Government to allow the necessary analysis to be produced.

Note that candidate entries from colleges and independent schools are excluded from the analysis as the relevant pupil characteristic data is not held by the Scottish Government.

The characteristics considered for this EQIA were:

• sex, 
• whether the pupil had an additional support need, 
• whether the pupil was declared or assessed disabled, 
• whether the pupil’s main language was English
• the Scottish Index of Multiple Deprivation (SIMD) group of the pupil’s home address,
• the urban/rural category of the pupil’s home address, and
• the pupil’s ethnic group.

For comparison purposes the analysis described above was conducted for 2016-2019 results. Note that these were based on a different certification process and caution should be applied when making comparisons of results between the two.

SQA analysis for the EQIA on alternative certification model, published on 4^th August, included analysis based on age groups.  An updated analysis by age has not been possible. The small numbers of publicly funded school candidates in certain age groups would have resulted in most of the figures being suppressed due to disclosure control. Similarly, it was not possible to separately identify gypsy/travellers in the ethnic group analysis due to the small numbers involved. Analysis of care experienced children could not be performed as, at the time of writing, we do not have information on care experienced children in the 2019-20 academic year.

Summary analysis showing the pass rate in each year and for each of the characteristics set out above can be found in Annex A. The analysis includes a comparison of 2020 pass rates with the average pass rate amongst candidates between 2016 and 2019. When comparing the results for each of the equality characteristics, please be aware there can be large differences between the number of entries for each category. For example, the largest ethnicity groups represented in the data were White Scottish and White Non-Scottish, for National 5, Higher and Advanced Higher. Together these represented around 90 per cent of entries from candidates in 2020.

In summary:

• Across National 5, Higher and Advanced Higher and for all equality characteristics the 2020 pass rate was higher than the average pass rate amongst candidates between 2016 and 2019.
• The size of this increase differed between different groups, ranging from 6.0 to 13.9 percentage points at National 5, between 7.4 and 17.3 percentage points at Higher and between 5.7 and 18.2 percentage points at Advanced Higher.
• In general, where one group outperformed another historically this remained the case in 2020. For example in 2020, as in previous years, the pass rate was greater for:
• females,
• those with no Additional Support Needs,
• those for whom English was their main language,
• Asian – Chinese pupils,
• those who were not declared or assessed disabled, and
• those from less deprived areas.
• However, the increase in pass rate between 2020 and the 2016-2019 average tended to be greater amongst those groups where historic attainment was lower. For example, at National 5 the 2020 pass rate amongst pupils from the most deprived areas was 13.9 percentage points higher than the average for 2016-2019 whilst for those from the least deprived areas the difference was 6.7 percentage points.

The Scottish Government produce data on school leaver attainment covering the attainment of young people by the time they leave school. School leaver attainment from 2009/10 to 2018/19 (pre COVID) shows similar results to the summary above, where one group often outperforms others. The data shows that females are continuing to outperform males at SCQF Levels 5 to 6 or better. Pupils with a recorded additional support need (ASN) are less likely to achieve SCQF Levels 5 to 6 or better, than pupils without an ASN. The school leaver data shows pupils from the least deprived areas outperform those from most deprived areas for SCQF Levels 5 to 6 or better. The National Improvement Framework (NIF) policy seeks to close the attainment gap between most and least deprived pupils.

The latest School Leaver attainment data can be found here.

Key findings

While around three quarters of all teacher and lecturer estimates were upheld through the SQA certification model, for the cases where grades were adjusted downwards there was a focus on how this was profiled across young people, with  the estimated grades of pupils in the most deprived areas adjusted to a larger degree than those of pupils in the least disadvantaged areas, leading to questions on the fairness of the process.

Following a reconsideration of the impact this has had on learners already facing unprecedented challenges as a result of Covid 19, the SQA was directed to re-issue awards which were downgraded through moderation back to the teacher estimated grade. This was done on a basis of no detriment for individual learners; where a grade was increased through moderation the higher grade will stand.  The revised certification process means that, across National 5, Higher and Advanced Higher, for all equality characteristics the 2020 pass rate was higher than the average pass rate amongst candidates between 2016 and 2019.

Conclusion 

This document has set out an overview of the range of impacts which the decision to award the majority of qualifications in 2020 by teacher and lecturer estimated grade will have had on different groups of learners. Whilst there are differences between how awards were made in 2020 compared to previous years, which means direct comparisons should not be made, this analysis shows that for all levels of qualifications pass rates for all groups are higher in 2020 than in recent years. The data also shows that for all levels of qualifications the changes in pass rates are larger for those learners in more deprived areas than in less deprived areas. The same is true for those learners with Additional Support Needs, and for both National 5s and Highers, the increase in pass rates was higher for those learners assessed as or declared as disabled.

In addition to the analysis set out in this document, the Scottish Government has taken further action after listening to the views of learners and other education stakeholders.

An Independent Review, led by Professor Mark Priestley of Stirling University, looked at events following the cancellation of coursework submission and the examination diet and the alternative certification model put in place by SQA. Areas considered in the review included:

• the approach developed in relation to estimating learners’ grades;
• teachers’ estimates;
• the moderation methodology used by the SQA;
• the impact on young people, and their families;
• any lessons from the process of awarding qualifications this year that will help to inform any future actions. 

The outcome of the Review^[11] and the Scottish Government response to the recommendations^[12] were published on 7 October, with the Scottish Government working with national and local bodies, practitioners, parents, young people and other stakeholders on taking these forward.

The Scottish Government has agreed with the OECD to expanding the remit of the Curriculum for Excellence review currently underway in order that it can look more explicitly at assessment and qualifications issues, including assessment practices, drawing on best practice globally. This work will primarily be a desk-based exercise, plus a discussion with stakeholders.

Following SQA’s consultation on the options for modifying course assessments in 2020-21, SQA have begun publishing subject specific guidance and assessment resources.  This includes ‘Understanding Standards’ material to support practitioners in making grading decisions.  Furthermore, a National Qualifications 2021 Group has been established, chaired by the Chief Executive of SQA, to consider qualifications arrangements for the 2020-21 session and monitor progress.