National Qualifications experience 2020: rapid review

Executive summary

Summary of findings

• This report draws upon a range of evidence, including stakeholder testimony (generated in panel and individual interviews) and analysis of relevant documentation (including government and SQA emails).
• SQA, the government, local authorities and schools faced an extremely difficult set of circumstances, within which there were no easy solutions. In this context, a workable system for qualifications, the Alternative Certification Model (ACM), was developed. This was based on three core principles and four stages.
• All parties involved in the process were found to have acted with integrity, with the best interests of students in mind.
• Respondents (teachers, lecturers, head teachers and local authority officials) generally found that SQA guidance was clear and useful.
• The generation of estimated grades, while clearly undertaken with integrity in the majority of centres, has been subject to variation (in the types of evidence available, the processes followed for internal moderation and the support given by local authorities), which has impacted on reliability and consistency of assessment at this stage.
• The statistical approach to moderation could have been more transparent earlier in the process, and moreover it has led to anomalies in grade adjustment, especially at the level of subject cohorts within centres and individuals.
• There is widespread criticism by respondents of SQA for a perceived lack of transparency and a failure to engage in participative development of solutions with stakeholders.
• While the application of the appeals process offered an in-principle technical solution to address these anomalies, it paid insufficient attention to the severe impact on those students obliged to undergo it (in terms of mental health and wellbeing, missed opportunities to transition into Higher Education, etc.).
• Principles relating to what data is appropriate to be held by certain organisations at certain points in time.(i.e. SQA, the Scottish Government), which make perfect sense in normal times (e.g. arrangements around data sharing), appear to have impeded the development of actions that might have led to an earlier anticipation and mitigation of subsequent problems.
• The equalities implications of an over-reliance on a statistical approach, premised on comparison with historical cohort data, had been raised repeatedly from April onwards, but seem to have been under-emphasised by both the government and SQA until late in the process.
• Many stakeholders believe that, subsequently, opportunities were missed (or dismissed) to engage in qualitative moderation of the statistical process (e.g. sense-checking of anomalous cohort patterns by local authorities).
• There has been an erosion of trust/confidence in SQA amongst teachers and young people, and damaged relations in some cases between young people and their teachers.
• Communications (with professionals and with young people and their families) has been a constant source of criticism.
• Our overall assessment is that, despite the extremely difficult environment for decision making, there are points in the process where different decisions may have led to better outcomes and at least partially avoided the controversy that ensued in August 2020. Of course, we are making this observation with the benefit of hindsight, thus our primary intention is to illustrate how the system can benefit from lessons learned in 2020 to avoid a similar predicament in 2021.

Summary of recommendations

1. Suspension of the National 5 examinations diet in 2021, with qualifications awarded on the basis of centre estimation based upon validated assessments.
2. The development of a nationally recognised, fully transparent and proportionate system for moderation of centre-based assessment.
3. The development of more extensive approaches to collaborative decision making and co-construction by professional stakeholders of assessment practices related to National Qualifications.
4. A commitment to embedding equalities in all aspects of the development of qualifications systems.
5. The development of more systematic processes for working with and engaging young people, as stakeholders and rights holders in education.
6. The development of a clear communications strategy, co-constructed with stakeholders, to ensure that the extraordinary arrangements for 2021 are as fully as possible understood by all parties.
7. A review of qualification appeals systems, including consideration of the rights and roles of young people, in the context of the incorporation of the UNCRC into Scottish law.
8. The commissioning of independent research into the development and application of the 2020 ACM, involving full access to anonymised attainment data and the statistical algorithms used to moderate grades.
9. The development by SQA and partners of digital materials and systems for producing, assessing and moderating assessment evidence, to ensure that operational processes for gathering candidate evidence for appeals is less reliant on paper-based systems.