National Qualifications experience 2020: rapid review

The Review

Context

In March 2020, in the face of the disruption caused by the COVID-19 pandemic, and following the cancellation of the 2020 examinations diet, the Scottish Qualifications Authority (SQA) was commissioned by the Deputy First Minister and Cabinet Secretary for Education and Skills, John Swinney, to provide alternative means for awarding qualifications, based on three principles.

• fairness to all learners;
• safe and secure certification of qualifications, while following the latest public health advice;
• maintaining the integrity and credibility of the qualifications system, ensuring that standards are maintained over time, in the interest of learners.

The SQA subsequently developed the Alternative Certification Model (ACM), comprising the following steps:

• Step 1 — Estimates
• Step 2 — Awarding
• Step 3 — Results and certification
• Step 4 — Appeals

The release of results on 4 August, 2020, was accompanied by controversies and considerable media attention, centred around issues of equity. Subsequently, Professor Mark Priestley of the University of Stirling was commissioned by the Scottish Government to lead an independent review of the processes through which qualifications were awarded.

Professor Priestley established the following research team to undertake the review:

• Professor Mark Priestley – Principal Investigator
• Dr Marina Shapira – Co-Investigator (with responsibility for the statistical aspects of the review)
• Dr Andrea Priestley – Co-Investigator (leader of the strand investigating the experiences and perspectives of young people)
• Michelle Ritchie – Research Assistant
• Dr Camilla Barnett – Research Assistant

Additionally, the Review employed two independent external reviewers, to provide advice on process and preliminary findings and to review the final report.

• Professor Robert Davis – Professor of Religious and Cultural Education, and Director of the Robert Owen Centre for Educational Change at the University of Glasgow
• Associate Professor Gill Wyness – Associate Professor of Economics, and Deputy Director of the Centre for Education Policy and Equalising Opportunities (CEPEO) at the UCL Institute of Education.

Work on the review commenced on 17^th August, with an interim report to the Deputy First Minister on 15^th September and completion of a final report on 30^th September.

Remit

The following remit was agreed with the Scottish Government:

The review will include considering evidence, providing commentary and recommendations around the following themes. A focus on those issues which are most pertinent to consideration of awarding methodology in 2021 if there is further significant disruption to learning and teaching and/or the cancellation of exams is key:

1. Events following the cancellation of the 2020 examination diet.
2. Advice and support given by SQA and Local Authorities to awarding centres on determining and quality assuring of estimates
3. Approaches to the gathering and quality assurance of teacher/lecturer estimates, including where possible feedback from teachers/lecturers/Directors of Education, prior to submission to SQA about the perceived rigour in the evidence base for making estimates, e.g. prelim marks, classwork, summative and formative assessment until the schools closed in March. This will include consideration of local quality assurance approaches taken by centres and Local Authorities to aid estimation; the conclusions reached by centres about estimated grades; and decisions about whether or not to share estimates with learners at that time.
4. Exploration of alternative approaches to grading and moderating national qualifications in the context of the disruption caused by Covid-19, that would maintain standards and the credibility of qualifications in Scotland and deliver public confidence.
5. Impact on young people (and their families) who did not receive what they believed their estimated grade submitted to be.
6. Feedback from teachers/lecturers on the estimation process and the moderated grades which were awarded on 4^th August
7. Consideration of the post certification review process as a means to address the issues in 5 and 6 above.
8. Confidential draft report to ministers on findings by 15^th September 2020.
9. Final report published by 30^th September 2020.

Methodology

The Review has been treated as a research project, involving the collection of primary data and review of secondary data, as well as due consideration of ethical issues. The following approaches were used to generate a wide range of data to inform the review.

1. Review of documentation, including published materials, emails and other communication between Government officials, SQA, local authorities and other stakeholders.
2. Panel discussions with key stakeholders, including young people and parents/carers, teachers, senior school leaders, local authorities, SQA and government officials.
3. Interviews with key individuals (e.g. SQA, academics with specialism in assessment/statistics).
4. Analysis of short position papers, submitted by stakeholder organisations. These are listed in Appendix A of the report.
5. Consideration of how moderation was applied to centres' estimated grades. This included examination of processes set in place by local authorities for supporting and moderating grades at the estimation process, and consideration of the national moderation processes applied by SQA. The time scale and resources available for the Review have not permitted an in-depth analysis of the statistical approach used for moderation, and we have not had access to the algorithms or anonymised datasets necessary to undertake such a review^[1].

The primary source for recruitment of panel members was national stakeholder groups, with a focus on people with direct experience of the awarding process. These groups nominated people for the panel in question. With the exception of one group of teachers (see below), and a small group of parents/carers, where contact was facilitated by the parents' advocacy group Connect, we have not been directly involved in selecting participants for panel discussions. We note here that the views expressed by panel members may not always agree with one another, or with the SQA view of events. We report stakeholder views presented in our evidence as perceptions of the process. It is important to do so, as these perceptions provide a clear indication of how the process was experienced by different people, thus providing insights into how the system might be operated differently in the coming year, when COVID-19 is likely to remain a factor. The following illustrates the range of stakeholders engaging with the review through panel discussions.

All panels and interviews were conducted via Microsoft Teams and recorded with the permission of participants, who underwent a formal process of informed consent. The research was conducted in accordance with the 2018 BERA Ethics guidelines^[2], with due regard for the human dignity and safety of all participants, following approval by the General University Ethics Panel at the University of Stirling.

Participants were guaranteed confidentiality as far as is possible in group interviews. We have not attributed any statements made in the interviews to individuals and/or particular schools and local authorities. Participants in group interviews were asked to refrain from identifying co-participants or divulging details of others' testimonies. Scottish Government and national agency staff were not present at panel discussions and interviews, and they will not have access to primary data (e.g. interview recordings and transcripts) or details about participants other than that which is public knowledge (i.e. named individuals publicly representing organisations). Different stakeholders were interviewed in discrete groups, avoiding, for example, a situation where teachers are nervous to testify frankly in the presence of local authority officers, or young people in the presence of teachers. All interviews and panel discussions were led by university researchers, who are independent of the qualifications system and processes.

We were cognisant of the need for additional sensitivity in the case of some groups of young people regarding confidentiality and anonymity, and access to the technology required to participate in the discussions. We were also aware of the potential for this research to cause emotional distress for some participants, who have been disadvantaged in the granting of awards and subsequent destinations (e.g. missed university places). The researcher leading this strand, Dr Andrea Priestley, is highly experienced at working with young people, including those in care and other vulnerable situations, and was able to address these issues. As all young people were representing third party organisations, they could usually also receive support from those organisations. A representative from the young people stakeholder organisation was permitted to attend the applicable session with the permission of all participants, in order to provide support for the young people. Where applicable, we referred young people to CYPCS for additional support.