National Qualifications experience 2020: rapid review

Recommendations

Summary of recommendations

1. Suspension of the National 5 examinations diet in 2021, with qualifications awarded on the basis of centre estimation based upon validated assessments.
2. The development of a nationally recognised, fully transparent and proportionate system for moderation of centre-based assessment.
3. The development of more extensive approaches to collaborative decision making and co-construction by professional stakeholders of assessment practices related to National Qualifications.
4. A commitment to embedding equalities in all aspects of the development of qualifications systems.
5. The development of more systematic processes for working with and engaging young people, as stakeholders and rights holders in education.
6. The development of a clear communications strategy, co-constructed with stakeholders, to ensure that the extraordinary arrangements for 2021 are as fully as possible understood by all parties.
7. A review of qualification appeals systems, including consideration of the rights and roles of young people, in the context of the incorporation of the UNCRC into Scottish law.
8. The commissioning of independent research into the development and application of the 2020 ACM, involving full access to anonymised attainment data and the statistical algorithms used to moderate grades.
9. The development by SQA and partners of digital materials and systems for producing, assessing and moderating assessment evidence, to ensure that operational processes for gathering candidate evidence for appeals is less reliant on paper-based systems.

Rationale for recommendations

1. Suspension of the National 5 examinations diet in 2021, with qualifications awarded on the basis of centre estimation based upon validated assessments.

This recommendation draws on the oft-repeated statement in our panel discussions that schools lack the capacity, especially in the context of an already disrupted school year, to both prepare students for exams and develop a robust evidence base for centre estimations, should they be needed. It reflects to some extent the BOCSH position paper submitted in response to the August SQA consultation on arrangements for 2021. It is based on the following rationales:

• National 5 is not a leaving qualification for the majority of candidates^[24], and therefore less high-stakes for most.
• National 5 involves large numbers of candidates – cancelling the examinations diet would enable considerable space to be freed for the arguably more important Higher and Advanced Higher examinations (both of which involve smaller numbers of students, and can be dispersed more readily across school building).
• The cohorts currently entering school year S3 and S4 have already experienced considerable disruption to teaching time; cancelling the National 5 examinations would allow for additional teaching time in the summer of 2021 for both cohorts.

2. The development of a nationally recognised, fully transparent and proportionate system for moderation of centre-based assessment.

There is a need for a nationally recognised, fully transparent and proportionate system for moderation of estimation (and teacher assessment in general if required), with a number of stages and co-constructed by stakeholders with SQA. This will develop validated assessments for use in National 5 (see recommendation 1), address the current high variance in the capacity and expertise of local authorities, and draw upon the expertise that clearly resides in many LAs. We stress here that moderation is much more than simply adjusting grades. It should include: 1] clear identification and validation of evidence sources, along with development of protocols for their use (including protocols for using historical data showing both individual prior attainment and cohort variance); 2] proportional internal verification procedures (e.g. sampling of decisions and underpinning evidence); 3] nationally agreed external verification procedures, based on sampling of decisions and underpinning evidence; and 4] statistical moderation to identify variance from trends, accompanied by further qualitative verification (with clear messaging that this will focus on candidates not the system). We note here that such a system needs to be 'owned' by teachers, including teacher unions (due to concerns about workload and professional trust in teachers). Its development could be enhanced by fully utilising the experience and expertise of continuous assessment and moderation that resides with the FE college sector. Effective professional education to enhance assessment literacy is essential, and should be developed by SQA working with local authorities and the Regional Improvement Collaboratives as a matter of priority.

3. The development of more extensive approaches to collaborative decision making and co-construction by professional stakeholders of assessment practices related to National Qualifications.

A clear message from the review is that because COVID-19 is an unprecedented threat, normal processes are inadequate to deal with this. We heard strongly expressed sentiments that no single organisation can solve this issue. We have seen evidence that normal protocols and ways of working, including a perceived tendency for SQA to eschew external involvement in its technical processes, have actively hindered actions which might have mitigated the problems experienced this year. For the coming year, there needs to be proactive approach developed which enables a greater degree of collaborative decision making by actors across the system, including where necessary the establishment of systems for data sharing and analysis. Such an approach will ensure higher degrees of ownership of solutions developed in the coming months, and moreover ensure that knowledge – different ways of thinking, local contextual knowledge and expertise, et cetera. – can be pooled more readily.

4. A commitment to embedding equalities in all aspects of the development of qualifications systems.

Considerations of principles of equity, ensuring just decisions for all individual students, should inform all processes for the award of qualifications. This means ensuring that Equalities and Children's Rights Impact Assessments are conducted at the outset of developing arrangements for the coming year, and that the principles contained therein should be at the heart of all planning and development. We would strongly endorse the idea of the government and SQA continuing to work closely with CYPCS and EHRC to realise these principles

5. The development of more systematic processes for working with and engaging young people, as stakeholders and rights holders in education.

Young people, as stakeholder and rights holders, are at the heart of a qualifications system for schools and colleges, and need to be involved fully in decisions that affect them, in line with Scotland's obligations to the UNCRC. This entails thorough consultation with young people as both a precursor for development and as a sense-checking mechanism during development. We recommend that young people are involved in the co-construction of a more effective communications system, whereby SQA, Local Authorities and schools might better disseminate key messages about qualifications. We also recommend that the final report of this review is accompanied by a young people/family-friendly briefing that clearly communicates the key messages in the review. Young people should be actively involved in the publication of this briefing.

6. The development of a clear communications strategy, co-constructed with stakeholders, to ensure that the extraordinary arrangements for 2021 are as fully as possible understood by all parties.

The effectiveness of communication is ultimately measured by how it is experienced by its recipients. Despite extensive channels of communication developed by SQA, many respondents reported that they did not experience this communication as effective. We therefore recommend that SQA engages in dialogue with practitioners and young people to develop their channels of communication further, to highlight the key priorities for communication in the coming year (i.e. what recipients would like to know) and to identify key points in the year when messaging is needed.

7. A review of qualification appeals systems, including consideration of the rights and roles of young people, in the context of the incorporation of the UNCRC into Scottish law.

Current appeals processes should be reviewed to consider the following: 1] appeals which allow estimation to be revisited where new evidence becomes available; 2] the possibility of direct appeals by young people, in line with the principles of the UNCRC, particularly Article 12. According to CYPCS (position paper), 'Many of the negative impacts relating to the cancellation of the 2020 exam diet are the result of a failure to recognise young people as rights holders and as the key stakeholders of the Scottish education system. Similar failings must be avoided in the future through transparent and pro-active consideration of children's human rights at all stages of SQA and Scottish Government decision-making in future models for assessment and certification of young people's achievements.' We endorse this view.

8. The commissioning of independent research into the development and application of the 2020 ACM, involving full access to anonymised attainment data and the statistical algorithms used to moderate grades.

As stated in the Learned Societies position paper, 'While we appreciate that it would be very challenging to undertake such an evaluation within the timeframe of the review, there is a need to generate a detailed understanding of the methodological approach used in order to plan improvements for the future.' We endorse this view. There needs to be a thorough independent analysis of the application of the ACM, using the estimation/attainment data and the specific codes/algorithms employed by SQA in 2020. This should include the following: 1] modelling of the algorithms used and alternatives; and 2] analysis of patterns of attainment, linked to demographic characteristics. This will enable fuller understanding of the issues relating to the use of statistical approaches to moderation (strengths and limitations), avoiding problems in future cohorts, and especially its impacts on the cohort of 2020. Analysis should include the effects of the 2020 awards on student transitions and destinations. This process should be achieved through the commissioning of independent fully-funded, accompanied by the publication of the full and transparent technical details of the approach employed, alongside the anonymised dataset, which would also facilitate analysis by independent researchers. We believe that the transparency involved in such an exercise would go a long way to restoring trust in the system.

9. The development by SQA and partners of digital materials and systems for producing, assessing and moderating assessment evidence, to ensure that operational processes for gathering candidate evidence for appeals is less reliant on paper-based systems.

Young people and other stakeholder groups (e.g. College Scotland, University of Glasgow Educational Assessment Network) have called for digitisation of coursework and other centre-based assessment materials, which would mitigate the problems caused by hard copies being inaccessible for estimation in 2020. This development would need to be accompanied by an evaluation of whether and how a digital divide in terms of access to hardware and software by young people might impact on disadvantaged young people.