Scottish National Standardised Assessments: national report for academic year 2017-2018

Appendix 4: Quality assurance

In preparation for first presentation of the assessments in the 2017 to 2018 academic year, quality assurance activities took place both in relation to the proposed assessment construct and the questions proposed for inclusion within the assessments. The assessments and resulting data on learner results, which is summarised within this report, are underpinned by thorough quality assurance processes that were completed prior to the 2017 to 2018 academic year. These processes are summarised below.

6.4.1 Scottish Government and Education Scotland reviews

Several meetings and workshops were held between Scottish Government (SG), Education Scotland (ES) and the ACER Partner Group (ACER UK in conjunction with ACER Group, SCHOLAR and TWIG) during the last months of 2016, to agree on refinement of the structures to be used in delivering the assessment instruments for P1, P4, P7 and S3. Among these meetings was a workshop held in Edinburgh, on 15 November 2016, to agree the approach, targeting, construct and number of clusters to be used in the adaptive design.

Further refinements to the construct took place in meetings subsequent to this.

An SNSA Assessment Content Design workshop was held in Glasgow, on 13 December 2016. Following this discussion, a table of weightings comprising organisers and subcategories drawn from the provisional benchmarks^[12] for Literacy and English, and for Numeracy and Mathematics, was forwarded to ACER by ES, on 16 December 2016. During the course of reviewing items with ES and SG literacy and numeracy nominees, and other SG-sponsored discussions with Scottish literacy and numeracy experts, several suggestions about adjustments to the construct received on 16 December emerged, and adjustments to the construct were progressively agreed.

On the basis of a review of Curriculum for Excellence and an audit of items in the ACER pool, ACER initially selected items for review by literacy and numeracy panels assembled by ES and SG. Two intensive face-to-face workshops were held at ES’s Glasgow offices, to undertake the review of prospective items for the 2017 to 2018 scored assessments. The first workshop was held from 9 to 13 January 2017 and the second from 27 to 30 March 2017.

Table 7 shows the main activities undertaken in the development, review and customisation of items for the first live assessments, and development and review of replenishment items for trialling in the 2017 to 2018 academic year, in the period up to the end of June 2017.

Table 7: Development and review of SNSA item pool, October 2016 to June 2017

6.4.2 Small-scale trialling – Local authority trialling phase

A ‘Local Authority Trialling Phase’, comprising a trial of assessments in reading, numeracy, spelling, and grammar and punctuation was conducted in five local authorities, from 13 February to 3 March 2017.

The purposes of this trialling were:

• to check the appropriateness of prospective assessment material for Scottish learners, particularly in terms of targeting difficulty levels
• to field trial the technical characteristics of the online system and its use in schools
• to collect local authority and teacher feedback on different aspects of the assessments.

Additionally, local authorities were asked whether they were willing to approach schools, asking whether some pupils might participate in ‘cognitive laboratories’ (sometimes known as ‘usability testing’). The purpose of the cognitive laboratories was to gather information on learners’ interaction with the assessments and assessment platform, including information at individual item level.

By the end of June, usability testing had been held in nine local authorities. The sessions were conducted by ACER Partner Group staff, on some occasions accompanied by an SG accessibility expert.