National Standardised Assessments for Scotland: national report 2022 to 2023

1 Introduction

1.1 What are the NSA?

In January 2016, the Scottish Government published The National Improvement Framework for Scottish Education (hereafter 'the Framework'). The Framework set out the Scottish Government's vision and priorities for Scotland's children and young people. It was developed to support high-quality learning and teaching – the core principle of Curriculum for Excellence (CfE). Over time, it was intended that the Framework would provide a level of robust, consistent and transparent data across Scotland to extend the understanding of what works, and drive improvements across all parts of the system.

To meet the aims of supporting high-quality learning and teaching for Scottish children and young people, it was determined that gathering data on children's progress at key points in their education, including data on differences between those from the least and most deprived areas, was essential. Improved data of this kind would support the planning of further interventions to ensure that all learners achieve as well as they can. Part of this information would be provided by SNSA (English Medium) and MCNG (Gaelic Medium).

The SNSA have been available for use in publicly funded schools in Scotland since August 2017 and MCNG have been available since December 2018. They are undertaken by children and young people in Primary 1, Primary 4, Primary 7 and Secondary 3 (P1, P4, P7 and S3) across Scotland, once in each school year at any point in time. Learners at P7 and S3 in Gaelic Medium Education may also undertake the SNSAs for Reading and Writing in addition to the MCNG assessments in the same areas; in S3, GME Learners may sit their numeracy assessment through either SNSA or MCNG depending on the language in which their numeracy education is delivered. Reports to schools and teachers are available as soon as a learner completes an assessment. Additional reports are available for local authorities. This is the first academic year of operation for the phase two NSA system, with the SNSA and MCNG being brought together into a single platform for the first time.

Outcomes from National Standardised Assessments provide one source of formative assessment evidence as part of a range of evidence to support teachers' professional judgement of children's and young people's progress in learning. The Achievement of CfE Levels (ACEL) data is based on this teacher professional judgement – this data reports on the percentage of children and young people in P1, P4, P7 and S3 who achieve the expected CfE level in literacy and numeracy. ACEL data is published annually at school, local authority and national level.

The NSA:

• are inclusive, accessible and adaptive;
• provide teachers with immediate feedback to support them in guiding teaching and learning; and
• can be used to monitor and evaluate educational interventions.

The user reports provided for NSA support a number of these points by providing teachers, school leaders and local authorities with diagnostic information about learners' strengths and areas of challenge that can be used as part of a wider formative approach to assessment to plan next steps in learning. Alongside other assessment evidence, the information reported in NSA can also be used to inform teachers' professional judgement on achievement of CfE levels. A central aim of NSA is also to provide information on the outcomes of Scottish children and young people in literacy and numeracy over time.

1.2 Key features of NSA

The NSA have a range of important and innovative features:

• they are delivered online

Children and young people complete the assessments using a digital device: a desktop computer, laptop or tablet. The assessments are delivered online and, because all items (questions) are scored automatically, teachers can access their learners' reports as soon as an assessment is completed.

NSA are designed to be undertaken on a range of devices, and delivery on the most commonly available browsers is supported. This flexibility in mode of delivery is designed to support administration of NSA across a range of different classroom settings, enabling schools to choose the method of presentation that best suits them. There is a section in the Quick Start Guide which outlines the operating systems on which the NSA will work. With this flexibility of delivery, the content of the assessments, within the adaptive design model, remains consistent. The Quick Start Guide can be found in the Help section of the NSA system which can be accessed via this link.

• they are adaptive

The items presented to children and young people vary according to how well they are performing on the items they have answered so far. All learners begin an assessment with a set of items of mid-level difficulty for their stage. If a learner does well on these, the next set of items presented will be more challenging. If a learner is not succeeding on early items, the items become easier – and so on, throughout the assessment. The adaptive nature of NSA means that the experience for each learner is modified so that the assessment is appropriate for their current level. The adaptive design also means there are increased opportunities to benefit from the diagnostic value of the assessment. An assessment is most useful as a formative tool when there are no 'floor' or 'ceiling' effects. A 'floor effect' occurs when an assessment is too hard, so it tells only what a learner cannot do. If this happens, it is impossible to see a starting point on which future learning can build. A 'ceiling effect' occurs if an assessment is too easy and a learner gets every question right. When this happens, it is impossible to judge the upper reach of their attainment and thus to help this learner make the next step. The adaptive design, when working well, enhances the learner's experience of the assessment and serves to support identification of where children and young people are in their learning development.

• they have a carefully judged number of items per assessment

Each assessment has from 27 to 30 scored items^[1]. These numbers of items allow coverage of different aspects of each of the assessed subject areas, without excessive time being required by any learner. On average, in the 2022 to 2023 academic year, children and young people completed each of the SNSA within 16 to 35 minutes (less than 20 minutes for Primary 1 children). MCNG learners completed each of the assessments within 15 to 48 minutes (less than 20 minutes for Primary 1 children). However, there is no time limit for completing NSA and, where a teacher judges it necessary, a learner may take a break and come back to pick up the assessment where he or she left off.

For assessment stages and types where there are thousands of learners, each question has been tested empirically to make sure it 'works'. Before being included in SNSA, every item has been trialled and has been presented to several hundred learners of a similar age and stage to the ultimate respondents, to ensure that it has sound measurement characteristics and will yield statistically consistent outcomes. (See footnote 1.)

For assessment stages and types where the learner population that items can be trialled with is smaller, a 'rolling' approach to calibration is implemented. This is common in other settings with small sample sizes and is particularly relevant to the MCNG assessments due to the small population of GME learners. Items are trialled until they reach a minimum threshold of responses needed to be 'tentatively' calibrated, and then enter live use on this basis. Such items' calibrations are then reviewed on an ongoing basis, until sufficient responses to them have been received that they can be firmly calibrated.

In addition, every item has been quality assured by Education Scotland.

• responses are scored objectively

The majority of items in NSA are in 'selected response' format, mostly multiple choice. The advantages of this format are both educational and technical. First and foremost, it is an advantage that all responses are marked consistently, so there is no question about the reliability and standardisation of the outcomes at the item level. A second advantage, in terms of curriculum, is that, because learners can complete items relatively quickly, a wider range of curriculum content can be covered in a limited time than would be possible if children have to generate their responses. A third advantage is that the assessments can be marked instantly, allowing the allocation of assessment items of an appropriate difficulty level within the adaptive design. Additionally, reports can be accessed as soon as an assessment is completed, so teachers can use the formative feedback immediately. There is, of course, much to be learned about children's understanding and skills from other modes of assessment, such as what they can say, make, write and do. However, assessments using selected response formats serve the purposes of NSA well in its role as one element in the wider array of evidence that teachers will use to evaluate children's and young people's learning.

Other features of the NSA programme are specific to the Scottish education context.

• they cover agreed elements of Curriculum for Excellence

The assessments have been constructed to align with CfE. For the academic year 2022 to 2023, the final version of the Benchmarks (published in June 2017) is used as a reference point for the assessments, along with CfE Experiences and Outcomes. The content areas covered are described in more detail in the sections of this report dedicated to numeracy, reading and writing.

• they have a flexible delivery model

The flexible delivery model is intended to allow children and young people to be assessed at any time in the school year that is judged suitable by the school for the class and individual learner. This flexibility is central to the formative purpose of the assessments, as it allows practitioners to plan assessments for the point in the school year where it will best inform further teaching and learning.

Assessments can be paused and restarted to fit around other learning activities, and to allow for breaks where appropriate. A consequence of the flexible timing is that, when interpreting the outcomes of the assessment at individual, class, school, local authority or national level, the point in the school year that the assessment was taken needs to be taken into account.

For SNSA, there is evidence that children's and young people's outcomes – their literacy and numeracy skills, knowledge and understanding – develop progressively, on average, over the 10 (effective) months of an academic year. Amongst the stages presenting for SNSA, children in Primary 1 showed an increase in outcomes in both literacy and numeracy: this can be seen when comparing outcomes from 2022 (August to December) with those from 2023 (January onwards). The same pattern was observed for P4 and P7 across all subject areas, but with diminishing increases in performance in 2022/2023 for each successive stage. Within each stage, the rate of improvement between the first half and second half of the 2022 to 2023 academic year was similar, regardless of subject area. At S3, across all subject areas, there were some increases in performance seen for some groups of learners between the two halves of the academic year but there were also some groups of learners where no difference in outcomes was seen.

While the findings described above might be as expected, they also constitute a positive outcome, confirmed empirically with SNSA data. However, given the possibility of undertaking SNSA throughout the school year, outcomes from all learners should be interpreted with some caution when making any comparative judgements about individuals or groups. Each learner is presented only once and, because the timing of SNSA was determined locally, it cannot be assumed that the profile of children and young people who presented in the first half of the school year was the same as that of those who presented in the second half. For example, it is possible that teachers chose which learners should undertake the assessment when, based on their judgement of their learning progress. Norming studies were completed pre- and post-Christmas for the 2022-2023 academic year and these national norms are used within the reports to show learner achievement in line with when each learner completed their assessments.

For MCNG, the number of learners taking assessments was too small to complete separate norming studies for assessments completed pre- and post- Christmas.

• they are designed to be accessible to all learners

To support learners when completing the assessments, the system is designed to be compatible with a range of assistive devices, so that learners can use the devices with which they are familiar from everyday use in the classroom, including software and devices such as text readers, screen readers and switches. In the case of screen readers, the assessments have been developed to include alternative text descriptions of images, charts and graphs that are integral to answering an item. For those learners already identified as following a curricular path other than that typically suggested by their stage, out-of-stage assessments are available – that is, assessments designed for the stage appropriate to their curricular path. Learners are also entitled to access the same level of support when undertaking assessments that they would in class. Detailed guidance is available for teachers in relation to additional support needs (ASN) and English as an additional language (EAL) via the Help section of the NSA portal which can be accessed via this link. The information gathered from across the school year, on which the analysis within this national report is based, includes data from learners with ASN and EAL.

Accessibility is central to the design of assessments. Authors are trained on the guidelines for accessibility, equity and inclusion according to the subject and stage for which they are writing.

Measures in place to promote the accessibility of NSA items include:

• Carefully considering use of images to avoid introducing accessibility issues. All essential images in the assessments have good contrast and avoid excess background imagery, in line with accessibility standards. Images are classified into three categories:
• Essential images – these are used to assess Experiences and Outcomes from the CfE that specifically require reference to an image. All essential images (for example pictographs and charts in numeracy) avoid red and green to make images meaningful for learners with deutan or protan colour-blindness. Colour is no longer used for the determination of item responses (e.g., “Choose the blue ball”) and non-compliant items are being phased out.
• Answer images – these are found largely at P1, where an image eliminates the need to read an answer or makes the task more engaging.
• Decorative images – these are used to add colour and interest to an item. For example, if an item is about finding the correct punctuation mark to use in a sentence about a car, there may be a picture of a car.
• Descriptions of essential and answer images are added as alternative text for use by digital screen readers.
• The layout of items on screen is optimised for screen reader use, with limited use of tables for layout.
• A common feature is for learners to be able to ask the teacher for support – this instruction is attached to the first image in an item. Teachers can describe images or have materials to support candidates. For example, if a numeracy item showed three sticks of different lengths and asks which is the shortest, teachers would be able to give three different length objects to allow them to answer. The approach of asking the teacher for help is also used at higher stages in numeracy, where it is the most accessible approach.

The NSA feature numeracy, literacy and writing contexts that are inclusive, with reading texts for P4 to S3 covering diverse themes, so that learners from a range of backgrounds can access and interpret them and they reflect the diversity of Scottish society. Texts and contexts are reviewed and edited to be varied and accessible to learners with additional support needs, well-written and rich. Extracts are self-contained, but context is provided where necessary.

All content production has been informed by Education Scotland’s guidelines on equity and inclusion, to ensure accessibility regardless of prior learning or experience.

1.3 Reporting NSA outcomes

In the academic year 2022 to 2023, six capacity bands per stage (12 overall, taking into account overlap between stages) were used in reporting the outcomes of NSA to schools and local authorities, and they are also used in this report. The exception to this is writing for which there are only nine bands, since P1 literacy is included on the reading scale. Band descriptors for each subject and stage are included in Appendix 6: Band descriptions from the 2022 to 2023 Individual reports.

1.3.1 Reporting on learner outcomes

The reports available to schools and local authorities for NSA 2022 to 2023 provide diagnostic information about each item presented to an individual or group of children or young people. This diagnostic information shows, for each item, the curriculum organiser to which each item belongs, the skills, knowledge and understanding assessed and the item's difficulty, as well as the individual's or group's outcomes against the item. This diagnostic information provides one piece of evidence to help the education profession identify areas of strength or challenge at the individual learner level or for groups.

Another key feature of the reports for schools and local authorities is information about learners' overall outcomes. Each stage's outcomes are reported in six bands. The outcomes of learners who achieved only a small degree of success on the assessment is reported in the lower bands. Similarly, the outcomes of learners who achieved a substantial degree of success on the assessment are reported in the top bands. These bands are related to a range of learner outcomes on the assessment. They are specific for each subject area and stage. Each of the 22 NSA capacity bands (11 SNSA and 11 MCNG) has a corresponding description unique to that assessment. These descriptions are based on a summary of the skills, knowledge and understanding assessed in the items included in the assessment in the academic year 2022 to 2023, which, in turn, are aligned with the Experiences and Outcomes. The region descriptions for each assessment and stage are shown in Appendix 5: Band descriptions from the 2022 to 2023 individual reports.

On an individual report, the learner's outcomes are located against these descriptions to show the kinds of skills, knowledge and understanding they demonstrated in the particular assessment.

The bands have a specific and different meaning for each of the assessments, according to subject area and stage. Accordingly, the dot on an individual's report, locating the learner's outcomes, shows what kinds of skills, knowledge and understanding they demonstrated in the particular assessment, rather than any fixed judgement about the learner's aptitude.

The outcomes on the assessment of an individual, a class or a school are intended as one piece of evidence – a fair and objective piece of evidence – in an evaluation of learners' progress in literacy and numeracy. The holistic outcome on the assessment is intended to be used by teachers to corroborate or, sometimes, to raise questions about, other reference points in their overall assessment of a learner's progress.

Figures 1 and 2 show example extracts from an SNSA and an MCNG individual report (for a fictitious learner). To the left of the SNSA report is a scale labelled 7–12, accompanied by the band descriptor text referred to above, for each of the bands; on the left of the MCNG report is a scale labelled 4-9, also accompanied by the band descriptor text referred to above, for each of the bands. . The easiest content is summarised in the paragraph at the bottom and the most difficult summarised in the paragraph at the top.

1.3.2  Reporting on assessment item difficulty

Just as each learner's overall outcome was expressed as a capacity band, each item in the assessment was also categorised as belonging to a certain band of difficulty. An item categorised as having the lower band difficulty in NSA was one that learners of this age and stage generally tended to be more likely to answer correctly. An item categorised as having the bands' mid-range difficulty was one that fewer learners were able to answer correctly. An item categorised as having the top bands' difficulty was one that relatively few of the learners in a particular stage were able to answer correctly. The ratings of item difficulty appeared next to a brief description of each item on reports to schools, to support a diagnostic interpretation of the challenge of the assessment items presented to the individual child or young person.

More detail about the content of the assessments is provided in the subject area sections of this report.

1.4 Academic Year 2022- 2023

The 2022 to 2023 academic year was the sixth year of implementation for SNSA and the fifth year of implementation for MCNG. 544,688^[2] SNSA assessments were completed across Scotland over the course of the year. This number is equivalent to about 89.3% of the possible maximum number of assessments available for children and young people in P1, P4, P7 and S3^[3].

4272^[4] MCNG assessments were completed across Scotland over the course of the year. This number is equivalent to approximately 66.7% of the possible maximum number of assessments available for children and young people in P1, P4, P7 and S3^[5].

The Scottish Government's policy and practice of continuous improvement applies not only to educational attainment but also to the NSA programme itself. In the 2022 to 2023 academic year, both the SNSA and MCNG were housed on the same assessment platform for the first time, although they remained two distinct assessment sets. The assessments for the Gaelic Medium Education cohort and English Medium cohort are now more closely aligned, due to the move to a shared approach to the assessment design, standardisation principles, and reporting functionality.

Further enhancements will be implemented over the next academic year including training courses and additional reports. One updated training course will focus on using NSA data to support improvement and will allow course delegates to interrogate a range of data sets in order to draw conclusions and plan interventions for groups of learners. Other courses will be available to support the interpretation of future reports which are Outcomes By Academic Year (OBAY) and the Learning Progress Report (LPR).

The OBAY report provides an overview of outcomes for each year group and curricular area in any given academic year. This will enable practitioners and school leaders to compare the outcomes achieved by different cohorts, in the same curricular area, at the same curricular stage, year-on-year. This report, when considered as part of a range of evidence, could be used to support practitioners in identifying whether improvement has taken place, the impact of curriculum changes and the impact of interventions.

The Learning Progress Report is an additional and complementary report to the suite of NSA reports available. Within the context of wider assessment evidence, the Learning Progress report can:

• help to monitor the extent of improvement in outcomes for children and young people;
• help to monitor the impact of interventions and curriculum changes; and
• support decisions on next steps for individuals and groups.

1.5 Summary of findings

This summary of findings speaks in general terms about the assessments and categories of learners. The analysis here is based on SNSA outcomes as due to the number of MCNG assessments completed, a similar breakdown here would risk identifying individual learners^[6]. For the same reason, a number of graphical analyses in this report have been provided for the SNSA but not for the MCNG.

In general, a greater proportion of male learners tended to achieve outcomes in the higher capacity bands at all stages for the Numeracy assessments than female learners but this was reversed in both Reading and Writing. These findings are consistent across all stages.

In terms of the categories of Scottish Index of Multiple Deprivation (SIMD), Free School Meal Entitlement (FME), Additional Support Needs (ASN) and Looked After Children (both at Home and Away from Home) greater proportions of learners from the least deprived backgrounds, with no FME and ASN and those who are not Looked After Children, tended to achieve outcomes in the highest capacity bands. This was observed across all stages and subject areas. These differences tended to become more pronounced as learners moved through the stages of their education.

In terms of ethnic background, at P1, a greater proportion of learners from a White Scottish background achieved outcomes in the higher capacity bands but this was reversed at later stages, with greater proportions of learners from a non-White Scottish background achieving outcomes in the higher bands. This was also observed in learners who are categorised as having English as an Additional Language with a greater proportion of learners who have English as their first language tending to achieve outcomes in the higher capacity bands at P1 but a greater proportion of learners with EAL achieving outcomes in those higher bands at P4, P7 and S3.