Scottish National Standardised Assessments user review: year 1 – session 2017-2018

Scottish National Standardised Assessments User Review: Year 1 - Session 2017/18

The Scottish Government ( SG) commissioned the Scottish National Standardised Assessments ( SNSA) as part of the National Improvement Framework.

This report summarises a range of evidence that the SG has analysed based on the year one experience of using the SNSA. These are part of the approach to assessing children's progress in Scottish schools.

The report draws on a range of sources of evidence. These include statistics from the SNSA system, feedback from stakeholders including schools, teacher professional associations, inspection evidence and local authorities.

Summary of Key Findings

• More than 578,000 assessments were completed by children and young people in P1, P4, P7 and S3 which equates to 94% of the expected total.
• An equating and two norming studies were successfully undertaken with high levels of engagement.
• Training has reached more than 7,700 participants and help materials are available at all times on-line.
• The support desk has responded effectively to more than 6,000 enquiries.
• There is growing evidence that the information generated by the assessments and reports is being used to plan effective next steps in learning. This, in turn, will give teachers more confidence in assessing children's progress, with a more consistent understanding of the standards expected of CfE levels in literacy and numeracy.

There has also been a number of concerns raised about certain aspects of the assessments, in particular about P1 assessments.

A number of changes and enhancements have been made to the SNSA for session 2018/19 in response to the findings from the first year of operation (detail provided within Appendix 1).

Background

1. The use of standardised assessment is one part of the Broad General Education ( BGE) assessment model, which is based around teacher professional judgement of children's progress and the achievement of Curriculum for Excellence ( CfE) levels. There is a strong educational rationale for using a range of assessment evidence, rather than a single source, to judge children progress, particularly in early years and primary. In order to support teachers and strengthen the consistency of teacher judgement, the Scottish National Standardised Assessments ( SNSA) were developed after extensive engagement with teachers, parents, children and academics. A design group including teacher professional associations influenced the specification of assessments. The SNSA assess aspects of literacy and numeracy and highlight next steps in learning for individual children. Assessment which highlights next steps in learning is known as 'formative' assessment, as opposed to 'summative' assessment which provides a score or pass/fail. The use of standardised assessment in some other countries is summative and high stakes, whilst the SNSA approach is formative and is based around next steps in learning. The SNSA are not 'high stakes' but rather a part of normal classroom activity that help to identify next steps in learning for individual children, whilst supporting teacher professional judgements.

2. The SNSA were introduced to provide teachers with additional, consistent evidence to use when making their professional judgement on whether a child or young person has met a CfE level. The OECD review of Scottish schooling in 2015 encouraged Scottish education to clarify standards across CfE levels and bring greater consistency and confidence to approaches to assessment. The SNSA provide assessments at P1, P4, P7 and S3 stages in line with CfE levels. The SNSA provide Scottish teachers with diagnostic information on aspects of reading, writing and numeracy to support the teacher's assessment of children's progress and to plan next steps in learning. This information helps teachers to support individual children as early as possible and avoid attainment gaps widening as children move through school. The SNSA also provide information at a class, school and local authority level, which can be used for improvement purposes.

3. Teacher judgement of children's achievement of CfE levels as they move through school is currently badged as 'experimental' by the Scottish Government's Chief Statistician. The SNSA evidence is an important part of continuing to improve the consistency of teacher judgement. Accurate, consistent judgement of children's progress by teachers across Scotland is crucial if all children are to receive the right support at the right time to make the best progress they can in their learning.

4. The use of standardised assessment in Scotland is not new, and the SNSA build on current practice – 29 of the 32 local authorities were already using standardised assessments prior to the introduction of the SNSA. Not all local authorities were using the same supplier; they were not all using assessments in the same way; nor were they aligned to CfE. The introduction of the SNSA will help ensure consistency across Scotland.

5. For the first time all local authorities are using the same on-line assessments, aligned with CfE, with instant diagnostic reporting available on each child along with the ability to produce reports at class and school level.

6. In addition, the assessments are designed to be accessible to the majority of children and young people. The SNSA is therefore a substantial upgrade on what was previously available to and used by schools.

7. The SNSA were launched in August 2017. The assessments are delivered on behalf of the Scottish Government by the Australian Council for Education Research UK ( ACER) and their partners SCHOLAR (Heriot Watt University) and Twig World (Glasgow). The SNSA comprise an assessment and reporting system delivered through an on-line platform, an SNSA public website, a training programme for teachers and school staff (SCHOLAR), and a service desk (Twig World) providing advice by phone and email.

8. The SNSA were designed and developed specifically for CfE and all questions included in the 2017/18 assessments were reviewed and approved by panels of Education Scotland and Scottish Government staff. Each question was mapped to an organiser and subcategory in the CfE Literacy or Numeracy Benchmarks, at a relevant stage for the year group.

Assessment completions

9. By the end of the 2017/18 school year more than 578,000 assessments had been completed. Based on school roll data, there was a theoretical maximum of circa 613,000 assessments that could have been completed in the academic year. Therefore there has been a completion rate in excess of 94%.

Assessment completion by stage

Assessment sittings by month

10. This completion rate represents a tremendous undertaking by children and teachers, schools and local authorities across all sectors. It also demonstrates a significant success in the delivery programme. There is no formal assessment 'window'. Teachers and schools are empowered to use the SNSA at a time in the year which suits children in their care and when the assessments will be most useful in supporting and informing children's progress through the relevant CfE level.

11. It is worth noting that 100% completion of SNSA is unlikely for the following reasons:

• There will be a number of children and young people with a range of additional support needs for whom a standardised assessment is inappropriate (individual teachers use their professional judgement to decide the appropriateness of presentation)
• Children who move schools during the course of the school year may not always have undertaken the assessments

Review of user experience

12. The Scottish Government and ACER have used a number of different methods to gather information about the experience of learners and teachers in the first year of operation. Evidence obtained has been useful in improving and enhancing the SNSA for future years.

ACER system feedback

13. User feedback has been formally gathered by ACER and its partners in a variety of ways throughout year 1 of the assessments:

• Assessment completion statistics
• Norming study analysis
• Equating study analysis
• Training participation and evaluation
• Service desk calls and breakdown
• Psychometric data and analysis

Assessment completion statistics

14. Uptake of the SNSA increased gradually throughout the academic year with an acceleration throughout May and June. Both ACER and SG monitored the progress of assessment completions over time. This showed a picture of steadily increasing uptake with a peak coming at the end of May with over 16,000 assessments being undertaken in one day.

15. The pattern of take up at each stage has been different and it will be interesting to see how this changes as the assessments develop and mature and teachers consider the best time of year to undertake the assessments and receive the diagnostic information.

Norming and equating studies

16. An equating study and two norming studies were commissioned during the year to support the development of the SNSA long scales.

17. The new SNSA long scales are ready for the new academic session and will show individual progress over time and against the time-referenced national norms. Engagement by schools and local authorities with these additional studies was very encouraging and significantly exceeded the necessary requirements for ACER and the participation rate standards of major international surveys, such as PISA and TIMSS. The quality of norms established through this exercise is therefore an excellent estimate of capacity.

18. The psychometric data gathered shows how well children have responded to the questions. This is based on psychometric matching between question difficulty and learners' capacities in Scotland. The data suggest that the spread of questions in P1, P4, P7 and S3 is slightly light on the more difficult questions. ACER therefore proposes that some more challenging questions are now included so that the SNSA reflects the full range of learner capacity. In S3, it is also proposed that 4th level questions are introduced.

Training

19. The training programme was bespoke to each local authority ( LA) and has been extensive, reaching a significant number of practitioners. The programme evolved during last session and was very highly evaluated.

20. Training evaluation forms were completed at the end of each session for both Phase A (set up and administration) and Phase B (interpreting the reports and data).

Phase A summary

	Total
Total training sessions	83
Total participants trained	4,148
Evaluation response rate	68%

Evaluation responses 2017/18

	Strongly agree/agree	Strongly agree/agree/neutral
Training met my expectations	71%	94%
I will be able to conduct the SNSA	81%	96%

Phase B summary

	Total
Total training sessions	206
Total participants trained	3,637
Evaluation response rate	84%

Evaluation responses 2017/18

	Strongly agree/agree	Strongly agree/agree/neutral
Training met my expectations	91%	99%
I will be able to analyse school/class data	95%	99%

Service Desk

21. The service desk has handled a large number and range of calls. Feedback about the service desk has been consistently positive, with users commenting on how quickly issues have been resolved and the pleasant and helpful manner of the staff.

22. To the end of July 2018 the service desk has received 6,014 enquiries from teaching and local authority staff. 96% of calls were resolved within 1 hour and 98% of calls were resolved within 2 hours. Less than 2% of calls took longer than 2 hours to resolve.

23. Common themes of the calls received included account provisioning and password resets, queries about the two norming studies, assessment guidance and re-assignment, reporting guidance and queries about SEEMiS data.

User feedback

24. In addition to this system feedback, SG sought feedback from teachers and local authorities who administered the assessments and received reports. The SG consulted with members of the SNSA User Assurance Group and the professional associations. The SG also undertook a user review based on nine focus groups and six classroom observations. This review had a particular focus on P1 assessments, given emerging concerns about these assessments.

25. The user focus groups were selected to represent a variety of local authorities in terms of demography and history of using standardised assessments. They consisted of a mixture of Head Teachers, Depute Head Teachers, Principal Teachers and class teachers. On two occasions, children also provided a focus group for discussion. The discussions were conducted around a standard set of questions to provide a consistent framework for discussion. They were very helpful in gaining qualitative understanding of the user experience of the SNSA.

26. Written submissions were received from AHDS and Voice. Glasgow City Council and City of Edinburgh Council provided feedback that they had received from their staff who participated in the norming studies. Aberdeenshire Council provided feedback from their SNSA working group. Around 30 emails or letters were received from individual teachers or practitioners, half of which were specifically about P1 assessments. Discussions were held with local authority Directors of Education to get feedback on the operation and experience of the SNSA at authority level, what had worked well and any emerging issues. SG also met with the National Parent Forum of Scotland, Connect, and Upstart Scotland to gain a better understanding of their views.

27. The EIS submitted the findings of its own on-line members' survey based on 5 key questions about the SNSA:

• Alignment to CfE
• User friendliness
• Timing of the assessments
• Practical delivery
• Usefulness of SNSA data

Summary of feedback

28. Overall there was a mixed range of feedback. While concerns have been raised about both the operation and value of the assessments in P1 (this has been highlighted in the media over the summer) there has also been positive feedback, particularly on the P4, P7 and S3 assessments. As the largest volume of feedback was in relation to the P1 assessments these comments are reviewed in greater detail later in the report.

Written feedback

29. The majority of written feedback came through the EIS survey of their members, through which we received comments from around 460 teachers. This feedback revealed a variety of views about the assessments, the operation of the assessment system and the teacher reports generated by the system. Key points made in this feedback include:

• Unhappiness at the level of difficulty of the SNSA particularly in P1 and the ICT skills required by young children to use the on-line interface. This led in some cases to children needing additional adult support to take part in the assessments. Some teachers also expressed concern that the information provided did not relate well to the early level benchmarks and that the reports did not provide them with any new or helpful information.
• Further there was a concern that the level of resource required to support the assessments and the time taken to deliver the assessments particularly in P1 was excessive and disproportionate to their value. In some cases the teachers were not seeing the data immediately. Others felt that is was being used in a summative rather than diagnostic manner. Some felt the SNSA did not sit well with a play based approach to early learning.
• Where assessments were carried out on iPads children found them engaging and were often unaware that they were being assessed. Teachers used their effective management to administer assessments and only needed to rely on existing classroom support.
• Teachers commented that children were able to complete the assessments in an appropriate time and children with additional support needs were given extra time and breaks as needed. They appreciated that there wasn't the use of timers to ensure children were given thinking time and support if needed. They also liked the audio support that read the assessments to P1 children.
• For some, the P1 assessments were reflective of the curriculum benchmarks and link well to the CfE experiences and outcomes.
• Some teachers reported that children found the assessments upsetting. In contrast, other teachers commented that the children found them an engaging and positive experience.

Broader feedback

30. The majority of children in P4, P7 and S3 found the assessments straightforward to do once they had become familiar with the assessment platform. Some of the questions have been challenging but on the whole children found the assessments accessible and stimulating. Some commented that they were better than previous assessments they had experienced.

31. Some teachers reported that there had been classroom management issues concerning timetabling access to IT equipment and the supervision of groups of children. However almost all said they had been very pleased with the reports generated and with the diagnostic information provided. The information provided by the assessment platform enabled them to drill down into the curriculum organisers to identify areas for further development.

32. School managers found the group reports very helpful in providing an overview of children's progress and particularly the ability to analyse the reports using the pupil characteristics embedded in SEEMiS. The use of tags and filters and the immediacy of the reports were especially helpful.

33. Many schools are already using the diagnostic data provided by the SNSA to plan future learning and teaching at an individual, group and cluster level.

Accessibility feedback

34. SNSA were designed to be as inclusive as possible. An Accessibility Advisory Group was established when the assessments were being developed and continues to provide valuable assistance in ensuring the SNSA are accessible to most children. Feedback from the ASN sector illustrated the ongoing challenges in making an online assessment available to almost all children. Management of the support required is still inconsistent and the time taken to undertake the assessments could be demanding for a few children.

35. The Accessibility Advisory Group was especially pleased to find that many of their recommendations have been incorporated into SNSA provision, including the absence of automatic limits on the timing of the assessments, the provision of off-stage assessments, and browser controls to adjust font size, type and background contrasts. The help pages of the SNSA system have been expanded during the year to provide additional guidance to support the assessment of specific aspects of the curriculum.

36. Feedback is clear that the SNSA are an improvement on the existing standardised assessments for children with additional support needs and the accessibility features are really valued by the teachers and children.

Local authority ( LA) feedback

37. SG worked closely with LA representatives throughout the development and implementation stages of the SNSA. LAs were very helpful in supporting the norming and equating studies. Training was provided on a bespoke basis according to LA requirements and on the whole was well supported by LA reps. SNSA progress reports were provided at Education Scotland Assessment Coordinator events and at the ADES Performance and Improvement Network.

38. LA feedback was largely positive and constructive as illustrated by a comment from one local authority 'The successful implementation is a result of the highly effective communication strategy and support from the National Project Team. As a data rich authority we welcome the extensive source of information provided for our schools and local authority. This creative and innovative digital approach has increased the efficiency of our service. Individual schools, clusters and indeed authority moderation groups are sharing and using the data to collaborate on strategies for improvement. The consistent format of reports is invaluable in gaining consistency and confidence in our processes.

39. The training delivered to our workforce has been of a very high quality and practitioners have welcomed this. In determining achievement of a level, staff have found the detailed reports to be helpful for developing strategies to improve children's progress and closing the poverty-related attainment gap'.

40. The LA level reports were provided in the SNSA system, with additional information being issued separately providing LAs with a wide range of information on the SNSA in their council areas. Further engagement on how LAs are using this information is planned for early in the new academic year.

41. Telephone conversations with all 32 Directors of Education or equivalent post holders throughout May and June specifically included questions about the SNSA. The general response was positive about the operation of the assessments which had gone smoothly in most areas. There was positive feedback on the training for staff operating the assessments, but a sense that future training should provide additional support around the use of and interpretation of SNSA results/reports. In keeping with other feedback, some Directors raised concerns about P1 assessments particularly the time it had taken to administer them and whether the questions were pitched at the correct level.

42. Most LAs have indicated that they have already replaced their existing standardised assessments with the SNSA and others will do so for the 2018/19 session.

National feedback

43. Education Scotland provided some initial feedback from the sample of school inspections conducted throughout 2017/2018. They noted that 'positive steps are being taken to make effective use of assessment as part of learning and teaching in some schools SNSA were mentioned positively in that staff were using them to support identification of learning needs'. The inspection evidence highlights that schools need to make use of a range of assessment tools to support the improvement of children and young people's learning.

44. The Chief Inspector commented 'The SNSA provide teachers with an invaluable resource to identify key strengths in a child's progress and crucially identify areas where a child may need further support. This is especially valuable in the early years. P1 teachers use the information the assessments provide to tailor support and to personalise approaches to ensure children get the very best start in their education. It can also inform how the curriculum is shaped across a cohort. For example, where a gap has been identified for a number of children, appropriate learning opportunities and support can be developed in a more targeted way'.

45. Parent organisations worked with SG at the launch of the assessments to co-produce information leaflets for parents, carers and teachers. However, the parent organisations expressed concern that in some cases they had not received enough information from their schools about assessment, including the SNSA. They are keen to get clear information about children's progress and next steps in learning when discussing their children with teachers. They also sought clarity on whether there is an opt-out process for SNSA.

P1 assessment feedback - Key themes

46. The P1 SNSA in literacy and numeracy received the largest volume of responses. These have been carefully scrutinised and the issues organised into 4 categories:

• Technical challenges
• Classroom management
• Question style and difficulty
• Appropriateness of the assessments

Technical challenges

47. A number of teachers commented positively on the ease of access to the platform, stating that it was an improvement on their experience with other standardised assessments. Other user feedback suggested that the IT infrastructure, the type of devices available and the children's IT skills had an negative impact on the experience of the assessments. This led to some frustrations and management challenges in some cases.

48. Teachers reported that broadband coverage is variable across Scotland and within LA areas. Where it was slow or inconsistent, delays in administering the assessment were experienced. The assessments are designed so they can be stopped and restarted at any time with no detriment to the user, reducing the concerns if broadband coverage dropped unexpectedly.

49. Access to appropriate IT equipment was also variable and created a few management challenges. Finding an appropriate space to administer assessments to a whole class was a challenge in some schools. A few teachers expressed a preference for a paper based assessment as this was easier to administer. Many other teachers however really valued the ability of the assessments to adapt to individual needs, the real time marking of the assessments, the range of reports and the ability to analyse these, all of which are only available on a digital platform.

50. The assessments are designed to be delivered on any IT platform. Teachers appreciated this freedom and assessments were delivered successfully on a range of devices. There was some evidence that children using tablets found the assessments easy to use. However tablets are not available in all schools.

51. Local IT support, where it was available, helped support class teachers, and many also commented on the positive responses they received from the SNSA service desk.

52. The digital skills of children had an impact on their experience of the assessments. Some schools deployed adults or senior pupils to manage the user interface for younger children if needed. Teachers commented that this was a developmental issue that might inhibit children doing the assessments early in P1. The audio narration provided for P1 assessments was widely appreciated.

Classroom management

53. The SNSA are designed to give teachers and schools maximum flexibility on how the assessments are administered. This has led to a wide variation in classroom management of the assessments.

54. Many schools provided children with one to one support and, while this was generally a positive experience for the child, it was reported that this had a significant impact on staff time and workload. A range of staff from managers to support assistants was used depending on availability in the school. Some schools advised that several weeks were required to deliver assessments to a class, impacting on the deployment of support staff. Other schools used small group sessions successfully and some schools used senior pupils to act as buddies for their P1 learners. In good practice observed, children were presented with the SNSA as part of normal, day-to-day learning and teaching. Children were motivated and engaged with the activity like any other high-quality learning task presented to them. Where good practice was observed, the SNSA were often set up at a table or activity station for small groups of children to engage with as they moved round the classroom, completing different tasks which were often presented through play and active learning. The ways in which teachers present and introduce the SNSA and the extent to which this is incorporated in to day-to-day learning activities is therefore crucial.

55. For some, the length of the assessments also contributed to management challenges. Some schools commented that children were taking over 40 minutes to complete an assessment. This could indicate that not all teachers were aware that the assessments do not need to be completed in one session. Other schools provided children with breaks. Analysis available from ACER showed that:

Assessment	Average duration per assessment (minutes)	90% of learners completed within… (minutes)
P1 Numeracy	22	32
P1 Literacy	27	44

56. Some teachers were concerned that they could not preview the assessments and therefore could not adequately prepare their children. For others the practice assessment was found to be helpful in this regard. Other teachers recognised that as these are not high stakes assessments, there is no need to prepare children for them, as they are a normal part of learning and teaching intended to support children's progress and highlight next steps in learning.

57. Some teachers and children found logging on to the assessments difficult. Others found setting easier passwords and advance logging in of children eased the administration of the process. This would indicate that not all teachers were aware that passwords could be changed at the user end.

Question style and difficulty

58. Some teachers expressed concern that the assessments were too difficult for some children, that the style of questions was unfamiliar and some content was too complex. In these instances they reported that children became frustrated and demotivated. Some younger children found using a mouse challenging.

59. The design of some questions, with drag and drop and scrolling, caused some difficulty. Other teachers commented that children really enjoyed doing the assessments, they were easy to access and once they had done the practice assessments the children were familiar with how to navigate the assessment. Teachers appreciated the adaptive nature of the assessments but some felt that the first range of questions was too challenging for some children. Some teachers questioned the value of the feedback from the reports commenting that they didn't learn anything they didn't know already.

60. Some staff found the immediacy of the feedback very helpful, particularly as it is framed in terms consistent with CfE. They commented that the feedback was of a high standard and would be of good use, alongside teacher judgement for identifying next steps.

61. A number of teachers commented positively on question style and level of difficulty of the assessment, stating that is was easier and of a higher quality than other standardised assessments.

Appropriateness of assessments

62. Beyond issues of management and style, some teachers questioned the appropriateness of the assessments for P1 children. Some stated that online activities are in contrast to normal classroom activities, especially in a play based approach to learning. They felt that children's digital skills were not yet sufficiently developed and literacy and numeracy skills are better assessed through classroom observation alone.

63. Other teachers stated that children found the online assessments stimulating, easy to use and enjoyable. The assessments were part of a range of teaching and learning approaches. Young children are often familiar with interactive games and activities on computers or tablets as part of their early learning. The assessments provided useful information alongside a range of other means of gathering evidence of children's progress which helped to inform accurate, consistent judgements by teachers about children's progress and next steps in learning, including their achievement of the early level of CfE.

Planned enhancements

64. As part of the planned continuous improvement process and in light of the responses received from the different sources of evidence, a number of enhancements are being implemented for the SNSA during the second academic year of implementation.

Enhancements for P1 assessments

Replenishment of questions

65. Around a third of questions have been replenished as part of the planned annual update. ACER and Education Scotland have assessed the suitability of each question and new questions have also been trialled during the previous year's assessments.

66. Questions that have caused undue concern have been replaced by alternative questions that were trialled during year 1.

67. These new questions are at an equivalent level of difficulty to those they have replaced. The context of questions is carefully considered to ensure they match CfE and are aligned to early level benchmarks. This is part of the quality assurance process led by literacy and numeracy specialists at Education Scotland.

Ordering of questions

68. In light of feedback about the perceived difficulty of some questions and question types early in the assessments, particular attention has been paid to presenting questions in a sequence that offers more familiar content in the first instance. This will not change the diagnostic value of the assessment but may better ease the child into the experience, and increase their confidence.

Improving question design

69. In some instances questions have been re-designed or replaced. To minimise the range of digital skills required by young children, there has been a reduction in the number of drag and drop type questions. This does not impact on the diagnostic value of the questions but will have a positive impact by making the assessment more user friendly. Accessibility guidance available within the SNSA help pages is being further expanded. A range of developments to enhance the experience for learners with ASN will be taking place across 2018/2019.

Updating the practice assessment

70. The practice assessment is designed to help children to become familiar with the format and layout of the assessment platform. It can be done as a class, small group or individually. The practice assessments for P1 have been updated to ensure that they provide examples of all question types. At the same time, care has been taken not to make the practice activity too long.

Providing advice and exemplification on classroom management

71. Evidence shows that classroom management is one of the key areas to ensure that SNSA is integrated into day-to-day learning and teaching. The assessment design allows for flexibility so that local decisions can be made on the timings, groupings and range of devices used. A wide variety of approaches, with differing degrees of success, has been reported and observed across the country.

72. There are good examples from the first year of implementation of how careful classroom management and creative use of existing resources have supported effective delivery of the SNSA. These include simplifying the log in process, setting small groups, encouraging older children to be mentors and deployment of support staff if needed.

73. Education Scotland is preparing case studies for practitioners to assist with classroom management. This will include hints and tips as well as the development of videos which will be available on the platform featuring 'talking heads' providing practical advice on successful classroom management. In addition, case studies will be prepared on how to interpret the data provided by the SNSA and how that can be used for improvement purposes.

74. The online help section will be progressively updated to include these examples and the training package updated. The service desk will continue to provide support and advice to teachers and local authorities.

Establishing a P1 Practitioner Improvement Forum

75. A P1 Practitioner Improvement Forum will be established to share existing practice and consider how to enhance the overall assessment model in Scotland for the early level of CfE, including the SNSA experience for learners and teachers. This will be led by Education Scotland engaging with those who have a specialist interest and experience in early learning. The forum will engage with practitioners and bring forward recommendations that could be implemented for 2019/20 school year.

Retaining the standard of P1 questions

76. Given the concerns raised by practitioners, ACER and Education Scotland re-examined the difficulty levels of existing questions and the replenishment questions. They concluded that the questions were stage appropriate and aligned to the standard within the early years' benchmarks. Additional advice will be provided on the relationship between SNSA and the benchmarks at early level, and on how to use the diagnostic feedback in a formative way. This will help to ensure that we continue to improve the consistency of teacher judgement of children's achievement of the early level of CfE across Scotland.

77. In addition, the detailed psychometric data ACER gathered from the system for P1 assessment questions indicated that they are at an appropriate level of difficulty to fulfil their purpose of providing diagnostic and formative information on where learners are in their progress in literacy and numeracy. The proposal therefore is not to change the range of difficulty of questions at the P1 stage at this point. This will however be part of the remit for the new P1 Practitioner Improvement Forum.

Enhancements for P4 – S3 assessments

Replenishment of questions

78. As planned around a third of questions have been replenished as part of the annual update. ACER and Education Scotland assess the suitability of each question for appropriateness and questions are also trialled as part of the previous year's assessments.

Increasing the difficulty level of questions

79. The psychometric analysis by ACER suggests that the sets of questions in the P4, P7 and S3 assessments need to include a few more challenging questions to provide maximum formative information about Scottish children and young people's capacity. Consequently, in session 2018/19, while there will still be a range of questions to suit the capacity of lower-attaining learners, the overall difficulty of the assessments will be increased by presenting more challenging questions at the upper end, including the trialling of 4 ^th level questions at S3, and 'next stage' questions for P4 and P7, during the 2018/19 session.

Revision of practice questions

80. Accessibility functionality has been added to ensure the practice assessment experience fully reflects that of the assessment platform for learners with ASN. While a number of teachers commented positively on the value of the practice questions, this part of the system for P4, P7 and S3 is undergoing further review and will be revised during the Autumn term.

New developments for year 2 – 2018/19

SNSA long scale for reports

81. Following the success of the norming and equating studies in year one of the SNSA, the three progress bands have been replaced in the learner reports from August 2018 with long scales of twelve bands of formative description showing progression from P1 to S3. The bands are formed using psychometric analyses from the national norming and equating studies. Each contains discrete, formative information which shows the range and type of conceptual understanding demonstrated typically across a 'band' of standardised scores.

82. In addition to all reporting conforming to the long scale format from this period onward, all outcomes reported for individuals within year one will be converted to the long scale during the academic session. This will enable teachers to track children's progress over time on a consistent scale and against the time referenced national norms. Accurate tracking of children's progress to inform high-quality, responsive planning of learning and teaching is important if we are to achieve the ambitions of CfE for all learners.

User feedback mechanisms

83. In order to provide ongoing, high quality information to support the continuous improvement of the SNSA, the assessment platform will be enhanced with a more systematic user feedback process.

84. At the end of each assessment, a short age-appropriate survey will be available for children and young people that will encourage them to feedback on their experience. This survey will be available in the Autumn term.

85. Teachers will have opportunity to offer feedback at any time from within the system, thus providing a simple and direct means to report their views and experience.

86. An annual survey of users will be introduced in 2018/19 collecting feedback from a randomly sampled group of staff to provide an overview of the SNSA experience.

87. The ongoing use of focus groups and observations will allow the Scottish Government to gain further qualitative understanding of the user experience.

Parental Advice

88. Updated parental advice will be made available through the public SNSA website, Parentzone and to all schools for sharing with their parental groups.

89. The Scottish Government has made clear that the Scottish National Standardised Assessments are part of everyday learning and teaching. The assessments provide teachers with diagnostic information to help them plan next steps in children's learning. Similar to other methods of assessment in schools, there is no legal basis for a parent to withdraw their child from the SNSA. If parents have specific queries or questions about their child undertaking the assessments then they should discuss this matter with their child's school who will take the decision on whether to grant the request as they would for any other aspect of learning and teaching.

Assessment summit

90. Given the positive features of the CfE assessment model, which is based around formative assessment, SG propose to have an assessment summit in session 2018/19 to consider the educational benefits of our approach and how this can be strengthened further. This will include how we can reassure parents and the public around the use of SNSA, which is distinctive from the ways in which standardised assessment is used in many other countries. The summit will include global assessment experts, representative of teacher professional associations, local authorities and other key stakeholders.

Summary and Conclusion

91. The first year of implementation of the SNSA has been a success on a number of levels:

• More than 94% of children and young people have completed the assessments. An equating and two norming studies were successfully undertaken with high levels of engagement.
• Training has reached more than 7,700 participants and help materials are available at all times on-line.
• The support desk has responded effectively to more than 6,000 enquiries.
• There is growing evidence that the information generated by the assessments and reports is being used to plan effective next steps in learning. This, in turn, will give teachers more confidence in assessing children's progress, with a more consistent understanding of the standards expected of CfE levels in literacy and numeracy.

92. As may be expected, a number of issues have been raised in the first year of implementation. These have been carefully considered. Improvements have been and will continue to be made to the assessments to enhance user experience. In addition, the SNSA has been significantly upgraded with the implementation of the SNSA long scales.

93. There is growing international interest in what Scotland is doing and how its model of teacher professional judgement, informed by a range of evidence including standardised assessment, is improving children's progress in learning and consistency in the understanding of standards within CfE levels.

94. The SNSA should continue to progress with confidence based on the successful foundations of the first year of implementation.

95. We must continue to emphasise the benefits of our overall assessment model for children and young people, and the ways in which it will support our shared mission of achieving excellence and equity for Scotland's children and young people. The place of SNSA within this, which has the same status as other sources of assessment evidence which inform teacher judgement, must continue to be emphasised.