Education - Independent Review of Qualifications and Assessment: data analysis – phase 1

Do respondents support the principles?

As with the vision statement, there was very little challenge to key ideas contained in the principles. Most comments were suggestions for improvement to the language. However, there were also more significant issues raised, e.g., suggestions to merge two principles.

Figure four (below) details Collaborative Community Groups and allied discussion groups and Schools & Colleges Groups' views about each principle.

The most significant difference between Collaborative Community Groups and allied discussion groups, and Schools & Colleges Groups was in the percentage of people who suggested the need for amendments to each principle.

II. Proposed amendments to the text of the principles: What are the main themes and issues?

Most Collaborative Communities and allied discussion groups, and Schools & Colleges Groups agreed with the principles with amendments, indicating that some amendment would enhance the message. Table 3 (below) presents the key findings (themes, subthemes and codes) based on qualitative analysis, where main themes illustrate the aspects to be reviewed concerning each principle.

Many respondents referred to the Language of the Principles and made proposals for revision: Collaborative Communities and allied discussion groups, and School & College Groups provided diverse views about each principle. Participants highlighted language issues in some principles (1, 2, 3, 4 and 5), mainly focusing on terms, clarity of language, practical implications, the need for brevity and writing style.

• "I agree with the sentiment expressed but it a bit unclear what this would mean in practice" Principle 2, Collaborative Communities Groups & allied discussion groups
• "Pupils struggling to understand wording of the statements: Need to have easier statements, and shorter sentences" Principle 5, Schools & Colleges Groups
• "Language and layout feels quite cluttered and too long" Principle 2, Collaborative Communities Groups & allied discussion groups
• "Principle is sound; articulation requires further work" Principle 4, Schools & Colleges Groups
• "Principle could be worded a lot simpler" Principle 1, Collaborative Communities Groups & Allied Discussion Groups
• "It's hard to imagine this principle would speak clearly to people who are currently disengaged!" Principle 3, Schools & Colleges Groups
• "The majority of Principles were too wordy" Principle 1, Collaborative Communities Groups & allied discussion groups

A second theme focused on a need to recognise practical implications.

'Constraints at work', 'An educational system for a sustainable world', 'Development of learning and innovation skills': In principle one, participants from Collaborative Communities and allied discussion groups, and Schools & Colleges groups highlighted the importance of this principle especially after the disruption of Covid in their settings. They noted the value of rethinking the educational system based on a sustainable world and developing learning and innovation skills for Scottish society. However, some respondents in the Schools and Colleges group identified potential barriers to the realisation of the principle, aspects that could constrain their day-to-day work.

• "The principle is particularly important after the disruption of Covid" Principle 1, Collaborative Communities Groups & allied discussion groups
• "Be mindful of the constraints on schools, e.g.: bound by staffing formulae; HR policies and procedures; budgets" Principle 1, Schools & Colleges Groups
• "We should be providing the skills like critical thinking, literacy, numeracy, finances, to all young people to apply in their own life as the world changes" Principle 1, Schools & Colleges Groups
• "I think this should be expanded to emphasise that the importance of the system preparing learners to face the social, economic and environmental crisis for a sustainable world" Principle 1, Collaborative Communities Groups & allied discussion groups
• "More focus on helping learners to learn how to learn independently, building resilience and a love of learning" Principle 1, Schools & Colleges Groups
• "It is a really important principle as the world is always changing and therefore Scotland's Qualifications need to change also" Principle 1, Schools & Colleges Groups

Some respondents also questioned whether a truly collaborative process was possible in the current context. Most Collaborative Communities and Schools & Colleges groups indicated that principle two is unrealistic for Scottish education based on the challenges of the time constraints of their daily lives. They also highlighted the issues of participatory democracy and questioned whether that process would be feasible in the near future. Some argued that this collaboration was key to the success of any innovation, suggesting that the principle should be stated more strongly.

• "This principle seems to conflict with the possibility of being responsive because proper consultation takes time" Principle 2, Collaborative Communities Groups & allied discussion groups
• "This is a crucial area and deserves more explicit and less mealy-mouthed language" Principle 2, Schools & Colleges Groups
• "This principle implies that qualification design and development could be a form of participative democracy. This will be very challenging to achieve in practice. It will not be possible to involve all learners and parents/carers" Principle 2, Collaborative Communities Groups & allied discussion groups
• "This principle should demonstrate how Scotland's qualification system will seek to understand current and future industry needs" Principle 2, Schools & Colleges Groups
• "This principle seems to conflict with the possibility of being responsive because proper consultation takes time" Principle 2, Collaborative Communities Groups & allied discussion groups

A third theme related to perceived tensions between CfE aspirations, and what is reflected in qualifications: the need for an educational system for a sustainable world.

In principle three, perhaps more than any other principle, participants of the collaborative Communities and allied discussion groups, and Schools & Colleges groups suggested offered a wide range of views. There were commonly articulated tensions between Curriculum for Excellence (CfE) aspirations and qualification. Respondents identified a range of barriers that would need to be addressed if a bridge were to be built between aspirations and practice. Most commonly, these included the need for more resources, particularly time for planning and designing activities, the need to strengthen communication amongst local authorities, and the development of clear routes for developing interdisciplinary learning. A small number of respondents challenged the concept of Curriculum for Excellence. For example,

• "Staff need time to reflect on aspirations throughout the academic year" Principle 3, Schools & Colleges Groups
• "Recognising the diverse achievements of learners and how these fit into the wider context of CfE and the Global Goals" Principle 3, Collaborative Communities Groups & Allied Discussion Groups
• "The curriculum for excellence is a good idea which has been implemented badly. Until more time and support is given to teachers to be able to deliver it properly, it should be ditched!" Principle 3, Collaborative Communities Groups & Allied Discussion Groups
• "We agree with the aspirations of CFE but feel it is very challenging to deliver due to funding, lack of staff time, location" Principle 3, Schools & Colleges Groups
• "Curriculum for Excellence is not a curriculum for excellence, so please do not allow the qualifications to reflect this, because it will only lower standards" Principle 3, Schools & Colleges Groups

There were strong statements from respondents about the need to develop a different relationship between academic and vocational qualifications. In responding to principle four, Collaborative Communities and allied discussion groups, and Schools & Colleges groups emphasised the value of both academic and vocational pathways, emphasising that these should have parity of esteem. More generally, there was a concern to move beyond a culture that was narrowly 'attainment driven'. For example,

• "A change in the culture of 'attainment driven' recognising student participation, achievements, and experiences throughout the academic year is required" Principle 4, Schools & Colleges Groups
• "The school sector has been reluctant to give parity of esteem to academic and vocational qualifications and the needs and aspirations of lots of learners in a school environment have been neglected" Principle 4, Collaborative Communities Groups & Allied Discussion Groups
• "We need to ensure that there is parity of all qualifications. By separating 'academic' and 'vocational' I'm not sure that we are doing this. Qualifications are all of equal value it is just that some are of a more practical nature and this needs to be reflected in their assessment methods" Principle 4, Schools & Colleges Groups
• "I think that in order to fully support and enable future goals, the qualifications system needs to recognise personal, social and emotional learning as well as academic and vocational skills and knowledge" Principle 4, Collaborative Communities Groups & Allied Discussion Groups
• "A change in the culture of 'attainment driven' recognising student participation, achievements, and experiences throughout the academic year" Principle 4, Schools & Colleges Groups
• "Creating a qualifications system which recognises the different achievements of every learner and provides parity of esteem to these which would be equally recognised as part of a transition" Principle 4, Collaborative Communities Groups & Allied Discussion Groups

There was strong support across all groups for the fifth principle, that the qualifications framework should be part of lifelong learning with progressive learning routes. For this aspiration to be realised, a number of respondents emphasised the need for the new system to be well understood by all parties, especially businesses and employers. Recording the broader learning' journey would be fundamental to allow learners to evidence the skills needed for the real world. Effective use of Technology, it was argued, would be of critical importance. Some examples appear below.

• "When qualifications change, there needs to be a common language, a robust and better communicated SCQF framework for businesses and industrial employers" Principle 5, Schools & Colleges Groups
• "It is important that learners and employers understand the connection between qualifications (the seamless learning journey). However, each qualification level must be valued in its own right, and not be seen only as a pathway to higher education" Principle 5, Collaborative Communities Groups
• "Pupils think that their final certificate should include all of their achievements" Principle 5, Schools & Colleges Groups
• "Progression routes from qualifications to continued learning opportunities should be built into the system" Principle 5, Collaborative Communities Groups & Allied Discussion Groups,
• "Digital platforms change every 5 or so years. It would be lovely if a person's whole learning journey was recorded safely and securely" Principle 5, Schools & Colleges Groups

There was similarly strong support from both Collaborative Communities and School & College groups for principle 6 where qualifications would include more flexible approaches to assessment. There were, however, a number of caveats. It was argued that this flexibility should be developed within an overall framework that was solid, reliable and based on standards. The need for strong communication between home and school was identified as a crucial component in making this work. There were also concerns about the time and resources available in the system to make these changes and about the extent to which there were strong national understandings of standards.

• "More effective communication between educational establishments and parents/carers is required to improve the possibilities of this outcome succeeding in practical terms" Principle 6, Schools & Colleges Groups
• "Collection and assessment of diverse evidence requires time and planning - lack of work time threatens this excellent principle" Principle 6, Collaborative Communities Groups & Allied Discussion Groups
• "A major part of our existing system due to the various difficulties of ensuring a national standard" Principle 6, Schools & Colleges Groups
• "There should be a robust national system for standardising evidence, to avoid criteria varying drastically across centres" Principle 6, Collaborative Communities Groups & Allied Discussion Groups
• "The system of recording needs to be available for all to access with a realistic budget for the equipment required to assess" Principle 6, Schools & Colleges Groups

There was almost unanimous support for principle 7, the establishment of a cyclical review process. Any reservations expressed about a cyclical review related to timescales for their development and in preparing the system appropriately to make best use of the system. Implicit in some of the responses was an implication that the model of the review was that of an external evaluation. Although this is one model, other approaches should be considered, approaches that are more collaborative in nature and designed to help the system to learn from findings rather than to judge those involved in the process.

• "Time frames need to be set for reviews" Principle 7, Schools & Colleges Groups
• "Review cycles are important, but the timescales need to be manageable" Principle 7, Collaborative Communities Groups
• "Historically, subjects were reviewed at times, especially if need methodology or theory came along. Some sort of review method is important" Principle 7, Schools & Colleges Groups
• "Reviews need to be sensibly scheduled and signposted in advance, and any changes communicated very timeously to all who require to respond to them" Principle 7, Collaborative Communities Groups
• "If the system is reviewed at more often intervals then changes can be made to help support everyone living in Scotland in education" Principle 7, Schools & Colleges Groups
• "Many felt that there needed to be something incorporated about timelines and the need for a cyclical review process, with adequate time given to allow change to bed in and for evaluation to be built in." Principle 7, Collaborative Communities Groups

In conclusion, Table 2. provides an overview of main themes, sub-themes and the codes used in the analysis in relation to the Principles statements among Collaborative Communities and Schools & Colleges Groups.

Table 2

N Principle	Themes	Sub - Themes	Codes
1, 2, 3, 4, 5	Language	Terminology	Too wordy Text overly complicated not sure of the word coherent Bit excessive and wordy
		Writing style	Shorter sentences
		Language used	Cluttered Too long worded a lot simpler language in the brackets vague language language is unfriendly
1	Practical Implications	Time	Heavy workloads Limited time to work on changes Responsiveness is limited
		Resource development	Limited books (English) regional needs requires budged Invest in small classroom size
		investment in technologies	Learning from Covid experiences Teach and assess online feasible Effective respond through online resources Online Courses/activities/assignment
	CfE – a gap between aspirations and practice: an educational system for a sustainable world	Developing strategies to face challenges	Social inclusion Economic and sustainability Environmental awareness Resilience Adaptability
		Qualifications for workforce	Attainment Achievement Awareness opportunities Developing skills for workforce Accessible for all Diverse learners
	Development of learning and innovation skills	Critical thinking	Problem solving Experiences in real world Literacy Numeracy Finance Logical thinking
		Collaboration	Partnership working Encouragement Collaborative space
		Flexibility and Adaptability	Making choices Diversity of learning styles Diversity of achievements
2	Is real collaboration currently possible in Scottish education?	Challenging to achieve in practice	School and business perspectives Time Resources Involving with external partners Communication with external partners Diverse learners' groups Yearly operationalisation of assessments Design stage of qualifications
		Challenges in a participatory democracy	Type of entity in charge Public vote Authorities Stakeholders
3	Tension between CfE aspirations and qualification	Barriers building a bridge	The focus on attainment-data model of assessment delivering curriculum for excellence Planning and lack of time clarifying capacities and meta-skills Developing interdisciplinary learning What matters in the curriculum' purpose Constrictions of the academic year Benchmarks and four capacities local authorities
4	Valuing academic and vocational pathways equally	Parity of esteem	Parity of esteem to academic and vocational qualifications Considering needs and aspirations of learners Prioritizing academic achievement as well as vocational Including wider courses Opportunities for practical experiences within the timetable Reviewing the culture of 'attainment driven' Recognising diversity in exam conditions
5	Developing a progression platform for progressive lifelong learning	The learning journey through digital platforms	Evidence recorded safely and securely Certificate based on all achievements Parity amongst assessment methods and learning
6	Flexible approaches to assessment in a reliable framework	Manageable standards in a qualification system	Time Resources Planning and designing Flexible approaches Discussion based on assessment Recognising wider achievement Assessment methods close to real-world Learning greater use of IT (technology)
7	A cyclical review process	Setting up timescales	constant review reactive review review method

III. Open Question: Finally, in relation to the Principles, respondents were invited to raise any further points they wished to make.

Are there any other proposed amendments not listed above which you regard as being of particular significance?

Participants from Collaborative Communities and Schools & Colleges groups suggested some ideas either to be added to the principles or given greater emphasis. (figure 5); also, these groups identified statements they considered unclear (figure 6). The following figures represent the frequency of these ideas. Only ideas identified by more than 5% of the sub-sample, based on the content analysis conducted are included.

When discussing principle one, perhaps the most interesting feature is the differences in the views of the Collaborative Community Groups and the Schools and Colleges Groups. The Collaborative Groups suggest a heightened profile for terms such as 'Scotland', 'complex' and 'globalised society. Whereas the Schools and Colleges Group propose greater emphasise words such as 'qualification', 'exam', 'time', and 'funding'.

With respect to principle two, there is greater consistency across both groups. For example, there is agreement about the importance of the term 'inclusive'. Similarly, both groups focus on ideas of putting ideas into practice. However, there were some differences in the language used in relation to the idea to practice relationship. For example, respondents from Collaborative Communities Groups & Allied Discussion Groups, more commonly used the words 'deliver' and 'by' when they referred to qualifications. On the other hand, School & College groups used words such as 'time', 'involve' and 'consult.

There were significant differences between the Community Collaborative groups and the Schools and Colleges Group in the changed advocated to Principle three. The CCGs advocated a stronger reference to the term Curriculum for Excellence, whereas the term Curriculum for Excellence is absent from the analysis of responses from the Schools and Colleges group. There is very little overlap between the desired changes across the CCGs and Schools and Colleges group.

Concerning principle four, both groups again suggested different ideas that could strengthen this principle. For example, participants from Collaborative Communities Groups & Allied Discussion Groups, emphasised the words 'qualification', 'recognise', 'value', 'every learner', 'aspiration' and 'goals'. Compared to the Schools & Colleges groups, where the focus has been the words 'vocational', 'skills', and 'aspirations'.

In the feedback for principle five, there was greater similarity in ideas across groups. such as 'clarity' and 'clear.' There were also, differences amongst these groups. For example, Collaborative Communities Groups & Allied Discussion Groups, highlighted the words 'learner', 'coherent' and 'support'. Contrast with the Schools & Colleges groups, which emphasise the words 'journey', 'transparent' and 'soft skills'.

When considering principle six, both groups used the words 'flexibility' and 'flexible' and emphasised the importance of a range of approaches to assessment. However, again there were differences in amendments proposed by the different groups. For example, Collaborative Communities Groups & Allied Discussion Groups, proposed the words 'assessment', 'time' and 'opportunity', whereas the Schools & Colleges groups emphasised the words 'IT', 'exam' and 'practical'.

Finally, for principle seven, again the two groups had different emphasises. For example, Collaborative Communities Groups & Allied Discussion Groups, suggested greater emphasis on ideas of adaptability, 'Scotland', and 'timelines' as necessary to better implement a cyclical review process. In contrast the Schools & Colleges groups, suggested including the words 'time', 'flexibility' and 'fair' would provide greater clarity to the principle.

Figure 5: Findings from participants' responses based on words highlighted or be added in principles.

Proposals for Changes to Language

Participants from both groups, Collaborative Community and Schools & Colleges groups, suggested rephrasing the principles or changing some words to improve clarity. See Figure (6)

Figure 6: Findings from participants' responses based on proposal amendments in principles.

The groups commonly made different suggestions but there were shared views about principles 4 and 5. For both groups, when discussing principle four, the words 'vocational' and 'academic' needed greater emphasis because of need to give parity of esteem to the diversity of achievements and skills developed in different pathways. In principle five, the word 'seamless' was felt to be ambiguous.

There were interesting differences noted between the groups in the feedback for principle seven. The Collaborative Communities Groups & Allied Discussion Groups underlined the terms 'needs' and 'young' compared to the Schools & Colleges groups, where many respondents referred to their view that there was 'too much change' in the Scottish system.

C. Principles as a whole

This third section reports on the responses of Collaborative Community Groups and Schools and Colleges Group responses to the principles as a whole.

I. Do respondents support the principles as a whole?

Participants' views about principles as a whole showed a similar pattern to those on the vision statement. 28% of participants in Schools & Colleges Groups approved without amendments to the statements compared to Collaborative Community Groups with 8%. Moreover, 56% of Collaborative Community Groups indicated that the principles as a whole should be approved with amendments, compared with 33% from Schools & Colleges Groups. Further, 36% of Collaborative Community Groups and 39% of Schools & Colleges Groups did not answer or comment on the statements.

II. Did respondents suggest the inclusion of further principles?

Several proposals were made.

There should be a principle about

- reducing the bureaucracy and workload around exams,

"Teachers need to be at the core of the design cutting bureaucracy and designing a curriculum that suits" Principles as a Whole, School and colleges

"it should be noted that teacher/lecturer workload will increase, so additional time for this should be allocated" Principles as a Whole, CCG

- the portability of qualifications internationally

"Stressed the importance of the portability of qualifications so that they are highly and easily transferable in international contexts, not simply for the learner but also in terms of promoting Scottish qualifications globally" Principles as a Whole, CCG

- the role of assessment, such as the promotion of excellence, the importance of data collection for accountability.

One respondent cited an OECD report and others shared their own experiences.

"If given the issues highlighted in the OECD report, should there be something more explicit about assessment and qualification design supporting effective learning and teaching / pedagogy instead of assessment driving learning and teaching" Principles as a Whole, CCG

Other participants highlighted words that could reinforce ideas in the existing principles. For example:

- 'Learner pathways', 'lifelong learning', 'progression' and 'transferability': Participants stressed the importance of promoting the wide variety of available learner pathways and highlighted that schools, colleges and educational bodies must work with parents to build better shared understand of the curriculum, the pathways and assessment options.

The relationship between national qualifications and vocational qualifications was questioned, potentially linking to the perceived need to promote wide learner pathways.

Other participants indicated that the current principles focus on young people and endpoints, neglecting the importance of lifelong learning. According to these groups, the principles should reflect a qualifications system that encourages learners to return to study, which has an intrinsic value at any age. It was also emphasised that learner journeys should be varied. That progression needs to be considered in developing any qualifications system, where one level of qualification provides a strong foundation for the next.

Other comments stressed the importance of the portability of capabilities so that they are highly and easily transferable in international contexts, not simply for the learner but also in promoting Scottish qualifications globally.

- 'Accessibility', 'inclusion', 'fairness' and 'attainment gap': Participants considered that the principles should focus more explicitly on inclusion and fairness, explicitly addressing the attainment gap. The attainment gap was a common topic amongst respondents.

A stronger commitment to accessibility for all learners was argued to be important within the principles, especially for learners with protected or special characteristics, including incorporating the government's anti-racism framework.

- 'Learning for sustainability': Other participants argued for a stronger reference to sustainability given Scotland's commitments in this field. Detailed reference to learning for sustainability and preparing learners for social, economic and environmental crises should be an explicit part of the qualifications or assessment system in the future.

- 'Co-creation', 'standard' and 'politicisation': Participants remarked that the principles should incorporate something related to system-wide responsibilities, such as co-creation, co-ownership or co-delivery of qualifications and assessment.

Decrease the political interference among the principles, highlighting that political interference should not be for promoting changes. Another relevant comment was that the current principles need to adequately cover the critical responsibility of setting and maintaining standards to provide a credible qualifications system that seeds confidence. Similarly, other participants highlighted the importance of validity, reliability, comparability and manageability, suggesting that validity, as educationalists understand it, should be incorporated into the principles.

III. Are there any other suggested amendments to the principles as a whole, for example should principles be combined?

Participants from Collaborative Communities and Schools & Colleges Groups suggested that principle one could be much broader and be integrated with principle seven. It was also suggested that principle one should be reworded to include the terms relevance and quality.

A second suggestion from respondent was to merge principles two and five. It was suggested that Principle five could be combined with principle two. A further suggestion was to connect principle two with principle six.