It's Our Future - Independent Review of Qualifications and Assessment: report

Appendix 9.2 Results of EQIA Framing Exercise

Overview of Policy - The Scottish Diploma of Achievement

As part of the Independent Review of Qualifications and Assessment, the Independent Review Group (IRG) led by Professor Louise Hayward has developed a new model for qualifications and assessment in the Senior Phase.

This model centres around the creation of a Scottish Diploma of Achievement (SDA) which the IRG states will: “gather evidence of learner achievement in a broader range of areas than is currently the case” and allow “learners to have opportunities to demonstrate achievements in three main areas: Study, Project and Personal”. The SDA would be awarded on completion of all three elements and at point of exit from the Senior Phase. Phased adoption of the Diploma is a key recommendation within the final report of the Independent Review, as is the need to continue to work with all communities with an interest in reform to qualifications and assessment in Scotland.

Who will it affect?

The Scottish Government will consider the recommendations of the Independent Review and will respond in due course. Should Ministers choose to accept, or partially accept, the recommendations of the Review including the Diploma, then this would directly affect learners, parents/carers, teachers, lecturers and a range of individuals employed in the education sector. There would also be wider implications across the education landscape including for the proposed new education bodies and the new inspectorate. There would likely be implications for wider society including further and higher education, recruitment and employment practices.

Inclusive Approach to Policy Design

The Independent Review states in its interim report that the Review has “adopted an inclusive approach to the design and development” of the Scottish Diploma of Achievement. There were three phases to the Review and three points at which comments and views were sought from stakeholders. The Review’s engagement centred around the IRG and associated Collaborative Community Groups (CCGs)

The IRG state that they have reflected on the three phases of engagement as well as a range of other evidence to develop the Diploma.

Framing Exercise - Scope

In the latter stages of the Review when the IRG had settled on a preferred model, an Equality Impact Assessment (EQIA) Framing Exercise was led by the Reviews’ Independent Secretariat and involved policy officials from within the Scottish Government’s Learning and Education Reform Directorates’. The task was to explore the Diploma using the Equality and Fairer Scotland impact assessment criteria. The aim of the exercise was not to provide a definitive assessment of potential impact, but to begin to explore some of the issues and provide advice around areas which require further exploration by Scottish Government, ahead of decisions being taken in respect of the Diploma. This short report prepared by the Review’s Independent Secretariat provides a summary of the exercise and the emerging conclusions.

It is important to be clear that the exercise focussed on the potential impacts of the Diploma on learners and did not explore potential impacts on those that will deliver the new qualification, or wider societal impacts.

It is also important to note that this summary was written before the final report and recommendations of the Review were written, thus final changes to the model have not been considered in this discussion.

The Framing Exercise – Summary of our discussion and findings

Framing Exercise: How

During the Framing Exercise we looked at evidence available under the protected characteristics as listed within the Equality Act 2010. In addition, the scope of the exercise was extended beyond the list of legally protected characteristics to include a number of other groups who may be impacted by the Diploma.

The task was to consider for each characteristic, relevant qualitative and quantitative evidence and consider a series of questions:

• The strength of the evidence available for that characteristic in respect of the policy and identify possible gaps in evidence.
• What areas of concern have been raised by stakeholders as part of the Review’s engagement?
• What opportunities might the Diploma present for advancing equal opportunities?
• What potential negative impacts might arise, and any mitigating actions?
• Which groups Scottish Government will need to engage with as part of the formal EQIA?

Evidence was primarily drawn from Scottish Government statistics and from the Review’s engagement. Links to the independent analysis of the three phases of the Review’s engagement can be found on the Independent Review Group’s page on gov.scot

Framing Exercise – Summary

The key conclusion from the exercise is that the Diploma, and the changes it would bring are likely to have an impact on learners in different ways. This includes learners with protected characteristics. The potential impacts, issues and opportunities identified during the exercise are summarised below. We recommend that a full EQIA is undertaken by Scottish Government as part of their considerations of the report and recommendations and in advance of any decision regarding adoption of the Diploma.

Protected Characteristic: Disability

The evidence considered during the exercise indicated that in principle the Diploma presents opportunities to improve educational outcomes for learners with disabilities and ASN. However, the ability to capitalise on these opportunities will depend on the detail of the Diploma and how it is implemented in practice.

During the Framing Exercise it was noted that the Diploma appears to align with one of the recommendations in the Morgan Review, Support for Learning: All our children and all their potential (2020) which called for, greater recognition of wider achievement beyond national qualifications.

“...the [qualifications] system should be set up to recognise the particular achievements of children and young people which goes beyond SQA qualifications....Children and young people for whom exam-based qualifications are not aligned to their learning needs and potential are not failures. The Review evidence is consistent that there must be recognition of individual achievement in Learning for Life. This means creating equally valued alternative pathways and ways of measuring individual progress.” (Morgan, 2020)

Evidence from Children in Scotland’s Inclusion Ambassadors received as part of Phase Three was considered. It was noted that the Inclusion Ambassadors were broadly “positive about the proposals”, and that they valued in the Diploma “the move towards recognising wider forms of success beyond the formal exams and assessments system and the opportunities for more personalised approaches.” (Inclusion Ambassadors / Children in Scotland – Phase three non-survey response)

However, areas were identified which the group felt would require further exploration by Scottish Government. These include:

• a detailed consideration of how the Diploma might impact on learners with a range of disabilities and additional support needs, including highly able learners;
• the potential impact on learners with multiple and complex disabilities and;
• the impact of teacher bias in assessment and the options for mitigating teacher bias.

There should be further engagement with disability organisations before a decision is taken around the Diploma. Further, should a decision be taken by Scottish Government to accept the proposals around the Diploma, there must be ongoing engagement with these groups to help design and develop the detail of the Diploma. This will be essential to ensure that the Diploma capitalises on the opportunity to improve the Senior Phase experience for learners with additional support needs and disabilities and crucially does not create unintentional disadvantage.

Protected Characteristic: Race

During the Framing Exercise consultation responses including from the Coalition for Racial Equality Rights (CRER) and Anti-Racism in Education Programme (AREP) were considered as well as notes from an Intercultural Youth Scotland led learner CCG discussion. These responses indicated particular concern around the proposal for greater reliance on teacher assessment in the Senior Phase.

“Teacher involvement in the setting of predicted grades in Scotland during the Coronavirus pandemic provides some insight into this. Learners from every ethnic group, on average, experienced over-estimated grades compared with actual performance within their group in previous years. With the exception of African, Caribbean and Black young people, however, every BME group of learners had lower levels of over-estimation than peers from white ethnic groups. This puts those from white ethnic groups at a relative advantage, as their attainment has been artificially raised to a greater extent.” (CRER)

“If Scotland is to move forward with more formative assessment relying on coursework marked by teachers, how will we ensure that racial bias (and other forms of bias) does not affect grades? Estimated grades for Black, Asian and Minority Ethnic learners are often lower than actual grades for blind-marked exams.” - A working group of anti-racist organisations and education stakeholders: Education Leadership and Professional Learning subgroup of the Scottish Government's Race Equality and Anti-Racism in Education Programme.

Possible mitigating steps to reduce teacher bias were discussed in the Framing Exercise. These include support and training for teachers as well as systems of school and national moderation. It is recommended that further research is undertaken to explore the potential for teacher bias and mitigating measures, before any decision is taken by the Scottish Government to increase teacher led assessment in the Senior Phase.

“I think from 2020 I believe it was scored on teacher student relationship. For some subjects I had higher scores but for some I know I failed but got good grade. The way it’s going there is definitely some bias.” - (CCG member - Learner, Intercultural Youth Scotland)

Another key concern in respect of racial equality was in relation to the Personal Pathway component in the Diploma:

“A range of known barriers impact Black and Minority Ethnic learners’ access to opportunities to gain awards and achievements outside of school. Racism within mainstream services targeted at young people is one such barrier... Black and Minority Ethnic households are twice as likely to experience poverty... Activities with a cost, including basic costs such as travel, may be inaccessible for many. Allied to this, the parents of Black and Minority Ethnic learners are more likely to be working in occupations with long or non-standard hours, making the practicalities of getting to and from clubs and activities more difficult.” - (CRER)

During the Framing Exercise it was noted that since the public consultation was issued, and in response to feedback during Phase Three of the Review, the proposed Personal Pathway in the Diploma has been amended. The focus of the Person component is not on the number of experiences the learner has undertaken but on what an individual has learned through an experience i.e., the reflection. It was noted that focussing on the reflection would have a lesser impact than focussing on activities. The proposal by the IRG that support from a suitable adult for example a Youth Worker or teacher is provided to complete the Personal Pathway, should be framed as an entitlement and should be guaranteed for each learner, was seen as a mitigating action. Notwithstanding this we felt that there may still be equalities issues in respect of the Personal Pathway which need to be explored further.

It is recommended that further work is undertaken with CRER and AREP to explore in more detail the opportunities and actions needed for the Personal Pathway component to support racial equality, before a decision is taken on this aspect of the Diploma.

It is recommended that before any decision is taken in respect of the Diploma further engagement is undertaken with representative organisation such as CRER, BEMIS and those representing Gypsy Traveller and Roma communities, to identify any other potential opportunities to advance equality of opportunity and crucially to ensure that the Diploma does not unintentionally create further disadvantage.

Protected Characteristics: Religion and Belief

It was noted that there may be equalities issues in respect of religion and belief. For example, the requirement in Roman Catholic schools to teach Religious and Moral Education throughout the Senior Phase was noted as something which needs to be considered. The Framing Exercise also flagged that that there could be equality implications in respect of the Personal Pathway.

It is recommended that engagement is undertaken with relevant faith groups such as Interfaith Scotland to explore the Diploma in respect of Religious and Belief equality.

Protected Characteristic: Sex

Qualitative evidence which indicates that there are ongoing structural barriers to girls accessing some subjects, in particular STEM subjects was considered as part of the Framing Exercise, and it was noted that the Diploma would not directly address this issue.

Overall, the group felt that the flexibility within the Diploma could potentially support both boys and girls but concluded that further work is needed by Scottish Government to explore in-depth the potential opportunities, and issues which the Diploma could present, in respect of Sex.

It is recommended that further research is undertaken in respect of assessment methods in relation to boys and girls. It is also recommended that there is engagement with the Gender Equality Taskforce in Education and Learning to ensure, that if accepted, the Diploma is designed and delivered in a way which will support gender equality in education.

Protected Characteristic: Age

The Framing Group noted that the Independent Review Group’s intention is that the Diploma would be available to all learners regardless of their age, and that for learners who have left school, they would have the opportunity to undertake the Diploma at a college.

It will be important for Scottish Government to engage with colleges to ensure that the proposal to deliver the Diploma in colleges can be realised in practice.

Protected Characteristic: Pregnancy and Maternity

The Framing Group are not aware of any relevant existing evidence at this time on pregnancy and maternity in relation to the policy. However, as part of the formal EQIA Scottish Government may wish to seek views on this.

Protected Characteristic: Gender reassignment

The Framing Group are not aware of any relevant existing evidence at this time on gender reassignment in relation to the policy. However, as part of the formal EQIA Scottish Government may wish to seek views on whether the proposals are likely to have any unintended consequences for this group.

Protected Characteristic: Sexual orientation

The Framing Group are not aware of any relevant existing evidence at this time on sexual orientation in relation to the policy. However, as part of the formal EQIA Scottish Government may wish to seek views on this.

Other Areas for Investigation:

In addition to the protected characteristics a number of other groups of learners were identified as requiring further consideration in respect of equality impacts.

Socio-Economic Aspects

Evidence considered in respect of socio-economic aspects drew on responses received during the three phases of engagement and in particular from the Equity CCG led by Dr Edward Sosu of Strathclyde University.

The main concern in relation to socio-economic equality was in respect of the Personal Pathway component. Many respondents in the public consultation and in the Phase Three engagement felt that for a variety of reasons it would be easier for learners from higher socio-economic backgrounds to complete this section of the Diploma than learners from lower socio-economic backgrounds, thereby exacerbating existing and longstanding inequalities in educational outcomes.

As outlined above, under Race, the Framing Group reflected that since the public consultation was issued, and in response to feedback during Phase Three the Personal Pathway component in the Diploma has been updated. The focus of the Personal Pathway component is not on the number of experiences the learner has undertaken but on what an individual has learnt through an experience i.e., the reflection. It was noted that focussing on the reflection would have a lesser impact than focussing on activities.

The proposal by the IRG that support from a suitable adult for example, Youth Worker or teacher, to complete the Personal Pathway component, would need to be guaranteed for each learner, was also welcomed and seen as a potential mitigating factor. Notwithstanding, the group felt that there may still be equalities issues in respect of the Personal Pathway component which will need to be explored further.

Other areas of concern in the Diploma noted by respondents in respect of socio-economic equality, include the possibility of teacher bias against learners from lower socio-economic backgrounds and concern around a move towards more digital learning, recognising that not all learners have access to digital technology and/or broadband.

“Digital inequity remains a serious issue for many learners, who lack access to IT devices and to wireless/broadband connectivity. We would further note that digital poverty can be hidden and is easily overlooked.” – (EIS)

On the other hand, the Framing Group reflected that during school visits the Review team had received positive feedback from a number of teachers and learners about aspects of the Diploma which they felt had the potential to reduce the poverty related attainment gap. For example, some teachers and learners reflected that:

• the move towards more teacher assessment was felt to be fairer since teachers know their student's ability and could provide more accurate grades than terminal examinations;
• an increase in flexibility and choice evident in the Diploma would support equality and;
• the Project and Personal Pathway component were viewed as opportunities to steer the focus away from success being purely about achievement in traditional academic subjects.

It is recommended that further engagement with organisations such as the Child Poverty Action Group and other similar organisations is a necessary next step to fully explore the potential socio-economic impacts of the Diploma.

Gaelic Medium Learners:

Bord na Gaidhlig, Comann nam Pàrant (Nàiseanta), CLAS (Comann Luchd-Teagaisg Àrd Sgoiltean/Gaelic Secondary Teachers' Association) and Education Scotland were amongst the responses received by the Review which considered the specific needs of Gaelic Medium learners.

It was noted that whilst those representing Gaelic interests were broadly supportive of the three components and the principle of the Diploma the following was emphasised:

Any new system needs to start from:

• an understanding across all indicators that Scotland provides education in either English or Gaelic depending on parental choice;
• opportunities to extend and develop the four capacities must be available equally for GME pupils and EME pupils and:
• all resourcing and training for education teams should be provided equally for the GME and EME systems.

If the Diploma or parts of the Diploma is adopted the GME sector including learners and teachers, must be involved in helping to design and develop the detail of the Diploma.

Looked after children:

The group reflected that whilst educational outcomes for looked after children have improved over the last decade there is still large gaps compared with all pupils. It was noted that it will be important to consider how the Diploma can be designed and developed to improve educational outcomes for looked after children. It will also be important to identify any unintended consequences on this group. Whilst organisations such as CELCIS and Who Cares? Scotland have provided views into the Review it is recommended, that should a decision be taken to adopt the Diploma, further engagement is undertaken with CELCIS and Who Cares? Scotland around the design and development of the model.

Home Educated Learners:

Home educated learners and their parents/carers involved in engagement as part of the Review expressed concern at any changes to the system which would make access to qualifications for home educated learners more difficult. Home educated learners and their parents/carers noted the current challenges with accessing national qualifications.

It is important to ensure that the needs of home educated learners are considered in the design and development of the Diploma and it is recommended that there should be further engagement with this sector, should a decision be taken to introduce the Diploma.

“It is important that the SDA is accessible to home educated young people. Needs checks and balances.” - (CCG member - Learners – Phase Three response)

Framing Exercise - Conclusions

As noted above this exercise is not intended to be a definitive statement or a full assessment of impacts. It does however present preliminary and indicative impacts which will require further exploration by the Scottish Government as part of its consideration of the IRG’s recommendations. It also highlights areas where there is a lack of evidence in respect of a number of protected characteristics.

Following this exercise, it is recommended that a full EQIA is required by Scottish Government and that this should be undertaken before Scottish Government respond to the proposals around the Diploma. This EQIA should include an assessment of the impact on learners and staff who will deliver the new qualification (predominantly teachers and college lecturers) as well as a broader consideration of wider societal impacts.