Schools - Regional Improvement Collaboratives: review

Chapter 5: Evaluation and impact

Key themes

• From this review, there is evidence that RICs are having an impact on:
  • developing the skills of school staff
  • delivery of lessons in the classroom - introducing new approaches and developing more consistency in pedagogical approach
  • skills and consistency around assessment and moderation
  • leadership and improvement planning skills
  • building a collaborative culture between local authorities
  • new online and blended learning opportunities for pupils.
• RICs have been on a journey around setting realistic and focused priorities which can be achieved through collaborative working within the RIC, identifying clear intended outcomes, setting appropriate measures and demonstrating impact.
• RICs have developed systems to track the impact of their work through to attainment and there is some evidence of increases in pupil attainment, achievement, equality and inclusion across regions. However, this connection is challenging to track, with the RICs forming one part of a large system, the context of a global pandemic since early 2020 and the very wide range of activity taking place to improve educational outcomes for young people in Scotland.
• Some stressed that in exploring impact, it was important to be realistic about what RICs could be expected to deliver given the amount of funding available to them, and the wide range of other activity to support improvement and equity in educational outcomes in Scotland.

Introduction

This chapter explores RIC evaluation and impact. It focuses on:

• processes for monitoring and evaluating effectiveness and impact
• the available evidence relating to the impact of the RICs so far.

When considering impact, it is important to understand that RICs remain at a relatively early stage in their development. The pandemic also impacted on the RICs in terms of shifting priorities and disrupting access to data.

For context, the interim review of RICs in 2018 found that all stakeholders felt it would take time to see an impact. There were some positive early examples of schools sharing best practice, and of skills development and practice change among school staff.

Monitoring and evaluation processes

Regional stakeholders felt that their monitoring systems were effective. They could monitor activities, who was taking part and where the gaps were. RICs were able to demonstrate:

• progress with activities
• outputs - events, activities, resources
• number of people engaging with activities
• demand for different activities
• profile of people and schools taking part
• participant satisfaction with activities.

However, many felt that while there was lots of evidence of RIC activity, there was a gap in terms of understanding the impact of this. Interviews with regional stakeholders demonstrated that many found it difficult to talk about the impact of the RIC. Most could talk about demand, activities and outputs, but found it harder to talk about the impact this had. Some regional stakeholders said that they could talk about impact based on their own impressions rather than wider evidence.

"If I'm entirely honest, we don't have evidence yet of impact." Regional stakeholder

National stakeholders also indicated that there was evidence about events, activities, participation and attendance, but less evidence about the impact of this on the classroom and on teaching practice.

Many regional stakeholders said that their RIC had been on a journey around demonstrating impact and were working hard on identifying clear outcomes or drivers and setting up appropriate measures. Most felt that they were beginning to make stronger use of data and evidence, and that this was built into 2020/21 RIC plans. Many indicated that while the RIC was able to demonstrate progress with activities and workstream outputs, it was much harder to establish wider change.

"We need to get better at measuring what is making a difference and evidencing this." Regional stakeholder

Some talked of the learning that had taken place around evaluating impact:

• setting realistic priorities - a few felt they had been over-ambitious
• focusing on a smaller number of priorities
• setting clear targets - being SMART about setting outcomes and how to measure progress
• using and understanding data - selecting a small group of measures that fit with the intended improvement and not trying to measure everything
• using a mix of quantitative and qualitative evidence
• setting time aside to explore impact.

"We want to be more slick regarding performance measures." Regional stakeholder

"How do we measure what we value?" Regional stakeholder

"There is so much data, we have to be clear about what it is we are trying to find out. Otherwise you can drown in the volumes of data." Regional stakeholder

Many regional stakeholders felt that it was also very challenging to demonstrate that impact was directly linked to the RIC. It was difficult to say if change had come about due to the RIC, the local authority, both or wider activity.

"Attribution continues to be an issue. It is hard to say conclusively that improvement is down to the RIC." Regional stakeholder

National stakeholders also commented on the ability to demonstrate impact of the RICs. Some national stakeholders stressed that in exploring impact, it was important to be realistic about what RICs could be expected to deliver in terms of improvement given the amount of funding available to them, and the wide range of other activity to support improvement and equity in educational outcomes in Scotland. A few felt that there was a lot of scrutiny related to RIC funding, compared with some other approaches.

"Are our expectations of RICs realistic?" National stakeholder

National stakeholders had varied views on what the intended impact of the RICs was, and how to measure this. One national stakeholder felt that there was a need to develop a system at national level to track and monitor change in the education system more widely and explore where RICs fit into this. Some national stakeholders highlighted that the focus should be on tracking the impact of RIC activity at classroom level, such as change, improvement or consistency in pedagogy while others felt that RICs should also demonstrate impact on pupil outcomes.

Each of the RICs was working on further developing its ability to demonstrate impact. However, the pandemic, re-focus of activities and gaps in data for 2020 and 2021 made this challenging. In some cases, the pressures relating to the pandemic meant that evidence had been gathered but not yet analysed.

Example: Northern Alliance - Small tests of change

The RIC has implemented 'tests of change' and explored impact. For example, for maths attainment the RIC is working with the University of Highlands and Islands on a project which involves work with 50 primary school teachers from across the 8 RIC authorities, working on fractions at Level 2. Key features include:

• development of a logic model and a peer mentoring approach
• data collated and presented in an anonymised spreadsheet, to facilitate professional dialogue around the data and share practice
• supporting children with maths anxiety – and will involve changes in pedagogy, interaction and assessment.

The project has measurable aims and seeks to build the confidence of teachers and pupils, as well as its impact on 300-400 learners.

More widely, Northern Alliance is collecting and analysing data to understand whether activities are leading to improvement. The RIC uses a range of processes to capture progress including a data dashboard, progress tracker and improvement studies. The RIC is finding that case studies and real life examples are particularly powerful, showing how the work they have done has made a difference to an individual. This sits alongside the data and provides lived experience and richness of communication. The RIC is also collaborating with the Data for Children's Collaborative (with UNICEF). They aim to identify 'meaningful measures' for practitioners to make use of in order to improve outcomes for children and young people across the Northern Alliance.

Example: Forth Valley and West Lothian RIC - Using data

The FVWL RIC Performance Group looks at data and evidence across the four RIC local authorities. It is led by an Education Scotland Attainment Advisor, and links with the work of Data Coaches within each local authority. The group is developing an evaluation framework with progress milestones, which will be used to monitor impact.

Evidence about impact on schools

As part of this evaluation, existing evidence about impact was reviewed, and new evidence gathered through 53 interviews with school staff from 50 schools. Through engaging with the RIC, school staff reported improvements in:

• skills of school staff
• delivery of lessons in the classroom
• assessment and moderation
• school leadership and improvement planning.

Skills of school staff

The RICs have supported school staff to develop their skills. Two thirds of the school staff involved in this review felt that they had developed skills and learned new things. Through this, staff became more empowered, inquiring and reflective in their practice.

"It's broadening their horizons, because they're influenced by research and evidence." Regional stakeholder

"Creativity and new thinking happen when you get together with other teachers." Headteacher, primary school

Example: Northern Alliance - Skills and confidence

Northern Alliance has evaluated many of its events, professional learning and opportunities for collaboration through exploring impact. This shows that teachers developed their skills - for example around facilitation and model for improvement - and reported changes in pedagogy and approaches to assessment as a result.

Delivery of lessons

Many school staff felt that through the RICs they had improved their delivery of lessons in the classroom due to:

• introducing new approaches to learning based on research and evidence
• being more consistent in their pedagogical approach across the region
• being more aware of issues such as the attainment gap and poverty proofing
• having access to a wider range of tools and resources

"It has had a really positive impact on the school. We have changed the way we think about CLPL and how to deliver lessons in the classroom." Acting headteacher, primary school

Learning through the RIC has helped to drive improvements within individual schools, as learning is used and cascaded. School staff gave examples of adopting approaches they had learned about across the whole school.

In a few cases, schools had adopted more research based and reflective practices, as a result of learning through the RIC. For example, one school had participated in a learning set, felt it would drive improvement, and so took the approach and used it with the school staff team, grouping staff into small teams of 4 or 5 who would not normally have the chance to have professional discussions. This is leading to better and more consistent approaches in the classroom.

Example: West Partnership - Evidence of impact on delivery

The Robert Owen Centre for Educational Change at Glasgow University undertakes external evaluation of the West Partnership. It produced an interim evaluation in February 2021. The evaluation highlighted that the West Partnership was still at an early stage in its development, had been impacted by COVID-19, and the evidence of impact on learners would take time. However, it found that the West Partnership was promoting a collaborative ethos and workstream activity was starting to impact on learning and teaching in the classroom.

Assessment and moderation

School staff also highlighted that through learning and leadership opportunities available through the RIC, they had become more skilled at professional judgement, assessment of progression and achievement of a level, quality assurance, evaluative writing, providing feedback and self-evaluation.

Example: West Partnership - Skills and confidence

The West Partnership undertakes evaluation of its events and programmes. Evaluation of events for headteachers and depute headteachers found that these would impact on practice through providing teachers with reassurance to try approaches, supporting teachers to get ideas to apply in their own contexts, and increase confidence in their role. For example, of the 150 staff who attended evaluative writing professional learning through the RIC 80% indicated that they felt their confidence levels improved.

Leadership and improvement planning

Some senior leaders felt that school leadership and improvement planning had also improved, resulting in a better experience for children and young people. Key areas of skills development included around management, coaching, strategic change and recovery. Leaders also improved their skills around professional dialogue and assuring quality of teaching.

"I was able to use the theory I had learnt on the leadership development course to lead in a time of change." Headteacher, primary school

Evidence about impact on pupils

New opportunities for pupils

Some regional stakeholders and school staff felt that the RICs had opened up new opportunities for young people. For example, a few elected members highlighted that young people would have a more positive education journey directly due to opportunities opened up by the RIC - including online learning, Easter study and contribution to the national e-learning offer.

Regional stakeholders also highlighted that because of the RICs, pupils would have better access to online and blended learning. Learning from this digital offer has encouraged the RICs to consider shared online learning opportunities, such as offering a selection of Advanced Highers online, which opens up new opportunities for pupils who may previously have been unable to access these courses or have faced challenges travelling and co-ordinating their timetable.

Pupil attainment

Ultimately, improvements in opportunities, teaching and leadership should result in longer term positive outcomes in terms of pupil engagement with school and attainment.

This link is harder to track, with the RICs forming one part of a large system. This has been made more complex with the pandemic affecting wider pupil health and wellbeing, learning outcomes and access to data about attainment. Both regional and national stakeholders stressed that it was challenging to track improvement due to gaps in robust and reliable data for 2020 and 2021.

However, the RICs are working to develop ways of tracking how improving teacher skills, teaching approaches and leadership and sharing best practice and experience is having an impact on attainment.

Example: South East Improvement Collaborative - Tracking improvement

The SEIC 2021 plan indicates that there has been an increase in the How Good is Our School 4? Framework quality improvement indicators between 2016/17 and 2019/20.

• learning, teaching and assessment - 17.2% increase
• leadership of change - 14.5% increase
• raising attainment and achievement - 13.5% increase
• ensuring wellbeing equality and inclusion - 11.5% increase.

Example: South West Education Improvement Collaborative - Leadership

High level indicators for SWEIC show increases between 2016/17 and 2019/20 in the proportion of early learning, primary and secondary establishments evaluated as good or better for:

• learning, teaching and assessment - from 52% to 76%
• leadership of change - from 54% to 73%
• raising attainment and achievement - from 62% to 86%
• wellbeing, equality and inclusion - from 70% to 82%.

Example: West Partnership 2020-2023 critical indicators

The West Partnership has agreed high level indicators which will allow it to report on the impact of its work with schools. Baselines were taken in 2016/17 and 2017/18. RICs were established in 2018 and information is available for 2018/19. However, there are gaps in most of the data for 2019/20 due to the pandemic, which means it is not yet possible to explore trends and the extent to which these have been driven by the RIC.

The types of measure being reported on for the 2020-2023 plan include:

• Primary - % of pupils P1, P4 and P7 achieving expected levels in literacy and numeracy
• Secondary - % of S3 pupils achieving third level or better and fourth level or better in literacy and numeracy
• Leavers - % of leavers achieving one or more and five or more awards at SCQF Level 6 or better
• Attendance - primary and secondary attendance and exclusion rates.

Indicators are also in place for the percentage of establishments evaluated as good or better for leadership of change, and learning teaching and assessment during HMIE Inspections, and changes in staff knowledge, understanding and confidence as a result of professional learning are also gathered and examined to identify next steps in planning.