Race Equality and Anti-Racism in Education Programme - Stakeholder Network Group minutes August 2021: paper - programme evaluation - breakout groups summary

Programme evaluation – break out groups – discussion write up

Thinking about an evaluation of the group’s work/impact: 

• what is the overall purpose of the evaluation? 
• at a high level, What do we need to know?
• when do we need to know it? 

What is the purpose of evaluation? Ensure that our actions work to achieve the vision and ask ourselves whether we did what we set out to do. Focus on the process of evaluation, not just impact (important to be clear about exactly how impact is measured).  

When looking at evaluating the change we want to see, it would be helpful to review the group’s theoretical perspective. Theory of change or theory of unlearning? The challenge will be to evaluate how old knowledge and behaviours which are so deeply entrenched are unlearned and changed in the long term.

What do we need to know and when? Evaluation should be ongoing. SG’s role is to bring partners together now and in 5 to 10 years to continue evaluating. Concerns were shared about the sustainability of the work if the programme ends. Ideally, partners should be empowered to keep the work going.

Need to ensure that all four work streams are working together.

Work of the group has a huge remit – difficult to capture in an evaluation. 

Important to identify key milestones for each work stream to be able to assess progress.

Both qualitative (particularly case studies) and quantitative methods have merit.

Data.  Some data is held but it’s not always detailed or nuanced. It doesn’t capture the gaps e.g people who fell out of the system and why that happened. 

Essential for data to link to all four workstreams.  Link collection of it to the workstream visions

Evaluation needs to have realistic timescales – change would be in the long term. Need to be clear about what we mean by long term:  5 years? Generational?

Need to have a baseline to measure against 

Need to have a focus on how things have changed for the individual learner

Need to evaluate continually, including the processes used to deliver interventions which will improve outcomes. This ensures continuous learning and improvement, taking practical action to demonstrate that concerns are being listened to, resulting in high levels of buy in and engagement, particularly in children and young people

Specific issues which were suggested for consideration as part of evaluation:

• recruitment and retention over a long period of time
• nebulous concepts e.g wellbeing – how do we measure these?
• growing Awareness - the step before Attitude Change which requires measuring, eg have you seen?  Have you heard? Have you engaged with? 
• policy and school improvement plans – is there a space for a focus on anti-racism to allow schools to develop and engage others?
• how could inspections be used to capture real change on the ground?

Thinking about the evaluation approach and the role of the group:

• do you have any comments on the process (Evaluability Assessment) identified in this paper to work through how best the Race Equality and Anti-Racism in Education programme can be evaluated?
• how should the REAREP group take forward the approach? This could involve sub-groups or the whole group (or a combination)
• would contracting an external facilitator to take forward evaluation in collaboration with the group be beneficial?

Specify methods of evaluation for each output: how did things chance? A lot of these outputs will be new since this type of data has not been collected before. This requires innovative steps to create new evaluations.

A complex approach e.g Theory of Change, is not always the optimum one

Differing views on the creation of an evaluation sub group:  generally, not keen on a separate evaluation sub-group, in order to be mindful of not putting additional burdens on participants, however the point was made that each workstream needs to input to any evaluation undertaken.

Agree with the recommendation on an external facilitator, but this should be someone/organisation with a relevant lived experience, anti-racist knowledge, and an education background. However, since this may be difficult to find, so consideration should be given to who has control and power over the process of evaluation. People with anti-racist knowledge, education background and lived experience could be driving the plans and analysing the data that the external evaluator gathers. Commissioning a separate researcher from a minority ethnic background and anti-racist knowledge to support analysis of data would be welcome and possibly more easily achievable.

Need to consider how the voice of children and young people will be reflected in any evaluation process, ensuring it benefits those that the group is trying to target and improve outcomes for.