Race Equality and Anti-Racism in Education Programme - Stakeholder Network Group minutes September 2021: paper - evaluation distillation

Summary: this paper:  

• captures the key points of the evaluation paper which was tabled at the previous Stakeholder Network Group meeting on 26 August (overview and background)
• captures the key points made in the discussion groups after the paper was presented (summary of key messages). (The full note of the discussion groups can be found as an annex to the minutes of the meeting on 26 August)
• will be used as the basis on which to have a short discussion under agenda item 4, in order to reach a decision on how to proceed with evaluating the Race Equality and Anti-Racism in Education Programme. Two possible options, which were discussed during the breakout sessions are that:

o    an evaluation sub-group is formed, consisting of Stakeholder Network Group members, which will establish an evaluation method. It would work in a similar way to the other existing sub-groups
o    an external (i.e. not part of the Scottish Government) organisation, for example a research body or an academic institution, would work closely with the Stakeholder Network Group to develop an evaluation model. A small number of members would be asked work with the external body, with the external body doing the majority of the development work

Overview

This paper aims to summarise:

• the issues discussed in the Race Equality and Anti-Racism in Education Programme paper on Evaluation (tabled at the meeting on 26 August) 
• feedback received during the breakout sessions at that meeting

Background 

The evaluation paper presented by Learning Directorate Analytical colleague Keith Dryburgh, set out information and points to consider when looking at how the Race Equality and Anti-Racism in Education programme can best be evaluated.

The paper covered the following key areas:

• what is evaluation
• why evaluate
• what does the evaluation process look like

And explained how this might be done using:

• evaluability assessment
• theory of change/logic models

 The paper noted that it was essential to keep in mind that the programme won’t have an immediate impact and will therefore be evaluated over a longer period of time. Work done on relationships and behaviours in schools requires a long term approach given that it is about changing culture and ideas. When evaluating impact, there is a need to be cautious and mindful of long term change.

It also posed questions for the group to discuss, as well as next steps

Summary of key messages

Stakeholder Network Group members were split into six break out groups to discuss the questions and each group was facilitated by a Scottish Government official. Key messages from the discussions have been captured in a fuller note that was circulated with the minutes of the meeting.

The evaluation work stream differs from the other four workstreams which the programme covers in that it requires a specific skillset, of which members may not have the same breadth and depth of experience compared to that found on the other four sub-groups. Given the existing demands on members’ time, it may be that there is an appetite to contract out the main body of the evaluation work, to an organisation who would work closely with members of the Stakeholder Network Group.    

Overarching points from the discussions can be found in the annex to the minutes of the meeting on 26 August but in summary it was agreed that there is a need to:  

• be clear about the process as well as the impact of what the Stakeholder Network Group is trying to achieve in terms of evaluation
• ensure sustainability – how do we evaluate in the long term
• retrospective consideration as well as forward look – in order to “unlearn”
• ensure that all four workstreams work collaboratively as part of the evaluation process as well as making sure that their agreed visions are embedded in it
• the importance of detailed and nuanced data cannot be underestimated

Next steps

At the meeting on 23 September, it would be helpful to consider and agree the mechanism for undertaking evaluation:  i.e.:

• do Stakeholder Network Group members volunteer to create an evaluation subgroup to take this forward
• is there a preference to contract a research body to work with members of the Stakeholder Network Group to develop an evaluation model