Conducting evaluation during times of change: Lessons from policy and community responses to the pandemic in Scotland

2. Introduction

In November 2021 the Deputy First Minister decided to convene a COVID-19 Learning and Evaluation Oversight Group to inform Scotland's recovery from COVID-19. The group's remit included synthesising evaluation evidence across policy and community responses to the pandemic (without a specific focus on NHS interventions or clinical studies) to distil learning for approaches to recovery. It also sought to identify key evidence gaps and advise on how these may be addressed.

The group was Chaired by Professor Linda Bauld, Chief Social Policy Adviser and included several Scottish Government Directors and experienced colleagues from a wide range of public, third sector and research organisations. The group has published a range of outputs drawing on evaluation evidence to inform policymaking^[1]. A subgroup was also convened to review existing evidence from evaluations of COVID-19 measures to develop key principles for embedding evaluation more systematically into policy making during times of rapid change or disruption. The sub-group included the following members:

• Dr Audrey MacDougall, Chief Social Researcher, Scottish Government
• Steven Marwick, Director, Evaluation Support Scotland
• Pamela Smith, Head of Poverty, Economy & Society, Public Health Scotland
• Professor Kay Tisdall, Chair of Childhood Policy, University of Edinburgh

This paper draws out implications for future evaluation arising from this work.

Approach

Policy teams in all Scottish Government portfolio areas were commissioned to provide details of all internal or commissioned evaluations of Scottish Government measures introduced in response to COVID-19. Studies of broadly defined impacts of the pandemic were not in scope unless they considered the difference made by Scottish Government interventions. Evaluations identified in this way were supplemented with details of Public Health Scotland and Third Sector evaluations of pandemic responses in Scotland that were known to members of the sub-group.

Reports identified in this way were reviewed by a member of the secretariat to identify 27 evaluation studies within scope for inclusion in this review. 24 were Scottish Government or Public Health Scotland evaluations of COVID-19 measures, while three were third sector reviews of COVID-19 responses (see Annex A)^[2]. The findings of published reports for these evaluations were summarised by the reviewer and discussed by the subgroup to draw out overarching observations and possible themes to explore. Following this a synthesis of key themes was drafted.

The wider COVID-19 Learning and Evaluation Oversight Group selected four themes to focus on:

1. Timing – evaluations were undertaken at different stages of the policy cycle and for different periods, and this theme was explored to understand what implications this had for their scope.

2. Equalities – the pandemic is known to have had unequal impacts, and this theme was included to consider whether and how evaluations took account of the implications of measures for different groups.

3. Evaluating Emergency Support Measures – the pandemic saw an unusually swift mobilisation of resources in response to emergent needs, creating an opportunity to learn how rapid policy development and deployment can be evaluated.

4. Evaluation Design – the subgroup considered it valuable to understand why some types of evaluation might be more common than others.

Findings under these themes are reported below.