Coronavirus (COVID-19) Learning and Evaluation Oversight Group minutes: February 2022

Attendees and apologies

Attendees:

• Linda Bauld, Interim Chief Social Policy Adviser (Scottish Government), and Professor of Public Health (University of Edinburgh) (Chair)
• Andrew Watson, Director for Budget and Sustainability, Scottish Government
• Kathy Johnston, Interim Deputy Director, Economic Analysis, Scottish Government
• Mary McAllan, Director for Covid Recovery, Scottish Government
• Michael Kellet, Director of Population Health, Scottish Government
• Andrew Kerr, Chair of SOLACE
• Tim Ellis, Head Performance and Outcomes, Scottish Government
• Claire Sweeney, Director of Place and Wellbeing, Public Health Scotland
• Jim McCormick, Chief Executive, Robertson Trust
• Nick Watson, Professor of Disability Studies, University of Glasgow
• Patricia Findlay, Professor of Work Employment and Organisation, University of Strathclyde
• Sarah Davidson, Chief Executive, Carnegie UK
• Steven Marwick, Director, Evaluation Support Scotland
• Kay Tisdall, Professor of Childhood Policy, University of Edinburgh
• Vittal Katikireddi, Professor of Public Health, University of Glasgow
• Tom Lamplugh, Head of Social Policy Unit, Scottish Government (Secretariat for the Covid-19 Learning and Evaluation Group)
• Graeme Wilson, Social Policy Unit, Scottish Government
• Fran Warren, Social Policy Unit, Scottish Government

Apologies were received from:

• Sarah Skerratt who is replacing Rebekah Widdowfield, Chief Executive, RSE
• Emily Lynch from the Improvement Service
• Maia Angel, University of Edinburgh
• Nicola Dickie, Director of People Policy and Health and Social Care COSLA

 

Items and actions

Introduction

The Chair welcomed everyone to the second meeting of the Covid 19 Learning and Evaluation Oversight Group. 

Actions from the first meeting of the group:

All action points from the first meeting have been completed. Key actions included updating the Terms of Reference, setting up a webpage on the Scottish Government website to publish papers and minutes and inviting comments on the Grant Funding Call for Proposals.

Terms of Reference

 A number of changes were made to the Terms of Reference this included:

1. Changing the reference in the opening paragraph to be about organisational learning from Scotland's approach to the pandemic.
2. Broadening the reference from Scottish Government to Scottish Government and partners  elsewhere in the document and including an additional sentence stating the group will support other sectoral and cross-cutting Covid response and recovery groups.
3. Addition of a new sentence clarifying the ambition that the work adds up to more than the sum of its parts.

 Members of the group confirmed that they were content with these changes.

Update on Grant Funding Call

Tom Lamplugh thanked group members for commenting on the previous grant funding call.  The call for applications has been advertised around 4 key themes:

• theme 1: mapping of key external evaluations of Scottish Government Covid-19 interventions completed or currently underway within Scotland
• theme 2: international pandemic recovery strategies
• theme 3: learning from public service innovation and creativity and
• theme 4: inequalities and human rights

The closing date for proposals is 28th February. The intention is to assess proposals quickly and be in a position to issue grant letters by the end of March. 

Requests for support in assessing tenders may to be made to group members where proposals relate to group members particular areas of expertise. 

Action: Tom Lamplugh to update the group on the grants in advance of the next meeting

Mapping Covid-19 related Scottish Government evaluations 

Graeme Wilson presented to the group on a mapping exercise to examine Covid 19 evaluation work underway or recently completed by the Scottish Government. For this exercise the Magenta Book definition of evaluation was used:

‘Evaluation is a systematic assessment of the design, implementation and outcomes of an intervention. It involves understanding how an intervention is being, or has been, implemented and what effects it has, for whom and why’.

Gaps in existing evaluation work were identified by several members of the group in discussion. These included gaps in the evaluation of:

• education interventions and young people’s transition into the labour market
• economic interventions and the targeting of business support interventions
• homelessness initiatives
• impacts of interventions on specific groups such as children with additional support needs, ethnic minorities, gypsy traveller children, young people with caring responsibilities, older people and disabled people
• gaps in economic evaluation and cost effectiveness of interventions
• intersectional analysis and the impacts on different groups (and overarching impacts) in the population of a range of government interventions

Points were made about the importance of understanding the connections between interventions and learning how we could better join up individual interventions in future crisis situations. It was suggested that we should be thinking about this work in relation to how people have been affected as communities rather than looking at interventions in isolation.  It was also suggested that it is important to have a broader focus on learning that goes beyond just formal evaluation.

Clare Sweeney identified a number of evaluations being undertaken by Public Health Scotland which would have relevance to the group.

Mapping non-Scottish Government Covid-19 evaluation work

Approaches to mapping non-Scottish Government Covid-19 evaluation work were discussed. Potential approaches discussed included:

- mapping work sponsored by the Research Councils

- looking across work done by National Core Studies

It was suggested that mapping of evaluation work would best be done in response to targeted questions. The starting position should be to ask ‘What do we need to know for to support recovery from Covid 19?’. Otherwise, there is a risk that we might end up trying to map everything. 

Action – It was agreed to give further consideration to how we could devise a more targeted approach to mapping evaluation work in order to learn from the pandemic.  

Priorities for the Covid-19 Learning and Evaluation Oversight Group

The group discussed and agreed the following actions for the first 6 months of the group:

Action: Map the range of Covid-19 evaluation work currently ongoing and recently completed in Scotland and develop parameters to give a clear focus to this work.

Purpose: To understand which Covid-19 related interventions are being evaluated, how they are being evaluated and potential gaps in evaluation work.

Action: Draw on the resources and experiences of the Group to develop a proposal to address priority gaps in Covid-related learning and evaluation (social and economic aspects in particular) drawing on prioritised gaps.

Purpose: To ensure that we learn from this unique period in Scotland’s history and, where practical, invest in learning and evaluation.

Action: Begin to synthesise and distil learning from completed evaluation studies in Scotland and identify a small number of priority themes for learning.

Will also think about how we can synthesise the impact on an individuals and / or communities from multiple interventions.

Purpose: To begin to develop a thematic framework for interpreting, synthesising and disseminating key findings from completed studies and evaluations to different audiences.

Action: Successfully commission a programme of small grants to provide a baseline foundation for the work of the Covid-19 Learning and Evaluation Group.  Once grants are decided, we may seek presentations on themes not covered in the grants that are directly relevant to the CRS work. 

Purpose: To support Scotland’s response to, and recovery from, Covid-19, and ensure that the value of our Covid-19 responses is fully captured to inform future policy in Scotland.

Action: The Secretariat will further develop a proposal for the longer term priorities for the group.

Any other business

Tom Lamplugh mentioned the UK Government are inviting bids for the Evaluation Accelerator fund, Bids are invited by 1 April.

Action: Tom Lamplugh to send around details about the fund to the group.

Date and agenda for next meeting

Suggested Items for the next meeting include:

1. Discussion of the grant funding call and gaps in the grant funding programme of work
2. Further discussion around prioritising mapping of evaluation work focused Covid 19 interventions
3. Developing a longer term plan for the Group
4. Presentations from group members on areas of work they are involved in that are relevant to the group
5. An update on the Covid Recovery Strategy

 Next meeting

The next meeting will be scheduled for late April / early May.