Conducting evaluation during times of change: Lessons from policy and community responses to the pandemic in Scotland

4. Theme 2 - Equalities

While some evaluations have compared outcomes for particular disadvantaged groups or intersectional impacts, many have highlighted methodological limitations preventing them from doing so. Given that the COVID-19 pandemic is known to have increased inequality for many marginalised groups, there is scope to learn from those studies that were able to say something about equality of outcomes, and to consider how this might be better achieved by evaluations in future crises, including through improvements in reaching seldom heard groups and gathering data on them that can be disaggregated.

4.1 Key findings

Use of administrative data allowed characteristics of target populations to be described where these data were routinely gathered. This was particularly evident in evaluations of health measures such as the COVID-19 Shielding Programme (Scotland), which employed data linkage between the list of shielding people and key socio-demographic and other COVID-19 datasets to profile the shielding group.³

Administrative data did not always support breakdowns for key groups. For example, the evaluation of the Extended Distress Brief Intervention Programme (EDBI) profiles service users according to age, sex and SIMD; but only aggregate service level data were available, preventing analysis of the impacts at an individual level or comparison between groups.⁵ The evaluation of the COVID-19 vaccine programme found that much of the ethnicity data linked to proved to be out of date.¹¹

Some diverse samples were achieved in surveys through multiple approaches to target excluded groups. For instance telephone follow-ups were employed in the Evaluation of the Near Me video consulting service in Scotland to reach digitally excluded groups (who may not have found or chosen to participate in online surveys); and in the phase one Universal Health Visiting Pathway (UHVP) evaluation to target parents in areas of multiple deprivation who were under-represented.^{6 4}

'Top-up' strategies to include under-represented groups sometimes had limited success. For instance, extensive efforts were made in an evaluation of perinatal care experiences to reach teenage mothers and women from low-income or minority ethnic households, including translated study materials and promotion through charity newsletters and websites. The final sample nevertheless under-represented those groups.¹⁶ Both waves of the Connecting Scotland evaluations reported low recruitment rates despite multiple measures to boost these.^[18][19]

Sample breakdowns or statistical comparisons based on area deprivation (SIMD) were achieved in some evaluations. However, where organisations rather than individuals were the unit of analysis (as in evaluations of emergency funds or the digitisation of care homes), findings were strongly caveated that many organisations operated in multiple areas and so could not be categorised by SIMD.

In the few evaluation reports providing breakdowns, sex and age were the protected characteristics most commonly analysed. No reports provide comparisons based on other protected characteristics, though target populations in evaluations of perinatal care and health visiting were characterised by pregnancy.^{16 4}

Where reported, reasons for not including analysis by groups experiencing inequality were either a lack of necessary data, or insufficient resources in terms of funding or staff available. A light touch approach to monitoring meant that recipients of emergency support measures could not be profiled; while some studies relying on routine data cited the lack of demographic detail available. Some studies that gathered primary data did not achieve a sufficiently large or diverse sample size to provide breakdowns (e.g. evaluations of Connecting Scotland¹⁹, Telemedicine Abortion at Home (TMAH)^[20]).

Other studies (e.g. evaluations of Connecting Scotland phase 1¹⁸, UVHP⁴) indicated that, while comparative analysis would have been desirable, this had not been undertaken due to project constraints such as time frames for reporting, even where (in a study on barriers to adherence with COVID-19 restrictions⁹) age, sex, ethnicity and disability data were gathered.

Augmenting quantitative studies with qualitative methods helped to understand the views and experiences of excluded groups; for instance in the evaluation of the UHVP, and qualitative research into the experiences of low income households accessing emergency support.^{4 12} In the evaluation of pandemic reforms in the Civil Courts¹⁷, impacts for vulnerable groups were reported on the basis of professionals' perceptions of these.

In qualitative research on the impacts of COVID-19 Mitigation Measures Among Children and Young People^[21], the contrast in views and experiences of children with different characteristics demonstrates the need to look beyond findings for people 'in general'.

4.2 Implications for evaluating in times of rapid change

• To understand impacts for particular groups during times of rapid change or disruption, there is a need to invest during times of relative stability in the collection of high quality administrative data that can support disaggregation. Several reports call for stronger routine data gathering on equality characteristics to facilitate similar evaluations in future, improvements that cannot simply be implemented once a crisis has hit. The Scottish Government's Equality Data Improvement Programme aims to address this need.
• Evaluations should be designed or commissioned taking account of specific information needs around inequality for particular groups. Studies often appear to have tried to answer questions about inequality from datasets designed to measure broader populations.
• Mixed methods should be considered a default approach; qualitative research may offer the best or only means of accessing the experiences of marginalised groups or sub-groups, such as people with relatively uncommon disabilities or ethnic communities that are small within Scotland.
• Cross-governmental research to evaluate the experiences and impacts of several different COVID-19 measures for a specific group experiencing inequality could address some gaps in equalities evidence. Most evaluations reviewed only one COVID measure, while studies that take a more holistic approach did not tend to single out the impacts of particular measures.
• Evaluations of measures to address rapid changes should be planned with sufficient resource to analyse in detail for particular groups. Some evaluations note a lack of capacity to take on detailed comparative analysis for disadvantaged groups. This could for instance be the function of a dedicated analytical team looking across a range of measures.
• Multiple strategies for reaching and recruiting disadvantaged groups are crucial to ensuring their experiences are captured. Although digital research had unique advantages in the pandemic, other options were important to reach excluded groups, for instance telephone interviewing.
• Advance planning and resourcing of a strategy with key organisations can ensure access to marginalised groups' views in times of rapid change. The engagement of stakeholder organisations for equalities groups has been vital to hearing those groups in evaluation work.
• Research and administrative data design should take account of how impacts across the full range of protected characteristics can be captured for evaluation during times of rapid change. As well as addressing the lack of understanding of impacts for some groups with protected characteristics (e.g. LGBTI+ or defined by religious belief) this can support a full understanding of intersectional impacts, which should be a priority.
• Evaluation strategy should consider process and impacts in relation to household income, or socio-economic background, where possible. Where socioeconomic disadvantage has been considered, this is almost exclusively on the basis of SIMD, which does not capture whether an individual is experiencing poverty themselves.