Early Adopter Communities: Evaluability Assessment

Executive Summary

This document summarises the findings of an evaluability assessment for the school age childcare Early Adopter Communities (EACs), including considerations and recommendations for process, impact, and economic evaluations.

Policy context and background

EACs are a key element in the Scottish Government’s School Age Childcare Transformational Change Programme. The Programme aims to design and build a system of school age childcare that provides care all year round, and supports parents and carers, particularly those on low incomes. The ambition to build a system of school age childcare sits alongside the Scottish Government’s wider commitment to eradicate child poverty. The Scottish Government recognises that affordable and accessible childcare is needed to enable parents and carers (from hereon in referred to as ‘parents’ for brevity) to access to training, study, or secure, stable employment and to increase household income.

Four EACs were established in October 2022 in areas within Clackmannanshire, Dundee, Glasgow and Inverclyde, supported by around £3.5 million in Scottish Government funding up to March 2024. A further investment of £16 million will be made over 2024/25 and 2025/26 and two new EACs will be established in Fife and Shetland. The EACs are intended to provide an opportunity to test and refine what an effective school age childcare system looks like in different communities. The EACs each provide funded childcare and wider family support for eligible families in their areas, working with a range of local partners. These included schools, childcare providers, various local authority teams, welfare advice, early years, colleges and the training sector. EACs took a place-based approach, therefore the way each EAC was set up varied depending on the local context and needs of local families.

Overview of early evaluation

The Scottish Government commissioned Ipsos Scotland to carry out work with the EACs to inform their approach to future monitoring and evaluation. The early evaluation comprised four stages: developing theories of change at both the community and national level; developing a monitoring and evaluation framework; conducting a process evaluation on early EAC implementation and delivery; and producing a preliminary evaluability assessment to inform future evaluation. The focus of this report is the evaluability assessment.

The evaluability assessment included reviewing options for obtaining quantitative and qualitative data to measure EAC outcomes and assess the implementation of EACs. This included identifying available monitoring information and outcomes data collected by EACs, secondary data sources that could be used to assess impact, and gaps that require further monitoring data and/or primary data collection.

The evaluability assessment also included examining the feasibility of a range of evaluation methods and designs to make recommendations for future impact, process, and economic evaluations. Proposed evaluation questions for each strand were also developed to guide future evaluations.

Impact evaluation

Future evaluation work would require the measurement of intended EAC outcomes, defined as part of the theory of change development. In collaboration with the Scottish Government, the following priority outcomes were agreed:

• System: improved understanding of how to deliver high quality childcare and continued improvement of EAC policy
• Family (parent): parents have more time to prepare for, start, sustain and/or increase hours of work or study; reduced cost of living; income from employment maximised; reduced financial pressure; more families engaged with support services; increased parental respite; and improved parental wellbeing (fewer crisis points)
• Family (child): children have increased social connections; increased opportunities for learning and new experiences; more opportunities to be physically active; access to nutritious food; improved social, emotional and behavioural development; and improved health and wellbeing.

The impact evaluability assessment involved three stages: 1) confirming that EACs met preliminary thresholds for future impact evaluation; 2) reviewing data sources for measuring priority outcomes; and 3) assessing the feasibility of impact evaluation methods and designs including experimental and quasi-experimental design (QED), and theory-based methods.

We concluded that the EACs met preliminary thresholds that warranted a future impact evaluation. Following this, a review of data sources found:

• Routinely-collected outcomes data – EACs do not currently capture comprehensive outcomes data, and there are inconsistences across EACs. If existing EAC data can be streamlined and enhanced, there is the potential for it to contribute to the measurement of priority outcomes. It is recommended that this includes capturing family income, employment status, and receipt of benefits in a consistent way at baseline and follow-up timepoints.
• Secondary datasets – secondary data to measure outcomes are limited, largely due to the small geographical areas of the EACs, meaning datasets do not report at a local enough level. A more robust option would be to access data that is not publicly available, such as household/individual level records from HMRC, DWP and/or Social Security Scotland. However, these require access applications, which are tightly regulated and time intensive. Future evaluation timelines would need to reflect this.
• Primary data collection – there are considerable challenges associated with establishing robust surveys to measure outcomes that are not available via other data sources. For example, this would either require consent for an evaluator to contact families directly (which would also require appropriate data sharing arrangements) or a heavy reliance on EAC staff to either administer the survey (adding burden and risk that this would affect relationships) or share an online survey link (risk for response rates). Given the range of providers and existing processes, this requires further consultation and ideally codesign. Alternatively, primary qualitative data collected via interviews with EAC families and staff is readily recommended.

The assessment of potential impact evaluation designs concluded that a theory-based impact evaluation is recommended as the most feasible and appropriate approach for evaluating the next stage of funding of the EACs (within the next two years). This timeframe has been chosen to align with existing funding commitments for the EACs. Multiple theory-based impact methods were considered, yielding two recommended approaches: realist evaluation (which seeks to understand how an intervention works by identifying the underlying causal mechanisms and how they operate in different contexts) and contribution analysis (which follows a step-by-step process that seeks to develop and test the theory of change and rule out alternative explanations). Both approaches involve testing the linkages and assumptions of the theory of change using data collection and analysis. To gather data, it is recommended that this includes building capacity within EACs to collect data on outcomes such as income and employment as well as primary qualitative data collection such as longitudinal interviews/focus groups with EAC and partner staff and longitudinal case studies with families.

There is potential to further scope options for a QED in longer-term evaluation, but this depends on how and when the programme is rolled out to other EACs and whether barriers to accessing secondary data and identifying a suitable counterfactual can be overcome.

Process evaluation

Process evaluation is critical when evaluating complex interventions because it helps assess how an intervention is implemented in practice and explain why an intervention works (or not). It should therefore be included in any future evaluation and build on the process evaluation undertaken as part of this early evaluation work. This would ideally further explore system change processes, while also retaining a focus on direct delivery with children and families. The process evaluation should assess the setup and delivery of childcare provision within the EACs to monitor what provision is delivered, gather information about the types of families that use the provision, and monitor attendance to understand frequency of use. In line with guidance on process evaluations, the process evaluation of EACs should aim to cover: implementation (what is implemented, and how?); mechanisms of impact (how does the delivered intervention produce change?); and context (how does context affect implementation and outcomes?).

While the proposed process evaluation questions have a different focus to the impact evaluation questions, the recommended methods are broadly similar. Analysis of monitoring data and documentation gathered by EACs should form a key data source. When families are referred to or register for childcare, EACs currently collect some monitoring data on family characteristics, engagement and reach. Most EACs also gather feedback forms to understand families’ experience. However, each EAC collects this information in different ways and at different timepoints. As such, it is recommended that a set of consistent indicators is agreed to support a more robust assessment of how EACs are implemented as well as how different factors (e.g. family characteristics or the amount of childcare used) affect the achievement of outcomes. Qualitative data collection is also recommended, including interviews with professionals (staff involved in the delivery of EACs and wider stakeholders) and interviews with families using EAC services.

Economic evaluation

The evaluability assessment considered the feasibility of undertaking different methods of economic evaluation for the EACs. These include the 4Es (Economy, Efficiency, Effectiveness and Equity) approach to assessing value for money (VfM), Cost-Benefit Analysis (CBA), Cost-Effectiveness Analysis (CEA) and Breakeven analysis. The results of the evaluability assessment suggest that the National Audit Office’s 4Es approach to VfM assessment is the most appropriate. In the absence of a quantitative impact evaluation, a full CBA is not feasible as the impacts of the intervention cannot be quantified. There is also a lack of cost information and benchmarking opportunities to support CEA. The 4Es approach is recommended by the UK Government’s Evaluation Taskforce in scenarios where benefits are difficult to quantify. The 4Es approach sets out four criteria to assess:

• Economy – the degree to which public spending on the EACs was the minimum required to achieve its objectives.
• Efficiency – the extent to which the outputs arising from the EACs were delivered efficiently (i.e. at minimum cost, using minimum resources, on the right things and at the right time).
• Effectiveness – how far the outputs arising from the EACs led to their intended outcomes and impacts, and the costs involved in producing these.
• Equity – how fairly the benefits of the EACs are distributed.

The recommended use of this approach involves answering a set of evaluation questions under each of the four “Es” through the process and impact evaluations, a programme of qualitative research, and the monitoring data known to be available. In this way, the economic evaluation builds upon the methods set out for process and impact evaluation and the robustness of the proposed approach relies heavily on the accuracy and robustness of the evidence they generate.

Conclusions

The recommendations in this report take into account the complex landscape within which EACs are operating; data collection challenges; and the need to access individual/household-level data to assess the potential for a suitable counterfactual. We recommend that a future evaluation should include an initial scoping phase to further explore and refine the recommended methods.