Early Adopter Communities: Evaluability Assessment
This report presents the findings of an evaluability assessment for the school age childcare Early Adopter Communities. This includes considerations and recommendations for process, impact, and economic evaluations.
6. Conclusions
In line with the intention for EACs to employ place-based approaches to adapt delivery to their local context and meet the needs of their target families, delivery of childcare and family support provision varies significantly across EACs. Monitoring data is also collected inconsistently, both across EACs and within them, in cases where there are multiple childcare providers. These factors add complexity to the future evaluation of EACs and limit the potential of existing data, at least in its current form, to contribute to a future evaluation. Further primary data collection, as well as the exploration of secondary data sources, are therefore required.
There are also several overarching considerations that should be factored into future evaluation plans and that have informed our recommendations. These are covered in detail above and include: the ongoing evolution of EAC design and delivery; the ability to attribute any impact to the EACs, given the complex landscape in which they operate; the extent to which it makes sense, given the differences in delivery and data available, to evaluate EACs individually or at an overall level; whether the evaluation focuses on measuring outcomes at an individual or local level (both of which present challenges); and evaluation timeframes (with our recommendations being based on the short term due to uncertainty around the pace and scale of EAC expansion).
Our recommendations for a future impact, process and economic evaluation are summarised in Table 6.1. However, as noted, this assessment should not be seen as an evaluation plan or protocol and any future evaluation should include an initial scoping phase to further explore and refine the range of recommended methods.
| Evaluation element | Summary of recommended approach |
|---|---|
| Impact evaluation | There are inconsistences across EACs in the data collected and the frequency with which it is updated. However, if existing data can be streamlined and enhanced, there is the potential for it to contribute to the measurement of priority outcomes. This includes income, employment status and receipt of benefits. Furthermore, primary qualitative data collection can provide a rich source of information about how the EAC investment affects families and systems. Overall, a theory-based approach is recommended in the short term, most likely taking a realist evaluation or contribution analysis approach. There is potential to further scope options for a QED in longer-term evaluation, but this depends on how and when the programme is rolled out to other EACs and whether barriers to accessing secondary data and identifying a suitable counterfactual can be overcome. |
| Process evaluation | A process evaluation should consider the implementation of the EACs, taking into account the way in which outcomes have been achieved and the specific contexts within which EACs are operating. This should include analysis of monitoring data and qualitative data collection. |
| Economic evaluation | A 4Es approach is recommended, supported by Breakeven analysis. In the absence of a quantitative impact evaluation or the likelihood of conducting a primary survey, and the lack of statistical causal inference provided by theory-based evaluation design, a CBA for the programme would not be feasible. A CEA would also not be feasible, due to lack of detailed cost data. |
Contact
Email: socialresearch@gov.scot
There is a problem
Thanks for your feedback