Early Adopter Communities: Evaluability Assessment

1. Introduction

This report presents the findings of an evaluability assessment for the school age childcare Early Adopter Communities (EACs). This includes considerations and recommendations for process, impact, and economic evaluations.

Policy context and background

EACs are a key element in the Scottish Government’s School Age Childcare Transformational Change Programme. The Programme aims to design and build a system of school age childcare that provides care before and after school, and during school holidays. As a result, it aims to support parents, particularly on low incomes, to have secure and stable employment.

School age childcare is care provided to children outside of normal school hours. The school age childcare system is made up of childcare and activity services that children attend outside of normal school hours, and parents and carers may use to support work, learning or rest. It can take place in breakfast clubs, after school clubs and holiday clubs, or at home, and encompasses various provision types including regulated providers, registered childminders, and organised activities such as sports clubs or cultural activities, which currently have no requirement to be registered with the Care Inspectorate. In addition, many families rely on friends and family, or consider at-home services such as nannies.

The School Age Childcare Transformational Change Programme was established to help support the Scottish Government’s wider commitment to eradicate child poverty. The Child Poverty (Scotland) Act 2017 set four legal targets for reducing child poverty by 2030. This included a target that less than 18% of children will be living in relative poverty by 2023/24 and less than 10% will be living in relative poverty by 2030. The latest figures show that 26% of children were living in relative poverty in 2022/23. Actions to achieve these targets are set out in Tackling Child Poverty Delivery Plans. In 2022, the Scottish Government published its second Delivery Plan, Best Start, Bright Futures. The plan recognises that affordable and accessible childcare is needed to enable parents and carers to gain access to training, study, or secure, stable employment and to increase household income. In the strategic early learning and school age childcare plan, the Scottish Government also recognises that high quality childcare can positively impact children’s development and family wellbeing.

More broadly, the Tackling Child Poverty Delivery Plan recognises the need to work differently in order to meet targets. It sets out a commitment to work collaboratively with partners to ensure that support systems work for the people who need them most, to trial and evaluate new approaches, adapt ways of working, and to monitor and understand how changes are impacting on priority families. The Delivery Plans set out the following six priority groups most at risk of experiencing child poverty:

• lone parent families;
• families with a disabled adult or child;
• larger families (three or more children);
• minority ethnic families;
• families with a child under one year old;
• families where the mother is under 25 years of age.

EACs are part of this landscape of innovative approaches to service delivery, alongside initiatives such as the new Fairer Futures Partnerships which build on existing place-based partnerships in Dundee and Glasgow (originally known, and referred to here, as the Pathfinders) and the Family Wellbeing Partnership in Clackmannanshire, the Whole Family Wellbeing Fund, and the No One Left Behind employability approach.

Four EACs were established in October 2022 in local areas within Clackmannanshire, Dundee, Glasgow and Inverclyde. These areas were chosen because they put themselves forward and submitted considered thoughts on how local childcare systems could support some of their most deprived communities. In terms of the specific location within each local authority, each project drew on a range of information to inform the decision on where to locate EAC services. Level of deprivation was a key consideration for each project, using the Scottish Index of Multiple Deprivation (SIMD) data. EAC funding proposals included a range of quantitative indicators of the level of need in the community. Commonly used indicators were: number of children in receipt of free school meals; number of children in receipt of school clothing grants; and percentage of children living in relative and/or absolute poverty. A wide range of other indicators were also mentioned, such as number of food banks, workless households, minority ethnic families, children with additional support needs (ASN) in the area; pupil attainment rates; and youth offending rates.

The EACs have been supported by around £3.5 million in Scottish Government funding over two phases up to March 2024. A further investment of £16 million will be made over 2024/25 and 2025/26 to continue to support the existing EACs and to develop new communities in Fife and Shetland. Setting up the projects was intended to provide an opportunity to test and refine what an effective school age childcare system looks like in different communities. More detail on the four existing EACs is included in Chapter 2. The report uses the terms ‘EACs’ and ‘EAC projects’ to describe the investment in local communities.

The EACs provide funded childcare and family support for families who fall into the six priority groups. They do this with a range of local partners, including schools, childcare providers, various local authority teams, welfare advice, early years, colleges and the training sector. The EACs build on learning from the Access to Childcare Fund (ACF), which tested different service-level approaches to childcare delivery for low income families. An evaluation of the Fund found that, in order to benefit fully from provision, some parents needed support with a range of wider wellbeing issues. Strong relationships between families and staff were seen as vital to this and, when projects employed a dedicated family support worker, they felt this had significantly enhanced their family support offer. Partnership working was also seen as key in maximising the reach and accessibility of projects via referrals. Partners also helped projects to achieve outcomes for families by providing additional sources of support to which staff could signpost.

Overview of early EAC evaluation activities

The Scottish Government commissioned Ipsos Scotland to carry out work with EACs to inform its approach to future monitoring and evaluation. This work comprised four key stages: 1) Developing theories of change at both a community level (for each EAC) and a national level; 2) Developing a monitoring and evaluation framework clearly setting out what data is currently, and could be, collected by current and future EACs to support effective evaluation; 3) Conducting a process evaluation on early implementation and delivery across current EACs; and 4) Producing a preliminary evaluability assessment to inform future evaluation.

This evaluability assessment report delivers on the fourth and final stage of the work and draws on evidence collected as part of the previous stages. While the EAC theories of change are included in this report and the monitoring and evaluation framework can be found as a supporting file, the process evaluation report is a separate output which should be read in conjunction with the evaluability assessment.

Aims and objectives of the evaluability assessment

The overarching aim of this evaluability assessment is to provide recommendations to inform the development of plans for future evaluation. More specifically, the objectives for this element of the work were to:

• Identify priority outcomes for future evaluation (based on the theory of change development work) and the expected causal pathways leading to those outcomes.
• Scope and explore the feasibility of a range of methodologies (including a Randomised Controlled Trial, Quasi-Experimental Designs, and Theory-Based Evaluation approaches) to evaluate the impact of the EACs.
• Identify existing and potential data sources (both primary and secondary).
• Scope the economic evaluation, identifying data requirements in relation to costs and impact measures.
• Provide recommendations on the most feasible approaches taking account of potential constraints, such as timelines, budget, and data collection options.

Methods used to inform the evaluability assessment

The following activities were conducted as part of the evaluability assessment:

• An initial meeting with the Scottish Government to confirm the overall aim, specific objectives, potential constraints, and other parameters for future evaluation.
• Desk research to identify other potential data sources and measures relevant for the EAC priority outcomes, and an interview with the Improvement Service to gain a better understanding of relevant secondary data.
• Assessment of different process and impact evaluation methodologies. This was based on the theories of change and priority outcomes, the feasibility of data collection (by EAC staff on a routine basis and/or external evaluators as part of an independent evaluation), the availability of secondary data, and options for identifying an appropriate counterfactual (comparison group).
• A value for money (VfM) assessment considering the feasibility of collecting information relevant to VfM, reviewing existing monitoring data and secondary data sources, and exploring the possibility of collecting data through the impact evaluation.

It is important to note that the evaluability assessment was also informed by the previous evaluation activities, specifically the development of EAC theories of change, an EAC monitoring and evaluation (M&E) framework, and a process evaluation of early set up and implementation across the EACs. More details on the methodologies used for each of these can be found in the EAC process evaluation report and the theories of change are discussed further in Chapter 2.

Ethics approach

Both Scottish Government and Ipsos Ethics Review Checklists were completed ahead of an evaluation work with the EACs. Researchers gathered informed consent from all participants prior to data collection, emphasising the voluntary nature of their participation. To minimise burden and barriers to participation, all fieldwork was conducted at a time convenient to participants. Since EAC project leads were being asked to give up the most time to take part in the evaluation study, the nature and extent of the time commitment was outlined in advance. Where it was not possible to cover everything during interviews, participants were able to share further thoughts via email. Ethical considerations for future evaluation are noted throughout the report.

Scope and limitations

When reading this report, the following considerations regarding its scope and limitations should be kept in mind.

• Each EAC has a unique childcare and family support offer informed by their local context, existing provision, partnerships, and more. Overall, we would recommend evaluating the EACs together, although it should be noted that some elements may be difficult to compare across projects.
• Throughout the study period, all four EACs evolved in terms of how they were designing and delivering provision. The evaluability assessment was based on their delivery plans during the study period. Any future evaluation should assess whether there have been any significant changes to delivery and refine evaluation plans as necessary to accommodate these.
• There are ongoing discussions about the future funding for EACs, which could influence how EACs deliver provision. As above, any evaluation will need to consider how this affects the recommendations set out in this report.
• EACs are only one element of broader efforts to tackle child poverty in these areas (including Child Poverty Pathfinder projects in Dundee and Glasgow and the Family Wellbeing Partnership in Clackmannanshire). As these wider initiatives also evolve, the contexts within which EACs deliver provision, and therefore the context for future evaluation, could change. For example, it may become increasingly challenging to unpick the impact of EACs on outcomes (i.e. attributing impact). This is further discussed in Chapter 3.

Due to the above, this evaluability assessment should not be considered an evaluation plan or protocol. The evaluation team strongly recommends that any future evaluation includes an initial scoping phase to further explore and refine the range of methods recommended by this evaluability assessment.

Report structure

The rest of this report is structured as follows:

Chapter 2: Theories of change and causal pathways provides information about the local-level EAC theory of change and core causal pathways that would form the focus of a future evaluation.

Chapter 3: Impact evaluation outlines current data collection within EACs and explores other options for measuring the intended outcomes, such as secondary datasets and primary data collection. It also provides an appraisal of the potential evaluation designs for assessing the impact of EACs.

Chapter 4: Process evaluation outlines the key factors that should be considered by a future process evaluation.

Chapter 5: Economic evaluation explores the requirements for completing an economic evaluation of the EACs.

Chapter 6: Conclusions draws together a final summary of this assessment.