Early Adopter Communities: process evaluation

Chapter 1. Introduction

This report presents the findings of an early process evaluation for the Early Adopter Communities (EACs).

Policy context and background

EACs are a key element of the Scottish Government's School Age Childcare Transformational Change Programme. The Programme aims to design and build a system of school age childcare that provides care all year round, and supports parents and carers –- particularly on low incomes –- to have secure and stable employment. The provision is targeted at the six priority groups identified by Scotland's Tackling Child Poverty Delivery Plan (Best Start, Bright Futures) as being most at risk of experiencing child poverty: lone parent families; families with a disabled adult or child; larger families (three or more children); minority ethnic families; families with a child under one year old; and families where the mother is under 25 years of age.

The school age childcare system includes different types of provision, such as breakfast clubs, after school clubs, holiday clubs and childminders. It is delivered by regulated providers, including registered childminders, or as part of organised activities (e.g. sports clubs or cultural activities) which are currently not required to be registered with the Care Inspectorate. In addition, many families use friends and family or at-home services such as nannies.

The school age childcare ambition sits alongside the Scottish Government's wider commitment to eradicate child poverty. In Scotland's Tackling Child Poverty Delivery Plan, the Scottish Government recognises that affordable and accessible childcare is needed to enable parents and carers (from hereon in referred to as 'parents' for brevity) to gain access to training, study, or secure, stable employment and to increase household income. In the Strategic Early Learning and School Age Childcare Plan, the Scottish Government also recognises that high quality childcare can positively impact children's development and family wellbeing.

More broadly, the Tackling Child Poverty Delivery Plan recognises the need to work differently to ensure that support systems work for those who need them most. It commits to trialling and evaluating new approaches, adapting ways of working, and monitoring how changes are impacting on priority families (lone parent families; families with a disabled adult or child; families with three or more children; minority ethnic families; families with a child under one; and families with a mother aged under 25). EACs are part of this landscape of innovative approaches to service delivery, alongside initiatives such as the new Fairer Futures Partnerships which build on existing place-based partnerships in Dundee and Glasgow (originally known, and referred to here, as the Pathfinders) and the Family Wellbeing Partnership in Clackmannanshire, the Whole Family Wellbeing Fund, and the No One Left Behind employability approach.

Four EACs were established in October 2022 in local areas within Clackmannanshire, Dundee, Glasgow and Inverclyde. The EACs have been supported by around £3.5 million in Scottish Government funding over two phases up to March 2024 (in this report, phase one refers to funding period October 2022 to March 2023 and phase two refers to funding period April 2023 to March 2024). A further investment of £16 million will be made over 2024/25 and 2025/26 and two new EACs will be established in Fife and Shetland. The EACs are intended to provide an opportunity to test and refine what an effective school age childcare system looks like in different communities. They built on learning from the Access to Childcare Fund, which tested service-level approaches to childcare delivery. More detail on the four EACs is included in Chapter 2. This report uses 'EACs' and 'EAC projects' to describe the investment in local communities.

Overview of the EAC evaluation study

The Scottish Government commissioned Ipsos Scotland to carry out work with the EACs to inform its approach to longer term monitoring and evaluation. This work comprised four key stages: 1) Developing theories of change at both a community level (for each EAC) and a national level; 2) Developing a monitoring and evaluation framework (M&E framework) clearly setting out what data is currently, and could be, collected by EACs to support effective evaluation; 3) Conducting a process evaluation on early implementation and delivery across current EACs; and 4) Producing a preliminary evaluability assessment.

This process evaluation report delivers on the third stage of the work, although it also draws on some evidence collected as part of the other stages. The theories of change are included in Appendix A and the M&E framework can be found as a supporting file. The evaluability assessment is a separate output, which can be read in conjunction with this report.

Aims and objectives of the process evaluation

The overarching aim of this process evaluation is to identify what worked well or less well in the early set-up and implementation of the four EACs, to inform ongoing design and delivery. This is assessed by looking at the extent to which activities identified in the EAC local level theory of change have been successfully implemented. This includes systems-level activities such as governance and partnership working; as well as activities relating to the delivery of services to families, such as co-design, referrals, application processes, and provision of subsidised childcare, food, transport and further family support where appropriate.

The following key evaluation questions guided the evaluation (note that a fuller set of evaluation questions can be found in Appendix B):

Delivery of services to families:

1. How are EACs implementing services for families?

2. How effective are EAC processes in reaching/engaging intended priority groups?

3. What are the enablers that have facilitated delivery of services to families and the challenges that have been experienced. How have challenges been overcome?

4. To what extent does provision meet the needs of local families?

5. What are the key differences across EACs?

Systems change delivery:

6. How are EACs implementing activities to achieve systems change?

7. To what extent are systems change activities being implemented as intended?

8. What are the views and experiences of EAC staff and stakeholders on the success/effectiveness of activities to achieve systems change?

9. What are the enablers that have facilitated system change and the challenges that have been experienced, and how have challenges been overcome?

10. What are the key differences across EACs?

11. What is needed for future scalability of the EACs?

Methods used to inform the process evaluation

The evaluation research has taken an iterative approach, whereby previous stages have informed later stages, including the development of this report.

Scoping phase

The scoping phase of this research began with an inception meeting with the Scottish Government in August 2023 to discuss policy updates and developments. This was followed by an initial desk review of project documentation, including EAC proposals and progress update reports. Finally, in September and October 2023, an initial video interview with each of the EAC projects leads was conducted to confirm the evaluation team's understanding of each EAC and fill any gaps in knowledge or understanding.

Theory of change development

Two overarching theories of change for the EACs were developed as part of this work:

• The national-level theory of change – this maps out Scottish Government and national stakeholder actions required to deliver the policy, and how these are expected to lead to intended policy outcomes.
• The local-level theory of change – this includes an overview of core activities that are expected to take place within each EAC, as well as the common intended outcomes at a local level.

In addition to these, each EAC area was supported to develop a community-specific theory of change to illustrate how each EAC was set up and how it was expected to lead to intended outcomes, within the local context of each community.

Once all these were agreed, the evaluation team produced an accompanying narrative for the overarching local-level theory of change (Appendix C). This included identifying anticipated causal pathways and mapping these against existing evidence.

Monitoring and Evaluation Framework development

The M&E framework was developed to provide an overview of what data is currently available in each EAC in relation to the overarching local-level theory of change. The evaluation team drafted an initial M&E framework based on information from the scoping phase, the theory of change workshops, and guidance in the Child Poverty Monitoring and Evaluation Policy Evaluation Framework.

To inform the next draft, the evaluation team conducted interviews with project leads and other EAC staff with oversight of monitoring and data collection in each EAC. These discussions were conducted virtually, during December 2023 and January 2024. Similar interviews were conducted with prospective EACs (Fife and Shetland) and some Access to Childcare Fund projects (St Mirin's, Indigo, Stepping Stones), to check that the framework could be applied more widely outside of the current four EACs.

Evaluability assessment

While the evaluability assessment was the fourth stage of the evaluation study, findings from interviews with project leads during this element of the project and documents shared on application and feedback processes also helped to inform the process evaluation. More information is available in the separate evaluability assessment report.

Process evaluation

This early process evaluation was qualitative in nature. This was considered to be the most appropriate design to meet the objectives of the research, as it provides an in-depth understanding of experiences of early set-up and delivery of the EACs. The aim of qualitative research is to identify as much diversity of experience as possible, rather than attempting to achieve a sample that is statistically representative of the wider population. Estimates of prevalence based on qualitative data are therefore inappropriate and this report avoids quantifying language, such as 'most' or 'a few' when discussing findings.

The process evaluation included individual or paired in-depth interviews with: each EAC project lead; EAC partners/stakeholders; and parents and children using each project. The evaluation team also conducted a review of the most recent monitoring reports.

Sampling and recruitment

The research team interviewed:

• Project lead(s) from each EAC (five interviews including six participants). Introductions between the research team and project leads were facilitated by the Scottish Government. Interviews were carried out in March 2024.
• Stakeholders involved in planning, delivery, or referral (five in Clackmannanshire, including one paired interview, and four in each of Dundee, Glasgow and Inverclyde). Stakeholders were suggested by project leads. The research team aimed to interview participants representing a range of organisations and roles. These included: childcare service managers, headteachers, and local authority partners. Interviews were carried out in March 2024.

Parents and children who attended each project. These were recruited with the help of project leads and/or family support workers. Project leads were briefed by Ipsos and provided with written information (Appendix D) to share with parents, inviting them to take part. Participants either contacted the research team to arrange an interview or the project lead organised this on their behalf. The aim was to speak to a range of families across the priority groups. Interviews took place in March and May 2024. Depending on the age, ability, and preferences of children who had attended the project and the preferences of their parent, the research team either spoke to the children on their own; the parent on their own; or conducted a joint interview with both parent and children. In total, the research team carried out 20 family interviews, covering the experiences of 22 parents and 27 children (between the ages of five to 11) who had attended an EAC. Tables 1.1, 1.2 and 1.3 show the profile of participating families.

Note that parents self-identified their and their child's ethnicity. In this report, 'minority ethnic' includes white minorities (i.e. where English is a second language). Relatively low numbers of families with a child under one year old, and where the mother is under 25 years old, may be due to the focus on school aged children.

Review of monitoring reports

Projects were asked to submit quarterly monitoring reports to the Scottish Government as part of their funding agreement. The reports covered topics such as: activities undertaken, number of families supported, and impact of provision. These documents were shared with the research team for analysis and to inform the development of research materials.

Data collection and analysis

Discussion guides were developed for each audience to ensure all relevant issues were covered in interviews (see Appendix E). Individual discussions were shaped by the processes outlined in the local-level theory of change and M&E framework. Interviews were conducted either in person, by telephone or by video call (depending on participants' preferences and availability). Families were given £35 (as a voucher or bank transfer) to thank them for their time. All interviews were facilitated by members of the research team and were recorded (and some transcribed) for subsequent analysis.

Three core members of the Ipsos research team, who had carried out fieldwork in the EACs, were involved in data analysis. The following processes were used:

• Each researcher independently coded data from their interviews. Data were summarised into thematic matrices (in Excel) in line with the evaluation questions.
• The team held interim and final analysis meetings to discuss emerging findings and to refine findings. Differences in interpretation were discussed and consensus reached among team members.
• Full data from the thematic matrices were reviewed and analysed systematically to identify the full range of views and experiences under each theme and any differences between different sub-groups.
• Team members reviewed each section of this report.

This research was carried out in accordance with the requirements of the international quality standard for Market Research, ISO 20252.

Ethics approach

The evaluation team carefully considered ethical risks for engaging parents and children. Both Scottish Government and Ipsos ethics reviews were completed for the study, and the researchers worked in line with the Ipsos disclosure policy. Researchers gathered informed consent from all participants prior to data collection, emphasising the voluntary nature of their participation. This involved providing information about the purpose, methods and intended uses of the research and what taking part would involve. Participants were able to discuss their accessibility needs with the research team. To minimise burden and barriers to participation, fieldwork was conducted at a convenient time for participants and if it was not possible to cover everything, participants could share further thoughts via email.

Scope and limitations

When reading this report, the following considerations should be kept in mind.

• Each EAC has a unique childcare and family support offer informed by their local context, existing provision, partnerships, and more. Therefore, what has worked well in one EAC is not necessarily transferable to another.
• As detailed in the EAC evaluability assessment, process evaluations typically cover implementation, mechanisms of impact (how the intervention brings about change) and context. However, as this is an early process evaluation and there is no related impact evaluation on which to draw, the focus is on implementation, with analysis of context where possible. This is reflected in the evaluation questions used to guide the evaluation, detailed above. While the evaluation assesses the extent to which implementation has happened as intended, and considers elements which are working well and less well, it was not within scope to make any definitive judgements of success or to consider whether the approaches taken were the most appropriate. In relation to the former, this was primarily due to that fact that, although there are clear outcomes for the EACs which could be assessed in an impact evaluation, measures of success for implementation of activities are less well defined.
• This report comments on early indications of outcomes in the theories of change where these were noted by participants. However, this was not the focus of the present evaluation and further work will be needed to assess impact of the EACs. It is important to note that early impacts commented on cannot be solely attributed to EACs, given they are only one element of local efforts to tackle child poverty.
• The evaluation relied on the EAC project leads and providers to recruit families to take part in in-depth interviews. While guidance was provided on recruiting families with a range of characteristics and levels of engagement, it is possible that those who agreed to take part were those who were more engaged with and/or more positive about the EACs.
• This research did not include families who were eligible for EAC projects but did not take the offer up, although barriers to engagement were discussed by EAC staff and partners. While this does not negatively affect the quality of the data, there may be further barriers to participation that the evaluation could not identify.

Report structure and conventions

The report includes the terms 'stakeholder' and 'partner' to refer to EAC partners and providers. Generally, 'stakeholder' is used as a catchall for all professional participants and partners is used where they were more directly involved in childcare provision. However, to minimise the risk of quotes being identifiable, the term 'stakeholder' has been more often used in quote attributions, regardless of the participant's role.

The remainder of this report is structured as follows:

Chapter 2: Overview of EAC provision provides an overview of the childcare and family support provided for families in each EAC.

Chapter 3: Targeting and reach considersthe extent to which EACs are reaching target families.

Chapter 4: Meeting the needs of families explores the extent to which EAC provision is meeting the needs of families.

Chapter 5: Implementation of systems change explores activities undertaken to achieve systems change, including enablers, challenges and early indicators of success.

Chapter 6: Conclusions draws together a final summary of this process evaluation.