School age childcare delivery framework

5. How will we measure success?

Embedding monitoring and evaluation in the programme development

Monitoring and evaluation is a core component of our school age childcare programme, with ongoing data gathering and shared learning built into projects. Alongside the policy design of our new system, we will also develop and publish an evaluation framework.

This will take account of the ELC Expansion Programme Evaluation Strategy and other relevant evaluations, particularly those with a focus on how we best support targeted families. This will include understanding links with the Tackling Child Poverty programme, Employability services, the Whole Family Wellbeing Funding and the Social Innovation Partnership projects. It will also link to learning from place-based initiatives such as the Regeneration Capital Grant Fund.

Programme logic model – developing a theory of change

The high-level programme logic model at Figure 8 will inform our evaluation, data development and collection plans, and provide the structure for our evaluation framework. The logic model sets out three high level outcomes our system of school age childcare is expected to contribute to – aligning with the key policy drivers of tackling child poverty and improving children’s outcomes. It underlines that, to contribute to these high-level outcomes, a number of intermediate outcomes must be achieved. In the initial stages of developing our evaluation framework we will work with stakeholders to develop a more detailed national level theory of change. This will set out how our planned actions to develop the system of school age childcare linked to intermediate and high-level outcomes.

Figure 8: School Age Childcare high level logic model

Inputs:

Appropriate funding and governance is provided at national and local level

Outputs:

A system of childcare, food and activities for school age children in targeted families is created across Scotland providing care before and after school all year round

Intermediate Outcomes:

• Targeted children take up the offer of available food as part of funded services
• Accessibility of school age childcare increases for targeted families
• School age childcare is more available to families who could not otherwise afford it
• Participation in a range of activities outside of school time is increased for targeted children
• Social interactions and social connections are improved for targeted children
• For targeted families, time available to parents is increased when their children are cared for

High-level Outcomes:

• Targeted groups of children are healthier and have increased wellbeing
• The poverty-related outcomes gap narrows
• More parents from targeted families take up, sustain and/or increase their hours of work, training or study

The policy and evaluation context

Within our Draft Framework for Out of School Care (2019), we summarised the existing evidence relating to childcare and the role of school age childcare in tackling poverty. This highlighted research showing that childcare can be a significant cost for households with children. A lack of affordable and flexible childcare can also limit opportunities for paid employment. Not being able to access flexible childcare may mean parents having to choose low-quality part-time jobs, take a role they are overqualified for or leave work altogether. Recent research with parents with lived experience of poverty commissioned by the Joseph Rowntree Foundation (2022) confirms these findings and in particular that the cost, inflexibility and/or inaccessibility of childcare is a significant factor in families on low incomes in Scotland and experiencing hardship. In December 2022, the Scottish Poverty and Inequality Research Unit (SPIRU) published a Rapid Review of Evidence on the Impact of Childcare on Parental Poverty, Employment and Household Costs in Low-income Families. This provides an overview of the existing evidence on the immediate and longer-term impacts of investing in childcare, as well as some of the barriers to achieving benefits.

Our Strategic Childcare Plan (2022) set out some areas where further research and data is needed and underlines how ongoing monitoring and evaluation work will develop and inform this evidence base. This work will also be an important part of the evaluation framework. As outlined in the Testing the Change section, we have evaluated our Access to Childcare Fund projects and the summer holiday programme. In line with our Programme logic model, these evaluations explore the extent to which projects have contributed to outcomes such as: making services more accessible, flexible and affordable; providing access to food; reducing household costs. This includes understanding the lived experience of families who use services. The evaluation research captured key learning on service- level models such as: what has and has not worked well; approaches taken to overcoming access and participation barriers; best practice in cross-sector working and community collaboration.

Findings and learning from these and previous evaluations will feed into testing and building a system of school age childcare. We are developing a monitoring and evaluation framework for the Early Adopter Community projects. This will involve collecting data to understand whether the community-level work is contributing to improved outcomes for children and families, as well as to learn what does and does not work well, for whom and why. Our aims, as we test the changes needed at a national level to support community-level systems of school age childcare, are to understand:

• who can benefit most from a school age childcare offer,
• what form that offer should take, to deliver most impact for target families, and
• how that can best be implemented to deliver change at a local level.

It is important to note that building the system of school age childcare involves changing a complex system and taking an iterative and locally variable approach. This means evaluating the outcomes and impact of the policy will also be complex, and careful thought will be needed in selecting appropriate and practical evaluation approaches and methods.

Data development

As we progress through the tests of change and system build action areas, we will gather data and learning to inform each phase of the programme. We will build a body of evidence to synthesise learning and produce recommendations to inform the design of a system of school age childcare in Scotland. The data gathered will also be used to monitor phases of implementation of the new system. It is envisaged that data will be obtained from a variety of sources, including:

• Care Inspectorate data on regulated school age childcare,
• a range of household/parent survey data to understand changes in e.g. living costs, childcare costs, economic activity, and
• management information showing uptake of provision through projects, alongside data related to workforce and infrastructure.

Available data sources such as the Scottish Household Survey will be utilised alongside new data management information, surveys, or commissioned research to address any gaps or limitations.

Evaluation Phases

A number of phases are planned for the data development and evaluation work.

Scoping and exploratory work (2022 – 2023):

• Completing and drawing together learning
• Working with Phase 1 Early Adopting Communities
• National level data scoping and development work
• Development of a national level theory of change
• Analysis to understand economic impacts of SACC

Evidence to underpin scaling up (2023 - 2025):

• Monitoring and evaluation of Early Adopter Community projects
• Drawing together data, research, evaluation and engagement evidence
• Evaluability Assessment of SACC offer for low income families
• Development of a monitoring and evaluation framework for SACC

National monitoring and evaluation (2026 onwards):

• Ongoing analysis and reporting of SACC monitoring data against national monitoring framework
• National-level evaluation of new targeted SACC offer for low income families