Early learning and childcare expansion programme: evaluation strategy

4. Evaluating the intermediate outcomes

4.1. Introduction

4.1.1. As set out in the Introduction, to achieve its high-level outcomes, the ELC expansion programme needs to realise the six intermediate outcomes of providing accessible, flexible, affordable and high quality ELC; maximising take up of expanded ELC; and supporting parental confidence and capacity. The approach to the first five outcomes is discussed in the following section, while the sixth – parental confidence and capacity – is discussed in the later section on evaluation outcomes for children, parents and families as the aspects and measures drawn on are strongly linked to other parent and family outcomes.

4.2. Accessibility

4.2.1. The Theory of Change highlights that the intermediate outcome of accessibility for families is a key determinant of take up of ELC. Whether or not ELC is accessible for families includes aspects such as: whether parents know about the entitlement and how to apply, and ease of registration; whether ELC is conveniently geographically located (e.g. in relation to home or work; travel-related barriers); whether there are barriers in terms of the needs of the child (e.g. ASN, language barriers); the ability to combine childcare for other children of different ages (e.g. finding a place for children who are too young to be eligible for funded ELC, combining collecting children from different childcare providers).

4.2.2. To supplement existing data sources, new questions on childcare and ELC use and affordability were added to the Scottish Household Survey (SHS) from 2018 onwards.^[51] These questions provide a source of annual, population level data to be used as part of the evaluation. They cannot, however, be linked to individual child experiences or attendance at ELC. In particular, the SHS provides annual data about accessibility, flexibility, affordability and satisfaction with funded ELC that are not available from other sources. The wide-ranging nature of the SHS also means that the childcare data can be explored by a number of other characteristics (where the sample size is large enough to allow breakdowns) e.g. household type, household income, SIMD (Scottish Index of Multiple Deprivation). As with other households surveys, the SHS data collection was disrupted by the pandemic and there is a break in the time series.^[52]

4.2.3. Table 1 below summarises the measures and data sources that will be used to understand changes in the accessibility of funded ELC. These focus on accessibility in terms of knowledge of the entitlement, ease of applying and geographical accessibility. The new questions in the SHS will provide annual population level data on these aspects of accessibility (and flexibility).^[53] We will also explore whether it is feasible to develop any additional measures of the geographical accessibility of ELC settings, to complement the data from the SHS.

4.2.4. Scottish Government research on 'Parents' views and use of ELC',^[54] published in 2018, provided baseline data on patterns of ELC use prior to the roll out of the expansion. The research explored parents' perceptions of quality, flexibility, accessibility and affordability. This included examining whether parents of children with ASN were satisfied that they had access to ELC that suits their child's needs. It reported on findings from a survey of over 10,000 parents across Scotland, alongside focus groups and interviews with particular groups of interest.

4.2.5. A follow up parent survey was undertaken in April-May 2022 to provide more detailed information on parents' views and experiences of flexibility and accessibility of 1140 hours of ELC after August 2021. As in 2018, the follow up survey collected data on the experiences of parents of children with ASN, including whether their level of satisfaction with, and any difficulties experienced, accessing, funded ELC that meets their child's needs. Qualitative research with parents was also undertaken during 2021-22 to explore parents' decisions about and experiences of using expanded ELC, including in relation to accessibility and flexibility (see Section 5.7 for more detail).^[55] Findings from these strands of research will be incorporated into the evaluation.

Table 1: Evaluating accessibility of funded ELC: questions, measures and data sources

Evaluation questions	Headline indicators	Data source
How has the accessibility of funded ELC changed following the expansion in terms of a) knowledge of the entitlement, how to apply and ease of registration? b) travel-related barriers? Do parents, in particular for those living in the most deprived and in rural areas, perceive that ELC is becoming more accessible, and meeting their needs in terms of accessibility?	% of households who do not use funded ELC because: 'I don't know if my child is eligible'; 'I was not aware of the availability of funded childcare'; 'Don't know how to apply/find applying too difficult'	SHS annual
	% of households who do not use funded ELC because of: 'transport difficulties'	SHS, annual

4.3. Flexibility

4.3.1. Flexibility is identified in the Theory of Change as an important aspect of whether or not childcare is accessible and supports parents in work, training, or study. The Children and Young People (Scotland) Act 2014 states that education authorities should provide ELC for all eligible children during at least 38 weeks of every calendar year, and that it should be delivered in sessions of 10 hours or less.

4.3.2. The Early Learning and Childcare Statutory Guidance sets out more on the requirements of the flexibility that should be offered.^[56] Education authorities should ensure that families are able to choose from a range of types of setting, offering different patterns of provision and support across their local area. This should reflect local demand and circumstances and it is not expected that all options will be available in all areas and settings. The range of models and patterns of delivery should be informed by consultation.

4.3.3. The provision within the 2014 Act to consult with parents is designed to ensure that there is an element of local flexibility in the availability of funded ELC – in line with the needs of families living locally. While this will not result in all families receiving exactly the provision of funded ELC that they want, it should ensure there is a reasonable balance and mix of provision available locally so parents can access provision that works for their family. For this reason, to provide a rounded understanding of progress with provision of appropriately flexible funded ELC, both objective measures – e.g. of whether specific aspects of flexibility are in place – and measures of parents' experiences of flexibility are required.

4.3.4. Table 2 below summarises the measures and data sources that will be used to evaluate changes in the flexibility of funded ELC. Care Inspectorate data on whether ELC settings are open for extended hours and during school holidays will be used as the headline indicator. Data from the SHS will provide additional information on whether lack of flexibility in terms of opening hours and the type of childcare available are barriers to accessing ELC for parents.

Table 2: Evaluating flexibility of funded ELC: questions, measures and data sources

Evaluation questions	Headline indicators	Data source
How has the flexibility of funded ELC changed following the expansion?	% of services providing funded ELC open: extended hours / during school holidays - disaggregated by local authority	Care Inspectorate data, annual
	% of households who do not use funded ELC because: 'Lack of flexibility/choice in opening hours' and/or 'Lack of flexibility/choice in types of childcare available' and/or 'lack of provision in school holidays' decreases	SHS, annual

4.3.5. As with accessibility, qualitative and survey research with parents will provide additional data on parents' perceptions and experiences of flexibility of funded ELC before and after the expansion.

4.4. Affordability

4.4.1. The intermediate outcome of affordability in relation to the expansion of funded ELC focuses on affordability for parents. The economic costs and benefits of the expansion of funded ELC will be considered in the economic evaluation described in Section 6. The provision of funded ELC sits alongside wider UK support with childcare costs that parents in Scotland may be entitled to, including childcare elements of Tax Credits and Universal Credit and the Tax-Free Childcare scheme for working parents.^[57] Therefore, the expansion of funded ELC will likely be one factor in the overall affordability of childcare for many families.

4.4.2. Expanded funded ELC will support affordability for parents if it reduces their need to purchase additional hours of childcare to meet their requirements and any additional costs of ELC. This intermediate outcome is, therefore, primarily relevant to those families who would have previously paid for additional hours (on top of the funded 600 hours). The costs of childcare for other older or younger children who are not entitled to funded ELC is also important in relation to overall household costs.

4.4.3. Table 3 below summarises the headline measures that will be used to evaluate changes in the affordability of ELC and wider childcare. Data from the SHS will be used to assess changes in the affordability of ELC for parents.

4.4.4. As with accessibility and flexibility, qualitative and survey research with parents will provide additional data on their perceptions affordability of funded ELC before and after the expansion, as well as additional data on expenditure from the survey.

Table 3: Evaluating affordability of ELC: questions, measures and data sources

Evaluation questions	Headline indicators	Data source
How has the affordability of both funded and unfunded ELC for parents changed following the expansion? What has been the impact on the affordability of overall childcare costs for parents of ELC age children?	% of households with ELC age children who say they find it 'easy' or 'very easy' to afford childcare	SHS, annual
	% of households with ELC age children saying they have no childcare costs, 'childcare is free/funded'	SHS, annual

4.5. Quality

4.5.1. Quality of ELC was identified in the evaluability assessment as the key determinant of improvement in children's outcomes. We will, therefore, examine any changes in the quality of ELC in funded ELC settings.

4.5.2. A distinction can be made between structural and process aspects of quality. Process quality refers to the actual experiences in ELC settings, such as children's interaction with staff and their day to day activities. Structural quality refers to how ELC delivery is organised, such as leadership, the number and qualifications of staff, physical space and regulations. These structural aspects create the conditions for high quality care and support. A PHS evidence review highlighted a number of structural and process indicators of quality including: staff qualifications; an experienced, competent and confident workforce; good working conditions; and an age-appropriate curriculum.^[58]

4.5.3. The National Standard sets out what children and families should expect from their early learning experience in Scotland regardless of where they access their child's funded hours.^[59] It covers 10 criteria:

1. staffing, leadership and management

2. development of children's cognitive skills, health and wellbeing

3. physical environment

4. self-evaluation and improvement

5. parent and carer engagement and involvement in the life of the setting

6. inclusion

7. business sustainability

8. fair work practices

9. payment processes

10. food

4.5.4. High quality ELC not only provides nurturing care for children but also supports their learning and development from an early age. Because of this, both the Care Inspectorate and HMIE have statutory functions which provide them with a role in relation to inspection of some services. All nurseries, childminders and school age childcare services must be inspected by the Care Inspectorate, regardless of whether they provide funded ELC hours, while local authority and private and voluntary nursery settings that provide funded ELC hours can be inspected by both the Care Inspectorate and Her Majesty's Inspectorate of Education (HMIE). The Care Inspectorate's quality evaluations are currently used to underpin aspects of the National Standard, while the development of other aspects was informed by How Good is Our Early Learning and Childcare? the self-evaluation framework used by HM Inspectors of Education.

4.5.5. Table 5 summarises the measures and data sources that will be used to examine any changes in quality of ELC. Care Inspectorate data provide information on the quality of ELC services through analysis of grading data.

4.5.6. SSSC data on ELC staff qualifications will be used to provide information on this key measure of structural quality. As noted in Section 3.2, work is ongoing to improve the workforce data available, and other measures will be drawn on as available and relevant.

4.5.7. Quality of funded ELC provision in terms of whether it meets the needs of children with ASN and parental perceptions of, and satisfaction with, quality are also important aspects of quality provision. The SHS will provide information on parental satisfaction with the overall quality of funded ELC provision. As for the other intermediate outcomes, qualitative and survey research with parents will provide additional data on perceptions of quality of funded ELC and ELC provision for children with ASN before and after the expansion, including factors that are important when choosing a funded provider and, for parents whose child has ASN, level of satisfaction with whether funded ELC meets their child's needs (see Section 4.2 on Accessibility for more detail).

4.5.8. The Scottish Study of Early Learning and Childcare (SSELC) is also collecting observational data on the quality of ELC settings. The SSELC is the main vehicle within this strategy for obtaining evaluation data on child, parent and family outcomes. It is measuring outcomes for children and parents receiving the existing funded entitlement (600 hours) and comparing them with those who receive the increased funded entitlement (1140 hours) (see Section 5.7 for more detail). The study includes observational assessments of settings attended by sampled children. The obervations are being conducted in partnership with the Care Inspectorate, using environment rating scales that focus primarily on process quality. The observations will provide Scotland-level data which will allow exploration of the relationship between a measure of setting quality and children's outcomes at a national level.

Table 4: Evaluating quality of ELC: questions, measures and data sources

Evaluation questions	Measures	Data source
How has the quality of ELC in funded ELC settings changed following the expansion? Is there variation over time, by type of setting and geographically? To what extent are parents satisfied with the quality of funded provision?	% of daycare of children services providing funded ELC achieving good, very good or excellent in Care Inspectorate inspections - disaggregated by type of provider / SIMD of the setting	Care Inspectorate data, annual
	% of ELC staff with a qualification condition[60] by provider sector and job function	SSSC data, quarterly
	% of parents who use funded hours reporting that they are very or fairly satisfied with the overall quality of funded ELC - disaggregated by SIMD	SHS, annual

4.5.9. The Care Inspectorate assesses both structural and process quality in its inspections of all childcare providers. Until 2022, grades were given on the basis of four 'Quality Themes': care and support, environment, staffing, and management and leadership.

4.5.10. The Care Inspectorate suspended their normal inspection programme in March 2020 in response to Covid-19 protective measures, instead refocusing inspections on a new 'Key Question 5' ('How good is our care and support during the Covid-19 pandemic?'). Full thematic inspections were only carried out during this period settings based on a risk and intellegence basis. Routine quality inspections recommenced on a revised basis during autumn and winter 2020 but were suspended again – unless risk and intellegence indicated an inspection should be undertaken – between December 2021 and March 2022. This means there is a gap in full thematic inspection data on the quality of ELC services for the majority of 2020 and the start of 2022.

4.5.11. In March 2022, the Care Inspectorate launched an updated quality framework for daycare of children, childminding and school age childcare settings that focuses on children's wellbeing.^[61] From 1^st June 2022 five 'Key Questions' replaced the quality themes. Four of the Key Questions directly link to the previous four quality themes (see Table 4 below).^[62] The framework also incorporates elements of Key Question 5 developed at the start of the pandemic to assess how settings support children's health and wellbeing and Covid safety.

Table 5: Care Inspectorate new Key Questions and previous Quality Themes

New Key Questions	Previous Quality Themes
1: How good is our care, play and learning?	Quality of care and support
2: How good is our setting?	Quality of environment
3: How good is our leadership?	Quality of management and leadership
4: How good is our staff team?	Quality of staffing

4.5.12. Through inspection, HMIs from Education Scotland aim to:

• provide assurance and public accountability
• promote improvement; and
• inform the development of educational policy and practice

4.5.13. ELC settings are familiar with the Education Scotland quality improvement framework, How good is our early learning and childcare? and are continuing to use the framework and quality indicators (QIs) for their own self-evaluation and planning for improvement. Therefore, HMIs will continue to use this framework as the focus for inspections for session 2022-2023.^[63] Inspection reports will be considered as part of the wider evidence for the evaluation.

4.5.14. Some inspections in ELCs are undertaken by both the Care Inspectorate and Education Scotland. Where a shared inspection takes place, this will be carried out by inspectors from both the Care Inspectorate and Education Scotland. The evaluations of both organisations will be included in the inspection report and the final letter to parents.

4.6. Take up

4.6.1. The Theory of Change (p9) highlights that whether parents take up the additional available hours of funded ELC is a key short-term indicator of successful implementation of the expansion and provides an early indicator of the likely impact on child, parent and family outcomes. So it is important to understand how take up of ELC has changed with the expansion. To understand take up of funded ELC requires measuring the proportion of three and four year olds and eligible two year olds who are registered for funded ELC and the number of days and hours of funded ELC used (during school holidays and term-time), as well as the type and amount of other childcare used (e.g. 'substitutes').

4.6.2. Table 6 below summarises the measures and data sources that will be used to evaluate take up of funded ELC.

4.6.3. The ELC Census provides data on children registered for funded ELC by age and, where relevant, by eligibility criterion. The Census is currently completed by the centres that provide funded ELC in Scotland and the analysis possible is limited since it collects aggregated data on children registered for funded ELC, not individual-level data. The current Census data does not provide details on how many hours children are registered for and only very limited information on the characteristics of children registered for funded ELC (e.g. not SIMD or ethnicity).^[64]

4.6.4. Scottish Government Children and Families Analysis is currently taking forward an ELC Data Transformation Project^[65] that aims to improve the data available to monitor ELC take up rates and analyse these by children's characteristics. It will move the collection of ELC data from an aggregated collection to an individual level census, and include data on ELC delivered by childminders. The new ELC Child Census will have data on how funded provision is being used, which centres children are enrolled at and for how many hours during census week, and how many weeks across the year they are registered for. It will contain information on the characteristics of children registered for the funded entitlement. The new Child Census process is due to take place for the first time in May 2024, and the Staff Census in Sept 2024, with trials in winter 2023/24.

4.6.5. As set out previously, to monitor progress on the delivery of the expansion, the Improvement Service collect data on progress of delivery across local authorities (until at least 2022). This includes data on the number of two year olds, three to four year olds, and five year olds (deferrals) accessing funded ELC; and (from 2020) the range of hours being accessed. The Improvement Service data collection bridges the gap until the new Census is in place, providing important data on the number of children accessing funded ELC and the number accessing the full 1140 hours.

Table 6: Evaluating take up of funded ELC: questions, measures and data sources

Evaluation questions	Measures	Data source
How has the take up of funded ELC changed following the expansion? How does the take up of ELC vary by age of child and geographically?	No. of registrations and % of two year olds, and three and four year olds registered for funded ELC annually - at a national and local authority level No. children registered whose home language is not English / with ASN / assessed or declared disabled	Current ELC Census, annual (September – to 2023)
	No. of two year olds, and three and four year olds and five year olds (deferrals) accessing funded ELC - range of hours accessed and setting type (LA, PVI, childminder) in which most hours accessed	Improvement Service, termly
	No. and % of children accessing funded ELC annually - at a national and local authority level - no. of hours being accessed / how many weeks across the year characteristics of children, including sex, home SIMD, ethnicity, ASN	New ELC Child Census, annual (May – from 2024)

4.6.6. The annual SHS data also includes information on the types of childcare (formal and informal) used by parents with a child aged between two and five years old (who has not yet started school) and the number of hours used during term-time and school holidays; as well as some data on the number of hours of funded ELC parents used.^[66] The 2017 and 2022 parent surveys also collected information on whether respondents use funded ELC, the hours of funded ELC used and setting type, and whether parents used their full entitlement.^[67]

Take up of ELC among two year olds

4.6.7. To fully understand take up rates of funded ELC among two year olds, it would be necessary to know the population who are eligible for funded ELC at the point the number of children accessing a place is counted. This would then allow the calculation of a percentage that is comparable over time. This has not been possible as the Scottish Government does not have access to the full UK Government administrative data required to provide a definitive figure for the number of eligible two year olds in Scotland at any one time. This means that meaningful analysis of changes to numbers of 2 year olds registered for a place, as reported in the census each year, is limited.

4.6.8. We have modelled an estimated eligible population of 14,500 children each year, based on population projections published by the National Records of Scotland and the eligibility criteria for the two year old offer. While this figure is used when necessary to provide an illustration of the scale of the eligible population against the total two year old population, it cannot be used accurately to estimate actual take up rates. We know the actual eligible population will change each year (and within years) as patterns of benefits take up and care experience will fluctuate.

4.6.9. However, having access to data on recipients of UK-level qualifying benefits (which we estimate make up around 90% of the eligible population), will allow an accurate measurement of the eligible two year old population.

4.6.10. We anticipate access to data for use by local authorities during 2022-23. UK Regulations that create a new 'public service delivery' objective under the Digital Economy Act 2017 were laid in Parliament on 15 July 2022. This will provide valuable information for local authorities to contact eligible families in their local area and allow them to more accurately estimate their eligible population. We are also working across Scottish Government to gain access to UK-level benefits data, including to allow us to calculate the proportion of eligible two year olds that are registered for funded ELC at the time of the annual ELC census.