Early learning and childcare trials discussion paper: analysis of responses

8. Approaches To Measurement of Trials

Question 7: What outcomes should we be measuring through this programme of trials?

8.1 Respondents were asked for their views on what a successful trial would look like and how it would be measured. Views were invited on what indicators will be needed to inform decisions on future delivery; what areas of children's and families' lives might the expansion of high quality, flexible ELC have a positive impact on; and how to measure the impact of a particular delivery model on providers and other stakeholders.

General views on trials and their evaluation
8.2 57 respondents addressed this question. Many provided general views on how the trials and their evaluations should be set up and executed. It was considered that the trials should be specific in what they aim to achieve in order that meaningful indicators can be devised and outcome measures established from the start. Although local initiatives were envisaged, many respondents called for evaluative material to be shared in order to promote learning more widely. Trials were recommended which encompass all sectors and settings. An electronic monitoring system was called for which could record evaluative data in addition to performing booking and accounts functions.

8.3 One broad recommendation was to use existing data collection mechanisms and build on these if possible in order to avoid unnecessary additional burden. The current inspection frameworks were amongst those mentioned in this regard, in addition to the adoption of GIRFEC principles and SHANARRI outcome measures. One local government respondent recommended aligning with Community Planning Partnership area outcomes. Specific references were made to the quality indicators in the Care Inspectorate's publication, "How well are we improving the lives of children and young people" which sets out a framework for measurement of impact including 22 quality indicators; Ferre Laever's Leuven Scale for measuring the impact of increased hours of ELC on children's wellbeing, engagement and involvement was also highlighted.

8.4 Respondents envisaged that evaluation would be based on both qualitative and quantitative data. Several identified approaches such as focus groups, case studies, self-evaluation and interviews as useful for generating information. Parents, staff, local authorities, providers and children were highlighted as potential research participants, with consultative methodologies advocated (such as Mosaic approach) to enable young children to provide views constructively.

8.5 Ten respondents identified assessment of longer-term outcomes as presenting a challenge to evaluation. A common view was that whilst aspects of process and short term outputs could be assessed in the time before roll-out to national implementation, there will not be time for longer-term impacts to emerge, nor time for experimentation and changes to original trial models to generate results:

"I think the true success of these trials will not be seen for many years. Whether this system is a success will not really be known until the children have grown up. If the rate of drop outs from school drops and the rates of success in education and employment go up only then will we really see if this system is working. The child's developmental progress can be measured to some extent but the true value of this system is whether it has a long term impact" (Registered Childminder).

Views on what should be measured
8.6 A wide range of views was provided on the outputs and outcomes to be measured in the trials. These are grouped by category in Table 8.1. The most frequently identified outcomes were child-focused and family/parent focused.

Table 8.1: Categories of outputs and outcomes recommended by respondents

Views on child-focused outcomes
8.7 Respondents across most sectors identified child-focused outcomes as integral to evaluation of trials. In particular they recommended outcomes related to children's developmental achievements; their wellbeing; and the quality of ELC provision received. Less frequently mentioned were outcomes relating to: positive attachments with parents; reduction of the attainment gap between children ( e.g. in expressive vocabulary); increased confidence; smooth transitions and pathways between settings; improved citizenship; and health outcomes such as reduction in obesity.

Views on family/parent-based outcomes
8.8 Eight sectors were represented by respondents recommending family/parent-based outcomes. Just under half of respondents who identified such outcomes called for family satisfaction with ELC provision to be measured.

8.9 Other less frequently identified outcomes were: positive parenting/confidence; flexibility in provision; family engagement and positive relationships with ELC staff; overall family wellbeing and quality of life; and parent choice over provision.

Views on economics/wider poverty outcomes
8.10 Respondents from eight sectors identified outcomes relating to economics and wider issues of poverty. Most recommended outcome measurement of impact of trials on the uptake of employment or training opportunities by parents. Others considered that outcomes should focus on decrease in levels of poverty and narrowing the economic gap between different areas. Four respondents suggested that outcomes relating to equality and equity issues should be established.

Views on workforce-focused outcomes
8.11 Respondents from seven sectors called for outcomes relating to the impact of the trials on the ELC workforce. A range of assessments was envisaged: quality of staff; qualifications of staff; availability of staff; ease of recruitment; retention of staff; morale of staff/feeling valued and satisfied; suitability of staffing arrangements including ratio of staff to children; empowerment of staff to make their own decisions.

Views on cost-related outcomes
8.12 Respondents from five sectors identified potential cost-related outcomes including: true cost of provision ( e.g. covering items such as additional staff training; expanding outdoor provision; providing meals and rest areas); financial viability of models of provision; impact on affordability to parents; and cost-effectiveness.

Views on process indicators
8.13 Seven sectors were represented amongst respondents who identified process indicators to include in the evaluation framework. Most frequently mentioned was measurement of impact of trials on take-up of hours, especially amongst the poorest families. Also identified were indicators relating to quality of partnership working; involvement of stakeholders in the design of the trials/co-production and engagement of different agencies; and the quality and availability of information on ELC trials for parents.

Views on physical infrastructure outcomes
8.14 Three local authority respondents, two private sector respondents and one individual raised specifically the importance of measuring suitability and quality of physical provision ( e.g. outdoor learning space; family learning area; dining and resting provision).

Views on community-based outcomes
8.15 Two voluntary organisations recommended that the integration between the ELC centre and staff and the wider community should be measured as a community-based outcome.

Implications for proposed trials
8.16 Respondents emphasised the need for trials to be clear in their aim in order for meaningful indicators to be devised and outcome measures established. Sharing of learning was recommended as a philosophy; with a wide range of settings and sectors to be tested.

8.17 Minimising bureaucracy was a general theme with respondents recommending building on existing data collection mechanisms wherever possible.

8.18 Respondents envisaged both quantitative and qualitative approaches to generating data, but from their recommendations on specific methods it is clear that qualitative data is expected to form the most significant aspect of evaluation, with innovation required to involve younger children in providing their views.

8.19 Child-focused outcomes were viewed as integral to the evaluation of the trials, with data to be collected on children's developmental achievements; wellbeing; and quality of ELC provision. Trials should, therefore, be set up with suitable evaluative frameworks which can capture such information, informed by research expertise and/or involving independent expert evaluators