Access to Childcare Fund: phase 2 - evaluation report

3. Monitoring, reflection and change over time

Summary:

• Project staff had varying levels of experience in relation to formal monitoring required by the Scottish Government. However, all projects were responsive to the needs of, and feedback from, families.
• Monitoring had benefited projects in various ways, including: helping to identify and reach families; identifying ways to improve approaches to delivery; learning from and strengthening relationships with partners; and keeping projects “on track” in relation to their intended aims.
• However, challenges included: limited knowledge of evaluation; lack of staff capacity; low parent response rates; and parents feeling uncomfortable sharing personal information.
• Features that supported monitoring and evaluation included: support around identifying and measuring outcomes; having a peer network; and strong relationships between Scottish Government and projects.

This chapter will explore approaches to monitoring and reflection, including learnings to inform future planning and delivery of SACC.

The Fund had an emphasis on monitoring to facilitate the testing of different models of SACC and to help projects learn what works and adapt accordingly to make improvements. Projects were provided with reporting templates and were required to report quarterly to the Scottish Government on progress. Project staff had varying levels of experience in relation to formal monitoring, and this is reflected in data in the reports provided. However, it was clear that all projects were responsive to the needs of, and feedback from, families.

Information contained within monitoring reports varied. Some projects collected qualitative and quantitative information on the impact of provision from parents, children, and staff/partners at various points in time and produced visual outputs. This also included the use of systems to record support families received and resulting outcomes. Other projects focused more on recording the characteristics of families supported by the Fund.

A variety of methods were used to collect data from different audiences:

• From parents: via questionnaires (generally at the beginning or end of a term), at initial referral meetings with project staff to understand their needs, at (often informal) catch-ups with project staff.
• From children: by staff asking for feedback, through photos or videos.
• From staff: observations on the needs of families.

Benefits of monitoring

There were various ways in which project leads said ACF monitoring had benefitted their project. These included:

• Identifying and reaching families who might benefit from support:

  “…before the project, we would never in a million years have asked families lots of personal details like household incomes and things like that. So, that has helped us identify families.” (Project lead)

• Identifying ways to improve their approach to delivery:

  “[Reviewing our data] helps us to then sit down with [staff member] and our other service managers and say, ‘okay, what are we going to do differently? How are we going to change?’” (Project lead)

• Learning from and strengthening relationships with partners:

  “We have never really asked our partners for feedback before, so it was quite interesting to say ‘actually, how are we doing? What feedback have you had?’…we can have different conversations and build relationships with those partners and families and children too.” (Project lead)

• Making sure projects were “on track” in relation to their aims:

  “It has kept you on track, you know exactly where you…what you're doing, where you want to be, and how you're going to get there...” (Project lead)

Barriers and learnings

Project leads discussed challenges they had experienced when undertaking monitoring and reporting, and how they had/would address these, including:

• Limited knowledge of how to evaluate projects, including on how to use specific tools such as a driver diagram. Training provided by Evaluation Support Scotland was described as helpful, and there was a desire for more support in this area.
• Having staff capacity to spend time collecting and analysing data. There were comments that monitoring and reporting requirements were onerous and led to management time not being charged for. One project would have hired a member of staff for administration had they known the work involved. However, one project noted that having ACF funding had reduced the time they needed to spend on applying for funding from other sources.
• Low response rate from parents to requests for feedback. One project changed the platform they used to distribute feedback forms to make it easier to complete, as it allowed access from different devices such as mobile phones and tablets.
• Parents feeling uncomfortable sharing personal information such as household income. Strong relationships between families and staff were felt to help overcome this.

More generally, it was noted that, because projects support families in so many different ways, and often go ‘above and beyond’, it can be challenging for staff to reflect on everything they do and communicate this to others:

“Initially you thought this is obvious, obvious to you… because you forget how much is second nature, because that is your life.” (Project lead)

When it comes to monitoring and evaluation best practice, several features were identified by project leads and stakeholders as important:

• Identifying clear intended outcomes and having support around this, recognising staff’s varying level of experience:

  “This is the first time I’ve been to this stage of monitoring and reporting, so having the Evaluation Scotland sessions was really helpful.” (Project lead)

• Understanding how to measure impact, for example by identifying short, medium- and long-term outcomes and taking baseline measures.
• A peer network to share learnings. It was clear that staff valued having the opportunity to connect and share experiences with staff from the other projects as part of peer network meetings.
• Strong relationships between projects and the Scottish Government to help understand expectations and experiences of delivering funding.

Changes to SACC delivery

Projects’ key aims had generally not changed over the course of involvement in the Fund. Instead, some project leads described how delivery processes had “evolved” over time in response to their practical experience of delivery or in response to feedback from families. This is explored in more detail in later chapters, but examples include changing times of activities to reflect parental preference, updating booking systems to improve accuracy, or adapting referral processes to better reach target families.

Some project leads also commented on changes they would ideally like to make to future delivery. These included: employing family support staff to advise on welfare and other sources of support, increasing staff numbers to make SACC available to more families, and extending reach into rural areas.

Project leads commented more widely on their aims across the lifetime of the Fund. Where it had not been possible for projects to fully meet some of their aims, this tended to relate to external factors such as the labour market, the impact of COVID-19 (including longer-term impacts e.g., on the labour market), or transport barriers. These factors are discussed in more detail in later chapters where they related to phase 2 of the Fund.

Changes to organisations

Project leads were also asked what impact the Fund had had on their organisation more widely. These included having the time and resources to test new ideas and approaches to SACC delivery:

“…there was lots of little things that had been hanging around for a long time and this funding gave us the people to actually find out the hard facts and then test out assumptions and what we thought might work.” (Project lead)

There were also examples of projects being able to employ more staff or improve staff pay and conditions. However, there were concerns around these positive impacts being unsustainable without funding:

“[The funding] was used to actually improve the staff pay [and] conditions, and I think especially before COVID, I think childcare staff in general were starting to feel quite undervalued and taken for granted.” (Project lead)

“We have staff employed specifically for after school clubs…without the funding they wouldn't be employed, that is a big worry for the future.” (Project lead)

Projects also commented on the ways in which the Fund had helped to enhance their reputation and profile in the community. For example, one project lead felt that their family support worker role enabled them to be more consistent with their work within the community and to strengthen relationships. Another highlighted the opportunities afforded by the Fund to demonstrate the skill and pedagogy behind what SACC providers can offer:

“There are cultural challenges between people’s views of school age childcare. […] I think the funding has certainly helped them to understand better what we actually do and that there is clear knowledge and pedagogy and practice and thinking behind the work that we do.” (Project lead)