Summer holiday food, activities and childcare programme: evaluation report - summer 2022

1. Introduction and methods

About the 2022 Summer Programme

The overall aim of the 2022 Summer Programme was to ensure that eligible children and young people (CYP), and their families benefitted from a range of activities, food and childcare provision that met their needs during the school summer holidays. The Scottish Government provided local authorities with £10 million in additional funding to target provision at CYP (aged 5 to 14) from low-income families. Eighty-five per cent of the funding was to be targeted at the groups identified in the Tackling Child Poverty Delivery plan (TCPDP), with the remaining 15% available to be spent on other groups depending on local priorities. The priority target groups were:

• Lone parent families;
• Ethnic minority families;
• Families with a disabled adult or child;
• Families with a young mother (under 25 years old)
• Families with a child under 1 year old;
• Larger families (3+ children).

The 2022 Summer Programme built on learning from Get into Summer 2021, which aimed to mitigate negative impacts of the Covid-19 pandemic. The evaluation of the 2021 Programme showed that it had a positive impact on children and families in terms of providing opportunities to have fun, spend time outdoors and do physical activity and connect with peers, among other benefits. Range and choice of activities, food provision, co-creation, partnership working, and reducing barriers to participation were all key to its success.

Unlike the 2021 Programme, the 2022 Programme also had a focus on the provision of childcare. Its guiding principles were: partnership working, co-creation, and coordination and integration. It was intended that providers would deliver a range of activities, childcare and food options to give families choice and flexibility to meet their needs. This would be achieved through strong local partnerships to help reach families in an inclusive and non-stigmatising way, helping to overcome barriers to participation, including those identified in the 2021 evaluation. The programme was intended to include effective communication, appropriate food and transport provision, and referrals to wider support services. It was recognised that focusing on local delivery would help reach eligible families, particularly in the most deprived areas.

A logic model for the Programme, setting out the intended inputs, outputs and outcomes, was developed to help steer this evaluation by Ipsos in conjunction with the Scottish Government and wider programme stakeholders. This is included as Appendix A.

Evaluation aims and questions

The Scottish Government commissioned Ipsos to evaluate the 2022 Programme. The overall aim of the evaluation was to assess whether the Programme met its intended outcomes and met the needs of the CYP and families it intended to support, as well as identifying what has worked well or what could be improved in terms of delivery. The findings of the evaluation will be used to inform and shape the year-round school age childcare commitment and the accompanying monitoring and evaluation approach.

More specifically, the evaluation aimed to answer the following research questions:

1. How has the 2022 Summer Programme supported CYP and families from low-income households?

• How was the funding used? What type of activities and childcare models were delivered? Did it provide opportunities to play, socialise, be active, be outdoors? Did it enable parents to work, train or study? What was the attendance at these activities/childcare sessions?
• Was 85% of the funding used to support school age children (5-14-year-olds) from low-income households in the priority family groups? How was the remaining 15% used?

2. What has been the experience of service providers and CYP and families?

• How do CYP and families and service providers describe their experience of the 2022 Summer Programme?
• Was the local offer shaped around what CYP said they want and need? Did they feel they had choice/flexibility? Were there any barriers to attendance?
• How were activities targeted and promoted to priority family groups? Were referral systems used or direct communications? Were any groups hard to reach?
• Were there any steps taken to mitigate potential sources of stigma? What were these? Was stigma experienced by families?
• How did the provision impact on the poverty and food insecurity experienced by CYP and families?
• What were the opportunities and challenges in the joint working between local authorities and local delivery partners?

3. What are the key lessons from delivery of the 2022 Summer Programme?

• What worked well? What could be improved in the delivery of future summer programmes?
• What were the opportunities and challenges in engaging and delivering activities to the priority groups? Any future opportunities for scaling up?

Methods

The evaluation was primarily qualitative in nature. Given the available time and resources, and the key aims of identifying what worked well and less well from the perspectives of children and families who attended Summer Programme activities and providers involved in their design and delivery, this was felt to be the most appropriate design. It included three main elements:

• Stage 1: in-depth interviews/paired interviews with parents and CYP in case study areas (conducted first to minimise loss of recall among CYP);
• Stage 2: in-depth interviews with case study local authority representatives and provider organisation representatives, including an online workshop with stakeholders outside the case study areas to ensure that the case study findings were broadly aligned with those of other local authorities;
• Stage 3: Review of quantitative monitoring data on attendance and activity types submitted to the Scottish Government from local authorities.

The study was carried out in accordance with the requirements of the international quality standard for Market Research, ISO 20252.

Sampling and recruitment

CYP and parents

Parents and CYP were recruited via providers in a sub-sample of eight local authority case study areas (two or three programmes were selected in each area). These local authorities, and the specific programmes focused on within each, were selected to try to ensure diversity in terms of activities offered, key target groups, geography, and level of deprivation (using the Scottish Index of Multiple Deprivation).

Participants were recruited with the help of programme organisers, who were briefed over the phone and provided with written information (Appendix B) by Ipsos to share with attendees, inviting them to take part. Individual families then contacted the research team directly, who checked eligibility and arranged a suitable time for an interview.

Depending on the age and preferences of CYP who had participated in the programme and their parent or carer, the research team either spoke to the child(ren) or young person on their own (one interview, young person aged 14); the parent on their own (six individual interviews and two mini groups with three or four participants); or conducted a joint interview with both (31 interviews). In total there were 44 interviews/groups, covering the experiences of 71 CYP who had taken part in the Programme. Table 1.1 shows the profile of participating families, in terms of the age and gender of the CYP, and whether they were from one of a number of key target groups.

Providers

The research team also interviewed 44 providers involved in the planning or delivery of the Programme in the case study areas. This included 26 local authority representatives and 18 partner organisation representatives. Participants were given the option to take part individually (20 interviews), in pairs (five interviews) or in group discussions with colleagues (four discussions of between three and five participants).

Local authority workshop

Once fieldwork within the case study areas was complete, a representative from every local authority was invited to share their experiences with the Summer Programme as part of a workshop. The main purpose of the workshop was to check whether the interim findings from case study areas were broadly aligned with experiences of the funding in other local authorities and to ensure nothing significant was missing from the final evaluation. Participants were also asked about their views on the funding model and how this might work best for any future provision. The workshop took place on 13th December 2022 with 32 participants from 20 different local authorities (it should be noted that 5 of these were case study areas).

Review of monitoring reports

All local authorities were asked to submit quantitative monitoring data on attendance and activity types to the Scottish Government. Data from 28 out of 32 local authorities was submitted and suitable for analysis. The data was cleaned and combined into one dataset and then some summary statistics were produced (totals and ranges for key variables across all local authorities).

It should be noted that, due to some data quality issues (inconsistencies in counting, interpretation, and missing data) only key figures with data provided from most local authorities have been included in this report.

Data collection and analysis

Discussion guides were developed to ensure all relevant issues were covered in interviews (see Appendix C). In-depth interviews were conducted either by telephone or video call (depending on participants' preferences) between August and October 2022. Within the evaluation timescales and budget, it was possible to offer families in some local authority areas (that were easily accessible to the research team) the option to participate in-person. Nine families took part in face-to-face interviews/mini groups. Families were given £35 to thank them for their time. All interviews/mini groups were facilitated by members of the research team and were recorded for subsequent analysis.

Data from interviews were summarised into thematic matrices (using Excel, with each column representing a theme and each row an individual interview, so that the data could be sorted in different ways for further analysis). These were developed by the research team and drew on the research questions and logic model. These thematic matrices were then reviewed to identify the full range of views and experiences under each theme.

Scope and limitations

All research is subject to challenges and limitations. First, it is important to keep in mind that this evaluation took a largely qualitative approach. The aim in qualitative research is not to achieve a sample that is statistically representative of the wider population, but to identify as much diversity of experience as possible. Estimates of prevalence based on qualitative data are therefore inappropriate and this report avoids quantifying language, such as 'most' or 'a few' when discussing findings from qualitative interviews.

Similarly, the qualitative nature of this evaluation, and the fact it used a case study approach involving eight local authorities, means it is not possible to provide evidence showing exactly how the funding was used across all local authorities. This includes the extent to which the funding was used on additional elements of summer programmes over and above what would have been provided without the Summer Programme funding, as well as the prevalence and extent to which funding was used on childcare compared to other activities.

Overall, the range of programmes covered, and recruitment quotas set, enabled the evaluation to hear from families with a wide range of characteristics and circumstances, including those on low incomes, with disabilities or ASN, from ethnic minority backgrounds, or with lone parents or young mothers. The research also includes the perspectives of people representing each of the Scottish Government's six TCPDP target groups. However, within the available timeframe, resources and recruitment approach, it was not possible to hear from each of these target groups in equal proportions, nor from participants in all of the numerous types of activity across Scotland that were funded through the 2022 Summer Programme. This limits the ability of this evaluation to comment in detail on which specific activities might work best for different groups - although it does identify enablers and barriers across activities.

The opt-in nature of the recruitment of families and providers was another limiting factor. The ability of the research team to hear from a range of participants across a range of activities was dependent first on local authorities and partner organisations sending invites on our behalf, and then on families getting in touch with us directly. This may have biased the sample towards those who wished to share experiences of summer activities that were particularly positive or negative. The provision of information sheets to accompany the invitation sent by partner organisations, outlining the purpose of the research and the appeal to hear from a wide range of people whatever their experience, was intended to mitigate this risk.

Another limitation is that although providers were asked about any groups that they felt were more difficult to reach, the research did not include interviews with CYP and families who did not take part in activities. While this does not negatively affect the quality of the data gathered, it should be kept in mind that there may be further barriers to participation that the evaluation could not identify. It is possible that data collected from providers may be skewed towards more positive aspects of delivery, although providers did consistently reflect on areas for improvement. It should also be noted that only perspectives of providers involved in the Summer Programme took part in this evaluation and not those who were unable to or chose not to.

The timing for interviewing was, as far as possible, intended to reduce any recall bias around elements of programme delivery by providers or the experiences of CYP who attended. However, it is possible that recall - particularly among the youngest children interviewed - became more limited as fieldwork progressed.

There were challenges reported by local authorities and delivery partners around collecting quantitative data to complete monitoring and evaluation templates, including issues identifying which target groups families were part of. Therefore, there was not a consistent approach to collating the quantitative data and the figures on attendance in the reach section may not be comprehensive.

Finally, the timeframe for this evaluation (interviews for which were completed by mid-October 2022) meant that it focused more on the short- and medium-term outcomes from the Summer Programme, as outlined in the logic model (Appendix A). While there is some consideration of the perceived likelihood of the programme having longer-term impacts, further research would be required to assess the extent to which this actually happens in practice.

Report structure and conventions

The remainder of this report is structured as follows:

Chapter 2: Overview of Summer Programme and planning the offer. This chapter outlines the types of activities delivered using the funding and how providers approached planning the offer.

Chapter 3: Participation and reach assesses the level of engagement among eligible families with Summer Programme, including barriers and enablers to participation.

Chapter 4: Perceived impacts for children and young people considers to what extent the Summer Programme achieved the intended short-term outcomes for children and young people set out in the logic model.

Chapter 5: Perceived impacts for parents and carers examines evidence of short-term outcomes for parents and carers (from hereon in referred to as 'parents' for brevity), with a focus on those included within the logic model.

Chapter 6: Perceived impacts for local authorities and delivery partners. This chapter explores the perceived impacts of the Summer Programme funding on those involved in the planning and delivery of summer activities, with a focus on partnership working.

Chapter 7: Key learning from the 2022 Summer Programme. This chapter synthesises the key lessons learned from the previous chapters, as well as the views of local authorities on the funding model overall and suggested improvements.

Case studies are included throughout the report to illustrate important points and diverse experiences. These are based on real experiences, but pseudonyms have been used and some details have been changed in order to protect anonymity.

This research was carried out in accordance with the requirements of the international quality standard for Market Research, ISO 20252.