Scottish Mentoring and Leadership Programme: interim report - qualitative process and impact assessment

Chapter 1: Policy background, aims and methodology

Policy background

In July 2023, there were 12,206 Looked After Children under 18 in Scotland.^[1] While education outcomes for looked after children have improved considerably over the last 10-12 years, there is still a notable gap between them and their non-care experienced peers.

The Scottish Mentoring and Leadership Programme^[2] (SMLP) aims to support and improve the outcomes of care experienced young people, and those in ‘tough realities’. SMLP is very much in line with The Promise made in 2020 to the country’s most vulnerable children following the Independent Care Review. The Promise sets out what Scotland needs to do “to make sure its most vulnerable children feel loved and have the childhood they deserve.” One of the foundations of The Promise is People:

“The children that Scotland cares for must be actively supported to develop relationships with people in the workforce and wider community, who in turn must be supported to listen and be compassionate in their decision-making and care.”^[3]

SMLP is one of several other initiatives aiming to improve the wellbeing of children and young people, such as Developing the Young Workforce (DYW), and it aligns with the preventative agenda set out in Best Start, Bright Futures, the Scottish Government’s delivery plan for tackling child poverty.

Central to the SMLP is the idea that young people can be empowered by developing their relationships with family, friends and adults in the wider community, and that more relational forms of support will enhance well-being and develop the capabilities of young people and their families, thereby improving their short, medium and long-term outcomes.

Following early evidence from an independent evaluation^[4] that MCR Pathways (one of the main strands of SMLP) has had a positive impact on outcomes and capabilities, the Scottish Government and the Hunter Foundation agreed to invest additional funding to expand MCR Pathways and roll it out over a six year period (from 2021/22 to 2026/7) to a much larger cohort of young people. Alongside this, and in partnership with MCR, Columba 1400 would deliver values-based leadership training to young people in schools via their Young People’s Leadership Academies (YPLAs). It was envisaged that the two schemes would complement each other in many schools with, for example, schools involved in MCR being encouraged to participate in Columba 1400 as well – and vice versa.

The funding model was intended to act as a catalyst: supporting take-up in areas where MCR and Columba 1400 were not embedded already – by providing part funding for three years^[5]. It was intended that this would give sufficient time to demonstrate impact, after which schools/local authorities (or other funders) would commit funding to sustain the programme.

Intervention overviews

MCR Pathways

MCR Pathways^[6] is a school-based mentoring scheme that seeks to improve the life chances of care experienced young people. The overall aim of the intervention is to increase the number of young people in full-time work, college or university after leaving school (known as “positive destinations”). In doing so, the scheme aims to improve attainment, and staying on rates for S5 and S6 pupils, as well as supporting young people to develop key life skills.

There are two broad groups of eligible pupils:

• Group one: young people who are currently or have previously been in care. These pupils are prioritised for support. These pupils are initially identified via the school’s information management system, known as SEEMiS, and then each individual’s potential participation is discussed by the Co-ordinator and school staff who know the pupil (e.g. guidance and pupil support staff).
• Group two: other young people who may benefit from the intervention (i.e. those in ‘tough realities’). These young people are identified by school staff.

All eligible young people are free to decide whether or not to take part in the whole intervention, or parts of it.

Schools have an MCR Pathways Co-ordinator located in the school who supports pupils and mentors, working closely with other school staff members. In S1 and S2, young people take part in weekly group work sessions with the Co-ordinator. These sessions focus on building confidence, life skills and improving literacy and numeracy. It is intended that the Co-ordinator builds trust and an individual relationship with each pupil at an early stage in their secondary school journey. Co-ordinators can also support the transition of pupils from P7 to S1.

Pupils are then matched with a mentor from S3 onwards^[7]. This is intended to provide more one-to-one support. Mentors are volunteers from the local community with a range of different backgrounds who receive training from MCR. They are not required to have any specific qualifications or prior experience working with young people. Pupils and mentors are matched based on their personalities, skills and interests, and the pupils’ career aspirations.

Columba 1400

Columba 1400’s Young People’s Leadership Academies^[8] provide values-based leadership for young people, The Columba 1400 scheme is delivered in three phases over the course of S2/3. Some schools have ongoing engagement with Columba 1400 and have taken part in the scheme multiple times with different year groups.

Phase one

In phase one, the selected pupils meet with Columba 1400 staff, typically over three to four sessions (ranging from a couple of hours to a full day), in the weeks leading up to the phase two residential. The delivery of phase one is tailored, with more sessions, staggered sessions, or fewer combined sessions, depending on the needs of each school. The purpose of phase one is to introduce the cohort to values-based leadership, to provide opportunities for the young people to get to know each other and to help them prepare for the residential. Pupils take part in a range of activities, including team-building exercises, discussions about leadership and mental health strategies, learning about Columba 1400’s core values, and a ‘values walk’ designed to provide an informal environment for young people to explore their own values.

Phase two

In phase two, pupils, accompanied by three or four members of school staff, go on a residential trip to the Isle of Skye and stay at the community and international leadership centre in Staffin with Columba 1400 staff for five days. From the point of arrival, with the “welcome meal”, there is a focus on high quality treatment and providing a safe space for the young people where they will be treated with respect.

Each day focuses on one of the Columba 1400 values: awareness, focus, creativity, integrity, perseverance and service. The schedule includes a mix of group and individual exercises, indoor and outdoor activities (including a search and rescue activity where pupils are split into two teams and have to find a dummy and transport it back to the centre), as well as discursive and more reflective work (e.g. journalling). The activities are grounded in different leadership models and are designed to encourage young people to “live their values”.

Towards the end of the residential, the group is introduced to phase three which is a project they will work on when they return to school.

Phase three

After returning from the residential, the young people give a presentation to the school’s senior leadership team to summarise their experiences. Supported by Columba 1400, the young people decide on a community-based or school-based project that they want to work on, develop a plan for this, and deliver it together (sometimes as one group delivering one larger project or in smaller groups delivering two or more smaller projects). At this point, Columba 1400 staff typically take a step back to allow the young people to carry out their project – but they provide ongoing facilitation if needed. The purpose of phase three is to further embed values-based leadership by enabling young people to put the skills they learned in phases one and two into practice.

Evaluation aims

The aim of this evaluation is to assess the effectiveness of the Scottish Mentoring and Leadership Programme (SMLP), which includes both the Columba 1400 Young Peoples’ Leadership Academy and the MCR Pathways interventions. There are three main strands to the evaluation which explore:

• The delivery of the programme
• The impact on ‘hard’ outcomes (attendance, attainment, staying-on rates and positive post-school destinations^[9]) and on capabilities (such as increased social confidence, increased health and well-being, and better relationships) which young people have said matter to them
• Value for Money (VfM) - assessing the costs against monetised benefits (including the longer-term benefits to wider society)^[10].

The evaluation will identify ways in which the outcomes of care experienced young people can be further improved and what wider lessons can be learned. The research is guided by an Evaluation Framework (see Appendix 1), which sets out the evaluation questions and the primary sources of evidence that are being used to answer each question.

This report uses qualitative data to explore both process and outcome findings, addressing the evaluation questions in relation to the reach, implementation and impact of the programme. The evaluation will also include a quantitative assessment of impact and a VfM assessment (see Future outputs below).

To support evaluation of the programme, a Theory of Change was developed during the Evaluability Assessment undertaken in 2021. It sets out a theoretical underpinning for understanding how the programme is intended to impact on the capabilities of young people (e.g. social confidence, health and wellbeing, relationships). Split into ‘doing’ and ‘being’ capabilities, these were largely derived from what young people themselves said was important to them. The Theory of Change has been used in conjunction with the Evaluation Framework to guide the design of data collection materials, analysis and reporting. Both can be found in Appendix 1.

Methodology

This report is based on 75 qualitative depth interviews conducted with pupils, parents/carers, MCR mentors, Columba 1400 delivery staff, and teachers in relation to either the Columba 1400 Young Peoples’ Leadership Academy or MCR Pathways, as well as with local authority and national stakeholder representatives.

Sampling and recruitment

Four schools were visited in this phase of the research: two taking part in MCR Pathways and two taking part in the Columba 1400 YPLA. The aim had been to visit a further two schools during this phase. However, due to recruitment challenges, this has not yet been possible.

The schools were selected with the aim of achieving a mix in terms of local authority area, geographic location, rurality, proportion of care experienced pupils, proportion of pupils living in the most deprived areas (SIMD1), school size, and denomination. MCR schools were also selected on the basis that they had begun the scheme in 2021/22 or earlier in order that there were pupils at the school who had taken part in the group work element as well as those who had taken part in mentoring.

The aim is to visit the same schools again in year four of the evaluation in order to explore longer-term impacts.

After the schools were selected, the Scottish Government wrote to the Director of Education in each local authority to secure permission for Ipsos to contact the school directly. Ipsos then wrote to the headteachers to inform them about the research and to request their school’s participation. The research team then liaised with school staff to facilitate a visit to the school.

Each school helped with recruitment of pupils, parent/carers and school staff. MCR schools also helped recruit mentors. To recruit pupils, Ipsos asked the school to list pupils taking part in the relevant intervention alphabetically by gender and assign them a number. Schools were asked to exclude any pupils from the list who they felt would not be in a position to take part in an interview. Ipsos then randomly selected numbers from the list and asked the school to speak to pupils about the research. Parents/carers were given the option to opt their child out of the research, and also to take part themselves. Any pupil who did not wish to take part was replaced by a non-selected pupil on the list.

Schools were provided with written information to share with pupils, parent/carers, mentors and staff inviting them to take part in an interview. Pupils could also watch an information video about the research (see Appendix 2).

Representatives from local authorities and national stakeholders who were involved in implementation or oversight of the interventions(s) were suggested by the Scottish Government, MCR Pathways or Columba 1400. Individuals were contacted by Ipsos and invited to take part in a virtual interview.

MCR schools

MCR fieldwork covered two schools and included interviews with 20 pupils, four members of school staff (depute headteachers and MCR Co-ordinators), six mentors, and two parents. Among the pupils there were:

• 11 girls and nine boys
• 10 in S2 (who had done group work only) and 10 in S4 (who had received mentoring but not done any group work in S1 or S2 because the MCR Pathways had not yet started in their school when they were in S1/2)
• 9 in MCR Group One (i.e. care experienced young people), 11 in MCR Group Two (other vulnerabilities/reasons e.g. being young carers or being at risk of becoming care experienced).

Columba 1400 schools

Columba 1400 fieldwork covered two schools and included interviews with 16 pupils, five members of school staff (teachers, depute headteachers, and headteachers) and four parents. Among the pupils there were:

• 11 boys and five girls
• 6 in S3 (who had recently returned from the residential^[11]) and 10 in S4 (who had been on the residential the year before)
• 1 care experienced young person, and 6 young people who had been selected to take part due to having low social confidence.

Non-school based fieldwork

Other non-school based fieldwork covered interviews with:

• Eight national stakeholders
• Four local authority representatives in areas delivering one or both interventions
• Six MCR Pathways or Columba1400 staff members.

Data collection and analysis

Interviews took place between November 2023 and February 2024. The majority were conducted face-to-face at schools (including interviews with young people, school staff, mentors, MCR coordinators and parents/carers). Interviews with Columba 1400 staff, local authority staff, national stakeholders and a small number of parents and mentors were conducted by phone/video call. All interviews were facilitated by members of the research team and were recorded for subsequent analysis.

Interviews were structured around discussion guides designed by Ipsos, in collaboration with the Scottish Government. Separate discussion guides were developed for each participant group and explored different aspects of the SMLP to varying levels of detail, depending on the participant’s role, but they generally covered:

• Perspectives on the programme aims
• An overview of the programme’s delivery
• Perceived impact of the programme
• Aspects that are working well
• Aspects that could be improved

In interviews with pupils and parents, we asked first about the overall impact of the relevant intervention and what difference it had made to young people (either based on themselves or on the pupil(s) participants were talking about). We then explored each capability in more detail using a card sort exercise. A statement related to each capability was printed on a card and the participant was asked to sort the cards into piles depending on whether they agreed or disagreed that the intervention had made an impact in that way, or whether it had made no difference (see Appendix 2 for the list of statements and related capabilities). We then explored the reasons for the selections in more detail. School staff, stakeholders and mentors were shown the logic model and asked to pick out the capabilities where they felt the intervention had made the most difference and where it had made less of a difference.

Note that, when discussing the impact of the interventions with participants, it was apparent that there was some overlap among the different capabilities (for example, “I am more comfortable speaking to other people” and “I feel more confident in social situations”) and that improvements in some capabilities led to improvements in others (for example, feeling less anxious might lead to improved attendance at school).

Data from interviews were summarised into thematic matrices (using Excel, with each column representing a theme, and each row an individual interview, so that the data could be sorted in different ways for further analysis). These were developed by the research team and drew on the evaluation questions. These matrices were then reviewed to identify the full range of views and experiences under each theme.

A note on qualitative research

The aim in qualitative research is not to achieve a sample that is statistically representative of the wider population, but to identify as much diversity of experience as possible. Estimates of prevalence based on qualitative data are therefore inappropriate and this report avoids quantifying language, such as ‘most’ or ‘a few’ when discussing findings from qualitative interviews. The value of qualitative research. in a process and impact evaluation such as this. is that it provides the perspective of those involved on what aspects of the process have worked well/less well, what the perceived impacts have been and how they think those impacts have been achieved. Qualitative research does not attempt to measure the extent of the impact but to set out the range of perceived impacts. The quantitative impact report (see Future outputs below) will provide robust quantitative data on the extent to which MCR Pathways has had an impact on key outcomes.

The structure of this report

The remainder of this report covers emerging qualitative findings on the evaluation questions. Although there are commonalities between MCR and Columba 1400, this report covers findings for each intervention separately as they are different interventions, with different approaches and durations, and it would be inappropriate to compare them directly with each other.

Chapter 2 focuses on MCR Pathways and summarises the perceived reach, implementation, and impact of the intervention on the young people taking part. Chapter 3 looks at the same issues for Columba 1400 Young People’s Leadership Academy. Chapter 4 considers whether the individual interventions, and SMLP as a whole, have been implemented as intended and whether mentoring and leadership have been embedded across the school system. Chapter 5 sets out the conclusions and suggested improvements.

Future outputs

An Interim Quantitative Impact Report will be produced in autumn 2024. This will look at the impact of MCR Pathways on attainment, attendance, staying on rates and positive destinations. There will be no quantitative impact assessment of the Columba 1400 YPLA because the criteria for selection are such that it would not be possible to identify a suitable control group of pupils.

An Interim Value for Money Report will also be produced in late 2024. This will assess the Value for Money of MCR Pathways.

Following further qualitative fieldwork and quantitative/Value for Money analysis, the final evaluation report is due in spring 2026.