Young Person's Guarantee: measurement and evaluation framework

Young Person's Guarantee Measurement and Evaluation Framework

The Guarantee's Measurement and Evaluation Framework

The ambition of the Young Person's Guarantee (from here on Guarantee) is that "every person aged between 16 and 24 in Scotland has the opportunity, depending on their circumstances, to study; take up an apprenticeship, job or work experience; or participate in formal volunteering."​

To do this, the Guarantee builds on existing education, learning, employability, training and employment opportunities to tackle the long-term economic and social scarring of the Covid-19 pandemic.

The measurement of the Guarantee is structured around Key Performance Indicators (KPIs), Interim Measures and Programme data. Each has a specific role to play in understanding the Guarantee as highlighted in the chart in the next section.

We have developed an Evaluation Plan to sit alongside the Measurement Framework. As a multi-agency programme it was necessary to bring together key stakeholders to develop this plan, including, but not limited to, the Guarantee's Implementation Group, Chaired by Sandy Begbie CBE and whose membership includes partners from the Local Government, Department for Work and Pensions (DWP), learning and skills agencies, third sector, private sector, the Guarantee's Equalities Sub-Group, and Youth Leadership Panel.

Overall, the evaluation will help us understand the experiences of young people and employers who are accessing the Guarantee. This will include targeted work to understand the experiences of young people from different backgrounds.

The Guarantee's Measurement Framework

Graphic text below:

KPI

• 6 KPIs which cover the high level outcomes of the policy across five themes:  Participation in Employment, Education and Training; Skills; Employer Engagement; Equalities; and Fair Work.
• These will remain for the duration of the Guarantee and will be influenced by other programmes and external factors along with the Guarantee.
• They will be supplemented by “breakdown” measures which will capture equalities and other breakdowns of these high level KPIs where possible.
• Example:  overarching indicators which look to measure the overall aim of the policy which was defined as being “to mitigate the rise in youth unemployment rate”

Interim Measures

• Interim measures will look to highlight and monitor the progress that we are making towards the Outcomes. 
• Will highlight primarily quantitative information but will look to draw on Qualitative information where possible.
• These are likely to change as we move through the process and more information becomes available.
• Examples: uptake by equalities and Scottish Index of Multiple Deprivation (SIMD), short term outcomes such as successful completion of a college programme or positive destination.

Programme Data

• Management information that is required to be able to deliver the individual elements of the Guarantee and will generally highlight the input/output characteristics of participants.
• Number of opportunities and number of young people supported gathered here.
• Should be consistent measures across all streams of the Guarantee and align to the Employability Shared Measurement Framework where possible. 
• Examples:  number of people: being supported via Local Authorities; taking up an offer at college; or undertaking volunteering.  

Key Performance Indicators

In conjunction with stakeholders, the following KPIs have been developed. They look to capture the high-level outcomes that the Guarantee is looking to influence.

However, it should be noted that other policies and external factors, including wider economic and labour market trends, will also have an impact on these KPIs.

The Guarantee's KPIs are:

• Overarching KPI: Reduce the unemployment rate for young people aged 16 to 24 (excluding those in full-time education) to pre-covid levels by the end of the current parliamentary term.
• KPI 1: Increase the proportion of 18 to 24 year-olds with qualifications at SCQF level 5 and above
• KPI 2: Increase the proportion of employers recruiting directly from education
• KPI 3: Increase the employment rate for young people aged 16 to 24 with disabilities
• KPI 4: Increase the proportion of young people aged 16 to 24 in secure employment
• KPI 5: Improve Scotland's performance compared to EU countries in the participation of young people aged 16 to 24 in education, training and employment.

Further breakdowns of the KPIs will be reported alongside the KPIs. This will include equality breakdowns where possible. Further detail on the KPIs can be found in: Young Person's Guarantee: Key Performance Indicators

Interim Measures

We are also in the process of developing a series of interim measures which will look to highlight and monitor the progress that we are making towards the KPIs. The interim measures highlighted are based on the recommended KPIs in each section. Some interim measures are also likely to contribute to more than one KPI and we will look to assess equalities issues where possible. The table below provides an initial indication of what may be included. This work is ongoing and we will engage with stakeholders in their development.

KPI 1: Increase the proportion of 18 to 24 year-olds with qualifications at SCQF level 5 and above

Potential Interim Measures:

• Level of qualifications achieved in funded college courses
• School leaver data on vocational qualifications

KPI 2: Increase the proportion of employers recruiting directly from education

Potential Interim Measures:

• Proportion of employers recruiting apprentices
• Proportion of employers offering work experience and work inspiration activities
• Proportion of young people employed in the public sector

KPI 3: Increase the employment rate for young people aged 16 to 24 with disabilities

Potential Interim Measures:

• Equalities will be assessed across all interim measures where possible.

KPI 4: Increase the proportion of young people aged 16 to 24 in secure employment

Potential Interim Measures:

• Measures under KPI 2 will help support KPI 4, as the Guarantee's employer engagement work encourages employers to adopt Fair Work practices. Fair Work will also be considered in future evaluation plans.

KPI 5: Improve Scotland's performance compared to EU countries in the participation of young people aged 16 to 24 in education, training and employment.

Potential Interim Measures:

• Participation and outcome measures across all Young Person's Guarantee provision, where possible.

Programme Data

The Key Performance Indicators draw on existing data sources such as the Annual Population Survey and the Employer Perspectives Survey. However, the Interim Measures, while using some of these existing data sources, will be largely developed based on Programme Level data in order to align as closely as possible with the actual provision of the Guarantee.

There are five policy areas from which we will draw Programme Level data from to monitor the Guarantee. These are Local Authority (LA), Third Sector, Volunteering, Developing the Young Workforce (DYW) Employer Engagement, and College and University Graduates.

Across these programmes, we have built on existing data collection systems where possible, such as employability data reported from LAs delivered under No One Left Behind (NOLB), and streamlined this as much as we can for the individual policy areas. We are also drawing on stakeholder input to refine this data.

The types of data we are looking to capture is data on opportunities, equalities such as gender and disability, SIMD and outcomes for participants. However, due to the differences across the five policy areas noted above, this is complicated and varies due to a number of factors, such as when the programmes started.

We will be looking to publish programme data, as management information, over the next year, where possible.

The Guarantee's Evaluation Framework

The Guarantee's evaluation framework is centred around three key themes and is expected to span a number of years.

Initially we will commission process evaluations within these themes to inform programme improvement. As the Guarantee progresses we will also seek to commission studies that aim to identify and attribute impact. We will set-up Research Advisory Groups (RAGs) for all studies to ensure collaboration with stakeholders and the delivery of high-quality evaluation research.

To date we have worked with partners to design the studies to be delivered up until the end of 2022, this is described in the next section. We will continue to work with partners to define the Guarantee's future evaluative activity.

Guarantee's Evaluation Themes

Theme 1: Employability Services

We are intending to run a joint implementation evaluation of the Guarantee's employability interventions and Phase 1 of the No One Left Behind reforms. This research will seek to better understand the implementation of the Guarantee's additional investment in local employability services.

Theme 2: Employer Engagement

Employer engagement will be captured via both survey and wider social research activity. For example, biennial surveys such as the 'Employer Perspectives Survey' and 'Employer Skills Survey' will provide a key source of insight on employer's skills requirements and their views and actions regarding the recruitment and training of young people. This evidence will be used in the KPIs and Interim Measures. In addition, in 2022, to inform service improvement, we will engage employers to understand how well the Guarantee and the DYW programme leveraged employer's support of young people's education and employment.

Theme 3: Education Interventions

We plan to engage with education practitioners to understand how recent attempts to embed work-based learning opportunities throughout school's curricula have shaped Scotland's education system. The findings will help better understand how the Guarantee can address societal inequalities. This will include reflections for how work-based learning might further support those less well served by traditional academic learning pathways.

Outcome Evaluations

In later years, it is proposed that the Guarantee's evaluative studies will seek to identify and attribute impact. This work is to be developed but it is likely to prioritise research that seeks to better understand our ability to support those who face the greatest barriers to entering and sustaining employment.

Publication Plans

Published Measurement & Evaluation Activity

• July 2021 – Key Performance Indicators
• December 2021 – Evidence Paper

Planned Measurement Activity

• College reporting on year 1 – SFC published their Annual College Statistics Report on the 25th January 2022. Data for the Guarantee is not published separately in this report but is included in the overall college statistics. The plan is to publish the tables for the Guarantee by late Spring after the SFC College Performance Indicators (CPI) publication is released.
• NOLB reporting – first report detailing the Guarantee's activity – end of February 2022
• Interim measures – Spring 2022
• Reporting on other work streams not expected until late summer/autumn at the earliest

Evaluation Activity – Expected timescales

• Employer engagement – expected report in Autumn 2022
• Education professionals – expected report in Autumn 2022
• NOLB Implementation Evaluation – expected report in Winter 2022