Developing the young workforce: fifth annual progress report 2018-2019

Chapter 6: Performance Assessment & Evaluation

When we published Developing the Young Workforce (DYW) - Scotland's Youth Unemployment Strategy in December 2015, we made a commitment to report annually on the progress of its implementation over its seven year life span.

The DYW programme has 11 Key Performance Indicators (KPIs) which underpin the programme in terms of areas where we are focusing on improvement. Established at the programme's outset were deliberately ambitious. We were clear that implementation would require significant changes to how young people, practitioners, schools, colleges, training providers and employers behave and interact. The programme also required the collective support of a number of public bodies and organisations to provide further detail on each measure.

Alongside the programme's KPIs, a number of milestones were identified for each part of the system to demonstrate what we might expect to change each year. This chapter lists our progress to date against these 11 areas.

DYW Evaluation

To support us in better understanding the impact of the DYW programme, in addition to the KPIs, we have commenced a programme-wide evaluation. The evaluation focuses on 'priority' aspects of the programme that aim to provide us with evidence about the effectiveness of DYW overall. This approach has been tested with evaluation experts internal and external to the Scottish Government.

We have worked in collaboration with stakeholders and identified priority DYW outcomes. These were selected on the basis of:

• The likely significance of their impact;
• The anticipated level of public interest in the outcome;
• The resources being invested in the outcome;
• The feasibility of carrying out a robust evaluation; and
• Perceived scope for improvement (i.e. a distribution of outcomes that are close or less close to being achieved).

This process led to 18 outcomes being taken forward in the evaluation.

Workshops were then held with stakeholders to identify the DYW activities that are contributing to each priority outcome. This process identified a total of 46 activities for consideration in the evaluation.

DYW evaluation framework

Last year we developed a DYW evaluation framework in order to assess whether the ambitions of the Wood Commission have been fully achieved as we progress with implementation. This provides a 'roadmap' for evaluating the 18 priority outcomes and 46 priority activities, and includes :

• The evaluation questions to answer;
• Details of the evidence that needs to be collected to answer the questions; and
• The qualitative and quantitative criteria that will be used to judge success.

While aspects of this evaluation framework can be analysed using existing quantitative data from the DYW performance framework, many questions will require additional mixed and qualitative evidence to be answered robustly. Qualitative evidence will help us to consider nuanced longer term outcomes, such as attitude and behaviour changes towards parity of esteem between academic and vocational qualifications.

Case studies for the evaluation

To support our understanding of the extent and distribution of change, we have identified a set of in-depth thematic case studies. This was aided by selection criteria which sought to identify case studies that:

• Cover a range of DYW activities that are working well/less well;
• Cover multiple aspects of the education and skills system (interpreted as cutting across two or more DYW Change Themes);
• Offer a representative analysis of DYW across Scotland;
• Consider at least two protected characteristics^[10]; and
• Can be underpinned by a robust evidence base.

Application of the selection criteria and discussions with Scottish Government Officials and partners reduced a long-list of 15 potential case studies to a preferred short-list of 5, which are:

• Stakeholder perceptions of the DYW offer: To explore how stakeholder perceptions of all aspects of the 'DYW offer'^[11] have changed, especially in terms of a shift towards parity of esteem between academic and vocational pathways. The case study will consider drivers and motivations for behaviour change amongst all key stakeholders (e.g. parents / carers, young people, teachers / practitioners);
• Foundation Apprenticeships in the Senior Phase and beyond: This case study will consider the potential 'bridging' role Foundation Apprenticeships (FAs) have provided between lower level vocational qualifications available in school, and higher level vocational qualifications and / or vocational employment destinations. A particular focus will be on the added value FAs have provided throughout the Senior Phase and beyond;
• The rural experience of DYW: This case study will address the specific issue of young people living in rural areas in Scotland and their experience(s) accessing the full range of activities and outcomes that should be available as part of the 'DYW offer'. It will pay particular consideration to identifying the main challenges (e.g. transport, curriculum offer) and / or opportunities (e.g. local labour markets) in rural areas and how this relates to the urban experience;
• Addressing labour market needs: This case study will consider the role of DYW supporting an education and skills system providing a range of work based and work relevant learning that can meet the skills needs of the labour market, including at different scales (national / regional / local), in different regions, for different sectors / sizes of employer and for different occupation types; and
• Regional Groups - added value: This case study will consider how and to what extent the DYW Regional Groups have added value across the DYW programme in terms of: (a) education - employer engagement; (b) the 'DYW offer'; and (c) increased and improved recruitment / employment / retention / job quality of / for young people, at the regional and local level.

Case studies will be prepared using mixed methods, including participatory workshops, focus groups, interviews and surveys, to explore in detail why we are seeing the (quantitative) results suggested by the DYW performance framework. For example, KPI 4 shows that since 2014, an increasing percentage of 16 – 24 year old college qualifiers are moving into higher level study or employment; KPI 10 shows that since 2014, the employment rate for disabled young people has increased and decreased. We anticipate that young people, parents / carers, teachers / school / college staff, employers, training providers, and DYW Regional Groups will be involved as participants in the case study research.

The integrated nature of the case studies means that they will each help to answer multiple evaluation questions. Case study findings will also be considered alongside additional targeted primary data collection, quantitative data from the DYW performance framework and relevant secondary datasets (e.g. from SDS, Education Scotland, DYW Regional Groups) to provide a comprehensive evidence base to answer the full range of questions in the evaluation framework.

Evaluation activities with equalities groups

Alongside activities to develop a programme-wide evaluation framework and in-depth thematic case studies, we have been working with stakeholders from or representing different equalities groups.

In Spring 2019, officials attended a Youth Action Success^[12] forum meeting where a workshop to discuss issues around employment was held. Participants discussed work placements, routes to employment, employer understandings of their needs and experience, and ways to support young disabled people to take up and benefit from vocational qualifications and Apprenticeships. Recommendations from this session were presented to relevant Policy teams, and the programme-wide evaluation will consider the extent to which issues within the scope of DYW that affect young disabled people have improved (e.g. quantitative and qualitative evidence relating to KPI 10 Increase the employment rate for disabled young people to the population average by 2021).

We have also been working with third sector organisations that support care experienced young people. An initial workshop with staff highlighted a range of challenges faced by care experienced young people in relation to education, routes to work, and undertaking vocational qualifications, including Apprenticeships. In Autumn 2019, this will be further developed through DYW-focused sessions with care experienced young people, to establish: their knowledge of work placements, work-based learning and vocational qualifications and Apprenticeships; the extent to which they perceive these as useful and accessible; and to identify areas for change and improvement.

Sessions with young people from minority ethnic backgrounds and girls and young women considering or undertaking STEM subject pathways / careers are also planned as part of the programme-wide evaluation.

Key Performance Indicators

Overarching target

To reduce the level of youth unemployment (excluding those in full-time education) by 40 per cent by 2021

Data Source and Frequency:

Annual Population Survey, Jan-Dec datasets, ONS

1. Figures rounded to the nearest 1,000

2. Overarching target is now based on estimates from the Annual Population Survey as Labour Force Survey estimates are too small to provide reliable estimates for the latest data

Lines to take

• The number of 16-24 year olds who were unemployed (excluding those in full-time education) increased slightly from 24,000 in 2017 to 25,000 in 2018.

• Despite the small increase in the level of youth unemployment over the year, we have continued to achieve the headline target for the Developing the Young Workforce programme, with a decrease of 23,000 (47.4 %) since the baseline in 2014.

Please note: The most recent update to this KPI has required a revision of its data source.

As a result of this, the numerical target has been revised from 31,000 to 29,000, in order to maintain the overarching 40 per cent target specified in the KPI.

The data source for this KPI is now based on estimates from the Annual Population Survey (APS), rather than the previously used Labour Force Survey (LFS). Due to the substantial decrease in the unemployment level and rate (excluding those in full-time education) for 16-24 year olds over recent years, the LFS sample size is now too small to provide reliable estimates. APS has a larger sample size than LFS, and provides more robust information for smaller groups.

KPI 1

Be one of the top five performing countries in the EU for youth unemployment by reducing the relative ratio of youth unemployment to 25-64 unemployment to the level of the fifth best country in the EU by 2021

Data Source and Frequency: Scotland data: Annual Population Survey (ONS); EU data: OECD Stat. Scotland data: Annual, by calendar year; EU data: Annual

Lines to take

• The ratio of youth (16-24) unemployment to 25-64 unemployment has decreased from 3.7 to 2.8 since the baseline rates were measured in 2013. This is due to the reduction in the youth unemployment rate relative to the adult unemployment rate. (For every 1 percentage point drop in the adult unemployment rate, the youth unemployment rate needs to fall by about 2.8 percentage points for the ratio to remain at this level.)
• This ratio has remained at 2.8 since 2017.
• For 2018, the ratio of the fifth best performing country in Europe is 2.2 (Austria).
• For 2018, Scotland is the 14^th best performing country in Europe in terms of the ratio of youth to adult employment.

KPI 2

Be one of the top five performing countries in the EU for youth unemployment by reducing the youth unemployment rate to match the fifth best country in the EU by 2021

Data Source and Frequency:

Labour Force Survey, Jan-Mar datasets, ONS

1. Unemployment Levels rounded to the nearest 1,000

2. Unemployment Rates are based on unrounded figures.

3. EU data for February each year – Eurostat (http://epp.eurostat.ec.europa.eu), accessed 23 July 2019

Lines to take

• For the period January - March 2019 the youth unemployment rate has decreased over the year to 6.6%.
• Scotland is the third best performing EU country over this period.
• The fifth best performing EU country over this period has a youth unemployment rate of 8.1%. This is 1.5 percentage points higher than the Scotland youth unemployment rate for the same period.
• The youth unemployment rate in the UK is 10.3% for January - March 2019, 3.8 percentage points higher than the rate in Scotland for this period.
• Please note: As of July 2019, these figures have been revised to use latest and most accurate available Eurostat data for all time points. This has resulted in changes to previous years' figures.

KPI 3

Increase the percentage of school leavers attaining vocational qualifications^[13] at SCQF level 5 and above by 2021

Data Source and Frequency: annual School Leavers Statistics: data from Awarding Bodies (SQA for baseline data) and SG Pupil Census. Annual Data.

Lines to take

• There has been a year on year increase in the number of school leavers achieving vocational qualifications at SCQF level 5 and above.
• 14.8% of 2017/18 school leavers achieved a vocational qualification. This is an increase of 2 percentage points since 2016/17, and an increase of 7.5 percentage points since the baseline in 2013/14.

KPI 4^[14]

Increase the percentage of 16-24 year old college students who have successfully completed a full time course moving into employment or higher level study

Data Source and Frequency: College Leaver Destination Survey (SFC) Annual, by college year

Lines to take

• The number of 16-24 year old college students moving into employment or higher level study after successfully completing a full-time course has increased by 1.1 percentage points from 2016/17, and by 3.5 percentage points from 2014/15 (baseline).
• The data for 2014/15 has been used to baseline as this represents the % of confirmed destinations of 16-24 year olds in a positive progression. (The data for 2013/14 has not been used as this was experimental data.)

KPI 5

The number of Modern and Graduate Apprentices at SCQF Level 6 and above to be increased. The target is for 20,500 out of a total of 30,000 Modern and Graduate Apprentice starts to be at this level by 2021, i.e. two-thirds

Data Source and Frequency: Modern Apprenticeship Statistics: Full Year Report (SDS). Annual, by financial year (April-March)

Lines to take

Notes: When drawing conclusions about trends, it should be noted that as of 2018/19 the total number of Apprenticeship starts at VQ Level 3 / SCQF Level 6 + will be artificially increased by approximately 2.0% as a result of SVQ – SCQF Level changes.

In 2018/19 SCQF Levels replaced SVQ Levels for MA frameworks.

As of 2018/19, KPI 5 figures include Graduate Apprenticeships, all of which are SCQF Level 6+.

The number of Apprenticeship starts at SCQF Level 6+ increased by 1,996, from 18,524 in 2017/18 to 20,520 in 2018/19.

This is an increase of 4,865 since the baseline in 2013/14.

KPI 6

Increase the percentage of employers recruiting young people directly from education to 35 per cent by 2018

• Baseline figure (2014): 32%

• Previous figure (2016): 32%

• Current figure (2019): 30%

Data Source and Frequency: Employer Perspectives Survey, 2019

Lines to take

• The percentage of employers recruiting young people directly from education has reduced to 30% since the baseline was measured in 2014.

KPI 7

To reduce to 60 per cent the percentage of Modern Apprenticeship frameworks^[15] where the gender balance is 75:25 or worse by 2021

Data Source and Frequency: Modern Apprenticeship Statistics: Full Year Report (SDS). Annual, by financial year (April- March)

Lines to take

• The percentage of Modern Apprenticeship frameworks where the gender balance is 75:25 increased by 0.7 percentage points, but remains at 72% (71.6% in 2017/18 to 72.3% 2018/19).
• This is a decrease of 1 percentage point on the baseline (2013/14).

KPI 8

Increase by 5 percentage points the minority gender share in each of the 10 largest and most imbalanced college superclasses^[16] by 2021

Data Source and Frequency: Further Education Statistics collection (SFC). Annual, by college year

Lines to take

• The minority gender share currently ranges from 3% in Building/Construction Operations to 16% in Engineering/Technology (general).
• The minority gender share increased by at least 1 percentage point in 6 out of the 10 largest and most imbalanced college superclasses between 2015/16 and 2016/17.
• This has increased by at least 1 percentage point in 9 out of the 10 largest and most imbalanced college superclasses from the baseline measurement in 2012/13.
• The largest increases from the baseline have been in Building Services (2% in 2012/13 to 9% in 2016/17) and Engineering/Technology (9% in 2012/13 to 16% in 2016/17). The minority gender share in each of these superclasses has increased
  by 7%.

KPI 9

Increase the number of MA starts from minority ethnic communities to equal the population share by 2021

Data Source and Frequency: Modern Apprenticeship Statistics: Full Year Report (SDS). Annual, by financial (April-March). Annual Population Survey (ONS) (April-March).

Lines to take

• The percentage of MA starts from minority ethnic communities increased from 1.9% in 2017/18 to 2.3% in 2018/19. This is an increase of 1.2 percentage points since the baseline (2013/14).
• This target will change year on year as the population share figure is updated.

KPI 10

Increase the employment rate for young disabled people to the population average by 2021

Data Source and Frequency: Annual Population Survey (ONS) Annual, by calendar year

Lines to take

NB. the Labour Force Survey and Annual Population Survey data was recently reweighted based on updated population estimates. As a result, there are some small differences to previous estimates for KPIs 1, 2A, 2B, 9 (proportion of the Scottish population who are from a minority ethnic background) and 10.

• The employment rate for young disabled people decreased from 43.2% in January – December 2017 to 35.8% for the same period in 2018.
• This is an increase of 0.6 percentage points compared to the baseline figure of 35.2% (January - December 2014).
• The target for 2018 is 57.2% – the target will change each year as the youth employment rate changes.
• There is currently a gap of 21.4 percentage points between the rate and the target. This gap has increased by 3.3 percentage points from 18.1 percentage points in 2014 (baseline).

KPI 11

Increase positive destinations for looked after children by 4 percentage points per annum resulting in parity by 2021^[17]

Data Source and Frequency: Educational Outcomes for Looked After Children (SG). Annual, by school year

Lines to take

• The proportion of looked after children in positive destinations is 76% in 2017/18. There has been no change since 2016/17, and an increase of 6.7 percentage points since the baseline figures were recorded in 2012/13.
• The "looked after children‟ cohort reflects children who have been looked after for the full year only.