Developing the Young Workforce evaluation: evidence synthesis

3. Methodology

3.1 Approach of the review

This evidence review was conducted over three months in autumn-winter 2022. The purpose of the review was to obtain and synthesise evidence on DYW’s impact on work-relevant learning (between the ages of 3-18), apprenticeships, colleges, employer engagement and equality and inclusion. The evidence review comprised three phases: (1) review of key documentation; (2) data collection from internal and external partners, as well as publicly available and unpublished sources; (3) evidence analysis.

In the first phase, key policy and strategy documents relating to DYW were reviewed, and sources of potential evidence identified. In the second phase, DYW implementation experts such as regional group leads, local authority leads, and Education Scotland and Skills Development Scotland partners were contacted, and relevant evidence obtained. Internal consultation with Scottish Government officials working on DYW and related policy areas was also conducted. In the third phase, evidence was analysed through a desk review of the findings and descriptive analyses of statistical data.

The evidence sources reviewed were diverse, and included official statistics, case study material, surveys, infographics, brochures, websites, reports, policy documents, news publications, videos and communications. The evidence short-listed for inclusion in the evidence review was chosen on the basis of relevance to DYW, and reference to key milestones, change themes and KPIs. To provide a balanced overview of the DYW programme, both quantitative and qualitative data was included in the evidence review. Care was taken to systematically collate and analyse data from a wide range of successful initiatives, covering all Scottish regions and all years during which the DYW programme was active. Where possible, activity data was supplemented with data relating to the corresponding outputs of activities and the impact of such activities.

3.2 Research objectives

The documentary analysis was guided by two main research objectives:

• RO1. Summarise the available evidence on DYW activity and outputs between 2014-2019
• RO2. Identify good practice examples and service delivery lessons arising from DYW activity during the COVID-19 pandemic

3.3 Evidence sources

The data reviewed as part of the analysis originated from the following types of evidence sources:

• Websites – Scottish Government Publications, DYW website, Skills Development Scotland, Education Scotland
• Statistical publications – ONS Labour Force Survey, Annual Population Survey, Eurostat, OECD statistics, school leavers and looked after children data
• DYW annual progress reports
• Internal and external contacts at national, regional and local level
• Reviews commissioned by the Scottish Government: Researching the Impact of Scotland’s Developing Young Workforce Strategy on Education (2023a) and DYW Employer Evaluation (2023b)
• Surveys – Senior phase headteacher surveys, Scottish Employer Perspectives surveys, Scottish Employer Skills surveys
• News and magazine publications – Informed Scotland magazine

3.4 Limitations

Most of the evidence reviewed in preparation for drafting this report covered the first five years of the DYW programme. Due to the onset of the COVID-19 pandemic in 2019, official reporting about DYW activities was delayed, halted or limited between 2019 and 2021. As a result, there is limited insight into the scope of DYW activity during the COVID-19 pandemic. This limitation was addressed by the addition of a new heading to the report: “COVID-19 and DYW-related activity 2019-2021”.

There are three further limitations of this evidence review. Firstly, some of the evidence reviewed was not explicitly collected for the purposes of formally evaluating the impact of the DYW programme. To remedy this, findings from the official DYW annual progress reviews were synthesised and presented in the “annual progress” sections. Secondly, this evidence review incorporates analyses of secondary data to a greater extent than primary data. For this reason, commissioned reviews and surveys, which involved primary data collection, were included in the evidence review. Thirdly, there is little available evidence on young people’s views on, and experiences of, the DYW programme.