Education and Skills Impact Framework (ESIF) - modern apprenticeships provision: contextual summary report 2022

Considerations

When interpreting the findings within this paper, it should be noted that this is not considered to be an economic impact assessment of the university or college sector or apprenticeship offer. Several points should be considered, which include but are not limited to:

1. Benefits of the Longitudinal Educational Outcomes data set (LEO) - The LEO dataset presents a unique opportunity to analyse the long-term earnings, employment and welfare dependency of individuals who have undertaken post-school education and training. This is the first time that this data

has been used in Scotland to measure impact and as such represents a considerable leap forward in our knowledge of the long-term outcomes of investment in education and skills.

2. Individuals included in the analysis - The results refer to Scottish-domiciled learners who were working in the UK after qualification. Individuals working abroad are not included, as HMRC does not collect earnings and employment data for these individuals. HESA data shows around 3 per cent of Scottish university graduates are working outside the UK, and the figure is thought to be lower for college graduates and MA completers.

3. Prior attainment - We currently do not have data on the prior school qualifications held by individuals. This lack of secondary school information implies that the ability bias for individuals that progressed from secondary school to university, MAs or higher qualifications at college cannot be fully mitigated by controlling for any prior attainment scores in the econometric analysis (as a proxy of ability). As a result, the estimated returns may not estimate the true returns to qualification achievement, with the bias likely to overestimate impacts for those qualifications (e.g. first degrees) where prior academic ability is a key driver of enrolment and achievement.

4. Work experience - The data does not contain details of prior work experience, nor any information on individuals’ non-cognitive skills (meta skills), both of which are expected to impact earnings, employment, and welfare benefit dependency.

5. Reasons for non-completion - The main counterfactual (control group) used in the model is non-completers. Individuals may not complete their qualification for a variety of reasons: they may find the course too difficult; they may lose interest; an MA may have a poor relationship with their employer; or other personal issues. An individual may not complete training for a ‘positive’ reason, for example, because they have been offered a job elsewhere, or decide to pursue a different career. The reason for non-completion is not available for inclusion in the model, therefore the marginal impacts will reflect a variety of reasons for non-completion, both positive and negative. We also assume that non-completers drop out at the beginning of their studies in the ROI model.

6. Older students - LEO data on an individual’s highest qualification is collected from 2003/04 for university and college students, and from 2008/09 for MAs and therefore only provides a partial record of education for older individuals. These individuals may have obtained their highest qualification prior to the LEO collection date, which could overestimate the impact of more recent qualifications.

7. Causation versus correlation - The labour market returns estimated should not be interpreted causally, but only as associations. In other words, while certain qualifications may be associated with higher marginal earnings and/or ROI we can’t say for sure that it is the qualification that is driving these higher earnings.

8. Stepping-stone qualifications - The model looks only at the impact of an individual’s highest qualification, however lower- level qualifications obtained by the same individual may also impact on their employment and earnings. For example, an individual may have an MA level 3 as their highest qualification (SCQF 6/7) but may also hold an MA level 2, therefore we can’t estimate the value that each ‘stepping-stone’ qualification adds to the overall learner journey. In the same way, the model only accounts for the costs of the highest level of qualification achieved.

9. Economic conditions - The Return on Investment (ROI) model used is sensitive to several key economic conditions including inflation. As such, changes to prior trends, such as wage growth or employment, may lead to returns differing from estimates. While the best available projections of inflation were used at the time, these do not reflect recent rises above the predicted trend.

10. Aggregation of vocational courses - MA provision is grouped by SCQF level (previously SQA level), ensuring that an approximately equal level of skills and experience is assumed. However, this ignores differences in occupation framework and intended learning outcomes between frameworks at the same level. Results should be considered as reflecting an average result across provision at an SCQF level and do not provide insight into the outcomes for MAs from any singular framework.