Developing young workforce strategy - impact on education: evaluation

Appendix B: School sampling framework

Sampling requirement

The Scottish Government invited researchers to propose a sampling methodology for researching the impact of Scotland's Developing Young Workforce Strategy on education. As noted in the brief:

"Overall, we anticipate that the successful contractor will develop a sampling methodology which will allow the evaluation to attain an overview of DYW's impact across Scotland without the need to directly engage in fieldwork with each of the 32 Local Authority areas. The research should collect and analyse the perspectives of those working across the Education System".

In The Lines Between's (TLB) proposal, we put forward the following approach:

Following the inception meeting, we will immediately begin designing our sampling plan to select local authorities and schools to include in the research. We will select a diverse sample of schools, considering and reflecting in the selection: urban and rural locations; different school sizes; geographic spread across Scotland (i.e. not just central belt); and percentage of school roll living in 20% most deprived areas of Scotland (based on SIMD profiles).

We anticipate varying levels of engagement with DYW across schools and, while the Scottish Government has suggested undertaking fieldwork with 10 schools, we feel that this sample may not provide a full picture of how the strategy has been implemented across Scotland, and the range of factors that influence progress. A sample of 10 schools may also not achieve the segmentation the SG requires (e.g. observations of differences between schools in rural urban settings). Our suggestion is to increase the sample of schools to 15 – this will give us a greater understanding of the scope of the impact of the strategy and enable us to fully explore factors which have affected implementation. While this is ambitious, we are confident it is achievable given the experienced and senior team of 6 researchers working on the project. A considerable number of days have been allocated i to setting up fieldwork in schools, which reflects the careful, responsive and concentrated approach we will take to liaising with head teachers to plan the fieldwork.

Our proposed sample will include 9 secondary schools, 4 primary schools and 2 ASN schools – we have allocated a greater number of slots to secondary schools as we anticipate that they will have more staff with direct experience of implementing DYW, and therefore more relevant data on impact (i.e. primary schools have fewer people involved in designing curriculum and do not have careers advisors, some ASN schools may not have the labour market as their primary target for pupils). We will work with head teachers to identify which members of staff will participate in interviews. It is likely they will include head teachers, depute head teachers, careers advisors, department heads, DYW coordinators, curriculum leads and class teachers. We plan to speak to 5 members of staff from each school; a total of 75 members of staff working in 15 schools. There may be occasions where either less or more staff are available from each school and can be flexible with the allocation. In addition, we plan to survey school staff within the sample of 15 schools so that all staff in the school have an opportunity to share perspectives and experiences.

A 'backup' sample of schools will be drawn up that share characteristics with our initial sample (e.g. similar location, similar size school) in case any of the selected schools are unable to accommodate the research in the timeframe.

To establish the sample dataset

Immediately after the research contract had been awarded to TLB, the Scottish Government provided high level school level data derived from publicly available School level summary statistics - gov.scot (www.gov.scot), for the sample to be drawn from (n=407 schools).

Prior to the contract award, the Scottish Government's access protocols for social research in a school setting were applied, which resulted in opt-in from most Directors of Education (n=328 accessible schools).

Some schools with the sample frame were deemed ineligible for sampling purposes, for example, because of concurrent involvement in another Scottish Government funded research project (n=238 eligible schools).

First filter: Yes (opt in from Director of Education) + Eligible for Sample, resulted in 238 schools spanning 26 local authorities. This comprised:

• 9 ASN (Special) Schools [desired sample = 2]
• 33 Primary Schools [desired sample = 4]
• 147 Secondary Schools [desired sample = 9]

Sampling criteria for inclusion in the sample

In the TLB proposal and following further conversation with Scottish Government and a review of the school data, categories to incorporate within the sample were identified:

• Mixture of Rural/Urban
• Range of SIMD representation
• Diverse ethnicity
• ASN in mainstream context
• Pupils facing additional barriers to labour market; for example, temporary exclusions or high unauthorised absence rates
• Diverse levels of engagement with DYW, based on the proxy indicator for a school's DYW commitment (total no of Senior Phase pupils / school's total number of college enrolments)
• School Size

Method of selection

TLB took a pyramid approach to sorting the database; beginning with a focus on the cohort with smallest selection of schools to choose from – the ASN (special) schools; then widening our search to primary schools; then finally secondary schools – assessing each of these data sets against the criteria described above.

Sampling

The team identified broad aims for the sample, with specific criteria for each school cohort.

Across the entire sample:

• 15 different local authorities
• A selection of schools where a high proportion of pupils come from a particular SIMD quintile, with the intention that across the sample, a full spread of quintiles is achieved.

ASN Schools

In the sample of 2 ASN schools:

• 1 large pupil roll, 1 small pupil roll (compared to others in the sample pool).

Primary Schools

In the sample of 4 primary schools:

• 1 each of island, rural, town, city
• 1 each of small, medium, big schools
• 1 with high levels of ethnic diversity compared to others in the sample pool.

Secondary Schools

Within the sample of 9 secondary schools:

• 1 with high numbers of pupils taught in Gaelic compared to others in the sample.
• 3 with high levels of ethnic diversity compared to others in the sample pool.
• 1 school with high numbers of ESOL pupils compared to others in the sample pool.
• 2 each of island, rural, town, city
• 2 small pupil roll, 2 large pupil roll, others spanning this range
• 1 denominational school
• 1 with relatively high level of temporary exclusions compared to others in the sample.
• 1 with relatively high unauthorised absence rate compared to others in the sample.
• 1 high level of ASN compared to others in the sample pool.

Using the methodology described above an initial sample of 15 schools was selected, which meets all broad and specific criteria set out in the sampling overview.