Evaluation of the Impact of the Implementation of Teaching Scotland's Future

3 Methodology

Summary of the approach

3.1 This evaluation used a mixed-method approach. The first stage comprised an online quantitative survey carried out with teachers in primary, secondary, and special schools across Scotland (including both the state and independent sectors). The second stage involved in-depth interviews with representatives from all LAs and with key educational stakeholders, and in-depth interviews and focus groups with teachers.

3.2 All research materials, including the survey questionnaire and interview topic guides, were developed by Ipsos MORI in consultation with the Scottish Government and the Teaching Scotland's Future steering group. Terry Carr, an independent education consultant and former HM Inspector of Schools, worked with Ipsos MORI as an advisor. All fieldwork was carried out between September 2015 and February 2016. All fieldwork was conducted by the core Ipsos MORI research team (the four authors of this report).

Online survey

3.3 An online survey was conducted with 6,346 members of the teaching profession between 14th September and 9th October 2015.

3.4 A small pilot was conducted in early September 2015 with 35 teachers (these were sampled to ensure a mix of sector, teaching experience and LA). Following the pilot, minor amendments were made to the questionnaire. The final version used is shown in Appendix A.

3.5 The sampling approach is detailed in Appendix B. In summary, a stratified random sampling approach was used to select staff. The sample was stratified by sector and staff group and within each strata respondents were sorted by school level variables (rurality, percentage of BME pupils, SIMD, school size) and local authority. Finally, respondents were selected using a random, 1 in n approach^[2]. In total, 30,157 respondents were sampled.

3.6 The overall response rate was 21%. The profile of respondents was compared with the known profile of all staff (using Scottish Government data on school sector, staff group, age, sex, school size, proportion of BME pupils in school, rurality of school, deprivation of school, whether full-time/part-time and whether permanent/temporary). The profiles were very similar which indicates that the achieved sample was representative of all staff - at least in terms of those variables. The data were weighted to take account of the slight differences. Full details are contained in Appendix C.

LA interviews

3.7 Letters were sent to all Directors of Education requesting their support with the research and to nominate a representative to participate in an in-depth interview about the impact of the implementation of TSF on the teachers, schools and pupils in their area. Interviews were carried out by telephone, and all 32 LAs took part. The interviews were semi-structured and included qualitative and quantitative elements. Interviews were carried out between September and November 2015.

Key stakeholder interviews

3.8 Key stakeholders (shown below) were identified by the TSF steering group, and representatives from each organisation participated in a telephone interview. As with the LAs, these interviews were semi-structured, with both qualitative and quantitative elements. Fieldwork took place during November 2015.

Stakeholders

ITE providers

• University of Aberdeen
• University of Dundee
• University of Edinburgh
• University of Glasgow
• University of the Highlands and Islands
• The Open University
• University of Strathclyde
• University of Stirling
• University of the West of Scotland

Teaching unions

• EIS
• NASUWT
• Scottish Secondary Teachers' Association (SSTA)
• VOICE - the union

Other stakeholder bodies

• Association of Directors of Education in Scotland (ADES)
• Association of Head Teachers & Deputes in Scotland (AHDS)
• COSLA
• Education Scotland
• General Teaching Council for Scotland (GTCS)
• School Leaders Scotland (SLS)
• Scottish College for Educational Leadership (SCEL)
• Scottish Council of Independent Schools (SCIS)
• Scottish Teacher Education Committee (STEC)

Interviews with teachers

3.9 Face-to-face in-depth interviews and focus groups were carried out between November 2015 and February 2016 with a range of teachers in Scotland to gain further insight into the impact of TSF on their professional learning and teaching practice. Teachers who had taken part in the survey and agreed to be recontacted were invited to participate in this follow-up qualitative research. Head teachers who had taken part were also used to recruit staff from their school.

3.10 In cases when it was difficult to recruit from survey recontacts, recruitment was undertaken by 'snowballing^[3]' from confirmed participants, and working with LA Education Service contacts to reach out to teacher networks or groups. Teachers and schools were selected to ensure a wide range of different experiences was covered in relation to the following characteristics:

• Sector: early years, primary, secondary and special school
• Geography: urban/rural
• Number of years teaching experience
• Role (e.g. class teacher, principal teacher, depute head teacher, head teacher, probationer, early career teacher)

3.11 In addition, supply teachers, those who undertook ITE outwith Scotland and those who had returned from a career break and were interviewed (these interviews were largely undertaken by telephone though some took part in focus groups).

Table 3.1 Profile of qualitative interviews

Participant career stage/sector	Early Years	Primary	Secondary	Primary & Secondary mixed group	Independent	Special
Focus Groups: number of groups (total number of participants)
Depute Head/Head teacher		1(10)[4]	1(5)
Teachers with 10+ years' experience		1(6)	1(5)
Teachers with 5-10 years' experience		2(7)	2(11)
Mixed group: teachers with 5+ years' experience	1 (6)				1 (7)	1 (5)
Early Career				2 (10)
Probationer				2 (8)
Interviews: number of participants
Depute/Head teacher		4	5			2
Studied ITE outwith Scotland		4	4
Returned from a career break		4	4
Supply		2	3			1
Total	6	37	37	18	7	8

Interpreting the data

Quantitative data

3.12 Data tables were created from the online survey results. Each question in the survey was shown with results broken down by key analysis variables. The tables included significance tests to highlight differences between sub-groups. Throughout the report, differences between sub-groups are commented upon only where these are statistically significant i.e. where we can be 95% certain that they have not occurred by chance.

3.13 The analysis comprised question by question analysis examining both the frequency of response to each survey question and the extent to which responses varied by key demographics such as school sector, teacher type and teaching experience.

3.14 In the charts throughout the report, where percentages do not sum to 100%, this may be due to computer rounding, the exclusion of 'don't know' categories or multiple answers. Throughout the report, an asterisk (*) denotes any value of less than half a per cent and a dash (-) denotes zero.

3.15 Where possible, comparisons have been drawn between the 2010 TSF teacher survey and the survey in this study. Due to the changes made to the questionnaire, only a small number of questions were directly comparable - these questions are highlighted in the survey topline results in Appendix A.

3.16 As noted above, the online survey from the evaluation was weighted to reflect the teaching current teaching population, while the 2010 data was not. This meant that, in order to allow comparison, the 2010 results had to be weighted in line with the procedures in 2015. The resulting figures may, therefore, differ from those found in the 2010 TSF Report.

Qualitative data

3.17 Unlike survey research, qualitative social research does not aim to produce a quantifiable summary of attitudes or experiences, but seeks to identify and explore the different issues and themes relating to the subject being researched. The assumption is that issues and themes affecting participants are a reflection of issues and themes in the wider population concerned, and the way in which these impact on people.

3.18 With the permission of participants, interviews and groups were audio-recorded and transcribed for analysis. In addition, after completing each interview or group, detailed field notes were written up to capture the key points of the discussion.

3.19 Once the fieldwork was completed, the team held a series of meetings to enable a collaborative discussion of the findings and their implications.

3.20 This process culminated in the identification of themes and sub-themes. Transcripts and notes were then systematically analysed for key points within these themes. This method ensured that analysis and reporting of the data was rigorous, balanced and accurate, and that key messages were identified.