Scottish education system: knowledge utilisation study

2. Methodology

2.1 Introduction:

In order to address the research aims, the methodology entailed a three-strand approach (Figure 2.1).

• The first strand was a literature review with two aims: (i) to provide an overview of what is currently known about how research evidence is used within the Scottish education system and (ii) to provide an appropriate framework to inform the subsequent qualitative study.
• The second strand comprised a qualitative study and involved interviews and focus groups with key stakeholders at different levels of the education system. 
• The third strand involved an on-line validation survey to help complement and corroborate the findings emerging from Strands 1 and 2.

Figure 2.1. Overall timeline of the project

2.2 Strand 1: Literature review 

A literature review provided an overview of research engagement and evidence in the context of education. It identified evidence on the extent to which research is embedded in the education system and explored the factors that influence the nature and level of research engagement in education. The review adopted a variant of the systematic review approach. 

Systematic reviews summarise large bodies of evidence to help explain differences among studies on the same question. A systematic review “involves the application of scientific strategies, in ways that limit bias, to the assembly, critical appraisal, and synthesis of all relevant studies that address a specific question” (Cook et al., 1997). Our variation on the systematic review approach involved limiting the time spent reviewing the source databases, due to the time allotted, while still maintaining rigorous selection criteria and applying a systematic framework to the review approach. This approach is informed by a ‘best evidence’ model (Slavin, 2008), and generates a sufficiently comprehensive, criterion-based analysis of the available literature. It also uses a robust, consistent method in retrieving, appraising and synthesising the literature.

The review searched for relevant references from the year 2000 onwards in recognised bibliographic databases using relevant search terms. The search terms were focussed on the main research questions and remit of the study. Where the databases allowed, phrase searching was employed to locate literature across the following sources. Education Resources Information Centre (ERIC); British Education Index (BREI); Google Scholar; SCOPUS; The EPPI-Centre Library; The Australian Education Index (AUEI); Scottish Government website including archived pages and professional association outputs and publications etc. Given the very limited literature that focussed on the Scottish and UK context, the search included international sources that had concentrated on research engagement and evidence in the context of education. The research team drew on the literature review conducted for a previous study (Chapman et al 2015) to highlight relevant concepts of knowledge mobilisation and how other professions, mainly the medical sector, use data and evidence. 

The selection criteria were guided by the terms of reference of the project and, where possible, assessments about the quality/ strength of the evidence to support claims were made. Key criteria included:

• Relevance within the Scottish/ UK context
• Aim and design of the study
• Robust peer-reviewed methodology
• Quality of data and analysis
• Theoretical and/ or ideological stance
• Plausibility of claims and associations based on evidence presented.

It was not always possible to apply all of the above selection criteria to each source. In instances where it was not possible to filter for criteria such as peer reviewed literature, we took a pragmatic approach to assess the quality of the evidence and its relevance on the information provided in the text. Across the various literature sources, we referenced 63 papers and publications. These were mainly empirical in nature, ranging across quantitative and qualitative methods, with a minority including some theoretical and conceptual content and some discussing use of research evidence in policy contexts. Those that were empirical largely focused on factors influencing teachers’ use of evidence and its mobilisation.

2.3 Strand 2: Qualitative study 

Strand 2 addressed the main aims of the project: i.e. to explore and map the extent to which research evidence is used and how it flows through the education system. This strand also looked at identifying the factors that facilitate or inhibit how research evidence influences educational practice and decisions among school actors in Scotland. This strand did this by eliciting relevant information from key stakeholders including classroom practitioners and school leadership teams and other stakeholders such as Attainment Advisors working with local authorities and the strategic leads of the RICs.

Given the paucity of relevant literature, this strand was crucial in providing up to date information about how research influences education decisions at all levels of the system in Scotland. This strand provided insight into the perspectives and experiences of educational practitioners.  A qualitative approach was appropriate as it provides ‘in-depth, intricate and detailed understanding of meaning, actions, non-observable as well as observable phenomena, attitudes, intention and behaviours (Cohen et al., 2011; p.219). Similarly, Evans (2009) states that qualitative research is valuable because it acknowledges ‘the studied phenomena as complex, developing, [and] multifaceted’ (Evans, 2009: p.113).

2.3.1 Cases and participant selection

The approach for Strand 2 began at the strategic level, engaging with the leads across the six RICs. Five out of the six invitees were able to take part in an interview. 

The research team then identified an appropriate local authority within each RIC area. Table 2.1 provides the sampling rationale for the authorities in which to conduct the Strand 2 interviews and focus groups. This rationale was informed by the research teams’ knowledge of research engagement-related activity across Scotland gleaned from the teams’ previous and current national research and development work with schools and local authorities. In each of the selected local authorities, the team invited the Attainment Advisor to take part in an interview. Attainment Advisors were seen as key informants for this strand, particularly given their knowledge of developments across the schools in the context of the local authority and wider Scottish education policy, and their role in building capacity of practitioners and leaders to undertake self-evaluation, enquiry and collaboration in the context of raising attainment. The research team interviewed four Attainment Advisors in total. Two advisors had other commitments that prevented them taking part in an interview during the fieldwork period. 

The team worked with RIC leads to identify two schools (one primary and one secondary) in the selected local authorities. When the team contacted each school leader for interview, they also negotiated identification of an appropriate group of practitioners in the respective schools who were invited to take part in a focus group. A total of 6 primary schools and 5 secondary schools took part in the research. 

By working down from the strategic level, the qualitative research strand was able to frame the interview data within the various regional and local policy frameworks and contextual factors. This allowed for a more nuanced and informative analysis of the nature of research evidence use across levels of the Scottish education system and the key drivers and inhibitors involved.

Table 2.1: Strand 2 Sample rationale

RIC	Local Authority	Rationale
The West Partnership	Renfrewshire	Secondary schools operate whole school and collegiate working groups to develop a number of curricular and support areas.
Forth Valley & West Lothian Collaborative	Clackmannanshire	There is a history of engagement with research across a number of schools in these authorities springing from their SIPP* involvement.
The Northern Alliance	Argyll & Bute	There are a number of primary schools in the authority working together to build research capacity using CAR**. These represent good examples of school-led (as opposed to LA-led) initiatives.
South East Alliance	East Lothian	Schools are involved with Scottish Universities’ Insight Institute (supports programmes of knowledge exchange).
South West Collaborative	South Ayrshire	Head teachers have been sharing data among their family schools to identify and share good practice as well as identifying areas for improvement. Local authority intends all schools will be consistently data literate to drive improvement across the system.
The Tayside Collaborative	Dundee City	The council has a history of building research capacity across schools and partner organisations. Local evidence-based interventions and sharing evidence are a key priority for the authority.

* School Improvement Partnership Programme.  ** Collaborative Action Research

2.3.2 Methods and data gathering

The research methods employed within the qualitative strand, interviews and focus groups, were designed to gather relevant information from different tiers of informants that could provide insights on the research questions. These were:

• Interviews with RIC leads 
• Interviews with the Attainment Advisor responsible for the identified local authority

• Interviews with each school’s head teacher. Often the head teacher would either also invite other members of their senior management team such as depute head teachers to join the interview or invite them to be interviewed separately;
• Focus groups with practitioners including class teachers and CLD and nursery staff.

Interviews were typically conducted face-to-face, but for some of the Attainment Advisors and RIC leads telephone interviews were used when requested by the informant.

Table 2.2: Breakdown of informants involved in the qualitative strand

Individual interviews conducted
Head teachers	6
Depute head teachers / other senior management team members	8
Principal teachers	3
Attainment Advisors	4
RIC leads	5
Number of focus groups with practitioners (Inc. teachers, CLD, nursery staff)
Ten involving a total of 41 practitioners

The data gathering processes were completed between May and September 2018. The total numbers of interviews and focus groups completed during this timeframe are contained in Table 2.2. In total, 67 individuals took part in the qualitative strand.

2.3.3 Processing data and thematic analysis

All interviews and focus groups were audio recorded and transcribed into MS word documents. Transcriptions were analysed using the DEDOOSE 8.0.42 qualitative analysis program. Each file was initially coded using 54 codes generated from the research questions, including sub questions used in the interview and focus group topic guides, as well as themes emerging from the literature review. This initial analytical framework was further developed as other codes were generated from each transcript. The codes were organised into thematic clusters that corresponded to the research aims and questions of the study.

2.3.4 Limitations

Qualitative methods provide in-depth insights regarding participants’ perspectives and practices. However, even when the sampling and selection of participants is for the purpose of obtaining a range of insights and perspectives, the findings cannot be generalised to all of the participant groups. 

Two strategies were selected to address this limitation. First, Strand 3 of the study involves a national survey to validate the qualitative findings to assess the representativeness of the qualitative responses. Second, findings from the qualitative strand were triangulated to compare emerging themes within and across the participant groups. This analysis suggested that there was a strong level of consensus across participants in both the quantitative and qualitative strands of the project. The relatively high levels of engagement with data and research across participants and respondents could indicate that those participating in the research are not completely typical of the population of Scottish teachers. Nevertheless, the findings are salient, particularly those regarding factors that influence engagement with and use of data and research evidence. For example, those teachers motivated to use data and research findings report encountering challenges in this process which is pertinent to strategies to support all teachers, including those less motivated to engage with evidence.

2.4 Strand 3: Validation survey

The emerging qualitative findings from Strands 1 and 2 informed the design and content of an on-line survey of practitioners. The survey complemented and corroborated the findings of the literature review and qualitative strand. It also supported the generalisability of the findings from Strand 2. Teachers across Scotland were invited to participate via directors of education, professional associations such the Education Institute for Scotland (EIS) and the Robert Owen Centre’s educational networks as well as through Education Scotland contacts. In total, 1, 036 responses were received by the time the survey closed. Details of who responded are summarised in Chapter 5 of this report.

2.4.1 Analysis of quantitative data

Analysis of the survey data focussed on analysis of frequencies for each of the variables using SPSS (Statistical Package for the Social Sciences). This reflected the main purpose of the survey, namely, to obtain an indication of how common the main themes emerging from the qualitative strand of the research were in a larger sample of educational practitioners.