Literature Review on Teacher Education in the 21st Century

6 EVALUATING THE IMPACT AND EFFECTIVENESS OF TEACHER EDUCATION

Key points summary Building research capacity in teacher education is a recognised priority internationally. Reviews of teacher education research in different national contexts consistently indicate that the field of teacher education research is fragmented and non-cumulative. Research-based evaluations of teacher education systems are limited; however, there are other avenues that could be explored such as inspection and school-level self-evaluation. Inspection facilitates comparison between providers against pre-specified criteria and also provides a basis for national overviews of the quality of teacher education. Some studies indicate that the use of evidence to inform policy and practice in teacher education at school level requires further development.

Introduction

6.1 The previous section provided an overview of research exploring the link between teacher education, teacher quality and pupil outcomes internationally. This section seeks to explore the fourth objective that is concerned with identifying effective practice in evaluating the impact and effectiveness of teacher education. In conducting this review it is apparent that the literature on effective evaluative practices is limited. In this section some limitations of the research base on teacher education are outlined; and a framework of evaluation methodologies is offered that includes: research-based evaluations; and the evaluation of teacher education through inspection and school-level self-evaluation practices.

Research

6.2 Reviews of research in the UK (Menter et al, 2010; Murray et al, 2009), New Zealand (Cameron and Baker, 2004) and Australia (Murray et al, 2008) indicate that the field of teacher education research is fragmented and non-cumulative, with a high volume of 'one off' single cohort studies. Building capacity in research on and for teacher education has been identified as a priority for educational research in the USA (Wilson et al, 2001; Cochran-Smith and Zeichner, 2005), UK (Munn, 2008) and Continental Europe (Erixon Arreman, 2008; Lunenberg et al, 2007). There is little evidence of evaluative research in teacher education. Kirby et al (2006: 25) note that, ' While rife with rhetoric and innovative ideas, teacher education reform is sadly short on objective evaluations'.

6.3 At present, there are few large-scale quantitative studies with a longitudinal focus and little evidence of large-scale systematic research being undertaken to directly inform the on-going development of teacher education programmes in the UK and beyond (Wideen and Grimmett, 1997; Wilson et al, 2001). Cochran-Smith and Zeichner, (2005:5) note "this dearth of larger and longer studies is the case, at least in part, because teacher education has rarely been a research priority for funding agencies or a focus of well-supported programmatic research."

6.4 Given the concern with persistent underachievement in urban schools and schools facing challenging circumstances, there is a relative lack of attention to teacher effects in these constituencies. Moreover, analysis of large-scale datasets of official records raises equity issues regarding the distribution of 'accomplished teachers'. Based on an analysis of official records in North Carolina (1997-2000), research by Goldhaber et al (2004) suggests that regardless of whether districts offer explicit incentives, teachers are more likely to apply and be awarded advanced certification if they are employed in more affluent schools and in districts with higher-achieving students.

6.5 Evaluating the contribution of policy interventions to raise teacher quality through teacher education is methodologically challenging. Research reviews indicate a need for further major studies using matched pairs designs (to examine the relationship between certification status and pupil outcomes) and the development of sophisticated value-added models to examine teacher effects (Harris and McCaffrey, 2009). Current research contends with limitations of sample size, availability of reliable and comparable student assessments at stage levels and curriculum areas (regional/ national assessment), as well as the considerable challenge of building statistical models that are attentive to background variables influencing teacher and pupil performance (see Cavalluzzo, 2004; Cantrell et al, 2007; Creemers et al, 2010).

6.6 There are resource implications in operationalising in-depth case studies of classroom practice that are attentive to particular institutional contexts and their locale. Many studies do not use direct observation but rely on proxy indicators such as teaching samples from portfolio submissions (Silver et al, 2009), construction of teaching problems through scenarios (Hogan and Rabinowitz, 2009), or data packages of classroom artefacts and curriculum materials from participants (Borko et al, 2005; Clare and Aschbacher, 2001).

6.7 Research reviews (see 6.2) suggest a need for further research to develop and field test prototype instruments to investigate the features of enhanced professional practice in authentic settings. A mixed-method approach drawing on the complementary expertise of teacher education faculty in partnership with researchers from the school effectiveness tradition might generate new inter-disciplinary collaborations in teacher education research (see for example productive possibilities indicated in European models of educational design research suggested by van den Akker et al, 2006). At present, there remain very few examples of large-scale, sequential mixed-method designs relevant to the study of teaching quality. Notable exceptions include Sato et al (2008) on formative assessment practices, work by Day et al (2007) on teacher resilience and competence, Hobson et al (2009) on routes into teaching and Stronge et al's (2007) comparison of NBCTs with non- NBCT colleagues using pre-instructional, dispositional and in-classroom variables.

6.8 Multi-disciplinary mixed methods approaches are at an early stage of development within the Scottish Teachers for a New Era ( STNE) research programme. The relative dearth of such studies stands in contrast to the current emphasis placed on impact assessment and policy relevance in research commissioning (Research Councils UK, 2007).

Inspection and self evaluation

6.9 The main purposes of teacher evaluation are: (1) quality assurance and accountability ('assessment of teaching'), and (2) professional development and the improvement of teaching ('assessment for teaching') (Danielson and McGreal, 2000; Kleinhenz and Ingvarson, 2004; Isore, 2009; Stronge and Tucker, 2003). Unsurprisingly the literature contains different positions in relation to evaluation practice and the extent to which the dual purposes of evaluation are compatible (Ferguson et al, 2000; Bates, 2004; Perryman, 2006). A review of the characteristics of Inspectorates of Education in Europe, by Standaert (2000:56-7) notes tension between what he describes as an 'economic-technical' model of inspection that is concerned with output control and a 'pedagogic-didactic' approach that is more concerned with creating internal evaluation cultures.

6.10 The Standaert review (2000:44) notes the influence of the 'standards movement' on school inspection in Europe. The Report suggests there is a growing emphasis on outcome measures within an 'input-context-processes and output model':

• output (covering attainment, achievement, results, outcomes);
• teaching-learning processes (curriculum, pupil guidance, quality of teaching, assessment, ethos, school climate);
• management (school management, leadership, organisation, quality assurance, communication, staff management);
• context input (infrastructure, financial context, characteristics of incoming pupils, legal demands, support structures outside the school) (ibid).

6.11 Reviews of evaluation practices internationally note the need for systemic approaches that align teacher evaluation with mechanisms for (internal and external) school evaluation and school system evaluation (Faubert, 2009; Isore, 2009). Self-evaluation is increasingly used as a systematic form of reflection at team or school level (McNamara and O'Hara, 2004; Swaffield and MacBeath, 2005). Reporting the findings of a large-scale study of staff attitudes towards self-evaluation, Vanhoof et al (2009:27) note that 'working on successful self-evaluation requires, first and foremost, a sufficiently developed reflective capacity.'

6.12 Different approaches taken to maintaining the quality of initial teacher education in Europe are reported in Quality Assurance in Teacher Education in Europe (Eurydice, 2006).

6.13 In England, the quality of initial teacher education programmes is inspected by the Office for Standards in Education (Ofsted, 2009). The outcomes of the inspection bear directly on the future allocation of training places to the provider concerned. The results of the inspections are also included as a set of data within performance profiles that are made available by the TDA annually ¹³. The methodology for the inspections has evolved over a number of years but now relies heavily on a self-evaluation document ( SED) ¹⁴, especially for those providers deemed to be performing well. Where full inspections are carried out, then a series of meetings and observation visits are held with trainees, providers and school partners. One of the major concerns is that schools where students are placed are not directly affected by the outcomes of the inspection and the provider may have little or no control of the quality of support being provided ( UCET, 1999).

6.14 The TDA also carries out a survey of Newly Qualified Teachers annually (most recently, TDA 2009 ¹⁵) which is used to assess the overall effectiveness of initial teacher training ( ITT) provision as well as to identify any providers which may be doing exceptionally well or badly. The survey is distributed to all NQTs in February each year and was supplemented by an induction tutor survey from 2007. NQTs rate their training in relation to the following factors ( TDA, 2010 ¹⁶):

Curriculum, teaching skills, assessment and progression:

• helping them understand the National Curriculum;
• providing them with the relevant knowledge, skills and understanding to teach their specialist subject;
• providing them with the knowledge, skills and understanding to use information and communications technology ( ICT) in their subject teaching;
• preparing them to teach reading including phonics and comprehension (primary NQTs only);
• helping them plan their teaching to achieve progression for learners;
• helping them to establish and maintain a good standard of behaviour in the classroom;
• helping them use a range of teaching methods that promote children's and young people's learning;
• helping them to understand how to monitor, assess, record and report learners' progress.

Continuing professional learning:

• preparing them to begin their statutory induction period;
• preparing them to use the career entry and development profile ( CEDP);
• preparing them to share responsibility for their continuing professional development ( CPD).

Teacher preparation for diversity:

• helping them to teach pupils with special educational needs in their classes, with appropriate support;
• preparing them to work with learners with English as an additional language;
• preparing them to teach learners of different abilities;
• preparing them to teach learners from minority ethnic backgrounds.

Working with others:

• preparing them to work with teaching colleagues as part of a team;
• preparing them to work with other professionals ( e.g. social workers, health workers, police officers);
• preparing them to manage the work of others in their classroom to achieve learning objectives;
• preparing them to communicate with parents and carers;
• preparing them for their teacher's statutory responsibility for the welfare and safeguarding of children and young people.

6.15 Research papers concerning the inspection process in England have focused on concerns about consistency (Sinkinson, 2004), reliability (Campbell and Husbands, 2000; Sinkinson and Jones, 2001) and the 'surveillance' aspect of inspection (Wilkins and Wood, 2009).

6.16 In Scotland, a cyclical process of Collaborative Review was undertaken between 2002 and 2006 in accordance with guidelines set out in a Handbook for Collaborative Review, developed jointly by representatives of higher education institutions, the General Teaching Council for Scotland, Education Authorities, schools, teachers' professional associations, HMIE and the (then) Scottish Executive ( SE, 2001a). In contrast to England, these reviews did not lead to the award of resource allocation grades to particular institutions or courses. Recent 'aspect reviews' of teacher education provision in Scotland, led by HMIE, have focused on particular themes and have considered provision across the whole sector: the preparation of student teachers to teach literacy ( HMIE, 2002); student teacher placements ( HMIE, 2005; 2006); and mentoring arrangements ( HMIE, 2008). These were designed to provide helpful support and guidance for future development to all providers. In addition HMIE carried out a 'Scoping Review' of Initial Teacher Education ( HMIE, 2003) that preceded the Second Stage Review of Initial Teacher Education (see section 2). HMIE does have a mandate to inspect ITE if required, but currently chooses not to do so.

6.17 The international literature records attempts to bring the quality assurance and professional learning functions of inspection into closer alignment (summarised in Table 6). These include the involvement of the profession in the generation of evaluation standards; the specification of standards tailored to the needs of teachers at different career stages; the promotion of collaborative peer review and the use of multiple data sources within cyclical review processes.

Table 6. Evaluation: merging quality assurance and professional learning

Limitations of evaluation systems	Advances in evaluation practice
Some concerns have been raised regarding the lack of precision in systems designed to evaluate teaching quality. A review of local teacher evaluation policies in 7 Midwestern States in the USA found that less than one in ten school districts required evaluators to be trained in order to improve inter-rater reliability (Brandt et al, 2007).	Systems of evaluation will have more credibility and power if they are not only 'publicly known' but 'publically derived' (Danielson and McGreal, 2000:22).The involvement of teachers in the design and implementation of evaluation systems is recommended (Kyriakides et al, 2006). See, for example, the involvement of the profession in the generation of Standards and assessment of their achievement in the National Board for Professional Teaching Standards scheme in the USA and the Highly Accomplished and Lead Teacher levels proposed in Australia.
Evaluation systems have traditionally been characterised by top-down, one way communication, where the teacher's role is passive.	Contemporary understandings of adult professional learning have emphasised the importance of active involvement (see for example Shulman's Teacher Assessment Project at Stanford, USA) and a sense of agency e.g. through the provision of options within a range of activities (Shulman, 1991).
Teacher evaluation systems in some countries do not differentiate the roles, responsibilities and performance standards expected of teachers at different career stages (Isore, 2009). Public accountability demands that early career stage teachers are subject to the same evaluation procedures as their more experienced colleagues. Evaluation systems are often designed to assure minimal competency rather than assess and promote accomplishment (Peterson, 2000).	Self-assessment and self-directed inquiry in professional development might reasonably be expected of the experienced reflective practitioner, whereas guided self-assessment within supportive communities of practice may be more appropriate in meeting the development needs of novice teachers (Danielson and McGreal, 2000). The advantages of an integrated framework of professional standards lies in the specification of standards at different career stages and the requirement of reflection and development planning to support career progression ( TDA, 2007).
Teacher evaluation systems based on observable teaching behaviours reflect models of student achievement based on skills acquisition. Systems for teacher evaluation need to be congruent with current conceptions of what constitutes 'good teaching' and research-based understandings of how pupils' learn.	New approaches to teaching and professional learning require different approaches to the evaluation of teaching and teacher education. For example, teaching for understanding (Wiske, 1998) and the development of higher order critical thinking skills require different evaluation practices than those designed to assess the outcomes of teaching informed by transmission models of learning (Danielson & McGreal, 2000).
School principals are the most common school-level evaluators (Mathers et al, 2008). Based on research in Australia, Kleinhenz and Ingvarson (2004) suggest that evaluation by principal judgment is not reliable. Empirical studies in the USA also indicate that principal ratings of teacher performance are frequently inaccurate (Peterson, 2000).	Content knowledge and content-related pedagogy of observers are viewed as important by teachers subject to external evaluation. Assessment of evaluators' expertise influences teachers' expectations of likely learning from review processes. Peer evaluators matched according to background, knowledge and experience are more effective than models based on seniority alone.
Many evaluation systems are reliant on quantitative measures: ratings systems; pupil performance data. Over-quantification is often associated with attempts to assure 'objectivity'.	Professional judgment is needed to make sense of evaluation data - establishing warrant from evidence. Multiple sources of evidence from multiple perspectives enhance evaluation systems.

School-level evaluation of teacher education

6.18 In England, the Training and Development Agency for Schools funded school-led projects to explore the impact of ITT on school improvement, pupil attainment and staff development through the Partnership Development Schools ( PDS) programme (2006-09). Whilst some impact measures are described as 'anecdotcal', the TDA (2008:3) notes that PDS impact reports contain some evidence of attempts to systematically evaluate the impact of ITT by 'measuring student performance and behaviour and analysing student perceptions... [through] the use of questionnaires, surveys, focus group interviews and individual interviews with mentors and trainees'.

6.19 Some studies have noted concern regarding current capacity in schools to conduct systematic evaluations of CPD. The review of Teacher Education Curricula in the EU (Finnish Institute for Educational Research, 2009:7) notes that "in-service teacher education was hardly mentioned at all in the documents and only in a few documents were there skills and competences highlighted which should be taken into consideration when planning contents, methods, etc. for teachers' in-service education". An Ofsted evaluation of CPD in schools also reported weaknesses in evaluation, noting that, "Few of the schools evaluated successfully the impact of CPD on the quality of teaching and on pupils' achievement because they did not identify the intended outcomes clearly at the planning stage" (Ofsted, 2006:4).

6.20 A wide range of evaluation instruments are currently employed in the evaluation of teachers, schools and teacher education (see Table 7). At all three levels, a systematic approach involving observation of classroom practice, scrutiny of documents and pupil performance/achievement records and engagement with stakeholders is discernible.

Table 7. Evaluation instruments

Teacher evaluation	School evaluation	Teacher education
Systematic classroom observation - principal ratings using observation protocols. Teacher self-report: interviews, surveys, instructional logs. Teacher data gathering: portfolios/ teacher work samples/ teacher dossiers. Analysis of classroom artefacts. Collaborative peer evaluation: 'instructional rounds'. Analysis of pupil attainment, achievement data and value added. Stakeholder surveys from pupils, parents, community. (Isore, 2009; Goe et al, 2008; City et al, 2009, Harris and McCaffrey, 2004)	External evaluation: ranging from full inspections of schools and individual teachers, to inspections of specific subjects. Instruments: observation of lessons; interviews with school leaders; teachers, parents and pupils; questionnaires to be completed by school staff and parents; study of school documents. School self evaluation: Questionnaires. Target setting & achievement tracking. Value added measures. Peer observation. Portfolios ( e.g. Schools of Ambition) School enquiry groups. Pupils as researchers. (Faubert, 2009; Standaert, 2000; Kellett, 2005; Ruddock and Flutter, 2004)	External Inspection: single or multiple phase. - Off-site analysis and risk assessment. Pre-visit briefing. - - Site visit including observations of teaching; Joint observation with 'trainer'; individual and group interviews with current and post-qualification students; scrutiny of student files and evidence; interviews with and observation of teacher educators; visits to placement schools/colleges; interviews with managers/ coordinators; scrutiny of course documentation/ VLE resources; Provider self-assessment - internal evaluation report. (Ofsted, 2009; ETI, 2009; Estyn, 2007; Eurydice, 2006)

6.21 Evaluation through research, inspection and school-level inquiry each have strengths and limitations, these are outlined below in Table 8.

Table 8. Overview of strengths and limitations of evaluation through research, inspection and school-level inquiry

	Strengths	Limitations
Teacher education research	can investigate precise questions; plurality of approaches and designs that can be used together to address complex problems of practice (Borko et al, 2008); advances in value-added ( VAM) methodology (Kirby et al, 2006; McCaffrey et al, 2004; Harris and McCaffrey, 2004). The objective of VAM is, 'to determine how students' learning and achievement differ having been in their assigned teacher's classroom rather than being taught by an alternative teacher' (Harris and McCaffrey, 2010:254)	rarely cumulative, long-term or large-scale (Cochran-Smith and Zeichner, 2005); claims to generalisability in the applied field of teacher education are influenced by the contextual nature of teaching and learning (Florio-Ruane, 2002). effectiveness research in teacher education does not 'itself generate practical solutions, even on a small scale' (Burckhardt and Schoenfeld, 2003:5 cited by Borko et al, 2008:1024). High quality research findings add to the range of available evidence, supplementing professional expertise and local knowledge.
Inspection of teacher education	valuable basis for comparison within and review across whole systems; connects with self-evaluation practices (MacBeath, 2006).	less flexible and can be less sensitive to particular contexts.
School-level evaluation	strong basis for professional development; connects with developing models of inquiry-minded accomplished teaching (Cochran-Smith and Lytle, 2009) and aspirations for 'teaching as a research-informed and research-informing profession' (Lingard and Renshaw, 2010).	limited in its wider significance; weak capacity in data skills at school level and within professional preparation (Earl, Torrance and Sutherland, 2006). While schools are data rich organisations, capacity to interrogate data to inform decision making is not always well established among staff.

Implications for Scottish teacher education At present, literature on effective evaluative practices is limited. Increased flexibility and autonomy have accompanied enhanced attention to school self-evaluation. Curriculum for Excellence provides a strong vehicle for linking curriculum development and teacher development. Teacher enquiry in Scotland has a long history but remains piecemeal. Much work undertaken by teachers on the chartered teacher programme, the GTCS teacher researcher scheme and local authority learning communities is focused on systematic reflection through self-evaluation. Moving from a reflective capacity at an individual level to a team and school level remains a challenge. Research suggests that further collaborative work between teacher mentors and university-based teacher educators and researchers would enhance the research base on teacher education.