Attainment Scotland Fund evaluation: interim report (years 1 and 2)

12. Use of data, analysis and knowledge

12.1. This chapter explores the way in which schools and local authorities made use of data in selecting, developing and evaluating their Attainment Scotland Fund activities in order to drive improvements.

12.2. Evidence-based plans and approaches was a key criteria for the release of funding to authorities.

12.3. There are four key questions that this chapter aims to explore:

• To what extent did authorities and schools use existing local data and evidence to drive improvements?
• To what extent did they use data to identify their target groups and interventions? To what extent did they use data to measure which interventions were having the desired impact?
• What did and did not go well in collecting, recording and using data?
• To what extent did the fund increase the use, skills and knowledge of how to use data?

12.4. Evidence was collected from the progress reports submitted bi-annually by Challenge Authorities, the annual plans and reports from the Schools Programme, the annual headteacher survey and the qualitative research.

Chapter Highlights – Use of data, analysis and knowledge

• Overall, local authorities and schools appeared to have used data purposely and continuously.
• Data usage appeared embedded throughout the whole process of defining strategies and outcomes, choosing targeting strategies and planning and monitoring interventions.
• On the whole, awareness of the need of data and knowledge on how to access, understand and use it improved over the first two years of the fund.
• Support from local authorities and attainment advisors was a key factor in supporting the successful use of data.

To what extent did authorities and schools use existing data and evidence?

12.5. Overall, local authorities and schools appeared to have used data purposely and continuously. Data usage appeared embedded throughout the whole process of defining strategies and outcomes, choosing targeting strategies and planning and monitoring interventions.

12.6. The fund resulted in conversations about data with local authorities and schools. Teachers and local authority officers felt that their approach to accessing, gathering, understanding and using data was improving, with positive attitudes, increased confidence and greater skills around the use of data.

12.7. On the whole, appreciation of the importance of data, and knowledge on how to access, understand and apply it, improved over the first two years of the fund.

Summary of data sources used

12.8. Overall, local authorities and schools used a wide variety of data sources as shown in Figure 12.1.

Figure 12.1: Data sources used by local authorities and schools

12.9. Most commonly, data used by local authorities and schools included data from the newly introduced standardised assessments, Achievement of Curriculum for Excellence Levels or the New Group Reading Test.

12.10. The majority of authorities and schools complemented this data with local surveys (in some cases pre and post implementation), qualitative focus groups, feedback forms, pupil assessment and tracking of attendance rates/punctuality, amongst others.

12.11. Some authorities reported external commissioning evaluations for their specific programmes. In some cases, authorities provided evidence of outputs from the external evaluations. The progress reports submitted did not make clear whether outputs from data collection exercises were well understood or distributed efficiently.

12.12. Some authorities claimed to use evidence to drive improvements in their implementation of the programme. In some cases improvement methodologies appeared embedded in their strategic thinking.

12.13. Most headteachers reported use of evidence to measure impact of interventions. In Year 2 of the headteacher survey, 85% of respondents indicated that they used evidence to measure the extent to which their interventions were having a desired impact.

Figure 12.2: Use of available evidence, headteacher survey

To what extent did local authorities and schools use data to target and choose interventions?

12.14. This section focuses specifically on how local authorities and schools used data to target and choose interventions. Previously in the report, the broader approaches used by schools to choose (see Chapter 6) and target (see Chapter 7) their interventions was discussed. This chapter focuses specifically on the use of data.

12.15. Overall, data played a significant role in how both local authorities and schools targeted and chose interventions.

Targeting interventions - Local authority level

12.16. Progress reports varied in depth of detail about their targeting approaches and the evidence used to support these. Based on the progress reports alone, the degree of skills and use of evidence could not be assessed. Still, the majority of authorities stated the use of data and evidence to support their targeting approaches.

12.17. However, there appeared to be an increased awareness of the need to use evidence to inform their targeting approaches over time. References to data and evidence in improvement plans and progress reports increased over time. Furthermore, by the end of Year 2 all Challenge Authorities had resources (i.e staff) dedicated at least partly to the analysis and use of data.

Targeting of interventions - School level

12.18. At a school level, awareness of and skills in using evidence to target interventions also appeared to improve over time. The support provided by local authorities and Attainment Advisors seemed to be important for increasing awareness and understanding of data.

12.19. Most teachers interviewed in the qualitative research at the end of Year 2 stated that 'they now had the information, skills and support they required to be able to target pupils effectively'. Many indicated that it was a big learning curve for them. A few teachers indicated that they felt 'in the dark' in the early stages, and found it took time to get up to speed with the data.

12.20. Schools reported using a wide range of data and evidence to target their interventions. Data sources used for targeting interventions at a school level are shown in the figure below:

Figure 12.3: Data sources used for targeting interventions

12.21. Over time, schools appeared to have developed a systematic approach to engaging and using data. The qualitative research revealed that some teachers now had access to data for each pupil, on a spread sheet. This was a new approach for many. Teachers felt that the data from a variety of sources had helped both identify the target pupils, and inform the interventions.

Choosing interventions

12.22. Data played a significant role when deciding which interventions to choose; both at local authority and school level.

12.23. Local authorities largely took the lead in identifying evidence based approaches, as shown in the qualitative research (see Appendix A). Schools drew on the support from their local authority and Attainment Advisor to select appropriate interventions.

12.24. Local authorities reported that they felt informed about the research evidence underpinning key interventions. Many also reported using the Plan, Do, Study, Act model to monitor interventions.

12.25. Local authorities referred to accessing reviews of research through both partnerships with Universities and those produced by organisations, including:

• Organisation for Economic Co-operation and Development ( OECD)
• Dartington Social Research Unit
• Child Poverty Action Group
• Joseph Rowntree Foundation
• Education Endowment Fund

12.26. Teachers reported they were aware that evidence from these organisations and others had been used to inform the selection of interventions.

12.27. In addition, they acknowledged that pupil- and school-level data was used to identify interventions. This included: SIMD, free school meals, Insight data, standardised assessments, staged interventions, health and social work involvement, child protection, attendance, pupil tracking systems, ScotXed data, professional knowledge and judgement.

12.28. Headteachers confidence in using evidence to inform the development of initiatives was high. In Year 2 of the headteacher survey, 84% of respondents agreed that they felt confident in using evidence to inform the development of initiatives.

Figure 12.4: Confidence in using evidence (headteacher survey, year 2)

12.29. In addition to using data or research evidence to identify interventions, the qualitative research also indicated that the views and experiences of key stakeholders, and learning from previously implemented interventions influenced the selection process. This was discussed in more detail in Chapter 6.

What did and did not go well around data usage at a school and authority level?

12.30. By the end of Year 2, most local authorities and schools appeared to have some data skills to make evidence informed decisions. However, this had been a steep learning curve for them, and for some it continues to be.

12.31. During the first two years of the fund, stakeholders were able to identify successful approaches to the use of data, as well as challenges. Evidence from the qualitative research, headteacher survey and progress reports is summarised below.

Successful approaches to the use of data

12.32. The key success factors identified by local authorities around the use of data were as follows:

• Support from local authorities, Attainment Advisors and improvement advisors to schools
• Recruiting research staff and/or commissioning specialist data
• Use of improvement science methodology
• Having conversations about data at the outset
• Standardised assessment – including the development of locally standardised systems
• Including resources to support use of data within the funding application

12.33. Furthermore, some authorities mentioned in the progress reports that they enabled education leaders to engage in the critical analysis of the data. Others stated that they engaged with headteachers to provide feedback to drive improvements.

12.34. Headteachers responding to the survey in Year 2 had many observations about the positive developments and consequences of successful use of data. Some of them saw data as means to improve their initiatives, create a more tailored approach to the implementation of the fund, and introduce more effective and better informed strategic planning, which they stated had the potential to lead to long-term improvements in practice at the school.

12.35. Similar to the local authorities, headteachers also mentioned that support from Attainment Advisors, as well as resources made available at a national level, had helped to bring about successful use of data gathering and analysis.

12.36. Headteachers reported in the online survey positive consequences as a result of increased data use. These included:

• Increased leadership confidence and increased potential for decision making at a school level
• Long-lasting positive changes – improved strategic planning and improved staff morale with teachers being able to see the evidence of their progress
• Increased awareness and understanding of the impact of poverty on pupil outcomes
• Being more informed about how to select and target interventions

Challenges around the use of data

12.37. While there were recorded improvements around the use of data, local authority officers identified some outstanding barriers. These included:

• The lack of national baselines – a few suggesting that the Scottish Government should prescribe a set of measures or indicators around closing the attainment gap, to enable comparison.
• Measuring impact in the short term
• National reporting requirements
• A perceived focus nationally on quantitative rather than qualitative data
• Time lags in data being available at a national level
• The need for a more standardised teacher judgement data
• The time/work involved in using data effectively.

To what extent did the fund increase the use, and the skills and knowledge of how to use data?

12.38. Overall, there were some indications that the fund increased the use, skills and knowledge of how to use data amongst local authorities and schools.

12.39. In progress reports, many local authorities highlighted data usage as a priority area for teachers' Learning and Professional Development. Many authorities used part of the fund to organise specific training sessions to improve knowledge of data and evidence in schools.

"Great improvements being made. Electronic monitoring and tracking systems are now being used by the majority of schools. Schools are also receiving ongoing training and support." (Challenge Authority, Primary Programme, Progress report – Year 2)

12.40. Some authorities monitored progress carefully around the use, skills and knowledge of how to use data, and reported improvements over the two years of the fund around confidence and skills of teachers using data.

12.41. Teachers themselves also reported an increase in confidence using data. Almost all teachers, who participated in the qualitative research, indicated that they were much better at gathering, understanding and using data than they were previously.

"We didn't really know anything about how to gather and use data in the early stages of the attainment challenge. We are now in a much stronger position and have a better knowledge of the context and our pupils."

(Headteacher, Challenge Authority, Primary School, Qualitative Research – Year 2)

12.42. Attainment Advisors reporting indicated a high level of support given to schools and local authorities around the use of data. For example, in the period between September 2016 and January 2017, advisors across 15 local authorities recorded 75 activities relating to data use. These activities involved supporting individual schools (20), clusters (19) central authority (20), and sharing practice between authorities (9).

12.43. Overall, local authorities and teachers felt that the use of data had improved considerably through the fund. The qualitative research suggested that the fund had:

• enabled the development of systems which track each pupil, and allow easy comparison of outcomes within schools, and across some local authorities;
• embedded the use of improvement science methods within some schools;
• built teacher skills around data, evidence, monitoring and evaluation;
• encouraged teachers and schools to take ownership of monitoring and evaluation; and
• enabled teachers to interpret SIMD data, and other data, more carefully and more meaningfully, understanding changes over time.

12.44. The impact of the fund in improving the use of data was evident in data collected through the headteacher survey. In Year 2 of the headteacher survey, over two thirds (68%) of respondents agreed that, as the result of the fund, their skills and knowledge of how to use data had improved.

Figure 12.5: Improvement of data skills, headteacher survey

12.45. In addition, as reported in Chapter 6, 86% of respondents to the headteacher survey said that there had been strong or some emphasis on data skills within their interventions. Indeed, data use was noted in Chapter 9 as one of the factors that were helping the initiatives to succeed in improving attainment and Health and Wellbeing.

12.46. This improvement in data use, however, was not consistent across all authorities and schools. The qualitative research showed that secondary school teachers, who were newer to the fund, had more issues with data than primary school teachers. This was also supported in progress reports submitted by Challenge Authorities.

12.47. Progress reports submitted by Challenge Authorities over the first two years of the fund showed that the primary programme had data procedures well embedded throughout the whole process. Conversely, information on data usage was very limited for the secondary programme.