Attainment Scotland Fund evaluation: interim report (years 1 and 2)

8. Interventions – Short and medium term outcomes

8.1. The first section explains what short and medium term outcomes interventions were intended to achieve. The second section describes the evidence used to assess the extent to which interventions were achieving these outcomes. The final section explores what this evidence showed around any progress made, and what factors schools and authorities think contributed to whether interventions achieved their short and medium term outcomes.

8.2. Evidence in this chapter draws primarily from Year 1 and 2 Challenge Authority progress reports, and the qualitative research findings provide further detail on stakeholders' views of progress towards short and medium term outcomes.

Chapter Highlights – Outcomes of interventions

• Most Challenge Authorities had clear short and medium term outcomes and targets in place.
• Overall, authorities were, for the most part, able to highlight what evidence they used to measure progress towards achieving their outcomes. However, there was great variability in the breadth and quality of evidence used.
• Overall, there was considerable progress made in the primary programme amongst Challenge Authorities; with strong foundations being built around leadership, resources and training of the workforce.
• Reflecting the later expansion of the fund into secondary schools, there was less evidence of progress in the secondary programme.
• Whilst there was a clear belief across all stakeholders that the fund had provided leadership opportunities and improved teaching skills, views on progress around parental engagement were more mixed, with schools finding this particularly challenging.
• The three key factors contributing positively towards achieving outcomes were early engagement, collaboration and improvement methodologies.

What short and medium term outcomes were interventions intending to achieve?

8.3. Challenge Authorities outlined the short and medium term outcomes that each work stream sought to achieve. Four Challenge Authorities (out of nine) provided short and medium term outcomes in Year 2.

8.4. Broadly, outcomes fell into four main themes:

Figure 8.1: Short and Medium Term Outcomes

8.5. The outline below focuses on short and medium term outcomes reported by authorities relating to interventions focusing explicitly on key features of the Attainment Challenge: Numeracy and Literacy as well as Health and Wellbeing.

Literacy and Numeracy

8.6. Short term outcomes described around Literacy and Numeracy included:

Figure 8.2: Literacy and Numeracy - Short term outcomes

8.7. Medium term outcomes described around Literacy and Numeracy included:

Figure 8.3: Literacy and Numeracy - Medium term outcomes

Health and Wellbeing

8.8. Short and medium term outcomes described around Health and Wellbeing included are described in Figure 8.4 and Figure 8.5.

Figure 8.4: Health and Wellbeing - Short term outcomes

Figure 8.5: Health and Wellbeing - Medium term outcomes

What evidence was used to assess the extent to which interventions are achieving their intended outcomes?

8.9. For the most part, Challenge Authorities were able to highlight what evidence they used to measure progress towards achieving their outcomes and/or targets. However, there was great variability in the breadth and quality of evidence used across the authorities.

8.10. Overall, authorities reported using a range of measures, both qualitative and quantitative in nature. Not all authorities collected all of the following, but the main data sources referred to are summarised in Figure 8.6 below.

Figure 8.6: Data sources used to monitor interventions

8.11. In the sections that follow, the report describes evidence used to track progress in each of the key areas of the Attainment challenge: Literacy, Numeracy and Health and Wellbeing.

Literacy and Numeracy

8.12. Outcomes of interventions focusing on Literacy and Numeracy largely focused on quantitative measures of long-term outcomes through standardised attainment measurements. In addition, several qualitative measures were mentioned by authorities to measure short and medium term outcomes.

8.13. Below are some examples of the methodologies and data sources employed both in the short and the medium term by the range of different local authorities.

• Short Term: The most common themes were surveys, interviews and classroom observations. In the measurement of short term outcomes, surveys were commonly reported as measures of change in knowledge/awareness, skills, motivation/engagement and confidence (including training evaluations and 'attitudinal' surveys). Interviews were discussed specifically in relation to outcomes focusing on teachers' confidence and skills, and learners' motivation and skills.
• Medium Term: the most commonly reported measures were attainment data and qualitative measures. Assessments of attainment were often undertaken before and after an intervention. Data sources included questionnaires, focus groups and parent evaluations.

Health and Wellbeing

8.14. In general, a wide variety of measures were reported by authorities to measure outcomes in Health and Wellbeing interventions, perhaps reflecting the diversity of interventions.

8.15. The variety of measures used was reflective of the range of Health and Wellbeing interventions, ranging from nurture, to mental health, to physical activity. Broadly, most outcome measures were qualitative in nature, including interviews, focus groups (with staff, pupils and carers), surveys, observations (including analysis of video footage) and self-evaluation measures (such as 'How Nurturing is our School?'). Specific measurement tools varied, with some more common tools being Boxall profiling and resources such as My Class Inventory. In addition, quantitative measures included school statistics (such as school exclusion and attendance rates) and measures of physical activity levels.

8.16. Below are some examples of the methodologies and data sources employed both in the short and the medium term by the range of different local authorities.

• Short Term: Measures of short term outcomes in Health and Wellbeing were similar to measures of Literacy and Numeracy in relation to their use of qualitative data (questionnaires, interviews and observations). In addition, measures described for Health and Wellbeing interventions included three additional themes of evidence of 'action', feedback, and Health and Wellbeing assessments. Collecting evidence of action through counts included the numbers of staff accessing training, the number of children identified to receive interventions, and the number accessing services provided through interventions. In addition, authorities mentioned gaining feedback from a number of sources, including external collaborators, schools, headteachers, parents and service users. Assessments included, amongst others, the 'Strengths and Difficulties Questionnaire', 'My Class Inventory' and 'Boxall' profiling.
• Medium Term: Where authorities provided specific information about measures, three main themes were identified in addition to qualitative measures; feedback, Health and Wellbeing assessments and attainment measurements. 'Feedback' was used mostly to measure outcomes relating to pupils, including general improved wellbeing and specific outcomes such as improved relationships, self-esteem and engagement. Assessments of Health and Wellbeing included 'self-evaluation' activities by schools and pupil measures. This was done for example using for example Boxall profiling.

What does the evidence show on the extent to which each of the different types of interventions achieved their short and medium term outcomes?

8.17. Reported progress towards achieving outcomes varied across authorities. Progress reports submitted differed, with some Challenge Authorities providing more detail than others. The qualitative research also gathered stakeholders' views of progress towards short and medium term outcomes.

8.18. On the whole, progress made in the primary programme was documented to a greater extent. Some highlights of specific individual cases for the primary programme are provided below:

Figure 8.7: Success of interventions in the primary programme

8.19. Generally, reports submitted by authorities contained no specific detail as to the size of the reported increases in the primary programme.

8.20. Progress in the secondary programme was very limited, as on the whole it was felt by authorities that it was still early days to see any impact. It should be noted that by the end of Year 2, secondary schools had only benefitted from the fund for one school year.

8.21. The qualitative research explored stakeholder views on progress towards short and medium term outcomes. Specifically, stakeholders were asked to comment on the extent to which interventions had improved teaching skills, leadership, parental engagement and home learning. Taking each of these in turn:

• Overall, there was a clear belief across the range of stakeholders that interventions had improved teaching skills. It had provided access to training, encouraged reflection on skills, increased professional dialogue and improved collaboration.
• Stakeholders agreed that the fund was creating leadership opportunities, with opportunities to mentor, network and lead on new approaches.
• The qualitative research saw a more mixed view around progress made in parental engagement and home learning. Many stakeholders interviewed indicated that parental engagement was a challenge, and that it continued to be 'work in progress'.

8.22. Many stakeholders interviewed in the qualitative research felt that it would take longer to see and measure tangible outcomes. However, they felt that groundwork (in terms of data use, targeting and understanding the aims of the fund) had been laid in order to achieve outcomes in the coming years.

8.23. Three factors identified by some authorities in progress reports as contributing positively towards achieving the outcomes were:

• early engagement with staff giving strategic responsibility for implementation
• collaboration through clusters and schools; and
• improvement methodologies