Attainment Scotland Fund 2019: evaluation

3. Findings

Governance

1.9. Local authorities were asked for their views on what they thought was working well and what could be improved in their experience of working with Education Scotland, including Attainment Advisers, and with Scottish Government.

Education Scotland

1.10. In terms of working with Education Scotland (ES), local authorities were positive overall of their experience. Key highlights included:

• Advice, guidance and support;
• National Improvement Hub resources;
• Conferences and events;
• Positive relationships between local authority and ES;
• Strategic changes and restructure of ES welcomed, such as move to ensure ES resource more accessible outwith the central belt.

1.11. Some local authorities referenced more limited engagement or involvement with Education Scotland. A number of areas for improvement in working with Education Scotland were also suggested. This included the need for advice and guidance on impactful approaches to help headteachers achieve best value; greater understanding of rural issues/rural poverty; addressing challenge of taking forward small scale initiatives, such as case studies which demonstrate good practice in rural areas. There was hope that the structure and support offered by the newly formed Education Scotland regional teams would enhance provision of support and challenge between ES and local authorities, with Attainment Advisers becoming part of a wider regional team.

Attainment Advisers

1.12. Overall, local authority responses were similarly broadly positive with regard to the role of Attainment Advisers. Support from Attainment Advisers was welcomed in terms of Career Long Professional Learning (CLPL), data analysis, strategic planning, policy development support, and direct work with schools and learning communities. One local authority response described the support as 'invaluable'. Another described the role of Attainment Advisers as '…very supportive and provided advice and Professional Learning on closing the poverty related attainment gap, including developing strong, measurable outcomes which has helped in evidencing impact'. Similarly, support was variously described as: 'excellent', 'strong and focused', and 'very proactive in seeking ways to support at both school and LA level'.

1.13. Good working relationships were also highlighted by a number of local authorities with regard to Attainment Advisers. For example, one local authority response described the Attainment Adviser as '…[a] core part of our senior leadership team and has influenced policy and practice on a variety of levels'. Another described: 'excellent working relationship which has been invaluable to sustaining and maintaining the project within [local authority]'. The Attainment Adviser role was also viewed as providing a national perspective and bringing this into the local level, including brokering links outwith the local authority and enabling sharing of practice across schools and the local authority.

1.14. In terms of areas for further improvement, several local authority respondents reported some challenges regarding availability of expertise and resource from Attainment Advisers in terms of:

• Inconsistency of staffing/periods without an Attainment Adviser in post;
• Lack of clarity of role of Attainment Adviser (one local authority noted a more detailed remit would be helpful);
• Limited capacity of Attainment Adviser in terms of time available. This was mentioned by several non-Challenge Authority local authorities.

1.15. Several local authority respondents described improvements emerging in terms of work with Attainment Advisers, for example where stability of staffing was improving or where the additional time available from an Attainment Adviser was very much welcomed by local authorities, for example to support measuring the impact of PEF.

Scottish Government

1.16. Local authority perspectives on what was working well and what could be improved in working with Scottish Government were also broadly positive. In terms of what was working well, the following themes emerged from responses:

• Clear communication;
• High profile local events/sharing events to support networks and build capacity;
• Commitment to resources in terms of Attainment Advisers;
• Input to 'national direction';
• Relationships/face-to-face meetings and attendance, such as showcase events;
• Clear administration and reporting procedures/reducing bureaucracy (eg 'increasing refinement of the reporting and change request processes');
• Visits to schools were noted as positive, as was advice and guidance at SAC school meetings from Scottish Government representatives;
• Support from local Improvement Adviser noted by several respondents;
• Tools to support data use, such as the BGE toolkit.

1.17. A number of areas for improvement were also raised with respect to working with Scottish Government. Despite efforts to reduce bureaucracy being welcomed, one respondent noted that schools still feel 'quite heavily scrutinised in the process'. The potential for communication issues, particularly given the number of individuals and departments involved, was also raised. The need for clear guidance on timescales and key dates at the beginning of the year was noted, and for further opportunities for good practice sharing and networking. For example, as one local authority response highlighted, there was a need for '…more opportunities for networking around strategies which are demonstrating real impact'.

1.18. A number of wider concerns were also raised with respect to governance:

• Staffing: recruitment and retention pressures were noted, although there was considerable geographical variation in terms of local authorities who raised this as a concern. Local authorities outwith the central belt were more likely to raise this;
• Resource availability emerged as a concern, particularly amongst non-Challenge Authorities. This was related to a perception of the development of a 'two tier' system by a small number of authorities.

Approach for addressing poverty-related attainment gap

1.19. Local authorities were asked about their perception of the extent to which the approach within their local authority for addressing the poverty-related attainment gap had changed over the period of funding. Of twenty-seven local authority responses to this question, twenty viewed their approach as having changed either significantly (4) or to some extent (16). A further seven viewed their approach as having changed to a limited extent. None indicated there had been no change of approach.

1.20. In terms of Challenge Authority responses to this question, two indicated their approach had changed significantly, three changed to some extent and four limited change.

1.21. When asked why approaches had changed or not changed, overall there was emerging evidence of a shifting focus and streamlining of approaches towards approaches where there was evidence of effectiveness and impact. Improvement was a driver for change. Self-evaluation of effectiveness and impact of intervention programmes were viewed as the basis of decisions for continuation, amendment or discontinuation.

1.22. Emerging enhancements and adaptations in approaches were based on:

• Increasing use of broad range of data;
• Focus on measuring impact;
• Building sustainability.

1.23. One local authority viewpoint illustrates this combination of use of data, building on experience and evidence of impact: '... a shift in focus from use of a wide range of interventions to narrowing the range of approaches .[..]. informed by our experience and evidence gathering over the past 3 years' (Schools Programme local authority). Another highlighted the focus on interventions with evidence of impact: '…working to better connect successful interventions to plan our exit strategy and build sustainability. The aim is to build an empowered and connected system of education' (Challenge Authority local authority).

1.24. For those authorities which indicated limited or no change in approach, this appeared largely due to continuation of an existing approach. For example, one local authority specifically mentioned a programme to tackle poverty which had been in place prior to introduction of ASF.

Data and evidence

1.25. Local authorities were asked to provide an assessment of the extent to which data and evidence relevant to the local context had featured in decision-making processes regarding the Attainment Scotland Fund at the local authority level. Twenty-seven local authorities responded to this question, of which twenty-six indicated data and evidence featured in local authority decision-making either to a great extent (20) or to some extent (6). One local authority indicated data and evidence featured to a limited extent, and no authorities indicated data and evidence had not featured at all. All of the Challenge Authorities indicated that data and evidence had featured to a great extent in their decision-making.

1.26. Local authorities were invited to provide further detail in an open text follow up question. Analysis of open text responses indicate evidence of increasing use of and focus on data to support decision-making in terms of approaches and interventions. There was evidence of:

• Improved capacity;
• Improved capability;
• Increasing focus on local data;
• Improvements in data quality.

1.27. In terms of availability and accessibility of data, responses suggest the development of increasingly rich data environments, with local authorities engaging in combining data from different sources and different levels within the system. However, there were also indications that the volume of data was potentially overwhelming for some local authorities.

1.28. A range of mechanisms for using data were emerging. Responses also indicated the benefits of enhanced use of data and evidence, including: targeting resources for improvement; driving change/improvement of approaches; identifying gaps/priorities/areas to be addressed; measuring impact; data at the local level: tailoring planning to individual context; confirmatory value/accountability.

1.29. There was varied use of data across levels within local authorities as well as variation in terms of the extent of use of data between local authorities.

Collaboration

1.30. Perceptions of the extent of collaboration as a result of the fund emerged strongly from local authority responses, with twenty-two out of twenty-seven respondents indicating that the fund had increased collaboration either to a great extent (10) or to some extent (5). Seven (out of nine) Challenge Authorities perceived the fund has increased collaboration to a great extent. The remaining two Challenge Authorities perceived the fund has increased collaboration to some extent.

1.31. Analysis of open text follow up responses with regard to collaboration indicated:

• Funding was viewed as a driver of collaboration. This included PEF funding being viewed as a driver of collaboration in PEF-only authorities.
• Range of mechanisms developed to support more strategic and systematic collaboration in areas such as CLPL, leadership training, capacity building, local support networks;
• Collaboration enabling sharing of practice;
• Collaboration at different levels of the system and between different levels of the system (within schools/between school clusters/within local authorities/between schools and partners including third sector and universities/between local authorities);
• Benefits of collaboration identified (particularly enabling making best use of resources);
• Whilst collaboration appeared to be broadly viewed as positive, concerns and barriers regarding collaboration also exist (eg time consuming; staff availability).

1.32. An illustration of increased opportunities for collaboration at different levels, with a focus on improving outcomes, is provided in the following quote: 'The fund has brought new opportunities for teams to work together, particularly in literacy and numeracy and HWB. Our cross-council working has also increased and we have been able to develop new relationships with other services to strengthen outcomes for young people' (Challenge Authority response).

Sustainability

1.33. Local authorities were broadly positive regarding the extent to which the focus on closing the poverty-related attainment gap would be sustainable beyond the years of the fund. Of twenty-four local authorities who responded to this question, nineteen stated that the focus would be sustainable to either a great extent (8) or to some extent (11). A further five indicated the focus would be sustainable to a limited extent. There were no responses which indicated the focus would not be sustainable at all beyond the years of the fund. Seven Challenge Authorities responded to this question, of whom three viewed the focus would be sustainable beyond the years of the fund to a great extent and four to some extent.

1.34. A range of actions were described which were being taken to support/encourage/plan for sustainability. These ranged from broad statements affirming a commitment to sustainability (eg 'continued focus on plans for sustainability'), to more specific actions both at local authority and at school level. There was evidence both of strategic approaches and actions, as well as actions at a more local level, with regard to sustainability.

1.35. There were numerous mentions in local authority responses of schools being asked to clearly consider sustainability, and of sustainability being a key feature of discussions within schools. Examples were given of schools considering the sustainability of interventions when planning and implementing PEF in their school, with support of Attainment Advisers, and of schools being asked to consider using PEF in sustainable ways. There was also evidence of mainstreaming plans, both at local authority and school level. Sustainability was being considered at many levels, including within specific initiatives. One local authority provided an example of the role of a Quality Improvement Officer working to encourage and suggest items which might be a better use of funding and would create sustainability.

1.36. There was emerging evidence of the prioritisation of interventions with the greatest impact by local authorities, as mentioned by several respondents. One response described the local authority's use of data to understand approaches which are most impactful to ensure these are continued as follows: '…investing in approaches and initiatives which are underpinned by a strong evidence base'.

1.37. Several local authority responses specifically alluded to an exit strategy. For example, one respondent noted that planning was being undertaken together with Human Resources (HR) for an exit strategy. Another mentioned workforce planning which was at an 'early stage' but key principles for sustainability had been developed.

1.38. Local authorities were also invited to suggest factors which will impact on the sustainability of focus of the approach. Staffing and budget were predominantly mentioned, with concerns related to sustainability following withdrawal of funding. However, responses also indicated that collaboration, 'pooling' of resources, sharing of good practice, building capacity and focusing on staff professional development would be sustainable to some extent.

Use of resources

1.39. Local authorities were invited to indicate what changes (if any) had been introduced in terms of how the local authority used all its resources, including core education funding, to improve outcomes for pupils from the most deprived backgrounds. Twenty-three out of twenty-eight local authorities responded to this open question, with a number of themes emerging from responses. Four indicated no change, or no significant change, in how resources were used. Several others referenced already pre-existing focus or priority on equity, and others noted a shift in response.

1.40. Similar to themes which emerged in the 2018 local authority survey, there was evidence of a greater focus on deprivation and of a more collaborative focus in terms of the use of resources. However, there were also some concerns emerging in terms of the use of resources. These were primarily in terms of: reduction in central resource; potential for emerging differentials between schools in terms of available resources.

Pupil Equity Fund: planning, implementation and support

1.41. The survey questionnaire included a number of questions regarding local authority perspectives on planning, implementation and support with regard to Pupil Equity Funding (PEF) in 2018/19.

1.42. In terms of perceptions of what had worked well during the process of planning and implementing PEF in 2018/19, numerous themes emerged from responses:

• Local and national guidance for headteachers;
• Attainment Adviser role in supporting schools;
• Pooling resources and building partnerships;
• Collaboration;
• Increased use of data and evidence/more time to consider evidence base;
• Development of tools and approaches to support schools;
• Introduction of roles to provide co-ordination function (eg PEF officer roles; PEF co-ordination roles);
• Building on experience of previous year;
• Support from central services (HR, finance);
• Greater confidence at school level (eg schools 'coping better with responsibility and accountability').

1.43. Views on barriers encountered in the process of planning and implementing PEF were also gathered. The most commonly mentioned barrier related to staffing. Recruitment challenges, staff availability and retention of staff were noted by many respondents. Other barriers frequently highlighted related to procurement, pressures and cost burdens on central local authority services (such as HR and finance), issues related to timing of PEF funding cycle and planning cycle, and uncertainty over longer-term funding linked to the temporary nature of posts funded through PEF. Local authority respondents also highlighted challenges which were perceived as arising as a result of PEF funding going directly to schools such as the emergence of a degree of overwhelm in some schools.

1.44. The level and nature of support provided to schools by local authorities in 2018/19 to aid schools' implementation of PEF in particular was also explored. The following were most frequently mentioned:

• Guidance (including translating national guidance into local guidance);
• Attainment Adviser support;
• Support from central team;
• Co-ordination function (such as the creation of PEF principal officer role);
• Sharing of good practice;
• Improvement events and training;
• Central support (such as finance, HR, legal);
• Undertaking PEF audits/thematic reviews;
• Support with data use and data analysis;
• Facilitating partnerships and collaboration.

1.45. A number of specific models/mechanisms created by local authorities were also mentioned (eg PEF 'surgeries', developing bespoke authority-wide tracking and monitoring).

1.46. A number of local authority responses expressed a viewpoint that a more centralised funding and support model was required in terms of PEF, as illustrated by the following quotes:

'A central overview is the only way to ensure that schools are fully supported to be able to take forward the work they have agreed in their local communities. This must be recognised in the future and appropriately resourced'.

'… greater central funding would have provided a more stable source of central support'.

1.47. Authorities in receipt of Challenge Authority or Schools Programme funding were asked a supplementary question regarding their views of planning and implementing PEF alongside Challenge Authority/Schools Programme funding. Nineteen local authorities responded to this question. Several key themes emerged: increased integration/alignment between SAC and PEF; important role of Attainment Advisers supporting schools in terms of using SAC and PEF funding effectively; some tensions in terms of planning cycles and concerns regarding headteacher workload.

Factors supporting progress and mitigating (inhibiting) progress towards closing the poverty-related attainment gap

1.48. Local authorities were asked their view of factors which were supporting progress towards closing the poverty-related attainment gap, and factors which were mitigating or inhibiting progress. In terms of factors supporting progress, respondents referenced a range of structures, tools and processes including new central structures, new posts, greater use of data/improved data literacy, increased collaboration, and leadership. Also mentioned were greater emphasis on strategic links (such as to child poverty strategies), and greater attention, focus and understanding of the impact of poverty. Several respondents mentioned specific models and approaches. These included, for example, the introduction of specific approaches, creation of procurement frameworks, and increasing focus on working with families and communities.

1.49. A range of mitigating factors were also highlighted by respondents. Largely these referenced issues mentioned previously in survey responses, and included recruitment difficulties, staff turnover, funding uncertainty in the longer term, and levels of data literacy. However, responses also highlighted broader considerations related to addressing the poverty-related attainment gap including: the scale of the challenge in terms of levels of poverty and need for wider community/societal responses; factors related to rurality and rural poverty, including concerns regarding measurement of poverty in rural areas using existing SIMD measure; perceived inequity across schools and local authorities depending on funding stream; recognition of length of duration required and continuation of support required to address the scale of the challenge.

Care Experienced Children and Young People Fund: planning and implementation

1.50. Twenty of twenty-four local authority respondents were of the view that the Care Experienced Children and Young People (CECYP) Fund supported strategic decision-making to improve attainment or outcomes for care experienced children and young people either to a great extent (9) or to some extent (11). A further four viewed the CECYP Fund to have supported strategic decision-making to a limited extent.

1.51. There was recognition that it will take time to consult, plan, develop understanding and buy-in, and therefore it is too early to give more than initial indications of progress towards planning and implementation of the CECYP Fund. However, respondents' viewpoints were broadly positive and reflected, for example, the increased focus on care experienced children and young people and links to existing local authority priorities (eg Children's Services Plans).

1.52. There was also evidence of progress in terms of developing structures, processes and approaches. Models (eg Virtual Headteacher) and initiatives (eg MCR Pathways) were highlighted, as well as appointment of specific posts, such as CECYP support workers. There was also emerging evidence of decision-making informed by analysis of research and data, and of collaboration and sharing of good practice (eg CELCIS). As one local authority response described, the CECYP Fund represented '…[a] creative new approach to supporting Care Experienced Young People' at the local authority level.

Unintended consequences

1.53. Local authority respondents were asked to indicate positive and negative unintended consequences emerging as a result of the fund. A number of positive unintended consequences were reported by local authority respondents, including:

• Partnership working and collaboration;
• Culture change;
• Improved data use;
• Increasingly skilled classroom practitioners;
• Increased understanding of poverty and its impact.

1.54. Negative unintended consequences were also reported by respondents:

• Workload;
• Reporting demands/bureaucracy;
• Staffing concerns;
• Level of funding in more affluent areas where there is still need/hidden poverty;
• Concerns about future sustainability.

Any other comments

1.55. Finally, respondents were invited to provide any further comments on their experience of the Attainment Scotland Fund. Responses included:

• Need for further consideration of rurality in relation to equity going forward, including issues in terms of measurement;
• Need for further 'joining up' of agendas at the national level (eg with Speech and Language Therapy in Early Years);
• Need for ongoing consideration of how funding is allocated both in terms of the balance between local versus central allocation and across local authorities. For example, one local authority respondent described this as the need for '…[a] better balance of using the funds to empower the whole education system, this means all (local authorities) need some resource as well as maintaining some personalisation at school level';
• Need for consideration and planning over longer term;
• Need for consideration of Children's Services in planning for poverty-related attainment improvements; more social work support and family/parental involvement support for headteachers;
• Ongoing concerns related to sustainability and how to embed the approach.