Measuring the attainment gap: consultation

National Improvement Framework: Consultation on measuring the attainment gap and milestones towards closing it

1. The Scottish Government has been clear about its commitment to closing the poverty-related attainment gap between children and young people from the least and most disadvantaged communities. Ministers are committed to making demonstrable progress in closing the gap during the lifetime of this Parliament, and to substantially eliminate it in the next decade. We have undertaken to consult on proposals for measuring the gap and milestones towards closing it, and to publish our proposals as part of the 2018 Improvement Plan in December.

Measuring the gap

2. As the National Improvement Framework comes fully on stream, there is a wealth of data available from which to determine the gap, or gaps we intend to measure. These data make clear that there is a gap in achievement/ attainment/development between children and young people from the least and most disadvantaged backgrounds across the system measures that we have e.g. health and wellbeing, attendance, achievement of CfE levels, national qualifications and 16-19 year olds participation measure. It would therefore be possible to use one or more of these measures as the indicator(s) against which we measure progress.

3. Following discussion with stakeholders and analysis of approaches taken in other jurisdictions, we do not believe that it is realistic to assess the performance of our system via a single measure. Such a measure, e.g. one focused on senior phase or leavers' data, will not be sufficient to demonstrate progress if we are trying to measure the impact of the system as a whole. A single measure could also generate perverse behaviours by becoming the single focus of activity in schools. Nor do we favour the approach of using a complex algorithm to bring together a range of measures to produce a small number of indicators of progress – such an approach is neither straightforward nor transparent.

4. For that reason, we have based our proposals on using a range of measures that reflect the breadth of issues that can impact on attainment. Our proposal is to identify a basket of key, mainly attainment measures supported by a set of sub-measures which include detailed attainment measures and factors known to have an impact on attainment. These measures will allow us to assess progress in closing the attainment gap across the 3-18 age range. This is a complex area and therefore having relevant measures that are agreed and clearly defined for the correct range of areas that we want to focus on is considered the most effective way forward.

5. The sub-measures proposed include aspects of health and wellbeing which are known to impact significantly on a child's ability to do well at school. Health and wellbeing is a multi-facetted concept and only a small number of measures could be included, the choice of which was partly determined by data availability.

6. It is proposed that we identify 8 key measures, supported by 17 sub-measures of detailed attainment measures and of factors known to impact attainment. The suggested measures are set out in Tables 1 and 2 and discussed in more detail in paras 14-28 below. The choice of measures may be revised as national data collection evolves and develops.

7. More broadly, we will also consider information from other sources to help inform improvement, rather than for measuring the gap. In particular, it would be helpful to analyse information from each local authority and (once they are established) each regional improvement collaborative, as this may help to identify geographical difference and particular local practices which are having a positive impact on closing the gap. Local authority and regional data will also help to identify areas where progress is slower than expected and therefore help to provide targeted resources and support. This will be key information for the new Scottish Education Council which will be established in October 2017.

Milestones

8. It is also essential to have clear milestones to measure whether, and how quickly, the gaps in achievement between the most and least disadvantaged children and young people are closing.

9. We are not proposing to set milestones simply related to reducing the gaps identified for each measure, e.g. a 25% reduction in that gap by 2020. While achieving this milestone would demonstrate that particular gap was closing, it would not necessarily mean that attainment was increasing.

10. It is suggested therefore that the most effective way of measuring progress is to use stretch aims, similar to those set out in the Children and Young People Improvement Collaborative, which set aims that reflect improvement in every Scottish Index of Multiple Deprivation ( SIMD) quintile. Stretch aims for improvement purposes are specifically focussed on the improvement which a system needs to make in order to reach a particular goal ( i.e. closing the gap) – they do not generally articulate the goal itself, although achieving the aims would also mean significant steps towards achieving the goal.

11. The graphs in Annex A suggest possible stretch aims for each of the 8 key measures to show what this would look like and the positive impact achieving them would have on closing the gap.

Principles

12. Our proposals are based on a number of key principles:

• we are looking at the difference in attainment between those children and young people from SIMD quintiles 1 and 5. However, we recognise the importance of increasing attainment for all children and are therefore proposing to set stretch aims for all 5 SIMD quintiles.
• focussing on a single measure is neither helpful or meaningful and would provide a false and limited picture
• measures and milestones should be relatively simple to measure and report against
• there needs to be a clear line of sight from the agreed measures and milestones to the priorities set out in the National Improvement Framework
• there should be a focus on literacy, numeracy and health and wellbeing
• the focus should be across the age ranges – from 3-18
• they should be a credible set of measures – understood to fairly reflect progress in closing the poverty related attainment gap
• the need to avoid perverse incentives through whatever milestones or stretch aims are set.

Q1: Have we based these proposals on the right principles?

Most and least disadvantaged

13. It is proposed to use SIMD data to identify the most and least disadvantaged children and young people. There is no direct measure of poverty available within the attainment data. Of the two measures that could be used, free school meals ( FSM) registration and SIMD, SIMD is considered more appropriate. FSM registration is a binary measure and as such provides limited information on the relative position of pupils, whereas SIMD is a more nuanced measure. Analysis by SIMD quintiles allows us to look both at the gap between the most deprived, and attainment levels within all five quintiles. SIMD is not a measure of poverty, it is a measure of area-based deprivation and we recognise that there may be some concerns about using it for these purposes, for example a concern that this approach is not sensitive to those children and young people from very disadvantaged backgrounds living in more affluent quintile areas. It is our view, however, that it is preferable to using FSM registration. SIMD quintiles are already routinely used in reporting Scottish Government official statistics by deprivation.

Key measures

14. It is proposed that there is a basket of 8 key outcome measures specifically on the achievement and attainment of children (and the associated 'gap') ( Table 1). In addition, we propose to have a set of 17 supporting sub-measures, which provide further detail on the attainment measures and also related measures on factors known to influence attainment ( Table 2).

15. Our expectation is that a basket of 8 measures will give a broad enough picture of the attainment gap from early years to school leavers and, importantly, would be relatively simple to measure and report against. Having fewer indicators would risk losing important information about how the gap changes e.g. from P1 to P7, and how the drivers for improvement are affecting the gap at each key stage. Having additional measures would risk increasing the overall complexity of measuring the gap, and would run the risk of being too complicated and, consequently, reduce the value and transparency of the results.

16. Table 1 sets out the proposed key measures and, based on the most recent data we have (2015/16 for all measures except Participation, which is based on 2016/17), shows the gaps which we are looking to close.

Table 1

Measure	All children %	Most disadvantaged (bottom 20% SIMD) %	Least disadvantaged (top 20% SIMD) %	Gap (percentage points)
27-30 month review (Children showing no concerns across all domains)	63.7	54.8	71.7	16.8
Primary – Literacy* (P1, P4, P7 combined)	67.7	58.4	79.8	21.4
Secondary Literacy* (S3, 3rd level or better)	82.0	74.0	90.5	16.5
Primary – Numeracy* (P1, P4, P7 combined)	75.1	67.7	85.4	17.7
Secondary Numeracy* (S3, 3rd level or better)	85.8	77.2	93.7	16.5
SCQF 5 or above (1 or more on leaving school)	85.6	74.4	94.7	20.3
SCQF 6 or above (1 or more on leaving school)	61.7	42.7	81.2	38.5
Participation measure	91.1	84.8	96.3	11.5

*This data will not be used for baseline purposes, we will use 2016/17 BGE attainment data that will be published in December 2017.

Q2: Do you agree with having a basket of key measures to assess the progress made?

Q3: Are the proposed key measures the right ones?

Q4: Will this approach avoid the introduction of perverse incentives?

27-30 month review

17. While the data in the early years is not as rich as for school years, it is considered important to include some information that relates to children's development in the early years. Social, emotional and behavioural development is considered to be an integral part of achieving good outcomes and the impact of poor early development on later attainment is strong. It is proposed to use data on the gap in "developmental concerns" identified in the 27-30 month review. This is an annual return. The percentage of children for whom there are no developmental concerns recorded here is based on children who were entitled to a review in a given year. As such, this does not match how this information is published by ISD Scotland (who calculate the measure based on those children who had a review and exclude those who did not).

Attainment in the Broad General Education ( BGE)

18. Following the introduction of the National Improvement Framework we now have a lot of data about children and young people's progress in the BGE. We propose to use teacher judgement data on the achievement of CfE levels to show performance in literacy and numeracy in the BGE. We propose to create a single measure for primary literacy and one for primary numeracy, underpinned by the stage level sub-measures allowing progress at each stage to be tracked. It should be noted that for a child to be recorded as being at the expected level in literacy, they will need to have achieved that level in each of reading, writing and listening and talking which reflects the ambition we have for Scotland's children.

19. It is suggested that the only S3 indicator we should measure progress against is 3rd level, given the current expectation is that young people should definitely be achieving at least 3rd level by the end of S3.

20. From the 2017/18 school year, teachers' professional judgement will be informed by the Scottish National Standardised Assessments ( SNSA). SNSA results will not be used to measure the gap or progress towards closing it. These are diagnostic assessments that teachers can use at any point throughout the year to inform their professional judgement. They only cover some aspects of literacy and numeracy and do not assess the full breadth of CfE levels.

Q5: Is 3rd level the right measure to use of attainment at S3?

Senior phase attainment

21. Our preferred measure of achievement in the senior phase is the qualifications achieved by young people at the point which they leave school. It is clear from existing data that the gap in achievement between the most and the least disadvantaged children is wider the higher the qualification involved, as shown in Table 1. We want this measure to be broader than just SQA qualifications, for example the inclusion of foundation apprenticeships, hence the use of SCQF levels.

22. In terms of measuring progress beyond school – the indicator we have used is the Participation Measure which reports on the wider activity of the 16-19 cohort, including those still at school. This is an indicator of school success in preparing young people for access to future work or study. It is noted however that this is not an explicit measure of attainment.

Q6: Does the use of SCQF levels reflect a sound approach to measuring senior phase attainment? Are there other options such as Insight tariff points?

Q7: How best we can give more meaning/clarity to the terms " SCQF 5" and " SCQF 6" so they are accessible to all.

Sub-measures

23. These 8 key measures will be supported by 17 sub-measures. These sub-measures cover attainment in literacy and numeracy at each of P1, P4, P7 and S3 to ensure that we have a picture of progress at each of these levels, as well as a number of measures known to impact on achievement, such as attendance, exclusions and health and wellbeing. These have not been included in the key measures to ensure that we have a manageable number, and because measures such as attendance ( etc.) are not direct measures of attainment. Consideration could also be given to the inclusion of additional sub-measures, for example the take up of pre-school places by eligible 2 year olds.

24. There is currently less health and wellbeing data available than attainment data. We have therefore been considering a range of options for enabling the gathering of detailed information from children and young people (in late primary and secondary stages) in relation to their self-reported health and wellbeing. We have been conducting a feasibility study, the outcome of which is likely to result in the development of an IT 'platform' that will primarily enable schools and local authorities to collect information from their local children and young people for improvement purposes. It would also be used by the Scottish Government to more effectively and efficiently capture information directly from children and young people in schools to assist with informing and monitoring government policies in this area.

25. In the meantime, given the priority the NIF places on children and young people's health and wellbeing, we plan to include currently available data in the sub-measures to indicate progress in social, emotional and behavioural development, and mental wellbeing, which are considered to be integral parts of doing well at school.

26. We are therefore proposing to include data about social, emotional and behavioural development of children and young people aged 4-12 years via the Strengths and Difficulties Questionnaire ( SDQ). We also propose to include data on young people's mental wellbeing from the Scottish Schools Adolescent Lifestyle and Substance Use Survey ( SALSUS), using the Warwick-Edinburgh Mental Wellbeing Scale ( WEMWBS).

27. While not attainment measures, we propose to have 7 of the 17 sub-measures covering child development and aspects of health and wellbeing, given the positive impact that these will have on attainment. It also recognises the importance attached to health and wellbeing as one of the NIF priorities. Similarly, we believe there is value in having an attendance sub-measure given the positive impact of attendance on attainment.

28. Table 2 – proposed sub-measures of detailed attainment measures and of factors known to impact on attainment. (2015/16 data)

Table 2

Measure	All children %	Most disadvantaged (bottom 20% SIMD) %	Least disadvantaged (top 20% SIMD) %	Gap (percentage points)
Detailed attainment sub-measures
P1 – Literacy*	75	67	86	19
P4 – Literacy	66	56	78	22
P7 – Literacy	62	51	75	24
P1 – Numeracy	84	78	92	14
P4 – Numeracy	73	66	84	18
P7 – Numeracy	68	58	80	22
Sub-measures known to impact on attainment
HWB: 27-30 month review uptake	88	86	89	3
HWB: Children total difficulties score (4-12 year olds)	14	22	6	16
HWB: Children total difficulties score (13 and 15 year olds)	31	34	26	8
HWB: Mental wellbeing score: 13 year old boys	51.4	49.9	52.6	2.7
HWB: Mental wellbeing score: 13 year old girls	48.2	47.2	49.1	1.9
HWB: Mental wellbeing score: 15 year old boys	50.1	49.3	50.6	1.3
HWB: Mental wellbeing score: 15 year old girls	44.4	43.7	45.8	2.1
Primary attendance rates	95.1	93.3	96.7	3.4
Secondary attendance rates	91.8	88.7	94.5	5.8
Primary exclusion rates (rates per 1000 pupils)	9.0	19.0	2.1	16.9
Secondary exclusion rates (rates per 1000 pupils)	49.6	95.2	15.2	80

*Literacy could be broken down further to reading, writing, and listening & talking separately

Q8: Are these the right sub-measures? Are there others that should be included?

Milestones and stretch aims for closing the gap

29. In looking to "substantially eliminate" the gap within 10 years, we need to see a significant difference within a short timeframe for each of the key measures, and so milestones need to provide a clear sense of what it is we are trying to accomplish, as well as the level of improvement that we want to see and by when. As discussed at paragraph 10 above, we propose to use stretch aims to measure the progress we are making.

30. Stretch aims differ from targets which are very specifically set out to support accountability and scrutiny. Properly articulated stretch aims are essential both to provide a guide for those delivering improvement and to provide evidence of the progress made. They set a challenging ambition that harnesses the energy and motivation of those participating and, given that transformational change is likely to be required, are not achievable by simply working harder/faster. They should be measurable in order for those engaged in the improvement work to use them to guide them on what works and for the wider system to evidence progress towards the desired outcome.

31. Using stretch aims in this way would assist the Scottish Government, local authorities and schools to develop and implement the most appropriate improvement activities to secure educational improvement for all children and young people in Scotland.

32. For simplicity, and to keep the main focus on attainment measures, it is proposed to set stretch aims for just the 8 key measures, not the sub-measures. Stretch aims already exist for the 27-30 month check and the BGE attainment measures as part of the outcome aims for the Children and Young People Improvement Collaborative http://www.gov.scot/Resource/0051/00517520.pdf – at least 85% of children within each SIMD quintile achieving each of the outcomes, e.g. early level literacy and numeracy by the end of P1. It was recognised when these stretch aims were published in November 2016 that they would need to be reviewed once more data was available.

33. The graphs at Annex A set out the proposed new stretch aims for the purposes of measuring progress in closing the attainment gap. The level of the 27-30 month check stretch aim is the same as that included in the CYPIC stretch aims. Having considered the 2015/16 BGE attainment data, it is expected that the levels at which CYPIC stretch aims are set will be revised in future. We will work with stakeholders for both the National Improvement Framework and CYPIC to ensure that where the same measures are used, the levels at which stretch aims are set are reflected consistently through both the National Improvement Framework and CYPIC.

34. This will be the first time that we have stretch aims for achievement in the senior phase and for the participation measure, although the CYPIC stretch aims currently include school leaver destinations which is similar to the latter.

Baseline data

35. We will use the most recent data we have as the baseline for this exercise, as per Table 1 above. The only exception to this is that we will use 2016/17 BGE attainment data that we will be publishing in the NIF Evidence Report alongside the NIF and Improvement Plan in December 2017.

Q9: Is the use of stretch aims, by SIMD quintile, the right way to set milestones?

Q10: Are the stretch aims set at the right level?