Scottish Survey of Literacy and Numeracy 2015 (Numeracy)

Chapter 2: Assessment of numeracy

• Numeracy performance was highest at P4 and P7 with 66 per cent of pupils at each stage performing well or very well. At S2, 40 per cent of pupils performed well or very well.
• Boys outperformed girls at S2. At P4 and P7, there was no difference in performance of boys compared to girls.
• Pupils from the least deprived areas performed better in numeracy than other pupils, across all stages.

Figure 1 provides an illustration of a question taken from one of the Second Level pupil-teacher interactive assessments which focuses on the 'measurement' organiser. The figure contains the data sheet and accompanying questions.

Figure 1: Example of a Second Level (P7) practical assessment

Figure 2 provides an illustration of a short answer task taken from one of the Third Level written numeracy booklets. This task focuses on the 'money' organiser.

Figure 2: Example of a Third Level (S2) question from a written numeracy booklet.

The following analysis is taken from the results of pupils completing all three elements of the assessment (two written booklets and one pupil-teacher interaction). Effective sample sizes on this basis were 3,381 pupils in P4, 3,368 in P7 and 3,737 in S2. Results were weighted to account for different school sizes, a small number of non-participating schools and gender and deprivation differences between the sample and the population.

Pupils were assigned to one of four reporting categories based on the percentage of questions they answer correctly. Table 2.1 describes the categories used for each performance level in the numeracy assessment. These cut-off scores were set in consultation with Education Scotland, SQA and teachers, based on professional judgement and an analysis of the tasks involved in the assessment.

Table 2.1 Cut-off scores for SSLN numeracy reporting categories

SSLN numeracy reporting category	Percentage of items successfully completed in the SSLN
Performing very well at the level	75 per cent or more
Performing well at the level	50 per cent or more, but less than 75 per cent
Working within the level	P4: more than 9 per cent, but less than 50 per cent
	P7: more than 19 per cent, but less than 50 per cent
	S2: more than 34 per cent, but less than 50 per cent
Not yet working within the level	P4: 9 per cent or less
	P7: 19 per cent or less
	S2: 34 per cent or less

For example, pupils answering 75 per cent or more questions correctly across all tasks are described as 'performing very well at the level'. As the assessments are designed to cover the full range of outcomes within the curriculum at a given level, such a pupil might be expected, in general, to achieve at least 75 per cent across all tasks at their level. Pupils described as 'working within' a level can achieve some of the outcomes expected for their stage, but are still working on achieving the others.

For each stage, the differing cut-off scores between 'working within the level' and 'not yet working within the level' were determined by assessing the number of marks that could potentially be obtained in the assessment using only skills acquired at the previous level or, in the case of multiple choice items, by chance.

2.1 Overall distribution

Chart 2.1 displays the proportions of P4, P7 and S2 pupils in each reporting category (as defined in section 1.3).

Chart 2.1: Performance in numeracy, by stage and reporting category

At P4 and P7, 66 per cent of pupils performed well or very well when assessed against the relevant curriculum level for their stage (as defined in section 1.3). At S2, 40 per cent of pupils performed well or very well.

Though P4 and P7 saw the same proportion of pupils performing well or very well overall, P7 had the highest percentage of pupils performing at the highest level, with 26 per cent performing very well in numeracy (22 per cent of P4 pupils).

The proportion of pupils not yet working within the level was greatest for S2 pupils at 36 per cent, higher than in the primary sector where there was less than one per cent of P4 pupils and three per cent of P7 pupils not yet working within the level.

The difference in performance between the primary stages and S2, as highlighted in these results, is consistent with other educational evidence. A drop in performance associated with the transition from late primary and early secondary has been documented previously in the education sector, both in Scotland and elsewhere. This is apparent in the latest P7 and S2 results. Whilst there are only two years of schooling between these stages, the results show a substantially lower proportion of pupils working well or very well at the respective level by S2, though it should be noted that pupils are expected to reach Third Level (the level at which S2 pupils are assessed) by the end of S3, rather than S2.

Chart 2.2 shows the distribution of scores for each of the three stages assessed. The chart shows that the peak in P4 scores was around a score of 60-69 per cent (20 per cent of pupils achieved this score). The distribution of P7 scores was similar with a peak at 70-79 per cent (18 per cent of pupils). The pattern of S2 scores was flatter, concentrated around a score of 20-59 per cent, with a slight peak at 30-39 per cent.

Chart 2.2 Distribution of scores in each stage

In line with the assessment results, primary pupils reportedly also have a slightly higher perception of their numeracy ability, as shown by the SSLN pupil questionnaire results. Pupils in all stages were asked if they 'usually do well working with numbers': 93 and 90 per cent of P4 and P7 pupils agreed with the statement respectively, whereas in S2 this was 83 per cent.

2.2 Attainment by gender

Boys outperformed girls in numeracy at S2, with the proportion of boys performing well or very well five percentage points higher than that of girls. The differences in the proportions of girls and boys who performed well or very well were not statistically significant at P4 or P7.

Chart 2.3 Proportion of pupils performing well or very well in numeracy, by stage and gender

The difference observed in pupil performance between the primary stages and S2 is also evident in the results for both girls and boys. This suggests both genders may be affected by the challenge of transition from primary to secondary.

A differential in numeracy ability between boys and girls is also observed in the OECD's Programme for International Student Assessment ( PISA) results. PISA surveys 15 year olds (generally S4 pupils). Results from the most recent surveys, 2009 and 2012, found that boys performed statistically significantly better than girls in the Mathematics assessment.

2.3 Attainment by deprivation

The proportion of pupils who performed well or very well was higher for pupils from the least deprived areas than pupils from the middle and most deprived areas, across all stages.

Chart 2.4 shows the difference in performance by deprivation category. Across all stages the proportion of pupils who performed well or very well increased through the deprivation categories (from most, to middle to least). This results in a stepped pattern where performance improves as level of deprivation declines.

The performance gap (the difference between the proportion of pupils from the least and most deprived areas performing well or very well) in the primary stages was 21 and 22 percentage points (P4 and P7 respectively), whereas the largest gap was at S2 at 28 percentage points.

Chart 2.4 Proportion of pupils performing well or very well in numeracy, by stage and deprivation category

2.4 Attainment by numeracy organiser and type of task

The SSLN questions are categorised in line with the CfE experiences and outcomes for numeracy e.g. 'measurement', 'time' etc. For all stages the highest proportion of questions answered correctly were 'data and analysis' questions (and jointly, at S2, 'number and number processes' questions) (Chart 2.5). The questions pupils found most challenging were questions on 'fractions, decimal fractions and percentages' (P4 and P7) and 'measurement' (P7 and S2). In line with the overall attainment results, the performance at S2 was lower across all organisers than at the primary stages.

Chart 2.5 Proportion of correctly answered questions, by stage and organiser

The numeracy assessment consists of three task types: short answer, multi-item and pupil-teacher interaction tasks (as described in section 1.2). Pupils in P4 performed better at multi-item tasks, with 64 per cent of these tasks answered correctly (Chart 2.6); P7 pupils were most successful in pupil-teacher interaction tasks (64 per cent correct). For S2 both multi-item tasks and pupil-teacher interaction tasks had the highest proportion of correct answers at 47 per cent. At all stages, pupils performed least well on short answer tasks.

Chart 2.6 Proportion of tasks answered correctly, by stage and task type

2.5 Mental maths

The first section of the pupil-teacher interaction assessment included four mental maths questions covering each of the four numeracy operators: addition, subtraction, multiplication and division. Two questions were written on prompt cards for the pupils to see, the remaining two questions were read out to the pupil by the teacher.

At all stages, the percentage of correct answers was highest for mental maths questions where the operator was addition, with 68 per cent (P4), 70 per cent (P7) and 61 per cent (S2) of addition questions answered correctly (Chart 2.7). Multiplication questions were also well answered by P7 pupils, with 69 per cent of these questions answered correctly.

Chart 2.7 Proportion of mental maths questions answered correctly, by stage and numeracy operator

In the primary sector, mental maths questions based on division saw the lowest proportion of correct answers, at 44 and 49 per cent (P4 and P7 respectively). In S2, there was not the variability between the non-addition questions as there was in the primary sector. As shown in Chart 2.7, the percentage of non-addition questions answered correctly was around 40 per cent.