Programme for International Student Assessment (PISA) 2015: Highlights from Scotland's results

3. Performance in Science

Average scores

38. In science, Scotland's score of 497 in PISA 2015 was statistically lower than in the 2006 (515), 2009 (514) and 2012 (513) cycles, ( Table A.1). Comparisons are not possible with the 2000 and 2003 science scores. Chart 3.1 illustrates Scotland's scores since 2006 with the 95-per-cent confidence intervals ^[4] next to the scores for the OECD average. Note that Scotland, with a smaller sample, has larger confidence intervals than does the OECD average, where the combined sample makes for more certainty.

Chart 3.1: Comparison of Scotland and OECD science scores over time

39. Scotland's score was similar to the OECD average in 2015, having previously been above the OECD average since 2006. The OECD average itself fell significantly, from 501 points in 2012 to 493 points in 2015. The OECD average in 2015 was similar to 2006, when science was previously a main domain.

Comparisons with other countries

40. Compared to the 34 OECD countries ^[5] , plus the three other UK administrations, Scotland performed similarly to 13 countries, including Northern Ireland and the United States, and above 12 countries including Wales and Italy. Twelve countries performed above Scotland, including Canada, Germany, Korea and England. Table 3.1 below shows which countries were statistically significantly above, similar to and below Scotland in 2015. Table A.2a, located in the annex, shows each country's score.

Table 3.1: OECD countries and UK administrations, higher than, similar to and lower than Scotland in science

Higher score than Scotland	Similar score to Scotland	Lower score than Scotland
Australia	Austria	Chile
Canada	Belgium	Greece
England	Czech Republic	Hungary
Estonia	Denmark	Iceland
Finland	France	Israel
Germany	Ireland	Italy
Japan	Norway	Latvia
Korea	Northern Ireland	Luxembourg
Netherlands	OECD average	Mexico
New Zealand	Poland	Slovak Republic
Slovenia	Portugal	Turkey
Switzerland	Spain	Wales
United Kingdom	Sweden
	United States

41. This was a measure of relative performance. Broadly, a comparable number of countries perform significantly below Scotland in science as above. Chart 3.2 below illustrates the numbers of countries ( OECD members plus the UK administrations) that have been found to be significantly above, similar to and below Scotland in the comparable science assessments since 2006

Chart 3.2: Numbers of OECD countries and UK administrations scoring above, below or similar to Scotland in science in PISA since 2006

42. Although the OECD ^[6] average has declined since 2012, Scotland's relative position against the member states and UK administrations has also declined, with more countries outperforming Scotland and a smaller number performing significantly below. The number of countries who performed above Scotland in science has increased in each PISA cycle since 2006, whilst fewer countries performed less well than Scotland since 2009, also suggesting a decline in Scotland's relative performance.

43. The countries that have changed their relative position to Scotland since 2012 can be categorised as follows:

• Countries which have maintained their performance, but moved ahead of Scotland as its score declined. This includes England, New Zealand, Slovenia and Switzerland
• Countries whose performance declined, but not as much as Scotland's, and have thus moved ahead. This applies to the Netherlands
• Countries who maintained their performance and moved from below to alongside Scotland. This applies to France, Norway, Spain, Sweden and the United States.
• Countries who improved their performance and moved from below to alongside Scotland. This applies to Portugal
• Countries, whose performance declined, and moved from above to alongside Scotland. This applies to Ireland and Poland

• 44. Among the participating non- OECD countries and economies, Singapore (the highest scoring participant), Hong Kong-China, Chinese Tapei, Viet Nam, Macao-China and the four participating provinces of the Peoples' Republic of China ("B-S-J-G") were above Scotland. No country was similar to Scotland. Thirty-one countries and economies performed significantly below Scotland.
• 45. The countries recorded as above Scotland were unchanged, with the exception of B-S-J-G replacing Shanghai-China, and Liechtenstein not participating in PISA 2015 ( Table A.2b). The OECD reports published at the same time as this report have full details on all countries' scores, and also those for which data has been collected on a "regional" basis (including Scotland).

Performance by type of science

• 46. Tables A.4, A.5 and A.6 set out Scotland's and the OECD countries' performance in the subscales of science explained in Chapter 2. Scotland's performance in each of these was similar to the OECD average, except in "Living systems" where Scotland was significantly higher than the OECD.
• 47. Although the subscales do not map directly onto those used in 2006, there appeared to be significant declines in Scotland's performance for "Explaining phenomena scientifically" and in content knowledge for "Physical systems" and "Living systems". There appeared to be no decline in "Earth and Space systems".

Distribution of scores

• 48. As well as comparison between countries' mean scores, it is important to look at how these are distributed within a country. It is likely that there is much more variation within than between countries. Scotland's spread of science performance in 2015, as measured by standard deviation of the scores (95 points), was similar to the OECD average (94 points). Twelve countries had a narrower distribution, 14 similar and 11 greater. Between 2012 and 2015, Scotland's spread of attainment has increased (2012 = 89 points) and reverted to similarity with 2009 levels (96 points).

High and low achievers

Low performance (below Level 2)

• 49. As set out in Chapter 2, the OECD categorise students into Levels according to their ability to undertake certain tasks. However the group below Level 2 merits particular attention, as the OECD consider that Level 2 is the baseline of ability to participate effectively in society.
• 19.5 per cent of 15 year-olds in Scotland performed below Level 2 in 2015 - statistically similar to the OECD average of 21.3 per cent.
• This was a significant increase on the 2012 figure of 12.1 per cent. The OECD figure also increased significantly from 17.8 per cent in 2012.

High performance (Levels 5 and 6)

• 50. At the other end of the distribution, the proportion of students who were "higher" achievers (Level 5 and above) was 7.5 per cent, similar to the OECD average of 7.7 per cent, and similar to our performance in 2012 (8.8 per cent).
• 51. Chart 3.3 below shows the distribution of scores in Scotland compared to the OECD average. Table A.3 shows each OECD country and UK administration's distribution of scores by proficiency level.

Chart 3.3: Percentage of Scottish students by proficiency level in science performance (per cent)

Gender

Mean score

• 52. Boys' and girls' performance was statistically similar. The average score in science was 496 for female students, and 497 for male students. The gap (one point) was statistically similar to the OECD average (four points).

Change over time

• 53. The gap in performance in science by gender was statistically significant in 2012, but this was no longer the case in 2015. However, performance for both boys and girls was significantly lower than in 2012, with a 20-point fall for boys and a 14-point fall for girls.

High and low achievers

• 54. In terms of the gender share of higher and lower achievers, 6.6 per cent of girls and 8.4 per cent of boys achieved Level 5 and above (no significant difference) and 20.1 per cent of boys and 19.0 per cent of girls were below Level 2 (no significant difference). The share of high and low performers was similar to the OECD for both boys and girls.

Social background

Box 3.1 The OECD's measurement of the impact of social background

The OECD analyse social background using the Index of Economic, Social and Cultural Status ( ESCS). It is constructed from the responses given by students in their background questionnaire and collects information on parental education and occupation, learning resources in the home and cultural possessions. This index is not comparable to the measure commonly used in Scotland - the Scottish Index of Multiple Deprivation ( SIMD) - however it does have the advantage of being generated directly from information provided by the student on their own background, rather than being based on their home address, so avoiding issues of more affluent students being resident in areas which are disadvantaged, and vice versa. It is also consistent across all countries who participate in PISA, enabling comparable analysis.

The index is used to derive a number of measures, each of which tell us something different about the impact of social background on performance.

First of all, the percentage share of the variation in performance explained by social background ^[7] tells us how strong the relationship is. For example, we can see an illustration of Scotland's students' scores in maths for 2012 plotted against the ESCS on the bottom. In 2012, 12.9 per cent of the variation in maths score was explained by social background, and in fact it is possible to see that many students from less affluent backgrounds (towards the left of the graph) achieved high marks - outperforming the average for their circumstances, and vice versa.

The second indicator, the gradient, shows simply how much score varies on average with each step (one point) in social background and can be seen by the slope of the line on the graph. Despite many students "bucking the trend", there is still a positive relationship between affluence and performance in PISA overall. For maths in 2012, this was a gradient of 37 score points. A higher score would indicate a steeper gradient, and greater increases in score with background prosperity.

The final indicator, and the one which perhaps best explains "the gap" in performance by social background, is the length of the gradient. Looking at the students on the 5 ^th and 95 ^th percentiles means that the very extremes of wealth and poverty won't distort the comparison. In Scotland these two notional students were 2.6 points apart by social background measured by ESCS in 2012.

With a gradient of 37 score points, this implies a difference in their maths performance of 96 score points. Although translating this gap into school years of education is not straightforward, the OECD calculate that this could imply as much as three years' difference in learning achievement (with 30 points being equivalent to a year ^[8] ).

• 55. The share of variation in science test scores that was explained by students' background was 10.7 per cent. This was similar to the OECD average (12.9 per cent) and similar to 2012, but less than the 2009 figure of 16.3 per cent. This means that Scotland remained about average in terms of how much pupils break away from the pattern of background affecting performance. Although there was still a clear link between background and performance, there are other things that affect performance, and many pupils do not follow the pattern.
• 56. The gradient was 37 points in the science assessment for Scotland. This was statistically similar to the 2012 figure (36 points), and the OECD average of 38 points This still represents a reduction on the estimated impact in the 2009 survey (47 points), and was greater than in seven countries, similar to 22, and less than eight others.
• 57. The difference between the 5 ^th and 95 ^th percentiles by ESCS was 2.63 points. Combined with a 37-point gradient, this implies that their average scores in science are apart by nearly 97 points which implies a difference of around three years' schooling.
• 58. Table A.7 has each OECD country and UK administration's scores on these measures, as well as estimates of the "adjusted mean score" if a country's students were assumed to have a social background similar to the OECD average.

Students and immigration background

• 59. The survey also asks about students' background in terms of whether they or their parents were born outside the country of the test (for these purposes, the UK). In 2015, students without an immigrant-background performed similarly (a one point difference) to those with an immigrant-background (defined as both parents being born outside the UK). This contrasted with the OECD, with a larger, significant, gap of 43 points in favour of non-immigrant-background students. This indicator is also reported in Table A.7.