Programme for International Student Assessment (PISA) 2015: highlights from Scotland's results: collaborative problem solving

3. Performance in Collaborative Problem Solving

Average scores

19. In collaborative problem solving, Scotland scored 513 in PISA 2015. Chart 3.1 illustrates Scotland’s score with 95-per-cent confidence intervals ^[2] next to the scores for the OECD average. Note that Scotland, with a smaller sample, has larger confidence intervals than does the OECD average, where the combined sample makes for more certainty.

Chart 3.1: Comparison of Scotland and OECD collaborative problem solving scores

20. Scotland’s score was above the OECD average in 2015. The OECD average is set at 500, reflecting that collaborative solving is a new domain.

Comparisons with other countries

21. Compared to the 31 OECD countries ^[3] who participated, plus the three other UK administrations, Scotland performed similarly to six countries, including Northern Ireland and the United States, and above 19 countries including Wales, France, Norway and Italy. Nine countries performed above Scotland, including Canada, Germany, Korea and England. Table 3.1 below shows which countries were statistically significantly above, similar to and below Scotland in 2015. Table A.1a, located in the annex, shows each country’s score.

Table 3.1: OECD countries and UK administrations, higher than, similar to and lower than Scotland in collaborative problem solving

Higher score than Scotland	Similar score to Scotland	Lower score than Scotland
Australia	Austria	Belgium
Canada	Denmark	Chile
England	Netherlands	Czech Republic
Estonia	Northern Ireland	France
Finland	Sweden	Greece
Germany	United Kingdom [4]	Hungary
Japan	United States	Iceland
Korea		Israel
New Zealand		Italy
		Latvia
		Luxembourg
		Mexico
		Norway
		OECD average
		Portugal
		Slovak Republic
		Slovenia
		Spain
		Turkey
		Wales

22. Among the participating non- OECD countries and economies, Singapore (the highest scoring participant), Hong Kong-China, Chinese Tapei and Macao-China performed significantly higher than Scotland. No country was similar to Scotland. Sixteen countries and economies performed significantly below Scotland, including Brazil, the Russian Federation and the four provinces participating from the Peoples’ Republic of China (“B-S-J-G” ^[5] ).

23. Table A.1b (annex) records the mean scores for the non- OECD member states. The OECD reports published at the same time as this report have full details on all countries’ scores, and also where data has been collected on a “regional” basis (including Scotland).

Distribution of scores

24. As well as comparison between countries’ mean scores, it is important to look at how these are distributed within a country. It is likely that there is much more variation within than between countries. Scotland’s spread of performance in collaborative problem solving, as measured by standard deviation of the scores (99 points), was larger than the OECD average (95 points). Seventeen countries had a narrower distribution, eleven similar and six greater.

High and low achievers

Low performance (below Level 2)

25. As set out in Chapter 2, the OECD categorise students into levels according to their ability to undertake certain tasks. However, the group below Level 2 merits particular attention, as the OECD consider that Level 2 is the baseline of ability to participate effectively in society. 23.8 per cent of 15 year-olds in Scotland performed below Level 2 in 2015 – lower than the OECD average of 28.1 per cent.

High performance (Level 4)

26. At the other end of the distribution, the proportion of students who were “higher” achievers (Level 4) was 9.8 per cent, greater than the OECD average of 7.9 per cent.

27. Chart 3.2 below shows the distribution of scores in Scotland compared to the OECD average. Table A.2 (annex) shows each OECD country and UK administration’s distribution of scores by proficiency level.

Chart 3.2: Percentage of Scottish students by proficiency level in collaborative problem solving (per cent)

Gender

Mean score

28. Girls in Scotland performed better than boys. The average score in collaborative problem solving was 530 for female students, and 497 for male students. The gap (33 points) was similar to the OECD average (29 points).

High and low achievers

29. In terms of the gender share of higher and lower achievers, girls were more likely to record high performance (Level 4) than boys (12.4 per cent of girls and 7.3 per cent of boys). 29.0 per cent of boys and 18.5 per cent of girls were below Level 2 (a significant difference between boys and girls at Level 1, but not Below Level 1).

30. Girls recorded a higher share of Level 4 performers than the OECD average. Boys’ share was similar to the OECD average. Boys recorded a lower share of low performers than the OECD average for Below Level 1 only. Girls recorded a lower share of low performers for Level 1 only.

Social background

31. The OECD measure social background using their own Index of Economic, Social & Cultural Status ( ESCS) calculated from data given by students in the background questionnaire. A number of indicators are used to explain how much performance is related to student background. Further information is provided in the Scottish Government’s previous PISA report, and the OECD volumes.

32. The share of variation ^[6] in test scores for collaborative problem solving that was explained by students’ background was 5.6 per cent. This was lower than the OECD average (7.9 per cent) This means that Scotland’s pupils were more likely to break away from the pattern of background affecting performance. Although there was still a clear link between background and performance, there are other things that affect performance, and many pupils do not follow the pattern.

33. The gradient in Scotland, i.e. how much the mean performance score in collaborative problem solving changes with social background, was 28 points. This was statistically similar to the OECD average of 30 points and was greater than in five countries, similar to 21 and less than eight others.

34. The difference between the 5 ^th and 95 ^th percentiles by ESCS was 2.63 points. Combined with a 28-point gradient, this implies that the average scores in collaborative problem solving between the most disadvantaged and least disadvantaged students are apart by nearly 74 points. If, as the OECD surmises with maths, reading and science, a year’s schooling is equivalent to about 30 points, that would imply a difference of nearly two and half years’ education.

35. Table A.3 (annex) has each OECD country and UK administration’s scores on the gradient and strength of relationship.

Students and immigration background

36. The survey also asks about students’ background in terms of whether they or their parents were born outside the country of the test (for these purposes, the UK). In 2015, students without an immigrant-background performed similarly to those with an immigrant-background (defined as both parents being born outside the UK). This contrasted with the OECD average, where Scotland’s (statistically non-significant) gap of 4 points in favour of immigrant students, compares with a statistically significant gap of 36 points in favour of non-immigrant-background students. This gap in turn was significantly different to Scotland’s. This indicator is also reported in Table A.3 (annex) for each OECD country and UK administration.