Programme for International Student Assessment (PISA) 2015: highlights from Scotland's results: collaborative problem solving

1. Introduction and Methodology

What is PISA?

1. The Programme for International Student Assessment ( PISA) is an assessment of 15 year-olds’ skills carried out under the auspices of the Organisation for Economic Co-operation and Development ( OECD). The programme runs every three years across all OECD members and a variety of partner countries. Scotland has participated in all six surveys since the first wave of testing in 2000.

2. Each survey cycle focusses on one of three domains: reading, mathematics and science. In 2015 the main domain was science, with maths and reading as subsidiary domains. Data on these domains was published in 2016. In 2015, for the first time, Scotland participated in the “innovative” domain - collaborative problem solving - and this report represents the results of that assessment.

Methodology

3. The survey was carried out in Scotland between 3 and 28 March 2015 in 109 secondary schools. The 3,123 students tested are generally described as “15 year-olds” although the actual age range was 15 years and 2 months to 16 years and 2 months as of 1 March 2015. Students were mostly (87.5 per cent) in the S4 year group.

4. The assessment items were in six clusters so that approximately half were science, the main domain, with the remainder split between reading, maths and collaborative problem solving.

5. The assessments are also supplemented by background questionnaires. Pupils are asked about their motivations for study, attitudes to school, views on science and studying, and their socio-economic background. Headteachers are asked about the challenges facing their schools, organisation and factors that they believe affect their students’ performance. In 2015, we also participated in the Parents Questionnaire, sent to the parents of all student participants.

6. Further information on PISA worldwide, and how it was administered in Scotland, can be found in our 2016 report.

7. We have included some details on how collaborative problem solving was assessed in Chapter 2. Further details of how each domain was assessed can be found in the OECD volumes published on the PISA website, www.oecd.org/pisa.

Interpreting the results

8. It should be understood that PISA is a sample survey. Like all surveys of this type, it is subject to sampling error. The necessity of surveying only a sample of students, even when chosen at random, runs the risk that such a group will not necessarily reflect the larger population of students. We must therefore be cautious in assuming that the values found in the survey would be the same as those in the population.

9. This means that being confident that there is a difference between Scotland and the OECD average, or between groups and countries, will depend on both the size of the observed difference and the standard error associated with the sample sizes used. Significance tests are used to assess the statistical validity of comparisons made.

10. Therefore, it is not possible to produce individual country rankings based on the absolute (mean) score. Accordingly this report shows results divided into those countries whose scores are statistically significantly higher than, similar to or lower than Scotland. By “significant” we mean that we are 95 per cent certain that there is a difference (or similarity). Where this report states that two figures are different, this has been established to be statistically significant. Where the report states that figures are similar, any differences were found to be statistically insignificant.