Programme for International Student Assessment (PISA) 2018: highlights from Scotland's results

1. Introduction and Methodology

What is PISA?

1. The Programme for International Student Assessment (PISA) is an assessment of 15 year-olds' skills carried out under the auspices of the Organisation for Economic Co-operation and Development (OECD). The programme runs every three years across all OECD members and a variety of partner countries. Scotland has participated in all seven surveys since the first wave of testing in 2000.

2. Each survey cycle focusses on one of three domains: reading, mathematics and science. In 2018 the main domain was reading, with maths and science as subsidiary domains. Data and analysis on global competence (the "innovative domain" in PISA 2018) will be published during 2020.

Who participates?

3. Around 600,000 students participated in the study worldwide, representing about 32 million 15 year olds. In 2018, 79 countries and economies participated in PISA.

Fig. 1.1: Global coverage of PISA 2018

Table 1.1: OECD states and partner countries and "economies" participating in PISA 2018

OECD countries		Partner countries and economies
Australia Austria Belgium Canada Chile Colombia Czech Republic Denmark Estonia Finland France Germany Greece Hungary Iceland Ireland Israel Italy	Japan Korea Latvia Lithuania Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Slovenia Spain Sweden Switzerland Turkey United Kingdom United States	Albania Azerbaijan (Baku City) Argentina Belarus Bosnia & Herzegovina Brazil Brunei Darussalam Bulgaria China (People's Republic of) Chinese Taipei Costa Rica Croatia Dominican Republic Georgia Hong Kong (China) Indonesia Jordan Kazakhstan Kosovo Lebanon Macao (China)	Malaysia Malta Moldova Montenegro Morocco Panama Peru Qatar Republic of North Macedonia Panama Peru Qatar Romania Russian Federation Saudi Arabia Singapore Thailand Ukraine United Arab Emirates Uruguay Viet Nam

4. The United Kingdom is a member state of the OECD and its results are published in the main OECD publication. Scotland participates as an "adjudicated region", meaning that its results have full quality assurance from the survey contractors appointed by the OECD, and can publish its results separately. Within the UK, England, Wales and Northern Ireland have boosted samples as "non-adjudicated regions" which means they are able to produce country-level analysis within their reports. Regional results are published as annexes to the main OECD volumes.

5. Survey fieldwork is carried out separately in each participating state by "National Centres" according to strict quality standards set by the OECD.

6. Results based on reading performance are reported as missing for Spain. Spain's data met PISA 2018 Technical Standards, however due to some implausible response behaviour amongst students the OECD is unable to assure that international, subnational and trend comparisons of Spain's results lead to valid conclusions about students' reading proficiency. PISA 2018 reading results for Spain are therefore not available and are not included in OECD average results.

What does PISA measure?

7. PISA seeks to measure skills which are necessary for participation in society. Accordingly, it assesses how students apply the skills they have gained to the types of problem they may encounter in work or elsewhere. Pupils are assessed at the age of 15 as this is regarded as a reasonable point at which to test the impact of compulsory education throughout the developed world. After this point students will typically move onto more specialised studies or enter the labour market. Box 1.1 contains the definitions of the domains tested by PISA.

Box 1.1: The PISA domains and their definition

Reading literacy is defined as students' capacity to understand, use, evaluate, reflect on and engage with texts in order to achieve one's goals, develop one's knowledge and potential, and participate in society.

Mathematical literacy is defined as students' capacity to formulate, employ and interpret mathematics in a variety of contexts. It includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena.

Science literacy is defined as the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen. A scientifically literate person is willing to engage in reasoned discourse about science and technology, which requires the competencies to explain phenomena scientifically, evaluate and design scientific enquiry, and interpret data and evidence scientifically.

8. We have included some details on how reading, the main focus of the 2018 PISA survey, was assessed in Chapter 2. Further details of how each domain was assessed can be found in the OECD volumes published on the PISA website, www.oecd.org/pisa.

9. The assessments are also supplemented by background questionnaires. Pupils are asked about their motivations for study, attitudes to school, views on reading, and their socio-economic background. Headteachers are asked about the challenges facing their schools, organisation and factors that they believe affect their students' performance.

The survey in Scotland

10. The PISA survey was managed by an international consortium led by ETS. The Consortium developed the tests, questionnaires and survey documentation and ensured that all participating countries met quality standards. In Scotland, the National Foundation for Educational Research (NFER) was the "National Centre", responsible for local adaptations to the surveys, and administering the test in schools.

11. The school sample was randomly selected by NFER following submission of sampling forms to the consortium. The sample was stratified on the basis of previous exam performance (split into five categories), whether schools were publicly funded or independent, urban/rural location and school size, and whether schools were single-sex or mixed.

12. The survey was carried out in Scotland between 8 October and 14 December 2018. In total, 107 secondary schools participated in the survey. One hundred of these were from the main sample (86 per cent response rate), and seven from the back-up samples (resulting in a 90 per cent weighted participation rate after replacements were added in). This exceeded the OECD's minimum standard of 85 per cent participation.

13. Within each school 40 students were randomly sampled by NFER using software supplied by the Consortium. In total 4,265 students were drawn in the sample. Schools were able to withdraw a certain number of students where it was deemed that participation would be difficult due to additional support needs or language issues. Similarly students that had left the school in the interim were not considered part of the target sample. In total 3,767 students were deemed eligible participants. Of these a total of 2,969 students took part, with the balance being those who did not wish to take part (both students and their parents were given the opportunity to opt out of the survey), those who were absent on the day of the test or were withdrawn by the school because of their additional support needs.

14. The OECD had strict criteria for the level of exclusion that was acceptable, and the total exclusion rate of 5.39 per cent was deemed to be consistent with a robust sample. Similarly, the final weighted participation rate, calculated by the consortium, was 80.51 per cent, which met the OECD requirement of 80 per cent.

15. The assessment was administered in Scotland by computer. This was achieved using the existing facilities in schools with the support of school and Local Authority ICT services.

16. The software delivery system was provided by the international consortium. The assessment was administered in two one-hour sessions, with a further 35 minutes for the background questionnaire. Students spent one hour on the reading assessment (composed of a core stage followed by two stages of either greater or lesser difficulty) plus one hour on one or two other subjects – mathematics, science or global competence.

17. As in all previous cycles, there was a survey of headteachers within schools, which asked about their views on school organisation, teaching staff and resources. Eighty-seven headteachers responded – a response rate of 81.3 per cent.

18. In 2018, Scotland also participated in the Teacher Questionnaire, which was undertaken by 19 countries and economies in total. Questions asked about initial teacher education and professional development, their beliefs and attitudes, and their teaching practices. Separate questionnaires were developed for teachers of the main domain (for PISA 2018, this is teachers of English) and for other teachers in the school. The teacher questionnaire took 45 minutes to complete and was sent to 30 teachers in each of the schools included in the PISA assessment. 1,445 teachers completed the questionnaire, a response rate of 51 per cent. The results from the teacher questionnaire will be published in a separate report in 2020.

Interpreting the results

19. It should be understood that PISA is a sample survey. Like all surveys of this type, it is subject to sampling error. The necessity of surveying only a sample of students, even when chosen at random, runs the risk that such a group will not necessarily reflect the larger population of students. We therefore cannot assume that the values found in the survey are be the same as those in the population. Confidence intervals are presented around mean scores; we can be 95% sure that the true value lies within this range.

20. This means that being confident that there is a difference between Scotland and the OECD average, or between groups and countries, will depend on both the size of the observed difference in mean scores and the associated standard error. Significance tests are used to assess the statistical significance of comparisons made.

21. It is not possible to produce individual country rankings based on the absolute (mean) score. Accordingly this report shows results divided into those countries whose scores are statistically significantly higher than, similar to or lower than Scotland. By "statistically significant" we mean that we are 95 per cent certain that there is a difference (or similarity).

Change over time

22. This report covers, as in previous publications, the position of Scotland relative to other countries, and how this has changed over time. The mathematics assessment changed radically in 2003 and for science in 2006, as they became "full domains" for the first time, so we are unable to make comparisons before those waves.

23. One complication is that membership of the OECD has changed at various points. In 2010, Chile, Estonia, Israel and Slovenia were admitted to membership. This affected comparison of reading scores in 2009^[1]. Scotland was above the OECD average when those four countries were included, but similar to the average of the pre-2010 membership. Since PISA 2015, Latvia (2016), Lithuania (2018) and Colombia (2019) have joined or been invited to join the membership of the OECD. When making comparisons with the OECD average, this report defines this as the average of member nations of the OECD at the time.

24. Further, the measurement of performance can be affected by new test items, the change of administration from paper- to computer-based assessment and the statistical treatment of data. While the scales have been equated to allow for expression on the same basis between cycles, the OECD provide a "link error" to quantify the uncertainty when comparing scores over different waves of data. All estimates in this report have taken this into account.

Further analysis of PISA

25. Much of this report focusses on changes to Scotland's mean score and the relative position internationally. However, PISA is not just a snapshot of student attainment, but a comprehensive data-gathering exercise which enables analysis, not only of how well school systems around the world perform, but the factors that are behind this. The OECD publications present international analysis of students' abilities, motivations, attitudes, background, support at home and confidence. In addition, information is gathered on school structure and management, and the OECD analyse how various aspects of school organisation may be related to attainment.

26. The OECD will publish further volumes of PISA 2018 data, including on the Global Comptetence Inovative Domain, during 2020.

27. Periodically, the OECD also publish short reports in their "PISA in Focus" series at the following link: http://www.oecd.org/pisa/pisaproducts/pisainfocus.htm