Programme for International Student Assessment (PISA) 2018 - global competence: results - highlights

Chapter 1: Introduction and methodology

What is PISA?

1. The Programme for International Student Assessment (PISA) is an assessment of 15 year-olds' skills carried out under the auspices of the Organisation for Economic Co-operation and Development (OECD). The programme runs every three years across all OECD members and a variety of partner countries. Scotland has participated in all seven surveys since the first wave of testing in 2000.

2. Each survey cycle focusses on one of three domains: reading, mathematics and science with an additional 'innovative' domain. In 2018 the main domain was reading, with mathematics and science as subsidiary domains. Data on these domains was published in Programme for International Student Assessment (PISA) 2018: highlights from Scotland's results in 2019. In 2018, Scotland participated in the innovative domain for the second time - global competence - and this report represents the results of that assessment.

Methodology

3. The survey was carried out in Scotland between 8 October and 14 December 2018 in 107 secondary schools, with 2,969 students taking part. These students ranged in age between 15 years and 2 months and 16 years and 2 months.

4. The assessments are supplemented by background questionnaires. Pupils are asked about their motivations for study, attitudes to school, views on science and studying, and their socio-economic background. Head teachers are asked about the challenges facing their schools, organisation and factors that they believe affect their students' performance. In 2018, Scotland also participated in the Teacher Questionnaire, including a sample of 1,445 teachers in participating schools.

5. Further information on PISA worldwide, and how it was administered in Scotland, can be found in our 2019 report.

Interpreting the results

6. It should be understood that PISA is a sample survey. Like all surveys of this type, it is subject to sampling error. The necessity of surveying only a sample of students, even when chosen at random, runs the risk that such a group will not necessarily reflect the larger population of students. We therefore cannot assume that the values found in the survey are the same as those in the population.

7. This means that being confident that there is a difference between Scotland and the OECD average, or between groups and countries, will depend on both the size of the observed difference and the standard error associated with the sample sizes used. Significance tests are used to assess the statistical validity of comparisons made. In this report, figures in bold represent Scotland results which are statistically significantly higher than the OECD average. By "significant" we mean that we are 95 per cent certain that there is a difference.

International comparisons

8. Reports on Scotland's PISA results use international comparisons to provide additional analysis and context. This is usually in the form of comparisons with UK and OECD results. However, this is not possible for global competence as the other UK countries (England, Northern Ireland and Wales) did not take part, and a number of other OECD countries also did not participate in all aspects of the global competence survey. Therefore, this report uses comparisons with all participating countries for the cognitive assessment (27 countries) and with participating OECD countries/economies for the questionnaire. Chapter 4 in this report covers the results of the cognitive assessment for Scotland and the other 26 participating countries, and Chapter 5-9 includes findings from Scotland and the other OECD countries participating in the global competence questionnaire.

9. As this is the first time that the Global Competence assessment has been carried out in PISA, there is no data from previous years to compare to.

How results are displayed in this report

Global Competence indices - responses to the student questionnaire are used to construct a series of global competence indices which indicate whether a country or student characteristic is above or below the OECD average. Scores above zero are higher than the OECD average, while scores below zero are lower than the OECD average. Negative scores do not indicate a negative attitude or view, only that it is below the average.

Gender - results are generally broken down to show comparisons between girls and boys.

Immigration background - results are broken down by immigration background. In these data, non-immigrant students are those whose mother or father (or both) was/were born in the country/economy where the student sat the PISA test, regardless of whether the student him/herself was born in that country or economy. Immigrant students are students whose mother and father were born in a country/economy other than that where the student sat the PISA test.

Proficiency Levels - PISA scores can be grouped into different PISA Proficiency Levels. It is common to look at the proportion of students performing below Proficiency Level 2 and at Proficiency Level 5 or better.

Index of Economic, Social and Cultural Status (ESCS) - The OECD analyses socio-economic background using the Index of Economic, Social and Cultural Status (ESCS). It is constructed from the responses given by students in their background questionnaire and collects information on parental education and occupation, learning resources in the home and cultural possessions. This index is not directly comparable to the measure commonly used in Scotland - the Scottish Index of Multiple Deprivation (SIMD).