PISA 2022 Results (Volume II)

Reader’s Guide

PISA in the pandemic

This edition of PISA includes data from 81 countries and economies. The test was originally planned to take place in 2021 but was delayed by one year due to the COVID-19 pandemic. The exceptional circumstances throughout this period, including lockdowns and school closures in many places, led to occasional difficulties in collecting some data. While the vast majority of countries and economies met PISA’s technical standards (available on line), a small number did not. In prior PISA rounds, countries and economies that failed to comply with the standards, and which the PISA Adjudication Group judged to be consequential, could face exclusion from the main part of reporting. However, given the unprecedented situation caused by the pandemic, PISA 2022 results includes data from all participating education systems, including those where there were issues such as low response rates (see Annexes A2 and A4). The next section explains the potential limitations of data from countries not meeting specific technical standards. Readers are alerted to these limitations throughout the volume wherever appropriate.

It is important to note that the limitations and implications were assessed by the PISA Adjudication Group in June 2023. There may be a need for subsequent adjustments as new evidence on the quality and comparability of the data emerges. PISA will return to the standard ways of reporting for the 2025 assessment.

Adjudicated entities not meeting the sampling standards

The results of 13 adjudicated entities (i.e. countries, economies and regions within countries), listed below, will be reported with annotations. Caution is required when interpreting estimates for these countries/economies because one or more PISA sampling standards listed below were not met.

• Overall exclusion rate. Standard 1.7: The PISA Defined Target Population covers 95% or more of the PISA Desired Target Population. That is, school-level exclusions and within-school exclusions combined do not exceed 5%.

• School response rate. Standard 1.11: The final weighted school response rate is at least 85% of sampled schools. If a response rate is below 85% then an acceptable response rate can still be achieved through agreed upon use of replacement schools.

• Student response rate. Standard 1.12: The student response rate is at least 80% of all sampled students across responding schools.

The 13 entities can be grouped into two:

(i) Entities that submitted technically strong analyses, which indicated that more than minimal bias was most likely introduced in the estimates due to low response rates (falling below PISA standards): Canada, Ireland, New Zealand, the United Kingdom and Scotland.

(ii) Entities that did not meet one or more PISA sampling standards and it is not possible to exclude the possibility of more than minimal bias based on the information available at the time of data adjudication: Australia, Denmark, Hong Kong (China), Jamaica, Latvia, the Netherlands, Panama and the United States.

The Adjudication Group also noted that the bias associated with trend and cross-country comparisons might be smaller, if past data or data for other countries are biased in the same direction. Therefore, the deviations from the standards in PISA 2022 are compared with those in PISA 2018 where necessary.

(i) Entities that submitted technically strong analyses, which indicated that more than minimal bias was most likely introduced in the estimates due to low response rates (falling below PISA standards)

Canada

• Overall exclusion rate: 5.8%. Exclusions exceeded the acceptable rate by less than one percentage point; at the same time, the exclusion rates observed in 2022 remained relatively close to exclusion rates observed in 2018 (6.9%).

• Student response rate: 77%. School response rates: 81% before replacement, 86% after replacement. Student response rates decreased from 84% with respect to PISA 2018, and fell short of the target in 7 out of 10 provinces (all but New Brunswick, Prince Edward Island and Saskatchewan). A thorough non-response bias analysis was submitted, with analyses conducted separately for each province, using students' academic achievement data as auxiliary information. School response rates also fell short of the target, driven by low participation rates in two provinces (Alberta and Quebec). For these provinces, non-response bias was also examined at the school level. The analyses clearly indicate that school nonresponse has not led to any appreciable bias, but student nonresponse has given rise to a small upwards bias.

Ireland

• Student response rate: 77%. Student response rates decreased from 86% with respect to PISA 2018. A thorough non-response bias analysis was submitted, using external achievement data at student level as auxiliary information. The analysis provided evidence to suggest a residual upwards bias of about 0.1 standard deviations, after non-response adjustments are taken into account. On the PISA scale, considering that the standard deviation in Ireland ranged (in 2018) from 78 score points in mathematics to 91 score points in reading, this could translate in an estimated upwards bias of approximately 8 or 9 points.

New Zealand

• Overall exclusion rate: 5.8%. Exclusions exceeded the acceptable rate by less than one percentage point; at the same time, the exclusion rates observed in 2022 remained relatively close to exclusion rates observed in 2018 (6.8%).

• Student response rate: 72%. School response rate: 61% before replacement, 72% after replacement. Student response rates decreased from 83% with respect to PISA 2018. School response rates also fell short of the target. A thorough and detailed non-response bias analysis was submitted, using external achievement data at student level, but also information on chronic absenteeism, as auxiliary information, along with demographic characteristics. The analysis provided evidence to suggest a residual upwards bias of about 0.1 standard deviations, after non-response adjustments are taken into account, driven entirely by student non-response (school non-participation did not result in significant bias, in contrast). The analysis also suggested that chronically absent students are over-represented among non-respondents in PISA. On the PISA scale, considering that the standard deviation in New Zealand ranged (in 2018) from 93 score points in mathematics to 106 score points in reading, this could translate in an estimated upwards bias of approximately 10 points. The Adjudication Group also noted that the bias associated with trend and cross-country comparisons might be smaller, if past data or data for other countries are biased in the same direction. For more information, see educationcounts.govt.nz website.

The United Kingdom

The United Kingdom (excluding Scotland)

• Student response rate: 75%. School response rates: 66% before replacement, 80% after replacement. Student response rates decreased from 83% with respect to PISA 2018. School response rates also fell short of the target. An informative non-response bias analysis was submitted, using external achievement data at student level as auxiliary information, along with demographic characteristics; the analysis was limited to England as the largest subnational entity within the United Kingdom (excluding Scotland), and thus covered over 90% of the intended sample. The analysis provided evidence to suggest a small residual upwards bias of about 0.07 standard deviations for reading and 0.09 standard deviations for mathematics, after non-response adjustments are taken into account, driven entirely by student non-response (school non-participation did not result in significant bias, in contrast). On the PISA scale, considering that the standard deviation in England (in 2018) was about 101 score points in reading and 93 score points in mathematics, this could translate in an estimated upwards bias of approximately 7 or 8 points.

Scotland

• Overall exclusion rate: 6.6%. Exclusions exceeded the acceptable rate by a small margin; at the same time, the exclusion rates observed in 2022 remained relatively close to exclusion rates observed in 2018 (5.4%).

• Student response rate: 79%. Student response rates missed the standard by a small margin, but were otherwise similar to response rates in PISA 2018 (81%). A thorough non-response bias analysis was submitted, using several external achievement variables at student level as auxiliary information, along with demographic characteristics. The analysis provided evidence to suggest a residual upwards bias of about 0.1 standard deviations, after non-response adjustments are taken into account. On the PISA scale, considering that the standard deviation in Scotland (in 2018) was about 95 score points in reading and mathematics, this could translate in an estimated upwards bias of approximately 9 or 10 points. Given the similarity of response rates between 2018 and 2022, it cannot be excluded that a similar bias might be present in 2018 as well, and in many PISA 2022 participants whose response rates were similarly close to the target. For this reason, data were deemed to be comparable to previous cycles.

(ii) Entities that did not meet one or more PISA sampling standards and it is not possible to exclude the possibility of more than minimal bias based on the information available at the time of data adjudication.

Australia

• Overall exclusion rate: 6.9%. Exclusions exceeded the acceptable rate by a small margin; at the same time, the exclusion rates observed in 2022 remained relatively close to exclusion rates observed in 2018 (5.7%).

• Student response rate: 76%. Student response rates decreased from 85% with respect to PISA 2018. A technically sound non-response bias analysis was submitted; however, the strength of the evidence was limited by the fact that no external student-level achievement variables could be used in the analysis. Based on the available evidence, and on the experience of other countries participating in PISA, the Adjudication Group considered that while non-response adjustments likely limited the severity of non-response biases, a small residual upward bias could not be excluded.

Denmark

• Overall exclusion rate: 11.6%. Exclusions exceeded the acceptable rate by a large margin and showed a marked increase, with respect to 2018 (5.7%). The Adjudication Group noted that high levels of student exclusions may bias performance results upwards. In Denmark, a major cause behind the rise appears to be the increased share of students with diagnosed dyslexia, and the fact that more of these students are using electronic assistive devices to help them read on the screen, including during exams. The lack of such an accommodation for students with diagnosed dyslexia in the PISA assessment led schools to exclude many of these students. In order to reduce exclusion rates in the future, PISA may need to further accommodate dyslexic students, allowing the use of assistive devices.

Hong Kong (China)

• Student response rate: 75%. School response rates: 60% before replacement, 80% after replacement. Student response rates decreased from 85% with respect to PISA 2018. School response rates also fell short of the target (as they did in 2018). At the school level, the fact that a raw, but direct measure of school performance is used to assign schools to sampling strata (and therefore, differential non-response across strata is unlikely to cause bias), limits the risk of bias due to non-response. A non-response bias analysis was submitted; however, the strength of the evidence was limited by the fact that no external student-level achievement variables could be used in the analysis (only student grade information, already used in non-response adjustments, was available). The proxies for school and student achievement (school size and student grade) that were used in the analyses showed no or very limited relationship with participation rates. Nevertheless, based on the available evidence, and on the experience of other countries participating in PISA, the Adjudication Group considered that while non-response adjustments likely limited the severity of non-response biases, a small residual upward bias could not be excluded.

Jamaica

• Student response rate: 68%. Student response rates were substantially below the standard. A simple non-response bias analysis was submitted, analysing student response rates by school characteristics: this showed in particular lower response rates in rural schools and regions. A limited non-response bias analysis was also prepared by the Core C contractor, to compare respondent characteristics (both before and after nonresponse adjustment) to characteristics of the full eligible sample of students. This suggested that non-response was also related to students’ grade level and gender (both variables are used in non-response adjustments). Based on the available information, it is not possible to exclude the possibility of bias; considering the analyses on student non-response conducted in other countries, the residual bias after non-response adjustments are taken into account is likely to correspond to an upward bias. The Adjudication Group also noted that a number of issues encountered during the main survey data collection could have been prevented, had Jamaica been able to do a full field trial. This was not possible because of COVID-related disruptions to schooling in 2021. In particular, enrolment information available to the national centre for school-level sampling often turned out to be imprecise; and low student participation rates could have been anticipated, had a regular field trial been conducted. As a result of inaccurate sampling frames and low student response rates, the achieved sample size for the main survey was well below target, and sampling errors for Jamaica are larger than desired. The Adjudication Group noted that apart from the challenges around sampling operations, the quality of the data met expectations for reporting.

Latvia

• Overall exclusion rate: 7.9%. Exclusions exceeded the acceptable rate by a large margin and showed a marked increase, with respect to 2018 (4.3%). Most of these students were excluded because they were attending school in remote or virtual mode. The Adjudication Group noted that high levels of student exclusions may bias performance results upwards.

The Netherlands

• Overall exclusion rate: 8.4%. Exclusions exceeded the acceptable rate by a large margin and showed a marked increase, with respect to 2018 (6.2%). Most of these students were excluded because they had a physical or intellectual disability and no adaptation was available for them. The Adjudication Group noted that high levels of student exclusions may bias performance results upwards.

• School response rates: 66% before replacement, 90% after replacement. A non-response bias analysis was submitted, analysing differences in performance and in other characteristics between responding schools and the total population of schools, as well as differences between replacement schools and originally sampled, but non-responding schools. This supported the case that no large bias would result from non-response; furthermore, given the available evidence, there is no clear indication about the direction of any residual bias.

Panama

• Student response rate: 77%. In the challenging circumstances surrounding schooling in Panama in 2022 (teacher strikes, road blockades, and student absenteeism), student response rates decreased from 90% with respect to PISA 2018. No non-response bias analysis was submitted; the PISA national centre explained that non-response was potentially related to the agitated school climate the students found themselves when returning to their schools after the strikes. A limited non-response bias analysis was prepared by the Core C contractor, to compare respondent characteristics (both before and after nonresponse adjustment) to characteristics of the full eligible sample of students. This analysis suggested that (before non-response adjustments were taken into account), non-response was related to students’ grade level, and to special needs status. Based on the available information, it is not possible to exclude the possibility of bias; considering the analyses on student non-response conducted in other countries, the residual bias after non-response adjustments are taken into account is likely to correspond to an upward bias.

The United States

• Exclusion rates: 6.1%. Exclusions exceeded the acceptable rate by a small margin but showed a marked increase, with respect to 2018 (3.8%), in exclusion rates for students with functional or intellectual disabilities. The Adjudication Group invited the national centres to investigate the reasons for this increase in exclusion rates and take remedial action for future cycles. It is expected that exclusion rates will fall again in the future, as a result.

• School response rates: 51% before replacement, 63% after replacement. School participation rates missed the standard by a substantial margin, and participation rates were particularly low among private schools (representing about 7% of the student population). A non-response bias analysis was submitted, indicating that, after replacement schools and non-response adjustments are taken into account, a number of characteristics (not including direct measures of school performance) are balanced across respondents and non-respondents. The Adjudication Group also noted that the response rate for students was only slightly above the target (80%). Based on the available information, it is not possible to exclude the possibility of bias, nor to determine its most likely direction.

Adjudication entity not reaching a strong level of comparability

The ability to compare PISA results with those of other countries, and over time, depends on the use of common test items and of standardised test-administration procedures. In addition, the common items must consistently indicate high, medium, or low proficiency, regardless of the country/economy or of the language of the test. When this condition is met, a common set of (international) parameters is used to convert students’ correct, partially correct or incorrect responses into an estimated score on the PISA scale.

The PISA Technical Advisory Group issued a memo in December 2021 stating that, in each country and economy, over two-thirds of items are expected to use the international item parameters to ensure strong comparability of PISA scores across countries and economies. Where the proportion is lower, greater uncertainty (beyond the uncertainty of estimates reflected in standard errors) is associated with cross-country comparisons.

During the review of PISA 2022 results, invariance of item parameters with respect to the international ones was examined for each major language of assessment within a participating country/economy. For Viet Nam, 40% of the items were assigned unique parameters in reading (35 of 87). Viet Nam’s reading results are, therefore, reported in this volume with an annotation indicating that a strong linkage to the international PISA scale could not be established.

Data underlying the figures

The data referred to in this volume are presented in Annex B and, in greater detail, including additional tables, on the PISA website (www.oecd.org/pisa). Five symbols are used to denote missing data:

• a The category does not apply in the country concerned or economy; data are therefore missing.

• c There were too few observations to provide reliable estimates (i.e. there were fewer than 30 students or fewer than 5 schools with valid data).

• m Data are not available. There was no observation in the sample; these data were not collected by the country or economy; or these data were collected but subsequently removed from the publication for technical reasons.

• w Results were withdrawn at the request of the country or economy concerned.

• x Data included in another category or column of the table (e.g. x(2) means that data are included in Column 2 of the table).

Coverage

This publication features data from 81 countries and economies, including all OECD Member countries except Luxembourg and 44 non‑OECD Member countries and economies (see map of PISA countries and economies in “What is PISA?”). Specific territorial disclaimers and footnotes applicable to this publication are included in the copyright page (p.2).

The designation “Ukrainian regions (18 of 27)” refers to the 18 PISA-participating jurisdictions of Ukraine: Cherkasy Oblast, Kirovohrad Oblast, Poltava Oblast, Vinnytsia Oblast, Chernihiv Oblast, Kyiv Oblast, Sumy Oblast, the City of Kyiv, Zhytomyr Oblast, Odesa Oblast, Chernivtsi Oblast, Ivano-Frankivsk Oblast, Khmelnytskyi Oblast, Lviv Oblast, Rivne Oblast, Ternopil Oblast, Volyn Oblast and Zakarpattia Oblast. Due to Russia’s large-scale aggression against Ukraine, the following nine jurisdictions were not covered: Dnipropetrovsk Oblast, Donetsk Oblast, Kharkiv Oblast, Luhansk Oblast, Zaporizhzhia Oblast, Kherson Oblast, Mykolaiv Oblast, the Autonomous Republic of Crimea and the city of Sevastopol.

Following OECD data regulations, a visual separation between countries and territories has been used in all charts to reduce the risk of data misinterpretation.

International averages

The OECD average corresponds to the arithmetic mean of the respective country estimates. It was calculated for most indicators presented in this report.

In this publication, the OECD average is generally used when the focus is on comparing performance across education systems. In the case of some countries, data may not be available for specific indicators, or specific categories may not apply. Readers should, therefore, keep in mind that the term “OECD average” refers to the OECD Member countries included in the respective comparisons. In cases where data are not available or do not apply for all sub-categories of a given population or indicator, the “OECD average” is not necessarily computed on a consistent set of countries across all columns of a table.

In analyses involving data from multiple years, the OECD average is always reported on consistent sets of OECD Member countries, and several averages may be reported in the same table. For instance, the “OECD average-35” includes only 35 OECD Member countries that have non-missing values across all the assessments for which this average itself is non-missing. This restriction allows for valid comparisons of the OECD average over time.

The number in the label used in figures and tables indicates the number of countries included in the average:

• OECD average: Arithmetic mean across all OECD Member countries except Luxembourg.

• OECD average-35: Arithmetic mean across all OECD Member countries excluding Costa Rica, Luxembourg and Spain.

• OECD average-26: Arithmetic mean across all OECD Member countries excluding Australia, Canada, Denmark, Ireland, Latvia, Luxembourg, the Netherlands, New Zealand, Portugal, Spain, the United Kingdom and the United States.

• OECD average-23: Arithmetic mean across all OECD Member countries excluding Austria, Chile, Colombia, Costa Rica, Estonia, Israel, Lithuania, Luxembourg, the Netherlands, the Slovak Republic, Slovenia, Spain, Türkiye, the United Kingdom and the United States.

Rounding figures

Because of rounding, some figures in tables may not add up exactly to the totals. Totals, differences and averages are always calculated on the basis of exact numbers and are rounded only after calculation.

All standard errors in this publication have been rounded to one or two decimal places. Where the value 0.0 or 0.00 is shown, this does not imply that the standard error is zero, but that it is smaller than 0.05 or 0.005, respectively.

Reporting student data

The report uses “15-year-olds” as shorthand for the PISA target population. PISA covers students who are aged between 15 years 3 months and 16 years 2 months at the time of assessment and who are enrolled in school and have completed at least 6 years of formal schooling, regardless of the type of institution in which they are enrolled, and whether they are in full-time or part‑time education, whether they attend academic or vocational programmes, and whether they attend public or private schools or foreign schools within the country.

Reporting school data

The principals of the schools in which students were assessed provided information on their schools’ characteristics by completing a school questionnaire. Where responses from school principals are presented in this publication, they are weighted so that they are proportionate to the number of 15-year-olds enrolled in the school.

Focusing on statistically significant differences

This volume discusses only statistically significant differences or changes. These are denoted in darker colours in figures and in bold font in tables. Unless otherwise specified, the significance level is set to 5%. See Annex A3 for further information.

Abbreviations used in this report

 

ESCS	PISA index of economic, social, and cultural status
GDP	Gross domestic product
ICT	Information and communications technology
ISCED	International Standard Classification of Education
ISCO	International Standard Classification of Occupations
PPP	Purchasing power parity
Score dif.	Score-point difference
S.D.	Standard deviation
SDGs	Sustainable Development Goals
S.E.	Standard error
% dif.	Percentage-point difference

Box 1. Interpreting differences in PISA scores

PISA scores do not have a substantive meaning as they are not physical units such as metres or grams. Instead, they are set in relation to the variation in results observed across all test participants. There is, theoretically, no minimum or maximum score in PISA; rather, the results are scaled to fit approximately normal distributions (i.e. means around 500 score points, standard deviations around 100 score points). In statistical terms, a one-point difference on the PISA scale therefore corresponds to an effect size (Cohen’s d) of 0.01; and a 10-point difference to an effect size of 0.10.

Interpreting large differences in scores: proficiency levels

PISA scales are divided into proficiency levels. For example, for PISA 2022, the range of difficulty of mathematics items is represented by eight levels of mathematics proficiency: the simplest items correspond to Level 1c; Levels 1b, 1a, 2, 3, 4, 5 and 6 correspond to increasingly difficult items. Individuals who are proficient within the range of Level 1c are likely to be able to complete Level 1c items but are unlikely to be able to complete items at higher levels. See Chapter 3, Volume I – PISA 2022 Results, for a detailed description of proficiency levels in mathematics, reading, and science.

In mathematics, each proficiency level corresponds to a range of about 62 score points; in reading the difference between the cut points for each proficiency level is about 73 score points, and in science is about 75 score points. Hence, score-point differences of that magnitude can be interpreted as the difference in described skills and knowledge between successive proficiency levels.

Interpreting small differences in scores: statistical significance

Smaller differences in PISA scores cannot be expressed in terms of the difference in skills and knowledge between proficiency levels. However, they can still be compared with each other by means of verifying their “statistical significance”.

A difference is called “statistically significant” if it is unlikely that such a difference can be observed in the estimates based on samples when, in fact, no true difference exists in the populations from which the samples are drawn. The results of the PISA assessments are “estimates” because they are obtained from samples of students rather than from a census of all students (i.e. which introduces a “sampling error”), and because they are obtained using a limited set of assessment tasks rather than the universe of all possible assessment tasks (i.e. which introduces a “measurement error”).

It is possible to determine the magnitude of the uncertainty associated with the estimate and to represent it as a “confidence interval”, i.e. a range defined in such a way that if the true value lies above its upper bound or below its lower bound, an estimate different from the reported estimate would be observed only with a small probability (typically less than 5%). The confidence interval needs to be taken into account when making comparisons between estimates so that differences that may arise simply due to the sampling error and measurement error are not interpreted as real differences.

Interpreting differences in scores across PISA assessments

To ensure the comparability of PISA results across different assessment years, “link errors” must be used. The link error represents uncertainty around scale values (“is a score of 432 in PISA 2022 the same as 432 in PISA 2018?”) and is therefore independent of the size of the student sample. For comparisons between mathematics results in PISA 2022 and mathematics results in 2018, the link error corresponds to 2.24 score points. For detailed information, see Box I.5.3 in Chapter 5 and Annex A7 of Volume I – PISA 2022 Results.

Interpreting differences in scores in terms of learning gains over a year of schooling

Knowing the typical learning gain that students make as they progress from one grade-level to the next can be useful for interpreting differences in PISA results. 20 points represents the average annual pace of learning of 15-year-olds in countries that participate in PISA. Box I.5.1 in Chapter 5 of Volume I – PISA 2022 Results explores this topic.

Further documentation

For further information on the PISA assessment instruments and the methods used in PISA, see the PISA 2022 Assessment and Analytical Framework (OECD, 2023[1]) and PISA 2022 Technical Report (OECD, forthcoming[2]).

StatLink

This report has StatLinks for tables and graphs. To download the matching Excel® spreadsheet, just type the link into your Internet browser, starting with the https://doi.org prefix, or click on the link from the e-book version.

References

[1] OECD (2023), PISA 2022 Assessment and Analytical Framework, PISA, OECD Publishing, Paris, https://doi.org/10.1787/dfe0bf9c-en.

[2] OECD (forthcoming), PISA 2022 Technical Report, OECD Publishing, Paris.