PISA 2022 Results (Volume IV)

Reader’s Guide

PISA in the pandemic

This edition of PISA was originally planned to take place in 2021 but was delayed by one year due to the COVID-19 pandemic. The exceptional circumstances throughout this period, including lockdowns and school closures in many places, led to occasional difficulties in collecting some data. While the vast majority of countries and economies met PISA’s technical standards, a small number did not. In prior PISA rounds, countries and economies that failed to comply with the standards, and which the PISA Adjudication Group judged to be consequential, could face exclusion from the main part of reporting. However, given the unprecedented situation caused by the pandemic, PISA 2022 results include data from all participating education systems, including those where there were issues such as low response rates (see Annexes A2 and A4). The next section explains the potential limitations of data from countries not meeting specific technical standards. Readers are alerted to these limitations throughout the volume wherever appropriate.

It is important to note that the limitations and implications were assessed by the PISA Adjudication Group in June 2023. There may be a need for subsequent adjustments as new evidence on the quality and comparability of the data emerges. PISA will return to the standard ways of reporting for the 2025 assessment.

Adjudicated entities not meeting the sampling standards

The results of certain adjudicated entities (i.e. countries, economies and regions within countries), listed below, will be reported with annotations. Caution is required when interpreting estimates for these countries and economies because one or more PISA sampling standards listed below were not met. PISA sampling standards are referred to the PISA 2022 main sample (i.e., the sample of students assessed in mathematics, reading and science), and were not computed specifically for the financial literacy sample. Nevertheless, the same notes of caution apply when interpreting the financial literacy results.

• Overall exclusion rate. Standard 1.7: The PISA Defined Target Population covers 95% or more of the PISA Desired Target Population. That is, school-level exclusions and within-school exclusions combined do not exceed 5%.

• School response rate. Standard 1.11: The final weighted school response rate is at least 85% of sampled schools. If a response rate is below 85% then an acceptable response rate can still be achieved through agreed upon use of replacement schools.

• Student response rate. Standard 1.12: The student response rate is at least 80% of all sampled students across responding schools.

The full list of adjudicated entities not meeting the sampling standards is reported in Volume I. Considering the countries and economies that participated in the PISA 2022 financial literacy assessment, these entities can be grouped into two:

(i) Entities that submitted technically strong analyses, which indicated that more than minimal bias was most likely introduced in the estimates due to low response rates (falling below PISA standards): Canada.

(ii) Entities that did not meet one or more PISA sampling standards and it is not possible to exclude the possibility of more than minimal bias based on the information available at the time of data adjudication: Denmark, the Netherlands and the United States.

The Adjudication Group also noted that the bias associated with trend and cross-country comparisons might be smaller, if past data or data for other countries are biased in the same direction. Therefore, the deviations from the standards in PISA 2022 are compared with those in PISA 2018 where necessary.

(i) Entities that submitted technically strong analyses, which indicated that more than minimal bias was most likely introduced in the estimates due to low response rates (falling below PISA standards)

As PISA sampling standards are referred to the PISA 2022 main sample (i.e., the sample of students assessed in mathematics, reading and science), response and exclusion rates refer to the entire Canada. Nevertheless, the same note of caution applies when interpreting the results about the eight Canadian provinces that participated in the financial literacy assessment.

Canada

• Overall exclusion rate: 5.8%. Exclusions exceeded the acceptable rate by less than one percentage point; at the same time, the exclusion rates observed in 2022 remained relatively close to exclusion rates observed in 2018 (6.9%). When taking into account the fact that the PISA Une Heure (UH) forms (i.e., a special one-hour test prepared for students with special needs) were not available for the financial literacy assessment, the overall exclusion rate was 8.1% (Annex A2).

• Student response rate: 77%. School response rates: 81% before replacement, 86% after replacement. Student response rates decreased from 84% with respect to PISA 2018, and fell short of the target in 7 out of 10 provinces (all but New Brunswick, Prince Edward Island and Saskatchewan). A thorough non-response bias analysis was submitted, with analyses conducted separately for each province, using students' academic achievement data as auxiliary information. School response rates also fell short of the target, driven by low participation rates in two provinces (Alberta and Quebec). For these provinces, non-response bias was also examined at the school level. The analyses clearly indicate that school nonresponse has not led to any appreciable bias, but student nonresponse has given rise to a small upwards bias.

(ii) Entities that did not meet one or more PISA sampling standards and it is not possible to exclude the possibility of more than minimal bias based on the information available at the time of data adjudication.

Denmark

• Overall exclusion rate: 11.6%. Exclusions exceeded the acceptable rate by a large margin and showed a marked increase, with respect to 2018 (5.7%). The Adjudication Group noted that high levels of student exclusions may bias performance results upwards. In Denmark, a major cause behind the rise appears to be the increased share of students with diagnosed dyslexia, and the fact that more of these students are using electronic assistive devices to help them read on the screen, including during exams. The lack of such an accommodation for students with diagnosed dyslexia in the PISA assessment led schools to exclude many of these students. In order to reduce exclusion rates in the future, PISA may need to further accommodate dyslexic students, allowing the use of assistive devices. When taking into account the fact that the PISA Une Heure (UH) forms were not available for the financial literacy assessment, the overall exclusion rate was 14.3% (Annex A2).

The Netherlands

• Overall exclusion rate: 8.4%. Exclusions exceeded the acceptable rate by a large margin and showed a marked increase, with respect to 2018 (6.2%). Most of these students were excluded because they had a physical or intellectual disability and no adaptation was available for them. The Adjudication Group noted that high levels of student exclusions may bias performance results upwards. When taking into account the fact that the PISA Une Heure (UH) forms were not available for the financial literacy assessment, the overall exclusion rate was 11.6% (Annex A2).

• School response rates: 66% before replacement, 90% after replacement. A non-response bias analysis was submitted, analysing differences in performance and in other characteristics between responding schools and the total population of schools, as well as differences between replacement schools and originally sampled, but non-responding schools. This supported the case that no large bias would result from non-response; furthermore, given the available evidence, there is no clear indication about the direction of any residual bias.

The United States

• Exclusion rates: 6.1%. Exclusions exceeded the acceptable rate by a small margin but showed a marked increase, with respect to 2018 (3.8%), in exclusion rates for students with functional or intellectual disabilities. The Adjudication Group invited the national centres to investigate the reasons for this increase in exclusion rates and take remedial action for future cycles. It is expected that exclusion rates will fall again in the future, as a result. When taking into account the fact that the PISA Une Heure (UH) forms were not available for the financial literacy assessment, the overall exclusion rate was 10.5% (Annex A2).

• School response rates: 51% before replacement, 63% after replacement. School participation rates missed the standard by a substantial margin, and participation rates were particularly low among private schools (representing about 7% of the student population). A non-response bias analysis was submitted, indicating that, after replacement schools and non-response adjustments are taken into account, a number of characteristics (not including direct measures of school performance) are balanced across respondents and non-respondents. The Adjudication Group also noted that the response rate for students was only slightly above the target (80%). Based on the available information, it is not possible to exclude the possibility of bias, nor to determine its most likely direction.

Data underlying the figures

The data referred to in this volume are presented in Annex B and, in greater detail, including additional tables, on the PISA website (www.oecd.org/pisa). Five symbols are used to denote missing data:

• a The category does not apply in the country/economy concerned; data are therefore missing.

• c There were too few observations to provide reliable estimates (i.e. there were fewer than 30 students or fewer than 5 schools with valid data).

• m Data are not available. There was no observation in the sample; these data were not collected by the country/economy; or these data were collected but subsequently removed from the publication for technical reasons.

• w Results were withdrawn at the request of the country/economy concerned.

• x Data included in another category or column of the table (e.g. x(2) means that data are included in Column 2 of the table).

Coverage

This publication features data on 20 countries and economies, including 14 OECD countries and economies (Austria, the Flemish community of Belgium, eight Canadian provinces, Costa Rica, Czechia, Denmark, Hungary, Italy, the Netherlands, Norway, Poland, Portugal, Spain and the United States) and 6 partner countries (Brazil, Bulgaria, Malaysia, Peru, Saudi Arabia and the United Arab Emirates).

The Canadian provinces refer to the eight provinces of Canada that participated in the PISA 2022 financial literacy assessment: Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Nova Scotia, Ontario and Prince Edward Island.

Following OECD data regulations, a visual separation between countries and territories has been marked in italics in all charts to reduce the risk of data misinterpretation.

International averages

The OECD and overall averages correspond to the arithmetic mean of the respective country/economy estimates. They were calculated for most indicators presented in this report.

In this publication, the OECD and overall averages are generally used when the focus is on comparing performance across countries and economies. In the case of some countries and economies, data may not be available for specific indicators or specific categories may not apply. Readers should, therefore, keep in mind that the term “OECD average” or “Overall average” refers to the OECD or all countries and economies included in the respective comparisons. In cases where data are not available or do not apply for all sub-categories of a given population or indicator, the “OECD average” or “Overall average” is not necessarily computed on a consistent set of countries and economies across all columns of a table.

In analyses involving data from multiple years, the OECD and overall averages are always reported on consistent sets of countries and economies, and several averages may be reported in the same table. For instance, the “OECD average - 2018” includes only those OECD countries and economies that participated in both the PISA 2018 and 2022 financial literacy assessments. This restriction allows for valid comparisons of the OECD average over time.

The averages used are:

• OECD average: Arithmetic mean across all OECD countries and economies that participated in the PISA 2022 financial literacy assessment (Austria, the Flemish community of Belgium, the Canadian provinces, Costa Rica, Czechia, Denmark, Hungary, Italy, the Netherlands, Norway, Poland, Portugal, Spain and the United States).

• OECD average - 2012: Arithmetic mean across all OECD countries and economies that participated in both the PISA 2012 and 2022 financial literacy assessments (the Flemish community of Belgium, Czechia, Italy, Poland, Spain and the United States).

• OECD average - 2015: Arithmetic mean across all OECD countries and economies that participated in both the PISA 2015 and 2022 financial literacy assessments1 (the Flemish community of Belgium, Italy, the Netherlands, Poland, Spain and the United States).

• OECD average - 2018: Arithmetic mean across all OECD countries and economies that participated in both the PISA 2018 and 2022 financial literacy assessments2 (Italy, Poland, Portugal, Spain and the United States).

• Overall average: Arithmetic mean across all countries and economies that participated in the PISA 2022 financial literacy assessment (Austria, the Flemish community of Belgium, Brazil, Bulgaria, the Canadian provinces, Costa Rica, Czechia, Denmark, Hungary, Italy, Malaysia, the Netherlands, Norway, Peru, Poland, Portugal, Saudi Arabia, Spain, the United Arab Emirates and the United States).

• Overall average – 2012: Arithmetic mean across all countries and economies that participated in both the PISA 2012 and 2022 financial literacy assessments (the Flemish community of Belgium, Czechia, Italy, Poland, Spain and the United States).

• Overall average – 2015: Arithmetic mean across all countries and economies that participated in both the PISA 2015 and 2022 financial literacy assessments3 (the Flemish community of Belgium, Brazil, Italy, the Netherlands, Peru, Poland, Spain and the United States).

• Overall average – 2018: Arithmetic mean across all countries and economies that participated in both the PISA 2018 and 2022 financial literacy assessments4 (Brazil, Bulgaria, Italy, Peru, Poland, Portugal, Spain and the United States).

Rounding figures

Because of rounding, some figures in tables may not add up exactly to the totals. Totals, differences and averages are always calculated on the basis of exact numbers and are rounded only after calculation.

All standard errors in this publication have been rounded to one or two decimal places. Where the value 0.0 or 0.00 is shown, this does not imply that the standard error is zero, but that it is smaller than 0.05 or 0.005, respectively.

Reporting student data

The report uses “15-year-olds” as shorthand for the PISA target population. PISA covers students who are aged between 15 years 3 months and 16 years 2 months at the time of assessment and who are enrolled in school and have completed at least 6 years of formal schooling, regardless of the type of institution in which they are enrolled, and whether they are in full-time or part-time education, whether they attend academic or vocational programmes, and whether they attend public or private schools or foreign schools within the country.

Reporting school data

The principals of the schools in which students were assessed provided information on their schools’ characteristics by completing a school questionnaire. Where responses from school principals are presented in this publication, they are weighted so that they are proportionate to the number of 15-year-olds enrolled in the school.

Focusing on statistically significant differences

This volume discusses only statistically significant differences or changes. These are denoted in darker colours in figures and in bold font in tables. Unless otherwise specified, the significance level is set to 5%. See Annex A3 for further information.

Abbreviations used in this report

 

ESCS	PISA index of economic, social and cultural status
GDP	Gross domestic product
ISCED	International Standard Classification of Education
ICT	Information and communications technology
PPP	Purchasing power parity
S.D.	Standard deviation
S.E.	Standard error
Score-dif.	Score-point difference
%-dif.	Percentage-point difference

Further documentation

For further information on the PISA assessment instruments and the methods used in PISA, see the PISA 2022 Technical Report (OECD, 2023[1]).

StatLink

This report has StatLinks for tables and graphs. To download the matching Excel® spreadsheet, just type the link into your Internet browser, starting with the http://dx.doi.org prefix, or click on the link from the e-book version.

Box 1. Interpreting differences in PISA scores

PISA scores do not have a substantive meaning as they are not physical units such as metres or grams. Instead, they are set in relation to the variation in results observed across all test participants. There is, theoretically, no minimum or maximum score in PISA; rather, the results are scaled to fit approximately normal distributions (i.e. means around 500 score points, standard deviations around 100 score points). In statistical terms, a one-point difference on the PISA scale therefore corresponds to an effect size (Cohen’s d) of 0.01; and a 10-point difference to an effect size of 0.10.

Interpreting large differences in scores: proficiency levels

PISA scales are divided into proficiency levels. For example, for PISA 2022, the range of difficulty of financial literacy items is represented by five levels of financial literacy proficiency: the simplest items correspond to Level 1; Levels 2, 3, 4 and 5 correspond to increasingly difficult items. Individuals who are proficient within the range of Level 1 are likely to be able to complete Level 1 items but are unlikely to be able to complete items at higher levels. See Chapter 2 for a detailed description of proficiency levels in financial literacy.

In financial literacy, each proficiency level corresponds to a range of about 75 score points. Hence, score-point differences of that magnitude can be interpreted as the difference in described skills and knowledge between successive proficiency levels.

Interpreting small differences in scores: statistical significance

Smaller differences in PISA scores cannot be expressed in terms of the difference in skills and knowledge between proficiency levels. However, they can still be compared with each other by means of verifying their “statistical significance”.

A difference is called “statistically significant” if it is unlikely that such a difference can be observed in the estimates based on samples when, in fact, no true difference exists in the populations from which the samples are drawn. The results of the PISA assessments are “estimates” because they are obtained from samples of students rather than from a census of all students (i.e. which introduces a “sampling error”), and because they are obtained using a limited set of assessment tasks rather than the universe of all possible assessment tasks (i.e. which introduces a “measurement error”).

It is possible to determine the magnitude of the uncertainty associated with the estimate and to represent it as a “confidence interval”, i.e. a range defined in such a way that if the true value lies above its upper bound or below its lower bound, an estimate different from the reported estimate would be observed only with a small probability (typically less than 5%). The confidence interval needs to be taken into account when making comparisons between estimates so that differences that may arise simply due to the sampling error and measurement error are not interpreted as real differences.

Interpreting differences in scores across PISA assessments

To ensure the comparability of PISA results across different assessment years, “link errors” must be used. The link error represents uncertainty around scale values (“is a financial literacy score of 432 in PISA 2022 the same as 432 in PISA 2018?”) and is therefore independent of the size of the student sample. For comparisons between financial literacy results in PISA 2022 and financial literacy results in 2018, the link error corresponds to 2.20 score points. For detailed information, see Box IV.2.2 in Chapter 2 and Annex A5.

References

[1] OECD (2023), PISA 2022 Technical Report, OECD Publishing, Paris, https://doi.org/10.1787/01820d6d-en.

Notes

← 1.  The Canadian provinces are not included in this average as Alberta did not participate in the PISA 2015 financial literacy assessment, making comparisons unreliable with results from the PISA 2022 financial literacy assessment.

← 2.  The Canadian provinces are not included in this average as Alberta did not participate in the PISA 2018 financial literacy assessment, making comparisons unreliable with results from the PISA 2022 financial literacy assessment.

← 3.  The Canadian provinces are not included in this average as Alberta did not participate in the PISA 2015 financial literacy assessment, making comparisons unreliable with results from the PISA 2022 financial literacy assessment.

← 4.  The Canadian provinces are not included in this average as Alberta did not participate in the PISA 2018 financial literacy assessment, making comparisons unreliable with results from the PISA 2022 financial literacy assessment.