PISA 2022 Results (Volume V)

Annex A3. Technical notes on analyses in this volume

Standard errors, confidence intervals, significance test and p-values

The statistics in this report represent estimates based on samples of students, rather than values that could be calculated if every student in every country had answered every question. Consequently, it is important to measure the degree of uncertainty in the estimates. In PISA, each estimate has an associated degree of uncertainty, which is expressed through a standard error. The use of confidence intervals provides a way of making inferences about the population parameters (e.g. means and proportions) in a manner that reflects the uncertainty associated with the sample estimates. If numerous different samples were drawn from the same population, according to the same procedures as the original sample, then in 95 out of 100 samples the calculated confidence interval would encompass the true population parameter. For many parameters, sample estimators follow a normal distribution, and the 95% confidence interval can be constructed as the estimated parameter, plus or minus 1.96 times the associated standard error.

In many cases, readers are primarily interested in whether a given value in a particular country is different from a second value in the same or another country, e.g. whether students with immigrant background perform better than students without an immigrant background in the same country. In the tables and figures used in this report, differences are labelled as statistically significant when a difference of that size or larger, in either direction, would be observed less than 5% of the time in samples, if there were no difference in corresponding population values. In other words, the risk of reporting a difference as significant when such difference, in fact, does not exist, is contained at 5%.

Statistical significance of differences between subgroup means, after accounting for other variables

For many tables, subgroup comparisons were performed both on the observed difference (“before accounting for other variables”) and after accounting for other variables, such as the PISA index of economic, social and cultural status of students. The adjusted differences were estimated using linear regression and tested for significance at the 95% confidence level. Significant differences are marked in bold.

Statistical significance of performance differences between the top and bottom quartiles of PISA indices and scales

Differences in average performance between the top and bottom quarters of the PISA indices and scales were tested for statistical significance. Figures marked in bold indicate that performance between the top and bottom quarters of students on the respective index is statistically significantly different at the 95% confidence level.

Statistical significance of relationships between PISA items, indices and scales at the system level

Relationships between two variables at the system level (e.g. the relationship between cognitive activation practices and mathematics performance across education systems) were also tested for statistical significance. Figures marked in bold indicate that a positive or negative relationship between two variables is statistically significant at the 95% confidence level. Figures marked in italics indicate relationships between two variables that are marginally significant (90% confidence level).

Change in the performance per unit of an index

The difference in student performance per unit of an index was calculated in many tables. Figures in bold indicate that the differences are statistically and significantly different from zero at the 95% confidence level.

Odds ratios

The odds ratio is a measure of the relative likelihood of a particular outcome across two groups. The odds ratio for observing the outcome when an antecedent is present is simply:

where $p_{11}/p_{12}$ represents the “odds” of observing the outcome when the antecedent is present, and $p_{21}/p_{22}$ represents the “odds” of observing the outcome when the antecedent is not present.

Logistic regression can be used to estimate the log ratio: the exponentiated logit coefficient for a binary variable is equivalent to the odds ratio. A “generalised” odds ratio, after accounting for other differences across groups, can be estimated by introducing control variables in the logistic regression.

Statistical significance of odds ratios

Figures in bold in the data tables presented in Annex B1 of this report indicate that the odds ratio is statistically significantly different from 1 at the 95% confidence level. To construct a 95% confidence interval for the odds ratio, the estimator is assumed to follow a log-normal distribution, rather than a normal distribution.

In some tables, odds ratios after accounting for other variables are also presented. These odds ratios were estimated using logistic regression and tested for significance against the null hypothesis of an odds ratio equal to one (i.e. equal likelihoods, after accounting for other variables).

Use of student weights

The target population in PISA is 15-year-old students, but a two-stage sampling procedure was used. After the population was defined, school samples were selected with a probability proportional to the expected number of eligible students in each school. Only in a second sampling stage were students drawn from among the eligible students in each selected school.

Although the student samples were drawn from within a sample of schools, the school sample was designed to optimise the resulting sample of students, rather than to give an optimal sample of schools. It is therefore preferable to analyse the school-level variables as attributes of students (e.g. in terms of the share of 15-year-old students affected), rather than as elements in their own right.

Most analyses of student and school characteristics are therefore weighted by student final weights (or their sum, in the case of school characteristics), and use student replicate weights for estimating standard errors.

Some considerations when interpreting the PISA results

Cross-national and cross-cultural comparability of the PISA data

PISA 2022 asked students and teachers to answer questions about learning and teaching practices. These are reports provided by teachers and students themselves rather than external observations, and thus may be influenced by cultural differences in how individuals respond.

While PISA aims to maximise the cross-national and cross-cultural comparability of complex constructs, it must do so while keeping the questionnaires relatively short and minimising the perceived intrusiveness of the questions. Despite the extensive investments PISA makes in monitoring the process of translation, standardising the administration of the assessment, selecting questions and analysing the quality of the data, full comparability across countries and subpopulations cannot always be guaranteed.

In order to minimise the risk of misleading interpretations, a number of reliability and invariance analyses of the PISA indices used in this report have been carried out (see Annex A1 and the PISA 2022 Technical Report ( (OECD, 2024[1]) for more details), providing readers with an indication of how reliable cross-country comparisons are.

Interpreting correlations and changes over time

A correlation indicates the strength and direction of a linear relationship, either positive or negative, between two variables. A correlation is a simple statistic that measures the degree to which two variables are associated with each other; it does not prove causality between the two.

Comparisons of results between resources, policies and practices, and mathematics performance across time (trends analyses) should also be interpreted with caution. Changes in the strength of the relationship between characteristics of education systems and education outcomes (e.g. mathematics performance) cannot be considered causal because they can occur for two key reasons. First, a particular set of resources, policies and practices might have been chosen by higher-performing students (or higher-performing schools or high-performing systems) while that set of resources, policies and practices might not have existed in lower-performing students/schools/systems. Under this interpretation, the relationship between mathematics performance, and resources, policies and practices is stronger because they are available to higher-performing students/schools/systems. Second, a particular set of resources, policies and practices may have been used more extensively in 2022 than earlier, and may have promoted student learning more in 2022 than before. PISA trend data indicate where changes have occurred. However, in order to understand the nature of the change, further analysis is needed.

Interpreting results before and after accounting for socio-economic status

When examining the relationship between education outcomes and resources, policies and practices within school systems, this volume takes into account socio-economic differences among students, schools and systems. The advantage of doing this lies in comparing similar entities, namely students, schools and systems with similar socio-economic profiles. At the same time, there is a risk that such adjusted comparisons underestimate the strength of the relationship between student performance and resources, policies and practices, since most of the differences in performance are often attributable to both policies and socio-economic status.

Conversely, analyses that do not take socio-economic status into account can overstate the relationship between student performance and resources, policies and practices, as the level of resources and the kinds of policies adopted may also be related to the socio-economic profile of students, schools and systems. At the same time, analyses without adjustments may paint a more realistic picture of the schools that parents choose for their children. They may also provide more information for other stakeholders who are interested in the overall performance of students, schools and systems, including any effects that may be related to the socio-economic profile of schools and systems. For example, parents may be primarily interested in a school’s absolute performance standards, even if that school’s higher achievement record stems partially from the fact that the school has a larger proportion of advantaged students.

For the system-level analyses, correlations are examined before and after accounting for per capita GDP in order to account for the extent to which the observed relationships are influenced by countries’/economies’ level of economic development.

References

[6] Benítez, I., F. Van de Vijver and J. Padilla (2019), “A Mixed Methods Approach to the Analysis of Bias in Cross-cultural Studies”, Sociological Methods & Research, Vol. 51/1, pp. 237-270, https://doi.org/10.1177/0049124119852390.

[8] Frankel, D. and O. Volij (2011), “Measuring school segregation”, Journal of Economic Theory, Vol. 146/1, pp. 1-38, https://doi.org/10.1016/j.jet.2010.10.008.

[4] Harzing, A. et al. (2012), “Response Style Differences in Cross-National Research”, Management International Review, Vol. 52/3, pp. 341-363, https://doi.org/10.1007/s11575-011-0111-2.

[1] OECD (2024), PISA 2022 Technical Report, PISA, OECD Publishing, Paris, https://doi.org/10.1787/01820d6d-en.

[2] OECD (forthcoming), PISA 2022 Technical Report, OECD Publishing, Paris.

[3] Pekrun, R. (2020), “Self-Report is Indispensable to Assess Students’ Learning”, Frontline Learning Research, Vol. 8/3, pp. 185-193, https://doi.org/10.14786/flr.v8i3.637.

[5] Spooren, P., D. Mortelmans and P. Thijssen (2012), “‘Content’ versus ‘style’: acquiescence in student evaluation of teaching?”, British Educational Research Journal, Vol. 38/1, pp. 3-21, https://doi.org/10.1080/01411926.2010.523453.

[7] Van de Vijver, F. et al. (2019), “Invariance analyses in large-scale studies”, OECD Education Working Papers, No. 201, OECD Publishing, Paris, https://doi.org/10.1787/254738dd-en.