PISA 2022 Results (Volume II)

Annex A3. Technical notes on analyses in this volume

Standard errors, confidence intervals, significance test and p-values

The statistics in this report represent estimates based on samples of students, rather than values that could be calculated if every student in every country had answered every question. Consequently, it is important to measure the degree of uncertainty in the estimates. In PISA, each estimate has an associated degree of uncertainty, which is expressed through a standard error. The use of confidence intervals provides a way of making inferences about the population parameters (e.g. means and proportions) in a manner that reflects the uncertainty associated with the sample estimates. If numerous different samples were drawn from the same population, according to the same procedures as the original sample, then in 95 out of 100 samples the calculated confidence interval would encompass the true population parameter. For many parameters, sample estimators follow a normal distribution, and the 95% confidence interval can be constructed as the estimated parameter, plus or minus 1.96 times the associated standard error.

In many cases, readers are primarily interested in whether a given value in a particular country is different from a second value in the same or another country, e.g. whether students in public schools perform better than students in private schools in the same country. In the tables and figures used in this report, differences are labelled as statistically significant when a difference of that size or larger, in either direction, would be observed less than 5% of the time in samples, if there were no difference in corresponding population values. In other words, the risk of reporting a difference as significant when such difference, in fact, does not exist, is contained at 5%.

Statistical significance of differences related to type of school and differences between subgroup means

Differences in student performance by type of school or other indices were tested for statistical significance. Positive differences indicate higher scores for students in private schools while negative differences indicate higher scores for students in public schools. Generally, differences marked in bold in the tables in this volume are statistically significant at the 95% confidence level.

Similarly, differences between other groups of students (e.g. students in urban schools and students in rural schools, or socio-economically advantaged and disadvantaged students) were tested for statistical significance. The definitions of the subgroups can, in general, be found in the tables and the text accompanying the analysis. All differences marked in bold in the tables presented in Annex B of this report are statistically significant at the 95% level, unless otherwise indicated.

Statistical significance of differences between subgroup means, after accounting for other variables

For many tables, subgroup comparisons were performed both on the observed difference (“before accounting for other variables”) and after accounting for other variables, such as the PISA index of economic, social and cultural status of students. The adjusted differences were estimated using linear regression and tested for significance at the 95% confidence level. Significant differences are marked in bold.

Statistical significance of performance differences between the top and bottom quartiles of PISA indices and scales

Differences in average performance between the top and bottom quarters of the PISA indices and scales were tested for statistical significance. Figures marked in bold indicate that performance between the top and bottom quarters of students on the respective index is statistically significantly different at the 95% confidence level.

Statistical significance of relationships between PISA items, indices and scales at the system level

Relationships between two variables at the system level (e.g. the relationship between disciplinary climate and mathematics performance across education systems) were also tested for statistical significance. Figures marked in bold indicate that a positive or negative relationship between two variables is statistically significant at the 95% confidence level. Figures marked in italics indicate relationships between two variables that are marginally significant (90% confidence level).

Change in the performance per unit of an index

The difference in student performance per unit of an index was calculated in many tables. Figures in bold indicate that the differences are statistically and significantly different from zero at the 95% confidence level.

Odds ratios

The odds ratio is a measure of the relative likelihood of a particular outcome across two groups. The odds ratio for observing the outcome when an antecedent is present is simply

$\mathrm{O}\mathrm{R}=\frac{\left(p_{11}/p_{12}\right)}{\left(p_{21}/p_{22}\right)}$

where $p_{11}/p_{12}$represents the “odds” of observing the outcome when the antecedent is present, and $p_{21}/p_{22}$ represents the “odds” of observing the outcome when the antecedent is not present.

Logistic regression can be used to estimate the log ratio: the exponentiated logit coefficient for a binary variable is equivalent to the odds ratio. A “generalised” odds ratio, after accounting for other differences across groups, can be estimated by introducing control variables in the logistic regression.

Statistical significance of odds ratios

Figures in bold in the data tables presented in Annex B1 of this report indicate that the odds ratio is statistically significantly different from 1 at the 95% confidence level. To construct a 95% confidence interval for the odds ratio, the estimator is assumed to follow a log-normal distribution, rather than a normal distribution.

In some tables, odds ratios after accounting for other variables are also presented. These odds ratios were estimated using logistic regression and tested for significance against the null hypothesis of an odds ratio equal to one (i.e. equal likelihoods, after accounting for other variables).

Use of student weights

The target population in PISA is 15-year-old students, but a two-stage sampling procedure was used. After the population was defined, school samples were selected with a probability proportional to the expected number of eligible students in each school. Only in a second sampling stage were students drawn from among the eligible students in each selected school.

Although the student samples were drawn from within a sample of schools, the school sample was designed to optimise the resulting sample of students, rather than to give an optimal sample of schools. It is therefore preferable to analyse the school-level variables as attributes of students (e.g. in terms of the share of 15-year-old students affected), rather than as elements in their own right.

Most analyses of student and school characteristics are therefore weighted by student final weights (or their sum, in the case of school characteristics), and use student replicate weights for estimating standard errors.

Calculation of the coverage for items and indices about COVID-19 school closures

In the student questionnaire, the question about the duration of school closures because of COVID-19 (ST347Q01JA) filters access to all questions regarding students’ experiences during this period. This concerns most of the questions covered in Chapter 2 of this volume as well as the indices derived from students’ answers to those questions (items belonging to questions ST348, ST352, ST354 and ST355, and the indices SCHSUST, PROBSELF, FAMUPSL, FEELLAH and SDLEFF).

When students had answered “No” to question about COVID-19 school closures, their missing responses to any of the other questions about school closures were considered as “valid skips” or “not applicable” in the PISA 2022 database. In the case of students who had answered that their school was closed for some time because of the pandemic but did not respond to the other questions about school closures, their missing responses were treated as “invalid”, “no response” or “system”. All of these responses are considered as “missing” in the PISA 2022 database.

To calculate the correct coverage for the questions requires reducing the sample to students who had one of the three missing values “invalid”, “no response” or “system”. For indices, the sample is restricted to students who answered “yes” to the question about the duration of school closures because of COVID-19 (ST347Q01JA), as indices can be built even if not all items were answered. The restriction of the sample does not modify the value of a result, but only its coverage (i.e. the information regarding the proportion of the sample covered shown next to the standard errors in tables).

The PISA 2022 Technical Report explains the different types of missing data (OECD, forthcoming[1]).

Calculation of the isolation index

The isolation index used in the report corresponds to the normalised exposure indicator (Frankel and Volij, 2011[2]),

$I=1-\frac{\sum _{j=1}^J\frac{n_j^a}{N^a}\frac{(1-n_j^a)}{n_j}}{1-p^a}$

where $n_j^a$ and $N^a$ stand for the number of students of type a (for instance, those with an immigrant background) in school j and in the education system; $n_j$ is the total number of students in school j; and $p^a=\frac{n^a}{N}$ is the proportion of type a students in the education system. The index ranges from 0 (full exposure) to 1 (full isolation), meaning that the index increases with the concentration of type a students in a limited number of schools. In the report, the index is used for measuring the isolation in schools of the following groups of students:

• Socio-economically advantaged and disadvantaged students (defined as those in the first and the fourth quarters, respectively, of the national distribution of the ESCS index)

• Low and high achievers in mathematics (defined as those in the first and the fourth quarters, respectively, of the national distribution of mathematics performance)

• Students with an immigrant background

• Boys and girls

School-level results and modal grade schools

The interpretation of school-level results (e.g. comparisons between upper secondary education/ISCED level 3 and lower secondary education/ISCED level 2) depends on how schools are defined and organised within countries and by the units that were chosen for sampling purposes. For example, in some countries, some of the schools in the PISA sample were defined as managerial units (even if they spanned several geographically separate institutions, as in the Netherlands); in others, they were defined as those parts of larger educational institutions that serve 15-year-olds; in still others they were defined as physical school buildings; and in others they were defined from a management perspective (e.g. entities having a principal). The PISA 2022 Technical Report (OECD, forthcoming[1]) and Annex A2 provide an overview of how schools are defined. In Slovenia, for example, the primary sampling unit for students attending upper secondary/ISCED level 3 is defined as a group of students who follow the same study programme within a school (an education track within a school).

Some indicators in Chapter 5 (namely full-time, part-time teachers, certified teachers, student-teacher ratio), were calculated specifically for schools with the “modal ISCED level” for 15-year-old students. The “modal ISCED level” is defined here as the level attended by at least one-third of the PISA sample. As PISA students are sampled to represent all 15-year-old students, whatever type of schools they are enrolled in, they may not be representative of their schools. Restricting the sampling to schools with the modal ISCED level for 15-year-old students ensures that the characteristics of students sampled for PISA represent the profile of the typical student attending the school. Modal grade may be lower secondary, upper secondary or both (lower and upper secondary education can be provided in the same school). As the restriction is made at the school level, some students from a grade other than the modal grade in the country may also be used in the analysis. The PISA 2022 Technical Report (OECD, forthcoming[1]) and Annex A2 provide an overview of how schools are defined.

Overall ratios and average ratios

In this report, the comparisons of ratios related to teachers, such as student-teacher ratio or the proportion of fully certified teachers, are made using overall and modal-grade restricted ratios. This means that ratios are obtained by dividing the total number of students in the target population (either from all schools attended by 15-year-olds or restricted to schools with the modal ISCED level for 15-year-old students) by the total number of teachers in the target population. The ratios are computed by first computing the numerator and denominator as the (weighted) sum of school-level totals, then dividing the numerator by the denominator. Similar estimations are made for the proportion of teachers with at least a bachelor’s, master’s or doctoral qualification, the proportion of fully certified teachers, and the proportion of teachers working part time or full time. In most cases (i.e. unless all schools are exactly the same size) this overall ratio differs from the modal-grade restricted ratios.

Time in regular lessons

In PISA 2022, time in regular lessons was calculated by combining answers from the student and school principal questionnaire and is not comparable to PISA 2018. Each student reported the number of class periods she/he is required to attend in all subjects per week, and school principals reported the average number of minutes per class period attended by students in modal grade. This combination may create some noise induced by the potential misreporting or misunderstanding of the definition of a class period, either by students or school principals.

Calculation of relationships between school and system characteristics and indicators of education systems’ resilience

System-level correlations and partial correlations were used to examine whether characteristics of education systems are potential components of resilience. The relationships between various characteristics of education systems (e.g. the average disciplinary climate or the percentage of students who had attended pre-primary education for at least one year), on the one hand, and all indicators of resilience (e.g. average mathematics performance, socio-economic fairness and sense of belonging at school), on the other hand, were tested for their statistical significance and direction.

Correlational analyses were conducted separately for OECD countries and for all countries and economies that participated in PISA 2022. In addition, correlations were computed before and after accounting for per capita GDP, to account for the level of economic development of a country/economy. The system-level correlation tables show correlation coefficients and partial correlation coefficients as well as their significance. Figures display the percentage of the variation of the resilience indicators (e.g. average mathematics performance) that is explained by the system characteristic under study (e.g. average disciplinary climate). Relationships between trends (e.g. change in bullying and performance in mathematics) refer to the same period. While most trends refer to the 2018-2022 period, a few system characteristics were not measured in 2018 (e.g. disciplinary climate in mathematics lessons). In these cases, the trend relationship refers to the trend between 2012 and 2022 (e.g. change between 2012 and 2022 in disciplinary climate and mathematics performance). No trend relationships were reported for sense of belonging in these cases because the index of sense of belonging at school in 2012 and 2022 were not comparable.

Some considerations when interpreting the PISA results

Cross-national and cross-cultural comparability of the PISA data

PISA 2022 asked students and school principals to answer questions about the organisation of schools, and the social and economic contexts in which learning takes place. These are reports provided by principals and students themselves rather than external observations, and thus may be influenced by cultural differences in how individuals respond.

While PISA aims to maximise the cross-national and cross-cultural comparability of complex constructs, it must do so while keeping the questionnaires relatively short and minimising the perceived intrusiveness of the questions. Despite the extensive investments PISA makes in monitoring the process of translation, standardising the administration of the assessment, selecting questions and analysing the quality of the data, full comparability across countries and subpopulations cannot always be guaranteed.

The indicators of school climate and well-being analysed in this report are based on students’ and principals’ reports, which are susceptible to several possible measurement errors: memory decay; social desirability (the tendency to respond in a manner that is more acceptable in one’s own social and cultural context, reference-group bias (what the comparison group is); and response-style bias (e.g. straight-lining, over-reporting, modesty, heaping, acquiescence) (Pekrun, 2020[3]; Harzing et al., 2012[4]; Spooren, Mortelmans and Thijssen, 2012[5]). These biases can operate differently in different cultural contexts, thus limiting the cross-country comparability of responses (Benítez, Van de Vijver and Padilla, 2019[6]; Van de Vijver et al., 2019[7]). Above all, readers should be particularly cautious when interpreting indicators with a strong subjective component, such as life satisfaction and student feelings, which are more likely to be influenced by cultural norms and the personality of the respondent.

In order to minimise the risk of misleading interpretations, a number of reliability and invariance analyses of the PISA indices used in this report have been carried out (see Annex A1 and the PISA 2022 Technical Report (OECD, forthcoming[1]) for more details), providing readers with an indication of how reliable cross-country comparisons are.

Interpreting information from principals and school-level results

In addition to the general constraints of self-reported data, there are other limitations, particularly those concerning the information collected from principals or the interpretation of school-level results, that should be taken into account when interpreting the data.

• The learning environment examined by PISA may only partially reflect that which shaped students’ experiences in education earlier in their school careers, particularly in school systems where students progress through different types of educational institutions at the pre-primary, primary, lower secondary and upper secondary levels. To the extent that students’ current learning environment differs from that of their earlier school years, the contextual data collected by PISA are an imperfect proxy for students’ cumulative learning environments, and the effects of those environments on learning outcomes is likely to be underestimated. In most cases, 15-year-old students have been in their current school for less than three years (Table II.B1.2.3). This means that much of their academic development took place earlier, in other schools, which may have little or no connection with the school in which they were enrolled when they sat the PISA test. Parents may have fewer opportunities to interact with the school staff when their child has been attending the new school for just a few months. Students may have also spent too little time in the new school to develop a strong attachment to the school.

• In some countries, 15-year-old students have already transitioned into upper secondary education, while in others they are still in lower secondary education. Some of the questions may be influenced by the education level in which students are enrolled, especially in those countries where transitioning into upper secondary education means transferring into a new school.

• In some countries and economies, the definition of the school in which students are taught is not straightforward because schools vary in the level and purpose of education. For example, in some countries and economies, subunits within schools (e.g. study programmes, shifts and campuses) were sampled instead of schools as administrative units (see above).

• Although principals can provide information about their schools, generalising from a single source of information for each school and then matching that information with students’ reports is not straightforward. Also, principals’ perceptions may not be the most accurate source for some information related to teachers, such as teachers’ morale and commitment.

• The age-based sampling followed in PISA means that, in some education systems, students are not always representative of their schools. Interpreting differences between schools appropriately therefore requires specific knowledge about how school systems are structured.

• When presenting results by the socio-economic profile of schools, the location of schools, the type of school or the education level, the number of students and schools in each subsample has to meet the PISA reporting requirements of at least 30 students and 5 schools. Even when these reporting requirements are met, the reader should interpret the results cautiously when the number of students or schools is just above the threshold.

Despite these caveats, information from the school questionnaire provides unique insights into the ways in which national and subnational authorities seek to realise their education objectives.

Schooling and school effects

In using results from non-experimental data on school performance, such as the PISA database, it is important to bear in mind the distinction between school effects and the effects of schooling, particularly when interpreting the modest association between factors such as school resources, policies and institutional characteristics, on the one hand, and student performance, on the other. School effects are education researchers’ shorthand for the effect on academic performance of attending one school or another, usually schools that differ in resources or policies and institutional characteristics. Where schools and school systems do not vary in fundamental ways, the school effect can be modest. Nevertheless, modest school effects should not be confused with a lack of an effect of schooling (the influence on performance of not being schooled compared with being schooled).

Interpreting correlations and changes over time

A correlation indicates the strength and direction of a linear relationship, either positive or negative, between two variables. A correlation is a simple statistic that measures the degree to which two variables are associated with each other; it does not prove causality between the two.

Comparisons of results between resources, policies and practices, and mathematics performance across time (trends analyses) should also be interpreted with caution. Changes in the strength of the relationship between characteristics of education systems and education outcomes (e.g. mathematics performance) cannot be considered causal because they can occur for two key reasons. First, a particular set of resources, policies and practices might have been chosen by higher-performing students (or higher-performing schools or high-performing systems) while that set of resources, policies and practices might not have existed in lower-performing students/schools/systems. Under this interpretation, the relationship between mathematics performance, and resources, policies and practices is stronger because they are available to higher-performing students/schools/systems. Second, a particular set of resources, policies and practices may have been used more extensively in 2022 than earlier, and may have promoted student learning more in 2022 than before. PISA trend data indicate where changes have occurred. However, in order to understand the nature of the change, further analysis is needed.

Interpreting results before and after accounting for socio-economic status

When examining the relationship between education outcomes and resources, policies and practices within school systems, this volume takes into account socio-economic differences among students, schools and systems. The advantage of doing this lies in comparing similar entities, namely students, schools and systems with similar socio-economic profiles. At the same time, there is a risk that such adjusted comparisons underestimate the strength of the relationship between student performance and resources, policies and practices, since most of the differences in performance are often attributable to both policies and socio-economic status.

Conversely, analyses that do not take socio-economic status into account can overstate the relationship between student performance and resources, policies and practices, as the level of resources and the kinds of policies adopted may also be related to the socio-economic profile of students, schools and systems. At the same time, analyses without adjustments may paint a more realistic picture of the schools that parents choose for their children. They may also provide more information for other stakeholders who are interested in the overall performance of students, schools and systems, including any effects that may be related to the socio-economic profile of schools and systems. For example, parents may be primarily interested in a school’s absolute performance standards, even if that school’s higher achievement record stems partially from the fact that the school has a larger proportion of advantaged students.

For the system-level analyses, correlations are examined before and after accounting for per capita GDP in order to account for the extent to which the observed relationships are influenced by countries’/economies’ level of economic development.

References

[6] Benítez, I., F. Van de Vijver and J. Padilla (2019), “A Mixed Methods Approach to the Analysis of Bias in Cross-cultural Studies”, Sociological Methods & Research, Vol. 51/1, pp. 237-270, https://doi.org/10.1177/0049124119852390.

[2] Frankel, D. and O. Volij (2011), “Measuring school segregation”, Journal of Economic Theory, Vol. 146/1, pp. 1-38, https://doi.org/10.1016/j.jet.2010.10.008.

[4] Harzing, A. et al. (2012), “Response Style Differences in Cross-National Research”, Management International Review, Vol. 52/3, pp. 341-363, https://doi.org/10.1007/s11575-011-0111-2.

[1] OECD (forthcoming), PISA 2022 Technical Report, OECD Publishing, Paris.

[3] Pekrun, R. (2020), “Self-Report is Indispensable to Assess Students’ Learning”, Frontline Learning Research, Vol. 8/3, pp. 185-193, https://doi.org/10.14786/flr.v8i3.637.

[5] Spooren, P., D. Mortelmans and P. Thijssen (2012), “‘Content’ versus ‘style’: acquiescence in student evaluation of teaching?”, British Educational Research Journal, Vol. 38/1, pp. 3-21, https://doi.org/10.1080/01411926.2010.523453.

[7] Van de Vijver, F. et al. (2019), “Invariance analyses in large-scale studies”, OECD Education Working Papers, No. 201, OECD Publishing, Paris, https://doi.org/10.1787/254738dd-en.