PISA 2022 Results (Volume IV)

Annex A2. The PISA target population, the PISA samples and the definition of schools

This annex to the PISA 2022 results provides further technical details on how the assessment covered its target population of 15-year-olds, how its national samples represent this population across participating countries and economies, and how the sampling procedure was adapted to accurately represent diverse education systems worldwide. The information presented below is, for the most part, a summary of the information presented in Annex A2 of PISA 2022 Results (Volume I): The State of Learning and Equity in Education (OECD, 2023[1]); the reader is invited to refer to that volume for more details. This annex also includes information specific to the financial literacy sample.

Who is the PISA target population?

PISA 2022 assessed the cumulative outcomes of education and learning at a point at which most young people are still enrolled in formal education: when they are 15 years old.

International surveys of education outcomes must guarantee the comparability of their target population across participating countries and economies. One way to do this is to assess students at the same grade level. However, differences between countries in the nature and extent of early childhood education and care, age at entry into primary education, and the overall institutional structure of education systems do not allow for a definition of internationally comparable grade levels.

Other international assessments have defined their target population by the grade level that provides maximum coverage of a particular age cohort. However, this definition leads to a population particularly sensitive to the distribution of students across age and grade levels, where small changes – of assessment dates, or month of entry into primary education – can lead to the selection of different target grades. There also may be differences across or within countries in whether students who are older or younger than the desired age cohort are represented in the modal grade, further rendering such grade level-based samples difficult to compare.

To overcome these problems, PISA uses an age-based definition of its target population, one that is not tied to the institutional structures of national education systems.1 PISA assesses students who are aged between 15 years and 3 (complete) months and 16 years and 2 (complete) months2 at the beginning of the assessment period, plus or minus an allowed 1-month variation, and who are enrolled in an educational institution3 at grade 7 or higher. All students who met these criteria were eligible to sit the PISA test in 2022, regardless of the type of educational institution in which they were enrolled and whether they were enrolled in full- or part-time education. This also allows PISA to evaluate students shortly before they are faced with major life choices, such as whether to continue with education or enter the workforce.

Hence, PISA makes statements about the knowledge and skills of a group of individuals who were born within a comparable reference period, but who may have been exposed to different educational experiences inside and outside of school. These students may be distributed over different ranges of grades (both in terms of the specific grade levels and the spread in grade levels) in different countries and economies, or over different tracks or streams within their respective education systems. It is important to consider these differences when comparing PISA results across countries and economies. In addition, differences in performance observed when students are 15 may diminish or disappear entirely later in life.

If a country’s mean scores in mathematics, reading, science or financial literacy are significantly higher than those of another, it cannot automatically be inferred that schools or particular parts of the education system in the first country are more effective than those in the second. However, one can legitimately conclude that it is the cumulative impact of learning experiences in the first country, starting in early childhood and up to the age of 15, and including all experiences, whether they be at school, home or elsewhere, that have resulted in the better outcomes of the first country in the subjects that PISA assesses.4

How were students chosen?

The accuracy of the results from any survey depends on the quality of the information drawn from those surveyed as well as on the sampling procedures. Quality standards, procedures, instruments and verification mechanisms were developed for PISA that ensured that national samples yielded comparable data and that results could be compared across countries and economies with confidence. Experts from the PISA Consortium selected the samples for most participating countries and economies and monitored the sample-selection process closely in those countries that opted to select their own samples.

All samples in PISA 2022 were designed as two-stage stratified samples. The first stage sampled schools in which 15-year-old students may be enrolled. Schools were sampled systematically with selection probabilities proportional to the estimated size of their (eligible) 15-year-old population. At least 150 schools5 were selected in each country, although the requirements for national analyses often demanded a larger sample. Replacement schools for each sampled school were simultaneously identified, in case an originally sampled school chose not to participate in PISA.

The second stage of the selection process sampled students within sampled schools. Once schools were selected, a list of each sampled school’s 15-year-old students was prepared. From this list, 42 students were then selected with equal probability (all 15-year-old students were selected when less than 42 eligible students were enrolled). The target number of students in a school who were to be sampled could deviate from 42 when agreed by PISA’s sampling contractor but could not fall below 20 students.

Data-quality standards in PISA require minimum participation rates for schools and for students. These standards were established to minimise potential bias arising from non-response. Indeed, it was likely that any bias resulting from non-response would be negligible – typically smaller than the sampling error – in countries that met these standards.6

At least 85 % of the schools initially selected to take part in the PISA assessment were required to agree to conduct the test when accounting for the number of enrolled 15-year-olds. Where the initial response rate of schools was between 65% and 85%, however, an acceptable school-response rate could still be achieved using replacement schools.

Whenever a school is selected for PISA, two other schools – the most similar according to the statistical criteria used for sampling – are selected as replacement schools in case of non-response or other contingencies. However, statistical similarities notwithstanding, sampling bias is still possible if the replacement schools differ from sampled schools in ways that might not be considered for sampling. Therefore, countries and economies were encouraged to persuade as many of the schools in the original sample as possible to participate.

Schools that were included but where student participation rates of 25-50% were observed were not considered to be participating schools when determining participation rates; but data collected from these schools (from both the cognitive assessment and background questionnaires) were included in the database and contributed to the estimation of the various quantities derived from the assessment. Data from schools with a student participation rate of less than 25% were excluded from the database.

Among the countries an economies that participated in the PISA 2022 financial literacy assessment, five – the United States (51%), the Netherlands (66%), the Flemish community (Belgium) (72%), Brazil (81%), Canada (81%) – did not meet the standard of 85% weighted school participation rate; one did not meet the 65% threshold for schools initially selected for PISA. Even after replacement schools were included, three countries – the United States (63%), Canada (86%) and the Netherlands (90%) still failed to reach target participation rates;7 all other participating countries and economies reached the threshold for an acceptable participation rate after including replacement schools.

PISA 2022 also required that at least 80% of the students chosen in participating schools sat the PISA test. This threshold was calculated at the national level and did not have to be met in each participating school. Follow-up sessions were required in schools where too few students had participated in the planned assessment sessions. Student-participation rates were calculated over all originally selected schools and over all participating schools, including replacement schools. Students who participated in either the planned or follow-up sessions were counted in these rates; those who attended only the questionnaire session were included in the international database and contributed to the statistics presented in this publication if they provided at least a description of either parent’s occupation.

The standard of 80% student participation rate was not met by some countries and economies, including Canada (77%).

Table I.A2.6 in Volume I shows the response rate for students and schools, before and after including replacement schools.

What proportion of 15-year-olds does PISA represent?

All countries and economies attempted to maximise the coverage of 15-year-olds enrolled in education in their national samples, including students enrolled in special education institutions. As such, the technical standards used in PISA only allowed countries and economies to exclude up to 5% of the desired target population (i.e. 15-year-old students enrolled in educational institutions at grade 7 or higher) either by excluding schools or students within schools.

All countries and economies attempted to maximise the coverage of 15-year-olds enrolled in education in their national samples, including students enrolled in special education institutions. As such, the technical standards used in PISA only allowed countries and economies to exclude up to 5% of the desired target population (i.e. 15-year-old students enrolled in educational institutions at grade 7 or higher) either by excluding schools or students within schools.

Some countries and economies did not meet this standard in PISA 2022, including Denmark* (11.6%), the Netherlands* (8.4%), Norway (7.3%), the United States* (6.1%) and Canada* (5.8%). In 31 countries and economies, the overall exclusion rate was less than 2% (Table I.A2.1 in Volume I (OECD, 2023[1])). When language exclusions8 were accounted for (i.e. removed from the overall exclusion rate), some countries, including the United States no longer had exclusion rates greater than 5%. More details can be found in the PISA 2022 Technical Report (OECD, 2023[2])).

Exclusions that should remain within the above limits include:

• At the school level:

  • schools that were geographically inaccessible or where the implementation of the PISA assessment was not considered feasible

  • schools that provided teaching only for students in the categories defined under “within-school exclusions”, such as schools for students with special education needs.

The percentage of 15-year-olds enrolled in such schools had to be less than 2.5% of the nationally desired target population (0.5% maximum for the former group and 2% maximum for the latter group). The magnitude, nature and justification for school-level exclusions are documented in the PISA 2022 Technical Report (OECD, 2023[2]). In addition, due to differences in when schools re-opened and returned to full, in-person instruction after the COVID-19 pandemic, an additional code for student exclusions (Code 6) was used in PISA 2022 to account for those who were enrolled but received instruction virtually.

• At the student level:

  • students with an intellectual disability, i.e. a mental or emotional disability resulting in the student being so cognitively delayed that he/she could not perform in the PISA testing environment

  • students with a functional disability, i.e. a moderate to severe permanent physical disability resulting in the student being unable to perform in the PISA testing environment

  • students with limited assessment-language proficiency (these students were unable to read or speak any of the languages of assessment in the country at a sufficient level and were unable to overcome such a language barrier in the PISA testing environment; they were typically students who had received less than one year of instruction in the language of assessment)

  • students who were not attending in-person classes or going to school for tests/assessments during the PISA testing period but, rather, were receiving all of their instruction on line

  • other exclusions, a category defined by the PISA national centres in individual participating countries and approved by the PISA international consortium

  • students taught in a language of instruction for the major domain for which no materials were available.

Students could not be excluded solely because of low proficiency or common disciplinary problems. The percentage of 15-year-olds excluded within schools had to be less than 2.5% of the national desired target population.

Table I.A2.1 in Volume I describes the target population of the countries and economies that participated in PISA 2022. Further information on the target population and the implementation of PISA sampling standards can be found in the PISA 2022 Technical Report (OECD, 2023[2]).

A high level of coverage contributes to the comparability of the assessment results. For example, even assuming that the excluded students would have systematically scored worse than those who participated, and that this relationship is moderately strong, an exclusion rate of 5% would likely lead to an overestimation of national mean scores of less than 5 score points on the PISA scale (where the standard deviation is 100 score points).9

Given the significant disruption caused by COVID-19 global pandemic to education systems in general, and to the administration of the PISA 2022 Main Survey in particular, coverage is of particular concern in the 2022 cycle, as it is feasibly affected both by changes in student behaviour (e.g. not returning to school when those were reopened) and by operational factors of administering PISA itself (e.g. less participating students due to interference between PISA dates and a country/economy’s school reopening plan).

Table I.A2.2 in Volume I provides an across-cycle perspective on the estimated size of the 15-year-old cohort in a given country/economy, the estimated population size of 15-year-olds enrolled at school in grade 7 or above, the number of students that sat PISA 2022 weighted by how much they represent the population, and the coverage of the 15-year-old population (Coverage Index 3).

A decrease in the Coverage Index 3 between PISA 2018 and PISA 2022 was observed for 23 countries and economies. However, in only five of them this decrease was larger than 5%, including the Netherlands*. Nonetheless, these elevated drops in coverage are to be interpreted with due caution: sampling outcomes for the Netherlands struggled to meet PISA sampling standards.

Conversely, all other participating countries and economies either kept or increased their coverage of the population between PISA 2018 and PISA 2022. Small increases, up to 5%, were observed in 31 countries and economies, with others showing quite elevated increase in coverage in the 2022 cycle compared to PISA 2018.

The PISA Adjudication Group, comprising the Technical Advisory Group and the Sampling Referee, reviewed the PISA 2022 data. Overall, the review found that national implementations of PISA generally adhered to PISA’s technical standards despite the challenging circumstances that affected not only PISA operations but schooling more generally during the COVID-19 pandemic. Nevertheless, a number of deviations from the standards were noted and their consequences for data quality were reviewed in depth. The following overall patterns of deviations from sampling standards were identified:

• About one in five adjudicated entities had exclusion rates exceeding the limits set by the technical standards (Standard 1.7).

• Seven entities failed to meet the required school-response rates, with three of them failing to meet the stricter level of 65% before replacement (Standard 1.11). This is not inconsistent with earlier cycles of PISA, however.

• A significantly larger number of entities failed to meet the required student-response rates (Standard 1.12): ten entities did not meet this standard in PISA 2022, while only one entity did not meet the standard in PISA 2018.

Countries and economies that failed to meet the response-rate standards were requested to submit a non-response bias analysis (NRBA) report. These reports, evaluated by the PISA Adjudication Group, contained additional analyses using the national context and data sources to assess potential bias arising from school and student non-participation.

Details on the PISA Adjudication Group’s assessments of the deviations from PISA standards are described in the Reader’s Guide and Annex A4.

Definition of schools

In some countries, subunits within schools were sampled instead of schools, which may affect the estimate of the between-school variance. In countries like Austria, schools with more than one programme of study were split into the units delivering these programmes. In the Netherlands, locations were listed as sampling units. In the Flemish community (Belgium), each campus of a multi-campus school was sampled independently, whereas the larger administrative unit of a multi-campus school was sampled as a whole in the French community (Belgium).

Schools in the Basque Country (Spain) that were divided into sections by language of instruction were split into sections for sampling based on those languages. Some schools in the United Arab Emirates were sampled as a whole unit, while others were divided by curriculum and sometimes by gender. Some schools in Portugal were organised into clusters where all units in a cluster shared the same teachers and principal; each of these clusters constituted a single sampling unit.

Sampling for the financial literacy assessment

In all countries and economies, the default sampling design used for the PISA assessment was a two-stage stratified sample design. The first-stage sampling units consisted of individual schools having 15-year-old students, or the possibility of having such students at the time of assessment. The second-stage sampling units in countries and economies using the two-stage design were students within sampled schools.

While countries/economies that participated in the financial literacy assessment selected schools and students in the same manner as described above, these countries/economies were required to assess 1,650 additional students. This was typically achieved by increasing the number of students selected in the sampled schools.  

Financial literacy was administered only as a computer-based assessment (CBA). In CBA countries/economies, a sample of students was selected with equal probability from each list of eligible students within a school. For sampled schools that contained more eligible students than the target cluster size (TCS, the number of students to sample in a school), a sample of TCS students was selected. For schools with fewer than TCS eligible students, all students on the list were selected. The students selected for financial literacy were an additional sample of students above and beyond those needed for PISA. To accomplish this, the TCS was usually increased for countries/economies participating in the financial literacy assessment in PISA 2022. For example, a county/economy that would have sampled 42 students in each school generally increased its TCS to 53 to accommodate the financial literacy sample. In some instances, the country/economy opted to increase the school sample size to achieve the required number of students selected for financial literacy.

As in PISA 2018, the financial literacy assessment was administered to a separate sample of PISA eligible students who took, in addition to the financial literacy assessment, a combination of reading or mathematics items. The total testing time for each student was two hours (120 minutes). 

To increase the size of the financial literacy student sample, financial literacy scores were imputed for those students who were given forms involving only mathematics and reading (forms 1 to 12, see Figure 2.5 of the PISA 2022 Technical Report); these students were then included in the financial literacy sample. 

Table IV.A2.1 presents the number of students who comprised the financial literacy sample in each country/economy, and the number of 15-year-old students in each country/economy that the sample represented.

Table IV.A2.2 presents revised overall exclusion rates (analogue to the ones presented in Column 12 of Table I.A2.1 in Volume I) for the countries and economies that participated in the PISA 2022 financial literacy assessment. These revised overall exclusion rates take into account the fact that PISA Une Heure (UH) forms (i.e., a special one-hour test prepared for students with special needs) were not available for the financial literacy assessment, hence students answering UH forms were effectively excluded from the point of view of the financial literacy assessment. The table shows that the overall exclusion rates are above 5% in Belgium and Costa Rica, in addition to Canada*, Denmark*, the Netherlands*, Norway and the United States*. Estimates for all of these entities should therefore be interpreted with caution.

Table IV.A2.1. Sample size for the PISA financial literacy assessment

	Number of students who were part of the financial literacy assessment (unweighted)	Number of students who were part of the financial literacy assessment (weighted)
OECD
Austria	4 538	75 720
Flemish community of Belgium	2 735	70 626
Canadian provinces*	9 474	257 422
Costa Rica	3 279	53 068
Czechia	4 947	98 220
Denmark*	3 621	55 889
Italy	6 268	495 624
Hungary	3 690	88 268
Netherlands*	2 886	150 143
Norway	4 882	58 641
Poland	3 526	336 936
Portugal	4 075	97 702
Spain	3 789	457 980
United States*	3 206	3 504 432
Partners
Brazil	6 477	2 290 291
Bulgaria	3 614	53 249
Peru	4 092	493 760
Malaysia	4 161	387 945
Saudi Arabia	4 119	317 128
United Arab Emirates	14 604	60 848

Table IV.A2.2. Overall exclusion rates taking into account the percentage of students answering the PISA Une Heure forms

	Percentage of students answering the PISA Une Heure forms	Revised overall exclusion rates taking into account the percentage of students answering the PISA Une Heure forms
%	%
OECD
Austria	0.48	3.98
Belgium	3.43	5.77
Canada*	2.38	8.07
Costa Rica	7.34	7.40
Czechia	1.78	3.72
Denmark*	3.15	14.34
Hungary	0.00	4.68
Italy	0.00	3.07
Netherlands*	3.48	11.61
Norway	0.00	7.27
Poland	0.00	4.78
Portugal	0.00	4.02
Spain	0.00	4.02
United States*	4.62	10.48
Partners
Brazil	0.00	3.17
Bulgaria	0.00	2.70
Malaysia	0.00	1.46
Peru	0.00	3.29
Saudi Arabia	0.00	3.22
United Arab Emirates	0.00	2.57

References

[3] OECD (2023), PISA 2022 Assessment and Analytical Framework, PISA, OECD Publishing, Paris, https://doi.org/10.1787/dfe0bf9c-en.

[1] OECD (2023), PISA 2022 Results (Volume I): The State of Learning and Equity in Education, PISA, OECD Publishing, Paris, https://doi.org/10.1787/53f23881-en.

[2] OECD (2023), PISA 2022 Technical Report.

Notes

← 1.  To accommodate countries that requested grade-based results for the purpose of national analyses, PISA 2022 provided a sampling option to supplement the age-based sampling from the target population with an additional grade-based sample.

← 2.  More precisely, PISA assessed students who were at least 15 years and 3 complete months old and who were at most 16 years and 3 complete months old (i.e. younger than 16 years, 2 months and roughly 30 days old), with a tolerance of one month on each side of this age window. If the PISA assessment was conducted in April 2022, as was the case in many countries and economies, all students born in 2006 would have been eligible.

← 3.  Educational institutions are generally referred to as schools in this publication, although some educational institutions (in particular, some types of vocational education establishments) may not be referred to as schools in certain countries.

← 4.  Such a comparison is complicated by first-generation immigrant students, who received part of their education in a country other than the one in which they were assessed. Mean scores in any country or economy should be interpreted in the context of local student demographics. In addition, the PISA target population does not include residents of a country who attend school in another country. It does, however, include foreign nationals who attend school in the country of assessment.

← 5.  In education systems inherently too small (due to demographics for instance), all schools and all eligible students were included in the sample. In PISA 2022, all eligible schools were selected in North Macedonia and Qatar. All students in all schools were selected in Brunei Darussalam, Iceland, Macao (China) and Malta.

← 6.  Non-response and other standards enforced to achieve consistent, precise, generalisable, and timely data collection in PISA 2022 are available on its Technical Standards (OECD, 2023[2]).

← 7.  The threshold for an acceptable participation rate after replacement varies between 85 % and 100 %, depending on the participation rate before replacement.

← 8.  These exclusions refer only to those students with limited proficiency in the language of instruction/assessment. Exclusions related to the unavailability of test material in the language of instruction are not considered in this analysis.

← 9.  If the correlation between the propensity of exclusions and student performance were 0.3, then resulting mean scores would likely have been overestimated by 1 score point if the exclusion rate were 1 %; by 3 score points if the exclusion rate were 5 %; and by 6 score points if the exclusion rate were 10 %. If the correlation between the propensity of exclusions and student performance were 0.5, then resulting mean scores would likely have been overestimated by 1 score point if the exclusion rate were 1 %; by 5 score points if the exclusion rate were 5 %; and by 10 score points if the exclusion rate were 10 %. For this calculation, a model was used that assumed a bivariate normal distribution for performance and the propensity to participate.