Find out what is PISA, how it is done and what are its goals.
Created by the Organisation for Economic Co-operation and Development (OECD), PISA tests the skills and knowledge of 15-year-old students in mathematics, reading and science. Eighty-one countries and economies took part in the 2022 assessment, which focused on mathematics, and the data were released by the OECD on 5th December 2023.
Additional results on students’ financial literacy, creative thinking and readiness for life-long learning will be released in 2024.
PISA focuses on the assessment of student performance in mathematics, reading and science to measure the extent to which students can use what they learned in and out of schools for their full participation in societies. PISA also collects valuable information on student attitudes and motivations, and assesses skills such as collaborative problem solving, global competence and creative thinking. In order to better understand how students are prepared to solve complex problems, think critically and communicate effectively in the 21st century, PISA is constantly exploring new assessment domains, such as "learning in a digital world" or "critical media literacy".
PISA draws on content that can be found in curricula across the world and looks at students’ ability to apply knowledge and skills to examine, interpret and solve problems. PISA does not prescribe or promote any one curriculum nor is it constrained by the need to find a common denominator.
The 8th round of the PISA assessment was originally planned to take place in 2021. However, due to the disruption caused by COVID-19, including widespread lockdowns and school closures, the PISA Governing Board decided to postpone the assessment by a year.
Content: The PISA 2022 survey focused on mathematics, with reading, science and creative thinking as minor areas of assessment. PISA 2022 also included an assessment of young people’s financial literacy, which was optional for countries and economies.
Students: Some 690 000 students took the assessment in 2022, representing about 29 million 15-year-olds in the schools of the 81 participating countries and economies.
Tests: Computer-based tests were used in most countries, with assessments lasting a total of two hours. In mathematics and reading, a multi-stage adaptive approach was applied in computer-based tests whereby students were assigned a block of test items based on their performance in preceding blocks. Students also answered a background questionnaire, which took about 35 minutes to complete. Test items were a mixture of multiple-choice questions and questions requiring students to construct their own responses.
The items were organised into groups based on a passage of text describing a real-life situation. More than 15 hours of test items for reading, mathematics, science and creative thinking were covered, with different students taking different combinations of test items.
Questionnaires: The questionnaire sought information about the students themselves, their attitudes, dispositions and beliefs, their homes, and their school and learning experiences. School principals completed a questionnaire about school management, organisation and the learning environment. Questionnaires for students and principals also included questions asking about learning experiences during school closures due to the COVID-19 pandemic.
Some countries/economies also distributed additional questionnaires to elicit more information. These included: in 19 countries/economies, a questionnaire asking teachers to report about themselves and their teaching practices; and in 17 countries/economies, a questionnaire asking parents to provide information about their perceptions of and involvement in their child’s school and learning. Countries/economies could also choose to distribute two other optional questionnaires for students: 52 countries/economies distributed a questionnaire about students’ familiarity with ICT devices/resources; and 15 countries/economies distributed a questionnaire about students’ well-being.
Further information can be found in the section "What is PISA" in PISA 2022 Results (Volume I).
Four countries – Australia, Canada, Denmark and Switzerland – have conducted longitudinal studies that followed the cohort of students who sat the first PISA assessments in the early 2000s through their transition into adulthood. Across all of these countries, students who performed better in PISA at age 15 were more likely to attain higher levels of education by the age of 25. They were also less likely to be out of the labour market entirely, as measured by the percentage of students who were not in education, employment or training (NEET).
Of course, there is a link between the socio-economic status of a student’s household – including material living conditions and parents’ educational attainment – and the student’s performance in PISA. However, even after accounting for these factors, students who perform better in PISA at age 15 have stronger education and employment outcomes at the age of 25. In addition, what students say about themselves in the PISA questionnaires also appears related to young people’s future life prospects. For further details, please see the November 2019 PISA in Focus and the report Equity in Education.
In addition, the OECD has conducted analyses combining data from PISA (2000, 2003 and 2006 assessments) and the Survey of Adult Skills, a product of the OECD Programme for the International Assessment of Adult Competencies (PIAAC) (2012 and 2015 assessments) to examine the growth in numeracy skills between the ages of 15 and young adulthood. Such analyses suggest that students who perform better in mathematics in PISA at age 15 also show greater gains in numeracy skills after leaving compulsory education than their lower-performing peers. Further information can be found in Box I.2.2. of PISA 2022 Results (Volume I).
An updated framework for the mathematics assessment was developed to reflect large-scale social changes since PISA 2012, the last edition where mathematics was the main focus. Various social changes such as digitalisation and new technologies, the use of data in personal decisions, and globalisation have redefined what it means to be mathematically competent and well-equipped to fully participate in the 21st century. Mathematics competency is not just about being able to reproduce routine procedures. It is more about the use of mathematical reasoning; to think mathematically to solve complex real-life problems in a variety of modern contexts.
Adaptive testing was implemented in the mathematics assessment for PISA 2022. It was implemented in the reading assessment for PISA 2018. It will be implemented in the science assessment for PISA 2025. Students who performed well on earlier parts of the mathematics assessment were more likely to receive more difficult questions later in the assessment. Conversely, students who performed poorly in earlier parts of the assessment were more likely to receive easier questions later in the assessment.
Creative thinking was tested for the first time in PISA 2022; results for this subject will be published in mid-2024.
Several countries/economies participated in PISA for the first time in 2022: Cambodia, El Salvador, Guatemala, Jamaica, Mongolia, the Palestinian Authority, Paraguay and Uzbekistan. Cambodia, Guatemala and Paraguay had previously participated in PISA for Development.
PISA assesses students between the ages of 15 years and 3 months and 16 years and 2 months, and who are enrolled in an educational institution at grade 7 or higher. The lists from which schools and students are selected for the PISA assessment cover all students who meet these criteria, regardless of the type of educational institution in which they are enrolled and whether they are enrolled in full-time or part-time education.
However, some of the schools and students that are listed may be excluded from PISA samples. PISA standards specify that such exclusions should not represent more than 5% of the target population. Within that limit, students can be excluded for various reasons, including the remoteness and inaccessibility of their school, intellectual or physical disability, a lack of proficiency in the test language (e.g. if the student has received instruction in the test language for less than one year), or a lack of test material in the language of instruction. For countries where the overall level of exclusions exceeds the limit significantly, further information is requested to support the case that no bias in comparisons over time and with other countries would result from exclusions.
Except in very small countries and economies, not all remaining students sit the PISA assessment; rather, a sample of schools, and of students within schools, is drawn, and the students selected are assigned sampling weights so as to represent the entire PISA-eligible cohort.
The PISA sample represents all students between the age of 15 years and 3 months and 16 years and 2 months, and who are enrolled in an educational institution at grade 7 or higher, minus those that are excluded. PISA reports the proportion of the total 15-year-old population that is represented by the PISA sample (this is known as Coverage Index 3). This proportion ranges from close to 100% in countries with limited exclusions and universal enrolment in educational institutions for 15-16-year-olds, to less than 50% in some low- and middle-income countries. In most participating countries and economies, more than 80% of 15-year-olds are covered.
Coverage of the 15-year-old cohort has increased in many countries since earlier PISA assessments. Between 2012 and 2022, Indonesia added more than 1.1 million students to the total population of 15-year-olds that could take the PISA test (the total population of 15-year-olds increased only by about 300 000 over the same period). Cambodia, Colombia, Costa Rica, Morocco, Paraguay and Romania also increased the number of 15-year-olds eligible to sit the PISA test despite stable or, in some cases, shrinking populations of 15-year-olds.
Further information about the share and number of 15-year-olds covered by PISA can be found in Chapters 4, 6 and in Annex A2 of PISA 2022 Results (Volume I).
The age of 15 was chosen as it is when young people in most OECD countries are nearing the end of compulsory education. The selection of schools and students is as inclusive as possible, so that the sample of students comes from a broad range of backgrounds and abilities.
PISA benefits from its worldwide scope and its regularity. Over 100 countries and economies have taken part in PISA so far and the surveys, which are usually carried out every three years, allow participating countries and economies to track their progress in meeting key learning goals. PISA is the only international education survey to measure the knowledge and skills of 15-year-olds.
PISA is also unique in the way it looks at:
Through its website and publications, the OECD makes available to the public and specialists all the key information on the methods and processes associated with PISA. The following documents are available on the Publications page and the Data page: the assessment frameworks that explain what is to be assessed, why and how; examples of the questionnaire items; the data from the assessment; the survey-implementation tools for administration and language-quality assurance; and a comprehensive technical report for every cycle, which includes detailed technical information on every aspect of design, development, implementation and analysis.
For each assessment cycle a selection of PISA test materials is also made available to the general public. In order to allow countries to follow their performance over time, many questions are used in more than one PISA survey. These questions cannot be made public as long as they are in use.
In addition to OECD staff and contractors, hundreds of experts, academics and researchers in PISA-participating countries and economies are involved in PISA's development, analysis and reporting; details of these participants are provided in the PISA reports.
PISA is developed and implemented under the responsibility of education ministries through PISA’s decision-making body, the PISA Governing Board. The Board has representatives from all member countries plus partner countries with Associate status, currently Brazil and Thailand. Countries appoint representatives to the Board who are knowledgeable about large-scale student assessments and their interface with educational policy and practice. Representatives comprise a mix of government officials and staff of research and academic institutions.
The Board determines the policy priorities for PISA and oversees adherence to these priorities during implementation. This includes setting priorities and standards for data development, analysis and reporting, and determining the scope of work that will then form the basis for the implementation of PISA.
To ensure the technical robustness of PISA, a Technical Advisory Group (TAG) is appointed by the PISA Governing Board and directly managed by the OECD, comprising independent, world-renowned experts in the fields that underpin the PISA methodology, such as sampling, survey design, scaling and analysis. The TAG is regularly called upon to discuss and provide recommendations on the PISA design and methodologies and adjudicate the data of individual countries and economies to ensure that what is published from PISA is robust and internationally comparable.
The PISA Governing Board: The PISA Governing Board is composed of representatives of OECD members and PISA Associates.* Countries and economies that participate in PISA but do not have Associate status are welcome to participate in PGB meetings as observers. Representatives are appointed by their education ministries, and the PGB Chair is chosen by the Board itself. Guided by the OECD’s education objectives, the Board determines the policy priorities for PISA and makes sure that these are respected during the implementation of each PISA survey.
*Associates are countries or economies that are not OECD members but have membership rights and obligations with regard to specific OECD bodies and programmes.
The OECD Secretariat: The OECD Secretariat is responsible for the day-to-day management of PISA. This means that the PISA team monitors the survey’s implementation, manages administrative matters for the PISA Governing Board, builds consensus among countries and serves as an intermediary between the PISA Governing Board and the PISA Consortium.
PISA National Project Managers: Working with the OECD Secretariat, the PISA Governing Board and the international contractors, the PISA National Project Managers oversee the implementation of PISA in each participating country/economy. The PISA National Project Managers are appointed by their governments. A list of the 2015 National Project Managers is provided here; the 2018 National Project Managers is provided here; the 2022 National Project Managers is provided here and the 2025 National Project Managers is provided here.
International contractors (the "PISA Consortium"): For each PISA survey, international contractors (usually made up of testing and assessment agencies) are responsible for the design and implementation of the surveys. The contractors are chosen by the PISA Governing Board through an international call for tender. The contractors are typically referred to as the PISA Consortium.
Education authorities: PISA would not be possible without the support and guidance of the education ministries in the participating countries and economies.
PISA Technical Advisory Group: The Technical Advisory Group is regularly called upon to discuss and provide recommendations on the PISA designs and methodologies and adjudicate the data of individual countries/economies to ensure that what is published from PISA is robust and internationally comparable. The group is appointed by the PISA Governing Board and directly managed by the OECD, comprising independent, world-renowned experts in the fields that underpin the PISA methodology, such as sampling, survey design, scaling and analysis.
PISA Subject Matter Expert Groups: PISA has Subject Matter Expert Groups for its three key areas of testing – mathematics, reading and science – as well as for other subjects when appropriate (for example, global competence in PISA 2018, creative thinking in PISA 2022, and learning in digital world in PISA 2025). These groups are made up of world experts in each area. They design the theoretical framework for each PISA survey.
PISA Questionnaire Expert Group: The Questionnaire Expert Group provides leadership and guidance in the construction of the PISA context questionnaires. The members of the Questionnaire Expert Group are selected by the PISA Governing Board.
PISA is financed exclusively through direct contributions from the participating countries and economies’ government authorities, typically education ministries.
Although PISA is a triennial international survey, it is easiest to consider the costs for participating countries on an annual basis. The costs comprise international costs (participating countries contribute to the cost of the International Contractors and OECD staff and related expenses) and national costs associated with the implementation of the PISA survey in the respective country (establishment of a national centre, cost of sampling, translation etc.)
Participation
Cost of participation
a) International costs
Members and Associates: The cost for participation is determined by the Secretariat of PISA. The cost is composed of a floor contribution, determined by the PISA Governing Board (PGB); the remainder of costs are assigned to Members and Associates on the basis of the OECD principles and rules for determining the scales of contributions by Members, using as input statistics, national income data and exchange rates used in the calculation of scales for the previous year. As a result, the cost of participation varies between countries.
Partners: International PISA costs for partner countries/economies are determined by the PISA Secretariat at the beginning of each PISA cycle, following signature of the contracts with international contractors. The cost is 171 000 EUR spread over a three-year period (costs for participation in PISA 2025).
b) National costs
National costs for the implementation of PISA vary by country, according to factors such as sample size, the number of languages to be tested in the assessment, national income statistics and living standards. For example, a small country might spend approximately EUR 100 000 per annum while a medium-sized country EUR 350 000 per annum; a large country could spend up to two or three times the latter amount.
PISA uses multiple-choice testing as the primary feature of its assessments because it is reliable, efficient, and supports robust and scientific analyses. Multiple-choice questions in PISA have a variety of formats, including highlighting of a word within a text, connecting pieces of information and making multiple selections from drop-down menus. In addition, typically up to one-third of questions in a PISA assessment are open-ended.
A few PISA test items are released after every assessment, while most test items are kept confidential in order to measure trends over time. PISA released items are publicly available on the PISA website.
Students also answer a background questionnaire, providing information about themselves, their attitudes to learning and their homes. In addition, school principals are given a questionnaire about their schools. Countries and economies can also choose to distribute several optional PISA questionnaires: the computer familiarity questionnaire, the well-being questionnaire, the parent questionnaire and the teacher questionnaire. Download PISA 2022 questionnaires are available here. In addition, many countries and economies choose to gather further information through national questionnaires. The information collected helps countries and economies explore connections between how students perform in PISA and factors such as immigrant background, gender and students’ socio-economic status, as well as students’ attitudes towards school and learning.
Participating PISA countries and economies are invited to submit questions that are then added to items developed by the OECD’s experts and contractors. The questions are reviewed by the international contractors and by participating countries and economies, and are carefully checked for cultural bias. Only those questions that pass these checks are used in PISA. Further, before the main test is conducted there is a trial test run in all participating countries and economies. If any test questions prove to be too easy or too difficult in certain countries and economies, for reasons not related to the overall level of proficiency of students (as demonstrated on the remaining questions), they are dropped from the main test in all countries and economies.
The PISA test is designed to provide an assessment of performance at the system (or country) level. It is not designed to produce scores for individual students, so it is not necessary for each student to receive exactly the same set of test items. Thus, PISA adopts an efficient design in which the full set of test material, covering all aspects of the framework, is distributed over a large number of test forms. This procedure enables the OECD to obtain a much greater coverage of the content than if all students had completed the same version of the test.
In science and creative thinking, the assignment of students to test forms was entirely random. In mathematics and reading, which were assessed through adaptive testing, the initial assignment was random, but the precise form was only determined during the assessment, taking into account students’ performance in the early portions of the assessment. As a result of this multi-stage adaptive design, high-performing students were more likely to be directed to more difficult questions (and low-achieving students to easier questions). This provided greater accuracy in measurement of both test scores and the relationship between test scores and other factors, such as socio-economic status.
In PISA 2022, most students took an hour-long sequence of mathematics questions combined with an hour of either reading, science or creative thinking questions. A small proportion of students took a combination of reading, science or creative thinking questions.
Computers and computer technology are part of our everyday lives and it is appropriate and inevitable that PISA has progressed to a computer-based delivery mode. Over the past decades, digital technologies have fundamentally transformed the ways we read and manage information. Digital technologies are also transforming teaching and learning, and how schools assess students. To reflect how students and societies now commonly access, use and communicate information, starting with the 2015 assessment, the PISA test was delivered mainly on computer. Existing tasks were adapted for delivery on screen; new tasks (initially only in science, then, for PISA 2018 also in reading, and for 2022 in mathematics) were developed that made use of the affordances of computer-based testing and that reflected the new situations in which students apply their mathematics, reading and science knowledge and skills in real life. For the comparability of computer and paper-based tests, see a FAQ under “Results”.
PISA applies strict technical standards including for the sampling of schools and students within schools. The sampling procedures are quality assured and the achieved samples and corresponding response rates are subject to an adjudication process that verifies that they comply with the standards set. If any country's response rate falls below the specified threshold, it is reported. Further information about response rates in PISA can be found on the OECD's PISA website, and specific information on participation rates for individual countries can be found in Annex A2 of PISA 2022 Results (Volume I). Regarding weighting adjustments in order to mitigate the impact of school non-response and student non-response on their under-representation in the data, see the PISA 2022 Technical Report.
The exceptional circumstances throughout PISA 2022, including lockdowns and school closures in many places, led to occasional difficulties in collecting some data. While the vast majority of countries and economies met PISA’s technical standards on sampling, a small number did not. In prior PISA rounds, countries and economies that failed to comply with the standards and the deviations are judged by the PISA Adjuration Group to be consequential for comparability could face exclusion from full reporting. However, given the unprecedented situation caused by the pandemic, PISA 2022 results include data from all participating education systems, including those where there were issues such as low response rates. See more in the Reader’s Guide in PISA 2022 (Volume I).
It is important to note that the limitations and implications were assessed by the PISA Adjudication Group in June 2023. There may be a need for subsequent adjustments as new evidence on the quality and comparability of the data emerges. PISA will return to the standard way of reporting for the 2025 assessment.
The following lists cases in which the OECD, on the basis of technical advice from the PISA Consortium, removed or annotated national data in the report because of technical anomalies or because the data did not meet the OECD technical standards for PISA. Such cases are listed below for earlier assessments.
For PISA 2022, see Reader’s Guide of PISA 2022 (Volume I).
Two first volumes of the PISA 2022 Results are available on line (Volumes I and II). Volume III on creative thinking, Volume IV on financial literacy and Volume V on students’ readiness for life-long learning will be available in 2024. Results from previous cycles are available on the Publication page. All data from the PISA surveys can be found on the Data page.
PISA scores can be located along specific scales developed for each subject area, designed to show the general competencies tested by PISA. These scales are divided into levels that represent groups of PISA test questions, beginning at Level 1 with questions that require only the most basic skills to complete and increasing in difficulty with each level. (More specifically, the mathematics scale starts at Level 1c, followed by Levels 1b and 1a; the lowest level of the science scale is Level 1b, and is followed by Level 1a; and the lowest level of the reading scale is Level 1c, followed by Levels 1b and 1a.)
Once a student’s responses to the test have been scored, his or her overall score in mathematics, reading and science can be located on the appropriate scale. For example, a student who lacks the skills needed to correctly complete the easiest questions on a PISA test would be classified as below Level 1b or 1c, as per the domain, while a student whose answers show mastery of these skills would be classified at a higher level.
In each test subject, there is theoretically no minimum or maximum score in PISA; rather, the results are scaled to fit approximately normal distributions, with means for OECD countries around 500 score points and standard deviations around 100 score points. About two-thirds of students across OECD countries score between 400 and 600 points. Less than 2% of students, on average across OECD countries, reach scores above 700 points, and at most a handful of students in the PISA sample for any country reach scores above 800 points.
PISA ranks participating countries and economies according to their performance in mathematics, reading and science. PISA does not give a collective score for all subjects combined; rather it gives a score for each subject area, and the mean score in each subject can be used to rank countries. However, even within a single subject, it is not possible to assign an exact rank to each country or economy based on its mean score. This is because PISA tests only a sample of students from each country or economy, and estimates, based on their results, the distribution of performance in the whole population of 15-year-old students. The statistical uncertainty associated with these estimates means that it is only possible to report the range of positions (upper rank and lower rank) within which a country or economy can be placed. For example, in PISA 2003 Finland and Korea were widely reported as ranking 1st and 2nd in PISA, when in fact we can only say that, among OECD countries, Finland’s rank was between 1st and 3rd and Korea’s was between 1st and 4th.
Computer-based administration (CBA) was the primary mode of delivery for PISA 2022. In PISA 2022, 77 out of 81 countries and economies took the CBA version of the PISA test. The computer-based delivery mode allows PISA to measure new and expanded aspects of the domain constructs. In mathematics, new material for PISA 2022 included items developed to assess mathematical reasoning as a separate process classification, and items that leveraged the use of the digital environment (e.g. spreadsheets, simulators, data generators, drag-and-drop, etc.). A mixed-design that included computer-based multistage adaptive testing was also adopted for the mathematics literacy domain to further improve measurement accuracy and efficiency, especially at the extremes of the proficiency scale (on adaptive testing, see Annex A9 of Volume I of PISA 2022 Results.
Four countries used a paper-based version of the assessment (PBA) in PISA 2022. Cambodia, Guatemala and Paraguay administered a “new” PBA instrument that was first used in the PISA for Development (PISA-D) assessment. This “new” paper-based instrument contained a substantial amount of material that was first used in the PISA 2015 computer-based tests, or taken from other assessments, including the “Literacy Assessment and Monitoring Programme” (LAMP), the OECD Survey of Adult Skills (PIAAC), and PISA for Schools. Viet Nam used the same paper-based materials as in the 2015 and 2018 cycles (based on items that were first used in PISA 2012 or earlier).
Student performance is comparable between the computer-based tests and the pen-and-paper tests used in PISA 2022. At the same time, a large number of items used in the PISA 2022 computer-based tests of mathematics, reading and, to a lesser extent, science, were not delivered on paper. Therefore, caution is advised when drawing conclusions about the meaning of scale scores from paper-based tests, when the evidence that supports these conclusions is based on the full set of items. For example, the proficiency of students who sat the PISA 2022 paper-based test of mathematics should be described in terms of the PISA 2012 proficiency levels, not the PISA 2022 proficiency levels. This means, for example, that even though PISA 2022 developed a description of the skills of students who scored below Level 1b in mathematics, it remains unclear whether students who scored within the range of Level 1c on the paper-based tests have acquired these basic mathematics skills. More information on the comparability of computer- and paper-based tests can be found in Annex A5 of PISA 2022 Results (Volume I).
PISA scores are reported on the same scale over time. The scale was set in the first year in which each subject was the major focus of the assessment (2000 for reading, 2003 for mathematics and 2006 for science).
To determine whether performance improved, remained stable, or declined, PISA uses both year-to-year comparisons (e.g. between 2018 and 2022, or between 2012 and 2022) and average trends over three or more assessment years (i.e. over a period of 6-7 years or longer).
Year-to-year comparisons are most robust when comparing assessments that share a large number of items and other design features. For example, comparisons between 2018 and 2022 for all subjects, or (to a lesser extent) between 2022 and the previous year in which the domain of interest was a major domain (2012 for mathematics, 2018 for reading and 2015 for science). In some cases, however, year-to-year comparisons can be confounded by particular circumstances affecting the test administration in one of the two years only.
Average trends that combine data from three or more measurements can reduce the uncertainty involved in trend comparisons and limit the possible confounding effect of unique circumstances that may have affected the test administration in a single year. In PISA reports, average trends are the preferred measure of change in mean performance over time spans of ten years or more. Average trends also allow for meaningful comparisons of trends between countries that cannot compare data across all assessment years, although differences in participation must be considered when interpreting these comparisons. For more information about measuring trends and the sources of uncertainty in these measurements, see Annex A7 of PISA 2022 Report (Volume I).
Real change, in education, is cumulative and often slow; with good measurement, one should expect gradual improvement or decline. In this situation, year-to-year comparisons may not be significant, taken individually, but the accumulation of short-term changes that point in the same direction represents a significant trend. Fitting a regression line to multiple measurements allows for the extraction of a signal from data that are, at times, noisy, and to detect if there is an underlying pattern of improvement or deterioration.
Changes in test design are introduced only gradually in PISA, and, where possible, include variants which make it possible to isolate a sub-set of students or of test questions that are not affected by the change (or are affected in different directions). For example, the 2022 test design for mathematics was adaptive, for the first time, and included a large number of newly developed questions. Still, it assigned 25% of students to a “non-adaptive” version, so that it is possible to study the effect of adaptivity on student behaviour and test performance. New mathematics questions were administered alongside “trend” questions, which remained identical to those used in previous years, and could be used to monitor the robustness of the reported trends to changing test contents. For all students, the items included in the first section of the test were determined at random, as in previous years, meaning that performance changes are unlikely to be driven by changes in test design if they can be already observed when considering only the first questions seen by each student. And, as usual, test domains and test positions are rotated across students, at random, so that it is possible to test for potential order effects (for example, to test whether mathematics performance is lower after a test of creative thinking compared to when mathematics is administered first, or after a test of science).
Technical annexes in Volume I (and, in particular, Annex A8) include several analyses to confirm that the conclusions that are drawn from test scores are valid, and robust to the evolving methods and design used in PISA.
Confidence in the robustness of PISA is based on the rigour that is applied to all technical aspects of the survey design, implementation and analysis, not just on the nature of the statistical model, which has developed over time and will continue to do so. Specifically regarding test development, the robustness of the assessment lies in the rigour of the procedures used in item development, field trials, analysis, review and selection.
The task for the experts developing the assessment is to ensure that all these aspects are taken into account, and to use their expert judgement to select a set of test items such that there is a sufficient balance across all these aspects. In PISA this is done by assessment specialists who work with advisory groups made up of international experts. Participating countries and economies also play a key role in this item-selection process.
The details of the test-design and test-development processes are available in the PISA 2022 Technical Report. Technical reports for earlier assessments are also available on the PISA website. For example, the PISA 2018 Technical Report and PISA 2015 Technical Report.
The value of cross-country comparisons is at the heart of large-scale international surveys such as PISA. PISA follows several standards and procedures concerned with ensuring fair and valid comparisons of PISA results across countries. These include the consistent implementation of PISA and the use of representative samples in each participating country and economy. Much effort also goes into ensuring that the questions used in PISA maintain their measurement properties in each of the many language versions (in PISA 2022, over 100 different language versions were used). The steps to ensure that the resulting measures are equivalent include:
PISA is the world's foremost international, high-quality and high-impact education assessment programme, so it is vitally important that the PISA data are accurate and authentic. Therefore the OECD applies strict conditions at all levels to make sure student data accurately reflect students’ ability and performance, and do not involve any form of cheating. This assurance starts with the Agreement for Participation between the OECD and each country. Article 4 of the agreement requires countries to comply with the comprehensive Technical Standards for PISA, including the secure management of test materials and secure administration of the assessment. These requirements are then reinforced through the PISA Operations Manual, the School Co-ordinators Manual and the Test Administrators Manual. These manuals have explicit instructions for the secure receipt, handling and storage of all test-related materials, and for the secure administration of the test itself. No one other than approved project staff has access to secure PISA data, and embargoed material and formal confidentiality arrangements are in place for all approved project staff.
Adherence to the standards is monitored throughout all project-implementation phases, and all deviations (e.g. deviations from the agreed test-administration protocol), both minor and major, are recorded for further review. After students sit the assessment, each national dataset is reviewed and any inconsistencies flagged for further analysis. Such inconsistencies include, for example, differences between one country and others in how items functioned, and differences in a country between the field trial and the main study in how items functioned. Data adjudication is the quality-control process through which each national dataset and quality-monitoring report are reviewed, and through which a judgement about the appropriateness of the data for the main reporting goals is formed.
Data for adjudicated regions are also reviewed, although not all sub-national jurisdictions choose to undergo the data-adjudication process.
The process and outcomes of data adjudication for PISA 2022 are described in the PISA 2022 Technical Report.
Over recent cycles, the national ministry has been piloting PISA in several provinces/municipalities in preparation for fuller participation across China. Shanghai has participated since PISA 2009. In PISA 2015, the municipalities of Beijing and Shanghai and the provinces of Jiangsu and Guangdong participated. In PISA 2018, the municipalities of Beijing and Shanghai and the provinces of Jiangsu and Zhejiang participated.
Chinese provinces/municipalities (Beijing, Shanghai, Jiangsu and Zhejiang) China are participants in PISA 2022 but were unable to collect data because schools were closed during the intended data collection period.
In a 2012 OECD survey of PISA-participating countries and economies, a large majority of respondents reported that the policies of high-performing countries or improving systems had been influential in their own policy-making processes. The same number of countries and economies also indicated that PISA had influenced the development of new elements of a national or federal assessment strategy. In relation to curriculum setting and standards, many countries and economies cited the influence of the PISA frameworks on: comparisons of national curricula with PISA frameworks and assessments; formation of common standards nationally; impact on their reading frameworks; the incorporation of PISA-like competencies in their curricula; and for setting national proficiency standards.
The OECD strives to identify what policies and practices appear to be “working” in countries and economies that are recording high performance or show evidence of significant improvement over time on PISA. It then reports those findings and supports countries and economies that wish to investigate and explore the extent to which they would benefit from similar programmes. The OECD is aware of the different circumstances in different countries and economies (with 81 participating in PISA 2022). There is no ‘one size fits all” education model for countries and economies. It is not possible or appropriate to “cut and paste” one country's education system into another country or economy.
The initial PISA 2022 results will be released with two volumes on 5 December 2023. The first volume, The State of Learning and Equity in Education, assesses how student performance in mathematics, reading and science evolved before and after the pandemic. Did student performance suddenly shift between 2018 and 2022 or stay the same? Are there long-term trends which have continued or sharply changed direction? The PISA team are examining these issues and whether specific student groups, such as socio-economically advantaged and disadvantaged students, boys and girls, and immigrant and non-immigrant students have exhibited varying performance trends.
The second volume of PISA 2022, Learning During – and from – Disruption, goes beyond academic ability and examines student well-being. It identifies “resilient education systems” that maintained or promoted student learning, equity and well-being amid the disruption caused by COVID-19. Education systems are scrutinised to see which maintained performance scores for all students, regardless of their socio-economic profiles, while keeping or boosting their sense of belonging at school. For more information, read PISA in Focus.