PISA 2022 Assessment and Analytical Framework

1. What is PISA?

Abstract

This chapter introduces the PISA 2022 Assessment and Analytical Framework. It describes what makes PISA unique, the key features of the PISA 2022 test, an overview of what is assessed in each domain, the evolution of reporting student performance and the context questionnaires. Finally, it presents how PISA is the result of a collaborative effort.

The OECD Programme for International Student Assessment (PISA), now in its eighth cycle, seeks to determine what is important for citizens to know and be able to do. PISA assesses the extent to which 15‑year‑old students near the end of their compulsory education have acquired the knowledge and skills that are essential for full participation in modern societies.

The assessment focuses on the core school subjects of reading, mathematics, and science. Students’ proficiency in an innovative domain is also assessed; in 2022, this domain was creative thinking. The assessment does not just ascertain whether students can reproduce knowledge; it also examines how well students can extrapolate from what they have learned and can apply that knowledge in unfamiliar settings, both in and outside of school. This approach reflects the fact that modern economies reward individuals not for what they know, but for what they can do with what they know.

PISA is an ongoing programme that monitors trends in the knowledge and skills that students around the world, and in demographic subgroups within each country, have acquired. In each round of PISA, one of the core domains, denominated major domain, is tested in detail, taking up roughly one-half of the total testing time. The major domain in 2022 was mathematics as it was in 2003 and 2012. Reading was the major domain in 2000, 2009 and 2018, and science was the major domain in 2006 and 2015.

With this alternating schedule of major domains, a thorough analysis of achievement in each of the three core areas is presented every nine years; an analysis of trends is offered every three years. However, due to the decision to postpone the assessment from 2021 to 2022 to reflect post-COVID difficulties, the analysis of achievement and trends from the results of this cycle will be offered one year later than in previous cycles.

Through questionnaires distributed to students and school principals, and optional questionnaires distributed to parents and teachers, PISA also gathers information about students’ home background, their approaches to learning and their learning environments. Combined with the information gathered through the various questionnaires, the PISA assessment provides three main types of outcomes:

• basic indicators that provide a profile of the knowledge and skills of students;

• indicators derived from the questionnaires that show how such skills relate to various demographic, social, economic and educational variables;

• indicators on trends that show changes in outcomes and their distributions, and in relationships between student-, school- and system-level background variables and outcomes.

Policymakers around the world use PISA findings to gauge the knowledge and skills of the students in their own country/economy compared with those in other participating countries/economies, establish benchmarks for improvements in the education provided and/or in learning outcomes, and understand the relative strengths and weaknesses of their own education systems.

This publication presents the theory underlying the PISA 2022 assessment – the eighth since the programme’s inception. It includes the frameworks for assessing the major domain, mathematics (Chapter 2); students’ financial literacy (Chapter 3); and the innovative domain, creative thinking (Chapter 4). These chapters outline the knowledge content that students need to acquire in each domain, the processes that students need to be able to perform, and the contexts in which this knowledge and these skills are applied. They also discuss how each domain is assessed. The publication concludes with the frameworks for the various questionnaires distributed to students, school principals, parents and teachers (Chapter 5), and the framework for the information and communications technology (ICT) familiarity questionnaire distributed to students (Chapter 6). The frameworks for the reading and science assessments are not included in this publication as they received their last major updates when they were the major domain of assessment (2018 for reading, 2015 for science), links to access their full versions are included as Annexes (Annex A and Annex B, respectively).

Box 1.1. Key features of PISA 2022

The content

PISA not only assesses whether students can reproduce knowledge, but also whether they can extrapolate from what they have learned and apply their knowledge in new situations. It emphasises the mastery of processes, the understanding of concepts, and the ability to function in various types of situations.

The PISA 2022 survey focused on mathematics, with reading and science as minor domains of assessment. For the first time, creative thinking was assessed as an innovative domain. PISA 2022 also included an assessment of young people’s financial literacy, which was optional for countries and economies.

The students

Some 690 000 students completed the assessment in 2022, representing about 29 million 15-year-olds in the schools of the 81 participating countries/economies.

The assessment

Computer-based tests were used, with assessments lasting a total of two hours for each student.

Test items were a mixture of multiple-choice questions and questions requiring students to construct their own responses. The items were organised in groups based on a passage setting out a real-life situation. Over 15 hours of test items were used, with different students taking different combinations of test items.

Students also answered a background questionnaire that took 35 minutes to complete. The questionnaire sought information about the students themselves, their homes, and their school and learning experiences. School principals completed a questionnaire that covered the school system and the learning environment. In addition, a new Global Crisis Module was included to collect information on COVID-19-related disruptions to students’ learning and well-being in participating education systems.

To obtain additional information, some countries/economies decided to distribute a questionnaire to teachers to learn about their training and professional development, their teaching practices and their job satisfaction. In some countries/economies, optional questionnaires were distributed to parents, who were asked to provide information on their perceptions of and involvement in their child’s school, their support for learning in the home, and their own engagement with mathematics.

Countries/economies could also choose an optional questionnaire for students about their familiarity with and use of information and communications technologies. A financial literacy questionnaire was also distributed to the students in the countries/economies that conducted the optional financial literacy assessment.

What makes PISA unique

PISA is the most comprehensive and rigorous international programme to assess student performance and to collect data on the student, family and institutional factors that can help explain differences in performance. Decisions about the scope and nature of the assessments and the background information to be collected are made by leading experts in participating countries, and are steered jointly by governments on the basis of shared, policy-driven interests. Substantial efforts and resources are devoted to achieving cultural and linguistic breadth and balance in the assessment materials. Stringent quality-assurance mechanisms are applied in translation, sampling, and data collection. As a consequence, results from PISA have a high degree of validity and reliability.

PISA’s unique features include its:

• policy orientation, which links data on student learning outcomes with data on students’ backgrounds and attitudes towards learning, and on key factors that shape their learning in and outside of school; this exposes differences in performance and identifies the characteristics of students, schools and education systems that perform well;

• innovative concept of “literacy”, which refers to students’ capacity to apply knowledge and skills, and to analyse, reason and communicate effectively as they identify, interpret and solve problems in a variety of situations;

• relevance to lifelong learning, as PISA asks students to report on their motivation to learn, their beliefs about themselves and their learning strategies;

• regularity, which enables countries to monitor their progress in meeting key learning objectives;

• breadth of coverage, which, in PISA 2022, encompassed 37 OECD countries and 44 partner countries and economies.

The PISA 2022 test

The PISA 2022 assessment was conducted principally via computer, as was the case, in 2015 and 2018. Paper-based assessment (PBA) instruments were provided for countries that cannot test their students by computer; but the paper-based assessment was limited to reading, mathematics and science trend items only (i.e. those items that had already been used in prior paper-based assessments). New items were developed only for the computer-based assessment (CBA). Regardless of the delivery mode, the assessment consisted of a cognitive testing session of 120 minutes, followed by a student questionnaire session of approximately 35 minutes.

The PISA 2022 design continued to leverage the innovations made possible by change to CBA as a main mode of assessment in PISA 2015, such as the multistage adaptive testing (MSAT) design implemented in 2018 for reading. The MSAT design for PISA 2022 mathematics builds on and improves on the reading design by 1) presenting items across all stages of the adaptive design, fully balancing item positions and further mitigating possible position effects, 2) using a linear-adaptive hybrid design that assigns students to a fully adaptive MSAT or a linear form to further optimise the data collection for scaling, and 3) using mathematical optimisation methods to optimise the assembly of the cognitive test, both on its linear and its adaptive formats. The 2022 MSAT reading design is the same as in 2018, albeit in a reduced form to account for a smaller item pool since it is a minor domain in this PISA cycle. More detail on the MSAT design employed in PISA 2022 can be found on its Technical Report (OECD, forthcoming).

There were six different kinds of test forms representing various combinations of two of the four domains (i.e., the three core domains, plus the innovative domain). For the CBA design with creative thinking, ninety-four percent of students received test forms involving 60 minutes of mathematics as the major domain, and 60 minutes of one of the three minor or innovative domains (reading, science, or creative thinking). In addition, six percent of students received test forms composed of two minor domains, this aims to fully allow for the estimation of covariance between any two given pair of domains.

Each test form was completed by a sufficient number of students to allow for estimations of proficiency and psychometric analysis of all items by students in each country/economy and in relevant subgroups within a country/economy, such as boys and girls, or students from different social and economic backgrounds.

The assessment of Financial Literacy was offered again in PISA 2022 as an optional CBA component. It was based on a revised Financial Literacy framework founded on the PISA 2022 updated framework. The cognitive instruments included trend items plus a set of new interactive items that were developed specifically for PISA 2022.

An overview of what is assessed in each domain

Box 1.2 presents definitions of the three domains assessed in PISA 2022. The definitions all emphasise the functional knowledge and skills that allow one to participate fully in society. Such participation requires more than just the ability to carry out tasks imposed externally by, for example, an employer; it also involves the capacity to participate in decision making. The more complex tasks in PISA require students to reflect on and evaluate material, not just answer questions that have one correct answer.

Box 1.2. Definitions of the domains

Mathematics: In mathematics, PISA measures an individual’s capacity to reason mathematically and to formulate, employ, and interpret mathematics to solve problems in a variety of real-world contexts. The framework includes concepts, procedures, facts, and tools to describe, explain and predict phenomena. It assists individuals to know the role that mathematics plays in the world and to make the well-founded judgments and decisions needed by constructive, engaged and reflective 21st century citizens.

Reading: The reading assessment in PISA measures an individual’s capacity to understand, use, evaluate, reflect on, and engage with texts in order to achieve one’s goals, develop one’s knowledge and potential, and participate in society.

Science: The science assessment in PISA covers the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen. A scientifically literate person is willing to engage in reasoned discourse about science and technology, which requires the competencies to explain phenomena scientifically, evaluate and design scientific enquiry, and interpret data and evidence scientifically.

Mathematical literacy (Chapter 2) is defined as students’ ability to reason mathematically and to formulate, employ, and interpret mathematics to solve problems in a variety of real-world contexts.

PISA assesses students’ performance in mathematics through questions related to:

• Mathematical Reasoning and Problem-Solving Processes: Includes the mathematical processes that describe what individuals do to connect the context of the problem with mathematics and thus solve the problem.

  Mathematical reasoning (both deductive and inductive) involves evaluating situations, selecting strategies, drawing logical conclusions, developing and describing solutions, and recognising how those solutions can be applied. It is enabled by some key understandings that undergird school mathematics, is the core of mathematical literacy. Included among these key understandings are: understanding quantity, number systems and their algebraic properties; appreciating the power of abstraction and symbolic representation; seeing mathematical structures and their regularities; recognising functional relationships between quantities; using mathematical modelling as a lens onto the real world (e.g. those arising in the physical, biological, social, economic, and behavioural sciences); and understanding variation as the heart of statistics. Regarding problem solving, PISA defines three categories of processes: formulating situations mathematically; employing mathematical concepts, facts, procedures and reasoning; and interpreting, applying and evaluating mathematical outcomes.

• Content: There are four categories (change and relationships; quantity; space and shape; and uncertainty and data) that are closely aligned with the content that is typically found in national school mathematics curricula content strands, such as numbers, algebra, functions, geometry, and data handling.

• Context: The aspect of an individual’s world in which the problems are placed. The framework identifies four contexts: personal, occupational, societal and scientific.

Reading literacy is defined as students’ ability to understand, use, evaluate, reflect on and engage with text to achieve their purposes.

PISA assesses students’ performance in reading through questions that involve a variety of:

• Processes (aspects): Students are not assessed on the most basic reading skills, as it is assumed that most 15‑year‑old students will have acquired these. Rather, students are expected to demonstrate their proficiency in locating information, including both accessing and retrieving information within a piece of text, and searching for and selecting relevant text; understanding text, including both acquiring a representation of the literal meaning of text and constructing an integrated representation of text; and evaluating and reflecting on text, including both assessing its quality and credibility, and reflecting on content and form.

• Text formats: PISA uses both single-source and multiple-source texts; static and dynamic texts; continuous texts (organised in sentences and paragraphs); non‑continuous texts (e.g. lists, forms, graphs or diagrams); and mixed texts.

• Situations: These are defined by the use for which the text was constructed. For example, a novel, personal letter or biography is written for people’s personal use; official documents or announcements are for public use; a manual or report is for occupational use; and a textbook or worksheet is for educational use. Since some students may perform better in one type of reading situation than another, a range of reading situations is included in the test.

Scientific literacy is defined as the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen. A scientifically literate person is willing to engage in reasoned discourse about science and technology, which requires the competencies to explain phenomena scientifically, evaluate and design scientific enquiry, and interpret data and evidence scientifically.

PISA assesses students’ performance in science through questions related to:

• Context: This includes personal, local/national and global issues, both current and historical, that demand some understanding of science and technology.

• Knowledge: This is the understanding of the major facts, concepts and explanatory theories that form the basis of scientific knowledge. Such knowledge includes knowledge of both the natural world and technological artefacts (content knowledge), knowledge of how such ideas are produced (procedural knowledge), and an understanding of the underlying rationale for these procedures and the justification for their use (epistemic knowledge).

• Competencies: These are the ability to explain phenomena scientifically, evaluate and design scientific enquiry, and interpret data and evidence scientifically.

The evolution of reporting student performance in PISA

Results from PISA are reported using scales. Initially, the average score across OECD countries for all three subjects was 500 with a standard deviation of 100, which meant that two-thirds of students across OECD countries scored between 400 and 600 points. These scores represent degrees of proficiency in a particular domain. Scores in subsequent cycles of PISA are calibrated so as to be directly comparable to those in previous cycles; hence the average score across OECD countries in subsequent cycles has fluctuated slightly around the original 500.

Reading literacy was the major domain in 2000, and the reading scale was divided into five proficiency levels of knowledge and skills. The main advantage of this approach is that it is useful for describing what substantial numbers of students can do with tasks at different levels of difficulty. PISA 2003 built upon this approach by specifying six proficiency levels for the mathematics scale. There were four “content” subscales in mathematics: space and shape, change and relationships, quantity, and uncertainty.

In PISA 2012 mathematics was re-assessed as a major domain, and, in addition to the content subscales (with the uncertainty scale being renamed as uncertainty and data for improved clarity), three new subscales were developed to point to the three processes in which students as active problem solvers will engage. These three process subscales are formulating situations mathematically; employing mathematical concepts, facts, procedures and reasoning; and interpreting, apply and evaluating mathematical outcomes, abbreviated as formulating, employing, and interpreting.

Mathematics was once again the major domain of assessment in PISA 2022. The six proficiency levels reported for the overall PISA mathematics in previous cycles were expanded as follows: Level 1 will be renamed Level 1a, and the table describing the proficiencies will be extended to include Levels 1b and 1c. These additional levels have been added to provide greater granularity of reporting in students performing at the lower end of the proficiency scale.

The context questionnaires

To gather contextual information, PISA asks students and the principals of their schools to respond to questionnaires. These take about 35 and 45 minutes, respectively, to complete. The responses to the questionnaires are analysed with the assessment results to provide at once a broader and more nuanced picture of student, school and system performance. Chapter 5 presents the questionnaire framework in detail. The questionnaires from all assessments since PISA’s inception are available on the PISA website: www.oecd.org/pisa/.

The questionnaires seek information about:

• students and their family background, including their economic, social and cultural capital;

• aspects of students’ lives, such as their attitudes towards learning, their habits and life in and outside of school, and their family environment;

• aspects of schools, such as the quality of the schools’ human and material resources, public and private management and funding, decision-making processes, staffing practices, and the school’s curricular emphasis and extracurricular activities offered;

• context of instruction, including institutional structures and types, class size, classroom and school climate, and reading activities in class;

• aspects of learning, including students’ interest, motivation and engagement.

Additionally, the PISA 2022 context questionnaires collected information on COVID-19-related disruptions to students’ learning and well-being in participating education systems. This information can provide context for understanding PISA 2022 results, as well as serve to advance policy discussions about fostering the resiliency of students, schools, and education systems in responding to educational disruptions arising from ongoing and future global crises.

In PISA 2022, three additional questionnaires were offered as options:

• computer familiarity questionnaire, focusing on the availability and use of ICT and on students’ ability to carry out computer tasks and their attitudes towards computer use;

• parent questionnaire, focusing on parents’ perceptions of and involvement in their child’s school, their support for learning at home, school choice, their child’s career expectations, and their background (immigrant/non-immigrant);

• teacher questionnaire, which asks about teachers’ initial training and professional development, their beliefs and attitudes, and their teaching practices; separate questionnaires were developed for teachers of the test language and for other teachers in the school.

The contextual information collected through the student, school and optional questionnaires comprises only a part of the information available to PISA. Indicators describing the general structure of education systems (their demographic and economic contexts, such as their costs, enrolments, school and teacher characteristics, and some classroom processes) and their effect on labour market outcomes are routinely developed and applied by the OECD (e.g. in the annual OECD publication, Education at a Glance).

A collaborative project

PISA is the result of a collaborative between the OECD Secretariat, international contractors, experts, and teams at participating countries and economies. The assessments are developed co-operatively, agreed by participating countries/economies, and implemented by national centres in each participant. The co‑operation of students, teachers and principals in participating schools has been crucial to the success of PISA during all stages of development and implementation.

The PISA Governing Board (PGB), composed of representatives at the senior policy level from all participating countries/economies, determines the policy priorities for PISA. It also oversees adherence to these priorities during the implementation of the programme. The PGB sets priorities for developing indicators, establishing assessment instruments and reporting results. Experts from participating countries/economies also serve on working groups tasked with linking PISA policy objectives with the best available technical expertise in the different assessment domains. By participating in these expert groups, countries/economies ensure that the instruments are internationally valid and take into account differences in cultures and education systems.

Participating countries/economies implement PISA at the national level through National Centres managed by National Project Managers, subject to the agreed administration procedures. National Project Managers play a vital role in ensuring that the implementation is of high quality. They also verify and evaluate survey results, analyses, reports and publications.

The frameworks were developed under the guidance of expert groups (Annex C). The OECD Secretariat has overall managerial responsibility for the programme, monitors its implementation on a day‑to‑day basis, acts as the secretariat for the PGB, builds consensus among countries, and serves as the interlocutor between the PGB and the international contractors charged with implementation. The OECD Secretariat is also responsible for producing the indicators, and for the analysis and preparation of the international reports and publications, in co‑operation with the contractors and in close consultation with participating countries/economies at both the policy (PGB) and implementation (National Project Managers) levels.