PISA 2018 Results (Volume V)

What is PISA?

“What should citizens know and be able to do?” In response to that question and to the need for internationally comparable evidence on student performance, the Organisation for Economic Co-operation and Development (OECD) launched the Programme for International Student Assessment (PISA) in 2000.

PISA is a triennial survey of 15-year-old students around the world that assesses the extent to which they have acquired key knowledge and skills essential for full participation in social and economic life. PISA assessments include the core school subjects of reading, mathematics and science, and also innovative areas, such as creative problem solving (2012), collaborative problem solving (2015), global competence (2018) and creative thinking (2021). The assessments do not just ascertain whether students near the end of their compulsory education can reproduce what they have learned; they also examine how well students can extrapolate from what they have learned and apply their knowledge in unfamiliar settings, both in and outside of school. This approach reflects the fact that the modern world rewards individuals not just for what they know, but increasingly for what they can do with what they know.

What is unique about PISA?

PISA is unique because of its:

• policy orientation, which links data on student learning outcomes with data on students’ backgrounds and attitudes towards learning, and with key factors that shape their learning, in and outside of school; by doing so, PISA can highlight differences in performance and identify the characteristics of students, schools and education systems that perform well

• innovative concept of “literacy”, which refers to students’ capacity to apply their knowledge and skills in key areas, and to analyse, reason and communicate effectively as they identify, interpret and solve problems in a variety of situations

• relevance to lifelong learning, as PISA asks students to report on their motivation to learn, their beliefs about themselves, and their learning strategies

• regularity, which enables countries to monitor their progress in meeting key learning objectives

• breadth of coverage, which, in PISA 2018, encompassed all 37 OECD countries and 42 partner countries and economies.

Which countries and economies participate in PISA?

PISA is used as an assessment tool in many regions around the world. It was implemented in 43 countries and economies in the first assessment (32 in 2000 and 11 in 2002), 41 in the second assessment (2003), 57 in the third assessment (2006), 75 in the fourth assessment (65 in 2009 and 10 in 2010), 65 in the fifth assessment (2012) and 72 in the sixth assessment (2015). In 2018, 79 countries and economies participated in PISA.

What does the test measure?

In each round of PISA, one subject is tested in detail, taking up nearly half of the total testing time. The main subject in 2018 was reading, as it was in 2000 and 2009. Mathematics was the main subject in 2003 and 2012, while science was the main subject in 2006 and 2015. With this alternating schedule, a thorough analysis of achievement in each of the three core subjects is presented every nine years; an analysis of trends is offered every three years.

The PISA 2018 Assessment and Analytical Framework (OECD, 2019) presents definitions and more detailed descriptions of the subjects assessed in PISA 2018:

• Reading literacy is defined as students’ capacity to understand, use, evaluate, reflect on and engage with texts in order to achieve one’s goals, develop one’s knowledge and potential, and participate in society.

• Mathematics literacy is defined as students’ capacity to formulate, employ and interpret mathematics in a variety of contexts. It includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena.

• Science literacy is defined as the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen. A scientifically literate person is willing to engage in reasoned discourse about science and technology, which requires the competencies to explain phenomena scientifically, evaluate and design scientific enquiry, and interpret data and evidence scientifically.

Map of PISA countries and economies

* Puerto Rico participated in the PISA 2015 assessment (as an unincorporated territory of the United States).

** B-S-J-Z (China) refers to four PISA 2018 participating Chinese provinces/municipalities: Beijing, Shanghai, Jiangsu and Zhejiang. In PISA 2015, the four PISA participating Chinese provinces/municipalities were: Beijing, Shanghai, Jiangsu and Guangdong.

Box A. Key features of PISA 2018

The content

• The PISA 2018 survey focused on reading, with mathematics, science and global competence as minor areas of assessment. PISA 2018 also included an assessment of young people’s financial literacy, which was optional for countries and economies.

The students

• Some 600 000 students completed the assessment in 2018, representing about 32 million 15-year-olds in the schools of the 79 participating countries and economies.

The assessment

• Computer-based tests were used in most countries, with assessments lasting a total of two hours. In reading, a multi-stage adaptive approach was applied in computer-based tests whereby students were assigned a block of test items based on their performance in preceding blocks.

• Test items were a mixture of multiple-choice questions and questions requiring students to construct their own responses. The items were organised into groups based on a passage of text describing a real-life situation. About 930 minutes of test items for reading, mathematics, science and global competence were covered, with different students taking different combinations of test items.

• Students also answered a background questionnaire, which took about 35 minutes to complete. The questionnaire sought information about the students themselves, their attitudes, dispositions and beliefs, their homes, and their school and learning experiences. School principals completed a questionnaire that covered school management and organisation, and the learning environment.

• Some countries/economies also distributed additional questionnaires to elicit more information. These included: in 19 countries/economies, a questionnaire for teachers asking about themselves and their teaching practices; and in 17 countries/economies, a questionnaire for parents asking them to provide information about their perceptions of and involvement in their child’s school and learning.

• Countries/economies could also choose to distribute three other optional questionnaires for students: 52 countries and economies distributed a questionnaire about students’ familiarity with computers; 32 countries/economies distributed a questionnaire about students’ expectations for further education; and 9 countries/economies distributed a questionnaire, developed for PISA 2018, about students’ well-being.

How is the assessment conducted?

As was done in 2015, PISA 2018 delivered the assessment of all subjects via computer. Paper-based assessments were provided for countries that were not able to test their students by computer, but the paper-based assessment was limited to reading, mathematics and science trend items, which were originally developed for previous PISA assessments.1 Since 2015, new items were developed for the computer-based assessment only.

The 2018 computer-based assessment was designed as a two-hour test. Each test form allocated to students comprised four 30-minute clusters of test material. For the main subject of reading, material equivalent to 15 30-minute clusters was developed. This material was organised into blocks instead of clusters, as the PISA 2018 reading assessment took a multi-stage adaptive approach. The reading assessment was composed of a core stage followed by stage 1 and stage 2. At the beginning of stages 1 and 2, students were assigned blocks of items of either greater or lesser difficulty, depending on their performance in earlier stages (see Chapter 1 in PISA 2018 Results [Volume I]: What Students Know and Can Do, for more detailed information on the multi-stage adaptive approach). To measure trends in the subjects of mathematics and science, six clusters were included in each subject. In addition, four clusters of global competence items were developed.2 There were 72 different test forms.3 Students spent one hour on the reading assessment plus one hour on one or two other subjects – mathematics, science or global competence.

Countries that used paper-based delivery for the main survey measured student performance with 30 pencil-and-paper forms containing trend items in the three core PISA subjects. The reading items in these paper-based forms were based on the 2009 reading literacy framework and did not include any items based on the new 2018 reading literacy framework.

The assessment of financial literacy was offered as an option in PISA 2018. It was based on the same framework as that developed for PISA 2012, which was also used in PISA 2015.4 The financial literacy assessment lasted one hour (in addition to the regular PISA assessment) and comprised two clusters distributed to a subsample of students in combination with the reading and mathematics assessments.

To gather contextual information, PISA 2018 asked students and the principal of their school to respond to questionnaires. The student questionnaire took about 35 minutes to complete; the questionnaire for principals took about 45 minutes to complete. The responses to the questionnaires were analysed with the assessment results to provide both a broader and more nuanced picture of student, school and system performance. The PISA 2018 Assessment and Analytical Framework (OECD, 2019) describes the genesis of the questionnaires in detail. The questionnaires from all assessments since PISA’s inception are available on the PISA website: www.pisa.oecd.org.

The questionnaires seek information about:

• students and their family backgrounds, including their economic, social and cultural capital

• aspects of students’ lives, such as their attitudes towards learning, their habits and life in and outside of school, and their family environment

• aspects of schools, such as the quality of the schools’ human and material resources, public and private management and funding, decision-making processes, staffing practices, the school’s curricular emphasis and the extracurricular activities it offers

• the context of instruction, including institutional structures and types, class size, classroom and school climate, and reading activities in class

• aspects of learning, including students’ interest, motivation and engagement.

In PISA 2018, five additional questionnaires were offered as options:

• computer familiarity questionnaire, focusing on the availability and use of information and communications technologies (ICT), and on students’ ability to carry out tasks on computers and their attitudes towards using computers

• well-being questionnaire, (new to PISA 2018) on students’ perceptions of their health, life satisfaction, social connections and activities in and outside of school

• educational career questionnaire, which collects additional information on interruptions in schooling, preparation for students’ future career, and support with language learning

• parent questionnaire, focusing on parents’ perceptions of and involvement in their child’s school, their support for learning at home, school choice, their child’s career expectations, and their background (immigrant/non-immigrant)

• teacher questionnaire, which asks about teachers’ initial training and professional development, their beliefs and attitudes, and their teaching practices. Separate questionnaires were developed for teachers of the test language and for other teachers in the school.

The contextual information collected through the student, school and optional questionnaires is complemented by system-level data. Indicators describing the general structure of each education system, such as expenditure on education, stratification, assessments and examinations, appraisals of teachers and school leaders, instruction time, teachers’ salaries, actual teaching time and teacher training are routinely developed and analysed by the OECD. These data are extracted from the annual OECD publication, Education at a Glance: OECD Indicators, for the countries that participate in the annual OECD data collection administered through the OECD Indicators of Education Systems (INES) Network. For other countries and economies, a special system-level data collection was conducted in collaboration with PISA Governing Board members and National Project Managers.

Who are the PISA students?

Differences between countries in the nature and extent of pre-primary education and care, the age at entry into formal schooling, the structure of the education system, and the prevalence of grade repetition mean that school grade levels are often not good indicators of where students are in their cognitive development. To better compare student performance internationally, PISA targets students of a specific age. PISA students are aged between 15 years 3 months and 16 years 2 months at the time of the assessment, and they have completed at least 6 years of formal schooling. They can be enrolled in any type of institution, participate in full-time or part-time education, in academic or vocational programmes, and attend public or private schools or foreign schools within the country. (For an operational definition of this target population, see Annex A2). Using this age across countries and over time allows PISA to consistently compare the knowledge and skills of individuals born in the same year who are still in school at age 15, despite the diversity of their education histories in and outside of school.

The population of PISA-participating students is defined by strict technical standards, as are the students who are excluded from participating (see Annex A2). The overall exclusion rate within a country is required to be below 5% to ensure that, under reasonable assumptions, any distortions in national mean scores would remain within plus or minus 5 score points, i.e. typically within the order of magnitude of 2 standard errors of sampling. Exclusion could take place either through the schools that participated or the students who participated within schools (see Annex A2).

There are several reasons why a school or a student could be excluded from PISA. Schools might be excluded because they are situated in remote regions and are inaccessible, because they are very small, or because of organisational or operational factors that precluded participation. Students might be excluded because of intellectual disability or limited proficiency in the language of the assessment. In 31 of the 79 countries and economies that participated in PISA 2018, the percentage of school-level exclusions amounted to less than 1%; it was 4% or less in all except five countries. When the exclusion of students who met the internationally established exclusion criteria is also taken into account, the exclusion rates increase slightly. However, in 2018, the overall exclusion rate remained below 2% in 28 participating countries and economies, below 5% in 63 participating countries and economies, and below 7% in all countries except Sweden (11.1%), Israel (10.2%), Luxembourg and Norway (both 7.9%). For more detailed information about school and student exclusion from PISA 2018, see Annex A2.

Where can you find the results?

The initial PISA 2018 results are released in six volumes:

• Volume I: What Students Know and Can Do provides a detailed examination of student performance in reading, mathematics and science, and describes how performance has changed over time.

• Volume II: Where All Students Can Succeed examines gender differences in student performance, the link between students’ socio-economic status and immigrant background, on the one hand, and their performance and other outcomes, on the other, and the relationship between all of these variables and students’ well-being. Trends in these indicators over time are examined when comparable data are available.

• Volume III: What School Life Means for Students’ Lives focuses on the physical and emotional health of students, the role of teachers and parents in shaping the school climate, and the social life at school. The volume also examines indicators of student well-being, and how these are related to school climate.

• Volume IV: Are Students Smart about Money? examines 15-year-old students’ understanding about money matters in the 20 countries and economies that participated in this optional assessment. The volume explores how the financial literacy of 15-year-old students is associated with their competencies in reading, mathematics and science, with their socio-economic status, and with their previous experiences with money. It also offers an overview of financial education in schools in the participating countries and economies, and provides case studies.

• Volume V: Effective Policies, Successful Schools analyses schools and school systems and their relationship with education outcomes more generally. The volume covers school governance, selecting and grouping students, and the human, financial, educational and time resources allocated to teaching and learning. Trends in these indicators are examined when comparable data are available.

• Volume VI: Are Students Ready to Thrive in an Interconnected World? examines students’ ability to consider local, global and intercultural issues, understand and appreciate different perspectives and world views, interact respectfully with others, and take responsible action towards sustainability and collective well-being. It does so through both an assessment completed by students and questionnaires completed by students and school principals.5

Volumes II and III are published at the same time as Volume I, in December 2019; Volumes IV, V and VI are published in 2020.

The frameworks for assessing reading, mathematics, science, financial literacy and global competence in 2018 are described in the PISA 2018 Assessment and Analytical Framework (OECD, 2019). They are also summarised in Volume I.

Technical annexes at the end of this volume describe how questionnaire indices were constructed and discuss. Many of the issues covered in the technical annexes are elaborated in greater detail in the PISA 2018 Technical Report (OECD, forthcoming).

A selection of key tables referred to in the analyses are included at the end of the respective volume in Annex B1, and a set of additional data tables is available on line (www.oecd.org/pisa). A Reader’s Guide is also provided in each volume to aid in interpreting the tables and figures that accompany the report.

References

OECD (forthcoming), PISA 2018 Technical Report, OECD Publishing, Paris.

OECD (2019), PISA 2018 Assessment and Analytical Framework, OECD Publishing, Paris, https://doi.org/10.1787/b25efab8-en.

Notes

← 1. The paper-based form was used in nine countries/economies: Argentina, Jordan, Lebanon, the Republic of Moldova, the Republic of North Macedonia, Romania, Saudi Arabia, Ukraine and Viet Nam.

← 2. The global competence assessment was not available in the countries/economies that conducted the PISA 2018 assessment on paper. It was conducted in Albania, Brunei Darussalam, Canada, Chile, Colombia, Costa Rica, Croatia, Greece, Hong Kong (China), Indonesia, Israel, Kazakhstan, Korea, Latvia, Lithuania, Malta, Morocco, Panama, the Philippines, the Russian Federation, Serbia, Singapore, the Slovak Republic, Spain, Chinese Taipei, Thailand and Scotland (United Kingdom). However, the global competence module was included in the student questionnaire, which was distributed in 56 of the countries/economies that took part in PISA 2018.

← 3. Thirty-six test forms were prepared for countries that did not participate in the global competence assessment.

← 4. The financial literacy assessment was conducted in Australia, Brazil, Bulgaria, Canada, Chile, Estonia, Finland, Georgia, Indonesia, Italy, Latvia, Lithuania, the Netherlands, Peru, Poland, Portugal, the Russian Federation, Serbia, the Slovak Republic, Spain and the United States.

← 5. The global competence assessment was conducted in 27 countries and economies, while the global competence module was included in questionnaires distributed in 56 countries and economies.