PISA 2018 Results (Volume I)

Annex C. Released items from the PISA 2018 computer-based reading assessment

Items from Rapa Nui, the released unit from the PISA 2018 reading assessment, and items used in the assessment of reading fluency

One new unit, Rapa Nui, was released from the main survey of the PISA 2018 computer-based reading assessment; the seven items from this unit are presented in this annex. Two other units, Chicken Forum and Cow’s Milk, were tested in the field trial but not used in the PISA 2018 main survey. These units, along with the untested unit The Galapagos Islands, are available on line at www.oecd.org/pisa. All four of these units were developed in accordance with the new PISA 2018 reading literacy framework. The annex concludes with sentences that illustrate those used in the reading-fluency assessment.

Screenshots of the interface used in PISA 2018 are shown to give readers an understanding of how students interacted with the assessment and its items. Interactive versions of all of these units are also available at www.oecd.org/pisa.

Unit CR551: Rapa Nui

Rapa Nui scenario

 

In this unit’s scenario, the student is preparing to attend a lecture about a professor’s field work, which was conducted on the island of Rapa Nui. The situation is classified as educational because it represents a student conducting background research on Rapa Nui in preparation to attend a lecture.

Rapa Nui is a multiple-source unit. It consists of three texts: a webpage from the professor’s blog, a book review, and a news article from an online science magazine. The blog is classified as a multiple-source text; dynamic (the webpage contains active links to the other texts in the unit); continuous; and narrative. The blog post is an example of a multiple-source text because the comment section at the bottom of the blog page represents different authors. Both the book review and the news article are classified as single text; static; continuous; and argumentative.

Initially, the student is provided with the blog post only. Several questions are presented that focus only on the content of this blog. Once those questions have been answered, the student receives the second text – the book review. After reading the book review, the student responds to a question that focuses solely on its content. The student then receives the third text – the article from the online science magazine. The student sees questions that focus only on the article. After that, the student is given items that require integrating the information from all sources.

This model was used for several of the multiple-text units in the new material developed for reading literacy. This approach was chosen because it allows the student first to demonstrate proficiency on questions that are related to one text and then to demonstrate the ability to handle information from multiple texts. This is an important design feature because there may be readers who can succeed with information when it is presented in a single text and even integrate information within one text, but who struggle when asked to integrate across multiple texts. Thus, this design allows students with varying levels of ability to demonstrate proficiency on at least some elements of the unit.

The “Rapa Nui” unit was intended to be of moderate to high difficulty. The three texts result in a larger amount of information to work through within the unit compared to a single-text unit. In addition, the student needs to consider the way the texts are related to one another, requiring him or her to recognise whether the texts corroborate each other or whether they differ in their stances. This kind of cognitive engagement with the material and the unit overall is expected to require more effort than a unit that presents all the information within one text.

Please note that the screenshot provided for released item #1 shows the full text of the blog for the purposes of this report. The student had to scroll to see the full text in the programmed version, which was programmed uniformly across language versions so that all students would have to scroll to see the full text.

Rapa Nui released item #1

 

In this item, the student must locate the correct information within the blog post. The difficulty of the item likely stems from the existence of other time-related information within the blog, i.e. the date it was posted and the time period in which the first mystery of the moai was solved (the 1990s). Here, the correct answer is (B) Nine months ago.

 

Item number	CR551Q01
Cognitive process	Accessing and retrieving information within a piece of text
Response format	Simple multiple choice – Computer scored
Difficulty	559 – Level 4
Source type	Single source

Rapa Nui released item #2

 

In this item, the student must understand that the second mystery mentioned in the blog post: what happened to the large trees that once grew on Rapa Nui and were used to move the moai? This is an open response/human coded item, and the coding guide used in the main survey is provided below. For this item, the student could provide a direct quotation from the blog ( “What happened to these plants and large trees that had been used to move the moai?”) or an accurate paraphrase. This item was coded with high reliability in the main survey.

 

Item number	CR551Q05
Cognitive process	Representing literal meaning
Response format	Open response – Human coded
Difficulty	513 – Level 3
Source type	Single source

For full credit, responses must refer to the disappearance of the materials used to move the statues (moai).

• What happened to these plants and large trees that had been used to move the moai? [Direct quotation]

• There are no large trees left that could have moved the moai.

• There are grasses, shrubs and some small trees, but no trees large enough to move the large statues.

• Where are the large trees? [Minimal]

• Where are the plants? [Minimal]

• What happened to the resources that were needed to transport the statues?

• She was referring to what moved the Moai because when she looked around there were no big trees or plants. She is also wondering what happened to them. [Although this response begins by referring to the wrong mystery, it contains the correct elements.]

Rapa Nui released item #3

For this item, the student is presented with the second text in the unit, a book review of Collapse, which was referenced in the blog post. The student must complete a table by selecting “Fact” or “Opinion” for each row. The question asks the student to identify whether each statement from the book review is a fact or an opinion. The student must first understand the literal meaning of each statement and then decide if the content was factual or represented the perspective of the author of the review. In this way, the student must focus on the content and how it is presented rather than just the meaning. To receive full credit for this item, the student was required to get all 5 rows correct. For partial credit, students were required to get 4 out of the 5 rows correct. If students got fewer than 4 rows correct, they received no credit. The correct answers are: Fact, Opinion, Fact, Fact, Opinion.

 

 

Item number	CR551Q06
Cognitive process	Reflecting on content and form
Response format	Complex multiple choice – Computer scored
Difficulty	For full credit, 654 – Level 5; for partial credit, 528 – Level 3
Source type	Single source

Rapa Nui released item #4

 

For this item, the student is presented with the third text in the unit – an article from an online science magazine. Note that at this point in the unit all three texts are available to the student using a tab structure; the student can click on any tab to toggle back and forth between the texts. The item itself remains fixed on the left side of the screen during any toggling action. In this item, the student is required to locate the section of the article that contains the reference to the scientists and Jared Diamond (paragraph 2) and identify the sentence that contains the information agreed upon. While texts are available to the student, this item is not classified with a cognitive process that reflects the use of multiple sources. This is because the student can find the answer within this text, and the item instructions on the upper left corner instruct the student to refer to this article only. Thus, the support from the item instructions eliminates the need to consider the other sources. The difficulty of this item is likely driven by the existence of plausible (but incorrect) distracting information within the paragraph with respect to human settlement. Here, the correct answer is (B) Large trees have disappeared from Rapa Nui.

 

Item number	CR551Q08
Cognitive process	Accessing and retrieving information within a piece of text
Response format	Simple multiple choice – Computer scored
Difficulty	634 – Level 5
Source type	Single source

Rapa Nui released item #5

 

In this item, the student is required to understand what information in the text supports, or corroborates, the theory put forward by the scientists. The correct answer is (D) The remains of palm nuts show gnaw marks made by rats. Here, the student must go beyond an understanding of the text and identify which element of the text can be used as evidence to support a claim. All other items classified as detect and handle conflict require detecting a conflict between two sources or recognising that the information is in two or more sources and is corroborated. However, in discussing this item prior to the field trial, the experts felt that the act of identifying which piece of information supports the theory proposed by Carl Lipo and Terry Hunt was most appropriately identified by the cognitive process of detect and handle conflict. Furthermore, while this item could have been classified as requiring only a single source in order to be solved, the requirement for the student to first consider the theory proposed by Lipo and Hunt and then to determine which piece of evidence supports this theory is akin to working with multiple sources.

 

Item number	CR551Q09
Cognitive process	Detecting and handling conflict
Response format	Simple multiple choice – Computer scored
Difficulty	597 – Level 4
Source type	Multiple source

Rapa Nui released item #6

In this item, students must integrate information across the texts with respect to the differing theories put forward by Jared Diamond on the one hand and Carl Lipo and Terry Hunt on the other. The student must identify the shared effect (the disappearance of the large trees) by rejecting information presented in the blog post about where the moai were carved (in the same quarry). Further, the student must understand what each scientist believes is the cause of the disappearance. To receive credit for this item, the student was required to get all three answers correct. The correct answers are: Cause (Jared Diamond) – Humans cut down trees to clear land for agriculture and other reasons. Cause (Carl Lipo and Terry hunt) – Polynesian rats ate tree seeds and as a result no new trees could grow. Effect (shared) – The large trees disappeared from Rapa Nui.

 

 

Item number	CR551Q10
Cognitive process	Integrating and generating inferences across multiple sources
Response format	Complex multiple choice – Computer scored
Difficulty	665 – Level 5
Source type	Multiple source

Rapa Nui released item #7

 

In this item, the student must integrate information from across the texts and decide which theory to support. In this way, the student must understand the theories – and that they are at odds with one another – and must present a response that contains support from the texts. To receive credit, a student could choose to support either theory or could choose neither theory as long as the explanation is focused on the need for additional research. This is an open response/human coded item, and the coding guide used in the main survey is provided below. This item was coded with high reliability in the main survey.

 

Item number	CR551Q11
Cognitive process	Detecting and handling conflict
Response format	Open response – Human coded
Difficulty	588 – Level 4
Source type	Multiple source

For full credit, at least one of the following descriptions had to be included:

The people cut down or used the trees (to move the moai and/or cleared the land for agriculture).

The rats ate the seeds of the trees (so new trees could not grow).

It is not possible to say exactly what happened to the large trees until further research is conducted.

Sample responses that would receive full credit include:

• I think the trees disappeared because people cut too many of them down to move the moai. [1]

• People cleared the land for agriculture. [1]

• Trees were used to move moai. [1]

• People cut the trees down. [1]

• It was the people’s fault because they wanted to move the moai. [1 – this response doesn’t explicitly refer to cutting down the trees, but it is acceptable because they refer to people and one reason they cut down the trees (to move the moai)]

• People’s fault. They destroyed the environment. [1 – this response doesn’t explicitly refer to cutting down the trees, but it is an acceptable way of summarizing the results of cutting down the trees.]

• I think the rats probably caused the most damage by eating the seeds of the trees. [2]

• The rats ate the seeds. [2]

• There is no proof that either one is correct, so we have to wait until there is more information. [3]

• Both. The people cut down the big trees for farming, and then the rats ate the tree seeds! [1 and 2]

Reading fluency

In PISA 2018, the Reading Expert Group recommended including a measure of reading fluency to better assess and understand the reading skills of students in the lower proficiency levels. PISA defines reading fluency as the ease and efficiency with which one can read and understand a piece of text. Reading fluently requires that one can recognize words within a text accurately and automatically and can then parse and process the words into a coherent whole in order to comprehend the overall meaning of the text. When these processes are done efficiently, students’ cognitive resources are available for higher-level comprehension tasks, allowing students to engage with texts more deeply.

In the PISA 2018 assessment of reading fluency, students were given three minutes to evaluate the sensibility of as many sentences as they could (i.e. Does the sentence make sense – Yes or No). The number of sentences was restricted to 21 or 22 sentences per student so that most students would be able to complete the task within the three minutes. Students were not cut off in the middle of an item or notified that they did not complete all the sentences. Instead, if a student reached the three minutes while viewing a sentence, the task ended after they completed that sentence’s sensibility judgment. This was done so that students would maintain motivation for the remaining sections of the PISA assessment.

Items in this task were the easiest items in the reading-literacy assessment in PISA 2018. Difficulty information is not provided in this report for the practice items because data for these items was not analysed. However, in the assessment of reading fluency, the items fell into proficiency Level 1c and Level 1b. One item was in Level 1a. Items that did not make sense and required a “No” response were more difficult than items that made sense and required a “Yes” response.

The introduction to and practice items for the reading-fluency task are provided below along with an explanation for how students were oriented to the task.

Reading fluency: Introduction

In this introduction, students are given the basic instructions for what they will do in the fluency task. Students are notified that the next sentence will appear as soon as they respond so that they are prepared for this style of presentation.

 

Reading fluency: Static examples

Students are given a set of static examples so that the sensibility judgements are understood prior to interacting with dynamic practice items. Here, three example sentences are provided, two that make sense (a Yes response is correct) and one that does not make sense (a No response is correct).

 

Reading fluency: Dynamic practice

The next three images show three dynamic-practice items. Students complete these dynamic-practice items prior to receiving the first fluency item so that they understand the response mode for the item. For each example, as soon as the student clicks on “Yes” or “No”, the next item appears.

Reading fluency: Dynamic-practice item #1

Here, the correct answer is “Yes”.

 

Reading fluency: Dynamic-practice item #2

Here, the correct answer is “No”.

 

Reading fluency: Dynamic-practice item #3

Here, the correct answer is “Yes”.

 

Reading fluency: End of practice

Students are told that they have completed the practice sentences. They are also given the time limit for the task – three minutes – and they are told to complete as many sentences as they can within the time limit. Once the student clicks on the NEXT arrow, the task begins and is carried out in the same way as the dynamic-practice items. Once students have completed the task, they are notified that the first section of the test is complete and the answers have been saved.