PISA 2022 Results (Volume III)

1. Measuring creative thinking

Abstract

This chapter summarises the conceptual foundations of the PISA 2022 Creative Thinking assessment and presents a selection of released items from the test. The chapter also describes how well students around the world demonstrate creative thinking at different levels of proficiency.

For Australia*, Canada*, Denmark*, Hong Kong (China)*, Jamaica*, Latvia*, the Netherlands*, New Zealand* and Panama* caution is advised when interpreting estimates because one or more PISA sampling standards were not met (see Reader’s Guide, Annexes A2 and A4).

For Albania** and the Dominican Republic**, caution is required when comparing estimates with other countries/economies as a strong linkage to the international PISA creative thinking scale could not be established (see Reader's Guide and Annex A4).

“Creativity is seeing what others see and thinking what no one else ever thought.”

Albert Einstein

For the first time, in its 2022 cycle, PISA has measured the creative thinking skills of 15-year-olds in 64 countries and economies. This chapter first presents the rationale for assessing creative thinking in PISA, and then describes how the creative thinking construct is defined and measured in the PISA 2022 test. The chapter then presents a selection of released items from the test to illustrate how students were asked to demonstrate creative thinking across different domain contexts. Finally, the chapter describes creative thinking at different levels of proficiency and summarises how the creative thinking scale was constructed to assess and describe students’ performance in the test.

Why measure creative thinking?

Creativity has driven innovation in human culture and society for millennia – from the sciences and technology, to philosophy, the arts and the humanities. One fundamental goal of education is to equip individuals with the competencies they need to succeed in life and society, for both their own and collective well-being (OECD, 2018[1]). Creativity, creative thinking and innovation are amongst these important competencies.1

Creative thinking helps prepare young people to adapt to a rapidly changing world that demands flexible and innovative workers. Beyond preparing students for the labour market, creative thinking in education contributes to students’ holistic development – it supports learning, problem solving and metacognitive skills through exploration and discovery, helping students to interpret information in personally meaningful ways. It has also been found to support a range of other important aspects of students’ development and achievement.2

The importance of developing creative thinking in education is reflected in national curricula worldwide. Nearly all PISA participating countries or economies with data available reported creativity as an intended student outcome in secondary education (Figure III.1.1).3

Figure III 1.1. Creativity in curricula worldwide

Percentage of countries/economies in which various subject areas refer to creativity in secondary education curricula

Notes: For each subject, the share of countries/economies is based on the number that reported the subject containing a reference to creative thinking (or related terms) over the number that reported including that subject within their relevant curriculum or learning standards (see N reported in brackets).

Where it was not possible to establish whether a subject referred to creativity, responses were counted as missing responses and excluded from the total response count (N).

Secondary education refers to ISCED Levels 2 and 3. In some jurisdictions, the curriculum or learning standards for primary education (ISCED Level 1) and lower secondary education (ISCED Level 2) are integrated; in these cases, secondary education refers only to upper secondary education (ISCED Level 3).

Source: OECD (2023[2]), Supporting Students to Think Creatively: What Education Policy Can Do. The StatLink URL of this figure is available at the end of the chapter.

How PISA 2022 defines creative thinking

In its 2022 cycle, PISA defines creative thinking as “the competence to engage productively in the generation, evaluation and improvement of ideas that can result in original and effective solutions, advances in knowledge and impactful expressions of imagination”. It focuses on the cognitive processes required to engage in creative work and is aligned with the concept of “little-c” creativity – in other words, a malleable capacity that can be developed through practice and that can be reasonably demonstrated in everyday contexts (see Box III.1.1).

Box III 1.1. “Big-C” vs. “little-c” creativity

Creativity can manifest in different ways, but research generally distinguishes between “big-C” and “little-c” creativity (Csikszentmihalyi, 2013[3]; Simonton, 2013[4]). “Big-C” creativity is associated with intellectual or technological breakthroughs or artistic or literary masterpieces that require deep expertise in a given context. In contrast, all people can demonstrate “little-c” (or “everyday”) creativity by engaging in creative thinking. This is the type of creativity people manifest when, for example, they arrange photos for display, combine leftovers to make a tasty meal, or find solutions to day-to-day problems. “Little-c” creativity can be developed through practice and honed through education (Kaufman and Beghetto, 2009[5]).

This definition of creative thinking includes both divergent cognitive processes (i.e. the ability to generate diverse ideas and creative ideas) and convergent cognitive processes (i.e. the ability to evaluate ideas and identify improvements to those ideas). For measurement purposes in PISA 2022, the construct of creative thinking consisted of three ideation processes (see Figure III.1.2 and Box III.1.2).

Figure III 1.2. The PISA 2022 competency model for creative thinking

Source: OECD (2022[6]), Thinking Outside the Box: The PISA 2022 Creative Thinking Assessment.

Box III 1.2. The three ideation processes involved in creative thinking in PISA 2022

In PISA 2022, creative thinking was premised on three ideation processes (Figure III.1.2). These ideation processes reflect the PISA definition and encompass the cognitive skills that are relevant to creative thinking in the classroom (see Annex A1 for a more detailed description). The distribution of test items across the three ideation processes is as follows: 12 items correspond to “generate diverse ideas”, 11 items correspond to “generate creative ideas”, and 9 items correspond to “evaluate and improve ideas”.

Generate diverse ideas

This ideation process refers to a student’s capacity to think flexibly by generating ideas that are different to each other. In the context of measuring creative thinking ideation skills, both ideational fluency (i.e. the total number of ideas produced) and ideational flexibility (i.e. how fundamentally different ideas are) are important factors for estimating creative potential (Guilford, 1956[7]; Runco and Acar, 2012[8]).

Generate creative ideas

Creative ideas are usually defined as being both novel and useful. Expecting 15-year-olds to think of unique and novel ideas would be neither feasible nor appropriate in the context of PISA; however, originality is a useful proxy for measuring the extent to which students can think outside of the box. Defined by Guildford (1950[9]) as “statistical infrequency”, originality encompasses the qualities of newness, remoteness, novelty or unusualness, and generally refers to deviance from patterns that are observed within a population. In the PISA assessment, originality is measured in relation to the responses of other students who complete the same task – if relatively few other students suggest the same idea, then a response is considered original.

Evaluate and improve ideas

This ideation process refers to a student’s capacity to evaluate limitations in ideas and improve their originality. Evaluative processes help to identify and remediate deficiencies in initial ideas as well as ensure that ideas or solutions are appropriate, adequate, efficient and effective (Cropley, 2006[10]). They often lead to further iterations of idea generation that can ultimately improve creative outcomes.

The PISA definition of creative thinking focuses on those ideation processes that can be engaged in different learning and problem-solving contexts. These include learning contexts that require imagination and expression, such as creative writing or the visual and performance arts, as well as those in which generating and improving upon ideas is functional to investigating problems or phenomena, or to designing innovative solutions.

Sample items

Students who took the creative thinking test in PISA 2022 spent one hour on creative thinking items, with the remaining hour of PISA testing time assigned to mathematics, reading or scientific literacy items. Creative thinking items were organised into units based on a common stimulus. Each unit varied according to the ideation process involved, the unit length, the number of items in the unit, and the domain context (see Box III.1.3).

Selected items from 9 of the 18 creative thinking units developed for the PISA 2022 test are described below. At least one unit from each domain context is presented. For each unit, a brief description of the unit context and scenario is provided, followed by a screenshot and description of the sample item(s) from that unit. For some items, genuine student responses are also presented, as well as a description of the item-specific coding criteria. For more detailed information on the scoring processes and the general approach to awarding full or partial credit across items, see Annex A1. Information on the empirical difficulty of select items presented here, at different credit levels, is also included in Table III.1.2 towards the end of this chapter. For more information on the released items, see also Annex C.

Box III 1.3. The four domain contexts in the PISA 2022 Creative Thinking assessment

Researchers now recognise that, to some extent, the internal resources needed to engage in creative work differ by domain (Baer, 2011[11]; Baer and Kaufman, 2005[12]). Situating creative thinking tasks across different domain contexts has several advantages in the context of the PISA assessment: it contributes to the generalisability of claims about overall performance on the test; it allows variation in student performance by domain to be analysed (see Chapter 4); it acknowledges that cultural preferences may exist for certain forms of creative engagement; and it acknowledges that creative work is supported by some degree of domain readiness.

Given the age of PISA test takers and the amount of available testing time, tasks in the PISA 2022 creative thinking test were situated in four different domain contexts:

• written expression, which involves communicating ideas and imagination through written language;

• visual expression, which involves communicating ideas and imagination through a range of different media;

• social problem solving, which involves understanding different perspectives, addressing the needs of others, and finding innovative and functional solutions for the parties involved; and

• scientific problem solving, which involves generating new ideas, designing experiments to probe hypotheses, and developing new methods or inventions to solve problems.

The distribution of items in the test across the four domain contexts is as follows: 12 items in written expression; 4 items in visual expression; 10 items in social problem solving; and 6 items in scientific problem solving (see Table III.4.1 in Chapter 4).

Written expression

In the PISA 2022 Creative Thinking test, students were asked to express their imagination in a variety of written formats. For example, students captioned an image, proposed ideas for a short story, or wrote short dialogues between characters in a movie or comic book.

Sample Unit 1: Illustration Titles

The unit Illustration Titles included two items. In the two items, students were asked to come up with original and diverse titles, respectively, for abstract illustrations.

Illustration Titles: Item 2 (Generate diverse ideas)

The second item in the Illustration Titles unit asked students to write three different titles for an abstract illustration of an oversized book embedded in nature (Figure III.1.3). To achieve full credit on the item, the ideas must all be appropriate and sufficiently different from one another. Box III.1.4 provides coded examples of genuine student responses and describes how ideas for this item would be considered “sufficiently different”.

Figure III 1.3. Illustration Titles: Item 2

Box III 1.4. Illustration Titles: Item-specific coding criteria and example responses

Item 2 (Suggest three different illustration titles)

Figure III.1.4 provides three example student responses for Item 2 of the Illustration Titles unit, in which students were asked to suggest three different titles for a given illustration (Figure III.1.3). Scorers must decide whether to award responses no credit, partial credit or full credit, depending on whether the three ideas are sufficiently different from each other.

Figure III 1.4. Coded examples for item 2 in Illustration Titles

In Example Response A, all three ideas provide a literal description of the illustration and synonyms describe the same idea (the size of the book); this response did not demonstrate skill in generating diverse ideas and was awarded no credit. In Example Response B, the foci of all three ideas reference a different element of the illustration (the book, the trail and the tree). The titles each include adjectives with distinct meanings (perfect, written and lonely) to further differentiate their meaning from each other. This response was awarded full credit. Example Response C includes two ideas that are structured identically (Title 1 and 3) and that focus on an abstract attribute of a story (freedom and power); although the attributes change, they both focus solely and explicitly on the book element of the illustration. The second title also references a story but focuses on the idea of life as a story. The structure of the title is significantly different, and it also implicitly connects to other elements of the illustration (e.g. nature or the path). Example Response C was awarded partial credit for including three appropriate ideas, but only two different ideas.

Sample Unit 2: Robot Story

In the unit Robot Story, students were asked to think of ideas for the plot and dialogue of a short film about an intelligent robot (“Rob”) and a human character (“Leo”). The unit included three items.

Robot Story: Item 1 (Generate diverse ideas)

The first item of the Robot Story unit asked students to write two different story ideas for the film based on a short prompt (see Figure III.1.6). To achieve full credit, students must provide two appropriate ideas that are different from each other. The scoring process is similar to the one described in Box III.1.4. There is no partial credit available for this item as students must provide only two different ideas.

The item-specific criteria outline examples of distinct plot developments: for example, the story might focus on how the robot “Rob” was created; a friendship between the two characters; or the human “Leo” becoming a robot. Stories with similar plots could also achieve full credit if the student sufficiently changed the focus or representation of ideas. For example, the narration of the story might occur from two different perspectives or very different settings or contexts.

Figure III 1.5. Robot Story: Item 1

Sample Unit 3: Space Comic

There were two items in the unit Space Comic. Students had to write a dialogue and suggest titles for a comic strip that shows the Sun and the Earth in conversation with each other.

Space Comic: Item 1 (Generate creative ideas)

The first item of the Space Comic unit asked students to write an original dialogue between the Sun and the Earth (Figure III.1.7). The comic strip includes six empty dialogue boxes in a fixed order that students must fill in. To achieve full credit, students must compose a dialogue with an original theme; conventional (i.e. non-original) themes for this item and example coded responses are described in Box III.1.5. Responses corresponding to conventional themes were awarded partial credit, unless combined with an innovative approach or implementation.

Figure III 1.6. Space Comic: Item 1

Box III 1.5. Space Comic: Item-specific coding criteria and example responses

Item 1 (Create an original dialogue)

Figure III.1.8 provides examples of three student responses to the first item in the Comic Strip unit Figure III.1.6). Scorers must decide whether to award no credit, partial credit or full credit for the response depending on whether the dialogue is original.

The item-specific coding criteria describe two conventional themes for this unit:

• Conventional Theme 1: Dialogue focusing on heat, temperature, weather or seasons (excluding a focus on environmental degradation or global warming);

• Conventional Theme 2: Dialogue focusing on environmental degradation or global warming.

In contrast, original themes included (but were not limited to) the Earth's ability to sustain life, observable or physical aspects of the Earth/Sun (e.g. colour, size, etc.), conversations about love or friendship, or about (other) celestial bodies.

Figure III 1.7. Coded examples for Item 1 in Space Comic

The dialogue in Example Response A focuses on the topic of heat. This response was awarded partial credit because it corresponded to Conventional Theme 1 but did not further develop the theme in an innovative or unconventional way. In Example Response B, the student focuses on the relationship between the Earth and Sun and references their gravitational attraction. This idea was awarded full credit as it corresponded to an original theme. Example Response C focused on seasons (also Conventional Theme 1) but introduced original details about the Sun’s brightness and developed the dialogue in a humorous way; the response was thus awarded full credit.

Sample Unit 4: 2983

The unit 2983 is a single-item unit in which students were asked to think of an original story idea for a book titled “2983” (Figure III.1.9). The item is classified as a “Generate creative ideas” item. Students must associate the number 2983 to a relevant detail in their story idea.

The scoring process is similar to that described in Box III.1.5 for Item 1 of the Space Comic unit. To achieve full credit, the response must correspond to an original theme. Conventional (i.e. non-original) themes included: stories about the future of humanity set in the year 2983; or stories in which the number 2983 identifies a person, a place or an object. Responses that corresponded with conventional themes were awarded partial credit unless combined with an innovative approach or implementation. For example, an unconventional reference to the number 2983 in the story was its use as a code for unlocking a device.

Figure III 1.8. 2983: Item

Visual expression

In the PISA creative thinking test, students created visual compositions from a library of images and shapes using a simple graphic tool. Students were able to resize, rotate and change the colour of shape elements. Students created visual designs for a variety of purposes, such as logos or posters for an event or designs for merchandise.

Sample Unit 5: Science Fair Poster

In the unit Science Fair Poster, students designed and improved posters for their school’s upcoming science fair. Students used a simple drawing tool that includes different shapes, colours and stamps to complete both items in the unit.

Science Fair Poster: Item 1 (Generate creative ideas)

The first item in the Science Fair Poster unit asked students to create an original poster for the science fair that represents the theme “Life in Deep Space” (Figure III.1.10). To achieve full credit, students must create a poster with an original theme. Box III.1.6 describes the conventional (i.e. non-original) themes for this item as well as coded example responses; responses that corresponded to conventional themes were awarded partial credit, unless combined with an innovative approach or implementation.

Figure III 1.9. Science Fair Poster: Item 1

Science Fair Poster: Item 2 (Evaluate and improve ideas)

The second item in the Science Fair Poster unit provides students with a simple poster design (the Sun and one planet) and asks them to improve it by connecting it to the topic of “Life in Deep Space” in an original way (Figure III.1.11). The coding process for this item is similar to that of Item 1: to achieve full credit, students must modify the poster with an original idea. Modifications that corresponded to conventional (i.e. non-original) theme ideas were awarded partial credit, unless combined with an innovative approach or implementation (see Box III.1.6 for coded examples of student responses).

Figure III 1.10. Science Fair Poster: Item 2

Box III 1.6. Science Fair Poster: Item-specific coding criteria and example responses

Item 1 (Design an original poster)

The item-specific coding criteria for Item 1 in the Science Fair Poster unit describes two conventional themes. These themes refer to students’ dominant representation of the idea of “Life in Deep Space”:

• Conventional Theme 1: The Earth;

• Conventional Theme 2: Elements related to human space exploration (e.g. astronauts, spacecraft, satellites).

Original themes included (but were not limited to) the use of text or script elements to communicate the theme, the inclusion of animate figures (e.g. humans or aliens) other than astronauts, and scientific models or notations related to life (e.g. molecules).

Figure III.1.12 provides examples of coded student responses for this item. Example Response A represents the idea of “Life in Deep Space” through two stickers: an astronaut and a spacecraft. Since the elements of the poster correspond to Conventional Theme 2, the response is awarded partial credit. Example Response B displays a molecule, created through combining shapes, and which the student has clarified to be a carbon molecule in the poster description (carbon is the most common element to all known life on Earth). The response connects to the science fair and does not correspond to one of the two conventional themes; it is thus considered original and awarded full credit. Like Example Response A, Example Response C also represents “Life in Deep Space” through an astronaut and spaceship. However, in Example Response C, the student used different shapes to create a spaceship (rather than using the sticker) and has attached the astronaut to the spaceship as if conducting a moonwalk. This is an innovative implementation of Conventional Theme 2 and is thus awarded full credit.

Figure III 1.11. Coded examples for Item 1 in Science Fair Poster

Item 2 (Modify a poster in an original way)

For Item 2 of the Science Fair Poster, the item-specific coding criteria describe three conventional themes. These themes refer to elements that students must add to connect the existing poster to the idea of “Life in Deep Space”. In addition to the two conventional themes that constituted the coding criteria for Item 1 (the Earth, and human space exploration), the item-specific coding criteria for Item 2 included a third conventional theme:

• Conventional Theme 3: The use of plants or flora as the dominant representation of life.

Figure III.1.13 provides examples of coded student responses for Item 2. Example Response A does not connect to the Science Fair: concentric circle shapes have been added but with no clear association to the theme of “Life in Deep Space” (nor is there any clarification in the description provided by the student). The response does not achieve any credit. In Example Response B, two simple stickers of the Earth and the moon have been added. The response is awarded partial credit as it corresponds to Conventional Theme 1. While Example Response C also uses the Earth sticker to connect the poster to “Life in Deep Space”, the student also uses shapes to modify the Earth and add animate details to its surface (sunglasses and a mouth). The response integrates an innovative approach and thus receives full credit.

Figure III 1.12. Coded examples for Item 2 in Science Fair Poster

Social problem solving

Social problem solving can range from the small-scale, personal and interpersonal problems of individuals to wider school, community or even global problems. In the PISA creative thinking test, students suggested solutions for open problems that focused on issues affecting different groups within society (e.g. wheelchair users) or affecting society at large (e.g. the collection and use of waste materials).

Sample Unit 6: Library Accessibility

In the unit Library Accessibility, students were asked to consider creative ways to address the accessibility of a library for wheelchair users (a community problem). The unit involved two items.

Library Accessibility: Item 1 (Generate diverse ideas)

The first item of the unit Library Accessibility asks students to think of three different ideas for improving the wheelchair accessibility of a library (Figure III.1.14). The coding guide provided scorers with a non-exhaustive list of idea categories and sub-categories to classify whether ideas are fundamentally different from one another (see Box III.1.7). To achieve full credit, students had to provide three appropriate ideas that are sufficiently different; if students provided only two different ideas, then their response achieved partial credit.

Figure III 1.13. Library Accessibility: Item 1

Library Accessibility: Item 2 (Evaluate and improve ideas)

In the second item of the unit Library Accessibility, students were presented with an idea to install ramps in the library. They were asked to suggest an original modification or feature for the ramp that would further enhance the ability of wheelchair users to access books in the library (Figure III.1.15). To achieve full credit, the response had to correspond to an original improvement theme. Responses that corresponded with conventional themes were awarded partial credit, unless combined with an innovative approach or implementation. Box III.1.7 describes the conventional themes for this item, as well as coded example responses.

Figure III 1.14. Library Accessibility: Item 2

Box III 1.7. Library Accessibility: Item-specific coding criteria and example responses

Item 1 (Suggest three ideas to address the accessibility of a library building)

In general, for the social problem-solving and scientific problem-solving items, the coding guide provided scorers with guidelines for determining whether student ideas were “sufficiently different”. As items in the two problem-solving domains had a more constrained solution space than items in the written or visual expression, it was possible to provide scorers with a non-exhaustive list of idea category and sub-category groupings. Typically, category groupings differentiated ideas by their main approach or focus while the sub-categories differentiated ideas within the same larger category by their means of implementation.

The item-specific criteria for Item 1 in the Library Accessibility unit described the following categories of ideas (sub-categories in parentheses):

• Category 1 – Physical modifications to the library (e.g. integrating ramps, elevators, etc.);

• Category 2 – Providing human assistance to wheelchair users (e.g. staff or volunteers deliver library materials or bring customers to the materials);

• Category 3 – Providing technological assistance mechanisms (e.g. aid with retrieving materials, guiding customers, or requesting deliveries).

Figure III 1.15. Coded examples for Item 1 in Library Accessibility

Figure III.1.15 shows three example responses for this item. All three ideas in Example Response A suggest hiring more staff for the library (Category 2) without further detail that could be considered evidence of a distinct focus or method of implementation. This response does not demonstrate skill in generating diverse ideas and was awarded no credit. In Example Response B, Ideas 1 and 2 both propose physical modifications to the library building (Category 1), effectively installing an elevator in both cases. The third idea of the response refers to integrating some technological assistance mechanisms (Category 3) and therefore displays a different focus. With two similar ideas and a third different idea, the response was awarded partial credit. In Example C, while all three ideas focus on providing human assistance (Category 2), each idea proposes a different method of implementation to assist the wheelchair users. They therefore correspond to different sub-categories and the response was awarded full credit.

Item 2 (Suggest an original modification to an existing solution)

The item-specific coding criteria for Item 2 in the Library Accessibility unit describes two conventional (i.e. non-original) themes. These themes include:

• Conventional Theme 1: Automating the ramp using a conveyer belt mechanism;

• Conventional Theme 2: Automating the ramp in other ways to move people (e.g. push/pull “on-demand” mechanisms, or mobile ramps).

Original themes included (but were not limited to) modifying the ramp’s gradient, adding a braking mechanism or an anti-slip surface to the floor of the ramp, adding extra lanes or adjusting the width of the ramp, or using the ramp as a bookshelf.

Figure III.1.17 provides examples of coded student responses for this item. Response A clearly corresponds to Theme 1 without adding further detail that could be considered an innovative approach or implementation; it was thus awarded partial credit. In Example Response B, the focus of the idea (adding an anti-slip surface) did not correspond to any of the conventional themes and was awarded full credit. Response C also corresponded to Theme 1 but introduced an original tool (voice automation) to facilitate the automation of the ramp. The response was awarded full credit.

Figure III 1.16. Coded examples for Item 2 in Library Accessibility

Sample Unit 7: Save the Bees

In the unit Save the Bees, students were asked to help the “Save the Bees” club at their school conduct an awareness-raising campaign focused on bees’ ecological importance. The unit includes three items in total.

Save the Bees: Item 2 (Generate creative ideas)

In the first item of the Save the Bees unit, students were asked to suggest three different ideas to raise awareness about the importance of bees; in the second item of the unit, students must suggest one original idea to achieve this goal (Figure III.1.18). Students could provide a completely new idea or choose one of the ideas they provided in the previous item.

Like all “generate creative ideas” items, the response must correspond to an original theme to achieve full credit. Conventional themes for this item included: efforts to amplify the verbal communication of club members, the creation of informative visual materials, or organising the observation of live bees. Responses that corresponded with conventional themes were awarded partial credit unless combined with an innovative approach or implementation.

Figure III 1.17. Save the Bees: Item 2

Sample Unit 8: Carpooling

The unit Carpooling is a single-item unit in which students must think of an original idea to further incentivise carpooling (Figure III.1.19). The item is classified as an “evaluate and improve ideas” item because granting discounts on fuel or tolls are existing incentives that need to be further strengthened. To achieve full credit, the response must correspond to an original idea theme. For this item, there is only one conventional (i.e. non-original) theme: introducing additional financial incentives, for example making the shared purchase of cars more affordable. Responses that corresponded to the conventional theme were awarded partial credit unless combined with an innovative approach or implementation.

Figure III 1.18. Carpooling: Item

Scientific problem solving

In the PISA creative thinking test, students investigated open scientific or engineering problems. Although creative thinking in scientific contexts is related to scientific inquiry, the tasks in this domain context differed fundamentally from the PISA scientific literacy tasks – in the creative thinking test, students were asked to generate multiple ideas or solutions, or an original idea or solution, for an open problem with no pre-defined “correct” response. For example, in a task asking students to think of explanations for a given phenomenon, they would be rewarded for proposing multiple plausible ideas regardless of whether these constituted the right explanation.

Sample Unit 9: Save the River

In the unit Save the River, students were asked to think creatively about a problem related to frogs in a local river. The two items in the unit focus on finding and verifying ideas about the cause of the problem.

Save the River: Item 1 (Generate diverse ideas)

The first item in the Save the River unit describes the problem to students – a declining frog population in a part of the river downstream from the city compared to the rest of the river – and asks them to provide two different, testable ideas for possible causes (Figure III.1.20). Students were explicitly instructed to think of causes other than pollution. Students could only achieve full credit or no credit for this item, as it required only two different ideas. The item-specific coding criteria provided several different possible causes of the problem (see Box III.1.8).

Figure III 1.19. Save the River: Item 1

Save the River: Item 2 (Evaluate and improve ideas)

The second item of the Save the River unit asks students to improve a proposed experiment aiming to test whether pollution is the cause of the problem with the declining frog population (Figure III.1.21). To achieve full credit, the response must correspond to an original improvement theme; conventional (i.e. non-original) themes and coded examples for this item are described in Box III.1.8. Responses that corresponded with conventional themes were awarded partial credit, unless combined with an innovative approach or implementation.

Figure III 1.20. Save the River: Item 2

Box III 1.8. Save the River: Item-specific coding criteria and example responses

Item 1 (Suggest two different ideas unrelated to pollution)

The item-specific coding criteria for Item 1 of the Save the River unit provides coders with guidelines on “sufficiently different” ideas. Ideas are classed into different categories and sub-categories based on their main focus and method of implementation. Among the possible different categories of ideas are:

• Category 1 - Changes to the water habitat (e.g. colder or warmer temperature, changes in oxygen or mineral levels, etc.);

• Category 2 – Changes to the surrounding fauna (e.g. a localised predator, lack of food);

• Category 3 – Changes to the local flora (e.g. a new invasive plant species, or absence of important flora);

• Category 4 – Changes to the frogs themselves (e.g. infection, disease, or mutation);

• Category 5 – Changes to the behaviour or activities of humans in the area (e.g. noise, ground vibrations, or humans capturing frogs).

This list of idea categories and sub-categories is not exhaustive, but intended to provide coders with informative guidelines to help determine whether the two ideas proposed by students are “sufficiently different”.

Item 2 (Suggest an original way to improve the experiment)

The item-specific coding criteria for Item 2 describes three conventional (i.e. non-original) themes for improving the given experiment idea (Figure III.1.20). These are:

• Conventional Theme 1: Providing more specific information about ways to test the water for chemicals or pollution;

• Conventional Theme 2: Testing the frogs for chemicals;

• Conventional Theme 3: Including a control measure in the experiment (e.g. comparing results to an unaffected group of frogs).

Original themes included (but were not limited to) conducting additional tests to rule out changes or anomalies in the frogs, to rule out environmental changes or anomalies, or to focus enquiries on identifying the chemicals that farms or factories are emitting.

Figure III.1.22 provides examples of coded student responses for this item. Response A suggests testing water from a different source as a type of control measure. It corresponds to Conventional Theme 3 without including any further information about how to test the water that could be considered evidence of an innovative approach or implementation. The response was thus awarded partial credit. Response B refers to also investigating the presence of invasive species that might be an alternative cause of the problem. It is an example of an original experiment improvement and was awarded full credit.

Figure III 1.21. Coded examples for Item 2 in Save the River

Reporting student proficiency in creative thinking

Like all PISA scales, student scores on the creative thinking test are summarised according to a unidimensional scale that estimates their overall creative thinking proficiency. However, the creative thinking scale for PISA 2022 has been constructed differently: this scale has been constructed as a bounded scale between 0 and 60 score points. The maximum sum-score of 60 points represents the total number of points available in a hypothetical test containing all 32 items within the creative thinking test-item pool. Student scores on the creative thinking scale can therefore be interpreted in terms of their estimated score (i.e. the sum of their partial and full credit responses) if they were to sit a test containing all 32 items in the test-item pool.

This two-digit scale addresses the relatively lower measurement precision of the creative thinking test compared to the PISA assessments of mathematics, reading and science, given the smaller number of items in the creative thinking test-item pool. A 1-point change in the creative thinking scale signals about 10% of a standard deviation of proficiency. This approach to scaling the PISA creative thinking data also means that results will be more sensitive to performance differences where there is more information available about students’ performance in the test. For more information on the construction of the creative thinking scale and its supporting rationale, see Annex A3 or Chapter 18 of the PISA 2022 Technical Report (OECD, 2023[13]).

Creative thinking proficiency levels

To help interpret what student scores mean on the creative thinking scale in substantive terms, the scale is divided into seven proficiency levels. Six levels are described based on the skills needed to successfully complete the tasks that are located within them; the seventh level refers to students who perform below Level 1. Level 1 is the lowest described level and Level 6 corresponds to the highest described level of creative thinking skills.

Table III.1.1 describes the six proficiency levels in detail and shows the OECD average percentage of students at or above each proficiency level.4

Mapping of select sample items to the creative thinking proficiency levels

The difficulty of each item in the PISA assessment, at both partial credit (where available) and full credit, can be located on the same scale as the proficiency levels (OECD, 2023[14]). This mapping of items to a value on the scale is based on response probabilities.5 The sample items described earlier in this chapter provide information about students across the entire range of the creative thinking scale. A selection of these sample items, at different credit thresholds (partial or full credit), have been mapped to each of the six described proficiency levels of creative thinking. Table III.1.2 presents this mapping, along with a brief description of the nature of the task at a given credit threshold and its drivers of difficulty (see Annex C for the technical information on each of the released items).

While the difficulty of the creative thinking items is established empirically based on response probability data at the international level, a combination of factors is likely to affect the difficulty of tasks in the creative thinking test. These include the familiarity of the item content to students, the task demands (e.g. generate two or three ideas), the task constraints (e.g. how open or closed the "solution space" is), the response type (e.g. a single word answer or an elaborated story idea), and the item-specific coding criteria (e.g. how many themes are designated as "conventional", or the scope of each theme/category). In general, tasks that require shorter response types and that focus on more familiar task contexts with an open solution space (i.e. with many possibilities and few appropriateness constraints) tend to be easier for students to demonstrate creative thinking.

Table III 1.1. Description of the six levels of proficiency in creative thinking

Source: OECD, PISA 2022 Database, Table III.B1.2.2. The StatLink URL of this table is available at the end of the chapter.

Table III 1.2. Mapping of select creative thinking items to the proficiency levels

Source: OECD, PISA 2022 Database. The StatLink URL of this table is available at the end of the chapter.

Table III 1.3. Measuring creative thinking in PISA: Chapter 1 figures and tables

Figure III.1.1	Creativity in curricula worldwide
Table III.1.1	Description of the six levels of creative thinking proficiency in PISA 2022
Table III.1.2	Mapping of select creative thinking items to the proficiency levels

Statlink

References

[11] Baer, J. (2011), “Domains of creativity”, in Runco, M. and S. Pritzker (eds.), Encyclopedia of Creativity (Second Edition), Elsevier Inc.

[12] Baer, J. and J. Kaufman (2005), “Bridging generality and specificity: The amusement park theoretical (apt) model of creativity”, Roeper Review, https://doi.org/10.1080/02783190509554310.

[28] Barbot, B. and B. Heuser (2017), “Creativity and identity formation in adolescence: A developmental perspective”, in Karwowski, M. and J. Kaufman (eds.), The Creative Self: Effect of Beliefs, Self-Efficacy, Mindset, and Identity, Elsevier, https://www.sciencedirect.com/science/article/abs/pii/B9780128097908000054?via%3Dihub.

[29] Barnes, J. (2016), “Creativity and promoting wellbeing in children and young people through education”, in Clift, S. and P. Camic (eds.), Oxford Textbook of Creative Arts, Health, and Wellbeing: International Perspectives on Practice, Policy and Research, Oxford University Press, Oxford.

[27] Beghetto, R. and J. Plucker (2006), “The relationship among schooling, learning, and creativity: “All roads lead to creativity” or “You can’t get there from here“?”, in Kaufman, J. and J. Baer (eds.), Creativity and Reason in Cognitive Development, Cambridge University Press, Cambridge, https://doi.org/10.1017/CBO9780511606915.019.

[16] Bungay, H. and T. Vella-Burrows (2013), “The effects of participating in creative activities on the health and well-being of children and young people: a rapid review of the literature.”, Perspectives in Public Health, Vol. 133/1, pp. 44-52, https://doi.org/10.1177/1757913912466946.

[22] Clarke, T. and M. Basilio (2018), “Do arts subjects matter for secondary school students’ wellbeing? The role of creative engagement and playfulness”, Thinking Skills and Creativity, Vol. 29, pp. 97-114, https://doi.org/10.1016/j.tsc.2018.06.005.

[23] Connor, T., C. DeYoung and P. Silvia (2018), “Everyday creative activity as a path to flourishing”, The Journal of Positive Psychology, Vol. 13, pp. 181-189, https://doi.org/10.1080/17439760.2016.1257049.

[10] Cropley, A. (2006), “In praise of convergent thinking”, Creativity Research Journal, Vol. 18/3, pp. 391-404, https://doi.org/10.1207/s15326934crj1803_13.

[3] Csikszentmihalyi, M. (2013), Creativity: The Psychology of Discovery and Invention, Harper Collins, New York.

[18] European Commission (2019), Key Competences for Lifelong Learning, Publications Office of the European Union, Luxembourg, https://data.europa.eu/doi/10.2766/569540.

[30] Fadel, C. and J. Groff (2018), “Four-dimensional education for sustainable societies”, in Cook, J. (ed.), Sustainability, Human Well-Being, and the Future of Education, Springer International Publishing, Cham, https://doi.org/10.1007/978-3-319-78580-6_8.

[24] Gajda, A., M. Karwowski and R. Beghetto (2017), “Creativity and academic achievement: A meta-analysis”, Journal of Educational Psychology, Vol. 109/2, pp. 269-299, https://doi.org/10.1037/edu0000133.

[21] Griffin, P., B. McGaw and E. Care (eds.) (2011), Defining twenty-first century skills, Springer Netherlands, Dordrecht, https://doi.org/10.1007/978-94-007-2324-5_2.

[7] Guilford, J. (1956), “The structure of intellect”, Psychological Bulletin, Vol. 53/4, pp. 267-293, https://doi.org/10.1037/h0040755.

[9] Guilford, J. (1950), “Creativity”, American Psychologist, Vol. 5/9, pp. 444-454, https://doi.org/10.1037/h0063487.

[15] Higgins, S. et al. (2005), “A meta-analysis of the impact of the implementation of thinking skills approaches”, Research Evidence in Education Library, https://eppi.ioe.ac.uk/cms/Portals/0/PDF%20reviews%20and%20summaries/t_s_rv2.pdf?ver=2006-03-02-125128-393.

[5] Kaufman, J. and R. Beghetto (2009), “Beyond Big and Little: The Four C model of creativity”, Review of General Psychology, Vol. 13/1, pp. 1-12, https://doi.org/10.1037/a0013688.

[13] OECD (2023), PISA 2022 Innovative Domain Test Design and Test Development, OECD Publishing, Paris, https://www.oecd.org/pisa/data/pisa2022technicalreport/PISA-2022-Technical-Report-Ch-18-PISA-Proficiency-Scale-Construction-Domains-Creative-Thinking.pdf.

[14] OECD (2023), Scaling Outcomes, OECD Publishing, Paris, https://www.oecd.org/pisa/data/pisa2022technicalreport/PISA-2022-Technical-Report-Ch-14-Scaling-Outcomes.pdf.

[2] OECD (2023), Supporting Students to Think Creatively: What Education Policy Can Do, https://issuu.com/oecd.publishing/docs/supporting_students_to_think_creatively_web_1_ (accessed on 28 March 2024).

[6] OECD (2022), Thinking Outside the Box: The PISA 2022 Creative Thinking Assessment, https://issuu.com/oecd.publishing/docs/thinking-outside-the-box (accessed on 4 March 2023).

[1] OECD (2018), The Future of Education and Skills 2030, OECD Publishing, Paris, https://www.oecd.org/education/2030-project/about/documents/E2030%20Position%20Paper%20(05.04.2018).pdf (accessed on 1 March 2023).

[26] Pellegrino, J. and M. Hilton (2012), Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century, National Academies Press, Washington, D.C., https://doi.org/10.17226/13398.

[8] Runco, M. and S. Acar (2012), “Divergent thinking as an indicator of creative potential”, Creativity Research Journal, Vol. 24/1, pp. 66-75, https://doi.org/10.1080/10400419.2012.652929.

[19] Scott, C. (2015), The futures of learning 2: What kind of learning for the 21st century?, UNESCO, https://unesdoc.unesco.org/ark:/48223/pf0000242996.

[4] Simonton, D. (2013), “What is a creative idea? Little-c versus Big-C creativity”, in Thomas, K. and J. Chan (eds.), Handbook of Research on Creativity, Edward Elgar Publishing, Cheltenham, https://doi.org/10.4337/9780857939814.00015.

[17] Spencer, E. and B. Lucas (2018), Understanding the role of creative self-efficacy in youth social action: A literature review, Centre for Real-World Learning, University of Winchester, Winchester, https://www.thersa.org/globalassets/pdfs/reports/teenagency-literature-review.pdf.

[25] Tamannaeifar, M. and M. Motaghedifard (2014), “Subjective well-being and its sub-scales among students: The study of the role of creativity and self-efficacy”, Thinking Skills and Creativity, Vol. 12, pp. 37-42, https://doi.org/10.1016/j.tsc.2013.12.003.

[20] World Economic Forum (2015), New Vision for Education: Unlocking the Potential of Technology, https://www3.weforum.org/docs/WEFUSA_NewVisionforEducation_Report2015.pdf.

Notes

← 1.  Many international frameworks on the future of education and skills identify creativity, creative thinking and innovation as among the most important skills that students need to develop (Binkley et al., 2011[21]; European Commission, 2019[18]; Fadel and Groff, 2018[30]; OECD, 2018[1]; Pellegrino and Hilton, 2012[26]; Scott, 2015[19]; World Economic Forum, 2015[20]).

← 2.  Various studies or research papers have focused on how creativity and creative thinking support students’ skill and personal development, for example identity formation (Barbot and Heuser, 2017[28]), academic achievement (Gajda, Karwowski and Beghetto, 2017[24]; Higgins et al., 2005[15]) and learning (Beghetto and Plucker, 2006[27]), various aspects of subjective well-being (Barnes, 2016[29]; Clarke and Basilio, 2018[22]; Connor, DeYoung and Silvia, 2018[23]; Tamannaeifar and Motaghedifard, 2014[25]) and physical well-being (Bungay and Vella-Burrows, 2013[16]), and social engagement (Spencer and Lucas, 2018[17]), amongst other things.

← 3.  In the PISA 2022 system-level survey, countries and economies were asked to report: i) how creativity is referenced within their jurisdiction’s curriculum or standards for both primary and secondary education (i.e. as a priority cross-cutting theme or competency, within the broader umbrella of 21st century competencies, within subject-specific contexts, or not at all); and ii) the specific subject areas in which curricula or standards reference creativity. For the purposes of the PISA 2022 system-level survey, “creativity” was understood to include any of the following terms: creative thinking, creative problem solving, and innovation. In contrast, “creativity” was not understood to include the terms entrepreneurship, critical thinking or collaboration.

← 4.  Students with a proficiency score within the range of Level 1 are expected to complete most Level 1 tasks successfully, but are unlikely to be able to complete tasks at higher levels; students with scores in the Level 6 range are likely to be able to successfully complete all tasks included in the PISA 2022 assessment of creative thinking.

← 5.  Response probabilities for a given item are calculated using the item’s international IRT parameters (discrimination and difficulty). Historically in PISA, a response probability of 0.62 (RP62) has been used to classify items into levels. Students with a proficiency located at or below this point have a probability of 0.62 or less of getting the item correct, while students with a proficiency above this point have a higher probability of getting the item correct higher than 0.62. Note that for polytomous items, the RP62 value is provided for partial credit as well as full credit responses. The partial credit RP62 has been defined as the minimum proficiency level a student needs to have an expected score that is 62% of the full credit. More information can be found in Chapter 14 of the PISA 2022 Technical Report (OECD, 2023[14]).