PISA 2022 Results (Volume III)

Annex A1. PISA 2022 Creative Thinking Framework

Defining the construct for the PISA 2022 assessment

The PISA 2022 definition of creative thinking

While closely related to the broader construct of creativity, creative thinking refers to the cognitive processes required to engage in creative work. It is a more appropriate construct to assess in the context of PISA as it is a malleable individual capacity that can be developed through practice and does not place an emphasis on how wider society values the resulting output. PISA defines creative thinking as “the competence to engage productively in the generation, evaluation and improvement of ideas that can result in original and effective solutions, advances in knowledge and impactful expressions of imagination”.

The PISA definition of creative thinking is aligned with the cognitive processes and outcomes associated with “little-c” creativity – in other words, it reflects the types of creative thinking that 15-year-old students around the world can reasonably demonstrate in “everyday” contexts. It emphasises that students need to learn to engage productively in generating ideas, reflecting upon ideas by valuing their relevance and novelty, and iterating upon ideas before reaching a satisfactory outcome. This definition of creative thinking applies to learning contexts that require imagination and the expression of one’s inner world, such as creative writing or the arts, as well as contexts in which generating ideas is functional to the investigation of problems or phenomena.

Unpacking creative thinking in the classroom

Confluent approaches to creativity emphasise that both “internal” and “external” resources are needed to successfully engage in creative work. To better understand children’s creative thinking and define what information is important to collect in the PISA assessment, it is necessary to contextualise these approaches in a way that is relevant to students in their everyday school life (Glaveanu et al., 2013[1]; Tanggaard, 2014[2]). This section describes what creative thinking in the classroom looks like and the interconnected internal and external factors that can promote or hinder it.

Schools can influence many of the internal resources students need to engage in creative thinking. Internal resources here essentially refer to the set of knowledge, skills and attitudes that enable creative thinking. These include: 1) cognitive skills; 2) domain readiness (i.e. domain-specific knowledge and experience); 3) openness to new ideas and experiences; 4) goal orientation and self-belief; 5) task motivation; and in some cases, 6) collaborative skills. In terms of external factors, features of students’ environments can also incentivise or hinder their capacity to engage in creative thinking. These include the classroom culture, the educational approach of schools and wider education systems, and broader cultural norms and expectations.

Schools are also important places in which students can think creatively, either as individuals or as part of a group, and where they can produce creative work. Creative achievement and progress in the classroom can take many forms, such as creative expression (communicating one’s thoughts and imagination through various media), knowledge creation (advancing knowledge and understanding through inquiry), or creative problem solving. Figure III.A1.1. summarises these elements that, together, define creative thinking in the classroom. The three sets of elements (internal resources, external factors, and creative achievement and progress) are strongly interconnected. For example, external factors include cultural norms and expectations, which in turn influence how students’ internal resources are developed and honed as well as the types of creative work that students might choose to produce. Each of the elements in Figure III.A1.1 are described in further detail in the following section.

Figure III A1.1. Unpacking creative thinking in the classroom: Internal resources, external factors and types of creative engagement

Internal resources

Cognitive skills

Both convergent and divergent thinking (Guilford, 1956[3]) are widely recognised as important skills for creative thinking. Convergent thinking refers to the ability to apply conventional and logical reasoning to information (Cropley, 2006[4]). As such, convergent thinking aids in understanding the problem space and identifying and evaluating good ideas (Reiter-Palmon and Robinson, 2009[5]; Runco, 1997[6]). By contrast, divergent thinking refers to the ability to think of original ideas, to make flexible connections between ideas or pieces of information, and to apply fluency of association and ideation (Cropley, 2006[4]). It also refers to the ability to break out of “fixed” performance scripts – in other words, to try new approaches, to look at problems from different angles, and to discover new methods of “doing” (Schank and Abelson, 1977[7]; Duncker, 1972[8]). In essence, divergent thinking brings forth novel, unusual or surprising ideas.

Creative thinking is often described in terms of divergent thinking, and most assessments to-date have focused on measuring divergent thinking cognitive processes. However, convergent thinking cognitive processes are also important for engaging in creative work. For example, Getzels and Csikszentmihalyi (1976[9]) found that art students’ success in “problem construction” was strongly correlated with measures of the originality and aesthetic value of their resulting paintings, and that these measures were also linked to long-term artistic success.

Domain readiness

Domain readiness conveys the idea that some prior domain knowledge and experience is needed to successfully produce creative work (Baer, 2016[10]). A better understanding of a domain is more likely to help with generating and evaluating ideas that are both novel and useful (Hatano and Inagaki, 1986[11]; Schwartz, 2005[12]). However, this relationship may not be strictly linear – well-established routines for deploying knowledge or skills within a domain may also result in idea fixation and a reluctance to think beyond those established routines.

Openness to experience and to intellect

Several studies have shown that creative people share a core set of tendencies, particularly the trait of “openness”, part of the “Big Five” personality dimensions (Kaufman et al., 2010[13]; 2016[14]; McCrae, 1987[15]; Prabhu, Sutton and Sauser, 2008[16]; Werner et al., 2014[17]). In general, such empirical studies examining the personality and behaviour of creative individuals have typically employed questionnaire instruments that operationalise creativity as a relatively enduring and stable personality trait (Hennessey and Amabile, 2010[18]). Meta-analyses of studies on creativity and personality have also found that openness appears to be a common trait in creative achievers across domains, whereas other personality traits tend to interact with creativity only insofar as they benefit individuals within specific domains (e.g. “conscientiousness” seems to enhance scientific creativity but detract from performance in the arts) (Batey and Furnham, 2006[19]; Feist, 1998[20]).

Both “openness to experience” and “openness to intellect” are included under the broader openness trait. “Openness to experience” describes an individual’s receptivity to engage with novel ideas, imagination and fantasy (Berzonsky and Sullivan, 1992[21]). Its predictive value for creative achievement across domains is likely due to its inclusion of cognitive (e.g. imagination), affective (e.g. curiosity) and behavioural aspects (e.g. adventurousness), and the links between curiosity and creativity have been further supported by several researchers (Chávez-Eakle, 2009[22]; Feist, 1998[20]; Guastello, 2008[23]; Kashdan and Fincham, 2002[24]).

“Openness to intellect” describes an individual’s receptivity to appreciate and engage with abstract and complex information, primarily through reasoning (DeYoung, 2015[25]). In contrast to “openness to experience”, which is particularly correlated with artistic creativity, the trait “openness to intellect” seems particularly correlated with scientific creativity (Kaufman et al., 2016[14]).

Goal orientation and creative self-beliefs

Persistence, perseverance and creative self-efficacy influence creative thinking by providing a strong sense of goal orientation and the belief that creative goals can be achieved. Investing effort towards one’s goal and overcoming difficulty are essential for engaging in creative thinking, as they enable individuals to maintain concentration for long periods and deal with frustrations that arise (Cropley, 1990[26]; Torrance, 1988[27]; Amabile, 1983[28]).

Related to goal orientation is creative self-efficacy, which describes an individual’s belief that they are capable of successfully producing creative work (Beghetto and Karwowski, 2017[29]). Researchers consider creative self-efficacy essential in determining whether an individual will sustain effort towards their goals in the face of resistance and ultimately succeed in performing tasks creatively (Bandura, 1997[30]). These beliefs can in turn be influenced by one’s prior experience and performance history, mood and environment (Bandura, 1997[30]; Beghetto, 2006[31]).

Task motivation

The role of task motivation as a driver of creative work has been well documented, namely in the works of Teresa Amabile (1997[32]; 2016[33]; 2010[18]; 1983[28]). The basic notion is that, as with any task, an individual will not produce creative work unless they are sufficiently motivated to do so. This motivation can be both intrinsic and extrinsic.

Intrinsic task motivation drives individuals who find their work inherently meaningful or rewarding, for reasons such as enjoyment, self-interest or a desire to be challenged. This type of task engagement is relatively insensitive to incentives or other external pressures. The experience of “creative flow” – being fully immersed in a task and disregarding other needs – is a powerful driver of creativity because individuals in flow are intrinsically motivated to engage in a task (Csikszentmihalyi, 1996[34]; Nakamura and Csikszentmihalyi, 2002[35]).

On the other hand, extrinsic task motivation refers to external incentives, goals or pressures that motivate people to engage in a particular task. Although research emphasises the importance of intrinsic task motivation in creative performance, extrinsic motivators such as deadlines or recognition can also motivate people to persist in their creative endeavours (Eisenberger and Shanock, 2003[36]; Amabile and Pratt, 2016[33]).

Collaborative engagement

Creative work often results from interactions between individuals and their environment – including interactions with others. Research has also increasingly examined creative thinking as a collective endeavour, for example by examining the actions and benefits of teams in generating new knowledge (Thompson and Choi, 2006[37]; Prather, 2010[38]; Grivas and Puccio, 2011[39]; Scardamalia, 2002[40]). Collaboration can help individuals to explore and build upon the ideas of others as well as improve weaknesses in ideas. This can drive forward knowledge creation by facilitating the development of solutions to complex problems that are beyond the capabilities of any one person (Warhuus et al., 2017[41]).

External factors

Cultural norms and expectations

Creative work is embedded within social contexts that are inherently shaped by cultural norms and expectations. Cultural norms and expectations can influence the skills that individuals develop, the values that shape personality development, and the differences in performance expectations within societies (Niu and Sternberg, 2003[42]; Wong and Niu, 2013[43]; Lubart, 1998[44]). Some studies have investigated how cultural differences affect national measures of creativity and innovation, concluding that differences along the individualism-collectivism spectrum can significantly shape how creative work is defined and valued (Rinne, Steel and Fairweather, 2013[45]; Ng, 2003[46]).

Educational approaches

Cultural norms affect educational approaches, in particular the outcomes an education system values for its students and the content it prioritises in the curriculum. In some cases, these approaches might actively hinder creative thinking and achievement at school (Wong and Niu, 2013[43]). For example, the pressures of standardisation and accountability in educational testing systems often reduce opportunities for creative thinking in schoolwork (DeCoker, 2000[47]). Some have even claimed that increasingly narrow educational approaches and assessment methods are at the root of a “creaticide”, i.e. death of creativity, affecting today’s young people (Berliner, 2012[48]). Schools and educational systems therefore play an important role in combatting this effect and should seek to implement policies and practices that increase the opportunities and rewards for producing creative work.

Classroom climate

Beyond broader cultural norms and educational systems, certain classroom practices can also stifle creative thinking – for example, perpetuating the idea that there is only one correct way to learn or solve problems, cultivating attitudes of submission and fear of authority, promoting beliefs that originality is a rare quality, or discouraging students’ curiosity and inquisitiveness (Nickerson, 2010[49]). Conversely, findings from organisational research have demonstrated that informal feedback, goal setting, teamwork, task autonomy, and appropriate recognition and encouragement to develop new ideas are all important enablers of creative thinking (Amabile, 2012[50]; Zhou and Su, 2010[51]).

Teachers’ beliefs about creativity are also important – they need to value creative work and consider it a fundamental skill that should be developed in the classroom in order to promote creative thinking. Teachers can actively cultivate an environment that helps students learn when creative thinking is appropriate and how to take charge of their own creativity – for example, by encouraging students to set their own goals, identify promising ideas, and take responsibility for contributing to creative teamwork (Beghetto and Kaufman, 2010[52]; 2014[53]). Employing “questions of wonderment” – or encouraging students to try to understand the world and put forth their ideas about different phenomena – can also help to promote knowledge creation in the classroom (Bereiter and Scardamalia, 2010[54]). These approaches are all supported by teachers’ beliefs that creative thinking is something that can be developed in the classroom, even if this development takes time.

Creative engagement

Creative products are both novel and useful, as defined within a particular social context. Examining the outputs of students’ creative work can provide indicators of their capacity to think creatively, particularly in tasks where much of the creative thinking process is not visible (Amabile, 1996[55]; Kaufman and Baer, 2012[56]). Students can produce different kinds of “everyday” creative work at school, either as individuals or as part of a group. These forms of creative work in the classroom are multi-disciplinary and extend beyond traditional subjects.

Creative expression

Creative expression refers to both verbal and non-verbal forms of creative engagement where individuals communicate their thoughts, emotions and imagination to others. Verbal expression involves the use of language, including both written and oral communication, whereas non-verbal expression includes drawing, painting, modelling, music, and physical movement and performance.

Knowledge creation

Knowledge creation refers to the advancement of knowledge and understanding, with a focus on making progress rather than achievement, e.g. improving an idea rather than achieving the optimal solution or complete understanding. Knowledge creation refers not only to important discoveries or advances but also to the purposeful act of building upon and iterating on ideas, which can happen at all levels of society and across all knowledge domains (Scardamalia and Bereiter, 1999[57]).

Creative problem solving

Not all cases of problem solving require creative thinking: creative problem solving is a distinct class of problem solving characterised by novelty, unconventionality, persistence and difficulty during problem formulation (Newell, Shaw and Simon, 1962[58]). Creative thinking becomes particularly necessary when students are challenged to solve problems outside of their realm of expertise and where the techniques with which they are familiar do not work (Nickerson, 1998[59]).

Measuring creative thinking in the PISA test: Task design and scoring approach

The competency model of creative thinking

The competency model is composed of three distinct ideation processes (or facets) for measurement purposes in the PISA 2022 test. These three facets are: 1) generate diverse ideas; 2) generate creative ideas; and 3) evaluate and improve ideas. These reflect the PISA definition of creative thinking and encompass the cognitive skills required for creative thinking in the classroom. The competency model incorporates both divergent cognitive processes (the ability to generate diverse ideas and to generate creative ideas) and convergent cognitive processes (the ability to evaluate other people's ideas and identify improvements to those ideas).

“Ideas” in the context of the PISA assessment can take many forms. The test units provide a meaningful context and sufficiently open tasks in which students can demonstrate their capacity to produce different ideas and think outside of the box.

Generate diverse ideas

Typically, attempts to measure creative thinking have focused on the number of ideas that individuals are able to generate – often referred to as “ideational fluency”. Going one step further is “ideational flexibility”, or the capacity to generate ideas that are different to each other. When it comes to measuring the quality of ideas that an individual generates, some researchers have argued that fundamentally different ideas should be weighted more than similar ideas (Guilford, 1956[3]).

The facet “generate diverse ideas” of the competency model encompasses these ideas and refers to a student’s capacity to think flexibly by generating multiple distinct ideas. Test items for this facet will present students with a stimulus and ask them to generate two or three appropriate ideas that are as different as possible from one another.

Generate creative ideas

The literature on creative thinking generally agrees that creative ideas and outputs are defined as being both novel and useful. Clearly, expecting 15-year-olds around the world to generate ideas that are completely unique or novel is neither feasible nor appropriate for the PISA assessment. In this context, originality is a useful concept as a proxy for measuring the novelty of ideas. Defined by Guildford (1950[60]) as “statistical infrequency”, originality encompasses the qualities of newness, remoteness, novelty or unusualness, and generally refers to deviance from patterns that are observed within the general population at hand. In the PISA assessment context, originality is therefore a relative measure established with respect to the responses of other students who complete the same task.

The facet “generate creative ideas” focuses on a student’s capacity to generate appropriate and original ideas. “Appropriate” means that ideas must comply with the task requirements and demonstrate a minimum level of usefulness. This dual criterion ensures the measurement of creative ideas – ideas that are both original and of use – rather than ideas that make random associations that are original but not meaningful. Test items for this facet will present students with a stimulus and ask them to develop one original idea.

Evaluate and improve ideas

Evaluative cognitive processes help to identify and remediate deficiencies in initial ideas as well as ensure that ideas or solutions are appropriate, adequate, efficient and effective (Cropley, 2006[4]). They often lead to further iterations of idea generation or the reshaping of initial ideas to improve a creative outcome. Evaluation and iteration are thus at the heart of the creative thinking process. Being able to provide feedback on the strengths and weaknesses of others’ ideas is also an essential part of any collective knowledge creation effort.

The facet “evaluate and improve ideas” focuses on a student’s capacity to evaluate limitations in ideas and improve their originality. To reduce problems of dependency across items, students were not asked to iterate upon their own ideas but rather to modify someone else’s work. Test items for this facet present students with a given scenario and idea and ask them to suggest an original improvement, defined as a change that preserves the essence of the initial idea but that adds or incorporates original elements.

Domains of creative thinking

The literature on creative thinking suggests that the larger the number of domains included in an assessment of creative thinking, the better the coverage of the construct. However, certain practical and logistical constraints limit the number of possible domains that can be included in the PISA 2022 assessment of creative thinking. These constraints include:

• The age of test takers. 15-year-olds have limited knowledge and experience in many domains, meaning those included in the assessment must be familiar to most students around the world and must reflect realistic manifestations of creative thinking that 15-year-olds can achieve in a constrained test context.

• The available testing time. Students will sit a maximum of one hour on creative thinking items, meaning the range of possible domains must be limited to ensure sufficient data are collected from tasks in each domain. As PISA aims to provide comparable measures of performance at the country level rather than the individual level, it is possible to apply a rotated test design in which students take on different combinations of tasks within domains.

• The available testing technology. The PISA test is administered on standard desktop computers with no touchscreen capability or Internet connection. Although the test platform supports a range of item types and response modes, including interactive tools and basic simulations, the choice of domains and the design of the tasks needed to take into consideration the technical limitations of the platform.

Taking these main constraints into account and building upon the literature exploring different domains of creativity, the PISA 2022 test included tasks situated within four distinct domains: 1) written expression; 2) visual expression; 3) social problem solving; and 4) scientific problem solving. The written and visual expression domains involve communicating one’s imagination to others, and creative work in these domains tends to be characterised by originality, aesthetics, imagination, and affective intent and impact. In contrast, social and scientific problem solving involve investigating and solving open problems. They draw on a more functional employment of creative thinking that is a means to a better end, and creative work in these domains is characterised by ideas or solutions that are original, innovative, effective and efficient.

These four domains represent reasonable and sufficiently diverse coverage of the different types of “everyday” creative thinking activities in which 15-year-olds engage. Given that differences in cultural norms exist for certain forms of creative engagement as do differences in what is valued in education across the world, in addition to the fact that creative engagement in each domain is supported by some degree of domain readiness, we can also expect variation in student performance across domains. By having students work on more than one domain during the test, it is possible to gain insights on country-level strengths and weakness by domain of context. Each of the four domain contexts are described in further detail below.

Written expression

Creative writing involves communicating ideas and imagination through written language. Good creative writing requires that readers understand and believe in the author’s imagination, including the rules of logic within the universe the author has created. Both fictional and non-fictional writing can be creative, and learning how to express oneself creatively can help students to develop effective and impactful communication skills that they will need throughout their lifetimes.

In the PISA test, students express their imagination in a variety of written formats. For example, students caption an image, propose ideas for a short story using a given text or visual as inspiration, or write a short dialogue between characters for a movie or comic book plot.

Visual expression

Visual expression involves communicating ideas and imagination through a range of different media. Creative visual expression has become increasingly important as the ubiquity of desktop publishing, digital imaging and design software means that nearly everyone will need to design, create or engage with visual communications at some point in their personal or professional lives.

In the PISA test, students express their imagination by using a digital drawing tool. The drawing tool does not enable free drawing, but students can create visual compositions by dragging and dropping elements from a library of images and shapes. Students were also able to resize, rotate and change the colour of elements. Students create visual designs for a variety of purposes, such as creating a clothing design, or a logo or poster for an event.

Social problem solving

Young people use creative thinking every day to solve personal, interpersonal and social problems. These problems can range from the small-scale, personal level (e.g. resolving a schedule conflict) to the wider school, community or even global level (e.g. finding ways to improve sustainable living). Creative thinking in this domain involves understanding different perspectives, addressing the needs of others, and finding innovative and functional solutions for the parties involved (Brown and Wyatt, 2010[61]).

In the PISA test, students solve open problems that have a social focus. These problems focus on issues that affect subsets of society (e.g. young people) or on issues that affect society at large (e.g. the use of global resources or the production of waste materials). Students were asked to propose ideas or solutions in response to a given scenario, or to suggest original ways to improve others’ solutions.

Scientific problem solving

Scientific problem solving involves generating new ideas and understanding, designing experiments to probe hypotheses, and developing new methods or inventions (Moravcsik, 1981[62]). Students can also demonstrate creative thinking as they engage in a process of scientific inquiry by exploring and experimenting with different ideas or materials to make discoveries and advance their knowledge and understanding (Hoover, 1994[63]).

Although creative thinking in science is related to scientific inquiry, the tasks in this domain differ fundamentally from the PISA scientific literacy tasks. In this test, students were asked to generate multiple distinct ideas or solutions, or an original idea or solution, for an open problem for which there is no predefined correct response. In other words, the tasks measure students’ capacity to produce diverse and original ideas, not their ability to reproduce scientific knowledge or understanding. For example, in a task asking students to formulate different hypotheses to explain a phenomenon, they would be rewarded for proposing multiple plausible hypotheses regardless of whether one of those hypotheses constituted the correct explanation for the phenomenon. Nonetheless, domain readiness may affect performance in this domain more than others, as most tasks that can be imagined imply a minimum level of knowledge of basic scientific principles.

In the PISA test, students engage with open problems that have a scientific or engineering basis. Students were asked to propose hypotheses to explain a given scenario, or to improve or generate new methods for solving problems.

Scoring the tasks

Every task in the PISA test is open-ended, meaning there are essentially infinite ways of demonstrating creative thinking. Scoring for this assessment therefore relies on human judgement following detailed scoring rubrics and well-defined coding procedures. All items corresponding to the same facet of the competency model apply the same general coding procedure. However, as the form of response varies by domain and task (e.g. a title, a solution, a design, etc.), so do the item-specific criteria for evaluating whether an idea is different or original. The detailed coding guides describe the item-specific criteria for each item and provide annotated example responses to help human coders score consistently (see Annex C for excerpts of the item-specific coding criteria for select released items from the PISA 2022 Creative Thinking test described in Chapter 1).

Scoring of “generate diverse ideas” items

All items corresponding to the “generate diverse ideas” facet of the competency model required students to provide two or three responses. The general coding procedure for these items involves two steps, as summarised in Figure III.A1.2. First, coders must determine whether responses are appropriate. Appropriate in the context of this assessment means that student responses respect the required form and connect (explicitly or implicitly) to the task stimulus. Second, coders must determine whether responses are sufficiently different from one another based on item-specific criteria described in the coding guide.

Figure III A1.2. General coding process for “generate diverse ideas” items

The item-specific criteria are as objective and inclusive as possible of the range of different potential responses. For example, for a written expression item, sufficiently different ideas must use words that convey a different meaning (i.e. are not synonyms). For items in the problem-solving domains, the coding guides list pre-defined response categories to help coders distinguish between similar and different ideas. The coding guides provide detailed example responses and explanations for how to code each example.

Full credit is assigned where all the responses required in the task are both appropriate and sufficiently different from each other. Partial credit is assigned in tasks requiring students to provide three responses, and where two or three responses are appropriate but only two are different from each other. No credit is assigned in all other cases.

Scoring of “generate creative ideas” items

All items corresponding to the facet “generate creative ideas” of the competency model require a single response. The general coding procedure for these items involves two or three steps, depending on the content of the response. First, as with all items, coders must determine whether the response is appropriate. Then, coders must determine whether the response is original by considering two criteria (see Figure III.A1.3).

An original idea is defined as a relatively uncommon idea with respect to the general pool of responses. The coding guide identifies one or more conventional themes for each item according to the patterns of genuine student responses revealed in multiple validation studies. If a response does not correspond to a conventional theme as described in the coding guide, it is directly coded as original. However, if an idea does correspond to a conventional theme, then coders must determine whether it is original based on its elaboration. The coding guide provides item-specific explanations and examples of original ways to elaborate on conventional themes. For example, a student might add an unexpected twist to a story idea that otherwise centres on a conventional theme.

Figure III A1.3. General coding process for “generate creative ideas” and “evaluate and improve ideas” items

This two-fold originality criteria ensures that the scoring model takes into account both the general idea and the details of a response. While this approach does not single out the most original responses in the entire response pool, it does ensure that the coding process is less susceptible to culturally-sensitive grading styles that favour middle points or extremes, and it provides some mitigation against potential cultural bias in the identification of conventional themes across countries.

Full credit is assigned where the response is both appropriate and original. Partial credit is assigned where the response is appropriate only, and no credit is assigned in all other cases.

Scoring of “evaluate and improve ideas” items

All items corresponding to the facet “evaluate and improve ideas” of the competency model require a single response and generally ask students to adapt a given idea in an original way rather than coming up with an idea from scratch. The general coding procedure for these items involves the same steps as those for the “generate creative ideas” items, described above and in Figure III.A1.3.

However, appropriate responses for these items must be both relevant and constitute an improvement. The threshold for achieving the appropriateness criteria for these items is thus somewhat strengthened with respect to items measuring the other two facets, as responses must explicitly connect to the task stimulus and attempt to address its deficiencies. The coding guide provides item-specific criteria, examples and explanations to help orient coders. For responses considered appropriate, coders must establish the originality of the improvement by considering the same two originality criteria as for “generate creative ideas” items.

Full credit is assigned where the response is both appropriate and an original improvement. Partial credit is assigned where the response is only appropriate, and no credit is assigned in all other cases.

References

[50] Amabile, T. (2012), Componential Theory of Creativity, http://www.hbs.edu/faculty/Publication%20Files/12-096.pdf.

[32] Amabile, T. (1997), “Motivating Creativity in Organizations: On Doing What You Love and Loving What You Do”, California Management Review, Vol. 40/1, pp. 39-58, https://doi.org/10.2307/41165921.

[55] Amabile, T. (1996), “Creativity in Context: Update to the Social Psychology of Creativity (Boulder, CO”, Westview Press.Baer, J.(1998).The case for domain specificity of creativity.Creativity Research Journal, Vol. 11.

[28] Amabile, T. (1983), “The social psychology of creativity: A componential conceptualization.”, Journal of Personality and Social Psychology, Vol. 45/2, pp. 357-376, https://doi.org/10.1037/0022-3514.45.2.357.

[33] Amabile, T. and M. Pratt (2016), “The dynamic componential model of creativity and innovation in organizations: Making progress, making meaning”, Research in Organizational Behavior, Vol. 36, pp. 157-183, https://doi.org/10.1016/j.riob.2016.10.001.

[10] Baer, J. (2016), “Creativity Doesn’t Develop in a Vacuum”, New Directions for Child and Adolescent Development, Vol. 2016/151, pp. 9-20, https://doi.org/10.1002/cad.20151.

[30] Bandura, A. (1997), “Self-efficacy: The exercise of control. New York: W.H. Freeman and Company”, American Psychological Association, Vol. 23.

[19] Batey, M. and A. Furnham (2006), “Creativity, Intelligence, and Personality: A Critical Review of the Scattered Literature”, Genetic, Social, and General Psychology Monographs, Vol. 132/4, pp. 355-429, https://doi.org/10.3200/mono.132.4.355-430.

[31] Beghetto, R. (2006), “Creative Self-Efficacy: Correlates in Middle and Secondary Students”, Creativity Research Journal, Vol. 18/4, pp. 447-457, https://doi.org/10.1207/s15326934crj1804_4.

[29] Beghetto, R. and M. Karwowski (2017), “Toward Untangling Creative Self-Beliefs”, in The Creative Self, Elsevier, https://doi.org/10.1016/b978-0-12-809790-8.00001-7.

[53] Beghetto, R. and J. Kaufman (2014), “Classroom contexts for creativity”, High Ability Studies, Vol. 25/1, pp. 53-69, https://doi.org/10.1080/13598139.2014.905247.

[52] Beghetto, R. and J. Kaufman (2010), Nurturing Creativity in the Classroom, Cambridge University Press.

[54] Bereiter, C. and M. Scardamalia (2010), ““Can Children Really Create Knowledge?””, Canadian Journal of Learning and Technology, Vol. 31/1.

[48] Berliner, D. (2012), “Narrowing curriculum, assessments, and conceptions of what it means to be smart in the U.S. schools: Creaticide by design”, in How Dogmatic Beliefs Harm Creativity and Higher-Level Thinking.

[21] Berzonsky, M. and C. Sullivan (1992), “Social-Cognitive Aspects of Identity Style”, Journal of Adolescent Research, Vol. 7/2, pp. 140-155, https://doi.org/10.1177/074355489272002.

[61] Brown, T. and J. Wyatt (2010), “Design Thinking for Social Innovation”, Stanford Social Innovation Review, Vol. 8/I, pp. 31–35.

[4] Cropley, A. (2006), ““In Praise of Convergent Thinking””, Creativity Research Journal, Vol. Vol. 18/3, pp. pp. 391-404.

[26] Cropley, A. (1990), “Creativity and Mental Health in Everyday Life”, Creativity Research Journal, Vol. 3/3, https://doi.org/10.1080/10400419009534351.

[34] Csikszentmihalyi, M. (1996), Creativity: Flow and the Psychology of Discovery and Invention, Harper Collins Publishers.

[47] DeCoker, G. (2000), “Looking at U.S. education through the eyes of Japanese teachers”, Phi Delta Kappan, Vol. 81/10.

[25] DeYoung, C. (2015), “Openness/intellect: A dimension of personality reflecting cognitive exploration.”, in APA handbook of personality and social psychology, Volume 4: Personality processes and individual differences., American Psychological Association, Washington, https://doi.org/10.1037/14343-017.

[8] Duncker, K. (1972), On problem-solving., Greenwood Press,, https://books.google.fr/books/about/On_problem_solving.html?id=dJEoAAAAYAAJ&redir_esc=y (accessed on 27 March 2018).

[36] Eisenberger, R. and L. Shanock (2003), Rewards, Intrinsic Motivation, and Creativity: A Case Study of Conceptual and Methodological Isolation, https://doi.org/10.1080/10400419.2003.9651404.

[20] Feist, G. (1998), “A Meta-Analysis of Personality in Scientific and Artistic Creativity”, Personality and Social Psychology Review, Vol. 2/4, pp. 290-309, https://doi.org/10.1207/s15327957pspr0204_5.

[9] Getzels, J. (1976), The Creative Vision: A Longitudinal Study Of Problem Finding in Art, John Wiley & Sons.

[1] Glaveanu, V. et al. (2013), “Creativity as action: findings from five creative domains”, Frontiers in Psychology, Vol. 4, https://doi.org/10.3389/fpsyg.2013.00176.

[39] Grivas, C. and G. Puccio (2011), The Innovative Team: Unleashing Creative Potential for Breakthrough Results, Jossey-Bass.

[23] Guastello, S. (2008), “Creativity and personality”, in The Routledge Companion to Creativity, https://doi.org/10.4324/9780203888841-30.

[3] Guilford, J. (1956), “The structure of intellect.”, Psychological Bulletin, Vol. 53/4, pp. 267-293, https://doi.org/10.1037/h0040755.

[60] Guilford, J. (1950), “Creativity.”, American Psychologist, Vol. 5/9, pp. 444-454, https://doi.org/10.1037/h0063487.

[11] H. W. Stevenson, H. (ed.) (1986), Two courses of expertise, W H Freeman/Times Books/ Henry Holt & Co.

[18] Hennessey, B. and T. Amabile (2010), “Creativity”, Annual Review of Psychology, Vol. 61/1, pp. 569-598, https://doi.org/10.1146/annurev.psych.093008.100416.

[63] Hoover, S. (1994), “Scientific problem finding in gifted fifth‐grade students”, Roeper Review, Vol. 16/3, pp. 156-159, https://doi.org/10.1080/02783199409553563.

[24] Kashdan, T. and F. Fincham (2002), Facilitating creativity by regulating curiosity., https://doi.org/10.1037/0003-066X.57.5.373.

[56] Kaufman, J. and J. Baer (2012), “Beyond New and Appropriate: Who Decides What Is Creative?”, Creativity Research Journal, Vol. 24/1, pp. 83-91, https://doi.org/10.1080/10400419.2012.649237.

[13] Kaufman, J. et al. (2010), “Personality and Self-Perceptions of Creativity across Domains”, Imagination, Cognition and Personality, Vol. 29/3, pp. 193-209, https://doi.org/10.2190/ic.29.3.c.

[14] Kaufman, S. et al. (2016), “Openness to Experience and Intellect Differentially Predict Creative Achievement in the Arts and Sciences”, Journal of Personality, Vol. 84/2, pp. 248-258, https://doi.org/10.1111/jopy.12156.

[44] Lubart, T. (1998), “Creativity Across Cultures”, in Handbook of Creativity, Cambridge University Press, https://doi.org/10.1017/cbo9780511807916.019.

[15] McCrae, R. (1987), “Creativity, divergent thinking, and openness to experience.”, Journal of Personality and Social Psychology, Vol. 52/6, pp. 1258-1265, https://doi.org/10.1037/0022-3514.52.6.1258.

[62] Moravcsik, M. (1981), “Creativity in science education”, Science Education, Vol. 65/2, pp. 221-227, https://doi.org/10.1002/sce.3730650212.

[35] Nakamura, J. and M. Csikszentmihalyi (2002), “The Concept of Flow Optimal Experience and Its Role in Development”, Handbook of positive psychology.

[58] Newell, A., J. Shaw and H. Simon (1962), “The processes of creative thinking.”, in Contemporary approaches to creative thinking: A symposium held at the University of Colorado., Atherton Press, New York, https://doi.org/10.1037/13117-003.

[46] Ng, A. (2003), “A Cultural Model of Creative and Conforming Behavior”, Creativity Research Journal, Vol. 15/2-3, pp. 223-233, https://doi.org/10.1080/10400419.2003.9651414.

[49] Nickerson, R. (2010), “How to Discourage Creative Thinking in the Classroom”, in Nurturing Creativity in the Classroom, Cambridge University Press, https://doi.org/10.1017/cbo9780511781629.002.

[59] Nickerson, R. (1998), “Enhancing Creativity”, in Handbook of Creativity, Cambridge University Press, https://doi.org/10.1017/cbo9780511807916.022.

[42] Niu, W. and R. Sternberg (2003), “Societal and school influences on student creativity: The case of China”, Psychology in the Schools, Vol. 40/1, pp. 103-114, https://doi.org/10.1002/pits.10072.

[16] Prabhu, V., C. Sutton and W. Sauser (2008), “Creativity and Certain Personality Traits: Understanding the Mediating Effect of Intrinsic Motivation”, Creativity Research Journal, Vol. 20/1, pp. 53-66, https://doi.org/10.1080/10400410701841955.

[38] Prather, C. (2010), Manager’s Guide to Fostering Innovation and Creativity in Teams.

[5] Reiter-Palmon, R. and E. Robinson (2009), ““Problem identification and construction: What do we know, what is the future?“”, Psychology of Aesthetics, Creativity, and the Arts, Vol. Vol. 3/1, pp. pp. 43-47.

[45] Rinne, T., G. Steel and J. Fairweather (2013), “The Role of Hofstede’s Individualism in National-Level Creativity”, Creativity Research Journal, Vol. 25/1, pp. 129-136, https://doi.org/10.1080/10400419.2013.752293.

[6] Runco, M. (1997), The Creativity Research Handbook.

[57] Scardamalia, M. and C. Bereiter (1999), “Schools as knowledge building organizations.”, Today’s children, tomorrow’s society: The developmental health and wealth of nations.

[7] Schank, R. and R. Abelson (1977), Scripts, Plans, Goals, and Understanding : An Inquiry into Human Knowledge Structures, L. Erlbaum Associates.

[12] Schwartz, D. (2005), ““Efficiency and Innovation in Transfer””, Transfer of learning from a modern, multidisciplinary perspective, Vol. 3, pp. 1-51.

[40] Smith, B. (ed.) (2002), Collective Cognitive Responsibility for the Advancement of Knowledge, Open Court.

[2] Tanggaard, L. (2014), Fooling Around: Creative Learning Pathways, Information Age Publishing.

[37] Thompson, L. and H. Choi (eds.) (2006), Creativity and Innovation in Organizational Teams, Psychology Press, https://doi.org/10.4324/9781410615732.

[27] Torrance, P. (1988), “The Nature of Creativity as Manifest in its Testing”, in The Nature of Creativity: Contemporary Psychological Perspectives.

[22] Villalba, E. (ed.) (2009), Creativity and personality, European Commission Joint Research.

[41] Warhuus, J. et al. (2017), “From I to We: collaboration in entrepreneurship education and learning?”, Education + Training, Vol. 59/3, pp. 234-249, https://doi.org/10.1108/et-08-2015-0077.

[17] Werner, C. et al. (2014), “The Chinese Version of the Revised Creativity Domain Questionnaire (<scp>CDQ</scp>‐R): First Evidence for its Factorial Validity and Systematic Association with the Big Five”, The Journal of Creative Behavior, Vol. 48/4, pp. 254-275, https://doi.org/10.1002/jocb.51.

[43] Wong, R. and W. Niu (2013), “Cultural Difference in Stereotype Perceptions and Performances in Nonverbal Deductive Reasoning and Creativity”, The Journal of Creative Behavior, Vol. 47/1, pp. 41-59, https://doi.org/10.1002/jocb.22.

[51] Zhou, J. and Y. Su (2010), “A Missing Piece of the Puzzle: The Organizational Context in Cultural Patterns of Creativity”, Management and Organization Review, Vol. 6/3, pp. 391-413, https://doi.org/10.1111/j.1740-8784.2010.00192.x.