Samuel Greiff
University of Luxembourg
Art Graesser
University of Memphis
Dragos Iliescu
University of Bucharest
Jean-François Rouet
Centre national de la recherche scientifique, University of Poitiers
Katharina Scheiter
University of Tübingen
Ronny Scherer
University of Oslo
With assistance from Juliana Gottschling
and Jan Dörendal Gottschling
University of Luxembourg
This chapter defines the concept of adaptive problem solving (APS) in the second cycle of PIAAC. The concept of APS accounts for the fact that we need to be vigilant, adaptive, and willing to modify our plans when interacting with the social, physical, and technological world of the 21st century. In this framework chapter, the cognitive and metacognitive processes that successful people engage into when solving problems and when adapting to changing conditions are described. In this, the PIAAC assessment of APS draws from a large set of information contexts and task dimensions that drive overall APS performance and individual proficiency levels. Several example items, considerations on item scoring and data capturing as well as a thorough discussion of the relation between APS and other competencies provide a comprehensive overview of the APS measurement framework for PIAAC.
Rapid changes in the social, physical, and technological world require individuals to be more vigilant to changes, more adaptive, and more willing to modify their plans in pursuit of their goals. It is therefore indisputable that the competence to solve problems and to adapt to changing conditions is of crucial importance in the 21st century, where citizens are faced with increasingly complex technologies, social systems and subject matters (Levy and Murnane, 2006[1]; National Research Council, 2012[2]). The need for problem solving is ubiquitous in the workplace, as well as everyday life, for most adults. For instance, Felstead et al. (2013[3]) conclude that problem solving skills are more important than numerical or communication skills for a worker to be successful, a finding that is likely to generally apply to economies that are service-oriented. Problem solving is therefore generally important to assess as an overarching construct.
The Programme for the International Assessment of Adult Competencies (PIAAC) included a measure of problem-solving proficiency in its first cycle in 2011. In addition to core dimensions of adult skills, i.e., reading component skills, literacy, and numeracy, the survey assessed problem solving in technology-rich environments (PS-TRE) for individuals aged 16 to 65. PS-TRE focused on goal setting, monitoring, and planning in technology-rich environments (OECD, 2012[4]) and assessed proficiency in the use of specific digital applications to access, search, manage, interpret, and evaluate information. The second cycle of PIAAC in 2022 will focus on adaptive problem solving (APS). “Adaptive” underlines that problem solving is a process that takes place in complex environments and that this process is not a static sequence of a number of pre-set steps but rather a constant attempt to solve a problem. Hence, while problems themselves can either be static (i.e., with no changes in the given states or the goal states) or dynamic (i.e., with changes occurring in the problem situation), the process of problem solving when confronted with dynamic problems is adaptive (i.e., problem solvers need to adapt to the dynamic nature of such problems).
There are three important core aspects that distinguish APS from previous large-scale assessments of problem solving, such as PS-TRE or as implemented in the assessment of the Programme for International Student Assessment (PISA):
First, the competence to handle dynamic and changing problem situations has become increasingly important in today’s society, and therefore the need for skills that enable adults to adjust their thinking and reasoning to novel and changing information has grown crucially. The assessment of APS will therefore focus on dynamic problems that require problem solvers to monitor their problem solving and to adapt their initial solution to new information or circumstances.
Second, the characteristics of the typical problems that individuals encounter at work and everyday life have been changing over the last five decades, in part because of radical changes in digital technologies and communication media (Autor, Levy and Murnane, 2003[5]). The solutions to particular problems are also more distributed over time as people take advantage of social and digital resources that have particular constraints in access and timetables. This new wealth of information, and the shift in the information environment that people are confronted with, will be reflected in the characteristics of the tasks included in the APS assessment, i.e., the information environments (physical, social, and digital) and problem contexts (personal, work, and social community) in which tasks will be situated.
Finally, cognitive processes are inherently bound to the problem-solving process and have always been an important aspect of the problem-solving assessment. However, especially in highly adaptive and higher difficulty problems, problem solvers also need to strongly engage in metacognitive processes (i.e., the ability to calibrate one’s comprehension of the problem, evaluate potential solutions, and monitor progress towards the goals). Consequently, the assessment of APS in the second cycle of PIAAC will put emphasis also on metacognitive processes.
The purpose of this document is to provide an assessment framework following the conceptual framework paper for APS (Greiff et al., 2017[6]) to guide the construction of APS items to be used in the second cycle of PIAAC as well as the definition of the proficiency scale for APS.
The ability to quickly and flexibly adapt to new circumstances, learn throughout life, and turn knowledge into action has always been important for full participation in labour markets and society (National Research Council, 2012[2]). However, in a world that has become increasingly and dynamically changing, and which provides a plethora of information from different resources, the need to flexibly adapt to unexpected changes has become more and more important. Over the course of a single day, an individual can be a purchaser of consumer goods, an organiser of local transportation, a holiday planner (searching for flights and accommodation arrangements in hotels or house swaps), a financial planner, and a home decorator. These various activities address multiple goals in non-routine ways that require APS skills. People need to adjust, for example, to prices of commodities that change overnight, a strike of transportation workers, internet sites that go down, and people who cancel appointments. Adapting to these unexpected changes in these various environments requires problem solvers to consider different resources in the physical, social, and digital environments, in addition to their own mental activities. Therefore, APS is particularly important to assess as problems often dynamically change during the course of problem solving, which then requires constant monitoring and, if necessary, adaptation of the original problem solution. These changes occur because of unexpected physical and/or social events in the environment and because of unintended consequences of the problem solver’s actions.
It is important to emphasise that the assessment of APS in the second cycle of PIAAC goes beyond what was assessed in previous OECD international assessments of problem solving. For one, the problems assessing individual problem solving in PISA 2009 were entirely static (i.e., the given states and goal states did not change) and preceded the collection of data on computers. In PISA 2012, the assessment of problem-solving competency was computer-based and allowed the implementation of interactive problem situations in addition to static ones (OECD, 2014[7]). The items became dynamic in the sense that the problem solver needed to interact with the problem environment in order to find all the relevant information to solve the problem. PISA 2015 then focused on collaborative problem solving with computer agents that interacted with a problem solver through chat facilities and actions performed in shared workspaces (OECD, 2017[8]). It is important to stress that the term dynamic is broadened in the assessment of APS as it refers not only to the exploration of the environment, i.e. the interaction between the problem solver and the information context, but also to changes in the problem situation to which the initial solution needs to be adapted to. When we refer to “dynamic” in the following, we always use the term in this broadened manner.
As mentioned before, problem solving was already assessed in the first cycle of PIAAC. The PS-TRE assessment was conceived to monitor the problem solver’s information-processing skills when operating in technology-rich environments using information and communications technology (ICT) skills. Core to the PS-TRE assessment therefore was the understanding and evaluation of meaningful information available in technology-rich environments, including simulated websites, e-mail and spreadsheet environments (OECD, 2012[4]). The assessment of APS will also use technology-rich environments. However, these environments will rather form the context in which the problem unfolds dynamically and to which the problem solvers need to adapt their initial problem solution.
As mentioned before, successful problem solving requires the problem solver to engage in cognitive as well as metacognitive processes. Previous assessments of problem solving have incorporated core cognitive theories of problem solving (Funke, 2010[9]; Mayer and Wittrock, 2006[10]). They start with the definition of a problem as having a given state, a goal state, a set of legal operators to get from the given to the goal state, and plans for solutions to subtasks. The PISA 2012 and 2015 assessments identified the problem-solving components as 1) exploring, understanding, and representing the problem, 2) searching, planning, and executing potential solutions, and 3) monitoring and reflecting on the progress towards solving the problem. The assessment of APS in the second cycle of PIAAC will have the following cognitive problem-solving components that are similar but not exactly the same: defining the problem – the same as 1) –; searching for information, and applying a solution – these latter two components mapping onto 2) –, whereas the explicit assessment of metacognition will incorporate 3).
The cognitive processes become more complicated in APS where the problem solution might need to be adapted in reaction to dynamically changing situations. That is, physical, social, and digital worlds are frequently undergoing changes that an adaptive problem solver must accommodate. The problem solver faces the additional challenge of having to continuously monitor, often through conscious effort, whether the current problem state remains the same or changes throughout the course of problem solving, whether operators that are already known from similar problem-solving attempts are still available or whether new ones need to be identified, and which plans can be executed using the available resources at a given point in time. The second cycle of PIAAC will contain items that measure metacognitive processes in addition to cognitive processes. The role of metacognitive processes becomes more important to the extent that problems are more complex and difficult to comprehend (requiring comprehension calibration), the problems change dynamically (requiring evaluation and re-evaluation of the suitability of operators and plans), and progress towards the solution becomes more difficult to discern (requiring monitoring and reflecting on progress towards the goals).
Both cognitive and metacognitive processes will be assessed at three stages of problem solving: defining the problem, searching for a solution, and applying a solution. There are cognitive processes and metacognitive processes required at each stage, with some items tapping both processes and others focusing on either cognition or metacognition.
In a nutshell, in the second cycle of PIAAC, the APS assessment will put greater emphasis on individuals’ capacity to a) flexibly and dynamically adapt their problem-solving strategies to a dynamically changing environment, b) identify and select among a range of available physical, social, and digital resources, and c) monitor and reflect on their progress in solving problems through metacognitive processes. The assessment tasks will therefore reflect the fact that solutions to problems in the modern world require a reflexive, flexible, and adaptive mind.
In the following, we will first define APS and introduce two tasks to exemplify how APS can be assessed. We then detail the task dimensions that define each APS tasks and describe the required cognitive and metacognitive processes. The next section describes the factors that may be used to describe the APS proficiency levels and is followed by a summary of the assessment of APS. We close with a comparison of APS with other core competencies, i.e., literacy, numeracy, and digital competency.
As mentioned above, there are three core aspects that are represented in the conceptual framework (Greiff et al., 2017[6]) and in the assessment framework of APS. First of all, in a dynamically changing world, it is essential to react to unforeseen changes and new information in a flexible and adaptive way. This is represented in the term “adaptive” in APS. Second, as the amount of information available in the world of the 21st century is ever increasing, we are faced with a wealth of information from different sources. This expansion of information environments needs to be taken in account and will be reflected in the tasks developed for APS, which will be situated in a range of information environments and contexts. Finally, whereas cognitive aspects have always been an important part of problem solving, the necessary change of plans and approaches to a problem and the adaptability and flexibility coming along with this require a stronger focus on metacognition in addition to the existing focus on cognition. Thus, APS puts a strong focus on metacognitive aspects throughout the process of problem solving.
The definition of adaptive problem solving in the second cycle of PIAAC is as follows:
“Adaptive problem solving involves the capacity to achieve one’s goals in a dynamic situation, in which a method for solution is not immediately available. It requires engaging in cognitive and metacognitive processes to define the problem, search for information, and apply a solution in a variety of information environments and contexts”. (Greiff et al., 2017[6])
Each part of this definition is explained in more detail below.
Adaptive problem solving…
The term “adaptive” stresses the adaptive nature of problem solving irrespective of the environment or the context in which the problem solving takes place. This underlines that problem solving is a process that takes place in complex environments and that this process is not a static sequence of a number of pre-set steps. Rather there could be an adaptive nature to the problem-solving process in each step. Put differently, problem solvers need to remain open and pay attention to changes in the situation and adapt their problem-solving approach accordingly. The term “adaptive” readily connects to notions such as cognitive flexibility or plasticity, but is broader in its meaning and encompasses the entire set of cognitive and non-cognitive components involved in APS.
“Problem solving” was chosen as a core term for the focus on situations that require non-routine solutions (as opposed to tasks, see below) independent of the specific content domain. Problem solving is generally regarded as one of the most ubiquitous activities that is necessary to successfully master challenges in unforeseen situations, be it in educational contexts, on the job, or in private life. Because problems can occur in a number of settings, the process of problem solving, including its different components, can be applied across different domains. In fact, a transversal understanding of problem solving has recently been included in several large-scale assessments, such as PIAAC and PISA, but those assessments differed in that they did not focus on the “adaptive” nature of problem solving in the 21st century.
…involves the capacity to achieve one’s goals in a dynamic situation…
The broad term of “capacity” is meant to convey that APS is a complex proficiency that is composed of a number of more specific sets of skills, most notably cognitive and metacognitive aspects that are explicitly targeted in the assessment. APS also includes the motivation to deal with the problem situation and to face the challenges of the problem situation and its unforeseen changes. Through this, the motivational aspect is implicitly part of the assessment, but it is not an explicit part of the core APS assessment.
Problem solving is a goal-directed activity, in which the problem solver is embedded into a situation that needs to be mastered successfully and this situation may be dynamic. That is, as opposed to static problem solving that takes place exclusively in situations that have no dynamic component, which implies that all relevant information is available at the outset and that there is no change in the problem setup, the constraints, or the goals have to be foreseen. When engaging into APS, problem solvers need to anticipate, incorporate, and deal with the many types of dynamic changes that might happen while moving from an initial state to a desired goal state. APS therefore refers to the process of problem solving in dynamically changing situations. More precisely, the dynamic aspect of the problem situation implies on the one hand that relevant information from different sources might need to be acquired throughout the process, something that has been considered relevant in previous assessments of problem solving (cf. the assessment of problem solving in PISA 2012). However, in addition to the capacity of exploring a problem situation, the problem solver also needs to deal with various types of changes in the situation and needs to react to these changes. Put differently, problem solvers need to monitor their progress, the problem state, and the environment and context in an attempt to pay tribute to the dynamic nature of the overall problem situation that might exhibit constant change or hardly any change at all. From an assessment perspective, the inclusion of the dynamic component relies on the use of technologically based assessments that allow for the type of items in which such dynamic changes can be implemented. In this, the second cycle of PIAAC is a technology-based assessment that allows a broadening of the scope of the proficiencies through the technical means and, through this, new item formats available to test developers.
…in which a method for solution is not immediately available.
This part of the definition alludes to a core component of virtually any problem-solving definition: at the outset, the path to the solution and the solution itself are not immediately clear and require that the problem solver initiates a process that, ultimately, leads to the goal state. This distinguishes problem solving from a mere task, in which a solution usually is readily available. It also shows that, even in specific domains such as mathematics or science not all items are problem solving items as some of them could be solved merely by knowing the correct answer, and it also stresses the non-routine aspect of the problems in this domain. In this, there is a direct link between existing frameworks of problem solving (e.g., problem solving in PISA 2012 or collaborative problem solving in PISA 2015), but the notion of a solution that is not immediately accessible is even more central to APS because changes in the problem setup or the problem situation require a re-examination of initial solutions and, in some cases, new approaches to solve the presented problem.
…It requires engaging in cognitive and metacognitive processes…
Cognitive and metacognitive components are both critical aspects of APS. Problem solving always requires some cognition such as organising and integrating information into a mental model or evaluating operators as to whether they are relevant for reaching the desired goal state. But metacognition, such as setting a goal or reflecting on progress, is equally important. In fact, both components are often intertwined in a way that makes it difficult to separate them and it will be a challenge in the assessment to do so. While the role of metacognition has been acknowledged in previous assessment frameworks, it has often not been targeted explicitly but rather been considered as a part implicitly included into the assessment. Here, APS differs in the sense that dealing with a dynamic situation in an adaptive way always requires a certain level of metacognition. For instance, if the situation changes, without a sufficient level of metacognitive awareness, this change will go by unnoticed and will not lead to a solution of the problem. Thus, the conceptual framework (Greiff et al., 2017[6]) stresses that the world of the 21st century cannot be successfully mastered without a certain level of metacognition. The assessment of APS will be designed in a way that it clearly reflects the need for metacognition and will also develop items that primarily target the problem solver’s metacognitive proficiency.
…to define the problem, search for information, and apply a solution…
The APS framework defines three broad problem-solving stages that are logically ordered from first defining the problem, second searching for information, and, finally, applying a solution. However, this is a schematic description and any problem-solving activity switches between the different stages or might even employ them simultaneously. The description here is meant to convey that usually one of those activities prevails. The assessment will aim to elicit problem solvers’ cognitive and metacognitive proficiencies along these three stages in a comprehensive way.
In each of the three stages, both cognitive and metacognitive processes are relevant and while there is some overlap, many of the processes are distinct for a specific stage. In fact, the delineation of the problem-solving process into different stages is ubiquitously found in the problem-solving literature even though there is some disagreement as to the number and the nature of the stages. In APS, the problem solver is faced with the challenge that a change in the setup might occur at any time, requiring constant monitoring and a readiness to react throughout these stages. That is, as compared to other problem-solving approaches, a once derived definition, a set of information, or a chosen path towards solution might become obsolete, but instead, a new definition, new information, or a new path towards the solution needs to be derived.
…in a variety of information environments and contexts.
This final part of the definition stresses that in information-rich environments – and virtually all of today’s problems are embedded into such – the different sources from which the information originates and the different contexts are of high relevance. Information can be gathered from physical, social, or digital environments, which is meant to cover the ubiquitous nature from which the problem solver derives the knowledge about a problem in today’s world. In this, APS differs from previous problem-solving assessments that focused on specific sources of information such as the social environment in collaborative problem solving in PISA 2015 or on knowledge gathered on websites in the assessment of problem solving in technology-rich environments in the first cycle of PIAAC. In addition, as situations that require APS may occur throughout different contexts, there can be problems that are embedded into a personal, a work, or a social community context because good adaptive problem solvers must be able to apply their proficiency across contexts and derive their information from a comprehensive set of sources.
The next section outlines two example tasks, “Dinner Preparation” and “Stock Market”, to give an exemplary understanding of what is meant by APS in terms of real-world situations. We then proceed with a more detailed description of the problem characteristics underlying APS tasks, the associated difficulty drivers, the cognitive and metacognitive processes involved, and define the assumed proficiency levels that determine the quality of the derived solution. We will link this formal description to both of the example tasks throughout this framework document to illustrate the process of APS.
The APS assessment in the second cycle of PIAAC will contain scenario-based tasks, that describe every day and working-life problems. In the following, we describe two examples of APS tasks, in order to illustrate how the principles of APS are transposed into practice. It is important to note that participants will learn how to interact with the provided environments before starting with the assessment. Also, the two units listed below are examples of how APS tasks can look like. None of the examples will be part of the final APS assessment.
The first example, Dinner Preparation, covers an everyday life scenario in which the problem solver has to plan and accomplish different goals over the course of a day. Because of the often encountered need to adapt initial plans by reacting flexibly to changing circumstances and upcoming impasses, and by incorporating and dealing with new information, navigating through everyday life might be seen as the prototype of an APS task.
The second example, Stock Market, describes a financial simulation in which the problem solver has to make buying and selling decisions for a number of companies, depending on their market evolution, in order to maximise profits. The problem is highly dynamic as the problem setup constantly changes and the problem solvers have to continuously adapt their solutions to the latest evolution of the problem environment.
In the example unit Dinner Preparation (see Box 4.1), the problem solver is asked to use an interactive map to accomplish a set of pre-defined goals. The initially static situation becomes dynamic through obstacles that present a change in the presented problem and the available solutions.
Box 4.1 shows two example items for Dinner Preparation. The unit starts with a static planning task. In the first item, the problem solver needs to use an interactive map to find the fastest route to accomplish three goals, keeping a set of time constraints in mind. The problem solver needs to: take a child to school by a designated time, purchase ingredients for dinner, and return home by a designated time. This could be considered a standard problem-solving task, in which a solution needs to be found given some constraints that need to be satisfied. In the second item, the situation becomes dynamic as the problem solver has to deal with new circumstances that interfere with the initial problem solution. Impasses must be overcome and additional constraints need to be taken into consideration when adapting the initial problem solution.
Planning and coordinating different, sometimes contradicting, goals are elementary parts of our everyday lives. This ranges from activities that involve single and multiple goals that have to be planned daily, to long-term goals, and they can arise in a variety of contexts, be it personal, work, or social. However, plans are also repeatedly thwarted by unforeseeable events, or changes in the initial situation. Successfully dealing with such dynamically changing situations, in which the solution is often not directly available requires everybody to engage into APS. More specifically, the emerging problem situation needs to be defined, information about how to approach the situation has to be considered, and the (new) solution has to be applied.
Imagine that you need to accomplish one single or even multiple goals over the course of the day, such as picking up the child from school, and getting the groceries for dinner. In order to accomplish both goals, you would plan the best route for the car trip, look up the driving times, and make a shopping list. At first, the situation seems to be manageable and quite predictable.
Example Item 1
Problem solvers are provided with a map that shows different locations and a sticky note that summarises the goals to be accomplished and the time constraints to be met. A clock shows the time of the day, information on the driving time can be viewed by clicking on the locations. In this first item, problem solvers need to navigate through the map by drawing lines in order to find the fastest way to a) take the child to school by 8.30 and to b) get to a market to buy the ingredients for dinner.
However, just as in real life, while on your way, you suddenly find that one of the local shops is closed and you need to come up with a different plan – you could for example go to a different store, call someone to get the missing ingredient, or change the dinner plans.
Example Item 2
When the problem solvers have planned their route, they get informed that their chosen market got closed due to a water leakage. Problem solvers need to adjust their route while keeping in mind the time constraints.
In the example unit Stock Market (see Box 4.2), problem solvers are provided with a stock market simulation, in which they begin with initial stock holdings in five companies, and a small disposable sum of cash that they can invest. They can sell stock for cash, or buy new stock with cash. Stock prices vary on a day-by-day basis. The situation describes a “continuous drip” problem, i.e., the problem is not turn-based, and does not progress to a new stage only after the problem solver commits to an action. It rather evolves in real-time, even if the problem solver does not perform an action – in this case, a new “day” comes on screen every 60 seconds. By judging the history of each company, problem solvers have to make a decision regarding the investment solution that will most likely yield a profit in the future. They then need to sell the undesirable investments that they hold in their portfolio and buy stock in the more promising companies, in order to maximise the value of their portfolio.
While the unit architecture may appear to be quite specialised (i.e., stock market, financial operations), the problem is, in fact, a knowledge-lean task. It does not contain any references to actual companies or industries, and the solution does not depend on specialised knowledge.
Box 4.2 shows two example items for Stock Market. In the first item, the problem solver needs to optimise an investment portfolio, while considering the current status and the performance of the five companies over a defined period of time. In the second item, the situation becomes complicated, as the previous pattern of performance for the five companies changes. An impasse is generated by having the two companies with a previous positive evolution turning to negative; this interferes with the initial problem solution and requires problem solvers to rethink their problem-solving strategy.
Most financially complex situations have a few characteristics in common: a limited number of options are assessed on the go, as part of a dynamically changing situation, in which the optimal state of the system, i.e., when to commit to a decision, is uncertain. Interestingly, financial transactions are typical in a large number of contexts, and are not limited to work, social, or community contexts. Complex financial transactions are now part of everyday life in virtually every culture and are consonant with the demands of the modern world. Throughout their lives, most people will have to solve problems having a complex financial component.
Imagine that you have to make a number of financial decisions over the course of a week or month, decisions that involve selling uncompetitive assets and buying more competitive ones. In order to accomplish the goal of maximising your money, you will have to consider the evolution of each of your assets each day and decide which ones have become less desirable and should to be sold, and which ones have become more attractive and should be bought to benefit you. The situation is complex from the start, and the problem unfolds day by day – not reacting in a meaningful way may already diminish your investments, as the worth of each share changes day by day.
Example Item 1
Problem solvers are provided with a stock market simulation, in which they begin with initial stock in five companies, and a small disposable sum of cash that they can invest. They can sell stock for cash, or buy new stock with cash. Stock prices vary on a day-by-day basis. A new “day” comes on screen every 60 seconds, with new information about the evolution of the five companies. A short history, i.e., the last few days in each company’s evolution are displayed on the screen. The pattern of change for some of the companies is transparent, i.e., future change is predictable.
In this first item, problem solvers need to decide, based on the past evolution history of each company, where to invest their money. They need to sell the stock they do not need, and buy stock in the more promising companies, in order to maximise the value of their portfolio.
Example Item 2
After the problem solvers have committed their portfolio to one or both of the more promising and predictable companies (Companies 2 and 3), the behaviour of these companies changes, and they begin to have negative yield. Problem solvers need to adjust their investment while keeping in mind the ultimate goal to generate as much money as possible.
So far, we have outlined the theoretical underpinnings of APS. This following section will now focus on the core dimensions that will provide the foundation for the APS assessment. Figure 4.1 illustrates the components of each of the core dimensions. The first panel shows the five task dimensions that define an APS task and their associated difficulty drivers. These are described in more detail below. As shown in the middle panel, and discussed in the next section, a second set of core components are the cognitive and metacognitive processes (i.e., defining the problem, searching for information, and applying a solution) that are crucial for the problem-solving process in greater detail. The third panel presents an overview of the features that define the quality of a solution, as associated with three levels of proficiency in adaptive problem solving. We will then outline the assumed proficiency levels of APS that will form the basis for analysis.
To really understand what forms an adaptive problem, it is crucial to identify specific characteristics that make a problem adaptive, and to ask whether there are any qualitative and/or quantitative differences between various adaptive problems. When decomposing a problem, it becomes apparent that each adaptive problem can be described by five problem characteristics, or “task dimensions”: (1) the problem configuration, (2) the dynamics of the situation, (3) the features of the environment, (4) the information environment, and (5) the problem context (see left panel of Figure 4.1). These five task dimensions are descriptive of any adaptive problem (see Box 4.3) and will guide the development of the APS assessment in the second cycle of PIAAC.
The first three of these five task dimensions permit changes in quantity, and thus can drive the difficulty of the problem. Each of these three task dimensions has four even more specific difficulty drivers and by tweaking these, a problem can become easier or more difficult, requiring different abilities from problem solvers. More specifically, these three dimensions, along with the respective difficulty drivers, can be characterised as follows:
1) Problem configuration:
This task dimension refers to the initial problem setup and the goal state(s) including the problem elements, the relations, and the resources/operators. A problem may have more or fewer elements, and these elements may interact with each other or be relatively independent. The different elements may be accessible with ease or with difficulty, and may be more or less salient. The various elements may interact with each other or be relatively independent. And the problem requirement may include the accomplishment of only one or of several goals. All these characteristics of the initial problem configuration drive difficulty in adaptive problems.
The four difficulty drivers that are typical for this task dimension, therefore, are:
(1a) the number of elements, relations, and operations
(1b) the salience and accessibility of operators
(1c) the interactions between problem elements
(1d) the number of parallel tasks and goals
2) Dynamics of the situation:
This task dimension refers to change (or absence of change) within the problem situation and the problem constraints across time, and how this affects the problem configuration.
For example, change may happen in one or more features of the problem, these features that change may be more or less relevant for attaining the goal, change may be more or less frequent, and change may generate a difficulty and impasse (or not). All these characteristics of the “dynamism” of the problem drive the difficulty of adaptive problems.
The four difficulty drivers that are typical for this task dimension therefore are:
(2a) the number of features that change and their relevance
(2b) the salience of change
(2c) the frequency of change
(2d) the degree of impasse
3) Features of the environment:
This task dimension refers to various features that are characteristic of the environment and the information and resources available from it. For example, the environment in which the problem is set and unravels may be rich in information, and that information may be more or less relevant to solving the problem, and may be more or less structured. These characteristics of the environment have a direct impact on the difficulty of the adaptive problem.
The four difficulty drivers that are typical for this task dimensions therefore are:
(3a) the wealth of information
(3b) the proportion of irrelevant information
(3c) the (lack of) structure of the environment
(3d) the number of sources of information
Task dimensions (1) to (3) and their respective difficulty drivers are the building blocks through which a purposeful construction of the units and items of the test is able to elicit the relevant cognitive and metacognitive processes in problem solvers. It is indispensable to understand their structure and role in the architecture of adaptive problems. It is also important to mention that we do not consider these difficulty drivers to be exhaustive in any way. The ones used here reflect important aspects of APS and can be manipulated with relative ease when constructing the test items. We have therefore settled on them, while explicitly acknowledging the possibility to also describe the problem configuration, the dynamics of the situation, and the features of the environment under other, different parameters. Annex 4.A. more specifically defines the respective difficulty drivers and relate them to how simple and difficult problems would look like.
The last two task dimensions only permit changes in the quality of the context in which the problem is set and therefore these two task dimensions do not drive the difficulty of the problem. Task dimensions (4), i.e., information environment, and (5), i.e., problem contexts, give context to the problems featured in the items. Contextualisation is important for any problem-solving effort: no actual problem that people encounter in their lives is free of context. Any problem occurs (and is solved in) an environment with its specific information that may not be directly part of the problem, but that may shape both, the "flavour" of the problem, and the resources that are available for a meaningful solution. More specifically, any problem occurs in a context that is related to people's lives: some problems are personal, other occur in work settings, or in community and social contexts. The goal in specifying these two dimensions is to ensure that the item pool reflects a range of information environments and contexts.
4) Information environment:
This task dimension refers to the sources for the resources that are available for solving the problem. The nature of the information environment can be physical, social, or digital. Of course, all these resources appear more or less simultaneously in a digital problem-solving effort, but the problem imposes the need to handle (at least mentally) a specific kind of resource. These resources will be simulated in the assessment tasks.
(4a) Physical resources are those that require hands-on handling: driving a car, operating a machine by pressing buttons and pulling levers, connecting pipes, and others.
(4b) Social resources are those that require the problem solver to engage in interpersonal and social interactions with other people, such as leading a group, planning an activity with friends or family, or presenting a speech to an audience.
(4c) Digital resources are those that require the problem solver to interact with digital features or devices and make use of digital knowledge and skills, such as sorting a table, sending an e-mail, searching the web, formatting a text and others.
5) Problem contexts:
This task dimension refers to the situational embedding of the problem, whereby people encounter problems in their personal life, at work or in social and community contexts.
(5a) Contexts that are personal may refer to one's home, family, career, education, hobbies, or financial investments; these problems will therefore require problem solvers to solve a problem that occurs in the context of their personal life.
(5b) Contexts that are work-related may require problem solvers to solve a work-related task, or place them in a work-related context, in which they work under supervision or with co-workers.
(5c) Contexts that are social and community related may refer to interaction with other people in leisure activities (e.g., going to a party or hiking in the mountains) or with community resources (e.g., police, firefighters, or administrative institutions).
The Dinner Preparation unit has a specific problem configuration: it asks test-takers to accomplish two goals at the same time, the problem elements are accessible and salient and presented in a visually ordered fashion. The information environment of this example is not rich, and not much information, relevant or irrelevant, is provided beyond the problem itself. The dynamic of the situation is average: when change is induced, test-takers are prompted to the change, and the specifics are explained; still, changes can produce an impasse. The problem is placed in a personal problem context and a mixed digital and physical information environment.
The Stock Market example also has a specific mix of these characteristics. The problem configuration requires solving of only one goal, and is based on a high number of elements, that are salient and easily accessible to problem solvers. The problem environment is not very rich and does not offer much information, relevant or irrelevant, beyond the problem itself. The dynamic of the situation is high, with frequent but salient change, that does not create an explicit impasse. The information environment is digital, and the problem context is personal.
The various task dimensions are critical in the description of any given adaptive problem, and the difficulty drivers are the operational building blocks through which task dimensions are implemented in the units and items of the test (right panel of Figure 4.1). However, the task dimensions only reflect the adaptive problem, and they do not directly describe in any relevant manner the cognitive and metacognitive processes underlying adaptive problem solving.
For the cognitive and metacognitive processes, it is assumed that three distinctive cognitive and metacognitive stages, i.e., definition of the problem, search for a solution, and apply the solution (second panel of Figure 4.1), are involved to differing degrees in the process of solving the respective problem tasks. These cognitive processes are inherently bound to the problem-solving process.
Purposeful construction of the units and items of the test uses the task dimensions and their respective difficulty drivers as building blocks with which to elicit the relevant cognitive and metacognitive processes in problem solvers. The next section will focus directly on these important, and often intertwined processes that are key to any APS task.
As stated in the definition of APS, there are multiple cognitive and metacognitive processes a problem solver has to accomplish in order to arrive at a problem’s solution. These processes can be organised with respect to three stages of problem solving, namely, defining the problem, searching for information relevant to its solution, and applying a solution. Figure 4.2 illustrates how APS is conceptualised according to these stages (shown as boxes organised from left to right to reflect the overall process of adaptive problem solving) and the processes embedded within each stage.
Source: Adapted from Greiff et al. (2017, p. 19[6]).
In the following, we will define the cognitive and metacognitive processes within each stage of APS from an assessment perspective and illustrate them by referring back to the example tasks provided in Boxes 4.1 and 4.2. For each process, we will make connections to the previous section on task dimensions to exemplify how they elicit cognitive and metacognitive processes and render them more or less challenging for problem solvers. Only few references will be made to task dimensions (4) and (5) since they refer to the contextual embedding of the problem and its solution-relevant information only; it is assumed that these task dimensions have no systematic influence on the quality of the cognitive and metacognitive processes that need to be conducted to solve a problem (e.g., constructing a mental model of the problem is not inherently different for problems embedded in either a physical or digital information environment nor do personal problems require different processes than social ones).
The present section ends with some general remarks regarding the relationship between the conceptual framework of APS (Greiff et al., 2017[6]) and the way the cognitive and metacognitive processes are considered when conceptualising them from an assessment perspective. While the present description is grounded in the conceptual framework (Greiff et al., 2017[6]), some amendments are necessary to take into account the specific requirements and constraints of the assessment context.
The remainder of this section will start with the definition of cognitive processes (shown in the upper part of Figure 4.2) and then turn to metacognitive processes (shown in the lower part of Figure 4.2). This is done because cognitive processes, which refer to reasoning about the problem and its solution, are involved in any kind of problem solving irrespective of how the task dimensions are implemented in the problem. In easy problems, cognitive processes may be conducted without considerable effort. Especially in more complex problems, however, these cognitive processes may require effortful monitoring and control to ensure that they are correctly executed. For instance, any change of information about the problem (as introduced in example item 2 of the Dinner Preparation example) will make it necessary for a problem solver to verify the understanding of what the problem is about and whether the initially derived solution plan still matches the current problem configuration. As a consequence, problem solvers also need to apply metacognitive processes by reasoning about the quality of their own thinking. Box 4.4 illustrates the cognitive and metacognitive processes necessary for the two example units.
In general, more complex problems are more likely to require metacognitive processes in order to be solved effectively. That is, the more (interacting) elements and relations are involved in the problem configuration (task dimension 1), the more dynamic a problem is (task dimension 2), and the richer, the more unstructured and less salient the information environment is (task dimension 3), the higher the likelihood that metacognitive processes will be involved. Of all these task dimensions with their respective difficulty drivers, task dimension 2 (dynamics of the situation) is likely to contribute most strongly to metacognitive requirements in APS, since any change in the problem configuration or the information environment always requires monitoring whether one’s reasoning is still aligned with the newly evolving situation and possibly modifying one’s cognitive structures (i.e., the mental model of the problem and/or the solution plan).
First of all, the problem would need to be defined on a cognitive and a metacognitive level. From a cognitive point of view, the Dinner Preparation example requires problem solvers to search for the relevant information about the goals by browsing the map, the problem requirements and by selecting, organising and integrating the information to plan the fastest route. The Stock Market example requires problem solvers to mentally organise and integrate the information about the companies and their histories in order to plan the most promising investment strategy. From a metacognitive point of view, the Dinner Preparation example requires problem solvers to set subgoals – for example, to first drive to school, then to the store. Both problems require problem solvers to monitor their problem comprehension.
On a cognitive level, the second stage of the adaptive problem-solving process, searching for solution, would involve the search for relevant information on the map and the sticky note in the Dinner Preparation example. For the Stock Market example, it would involve a continuous search of changes in the problem statement and the environment, and an analysis of these continuous changes. On a metacognitive level, problem solvers would need to evaluate different alternatives to accomplish both goals in time in the Dinner Preparation example. In the Stock Market example, problem solvers would need to constantly look at the most promising investment alternatives continuously opening up as a function of the “daily” changes in company prices.
In the apply the solution stage, in both of the examples, the plans would then be applied to solve the problem on a cognitive level, while, on a metacognitive level the progress would be monitored.
In the following we will describe the different cognitive processes as specified in Figure 4.2.
In order to define a problem, a person needs to construct a mental model of the state of affairs described in the problem (Mayer and Wittrock, 2006[10]; Nathan, Kintsch and Young, 1992[11]). This mental model comprises information on the initial state (i.e., the problem configuration, cf. task dimensions), the goal state to be achieved, the legal operators, and the set of intervening states that are required in order to move from the initial state to the goal state; together these various states make up the problem space (Klahr, 2002[12]; Klahr and Dunbar, 1988[13]; Newell and Simon, 1972[14]; Vollmeyer, Burns and Holyoak, 1996[15]). Accordingly, items assessing mental model construction need to provide an account of the accuracy and comprehensiveness regarding the problem solver’s understanding of what the problem is about. Three cognitive sub-processes were identified in the conceptual framework (Greiff et al., 2017[6]) as contributing to mental model construction (cf. lower left corner of Figure 4.2). In the following, these will be re-introduced and discussed from an assessment perspective.
1) Selecting, organising, and integrating problem information into mental model
To define the problem, one first needs to select relevant information about the initial problem state. This means that a problem solver will need to decide for every piece of available information whether it is necessary in order to understand the current problem configuration. The exploration of information will be rather broad and involve the use and evaluation of multiple sources of information as resources with respect to their reliability, relevance, adequacy, and comprehensibility. The selected information will then need to be organised and integrated into a coherent mental representation that comprises all information that is known about the problem configuration.
The more (interacting) elements and relations a problem contains, the less salient the problem information is (e.g., because problem-irrelevant information is also included in the problem statement, task dimension 1), and the more the problem information is subject to change over time (task dimension 2), the more difficult will it be for a problem solver to select, organise, and integrate problem information into an accurate mental model. Accordingly, items can be varied along these dimensions to make this cognitive process more or less challenging for problem solvers. Items assessing mental model construction need to reflect whether a problem solver considered all relevant information for defining the problem, while ignoring irrelevant information also embedded in the storyline.
For instance, in the Dinner Preparation example, an example item could consist of a list of options describing which information is available for solving the problem (e.g., driving times to reach a grocery store, its opening hours, availability of organic food). The problem solver is then asked to tick all information categories that s/he wants more details on. In example item 1, only driving time matters for the problem definition; hence, none of the other options should be ticked. Such an item provides information on a problem solver’s accuracy in solving the problem, while at the same time delivering information on the underlying cause of problem-solving failure, namely, a problem solver’s inability to construct an adequate mental model of the problem.
2) Retrieving relevant background information
In real-world problem solving, relevant background knowledge will help an individual to distinguish between relevant and irrelevant information as well as building a coherent mental model. Memories from past problem-solving activities are one important source of background knowledge. Thus, a problem solver has to activate these memories from past problem-solving activities, which has been shown to be difficult for many problem solvers who fail to recall these past activities and do not recognise that they possess potentially helpful past experiences (Ross, 1989[16]). Moreover, many problem solvers will fail to distinguish between a problem’s structural features, which will affect how the problem can be solved and superficial or contextual features that are irrelevant to its solution (Braithwaite and Goldstone, 2015[17]; Ross, 1989[18]). Therefore, they will activate memories of past problems that are only superficially similar to the problem at hand or construct a situation model that is heavily based on irrelevant information, which will misguide the subsequent problem-solving steps.
Accordingly, a problem solver’s ability to make effective use of his or her past experiences and knowledge is likely to have a profound impact on performance in real-world problem solving. However, assessing this sub-process in the second cycle of PIAAC is problematic for various reasons. It is not known what kind of prior or expert knowledge problem solvers already possess nor can it be comprehensively assessed; moreover, expert knowledge is likely to vary between individuals and countries. The goal in the assessment is to include problems that are accessible to most people, thereby also not confounding availability of expert knowledge with a person’s ability to solve problems. Accordingly, while problems cannot be totally free of background knowledge, problems in which expert knowledge is required or where those with expert knowledge will find that the scenario conflicts with what they know should be avoided.
3) Externalising internal problem representation
Even though problem solving itself is mostly an internal process (Mayer and Wittrock, 2006[10]), it can largely benefit from externalising one’s thoughts. With respect to the construction of a situation model, problem solving will benefit from forming an external representation of a problem’s main features [e.g., in a drawing or table; (Ainsworth, Prain and Tytler, 2011[19]; Fischer, Greiff and Funke, 2012[20]; Zhang, 1997[21])].
From an assessment perspective, these externalisations can provide important insights into the way a person conceptualises a problem and into his or her misconceptions or gaps in the mental model (Lee, Jonassen and Teo, 2011[22]). Hence, it is suggested to include externalising tasks in the assessment that ask problem solvers to make a drawing or create a table, where they would need to include all the relevant features and show the relationships among those features. Because problem solvers are explicitly instructed to create externalisations, such tasks do not assess spontaneous use and hence the cognitive process underlying it. Rather, such tasks are recommended because they are instrumental to the assessment of yet another, albeit pivotal process contributing to mental model construction, namely, selecting organising, and integrating problem-relevant information in a specific format. In consequence, the same task dimensions as for selecting, organising, and integrating problem information affect the difficulty by which a mental problem representation can be externalised.
This second stage heavily relies on the mental model that was built when defining the problem (cf. middle box in Figure 4.2). The solution of the problem can be described as the sequence of steps necessary to get from the initial state of the problem to the goal state. The process of searching for a solution marks the distinction between a task and a problem. A task is present if a solution can be directly retrieved from memory and applied to the situation at hand effortlessly and without modification. A problem, on the other hand, requires that a person breaks down a problem into parts, searches for a solution among different alternatives, plans a sequence of actions, and possibly tries out different ways of reaching the goal state (Gick, 1986[23]). The search for a solution thus requires cognitive strategy knowledge on different solution methods and the metacognitive skills to handle this knowledge (Fischer, Greiff and Funke, 2012[20]; Mayer and Wittrock, 2006[10]).
Two cognitive sub-processes were identified in the conceptual framework (Greiff et al., 2017[6]) as contributing to solution search. In the following, these will be re-introduced and discussed from an assessment perspective.
1) Searching for operators in the (mind and) environment
Whereas information search aimed at defining the problem is tailored towards understanding the problem with the goal of acquiring as much knowledge as possible about the problem, the search during this stage aims at identifying possible operators that will help to make the transition from the initial state to the goal state [cf. dual space theory (Klahr and Dunbar, 1988[13]); see also (Greiff, Wüstenberg and Avvisati, 2015[24])]. Operators may be in the mind of the problem solver (i.e., cognitive actions such as adding two numbers) or they may be located in the information environment. In general, the more complex the problem configuration and the features of the information environment (task dimension 3) are, the more difficult searching for operators will become.
For instance, in the Dinner Preparation example there is one overarching operator that refers to taking the car to go grocery shopping and that has different instantiations in that the stores are differentially suited to fulfill the problem’s constraints given the driving times to them. In the Stock Market example, on the other hand, there are two operators (i.e., buying and selling stocks) with various stock options, making this problem harder than the Dinner Preparation example (cf. task dimension 1). As for the complexity of the problem configuration, the map used in the Dinner Preparation example might not be as clean as the one presented above, but might be very cluttered with unnecessary information and occlude the relevant information on driving times, in which case searching for operators would be far more difficult.
Sequences of operators that are determined prior to executing solution steps make up problem-solving plans. In the remainder, we will always talk about operators only, even though in a specific problem they might be composed into a problem-solving plan.
Searching for operators involves using appropriate devices, tools or information as well as communicating and coordinating one’s activities with other parties [cf. collaborative problem solving (OECD, 2017[8])]. Resources for locating operators may hence be located in the social, physical, or digital environment. Due to the digital assessment to be implemented in the second cycle of PIAAC, access to resources is always embedded in a digital interface for the sake of representing the problem, but this does not mean that the resources would be necessarily digital in the real world as well.
For instance, in the Dinner Preparation example, the map to read off driving times to different grocery stores might as well be a physical map; on the other hand, the diagrams illustrating the dynamics of the stock market are likely to be digital even in the real world since they need to be updated in real time.
Because the situations in which 21st century citizens solve problems often undergo change over time (cf. task dimension 2), APS requires that they constantly update their knowledge about operators.
For instance, in the second, dynamic example item of the Dinner Preparation problem the problem solver receives a message while being on the road that there has been a water leak in the designated grocery store, thereby requiring a change of plans. Similarly, in the second example item of the Stock Market problem there is constant change in the performance of the different companies that needs to be considered when buying or selling stocks.
2) Evaluating operators with respect to how well they satisfy problem constraints
There may be many operators that come up during the aforementioned search for operators, but not all of them may be legal. That is, they may fail to satisfy the constraints as expressed in the problem configuration.
For instance, while grocery store A and B may both offer the required food choices, store A may have opening hours that conflict with the requirement of being home before 10 a.m. Hence, for every potential operator it has to be determined whether it is effective in principle (i.e., enables the transition from initial to goal state) and whether it satisfies all constraints.
Evaluation of operators becomes harder for problem solvers, if there are many potential operators and many constraints to be considered (cf. task dimension 1) as well as if information on these operators is embedded in a rich and unstructured environment (cf. task dimension 3). Moreover, whereas in static problems a problem solver can rely on the operators’ (un-) suitability for problem solving once it has been evaluated, in dynamic problems, a problem solver has to continuously re-evaluate whether either the operators or the constraints have changed, thereby affecting the effectiveness of the solution.
For instance, a grocery store is no longer available due to a water leakage or a formerly well-performing company does no longer make any profit, which is why its stocks should potentially be sold rather than bought (cf. the dynamic example items of the two sample problems).
In real-world problem solving, the sub-process of evaluating operators typically includes two aspects: evaluating whether the operator is in line with the options that have been provided (e.g., is store A better suited than store B?) and evaluating whether the problem solver is capable of using the operator. The prior evaluation refers to a cognitive process since it requires reasoning about the problem. The latter requires problem solvers to consider their own or the fictitious problem solver’s resources that they could invest into applying the solution, thereby addressing metacognitive aspects. From an assessment perspective, these two aspects are difficult to disentangle in an artificial problem-solving context. For this reason, it is recommended that items in this category might be coded on both dimensions for analysis purposes (see section on assessing APS below).
During this third stage, a problem solver applies plans to solve a problem and executes the specified operators (cf. right box in Figure 4.2). This stage relies on having procedural knowledge available (Mayer and Wittrock, 2006[10]). The nature of this procedural knowledge will depend on the requirements of the problem and may, for instance, comprise algebra skills to solve equations, logical reasoning skills or other domain-specific operators. In the context of simulating problem solving for the purpose of assessing problem-solving skills, this process must be confined to selecting an operator, as problem solvers do not actually perform any actions (i.e., they do not actually go grocery shopping).
Note that the conceptual framework (Greiff et al., 2017[6]) mentioned ‘predicting the environment’ as yet another cognitive sub-process relevant to applying a solution. However, the expert group agreed that this aspect was not well defined and could not be measured so it will be dropped as a process to be included in the assessment.
A summary of the cognitive processes of APS together with a brief definition is provided in Box 4.11.
(1) Selecting, organising, and integrating information into mental model: Constructing a mental representation of the problem space (initial state, goal state, legal operators).
(2) Retrieving relevant background information: Accessing memory to retrieve background knowledge (note: assessment tasks should be designed to avoid necessity of this process).
(3) Externalising internal problem representation: Creating an external representation (e.g., drawing, table) that illustrates the problem solver’s mental model of the problem.
(1) Searching for operators in the mind and environment: Locating information about available action options that might be suited to solve the problem.
(2) Evaluating operators with respect to how well they satisfy problem constraints: Determining which of the action options will be best to reach the goal while considering all possible constraints.
(1) Applying plans and executing operators: Implementing the selected operator(s) to solve the problem.
As already mentioned, metacognitive processes are also inherently bound to the process of problem solving. However, metacognitive processes become more important to the extent that problems are more complex and difficult to comprehend, that the problems change, and that progress towards the solution becomes more difficult.
Problem-solving situations in real life may differ in whether the goal (i.e., what is to be achieved) is clear and whether only the way to get there is not yet known. In particular, there may be ambiguous problems where the goal and hence the direction to take in order to solve the problem needs to be figured out first. Moreover, especially in complex problems, that is, problems whose solutions are composed of multiple steps (cf. task dimension 1) or that require adaptation to changing circumstances due to their dynamic nature (cf. task dimension 2), the problem solver has to constantly evaluate whether the current understanding of what the problem is about still matches the current state of affairs. Thus, s/he must monitor the quality of the cognitive processes regarding the definition of the problem. Due to the fact that goal setting and monitoring problem comprehension require thinking about one’s own state (‘what do I want to achieve?’) and mental representations rather than contemplating about the problem, these processes are metacognitive in nature.
Accordingly, the conceptual framework mentioned ‘goal setting’ and ‘monitoring problem comprehension’ as two important metacognitive sub-processes (Greiff et al., 2017[6]), which are shown in the lower left part of Figure 4.2. For reasons mentioned below, the assessment framework will consider only the latter process.
1) Goal setting
Goal setting refers to defining dimensions of the problem that require a change and identifying features that characterise the state one wants to achieve. Different from the initial problem state, the definition of the goal state crucially depends on the problem solver, his/her motives, and the resources that s/he has available, and also willing to invest these, for a favourable outcome. Hence, setting goals requires reflection about one’s own cognition and motivation, thereby making it a metacognitive process.
In real life, goal setting is an important metacognitive process when solving a problem for one’s own purpose, since it gives direction and is the motivational driving force behind many actions taken towards solving the problem. However, from an assessment perspective, letting problem solvers chose among different goals would impose immense challenges in terms of scoring their performance, since problem solvers would differ in their goals, which in turn determine which solution steps would be appropriate. Hence, every goal would require its own scoring rules; moreover, problem solvers might even set goals whose achievement is not supported by the information environment made available in the assessment. For these reasons, goal setting will not be assessed in the APS tasks as the goals will be given to the problem solver in the description of the units.
For instance, in a real-world situation, a problem solver faced with the Dinner Preparation problem might actually decide to give up the initial goal of preparing a healthy dinner and get take-out food instead; in the Stock Market problem s/he might contemplate between making a quick, but potentially risky bargain versus optimising profit in the long run at a medium level, but with less risk involved.
2) Monitoring problem comprehension
An accurate understanding of the problems’ initial and goal state (i.e., “where am I and where do I need to be?”) is crucial for all subsequent problem-solving steps. Hence, problem solvers need to monitor whether their understanding of the problem is sufficient in order to find a solution to it. An accurate comprehension monitoring is especially important, since it will determine whether the process of defining the problem is adequately regulated (Nelson and Narens, 1990[25]). For instance, overconfidence in one’s understanding of the problem may lead to a premature termination of the search for problem-relevant information, whereas underconfidence may yield an inefficient construction process, where information search is continued even after all relevant information has been identified. Research on metacognitive judgements has shown that many people, especially those with little prior knowledge, make rather inaccurate judgements of their level of comprehension and rely on invalid cues when making these judgements (Bjork, Dunlosky and Kornell, 2013[26]). Notably, monitoring becomes more difficult the more information needs to be considered when constructing a mental model of the problem (task dimension 1). Moreover, dynamic problems require constant monitoring of problem comprehension, since the problem configuration may be affected by the dynamics (task dimension 2).
In contrast to some of the other metacognitive processes, monitoring problem comprehension can be assessed relatively easily by administering items in which problem solvers have to indicate whether they would require additional information on the problem before they can start solving it.
For instance, in the Dinner Preparation example, only upon taking an action (e.g., activating an additional display option by clicking on it) would the map display not only the locations of the grocery stores but also the problem solver’s location, which is necessary to infer the driving distances. Problem solvers who take this action are aware of the fact that their understanding of the problem’s initial state is incomplete and that further information is necessary. Similarly, items could ask problem solvers whether they have understood the problem and relate their answers to their actual comprehension performance. Ideally, corresponding questions should be asked by an agent or problem-solving partner, thereby embedding the assessment into the story line and making the assessment of metacognition less evident. In the Dinner Preparation example, for instance, a problem solver may respond to a friend’s question that s/he has looked up the opening hours to grocery store A, so that s/he is ready to go – thereby not accounting for the fact that driving there would take far too much time in order to be back home at 10 a.m.
Operators need to be eventually selected based on an integrated evaluation of their effectiveness and their ability to satisfy problem constraints as well as internal constraints such as the problem solver’s ability to apply an operator (cf. middle box in Figure 4.2). Because these two evaluation criteria are difficult to disentangle in an artificial problem-solving context, it is suggested to code items in this category as reflecting both, cognitive and metacognitive processes for analysis purposes. Accordingly, metacognitive evaluation is affected by the same task dimensions, in that it becomes more difficult if there are many potential operators and many constraints (cf. task dimension 1) as well as if relevant information is embedded in a rich and unstructured environment (cf. task dimension 3). Moreover, the need to constantly update the evaluation process makes dynamic problems more challenging (task dimension 2).
For instance, to assess metacognitive evaluation processes, in the Stock Market example, the problem solver could be involved in a discussion with another broker who suggests two (or more) different plans that fulfill the problem constraints to different degrees. The problem solver could be asked to continue the discussion by making a decision regarding the suggested options and also providing a reason for this decision (e.g., possible answer options: ‘both options sound good to me. I will decide spontaneously which stocks to buy’; ‘I will go for option A, because …[right/wrong reason]’; ‘I do not think that either option will work, because …[right/wrong reason]’). Such a task requires that the problem solver reflects upon the adequacy of the cognition (solution plan) rather than about the problem, which is why such a task is assumed to mainly trigger metacognitive reasoning processes. Again, an agent is introduced to not make the need for metacognitive evaluation less evident and to not trigger processes that, in the real world, would have to be carried out spontaneously.
When applying a solution, problem solvers need to evaluate whether they are making progress towards the goal and/or take actions if this is not the case (cf. right box in Figure 4.2). Especially in dynamic problems (task dimension 2) there may be changes in the problem configuration or obstacles that may affect the availability of operators, thereby making it necessary to regulate the problem-solving process and to modify existing plans in order to steer towards goal achievement.
1. Monitoring progress
When executing a problem-solving strategy, a problem solver needs to constantly monitor the degree to which progress towards solving the goal has been made. To do so, it is important that the goal has been defined in a way that clear criteria for goal achievement exist against which the current problem state can be evaluated. In the case the goal state has been achieved, the problem-solving process can be terminated. However, monitoring will often lead to the detection and interpretation of unexpected events, impasses, or breakdowns. If there is no or too little progress towards the goal state, problem solvers will need to identify possible reasons for this in order to regulate their future efforts accordingly (see below). Importantly, again test items need to be designed in a way that they do not trigger monitoring.
For instance, a variant of the Dinner Preparation example could involve a more complex task where the problem-solving process is interrupted at a point where two subgoals (e.g., doing part A of the grocery shopping and picking up the child) have already been achieved. The problem solver could be asked what next options would be. If s/he decides to drive home to prepare dinner – thereby forgetting that part B of the shopping in a different store has not yet been accomplished – this suggests poor progress monitoring. Similarly, in the Stock Market problem the goal could be to buy and sell stocks so that at a given point in time the custody account is of a certain value. If a problem solver stops interacting with the simulation prior to having reached this value, this would denote poor progress monitoring.
2. Regulating the application of operators
The process of regulating the application of operators heavily depends on progress monitoring (Bjork, Dunlosky and Kornell, 2013[26]; Nelson and Narens, 1990[25]). When progress monitoring implies that the goal has been reached, the application process can be terminated. When a problem-solving failure due to an inadequate plan has occurred, the problem solver needs to devise a modified or entirely novel plan, thereby backtracking to earlier stages of the problem-solving process. Alternatively, the plan may have been adequate, but a problem solver may have failed to carry out the involved operators, because s/he lacked the procedural knowledge. In this case, the formerly devised plan can still be used to solve the problem, but its execution needs to be optimised. Finally, modifications might be necessary because of changes in the problem configuration and its constraints (cf. task dimension 2), which would be noticed by a problem solver if s/he was good at monitoring problem comprehension.
For instance, in a variant of the Dinner Preparation example impasses may occur during execution of the plan such as that the problem solver notices that store A actually ran out of fish, which is, however, a necessary ingredient for dinner. In contrast, there could also be other things on the shopping list that are not available at this moment as well, but that are not necessary for dinner on that day. Items can assess whether problem solvers in the first scenario will plan to go to a different store instead to fetch the missing ingredient there (correct option) or drive home instead; for problem solvers in the second scenario driving home without making a detour to a second store is the correct option. In the Stock Market problem, the change from example item 1 to 2 where suddenly formerly well-performing companies now show a dip in their performance requires that the problem solver notices that these companies should no longer be considering in buying stocks.
As can be seen, regulation also requires the comparison of different solutions, which is why the latter process that had been mentioned separately in the conceptual framework is subsumed here.
3. Reflection
People who are good at solving problems have been shown to reflect upon their problem-solving experiences and abstract strategy knowledge from it that can be put to use in future problem-solving situations. Thus, problem solving is assumed to leave memory traces, which can be used in the future. This sub-process involves the development of a principal or set of principals related to general problem solving. While being an important aspect for the development of problem-solving expertise, it is unlikely that this can be assessed in the context of a large-scale assessment.
A summary of the metacognitive processes in APS together with a brief definition is provided in Box 4.17.
(1) Goal setting: Deciding upon what the to-be-achieved state is about (cannot be considered in large-scale assessments because allowing problem solvers to set their own goals would yield too many degrees of freedom).
(2) Monitoring problem comprehension: Supervising whether one’s mental model of the problem matches the current state of affairs.
(1) Evaluating operators with respect to whether they can be executed: Determining which of the action options will be best to reach the goal while considering all possible constraints.
(1) Monitoring progress: Determining whether executing operators achieves the desired outcome.
(2) Regulating application of operators: Modifying selection of operators in case the problem configuration has changed (cf. monitoring problem comprehension) or impasses have been noted (cf. monitoring progress).
(3) Reflection: Deliberating about one’s own capabilities to solve problems with the goal of abstracting knowledge from it that can be applied in the future (cannot be considered in a large-scale assessment context because it requires repeated confrontation with similar problem-solving instances).
In the previous section we have attempted to illustrate the cognitive and metacognitive processes that constitute APS referring back to the example items provided in Boxes 4.1 and 4.2, to describe how they are affected by the different problem characteristics, i.e. task dimensions, described previously, and commented on their relevance and how well they can be assessed in a large-scale context. General principles regarding the design and scoring of items for the assessment of APS will be addressed in the next section; however, here we would like to point out some important issues that arise when attempting to consider cognitive and metacognitive processes underlying APS in a large-scale assessment such as PIAAC.
(1) Not all processes are equally important to APS. For instance, once a comprehensive mental model of a problem has been constructed and the correct operators identified, applying operators from a cognitive perspective may just be a technicality. On the other hand, metacognitive processes during the latter stage can play a major role for problem-solving success, especially if the problem solver faces impasses or the problem configuration changes. Hence, it is unlikely that processes will be equally distributed across problem-solving assessment scenarios without distorting their naturally occurring distribution in real-world problem solving.
(2) Not all processes can be considered in a large-scale assessment context. Some processes such as setting a problem-solving goal and managing this goal during problem solving (i.e., making sure it is maintained and shielded against distractions) are highly relevant from a metacognitive perspective in that they can provide substantial barriers for problem solvers; however, the test-taking situation requires that the goal is already pre-defined so that its accomplishment can be unambiguously scored as correct or incorrect. As a consequence, some processes, albeit important from a conceptual perspective are not considered in the assessment framework discussed here.
(3) Not all processes can be unambiguously disentangled in a large-scale assessment context. Some processes are difficult to tease apart in an assessment situation where no “real action” is required. For instance, selection of a set of problem-solving operators and its application appear to be the same in a test, where, for example, a problem solver does not actually need to drive the route to get to a shop. As a consequence, in some cases it is suggested to merge processes into one, where no separation in an assessment context seems possible. Moreover, in real life, cognitive and metacognitive processes can usually not directly be observed and they are tightly intertwined with each other. For this reason, in some cases it is suggested to devise items that can be scored both ways, as being evidence for cognitive and metacognitive processes.
(4) An explicit assessment of processes is likely to alter their occurrence. Especially metacognitive processes may often be implicit only. Thus, they may often be better reflected in the ease of problem solving (e.g., in response times, choices NOT made, or feelings of confidence in one’s decisions) than in a ratable response to an explicit question. Moreover, explicit questions tailored towards metacognitive processes may serve as trigger for these processes, which would otherwise not have been conducted spontaneously by the problem solver. For instance, explicitly asking a problem solver whether s/he has fully comprehended the problem will most likely make him or her monitor comprehension in that situation; however, the response will not be a good indicator of spontaneous monitoring. This problem pervades research on metacognition and a lot of effort is invested into identifying more implicit measures of metacognition. For the assessment context, it is suggested to embed tasks targeting the problem solver’s metacognition as much as possible into the storyline of the problem, so that their true purpose remains concealed.
So far, we have described the different task dimensions that define an APS task and specified the various cognitive and metacognitive processes that form the basis of the problem-solving process. We also outlined how these processes translate into the actual assessment of APS. In a next step we describe the way in which the quality of the solution of an adaptive problem depends on the problem solver’s proficiency to deal with the various demands. These demands are inherent in the quantitative task dimensions (1) to (3) and their respective difficulty drivers (see right panel of Figure 4.1 and previous section). Task dimensions (4) and (5) however, are only of qualitative nature and do not contribute to the actual process of problem solving.
More specifically, whether a problem solver scores high or low in APS will depend on how s/he deals with different problem configurations (task dimension 1), the dynamics of the situation (task dimension 2), and features of the environment (task dimension 3), whose respective difficulty is determined by the assumed difficulty drivers (see Annex 4.A. for a detailed description of the difficulty drivers and how they shape the difficulty of a problem). In the following, we differentiate high from low scorers in the three relevant task dimensions to build the ground for the specification of the assumed APS proficiency levels (see right panel of Figure 4.1).
Problem solvers may score low or high when confronted with different problem configurations (cf. task dimension 1). Low and high scorers will exhibit different levels of cognitive and metacognitive processes. In any possible adaptive problem,
A low scorer:
integrates in his/her mental model only a small number of elements, relations and operations;
accesses only that extra information that is readily available and that does not require the problem solver to take extra steps (such as pushing a button in the interface);
understands only simple, clear, direct and straightforward effects and understands incompletely or incorrectly those problems that contain indirect effects, or effects generated by interactions between various elements;
identifies operators that are not salient, i.e., resources that are not readily available and identifiable as such;
handles only one task at a time, has difficulties in handling several tasks in parallel;
considers only one of several goals (end states) at the same time for a problem; only focuses on a single goal at a time; if several goals are given for the problem, needs to accomplish them one after the other (consecutively).
A high scorer:
mentally manipulates and integrates in his/her mental model a large number of elements and the relations between them;
accesses information that is not immediately and readily available by taking the extra steps needed;
understands complicated effects based on non-linear relationships, and on interaction effects between operators;
identifies resources and relationships that are not salient, i.e., are not straightforwardly defined as such, but are “hidden” in the context;
handles multiple tasks at the same time, such as controlling multiple effects towards an end goal; considers several goals at the same time, as end states of the problem-solving process, and works towards their accomplishment in parallel (not consecutively).
For instance, low scorers in the Dinner Preparation example will have difficulty in keeping in mind the various elements of the problem, and will need to continuously check on the routes and on the sticky note. They will try to only handle one task at a time and will have difficulties in handling potentially competing goals. They will use the resources that are on screen, but in case the problem will permit invoking a calculator to aid in planning the route, they may not press the button that is needed in order to make use of this resource. In the same example, high scorers will handle various goals at the same time, will use the resources available on screen while also identifying those resources that are not readily available (such as the calculator), and will keep in mind all the various elements of the problem.
Problem solvers may also score low or high when confronted with different dynamics in a situation (cf. task dimension 2). Low and high scorers will have different abilities to cope with dynamic changes during the problem-solving process. In any possible adaptive problem,
A low scorer:
identifies only some of the features that change;
identifies only the most salient features, and may miss those that are less salient;
reacts only to change that is transparent, for example when s/he is prompted that something changed;
is based in reasoning on the current situation, has difficulties in predicting future change based on past changes (or prior information);
builds incomplete or incorrect mental models of the change process (to understand how and why "things" change);
adjusts the mental model to change incompletely or incorrectly (e.g., has difficulties in making adequate changes to resolution strategy).
A high scorer:
identifies all relevant features that change, irrespective of their number, salience, transparency;
predicts likely future changes based on past changes (prior information);
constructs a mental model of the actual change (not only of the problem) (i.e., understands how and why things change);
adjusts the mental model to changes (e.g., changes resolution strategy if needed).
For instance, low scorers in the Stock Market example might not identify that the prices for all stocks have changed. They will have difficulty predicting future changes in any of the stocks, and may only be able to predict how stocks will vary in the case of those that have a very transparent and univocal past evolution. They may build incomplete or incorrect mental models of the problem and its dynamics. In the same example, high scorers will quickly identify that change takes place in all the stocks, on a “daily” basis, will correctly predict future changes based on prior evolutions of these stocks, will build a correct mental model of the problem and its dynamics. Based on these abilities to constantly monitor the problem solution and to react to changes, they will easily adjust this mental model to any supplementary change, if induced, i.e., they will adapt to the new circumstances.
Finally, problem solvers may score low or high when confronted with different features of the environment (cf. task dimension 3). Low and high scorers will have different abilities. In any possible adaptive problem,
A low scorer:
works with only one or a small number of variables about the state of the environment;
integrates only one or a small number of variables from the environment in the conceptualisation of the problem;
filters out distractors with difficulty and incompletely; is distracted by irrelevant information; continuously manipulates variables that have no effect on anything;
is distracted by background material; does not recognise distractors; continues to consider all material, even if not relevant (e.g., reads through all the update notes);
interacts with structured environments, but interacts in an inefficient (and sometimes not meaningful) way with environments that are not structured.
A high scorer:
mentally manipulates and integrates in mental models a large number of variables about/from the environment;
integrates “the environment” (and its variables) in the conceptualisation of the problem;
filters out distractors (irrelevant information);
focuses on relevant variables from the environment, is not distracted by stimuli that are external to the task or are irrelevant for the task;
recognises the distracting background material;
interacts efficiently with unstructured environments (i.e., structures environment, constructs mental model of environment).
For instance, low scorers in the Dinner Preparation example will only integrate a small amount of the available information in their conceptualisation of the problem. They will be distracted by irrelevant background information and will operate the map in an inefficient way. In the same example, high scorers will integrate a large number of only relevant information into their mental model of the problem. They will recognise changes in the environment and will interact with the map efficiently even if the map would be cluttered with irrelevant information.
The described core task characteristics and their difficulty drivers form the basis upon which the high and low scorers of APS can be described. However, the final score of problem solvers is not directly interpretable, unless related to their proficiency level. Using the task characteristics and difficulty drivers identified in the framework, the expert group will define levels of proficiency and explain what each level means. In other words, what are the specific components of APS that can be performed with proficiency by a high scorer, but cannot be performed by an average scorer, and what are those components that are performed by average scorers and cannot be performed by low scorers? Further, what are the specific components that are expected to be performed even by low scorers?
The proficiency levels will define the scale and will provide a useful way to understand the progression of APS skills. These proficiency levels are associated with the competency of problem solvers, but are also associated with the complexity of items, i.e., the specific components of APS skills that are required by each progressively more difficult items. In Table 4.1, we present a preliminary proposal for APS skills, divided into three proficiency levels. This proposal is based on theoretical considerations about how proficiency may be distributed in the population with the task dimensions as well as the cognitive and metacognitive processes outlined in this framework in mind. This proposal is not based on actual data, and analysis of the main study data will require changes in the number of levels as well as the specific descriptions of those levels of the proficiency scale. The table contains four descriptions for each proficiency level:
a) a general statement of that proficiency level, that can help readers to quickly understand each level;
b) a description of how problem solvers at that specific proficiency level deal with (i.e., adapt to) dynamically changing problems – which is, after all, the basis of adaptive problem solving;
c) a description of the various cognitive processes that are typical for that proficiency level;
d) a description of the various metacognitive processes that are typical for that proficiency level.
The previous section presented the domain of APS and outlined the task dimensions, difficulty drivers, the cognitive and metacognitive processes involved in APS, and the proposed proficiency levels. These elements define the overall, conceptual framework of APS and form the basis for the development of test units and their corresponding items. Ensuring a sufficient match between the conceptual framework and what the APS units and items assess is critical to the crafting of a validity argument. Hence, achieving the greatest possible coverage of the task dimensions and APS processes is the key goal for the test development. The assessment of APS in the second cycle of PIAAC will emphasise the dynamic nature of problem-solving situations as defined earlier and will present problem solvers with newly developed test units that will be suited in information-rich environments.
This section provides an overview of the anchoring of the APS units in the task dimensions outlined in the previous section (see also Figure 4.1), describes overarching test design principles, and explains the scoring and capturing of data beyond item responses that will form the basis of the different proficiency levels.
The APS units will represent tasks that are comprised of multiple items (i.e., questions). In this sense, an APS unit contains the following key elements: a task stimulus (e.g., introduction to the task, description of functionalities of interactive elements) and multiple items that require the problem solver to adapt to changing situations. The design of the items within a unit will be guided by (1) the task dimensions, and (2) the cognitive and metacognitive processes, as described in previous sections.
Concerning (1), the following five task dimensions formed the development of APS items: problem configuration (i.e., the initial problem setup and goal states), dynamics of the problem situation (i.e., the degree to which the problem situations and its constraints change over time), the features of the environment (i.e., construct-relevant features of information and resources), the types of information sources (i.e., physical, social, and digital), and the contexts (i.e., personal, social community, and work; as defined in the first PIAAC cycle (OECD, 2012[4]). Each and every unit will be mapped onto these five dimensions. However, as we assume information environments and problem contexts in real life to be not equally distributed (cf. section defining APS), we propose to target slightly different proportions of all the problems to be placed in the various environments and contexts as displayed in Table 4.2.
Concerning (2), all items within the APS units are located within the framework of cognitive and metacognitive processes. These processes comprise defining the problem, searching for a solution, and applying the solution (see section on cognitive and metacognitive processes in APS and Figure 4.2). For a specific item, these three processes may be required, both on the cognitive and metacognitive side. Given that the cognitive and metacognitive processes are intertwined, a clear separation of these processes – for instance, in the form of empirically distinct indicators or scores – is hardly possible. As a consequence, the APS items may require problem solvers to engage in multiple processes rather than a single process within the APS framework. Besides, to successfully solve a problem that is subject to change over time, problem solvers have to understand the problem situation and develop a mental model about it (Ericsson and Pool, 2016[27]). Ultimately, the processes of understanding the problem form the basis for all subsequent processes of search and applying a solution. This dependence between the three processes of APS results in the anchoring of the APS items in multiple cognitive or metacognitive processes. However, for a given item, some processes may be more pronounced than others and these items will be assigned to the respective, dominant processes.
The proposed distribution of the three main processes in the APS item pool is shown in Table 4.3.
As stated earlier, for reasons of test fairness and validity, reference to expert knowledge should be avoided from an assessment perspective. Accordingly, items should be designed so that information on operators should be provided through them. In this regard, the Stock Market example is potentially a borderline case, since experience with buying and selling stocks may be very limited in some populations. To make this scenario accessible to problem solvers, it has to be simplified compared with its real-world counterpart.
The APS units will be administered on tablets and allow problem solvers to interact with the problem and information environments directly. The technology-based test administration further enables the implementation of problem situations that change over time or make new sources of information available to the problem solver during the problem-solving process. Moreover, in selected items and units, log-file data of specified actions may be used to inform the development of the described APS proficiency levels.
For the main study, the APS assessment will be administered together with the assessments of numeracy and literacy. Participants will be randomly assigned to two of the three domains. For these assessments, an adaptive test design is anticipated so that each participant does not work on all items within the respective domains. The adaptive testing procedure will be based on units, depending on the dependencies between items within a task. At the beginning of the assessment, participants will be assigned to one of three pathways based on their initial performance on a locator test of their literacy and numeracy skills. This design combines adaptive testing with multi-stage testing and is aimed at maximising the information about the participants gained from the assessments (OECD, 2013[28]).
The design of the APS units and items contains several elements that facilitate the assessment of adaptive problem solving and ensure the fairness of the test:
a) Explicitness of change: In some APS tasks, change in the problem situation is not made explicit so that problem solvers will have to recognise these changes. This design element is construct-relevant as it stimulates metacognitive processes of reflecting on the problem situation and initial mental models given the changes in the environment. This element, however, increases the difficulty of the items and is thus used sparsely. In fact, most APS items make explicit the changes in the problem environment.
b) Rescue elements: The design of APS units as a sequence of items that gradually introduce changes to the problem environment may create dependencies between items. In other words, if a problem solver does not succeed in one item, s/he may have a disadvantage in solving subsequent items. To circumvent this problem and to ensure the comparability of items among problem solvers, the APS units will contain rescue elements. These elements represent a certain decision or problem solution to the problem solver that are based on a previous item. However, these elements do not evaluate the problem solvers actual responses on previous items but are entirely independent from the correctness of these responses. In this sense, all problem solvers receive the items with these rescue elements to ensure test fairness.
c) Gradual introduction of changes: At the outset of an APS task, problem solvers will be presented with a static problem. The subsequent items will gradually unfold and introduce the dynamics of the problem situation. These changes are mostly made explicit (see above) and may be of discrete or continuous nature. The initial, static tasks will ensure that a measure can be established that forms the baseline for problem solvers’ performance on subsequent items.
The APS units and items will be designed in a way that the level of literacy required to successfully solve the problem is kept minimal [see Greiff et al. (2017[6])]. To accomplish this, the stimulus material and item statements will be formulated briefly and as clearly as possible, except when the complexity of the materials is construct-relevant (e.g., amount of distracting information for information-rich problems). Furthermore, APS units will not present problem solvers solely with written text but will also provide information in tables, schemes, diagrams, and interactive simulations to reduce the reading load and exploit the advantages of multiple representations of testing material. At the same time, a certain level of literacy will be required to successfully solve the problems, especially in order to understand the problem situation and the information material. How APS distinguishes from other core abilities, namely literacy, numeracy and ICT, will be described in detail in the following section.
Along similar lines, the technology-based administration of the APS assessment in the second cycle of PIAAC will require basic skills to deal with ICT. Whether problem solvers are likely to have these skills will be determined in the tablet training. It must be noted that the required level of ICT skills will be kept low, and APS units will mainly demand the navigation through items, switching between two to three information pages, selecting response options, inserting short responses into text boxes, and manipulating well-defined variables by operating a small number of buttons or sliders. In fact, participants will only need to tap on a selection with a stylus or finger, use drag and drop, and highlight (underline) text. To further assist problem solvers in maneuvering through the APS units, a tablet tutorial will be provided at the outset of the PIAAC test administration. This tutorial supports participants in familiarising themselves with the tools to navigate through the tests. Moreover, PIAAC Cycle 2 chose to administer the performance tests on tablets to facilitate an intuitive handling of the test environment (OECD, 2018[29]).
The main purpose of the APS assessment is to assess problem solvers’ capacity to successfully solve dynamic problems. To capture the broad variation of proficiency in the PIAAC population of 16- to 65-years old participants, APS units and items will need to vary with respect to their item difficulty. To achieve this, the items will be distributed along the difficulty drivers as described in detail earlier in this chapter (see also Table 4.A.1. in the Annex).
As the second cycle of PIAAC focuses on the adaptive component of problem solving, the manipulation of the dynamics of the problem situation is key to the item development. At the same time, the elements a problem situation is comprised of (i.e., its configuration and the characteristics of the information sources) also play an important role in driving item difficulty. Furthermore, in some instances, the instructions to solve a problem are not fully provided, for instance, when problem solvers interact with a simulation and thereby acquire knowledge about its functionalities. This design feature is relevant to the measurement of APS, as it presents problem solvers with an actual problem situation and triggers metacognitive processes to develop and refine a mental model about the problem situation (i.e., in this case, the functionalities of the simulation).
As noted earlier, metacognition plays an important role in the APS processes, especially as problem solvers monitor their comprehension of the problem, evaluate operators and solution plans, and monitor their progress towards the goal. As these metacognitive processes interact directly with the cognitive processes during problem solving, disentangling them from the measurements of cognition poses a challenge. For example, evaluating one’s personal resources and capabilities is an aspect of metacognition that cannot be addressed in a survey such as PIAAC that does not report individual results. Moreover, test questions that are aimed at making problem solver’s understanding of a problem explicit by asking them “How well do you think you understood the problem?” seem artificial (and may lack face validity) and could prompt problem solvers’ responses in following items or even units.
To obtain some measures of metacognition, the APS assessment provides implicit and explicit indicators that can be derived from item scores or log-file data. For instance, in some APS items, log-file data can provide information whether a problem solver accessed certain information sources (i.e., navigation behaviour). This information may serve as an indicator of metacognitive processes to evaluate certain information sources during “searching for a solution” – in some instances, it may also indicate whether problem solvers reconsider certain pieces of information during the “applying the solution” stage. In general, the navigation behaviour may indicate certain metacognitive strategies to solving the problem.
Next to these implicit measures, some APS items explicitly assess metacognition. For instance, at the end of a problem-solving process, problem solvers may be asked to evaluate a given solution to the problem according to pre-defined criteria. Additionally, problem solvers may be asked to evaluate certain problem-solving strategies according to their efficiency and applicability. Mastering the latter is indicative of problem solvers’ metacognitive strategy knowledge [e.g., (Antonietti, Ignazi and Perego, 2000[30]; Efklides and Vlachopoulos, 2012[31])]. Overall, the APS assessment will contain both explicit and implicit measures of metacognition. However, given the nature of metacognitive processes and the challenges inherent in their assessment, metacognitive processes, albeit essential to APS, will not form the major focus of the assessment itself.
For the two example units, metacognitive processes could be traced using several measurement approaches. These approaches are described below (Box 4.21).
Metacognition in the Dinner Preparation example is implicitly assessed in item 2 only. It can be assessed whether problem solvers adapt their initial solution according to the new information. Metacognition in the Stock Market example is not assessed explicitly in this unit, but implicitly. Item 2 requires the problem solvers to understand that the previously employed and efficient solution is not working any more, due to changes in the environment. They will need to detect the impasse, to understand the reason, and to adapt decisions accordingly.
Each APS item will be scored according to criteria that define the correctness of the responses. For most items, the answers provided by problem solvers (e.g., by selecting a response among given response options, or by selecting certain sets of values for a set of variables) are scored dichotomously as either correct (code: 1) or incorrect (code: 0). Missing responses are also coded (code: 9). For some items, the solution must fulfill multiple criteria so that partial credits may be given. Nevertheless, the item scoring is aimed at providing scores that allow the application of parsimonious item response models – hence, a dichotomous scoring is preferred.
To exemplify the item scoring, Box 4.22 describes how problem solvers’ responses are scored in the two sample units.
While the preferred scoring method is to dichotomise problem solvers’ performance in items (correct vs. incorrect), in some instances, the scoring may allow for partial credits. Partial credits will be used only if the different scores represent qualitatively different responses or processes. Field trial data will be used to evaluate the appropriateness of partial credit scoring for the main study. The key criterion for considering partial credit scores is therefore their construct-relevance.
As noted earlier, the cognitive and metacognitive processes stimulated by the APS items are intertwined, and, in most APS units, their indicators cannot be separated clearly. As a consequence, the scaling of problem solvers’ APS performance will not result in two distinct APS dimensions representing the two types of processes. Along the same lines, the APS assessment in the second cycle of PIAAC does not aim for distinguishing the three processes, define the problem, search for a solution, and apply the solution empirically into three correlated APS dimensions. The reporting of the APS performance scale will therefore most likely not be along these processes, and will most likely result in a single APS scale.
Given the variation of APS items and units across the task dimensions, a possible distinction between dimensions may be based on the dynamics of the situation (e.g., static vs. dynamic items) or the inclusion of metacognitive processes (e.g., items requiring metacognition vs. items not requiring metacognition to a substantial degree). These possible dimensions will, however, not be made psychometrically explicit, for instance, in the form of separate APS scores - they may be used to craft a validity argument for the APS assessment.
Item 1: “Plan the fastest route to accomplish these goals. Keep the time constraints in mind”
Code 1: Route from Home to School to Shop A selected
Code 0: Other responses
Code 9: Missing
Item 2: “Adapt your chosen route to accomplish the rest of the goals for the day. Keep the time constraints in mind”
Code 1: Route correctly adapted School to Shop A to Home OR School to Shop C to Home
Code 0: Other responses
Code 9: Missing
Item 1: “Based on the information provided, which shares should you buy or sell in order to maximise your chance for higher profits next day”
Code 1: The problem solver uses the correct investment pattern to maximise profit
Code 0: Other responses
Code 9: Missing
Item 2: “Based on the information provided, which shares should you buy or sell in order to maximise your chance for higher profits next day”
Code 1: The problem solver uses the correct investment pattern to maximise profit
Code 0: Other responses
Code 9: Missing
Next to the scoring of problem solvers’ item responses that they submitted directly after completing an item, log-file data are used to retrieve and evaluate certain behaviours while solving a problem. These data may include the sequence of actions, whether or not certain elements in the problem environment were selected or accessed, and the time spent on the tasks. Whereas the latter may be useful to identify test-taking effort or aberrant responses (Goldhammer, Martens and Lüdtke, 2017[32]; Marianti et al., 2014[33]), the former can provide insights into metacognition. Some of these behaviours may even be scored.
For instance, whether or not a problem solver makes use of a certain information source (e.g., a hyperlink to a text that contains relevant information) may be an indicator of both cognitive and metacognitive processes of search for information and understanding the problem. If, indeed, a problem solver does not access this information, the problem-solving success may only be limited due to missing information or a resultant solution that does not fully meet all criteria. For instance, considering the information about time restrictions in the Dinner Preparation example is essential to the APS performance. In this sense, log-file data aid the analysis or the description of problem-solving performance within the task. Overall, log-file data may provide data beyond the mere correctness of an item response to indicate test-taking behaviour and, in some cases, metacognitive processes.
Up on this point we have described the importance of APS in today’s changing world, defined and explained what is meant by APS and have introduced the core dimensions that form an adaptive problem before concretising how APS can be assessed. It is, however, also crucial to theoretically describe what differentiates APS from other core competencies, since APS addresses a set of higher-order cognitive skills that are related to other domains, such as literacy, numeracy, or digital competencies. For example, APS often relies on verbal and pictorial representations that the person has to be able to parse in order to acquire information that is needed to solve the problem. The Dinner Preparation example presented in Box 4.1 involves written instructions, a map and a sticky note; and the Stock Market example (Box 4.2) has a set of tables and graphs. Regardless of their ability to adaptively solve the problem, problem solvers need to be able to parse and make sense of the information in these representations, which is arguably related to their literacy skills.
In the present section we discuss the status of APS in relation to some of these overlapping domains. We review the similarities and differences between the domains and we list a number of distinctive features that differentiate APS as a construct. We also explain how the design of APS task intends to reduce the potential influence of these related domains.
The word literacy is sometimes used in the restricted sense of "knowing to read and write". However, over the past 20 years, the definition has been expanded to reflect abilities related to the functional use of documents, which reflects the growing pervasiveness of reading and writing in post-industrial societies (Rouet and Britt, 2017[34]). In turn, the functional use of a document often entails forms of reasoning that amount to problem solving (for instance, making a decision about which product to purchase based on two descriptions of competing products). Therefore, it is important to clarify the boundaries between APS and literacy.
Literacy is bound to overlap with most areas of assessment because most assessment procedures rely on natural language communication. Put in a concrete way, whatever the testing domain, participants always have to read and comprehend written instructions, questions, and stimuli in order to demonstrate their ability in the respective domains. Completing APS tasks is no exception to this rule as a minimum level of literacy is required to solve an adaptive problem. However, several dimensions contribute to making APS a distinct domain. Some of the main dimensions are the types of representations used in the testing materials, the level of problem specification, and the dynamics of the environment (Table 4.4).
Note: Other dimensions that are specific to literacy are not represented here.
In a reading literacy assessment, materials include by definition written texts sometimes with other, adjunct representations such as a graph or a picture. Materials included in the APS assessment will encompass a range of stimuli, some of them almost entirely non-verbal. In addition, reading literacy tasks are meant to be well defined, whereas some problem-solving tasks are intentionally left partially implicit. Finally, a reading literacy environment involves one or several passages of text that are provided at the onset and remain the same throughout the task. APS environments may change with time as a function of a range of factors including the problem solvers actions.
In order to maximise the specificity of APS assessment, care will be taken to develop tasks that do not pose significant challenges from a reading literacy perspective. For example, for those APS tasks that include written texts, these will be limited to short and simple passages in combination with non-verbal representations. For instance, the Dinner Preparation example involves a simple narrative and a short list of things to do. The Stock Market example contains no extended text passage. Difficulty in this unit clearly comes from the need to handle multiple dynamic sources of mostly non-verbal information, which arguably makes it distinct from a reading literacy task.
Throughout the second half of the 20th century, digital devices (e.g., mainframes, computers, laptops, iPads and smartphones) have spread rapidly and profoundly in developed societies. People's ability to handle these devices has had an increasingly important impact on their access to employment, civic participation and their personal life in general. Numerous calls have been made for governments and other organisations to assess people's ability to use computers and related devices, under various constructs ranging from "ICT literacy" (Eshet-Alkalai, 2004[35]), to "digital competence" [Ferrari (2013[36]), to cite just a few].
Proficient use of digital devices involves knowing how to perform basic operations such as opening a folder, naming a file or updating a piece of software, but also to perform more complex tasks such as managing a photo or e-mail archive, addressing issues with system or application compatibility, or contacting a customer service in order to obtain information. Surveys and assessments addressing people's use of computers have typically included tasks at various levels of difficulty.
Digital devices are used to perform an ever-increasing range of tasks, including non-routine ones. In addition, these devices are typically dynamic and interactive, offering numerous opportunities for adaptation. Therefore, it is relevant to ask how APS differs from an assessment of digital competence. Table 4.5 highlights two of these dimensions.
1. Here the phrase "Digital competence" subsumes the various constructs and frameworks that have addressed people's knowledge of and proficiency at using digital devices.
Source: Adapted from Greiff et al. (2017[6]).
Firstly, some APS tasks will require the use of digital devices and applications whereas others do not. For instance, the Dinner Preparation task uses a static map even though it could be set in the context of embarked information systems such as a GPS editor. The Stock Market example also uses simple representations although a spreadsheet application could be of some use to people with a high level of digital competence. Ideally, prerequisites in terms of digital competence should remain minimal in an assessment of APS.
Secondly, APS tasks involve non-trivial goals whereas assessments of digital competence may involve routine as well as non-routine uses. For instance, in the Stock Market example, information about two companies changes during the completion of the task, requiring the problem solver to adjust their investment decisions accordingly. The demand on ICT use is minimal, although the complexity in terms of goal management is expected to be moderate to high.
The prevalence of problem solving in ICT use has prompted efforts to understand what participants can or cannot do when faced with tasks involving non-routine uses of technology. Therefore, the assessment of traditional competencies, namely literacy and numeracy, was augmented by an assessment of individuals’ ability to effectively use information and communications technology to solve problems [i.e. PS-TRE; (OECD, 2012[4])]. The domain was defined as:
"using digital technology, communication tools and networks to acquire and evaluate information, communicate with others and perform practical tasks." (OECD, 2012, p. 47[4])
Since the assessment of APS will also use technology-rich environments in which the problem is embedded, it is important to also compare the APS with the assessment of problem solving in the first cycle of PIAAC.
PS-TRE focused on "non-routine" uses of technology, i.e., those in which individuals have to set up ad hoc goals and plans, and to access and use information presented on the computer. Thus, the assessment of PS-TRE in the first cycle of PIAAC was an assessment of problem-solving skills as they apply to technology-rich environments. The stimuli were presented in the context of simulated web browser, e-mail, and spreadsheet environments. The tasks required the participants to access information relevant to their needs by using the tools available in the computer applications(s). Depending on the task, one or several applications were available. For example, a task might require respondents to use a web-based reservation system to manage requests to reserve a meeting room and send e-mails to decline requests if reservations could not be accommodated. The environment typically included more information than was needed to solve the task.
In contrast, the assessment of APS in the second cycle of PIAAC will not systematically assess the proficiency of problem solvers to interact with technology-rich environments. Instead, APS focuses on problem solvers' ability to adapt to changing conditions, such as a change in the problem definition, unexpected difficulties when taking a path towards a solution, or simply a dynamic environment that changes in more or less predictable ways as a function of time (see section defining APS). Proficient problem solvers are expected to be able to detect and manage those changing conditions. This may include giving up an initial path towards a solution, backtracking to previous stages in the problem-solving process, and/or incorporating the new conditions into one's strategy to solve the problem.
In summary (Table 4.6), APS tasks will involve a variable amount of information, and most tasks will implement a constraint to adapt to changing conditions.
1. Both PIAAC Cycle 1 and 2 use simulations of mainstream computer applications such as a spreadsheet or a web browser. The simulations typically feature a limited set of functions (for instance, a sort function on the spreadsheet), which are presented in standard ways so as to maximise transfer from real-life applications.
In this section we have examined the relationship of APS with three related constructs and domains: literacy, digital competence and PS-TRE. Because of their breadth and the universal use of written language to convey instructions and stimuli, the domains are bound to overlap. However, we have listed a few aspects that make APS distinct from the other domains. One aspect is the diversity of the representations used in the problem-solving environment; another is the non-trivial and sometimes partly implicit nature of tasks. Finally, APS uniquely implements environments that are dynamic and interactive.
The domain of competencies that is implemented in APS reflects current demands on individuals, both at the workplace and in society in general. In particular, it addresses the need for individuals to adjust to conditions that may change at a rapid pace and sometimes in unpredictable ways.
[19] Ainsworth, S., V. Prain and R. Tytler (2011), “Drawing to learn in science”, Science, Vol. 333/6046, pp. 1096-1097, http://dx.doi.org/10.1126/science.1204153.
[30] Antonietti, A., S. Ignazi and P. Perego (2000), “Metacognitive knowledge about problem-solving methods”, British Journal of Educational Psychology, Vol. 70/1, pp. 1-16, http://dx.doi.org/10.1348/000709900157921.
[5] Autor, D., F. Levy and R. Murnane (2003), “The skill content of recent technological change: An empirical exploration”, Quarterly Journal of Economics, Vol. 118/4, pp. 1279-1334, https://doi.org/10.1162/003355303322552801.
[26] Bjork, R., J. Dunlosky and N. Kornell (2013), “Self-regulated learning: Beliefs, techniques, and illusions”, Annual Review of Psychology, Vol. 64/1, pp. 417-444, http://dx.doi.org/10.1146/annurev-psych-113011-143823.
[17] Braithwaite, D. and R. Goldstone (2015), “Effects of variation and prior knowledge on abstract concept learning”, Cognition and Instruction, Vol. 33/3, pp. 226-256, http://dx.doi.org/10.1080/07370008.2015.1067215.
[31] Efklides, A. and S. Vlachopoulos (2012), “Measurement of metacognitive knowledge of self, task, and strategies in mathematics”, European Journal of Psychological Assessment, Vol. 28/3, pp. 227-239, https://doi.org/10.1027/1015-5759/a000145.
[27] Ericsson, K. and R. Pool (2016), Peak: Secrets from the New Science of Expertise, Eamon Dolan/Houghton Mifflin Harcourt, New York.
[35] Eshet-Alkalai, Y. (2004), “Digital literacy. A conceptual framework for survival skills in the digital era”, Journal of Educational Multimedia and Hypermedia, Vol. 13/1, pp. 93-106.
[3] Felstead, A. et al. (2013), Skills at Work in Britain: First Findings from the Skills and Employment Survey 2012, Centre for Learning and Life Chances in Knowledge Economies and Societies, Institute of Education, London.
[36] Ferrari, A. (2013), DIGCOMP: A Framework for Developing and Understanding Digital Competence in Europe, European Commission, Joint Research Center, Institute for Prospective Technological Studies, Seville, http://publications.jrc.ec.europa.eu/repository/handle/JRC83167.
[20] Fischer, A., S. Greiff and J. Funke (2012), “The process of solving complex problems”, The Journal of Problem Solving, Vol. 4/1, pp. 19-42, http://dx.doi.org/10.7771/1932-6246.1118.
[9] Funke, J. (2010), “Complex problem solving: A case for complex cognition?”, Cognitive Processing, Vol. 11/2, pp. 133-142, http://dx.doi.org/10.1007/s10339-009-0345-0.
[23] Gick, M. (1986), “Problem-solving strategies”, Educational Psychologist, Vol. 21/1-2, pp. 99-120, http://dx.doi.org/10.1080/00461520.1986.9653026.
[32] Goldhammer, F., T. Martens and O. Lüdtke (2017), “Conditioning factors of test-taking engagement in PIAAC: An exploratory IRT modelling approach considering person and item characteristics”, Large-scale Assessments in Education, Vol. 5/18, http://dx.doi.org/10.1186/s40536-017-0051-9.
[6] Greiff, S. et al. (2017), “Adaptive problem solving: Moving towards a new assessment domain in the second cycle of PIAAC”, OECD Education Working Papers, No. 156, OECD Publishing, Paris, https://dx.doi.org/10.1787/90fde2f4-en.
[24] Greiff, S., S. Wüstenberg and F. Avvisati (2015), “Computer-generated log-file analyses as a window into students’ minds? A showcase study based on the PISA 2012 assessment of problem solving”, Computers & Education, Vol. 91, pp. 92-105, http://dx.doi.org/10.1016/j.compedu.2015.10.018.
[12] Klahr, D. (2002), Exploring Science: The Cognition and Development of Discovery Processes, MIT press, Cambridge.
[13] Klahr, D. and K. Dunbar (1988), “Dual space search during scientific reasoning”, Cognitive Science, Vol. 12/1, pp. 1-48, http://dx.doi.org/10.1207/s15516709cog1201_1.
[22] Lee, C., D. Jonassen and T. Teo (2011), “The role of model building in problem solving and conceptual change”, Interactive Learning Environments, Vol. 19/3, pp. 247-265, http://dx.doi.org/10.1080/10494820902850158.
[1] Levy, F. and R. Murnane (2006), “Why the changing American economy calls for twenty-first century learning: Answers to educators’ questions”, New Directions for Youth Development, Vol. 2006/110, pp. 53-62, http://dx.doi.org/10.1002/yd.167.
[33] Marianti, S. et al. (2014), “Testing for aberrant behavior in response time modeling”, Journal of Educational and Behavioral Statistics, Vol. 39/6, pp. 426-451, http://dx.doi.org/10.3102/1076998614559412.
[10] Mayer, R. and R. Wittrock (2006), “Problem solving”, in Alexander, P. and P. Winne (eds.), Handbook of Educational Psychology, 2nd edition (pp. 287-304), Erlbaum, Mahwah, NJ.
[11] Nathan, M., W. Kintsch and E. Young (1992), “A theory of algebra-word-problem comprehension and its implications for the design of learning environments”, Cognition and Instruction, Vol. 9/4, pp. 329-389, http://dx.doi.org/10.1207/s1532690xci0904_2.
[2] National Research Council (2012), Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century, National Academies Press, Washington, DC, http://dx.doi.org/10.17226/13398.
[25] Nelson, T. and L. Narens (1990), “Metamemory: A theoretical framework and new findings”, in Bower, G. (ed.), The Psychology of Learning and Motivation: Advances in Research and Theory, Vol. 26, Academic Press, San Diego, http://dx.doi.org/10.1016/s0079-7421(08)60053-5.
[14] Newell, A. and H. Simon (1972), Human Problem Solving, Prentice-Hall, Englewood Cliffs, NJ.
[29] OECD (2018), The use of tablets for collecting data in the 2nd cycle of PIAAC. 20th Meeting of the PIAAC Board of Participating Countries, 16-17 April 2018, OECD Headquarters, OECD, Paris.
[8] OECD (2017), “PISA 2015 collaborative problem‑solving framework”, in PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic, Financial Literacy and Collaborative Problem Solving, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264281820-8-en.
[7] OECD (2014), PISA 2012 Results: Creative Problem Solving (Volume V): Students’ Skills in Tackling Real-Life Problems, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264208070-en.
[28] OECD (2013), Technical Report of the Survey of Adult Skills (PIAAC), https://www.oecd.org/skills/piaac/_Technical%20Report_17OCT13.pdf.
[4] OECD (2012), Literacy, Numeracy and Problem Solving in Technology-Rich Environments: Framework for the OECD Survey of Adult Skills, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264128859-en.
[16] Ross, B. (1989), “Distinguishing types of superficial similarities: Different effects on the access and use of earlier problems”, Journal of Experimental Psychology: Learning, Memory, and Cognition, Vol. 15/3, pp. 456-468, http://dx.doi.org/10.1037/0278-7393.15.3.456.
[18] Ross, B. (1989), “Remindings in learning and instruction”, in Vosniadou, S. and A. Ortony (eds.), Similarity and Analogical Reasoning, Cambridge University Press, Cambridge, CA.
[34] Rouet, J. and M. Britt (2017), Literacy in 2030. Report commissioned by the OECD’s Education 2030 project, OECD, Paris.
[15] Vollmeyer, R., B. Burns and K. Holyoak (1996), “The Impact of Goal Specificity on Strategy Use and the Acquisition of Problem Structure”, Cognitive Science, Vol. 20/1, pp. 75-100, http://dx.doi.org/10.1207/s15516709cog2001_3.
[21] Zhang, J. (1997), “The nature of external representations in problem solving”, Cognitive Science, Vol. 21/2, pp. 179-217, http://dx.doi.org/10.1207/s15516709cog2102_3.