Dave Tout
Australian Council for Educational Research
Isabelle Demonty
University of Liège
Javier Díez-Palomar
University of Barcelona
Vince Geiger
Australian Catholic University
Kees Hoogland
HU University of Applied Sciences Utrecht
Terry Maguire
National Forum for the Enhancement of Teaching and Learning
This chapter presents the framework for conceptualising and assessing adult numeracy and developing a reporting scale for the direct assessment of numeracy as part of the OECD’s Programme for the International Assessment of Adult Competencies (PIAAC Cycle 2). Numeracy as described here refers to adults’ skills in accessing, using and reasoning critically with mathematical content, information and ideas represented in multiple ways in order to engage in and manage the mathematical demands of a range of situations in adult life. The framework describes the conceptual and theoretical foundations behind the adult numeracy construct and the principles applied for assessing numeracy in PIAAC and the distribution of the numeracy assessment items by a range of task characteristics.
This chapter presents the framework for conceptualising numeracy as part of the OECD’s Programme for the International Assessment of Adult Competencies (PIAAC Cycle 2). It builds on conceptual and assessment frameworks and cumulative wisdom developed in connection with prior surveys of adult skills, primarily the first cycle of the PIAAC, the Adult Literacy and Lifeskills project (ALL) and the International Adult Literacy Survey (IALS), but also surveys of school-age students e.g., the Programme for International Student Assessment (PISA).
This chapter has six separate sections, followed by references:
The assessment of numeracy in PIAAC
Conceptual and theoretical foundations
Numeracy assessment construct in PIAAC Cycle 2
Operationalisation of the PIAAC numeracy assessment
Relationship between PIAAC and PISA
Numeracy components.
The first section provides a summary of the 2017 review of the PIAAC numeracy framework and assessment, gives an indication of some other conceptual issues considered, and includes a brief rationale for assessing numeracy in PIAAC. The second section addresses the conceptual construct for numeracy. The third section addresses the assessment construct, and describes the different dimensions of numeracy being assessed, including contexts, expected responses, content areas of mathematical information and ideas, and representations, as a way of operationalising the numeracy construct for scale development. It also discusses enabling processes, both cognitive and non-cognitive or dispositional, which underlie numerate behaviour. The fourth section discusses the operationalisation of the construct of numeracy in a large-scale assessment such as PIAAC and how this is affected by many factors that determine and shape the extent to which the theoretical construct can be fully addressed by the actual collection of items used in the direct assessment. It describes what can, and what cannot, be assessed in PIAAC Cycle 2. Subsequent sections comment on differences and commonalities between PIAAC's numeracy assessment and the related construct of mathematical literacy assessed in the Programme for International Student Assessment (PISA); and another section is dedicated to describing the new numeracy components assessment.
An assessment framework and construct is required for any valid assessment. The assessment framework provides a definition of the domain and the features of the construct. Such a framework usually includes:
the background, purpose and rationale for, and description of, the assessment programme based on a theoretical and conceptual framework
the target groups for the assessment
a definition of the domain
the description of any variables that are part of the description and describe its depth and breadth (e.g., contexts, processes, content)
a blueprint for the test development against the above descriptions and variables, which might also include item types, representations, the length of the assessment, the number of items and the spread against the different variables.
Together, these aspects and content create the conceptual and assessment framework that will guide the assessment, as is the case with this document and its role for the assessment of numeracy components in PIAAC. It defines the construct of numeracy that steers the development of the test items and eventually the interpretation of results. The assessment construct provides a formal definition of the domain and the features of the construct in terms of any key parameters or dimensions of content, cognitive strategies and range of applications that need to be covered by the content of the assessment.
It is important to note that the PIAAC numeracy assessment describes the full range of numeracy capability in the adult population. This covers at one extreme, adults who have university level training and, at the other, adults who have very limited levels of education (e.g. who left school at or before the age of 15). At the same time, it covers both young adults still in education and adults who completed their formal education 30-50 years prior to undertaking the assessment.
The assessment of numeracy in the second cycle of PIAAC will link to the assessment used in the first cycle of PIAAC and also the earlier ALL study through the use of linking items. As a result, this revised conceptual framework for assessing numeracy in PIAAC will need to maintain key conceptual and pragmatic links to the numeracy framework used for the ALL study and PIAAC Cycle 1.
At the same time, it is important that the framework identifies a construct of numeracy that is relevant to the realities of the third decade of the 21st century as well as reflecting contemporary understandings of adult numeracy and that it incorporates relevant developments in testing practice and makes the best use of the available testing technologies.
This part of the report provides a summary of the 2017 review of the PIAAC Cycle 1 numeracy framework and assessment, gives an indication of some other conceptual issues to consider—identified by the new PIAAC Cycle 2 Numeracy Expert Group (NEG)—and finishes with a brief rationale for assessing numeracy in PIAAC.
The conceptual and assessment framework for the second cycle of PIAAC numeracy was expected to be updated and revised based on a review of the numeracy assessment framework used in the first cycle of PIAAC. This review was commissioned by the OECD Secretariat and published at the beginning of 2017 (Tout et al., 2017[1]). The aim of this review project was to prepare a paper reviewing the framework that guided the assessment of numeracy in the first cycle of PIAAC.
The review aimed to evaluate the extent to which the framework developed in 2009 reflected current understandings of adult numeracy and continued to be an appropriate basis for the assessment of the capacity of adults to successfully undertake the range of numeracy tasks that they will face in their everyday and working lives in the third decade of the 21st century. In particular, the review addressed the following:
theoretical developments in the understanding and conceptualisation of adult numeracy that are relevant for the assessment of numeracy in PIAAC
how to ensure that the assessment reflects the importance of digital information, representations, devices and applications as realities that adults have to manage in dealing with the numerical demands of everyday life
developments in the assessment of numeracy (particularly of adults) that could be relevant for PIAAC (e.g., item types and formats, use of animation, and modelling)
how the relationship between the PIAAC numeracy framework and the PISA mathematical literacy framework and assessment should be conceived, developed (if appropriate) and presented
the utility and feasibility of the development and implementation of an assessment of numeracy components equivalent to the PIAAC reading components assessment.
The review recommended a range of areas for potential improvements and enhancements, including the definition and elaborations of adult numeracy used in the framework, and of the assessment content. Many of the suggestions arose out of the concern that the existing Cycle 1 framework and assessment did not reflect some of the realities of the skills and knowledge adults needed to succeed in work, life, and citizenship in the 21st century. Some of the key elements arising from the review paper included:
addressing 21st century skills including critical thinking and reflection, reasoning and understanding of degree of accuracy
taking on board technology/ICT advancements while keeping a balance with more traditional modes and means of communication and undertaking numeracy tasks
making better use of technology for assessment in relation to both authenticity and making items accessible
addressing a number of issues regarding adults’ numeracy performance and understandings, including a person’s disposition to use mathematics and to see mathematics in a numeracy situation
developing an assessment of numeracy components, which would have parallel aims to the existing reading components assessment, and provide insights into the skills and knowledge of the significant number of adults with low levels of numeracy.
This report and the recommendation have been instrumental in the writing of this framework and assessment construct. The review and this document build on conceptual and assessment frameworks and cumulative wisdom developed in connection with prior surveys of adult skills, primarily the first cycle of PIAAC – see PIAAC Numeracy Expert Group (2009[2]) and Gal and Tout (2014[3]). The PIAAC Cycle 1 framework and assessment drew heavily on the numeracy assessment framework of the Adult Literacy and Life Skills Survey (ALL) (Murray, Clermont and Binkley, 2005[4]). It also built on the work in the International Adult Literacy Survey (IALS), and surveys of school-age students, especially the Programme for International Student Assessment (PISA).
In the process of considering the recommendations and content of the 2017 review paper by the new PIAAC Cycle 2 Numeracy Expert Group (NEG) at its first meeting in March 2018, a number of additional issues were identified which needed some further exploration. As a result, there was a literature review undertaken by the NEG exploring the following five conceptual issues: Big Ideas in mathematics, Number sense, Embeddedness, Authenticity, and Numerate behaviour and practices. While this work has been incorporated and embedded throughout this revised numeracy framework, a brief summary is included below.
Big Ideas in mathematics, is a term that is used to talk about powerful mathematical ideas (Jones, Langrall and Thornton, 2002[5]) central to the learning of mathematics, linking numerous mathematical understandings into a coherent whole [e.g., see (Charles, 2005[6]; Hurst, 2014[7]; Hurst and Hurrell, 2014[8]; Kuntze et al., 2009[9]; Kuntze et al., 2011[10]; Steen, 1990[11])]. Initially, the term “Big Ideas” referred to how mathematical information can be classified in different ways compared with the traditional school mathematics curriculum content areas. They often include the following content domains (which are elaborated in the PIAAC framework): quantity and number, space and shape, change and relationships, and data and chance. “Big Ideas” are also used as focal points to add some structure to sometimes “overcrowded curricula” (Siemon, 2017[12]; Siemon, Bleckly and Neal, 2012[13]).
In PIAAC Cycle 2, particularly in relationship to the challenge to develop a separate, new, numeracy components assessment, number sense is seen as relating to a person’s general understanding of different types of number and operations, and it involves a critical understanding in order to make decisions and solve problems using numbers in flexible ways in Personal, Work, and Societal/community contexts (Ontario Ministry of Education, 2006[14]; Peters, 2012[15]; Wagner and Davis, 2010[16]; Yang, Reys and Reys, 2009[17]). McIntosh, Reys and Reys (1992[18]) define number sense as: “It reflects an inclination and an ability to use numbers and quantitative methods as a means of communicating, processing and interpreting information” (p. 3). In addition, “using numbers is more than reasoning about number and more than skilled calculations. It is about making sense of the situation to which we apply numbers and calculations” (Thompson, 1995, p. 220[19]). Numbers, and quantitative expressions, may be presented in a range of different representational systems, including: text or symbols, images of physical objects, structured information and dynamic information. An understanding of number sense has been identified as a key element and is addressed throughout the framework, and has helped underpin the development of the new numeracy components assessment for PIAAC Cycle 2. This is elaborated further in the sixth section: Numeracy components.
These four issues are all interrelated, and the NEG has attempted to address them more explicitly throughout the new framework, its elaboration, and to some extent in the content of the assessment itself, including in the background questionnaire (BQ) questions that relate to numeracy and mathematics skill use. These issues of embeddedness, authenticity, numerate behaviour and numerate practices, all relate to an understanding of the vital and underpinning connection to the real-world context in which mathematics is utilised by adults in their daily lives as individuals, citizens, family members or as workers.
The embeddedness of mathematics refers to the deep connections the mathematics has to the context in which it is utilised. This means that the way mathematics is used to operate on a task is fundamentally shaped by the context in which it is employed, which includes socio-cultural influences that afford or constrain action in civic, personal or workplace environments. In this view there is a clear separation between school mathematical knowledge, how it is taught, learned and practised, and the use of this knowledge outside of schooling. The issue of authenticity is a significant issue in the development of the test questions in PIAAC as it relates to how alike a task is in an international assessment like PIAAC to the actual real-life situation that the assessment task has been adapted from. PIAAC items are developed on the basis of finding situations and tasks that are based on authentic stimuli and then composing sets of questions that someone would want answered about, or based on, the information in the stimulus. Numerate behaviours and practices are distinct, but complementary, issues. Numerate behaviours relate to the cognitive responses by an individual to situations where mathematics is embedded in a real-world problem where a response or action is expected. Numeracy practices relate to the different uses of mathematics within a context, defined not just by the problem itself, but also by the physical and social context in which it is situated. These issues are further elaborated and discussed throughout the framework paper.
As was argued in the PIAAC Cycle 1 numeracy framework (PIAAC Numeracy Expert Group, 2009[2]), this framework and description of numeracy is founded on the assumption that a direct assessment of numeracy in PIAAC is an essential and worthwhile undertaking for four separate but related reasons (PIAAC Numeracy Expert Group, 2009, pp. 8-9[2]):
Numeracy is essential for adults and for the societies in which they live.
Public policy in most countries includes separate investments in literacy and numeracy.
The policy and programme responses are different for numeracy than for literacy.
Numeracy skill levels are not measured well by literacy measures.
Basic computational and mathematical knowledge has always been considered as part of the fundamental skills that adults need to possess to function well and be able to accomplish various goals in their everyday, work, and social life. As will be demonstrated later in this framework, societies now present increasing amounts and a wider range of information of a quantitative or mathematical nature to citizens from all walks of life, in diverse contexts such as regarding health risk factors, environmental impacts, or financial planning and insurance purchasing, to name just a few. As workplaces are becoming more technology rich and concerned with involving all workers in improving efficiency and quality, the importance of numeracy and mathematical skills is growing. Numeracy-related skills have been shown to be a key factor in labour market participation, sometimes even more so than literacy skills. Adults with lower level skills in numeracy and literacy are more likely to be unemployed or require social assistance. Further, sound numeracy skills are deemed essential for post-secondary education in many areas, including but not limited to hard sciences, engineering and technology (Benn, 1997[20]; Bynner and Parsons, 2005[21]; Coben et al., 2003[22]; Coben, O’Donoghue and FitzSimons, 2000[23]; Condelli et al., 2006[24]; Coulombe, Tramblay and Marchand, 2004[25]; Forman and Steen, 1999[26]; Gal, 2000[27]; Gal et al., 2005[28]; Ginsburg, Manly and Schmitt, 2006[29]); (Hoyles et al., 2002[30]; Johnston, 1994[31]; Jonas, 2018[32]; Jones, 1995[33]; Murnane, Willett and Levy, 1995[34]; National Research and Development Centre (NRDC), 2006[35]; OECD/Statistics Canada, 2005[36]; Tout and Gal, 2015[37]; Tout and Schmitt, 2002[38]; Willis, 1990[39])
Public policy in most countries includes separate investments in literacy and numeracy. The separate acquisition of skills in these two fundamental areas is emphasised throughout both primary and secondary school systems, and in adult education or nonformal learning schemes. Countries expect that investment in literacy and numeracy will increase citizens’ ability to act independently towards their own progress and income security, thereby reducing future social expenditures as well as contributing to citizens’ participation in economic and social life in an information-laden society.
The policy and programme responses are different for numeracy than for literacy. Efforts to improve literacy and numeracy levels of specific population groups are not necessarily implemented via the same mechanisms—they often require different experts, resources, and learning systems because of differences in the underlying knowledge components and learning trajectories. It is vital that nations have information about their workers’ and citizens’ numeracy, independently of other competency areas, in order to evaluate the human capital available for advancement, to plan school-based and lifelong learning opportunities, and to better understand the factors that affect citizens' acquisition and usage of numeracy (Johnston and Maguire, 2005[40]).
It is not possible to represent the numeracy levels in a population solely via people’s performance on literacy measures that examine how well people read, process, and comprehend various types of texts and documents, or communicate about such texts. As found in PIAAC and other research that compares adult’s skills and performance in literacy with numeracy , there are substantial differences in the performances, outcomes and implications/consequences of lower or higher numeracy skilled adults compared to literacy skills [e.g., see (Bynner and Parsons, 2005[21]; Jonas, 2018[32]; OECD, 2017[41]; Tout and Gal, 2015[37])]. As is explained in more detail later, numeracy involves, among other things, the handling of not only arithmetical processes, but also the understanding of proportions and probabilistic ideas, the understanding of numerical, geometric, graphical and algebraic types and representations of mathematical information, and the critical interpretation of statistical or mathematical messages. Most of these elements and processes bear little relation to what is subsumed by literacy measures (Coben, O’Donoghue and FitzSimons, 2000[23]; Gal et al., 2005[28]).
It follows that a direct assessment of numeracy in PIAAC can provide policy makers and other stakeholders with a unique and sound basis for evaluating the distribution of the actual numeracy competence in the adult population.
The conceptualisation of numeracy in an international context is a challenging undertaking. Like literacy, the term numeracy has multiple meanings across countries and languages. In some countries the term numeracy relates to basic skills which school children are expected to acquire as a prerequisite to learning formal mathematics at higher grades. In other countries the term numeracy encompasses a broad range of skills, knowledge and dispositions that adults should possess but it does not necessarily relate to formal schooling. This is discussed further in the next section. Some countries do not even have a word such as numeracy; therefore, as part of educational or policy-oriented discourse in such countries, experts or translators either had to invent a special new word for it (e.g., Numératie in Canada, Numeralitet in Denmark), or use other phrases such as “mathematical literacy”, “functional mathematics”, or terms equivalent to “computational ability”. Such diversity in terminology, or the lack of an accepted term with which policy makers feel comfortable, can complicate the communication with and among policy makers and educators interested in PIAAC.
The range of meanings attached to the term numeracy and the lack of an equivalent term across languages may create miscommunications or gaps in expectations regarding what will be measured by a numeracy scale in PIAAC. This can affect the perceived policy relevance of a numeracy scale. Thus, attention has to be given to making sure that discussions regarding numeracy assessment in PIAAC are based on a clear description and consensus about the scope of the term and recognition of its centrality in a wide range of adult life circumstances.
It must also be remembered that what will be measured by a numeracy assessment is jointly determined by two interrelated factors:
the conceptual construct describing numeracy and its elements
the assessment construct describing how the general conceptualisation of numeracy is operationalised and manifested in the nature and range of tasks used in the assessment and the mode of administration and scoring.
Formulation of what numeracy encompasses has evolved since the term was first introduced in the 1959 Crowther Report in England and Wales [e.g., see (Karaali, Villafane-Hernandez and Taylor, 2016[42])], when it was defined as something “more than mere ability to manipulate the rule of three” (Crowther, 1959, p. 270[43]). Another significant milestone in the conceptualisation and description of numeracy was in the Cockcroft report of 1982, where it was defined as:
[n]umeracy is…an ‘at-homeness’ with numbers and an ability to make use of mathematical skills which enables an individual to cope with the practical mathematical demands of his everyday life…[and] an ability to have some appreciation and understanding of information, which is presented in mathematical terms, for instance graphs, charts or tables or by reference to percentage increase or decrease. (Cockcroft, 1982, p. 11[44])
The use and meaning of the term numeracy has gained momentum in the years since the Crowther and Cockcroft Reports. Some relevant papers and research include: (Baker and Street, 1994[45]; Benn, 1997[20]; Coben, O’Donoghue and FitzSimons, 2000[23]; Coben et al., 2003[22]; Condelli et al., 2006[24]; Forman and Steen, 1999[26]; Gal, 2000[27]; Gal et al., 2005[28]; Ginsburg, Manly and Schmitt, 2006[29]; Hoyles et al., 2002[30]); (Johnston, 1994[31]; Lindenskov and Wedege, 2001[46]; Maguire and O’Donoghue, 2003[47]; National Research and Development Centre (NRDC), 2006[35]; Tout and Gal, 2015[37]; Tout and Schmitt, 2002[38]; Willis, 1990[39]). In the United Kingdom in 2000, Coben, O’Donoghue and FitzSimons published a work titled Perspectives on Adult Learning Mathematics, in which they provided a review of research related to adults’ learning of mathematics. At the same time, a similar volume, Adult Numeracy Development: Theory, Research, Practice was published in the United States (Gal, 2000[27]). A conceptualisation of adult numeracy for the Adult Literacy and Life Skills (ALL) survey, the precursor to PIAAC, was developed around the same time (1998-2000) by an international team (Gal et al., 2005[28]). This was the first time the construct of numeracy had to be defined in an international comparative assessment context and not purely in an educational context. The ALL international team defined numeracy alongside a more elaborate definition of numerate behaviour. Coben, in 2003, led a team who wrote Adult Numeracy: Review of Research and Related Literature and noted that numeracy was increasingly defined as “mathematics in work and mathematics in everyday adult life” (Coben et al., 2003, p. 38[22]).
Maguire and O’Donoghue (2003[47]) reviewed and organised conceptions of numeracy from several countries (Australia, Canada, Denmark, Ireland, the Netherlands, United Kingdom and United States) along a continuum of increasing levels of complexity or sophistication: formative, mathematical and integrative. Formative conceptions view numeracy as related to basic arithmetic skills. Mathematical conceptions consider numeracy in a contextualised way, as a broader set of mathematical knowledge and skills (beyond basic computations) of relevance in everyday life. Finally, integrative conceptions consider numeracy as a multifaceted, sophisticated construct incorporating not only mathematics but also communicative, cultural, social, emotional, and personal elements which interact and pertain to how different people function in their social contexts.
At this time, formative conceptions were often associated with how numeracy was viewed in connection with goals of primary schooling, and reflected in how numeracy was defined when classifying literacy/numeracy levels worldwide e.g., UNESCO (1997[48]). Most extant conceptions which adult education, workplace training, and national and international assessments have adopted fall at different points across the mathematical and integrative phases described by Maguire and O’Donoghue. The range of conceptions and definitions of adult numeracy from late last century to more recent times illustrate that conceptions evolve over time and that variability can be noticed even within the same national system.
Lindenskov and Wedege (2001[46]) offer an interesting case study of defining numeracy. Based on their work in adult and mathematics education in Denmark, they imported numeracy from English-speaking countries and introduced a new term, Numeralitet, with a conceptual framework that was later adopted by the Danish Ministry of Education. According to this perspective, it is essential to distinguish between what numeracy is, or ought to be, from the individual’s and from society’s points of view. Lindenskov and Wedege advocated a societal view, whereby numeracy is seen as a competence that involves a dynamic interaction between functional mathematical skills and conceptions and operations on the one hand, and a series of activities and various types of data and media on the other. They argued that this skill- and activity-based view should be coupled with the understanding that in principle all people need to have this competence, and that numeracy is a competence determined by society and technology and that it changes in time and space along with social change and technological development.
The definition quoted from the United Kingdom’s Cockcroft Committee (1982[44]) earlier has been quite influential in that its conception of numeracy implied it is an ability to cope with various functional tasks in real-world contexts as well as interpretive tasks, but also pointed to the centrality of underlying supporting non-cognitive components. These key ideas are reflected, albeit with different terminologies and foci, in other views of numeracy, including in the definitions used in both ALL and PIAAC Cycle 1. Another important commonality is the presence of mathematical elements or ideas in real situations, and the notion that these can be used or addressed by a person in a goal-oriented way, dependent on the needs of the individual within the given context, i.e., home, community, workplace, societal action, etc.
Looking at publishing sources, the evidence is growing that the use and concept of the term numeracy has been displacing other terms such as mathematical literacy and quantitative literacy, and has become more popular, even though in many languages it does not have a direct translation. This is illustrated in Figure 3.1 below, which is based on work by Karaali, Villafane-Hernandez and Taylor (2016[42]), but updated.
Source: Data from Google Books Ngram Viewer. Retrieved on 15th December 2018.
The two terms Competency and Skill are both used in current PIAAC documents and reports. Competencies can mean different things in different situations, and in different cultures. For example, a reductive notion of competence is used as a synonym for skill in some adult education settings within Australia. Competencies can however also mean the combination of skills and aspects of higher order thinking such as strategic planning. The latter is reflected in the OECD’s Definition and Selection of Competencies project (DeSeCo) (OECD, 2005[49]), which was developed to provide a framework informing the identification of key competencies in international surveys measuring the competence level of young people and adults [see Rychen (2004, p. 321[50])].
In The Survey of Adult Skills – Reader’s Companion, Second Edition (OECD, 2016[51]), the OECD discusses this issue of terminology and the use of these two terms (skill and competency) and concludes that there is much overlap in their understanding and use, and while acknowledging that there can be differences in the use and meaning of the two terms, concludes:
In the context of the Survey of Adult Skills (PIAAC), however, no attempt is made to differentiate competency and skill, and the terms are used interchangeably ... Both terms refer to the ability or capacity of an agent to act appropriately in a given situation. Both involve the application of knowledge (explicit and/or tacit), the use of tools, cognitive and practical strategies and routines, and both imply beliefs, dispositions and values (e.g. attitudes). In addition, neither competency nor skill is conceived as being related to any particular context of performance, nor is a skill regarded as one of the atomic units that combine to form competency. (OECD, 2016, p. 17[51]).
This numeracy framework report similarly does not make or attach an explicit meaning or use to either term.
Establishing and extending numeracy capability requires the adoption, development or appropriation of both numerate behaviours and practices. These two constructs are distinct but complementary. Numerate behaviours are cognitive responses by an individual to particular situations where mathematics might provide advantage in addressing a real-world problem. On the other hand, numeracy practices relate to the use of mathematics within a context defined not just by the problem but also the physical and social context in which it is situated. The notion of situatedness is tied to ways of thinking, modes of reasoning and means of knowledge generation within communities that are defined by distinct social or cultural types of activity. From this perspective, numeracy is viewed as a social practice. As Yasukawa et al. (2018[52]) explain:
A NSP [numeracy as a social practice] perspective focuses on what people do with numeracy through social interactions in particular contexts, rather than on people’s performance of mathematical skills in isolation of context…Moreover, a focus on practice entails viewing numeracy activity as culturally, historically and politically situated. (p. 13)
Thus, employing numeracy behaviours to address real-world problems in different contexts requires the accommodation of unique ongoing activities, social relationships and community-based modes of thinking and reasoning (Lave, 1988[53]). This means that the use of mathematics within a practice requires that mathematical capability is nuanced by holistic strategies shaped by the specific contexts in which they are deployed (Geiger, Goos and Forgasz, 2015[54]).
The notion of numeracy as a social practice, however, has implications for the question of transfer—the use of numerate behaviours developed in one context in a new or different situation. This issue of transfer, according to Hoyles, Noss and colleagues (Hoyles, Noss and Pozzi, 2001[55]; Noss, Hoyles and Pozzi, 2002[56]), can be achieved through the abstraction of underlying invariants that are relevant across situations—a process they term situated abstraction.
Mathematical conceptualization may be finely tuned to its constructive genesis-how it is learned, how it is discussed and communicated—and to its use in a cultural practice, yet simultaneously can retain mathematical invariants abstracted within that community of practice. (Noss, Hoyles and Pozzi, 2002, p. 205[56])
While the nature of the assessment content in PIAAC limits its primary focus to cognitive aspect of numeracy, that is, the numeracy behaviours and skills that underpin the questions, their contexts and the specific items, the notion of practices has influenced the development of the skills use questions that form part of the background questionnaire. PIAAC's background questionnaire (BQ) includes collecting a wide range of information which can help to explain differences in performance among adults, further informing our understanding of factors that affect skill acquisition and retention or motivation for further learning. The skills use questions are structured around two themes, work practices and everyday practices, where questions elicit responses about the frequency of use a different numeracy practices. The Numeracy Expert Group (NEG) worked with the OECD and the PIAAC BQ Expert Group to revise and improve the consistency and research validity and usefulness of the BQ questions on numeracy practices at work and in everyday life for the second cycle of the PIAAC. The NEG’s work and recommendations helped to homogenise the set of questions, keep a sense of consistency between work/professional practices and personal uses, while also trying to preserve a continuity between the two cycles. The NEG used the descriptions of the four different content areas of the PIAAC numeracy framework to help guide them in this, alongside reviewing the existing research about numeracy practices at work and in everyday life from PIAAC Cycle 1.
Questions related to work practices and use include: the calculation of prices; counting stock; reviewing inventories; planning delivery routes; preparing budgets; undertaking measurements; interpreting charts or performing data analysis. Everyday practice questions are related to examples such as: calculations related to purchase and discounts; decisions regarding financial matters such as budgets, insurance, loans, or savings; measurements needed when you cook, garden, make clothes. Thus, while PIAAC’s capacity to assess numeracy activity from a social practice perspective is necessarily limited, the role of practices in documenting and researching numeracy capability and performance is recognised as vital.
The theoretical conceptualisation of numeracy for PIAAC Cycle 2, discussed and presented below, is built on the previous review of literature and research findings reported in the two previous numeracy frameworks for ALL survey (Gal et al., 2005[28]) and PIAAC Cycle 1 (PIAAC Numeracy Expert Group, 2009[2]). As well it incorporates the review of the numeracy assessment framework used in the first cycle of PIAAC commissioned by the OECD Secretariat (Tout et al., 2017[1]), along with further research done by the Numeracy Expert Group for PIAAC Cycle 2. This is then used as the basis for the elaboration of how the assessment of numeracy in PIAAC Cycle 2 will be implemented and what the key dimensions are that will be assessed. This is described in the third section: Numeracy assessment construct in PIAAC Cycle 2.
This conceptualisation operates on two levels. It relates to numeracy as a construct describing a skill or competence, and to numerate behaviour and practices, which is the way a person’s numeracy is manifested in the face of situations or contexts which have mathematical elements or carry information of a quantitative nature. In this way, inferences about a person’s numeracy are possible through analysis of performance on assessment tasks designed to elicit numerate behaviour. In congruence with the above view of a competence, numeracy will be described as comprised both of cognitive elements (i.e., various knowledge bases and skills) as well as non-cognitive or semi-cognitive elements (i.e., attitudes, beliefs, habits of mind, and other dispositions) which together shape a person’s numerate performance, behaviour and practices. This conceptualisation includes ways of knowing, the means of generating new knowledge, and using different modes of reasoning.
The following sections summarise some of the theoretical and conceptual foundations in the previous PIAAC framework document and adds more recent research and understanding about adult numeracy, mainly from the review paper. It starts with the same structure as the previous framework and first addresses the contexts and demands for numeracy, but then adds to this with a new section that adds in further research insights from the PIAAC Cycle 1 framework review and more.
But it should be noted that most recent reviews indicate that there continues to be a shortage of any empirical or theoretical developments in research on adult numeracy [see e.g., (Carpentieri, Litster and Frumkin, 2009[57]; Condelli et al., 2006[24]; Geiger, Goos and Forgasz, 2015[54]; Windisch, 2015[58])]. However, the 2017 PIAAC numeracy review team’s research (Tout et al., 2017[1]) included reading and reviewing recent reports about the teaching, learning and descriptions of adult numeracy practices; e.g., (Chisman, 2011[59]; Griffiths and Stone, 2013[60]; National Institute of Adult Continuing Education (NIACE), 2011[61]). The review team found that a number of issues should be considered and addressed in the review and rewriting of the PIAAC numeracy framework for this second cycle of PIAAC. These have been incorporated into the discussions and outcomes below.
What is encompassed by numeracy (and numerate behaviour and practices) can initially be addressed by identifying the nature of the contexts that contain quantitative and mathematical1 elements that adults face and which pose demands with which they have to cope. This in turn provides the basis for describing the knowledge elements and supporting processes which enable adults to cope with real-world numeracy tasks (Ginsburg, Manly and Schmitt, 2006[29]), and can later help to form a road map which can guide the design and selection of tasks for inclusion in the numeracy assessment in PIAAC.
The literature pertaining to the uses of numeracy in the real world can be divided into three strands:
the role of numeracy in adults’ lives
the mathematical demands of workplace settings
educational perspectives on the mathematical needs of school graduates and citizens.
These three areas are certainly intertwined but also offer complementary ideas; hence, each is reviewed separately below.
Research shows that 21st century skills requirements have changed compared with the previous century, and new ways of working, reasoning and thinking are required, and that increasingly the new skills interact with technology [e.g., see (Binkley et al., 2011[62]; Expert Group on Future Skills Needs (Ireland), 2007[63]; Foundation for Young Australians, 2017[64]; Griffin, McGaw and Care, 2012[65]; Partnership for 21st Century Skills, 2016[66]; Pellegrino and Hilton, 2012[67])]. In the literature, this is often referred to as ‘21st century skills’ or ‘21st century competences’ (Voogt and Roblin, 2012[68]), ‘global competences’ (OECD, 2019[69]) or ‘the 4th industrial revolution’ (Schwab, 2016[70]). Common is the acknowledgement that across education, government, and business, the skills and knowledge needed to succeed in work, life and citizenship have significantly changed in the twenty-first century. As has been argued and documented by many sources, and summarised in PIAAC planning documents, adults are presented with ever-increasing amounts of information of a quantitative or mathematical nature through Internet-based or technology-based resources. New means of communication and types of services have changed the way individuals interact with governments, institutions, services and each other, and social and economic transformations have, in turn, changed the demand for skills as well. More so than in prior decades, a wider range of quantitative and mathematical information is more readily available, but this information has to be located, selected or filtered, interpreted, at times questioned and doubted, and analysed for its relevance to the responses needed.
The implications of such 21st century skills and ICT demands on the numeracy needs for adults in their daily lives, as citizen and as workers are discussed in the following sub-sections.
Analysis of the purposes served by adults' numeracy skills has often focused on workplace numeracy practices or on the outcomes of school education, which are both discussed in the following two sub-sections. In the 21st century, young people and adults need to be able to cope with the aspects of the world as they encounter them, which includes the digital and technological aspects of information and society—and society already has all kinds of technological aspects that interact with numerical and mathematical information. Therefore, the focus must also be on both life as an individual, and as part of society and citizenship, and that includes with the digital aspects of information and society—the reality is that technology is now ubiquitous with all aspects of many societies. Services, interactions and communications outside the workplace have all changed in the 21st century, often driven by technological advances. This includes online processes such as banking, purchases, bookings, reviewing information (health, housing, etc.) and making decisions based on that information. It includes functioning in the bureaucratic world (applying for permits, social security applications and processes, applying for jobs, managing insurances, etc.), use of different media (e.g., the Internet, online news, Facebook, podcasts, videos, etc.), use of different aspects of communication (e-mail, SMS, apps, social media, etc.), and of a range of software and technology at home and in the community. Technology has meant greater market penetration and influence. The influences of social and mass media has implications for informed and participatory citizenship, and hence for citizens to be critical consumers of all forms of media.
Further, it has been argued that in a society in which the media constantly present information in numerical or graphical form, the ability to interpret and critically reflect on quantitative and statistical messages is vital for all adults [e.g., see (Benn, 1997[20]; Paulos, 1988[71]; Paulos, 1995[72]; Steen, 1990[11]; Utts, 2003[73]; Willis, 1990[39])]. It is seen as essential for all adults to possess the ability to critically reflect on quantitative information encountered in various media sources and documents (Frankenstein, 1989[74]), and to understand how to be a careful or critical consumer of statistical arguments of various kinds (Gal, 2002[75]; Utts, 2003[73]; Watson and Callingham, 2003[76]). This view of needing to be critical as part of being numerate was often espoused by adult education experts, focusing on the role of adults as reflective communicators and critical consumers of information in society who are involved in the exchange and interpretation of messages encountered in media or in political and community contexts (Frankenstein, 1989[74]). For example, Johnston (1994[31]) argued:
To be numerate is more than being able to manipulate numbers, or even being able to ‘succeed’ in school or university mathematics. Numeracy is a critical awareness which builds bridges between mathematics and the real-world, with all its diversity. (p. 34)
Efforts to formally describe numeracy use in society more generally have been undertaken in several countries [e.g., see (McLean et al., 2012[77]; Quality and Qualifications Ireland (QQI), 2016[78]; Tertiary Education Commission, 2008[79]; U.S. Department of Education, 2013[80])]. In Australia, for example, two frameworks (Kindler et al., 1996[81]; Victorian Curriculum and Assessment Authority (VCAA), 2008[82]) proposed four broad categories regarding the uses of numeracy. The four categories are Numeracy for practical purposes; Numeracy for interpreting society; Numeracy for personal organisation; and Numeracy for knowledge. Numeracy for practical purposes addresses aspects of the physical world that involve designing, making, and measuring. Numeracy for interpreting society relates to interpreting and reflecting on numerical and graphical information in public documents and texts. Numeracy for personal organisation focuses on the numeracy requirements for personal organisational matters involving money, time and travel. Numeracy for knowledge describes the mathematical skills needed for further study in mathematics, or other subjects with mathematical underpinnings or assumptions.
Another scheme was developed by Steen (1990[11]), who outlined five dimensions of numeracy:
practical, focused on mathematical and statistical knowledge and skills that can be put to immediate use to cope with tasks in daily life
professional, focused on the mathematical skills required in specific jobs
civic, focused on benefits to society
recreational, related to the role of mathematical ideas and processes in games, puzzles, sports, lotteries, and other leisure activities
cultural, concerned with mathematics as a universal part of human culture (and related to appreciation of mathematical aspects such as in cultural or artistic artefacts).
Overall, the purposes regarding numeracy use appear to be consistent and suggest that adults need to be able to apply their numeracy (and literacy) skills to tasks with a social or personal purpose in both informal and more formal contexts. Such perspectives supplement Bishop’s (1988[83]) proposal that there are six modes of mathematical actions that are common in all cultures and pertain both to children and to adults: counting, locating, measuring, designing, playing and explaining.
Mathematical and statistical skills that are important in adults’ work have in part been described in large-scale efforts to define "core skills" or "key competencies" that workers should have, usually in response to the need to maintain economic competitiveness and improve employability of adults and school graduates. In addition, several projects looked specifically at the mathematical skills of workers in a range of occupational groups or workplace clusters.
Basic computational knowledge has always been considered as part of the fundamental skills that adults need to possess, but recent research and skills framework developments claim that workers need to possess a much broader range of mathematical skills. Examples exist in many countries and the following selective description is indicative of the nature of such efforts. In the United States [see (Carnevale, Gainer and Meltzer, 1990[84]; Secretary’s Commission on Achieving Necessary Skills (SCANS), 1991[85])], reviews differentiated between mastery of basic arithmetical skills and a much broader and flexible understanding of mathematical skills. The higher level skills included “choosing appropriately from a variety of mathematical techniques; uses quantitative data to construct logical explanations for real-world situations; expresses mathematical ideas and concepts orally and in writing; and understands the role of chance in the occurrence and prediction of events” (Secretary’s Commission on Achieving Necessary Skills (SCANS), 1991, p. 83[85]). Forman and Steen (1999[26]) similarly argued that quantitative skills desired by employers are much broader than mere facility with the mechanics of addition, subtraction, multiplication, and division and familiarity with basic number facts. They also include some knowledge of statistics, probability, mental computation strategies, some grasp of proportional reasoning or modelling relationships, and broad problem solving and communication skills about quantitative issues.
In relation to work in the 21st century, research is showing that there is a significant and increasingly important and underpinning role that science, technology, engineering, and mathematics (STEM) skills play [e.g., (Foundation for Young Australians, 2017[64]; PwC, 2015[86])]. In their recent 2017 review, the National Council of Teachers of Mathematics (NCTM) (2017[87]) argued that mathematics is at the heart of most innovations in the information economy. They saw mathematical and statistical literacy as needed more than ever to filter, understand, and act on the enormous amount of data and information that we encounter every day.
As well, there is a significant amount of research that has looked into specific numeracy and mathematics practices in workplaces, including in relation to 21st century skills [e.g., see (Australian Association of Mathematics Teachers (AAMT) and Australian Industry Group (AiGroup), 2014[88]; Bessot and Ridgway, 2000[89]; Buckingham, 1997[90]; Coben et al., 2010[91]; FitzSimons, 2005[92]; Geiger, Goos and Forgasz, 2015[54]; Hoyles et al., 2002[30]; Hoyles et al., 2010[93]; Kent et al., 2011[94]; Marr and Hagston, 2007[95]); (Straesser, 2015[96]; Wake, 2015[97]; Weeks et al., 2013[98]; Zevenbergen, 2004[99])]. One of the key outcomes of the research is that because of the impact of technology and digital tools and processes, the mathematics or numeracy tasks that people undertake at work involve more than basic calculation skills or ‘by hand’ skills and straightforward procedural competence, consistent with the above research. These practices involve more sophisticated mathematical problem solving skills and understandings, new ways of reasoning and thinking, and entail the ability to recognise and engage with the mathematics that is fully embedded within complex and “messy” workplace settings. Many 21st century workplace mathematics and numeracy practice are integrated with technology, particularly information and communications technologies (ICT), and have profoundly altered what are considered to be the key knowledge and skills that individuals need as economies and society continue to evolve.
The skills required in the 21st century include a range of mathematical capabilities such as understanding and interpreting graphical information, interpreting measures in terms of what the data are saying about a manufacturing process, making use of spreadsheets, interpreting visual, computer-generated 3D representations or virtual images, and more. Hoyles et al. (2010[93]) argue that this requirement for mathematical capabilities will be driven by the need to improve production processes and productivity; that is, there will be greater demand for what they call techno-mathematical literacies:
We therefore decided to introduce the term Techno-mathematical Literacies, developing from the idea of mathematical literacy that was used in our previous research … This literacy involves a language that is not mathematical but ‘techno-mathematical’, where the mathematics is expressed through technological artefacts. (Hoyles et al., 2010, p. 14[93])
In relation to technology at work, a 2014 Australian study about the use of mathematics in the workplace found similar connections and entanglements between mathematics and technology:
Many people in the workplace are engaged with technology, particularly in using spreadsheets and graphical outputs. There is an inter-dependency of mathematical skills and the use of technology in the workplace in ways that are not commonly reflected in current teaching practice. The perception is that technology is transforming workplace practices and the use of technology has changed the mathematical skills required – while not reducing the need for mathematics. (Australian Association of Mathematics Teachers (AAMT) and Australian Industry Group (AiGroup), 2014, p. 2[88])
The same report (Australian Association of Mathematics Teachers (AAMT) and Australian Industry Group (AiGroup), 2014[88]) found that workers needed a blend of the following skills:
ability to recognise and identify how and when mathematics is used in the workplace
an understanding of mathematical concepts, procedures and skills
an understanding of the kinds of practical tasks they need to perform
the strategic processes they should be able to use in using and applying mathematics.
Overall, these studies complement the earlier research and studies, and suggest that employees need to possess a range of specific numeracy-related skills or knowledge, such as in the following (but not the only) areas of mathematics:
fast and accurate computations but also estimation, and knowing when each skill is required and why
ability to deal with proportions and percents
understanding measurement concepts and procedures
working with, or creating, simple formulas
a sense for the use of models and modelling in foreseeing future needs
understanding basic statistical concepts and interpreting data and displays.
However, it is not simply the demands of 21st century workplaces and practices that are driving the use of digital technologies in the workplace; workers themselves also use technology to support their thinking. That is, it is not just about the use of digital technologies and tools to replace traditional physical or cognitive skills. In particular, digital tools increasingly mediate young workers’ ways of reasoning, acting, and working (Jorgensen Zevenbergen, 2010[100]; Zevenbergen, 2004[99]). At the same time, these new ways of thinking and acting are reshaping the structuring practices and deployment of skills in workplaces. Zevenbergen argues that this allows young workers to solve problems in often more inventive ways than their more experienced co-workers do.
In addition, on a broader and less technical level, these studies argue that workers need to be able to make decisions in the face of uncertainty in real situations, prioritise actions and make choices regarding the approach to handling different tasks, depending on changing external demands. As well, there is a need for workers to be able to communicate with other workers or clients or understand written documentation (e.g., through text or with tables, charts, and graphs) about issues such as quantities, schedules, variation over time, results of quantitative projections, or analysis of different courses of action in this regard. Such findings echo the earlier distinctions made by the SCANS analysis between the need to attend both to basic arithmetical skills and more elaborate and complex mathematical skills in the workplace, including ways of reasoning and thinking, making connections within and between different aspects of mathematics, and also highlight some areas where specific literacy and communication skills are intertwined with numeracy skills.
Important research literature has also accumulated over the last decades regarding the ways in which people use mathematical skills or cope with mathematical tasks in both formal (i.e., school-based) and informal (i.e., everyday, workplace) contexts (e.g., (Carraher, Carraher and Schliemann, 1985[101]; Nunes, 1992[102]; Nunes, Schliemann and Carraher, 1993[103]; Presmeg, 2007[104]; Resnick, 1987[105]; Rogoff and Lave, 1984[106]; Saxe and Gearhart, 1988[107]). While too complex to discuss in detail here –see Greeno (2003[108]), for one of several reviews of this literature, among other things these studies highlight the situatedness of mathematical knowledge used in functional contexts and the need for actors in different contexts to develop situation-specific mathematical procedures and know-how.
Research suggests that, for adults as well as for children, mathematical knowledge develops both in and out of school [e.g., (Lave, 1988[53]; Saxe, 1992[109]; Saxe et al., 1996[110]; Schliemann and Acioly, 1989[111])]. Saxe and his colleagues have written about the importance of cultural practice in the development of mathematical thinking and how such practices profoundly influence an individual’s cognitive constructions and mathematical ideas, depending, e.g., on the artefacts or tools they use, the nature of the measurement systems in their culture, the counting or calculating devices (abacus, calculator) they use, the distribution of work among family members, or general patterns and types of social activity.
Further, numerous researchers [e.g., (FitzSimons and Coben, 2009[112]; Kent et al., 2007[113]; Marr and Hagston, 2007[95]; Straesser, 2003[114]; Wedege, 2004[115]; Wedege, 2010[116]; Williams and Wake, 2006[117])] have argued, based on ethnographic analyses of workers’ activities in diverse industries, that important portions of the mathematical activities at work are made “invisible” to occasional observers as well as to the workers themselves, or are disguised as non-mathematical. This means that mathematics can be fundamental to activities that are not obviously mathematical. This is most clearly apparent in the use of technology in the workplace where digital tools used to complete tasks often obscure underpinning mathematical activity. As Kent et al. (2007[113]) argues, within techno-mathematical situations in workplaces “there is a shift from fluency in doing explicit pen-and-paper mathematical procedures to a fluency with using and interpreting output from IT systems and software, and the mathematical models deployed within them” (p. 2-3). Building on this point, Wedege (2010[116]) defines two forms of invisible mathematics as (a) subjectively invisible mathematics where people do not recognise the mathematics that they do as mathematics and (b) objectively invisible mathematics in which mathematics is hidden in technology.
Various factors have been posited as causing this phenomenon, such as the encapsulation of many mathematical activities into routines or automated procedures; the use of tools and instruments or information technology (e.g., spreadsheets); the normative use of job-specific linguistic terms that are different than traditional school terms; or the division of labour among different workers.
Based on such and related findings, many reports have argued that mathematical skills as used in the workplace are often different and broader in scope than what is traditionally taught in school mathematics, but also take on different forms depending on the specific work context [e.g., (Australian Association of Mathematics Teachers (AAMT) and Australian Industry Group (AiGroup), 2014[88]; Marr and Hagston, 2007[95])]. The above Australian study (Australian Association of Mathematics Teachers (AAMT) and Australian Industry Group (AiGroup), 2014[88]) about the use of mathematics in the workplace summed up much of this research:
Although the skills observed appear to be fundamental, it is their use and application in work contexts that is not straightforward. (p. 1)
This report went on to describe more fully the differences between school mathematics and workplace mathematics use:
Mathematics is applied in both routine and complex tasks requiring sophisticated use of fundamental mathematical skills and ‘judgement’ or ‘problem-solving’ procedures. Workplace mathematics is performed differently to school mathematics. Mathematical demands may be present implicitly in the workplace tasks, often through tasks that are not obviously mathematical. (Australian Association of Mathematics Teachers (AAMT) and Australian Industry Group (AiGroup), 2014, p. 2[88])
This is consistent with earlier research by Steen in the United States:
“Mathematics in the workplace makes sophisticated use of elementary mathematics rather than, as in the classroom, elementary use of sophisticated mathematics. Work-related mathematics is rich in data, interspersed with conjecture, dependent on technology, and tied to useful applications. Work contexts often require multi-step solutions to open-ended problems, a high degree of accuracy, and proper regard for required tolerances. None of these features are found in typical classroom exercises.” (Steen, 2004, p. 55[118])
It needs to be emphasised that sense of number still underpins much of the mathematical thinking required—including fluency and flexibility in mental calculations and estimations.
The updated conceptualisation of numeracy for PIAAC Cycle 2 was derived with reference to the types of numeracy and mathematical demands as depicted in this sub-section. However, a working assumption has been made that it is not feasible to employ assessment items that are too workplace-specific (e.g., couched in the context of a single workplace or occupation) because mathematics or statistics as used in this context may not be visible or familiar to most other adults (Hoyles et al., 2002[30]).
A growing dialogue about the goals and impact of mathematics education in schools has intensified in recent years. This is in part due to economic pressures and industry expectations on the one hand, but also due to the realisation that mathematical knowledge and skills serve multiple and separate gateway functions on the other hand. Specifically, mathematical competencies affect chances of entry into key occupational tracks (mainly in science, technology, and economics) and may affect employability and labour-force participation, underlie some important aspects of civic participation, and may impact on the possibilities of certain population groups for social equality and mobility. While the dialogue about these issues admittedly overlaps to some extent the points raised earlier in discussing the role of numeracy in society, it is worth elaborating upon because it brings forward some additional points and broadens the understanding of contexts where demands on adults’ numeracy skills exist.
Various arguments have been forwarded over the last few decades to support a broadening of the conceptions regarding the mathematical skills and knowledge that school graduates should possess, and the ways in which learned knowledge serves adults. For example, Ernest distinguishes six different types of mathematical knowledge and capabilities for the results/outcomes of mathematics education in school (Ernest, 2004, p. 317[119]). These are not intended to be seen as mutually exclusive, but as a set of different foci for mathematics education:
utilitarian knowledge
practical, work-related knowledge
advanced specialist knowledge
appreciation of mathematics
mathematical confidence
social empowerment through mathematics.
Apart from the third capability, ‘advanced specialist knowledge’, often a key focus for school mathematics, the other five categories are all compatible with and consistent with the above arguments about how adults might use mathematics in their lives and be numerate individuals, workers and citizens.
Educators working both with school students and adults increasingly aim to assist learners in developing mathematical concepts and skills in ways that are personally meaningful but also functional. Such approaches usually assume that there is often more than one right way to cope with a real-world functional task, and that adults require access to a repertoire of strategies for solving functional problems. Adults’ personal methods of using mathematics are encouraged and valued. This is often a significant difference from traditional (pre-reform) school-based mathematics teaching, within which school students were often expected to solve a problem following the one correct method or algorithm, introduced by the teacher.
Several decades ago, ideas already began emerging in different countries that since mathematics is an essential aspect of society, mathematics education in schools should be derived from or prepare learners for broad real-life situations in family, work, community, and other contexts (National Council of Teachers of Mathematics (NCTM), 2000[120]; Willis, 1990[39]), beyond employers’ desire to focus mostly on practical or job-specific numeracy skills. Two early influential examples are the recommendations of the Cockcroft Committee in the United Kingdom [Department of Education and Science/Welsh Office, (1982[44])], and Freudenthal’s work in the Netherlands, which has led to the Realistic Mathematics Education movement (van den Heuvel-Panhuizen and Gravemeijer, 1991[121]). Over the last two or three decades, many countries have adopted adult education frameworks which give explicit attention to numeracy skills.
Indeed, the dialogue about the various demands on adults’ knowledge has been reflected in part in the emphasis in PISA on the assessment of mathematical literacy and science literacy. Such constructs pertain, broadly speaking, to school students’ readiness for entering adults’ life contexts; it is indicative that they have been chosen to be the focus of assessment rather than more traditional notions of formal curriculum-based knowledge in mathematics or science areas which were assessed primarily in earlier studies.
As outlined above, being numerate in the 21st century means being able to cope with the aspects of the world as we encounter it, which includes the digital and technological aspects of information and society—society generally already has all kinds of techno-mathematical aspects. The 2017 PIAAC numeracy framework review found that 21st century digital technologies provide tools and processes that mediate thinking as well as action and are not just devices that can be used to complete manual, hands-on tasks more efficiently. These tools and processes often change the numeracy task itself and so transform practices within adults’ lives and within the workplace. The use and application of a range of techno-mathematical literacies underpins much of this.
This aspect of 21st century representations and tools was missing from much of the existing PIAAC Cycle 1 numeracy framework discussions, and not adequately reflected in the definition and then in the elaborations. This is explicitly addressed in the new refinements and enhancements to the numeracy framework and construct elaborated later in this paper.
However, it is important to acknowledge that in addressing this issue PIAAC is a survey of adult competencies across all aspects of life, not just about workplace and employment, and not just about engaging with numeracy and mathematics actions within technologically rich environments. It is essential that a balance be kept between numeracy activities in digital and technological environments and those embedded in other, non-digital media; between numeracy demands and situations met as an individual and those encountered as part of society; and between work and employment settings and home and family activities. From the PIAAC numeracy assessment perspective, this can in part be addressed by the need to keep for trend purposes some of the existing former ALL numeracy items, which were originally developed at the end of last century and are not as technologically based, along with a number of the Cycle 1 PIAAC items. The new Cycle 2 items can, hence, contain a set of new items that are more targeted at 21st century digital representations.
In addressing numeracy in adults’ lives above there was reference to a set of formal based descriptions of numeracy for both adult and a youth curriculum, where numeracy use was described by four broad categories (Kindler et al., 1996[81]; Victorian Curriculum and Assessment Authority (VCAA), 2008[82]). The four categories are Numeracy for practical purposes; Numeracy for interpreting society; Numeracy for personal organisation; and Numeracy for knowledge. These categories were used to reflect on how these different purposes and uses might interact with digital information and technology. Table 3.1 below shows some possible connections between numeracy practices and 21st century digital information and technology.
The above reflection about numeracy use indicates a strong connection and entanglement of digital information and technology with literacy and numeracy use in adults’ lives. The ubiquitous presence of social and mass media also carries implications for informed and participatory citizenship, particularly the need for citizens to be critical consumers of such media. This issue of the connection of numeracy with digital information and technology will be addressed explicitly in the later descriptions, elaborations and dimensions of numeracy in PIAAC Cycle 2.
The research section in the review paper on the PIAAC framework (Tout et al., 2017[1]) raised a significant number of challenges, and pointed to the need for careful consideration in the revisions to the PIAAC numeracy framework and in the development of any new assessment items. The review considered not only new research but also looked at different descriptions and models for representing and describing numeracy. Some of these are considered below.
The 2017 review paper considered the PISA 2012 mathematical literacy framework and its descriptions (OECD, 2013[122]). It should be noted that the same mathematical literacy framework and assessment construct was used for the next two cycles of PISA in 2015 and 2018.
However, for PISA 2021, mathematical literacy is again the major domain for PISA, and hence the PISA framework and assessment construct is being updated and revised. This revision was happening in parallel with the development of this numeracy framework for PIAAC Cycle 2. The PIAAC Numeracy Expert Group was able to access a copy of the second draft of the PISA 2021 Mathematics Framework (OECD, 2018[123]) in November 2018. Because of the timing issues, most of the comparisons between PIAAC numeracy and PISA mathematical literacy have therefore been based on a comparison of the 2012 PISA framework and descriptions, but where possible the PIAAC NEG has also included comments and comparisons with the updated 2021 PISA mathematical literacy framework. It should be noted that it is therefore possible that information regarding 2021 PISA mathematical literacy may change from what was in the second draft of the framework paper.
The definitions of mathematical literacy in the 2012 and 2021 PISA frameworks are very similar and consistent, with some changes and updates to reflect some new perspectives. The two definitions are shown below.
Mathematical literacy is an individual’s capacity to formulate, employ, and interpret mathematics in a variety of contexts. It includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals to recognise the role that mathematics plays in the world and to make the well-founded judgments and decisions needed by constructive, engaged and reflective citizens. (OECD, 2013, p. 25[122])
Mathematical literacy is an individual’s capacity to reason mathematically and to formulate, employ, and interpret mathematics to solve problems in a variety of real-world contexts. It includes concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals to know the role that mathematics plays in the world and to make the well-founded judgments and decisions needed by constructive, engaged and reflective 21st century citizens. (OECD, 2018, p. 8[123])
The PISA 2012 (OECD, 2013[122]) definition and description of mathematical literacy was based around a model that assumed that when individuals use mathematics and mathematical tools to solve problems set in a real-world situation, they work their way through a series of stages as depicted in Figure 3.2 (OECD, 2013, p. 26[122]).
However, the PISA 2021 definition and description of mathematical literacy has extended the previous PISA 2012 model, and is based around a model that comprises two related aspects: mathematical reasoning and problem solving. When individuals use mathematics and mathematical tools to solve problems set in a real-world situation, they work their way through a series of stages as depicted in Figure 3.3 (OECD, 2018, p. 9[123]).
The formulating, employing, interpreting and evaluating processes are still key components of the mathematical modelling cycle that has underpinned the mathematical literacy construct in PISA since its beginnings. The mathematical reasoning process has been added as an explicit component in 2021 to highlight the centrality of mathematical reasoning to solving practical problems. The PISA mathematical reasoning aspect names these key understandings:
understanding quantity, number systems and their algebraic properties
understanding mathematics as a system based on abstraction and using symbolic representation
seeing mathematical structure and regularities
recognising functional relationships between quantities
using mathematical models as a lens into the real world
understanding variance as the heart of statistics (OECD, 2018, p. 16[123]).
As described in the draft PISA 2021 framework, these reasoning skills appear to mainly focus on reasoning skills within the world of mathematics, and mathematical reasoning is seen as a separate skill or process to the three problem solving processes of formulating, employing, interpreting and evaluating. As discussed further in the fifth section, this illustrates PISA’s interest in the ability of 15-year-olds to use and apply curriculum-based mathematical skills and knowledge, whereas this type of more formal mathematical knowledge is not generally assessed in PIAAC.
The PIAAC description and definition of numeracy can learn from the PISA definitions, descriptions and models in relation to the need to highlight different problem solving skills and processes, including reasoning skills, and being critical (making well-founded judgements and decisions) framed around using mathematical models as a lens into the real world. The relationships between the PIAAC and PISA frameworks and their descriptions and constructs are discussed further in the fifth section: Relationship between PIAAC and PISA.
Another existing model for numeracy in the twenty-first century is illustrated in Figure 3.4 below, and attempts to capture the multifaceted nature, and especially the critical dimension, of using mathematics to act in the real world. This model incorporates four dimensions of settings/contexts, mathematical knowledge, tools, and dispositions that are embedded in a critical orientation to using mathematics (Goos, Geiger and Dole, 2014, p. 84[124]). These dimensions are described more fully in other publications [e.g., (Geiger, Goos and Dole, 2014[125]; Goos, Geiger and Dole, 2014[124])]. Although primarily developed for use in relation to teacher education programmes and numeracy across the curriculum, the model and its components has some consistency with both the PISA model and PIAAC’s framework.
Both the PISA models and their sets of processes and this model for numeracy in the twenty-first century raise some issues to be considered in the redevelopment of the Cycle 1 definition and elaboration of numeracy for PIAAC Cycle 2.
Out of its research and review of conceptual and theoretical aspects of adult numeracy, the 2017 review recommended that there were four related issues to be explicitly addressed in updating and refining the existing PIAAC framework definition and description:
disposition to use mathematics
the ability to see mathematics in a numeracy situation
critical reflection
degree of accuracy.
Another issue that was raised in the review paper that is relevant here concerned the issue of authenticity, and this is also discussed below.
The issue of a person’s judgement on how to use mathematics (or not) in solving a numeracy problem is an important issue to address. The issue of choice or disposition when engaging with and solving a numeracy problem is an important factor to consider in an adult’s use and application of mathematics in a real-world situation (Geiger, Goos and Dole, 2014[125]; Goos, Geiger and Dole, 2014[124]). Are individuals able to choose to use mathematics when it is relevant and appropriate? This can also relate to mathematics anxiety and individual’s negative disposition to mathematics and their decision to avoid using mathematics, even when appropriate. Research about mathematics anxiety is well documented and demonstrates that it can have a significant impact on performance in mathematics [e.g., see: (Buckley, 2013[126]; Ma, 1999[127]; Tobias, 1993[128])].
There are three potentially related aspects behind this issue of disposition in relation to solving a numeracy problem where an adult is expected to use and apply some form of mathematical knowledge in a real-world situation:
using other means than mathematics to solve a problem when mathematics should have been the obvious and most sensible approach;
using formal mathematics when other sense-making methods would be more efficient; or
avoiding doing anything at all and not attempting to solve the numeracy problem at hand.
This issue of disposition is addressed more explicitly in the discussions about the elaboration of numeracy and numerate behaviours and practices at the end of this section.
Research indicates that an important aspect of a person’s numeracy or numerate behaviour is their capability to “see” or notice when mathematics is embedded in a real-world situation—how to recognise the mathematics and to potentially take the next step and act on it. The ability to see the mathematics that surrounds adults in their everyday life is an important issue in relation to being numerate—to potentially link the mathematics they learned in school with mathematics embedded in a real-world situation (Maguire and Smith, 2016[129]; Roth, 2012[130]).This issue is also identified as important in research about workplace numeracy, for example in calculating medication dosage (Weeks et al., 2013[98]).
Seeing mathematics in a numeracy situation relates to aspects of two of the processes described as part of PISA’s 2021 problem solving cycle for mathematical literacy: Mathematical reasoning and Formulating. In relation to Mathematical reasoning, before solving a problem, students need to “use their mathematics content knowledge to recognise the mathematical nature of a situation” (OECD, 2018, p. 9[123]). PISA also describes Formulating as: “seeing that mathematics can be applied to understand or resolve a particular problem or challenge presented” (p. 12).
As will be argued later, this aspect of being able to see and access the mathematics embedded in a numeracy situation and transposing the problem into a mathematical problem that can be solved is addressed explicitly in the revised numeracy framework and assessment construct through the new cognitive dimension.
While the current framework mentions the notion of critical reflection under the facet Responses in its elaboration of numerate behaviour, having a critical orientation or reflection are aspects of numeracy that could be emphasised and described further. It is important for individuals in their lives as citizens and workers to critically review the mathematics used and the outcomes obtained to reflect on and question real-world implications, to be capable of following up with appropriate actions, and to make decisions and judgements. A critical orientation is also about supporting an argument or position with mathematical evidence or challenging the argument or position of another person or organisation.
This capability to reflect critically and to act is named and described explicitly in some other models and frameworks [e.g., (Geiger, Goos and Forgasz, 2015[54]; Goos, Geiger and Dole, 2014[124])]. The third problem solving process in the PISA mathematical literacy problem solving cycle, which is called Interpreting and evaluating, includes elements of critical reflection: the need to reflect and make contextual arguments, to evaluate the reasonableness of solutions, and to critique and identify the limits of any models used. As well the new Mathematical reasoning aspect of PISA 2021 includes evaluating and making arguments, to evaluate interpretations and inferences related to statements and problem solutions (OECD, 2013[122]; OECD, 2018[123]).
The Cycle 1 PIAAC numeracy framework did not explicitly address the issue of the degree of precision or accuracy that may be required in the solution of a numeracy problem. It is expected that a numerate person would use estimation and other skills to check the outcomes and decide on the appropriate degree of accuracy required when solving a problem. This is particularly true within a workplace environment, where precision, accuracy and working within specified tolerances can be critical. On the other hand in other situations and applications, there are instances in being numerate where accuracy is not a critical component (e.g., in relation to some spatial skills, in graphical/data interpretation and analysis, or in estimating quantities, where an order of magnitude estimate can often suffice).
Another issue raised in the review paper concerns the issue of authenticity, and as mentioned in the Introduction, the NEG did further research into the related issues of embeddedness, numerate behaviour and numerate practices. They relate to the connection between the real-world context in which mathematics is embedded and to their roles as individuals, citizens, family members or as workers. This can mean that the way mathematics is used to operate on a task is fundamentally shaped by the context in which it is employed (Turner et al., 2009[131]). This includes socio-cultural influences that afford or constrain action in civic, personal or workplace environments. In this view, there is a clear separation between school mathematical knowledge, how it is taught, learned and practised, and the use of this knowledge outside of schooling. As Harris (1991[132]) notes:
In work [. . .] mathematical activity arises from within practical tasks, often from the spoken instruction of a supervisor and always for an obvious purpose which has nothing to do with the numbers working out well. Thus, students taught to react to isolated, abstract and written commands in the specialist language and carefully controlled figures of a school mathematics class, find themselves confronted with the urgent spoken, if not shouted, instructions in a completely different context and code. (p. 138)
Yasukawa, Brown and Black (2013[133]) make a clear connection between embeddedness and social practice arguing that numeracy practices cannot be understood independently of the social, cultural, historical and political contexts. They illustrate this point that make the comparison of students completing calculations individually, using paper and pen and perhaps a calculator against the use of mathematics in the supermarket, in which the same calculations completed at a checkout counter by the shop assistant using a cash register. In this situation the shopper might perform an estimation to avoid being overcharged. However, the shop assistant is equally concerned with charging the customer the correct price and recording accurate record of the items sold via the cash register. The calculations are fundamentally the same but the purpose—which is related to context—is different.
Authenticity of assessment tasks and word problems in mathematics education has been researched and documented [e.g., see (Hoogland et al., 2018[134]; Palm, 2006[135]; Palm, 2008[136]; Palm, 2009[137]; Stacey, 2015[138]; Verschaffel et al., 2009[139])]. In PIAAC it is important to have stimuli and questions that are based on authentic stimuli or scenarios with questions asked being ones that someone would want answered. While this is related to broader discussions about authentic assessment (Palm, 2008[140]) the focus in the PIAAC assessment programme is on the authenticity of the stimulus used and the questions asked. This matches what Palm describes as the “figurative context” where the context used in the assessment represents a situation taken from real life that has occurred, or might happen. PIAAC is interested in the ability of individuals to cope with tasks that are embedded in the real world, rather than assessing decontextualised mathematical tasks. This is in contrast to traditional school-based mathematical word problems which often disregard and challenge students’ sense making and only continue to distance students from the real world, and the usefulness and value of mathematics. The NEG believe that the assessment of numeracy in PIAAC is about promoting the belief that the value in mathematics is about its relationship with real-world things—whereas word problems often do the opposite. Another reason for PIAAC to utilise authentic situations in its questions is to encourage a more positive disposition towards solving relevant and engaging mathematics problems, not irrelevant, nonsensical word problems as can be met in school mathematics classrooms.
Hence, the PIAAC numeracy contexts and the items are developed by finding and identifying situations and tasks from different countries that provide authentic stimuli and then writing sets of questions using the information in the stimulus. Based on the description and understanding of numeracy in PIAAC, there have been deliberate attempts to avoid what are traditionally seen as school, curriculum-based word problems that are often contrived and have little real-world relevance or authenticity.
However, a challenge is that authentic situations and scenarios that involve mathematical concepts, and their related stimuli and materials, are often complex. In relation to textual components and reading demands, there are a range of issues in relation to the intersection between literacy and numeracy skills and the role that reading literacy aspects take in solving a numeracy problem. It is clearly acknowledged in the description of numeracy and its elaborations, and then reflected in the PIAAC complexity schema (PIAAC Numeracy Expert Group, 2009[2]; Tout et al., 2020[141]), that reading literacy is an integral and important aspect of numeracy. Certainly, in society and workplaces that adults occupy, tasks and challenges do not neatly divide into, or present as, discreet ‘literacy’ and ‘numeracy’ tasks. Real-world situations and demands cross those kinds of educationally defined boundaries.
The reality is that in using authentic situations as the basis for the numeracy assessment tasks where the mathematics is embedded in a real-world setting, the associated information and data can be very complicated, unfamiliar and involve a heavy reading load. This can create challenges in trying to focus the assessment on the mathematics and numeracy skills and knowledge. Hence a key goal in the item development process is to make the wording of numeracy items as simple and direct as possible, in order to help minimise the reading literacy demands.
Recent research that systematically compares descriptive mathematical assessment tasks with more depictive representations of problem situations through using illustrations and photos and minimising the use of words (Hoogland et al., 2016[142]; 2018[134]), gives an indication that even the use of simple supporting illustrations and images could improve performance by a small margin.
As introduced in the first section, Big Ideas in mathematics, is about describing powerful mathematical ideas central to the learning of mathematics, linking numerous mathematical understandings into a coherent whole [e.g., see (Charles, 2005[6]; Hurst, 2014[7]; Hurst and Hurrell, 2014[8]; Kuntze et al., 2009[9]; Steen, 1990[11])]. Initially, the term “Big Ideas” referred to how mathematical information can be classified in different ways compared with the traditional school mathematics curriculum content areas. Steen (1990[11]), for example, identified six broad categories pertaining to: quantity, dimension, pattern, shape, uncertainty, and change. Rutherford and Ahlgren (1990[143]) described networks of related ideas: numbers, shapes, uncertainty, summarising data, sampling, and Reasoning. Jones and his colleagues (2002[5]) provide a summary of the main contributions of the research to what they call powerful mathematical ideas, which included the following domains: whole number and operations, rational numbers, geometry, probability, data exploration and algebraic thinking and other underrepresented domains. It could be argued that being numerate means using the contents of all these domains not just as procedures (instrumental understanding) but in a critical/meaningful manner.
Charles (2005[6]) defines Big Ideas as “a statement of an idea that is central to the learning of mathematics, one that links numerous mathematical understandings into a coherent whole.” This view is also shared by other authors such as Hurst and Hurrell (2014[8]). In their article, they track the notion of Big Ideas in mathematics back to the work of Bruner (1960[144]), who inspired Clark’s (2011[145]) definition of a Big Idea as a “cognitive file folder” that we can file with “an almost limitless amount of information” (Clark, 2011, p. 32[145]). Big Ideas became conceptual structures (or schema) that can be used to provide a numeracy framework where content might be characterised by multiple connections. As Bruner (1960[144]), Hurst and Hurrell (2014[8]), Clark (2011[145]) and other authors claim, Big Ideas may become bridges for the transfer of learning.
Big Ideas in mathematics can also refer to processes (Kuntze et al., 2009[9]) where they include processes such as: Ordering; Classifying; Dealing with variation and uncertainty; Finding arguments and proofs; Formalising; Modelling; Generating and using algorithms, among others. Big Ideas have also been seen as a potential vehicle for making mathematics education a coherent and connected study. Descriptions of effective teaching of mathematics [e.g., (Ma, 1999[146]; Sullivan, 2011[147])] and related research (Askew et al., 1997[148]; Boaler and Humphreys, 2005[149]; Clarke and Clarke, 2004[150]) also consistently refer to the need for teachers to have a sense of how mathematics is a coherent and connected whole. In the Effective Teachers of Numeracy study (Askew et al., 1997[148]), this view was also supported, with highly effective teachers believing that being numerate requires “having a rich network of connections between different mathematical ideas.” This is in contrast to ways in which mathematical content knowledge is often reduced to lists of specific dot points in curriculum frameworks that Askew terms “death by a thousand bullet points” saying that “too much effort goes into specifying the knowledge that teachers need to know” (Askew, 2008, p. 21[151]). Hurst and Hurrell (2014[8]), quoting Charles (2005[6]), state that “Big Ideas” allow us to see mathematics as a coherent set of ideas, encouraging a deep understanding of mathematics.
These perspectives of Big Ideas in mathematics provide an overarching and integrative idea of mathematics and how mathematics is used in the world—that is, they are about framing and viewing mathematics as making connections with the real world, which is what underpins numeracy in PIAAC. It could be suggested that being numerate as defined within the PIAAC numeracy framework links to the idea of being able to access, use, interpret and communicate mathematical information around what the international scientific community calls Big Ideas in mathematics.
Reaching a consensus on a definition of numeracy that can fit an international programme of assessment is a challenging undertaking. First, as noted earlier, there are various country-specific connotations for numeracy, if such a term at all exists in a local language. Second, there are overlapping or competing constructs such as quantitative literacy, mathematical literacy, functional mathematics, and so forth [e.g., see (Gal and Tout, 2014[3]; Hagedorn et al., 2003[152]; Tout and Gal, 2015[37]; Tout and Schmitt, 2002[38])]. Third, an attempt to discuss the definition and meaning of numeracy is complicated by the fact different stakeholders already view it from within a given lens imposed by the historical and cultural aspects, whether organisational, social, economic, or linguistic, of the systems within which they operate. For example, some of the existing conceptions of numeracy were developed by educators working in delivery systems for schoolchildren, while other stakeholders link the term numeracy only to adult-related competencies.
As stated in the Introduction it is critical to note that the PIAAC numeracy assessment aims to describe the full range of numeracy capability in the adult population. This covers at one extreme, adults who have university level training and, at the other, adults who have very low levels of education (e.g. who left school at or before the age of 15). At the same time, it covers both young adults still in education and adults who completed their formal education 30-50 years prior to undertaking the assessment. As such, it incorporates a wide range of different mathematical and quantitative skills and knowledge, and is not based on a narrow view of numeracy that sees numeracy as only dealing with numbers and arithmetical operations. This will be expanded on later in the chapter.
In this section, the various aspects that are to be assessed, and eventually reported on, as part of the numeracy assessment in PIAAC are defined, described and elaborated. These aspects or characteristics of the assessment were called facets in ALL and PIAAC Cycle 1, but in PIAAC Cycle 2 they are called dimensions, which is consistent with the terminology used in literacy in PIAAC.
The initial sub-sections look at the refinement of the definition and description of numeracy from ALL through to PIAAC Cycle 1 and then to the new definition and description for this, the second cycle of PIAAC. For a backwards look at the development of numeracy definitions and developments it is best to read the PIAAC Cycle 1 framework (PIAAC Numeracy Expert Group, 2009[2]) or refer to the OECD Working Paper that compared the PISA and PIAAC frameworks (Gal and Tout, 2014[3]).
Next, an updated and refined definition of numeracy for PIAAC Cycle 2 is presented based on the research and review detailed in the second section, followed by a discussion of the dimensions of numerate behaviour and practices, including the core dimensions that comprise the numeracy assessment. This leads on to the next sub-sections, where the assessment construct is elaborated, described and defined in full.
The 2017 review report made a number of recommendations regarding the definition of the construct of numeracy and the priorities for development of the assessment framework for numeracy in the second cycle of PIAAC. Many of the suggestions arose out of the concern that the existing Cycle 1 framework and assessment did not reflect some of the realities of the skills and knowledge adults needed to succeed in work, life, and citizenship in the 21st century. These have been documented in the discussions in the previous section. The review and this document and its resulting definitions and elaborations, while building on the two previous conceptual and assessment frameworks and all the cumulative wisdom developed in connection with prior surveys of adult skills, have been able to enhance the numeracy framework and construct for PIAAC Cycle 2. The resulting framework and its associated definition, elaborations and assessment construct is contained below.
The conceptualisation of numeracy for the first international survey of adult numeracy, the Adult Literacy and Lifeskills (ALL) survey, was developed in 1998-2000 by an international team (Gal et al., 2005[28]). This was the first time the construct of numeracy had to be defined in a comparative assessment context and not purely in an educational context.
Numeracy was conceptualised and described in ALL as a much broader construct than Quantitative Literacy that was assessed in the earlier International Adult Literacy Survey (IALS). Quantitative Literacy was described in IALS as the knowledge and skills required to apply arithmetic operations to numbers embedded in printed materials. It was argued in ALL that numeracy requires more varied responses (order, count, estimate, compute, measure, interpret, explain) to a wider range of mathematical information (quantity, dimension and shape, pattern, change and relationships, and data and chance) that may be embedded in text in varying degrees.
Cognisant of the complexity and multifaceted nature of the numeracy construct, the ALL team developed a three-tier conceptualisation which attempted to reflect key perspectives of numeracy on the one hand, but also enable operationalisation of the construct in an assessment scale on the other. The three tiers are a brief definition of numeracy, a more elaborate definition of numerate behaviour, both presented below, and a detailed listing of components of the facets of numerate behaviour (Gal et al., 2005[28]).
Numeracy is the knowledge and skills required to effectively manage and respond to the mathematical demands of diverse situations.
Numerate behaviour is observed when people manage a situation or solve a problem in a real context; it involves responding to information about mathematical ideas that may be represented in a range of ways; it requires the activation of a range of enabling knowledge, factors, and processes.
Both the brief and elaborate definitions shown above were seen by the ALL numeracy team to be required, given the needs of a comparative assessment. A brief definition is essential to simplify communication with various stakeholders, such as policy makers and experts. However, as with most brief definitions of complex constructs, the language used is general and abstract, hence the definition cannot be explicit about what a numerate person can do in an assessment. With this in mind, a more detailed definition of numerate behaviour was developed as a way to emphasise different facets or dimensions that were seen by the ALL numeracy team as underlying numerate behaviour.
The advantage of using a more elaborate definition of numerate behaviour was that it is more explicit about what numeracy encompasses, and thus served as a springboard for developing an actual specification for an assessment. It is important to also note that the definition of numerate behaviour points to the presence of both cognitive and non-cognitive factors that underlie or enable effective numerate behaviour and practices. Ideally, coverage of both cognitive and non-cognitive aspects of numerate behaviour is essential in order to generate a full picture regarding the competence described by this view of numeracy. However, it needs to be acknowledged that the direct assessment component of PIAAC can only assess the cognitive aspects of numerate behaviour and that the non-cognitive aspects can only be addressed in proxy via responses to questions about skills use and data collected on respondents’ backgrounds.
The development of the conceptualisation and definition of numeracy for PIAAC went through several stages of work and consultation. An expert panel appointed to develop the overall assessment design for PIAAC presented in summer 2006 tentative recommendations regarding all competencies to be assessed in PIAAC (Gal, 2006[153]; Jones, 2006[154]; Murray, 2006[155]; Tout, 2006[156]) and then proposed to define numeracy as: "The ability to use, apply, and communicate mathematical information". Various perspectives on numeracy and its assessment were later examined by participants at the Canada-OECD Expert Technical Workshop on Numeracy, which met in November 2006 in Ottawa; a tentative working definition of numeracy was then proposed for PIAAC and included in a draft framework circulated for external review (Gal, 2006[153]). Further development of the numeracy framework was then undertaken by the Numeracy Expert Group for PIAAC appointed in April 2008, which released a revised framework for review by all participating countries in October 2008.
In general, work on the development of the numeracy framework for PIAAC Cycle 1, together with the assessment scale and related item pool, was conducted with two somewhat conflicting objectives in mind. One was the need to maintain compatibility with the conceptualisation of numeracy in the ALL survey, given the need for PIAAC to provide trend data related to ALL results. For this reason, PIAAC was designed with a specification that approximately 60% of the numeracy tasks that were to be employed in the final assessment would come from the item pool used in ALL. The other objective was the need to extend the ALL definition in light of PIAAC's overarching conceptualisation of “literacy competencies in the information age”, and consider new or emerging uses of numeracy in the adult world.
Taking all the above into consideration, numeracy was defined for PIAAC Cycle 1 as follows:
Numeracy is the ability to access, use, interpret, and communicate mathematical information and ideas, in order to engage in and manage the mathematical demands of a range of situations in adult life.
This definition captured essential elements in numerous conceptualisations of numeracy in the extant literature; was compatible with the definition used for ALL and provided a solid basis from which to develop an assessment for PIAAC with its emphasis on competencies in the information age. The inclusion of "engage" in the definition signalled that not only cognitive skills but also dispositional elements, i.e., beliefs and attitudes, are necessary for effective and active coping with numeracy situations.
As with ALL, the definition of numeracy developed for PIAAC Cycle 1 was not to be considered by itself, but again was to be coupled with a more detailed definition of numerate behaviour and with further specification of what were called the facets of numerate behaviour. This pairing was seen as essential in order to not only describe numeracy but to also enable operationalisation of the construct of numeracy in an actual assessment, and in order to further broaden the understanding of key terms appearing in the definition itself. Consequently, a definition of numerate behaviour similar in general terms to the one used for the ALL survey, but shorter, was adopted for PIAAC Cycle 1:
Numerate Behaviour involves managing a situation or solving a problem in a real context, by responding to mathematical content/information/ideas represented in multiple ways.
As with ALL, each of the different facets embedded within the definition of numeracy and the elaboration of numerate behaviour were defined and described. This included the same facets as ALL: contexts; responses; mathematical ideas/content; and representations.
Based on the discussions in the previous section: Conceptual and theoretical foundations, the PIAAC Cycle 2 NEG developed and agreed on a new definition for Cycle 2 of PIAAC numeracy:
Numeracy is accessing, using and reasoning critically with mathematical content, information and ideas represented in multiple ways in order to engage in and manage the mathematical demands of a range of situations in adult life.
In this updated definition and in the elaboration below, there are four core dimensions (previously called facets) described and used in PIAAC Cycle 2:
cognitive processes
content
representations
contexts.
The key words or terms that have changed or been introduced into the new definition include:
the use of the term ability has been deleted
access, use and reason critically has replaced access, use, interpret, and communicate
represented in multiple ways has been introduced into the definition.
The use of the term ability can imply that it is an “innate” ability that some people may not possess. The NEG firmly believe that adults (and children) all have the capacity to learn mathematics successfully and apply it successfully in their lives and hence be numerate.
Based on the views and research outlined in the previous section and raised in the review report, the NEG has substantially reworked the former Response facet (access, use, interpret, and communicate), and replaced with a more comprehensive description and elaboration of what is now called the core dimension of Cognitive Processes. This is named and described under three classifications:
access and assess situations mathematically (assess, identify, access and represent)
act on and use mathematics (order, count, estimate, compute, measure, graph and draw)
evaluate, critically reflect, make judgements (evaluate, reflect, justify and explain).
This has been done in the light of a number of the outcomes of the above research, both by the new NEG, but also from the recommendations made in the 2017 review paper. The major influences have been about the need to incorporate the aspect of being able to “see” mathematics in a real-world situation, and to include a critical aspect and the ability to reason and make judgements. This dimension has been able to be more explicitly used to drive the assessment development and the NEG believe this has greatly enhanced the item pool, helping to address a number of the concerns expressed in the review paper. It will also help develop and write the scale descriptions once the data and results are available. This dimension, along with the other core dimensions are elaborated further in the following section.
The third facet of the previous elaboration in ALL and PIAAC that has been added in and made explicit by including it in the new definition is that of representation. Although this was included in the previous elaborations it was not part of the definition, and did not help drive item development. Again, to address issues raised above, and specifically in relation to 21st century changes in how mathematical and quantitative information is now presented, the inclusion of represented in multiple ways has been introduced. This is named and described under four classifications:
text or symbols
images of physical objects
structured information
dynamic applications.
The latter two classifications have enabled the NEG, and hence the item writers, to explicitly address the issue of 21st century digitisation and technologically based materials and representations such as interactive websites, infographics, online calculators, spreadsheets and more.
The other facets in ALL and PIAAC Cycle 1, content and contexts remain, although there have been changes made to their labels and descriptions. All these core dimensions are named and elaborated below and in the following section.
As with previous cycles, the definition leads on to an elaboration of numerate behaviour, which is included in Box 3.2 below. This box first lists the direct assessment components of the four core dimensions of the definition, and these components are explained in more detail in the next sub-section. Additionally, the bottom part of Box 3.2 also lists several non-cognitive, enabling factors and processes, whose activation underlies numerate behaviour and successful numeracy practices. Most of these enabling factors and processes appeared in the ALL conceptual framework and PIAAC Cycle 1. Overall, the definition of numeracy and the description of numerate behaviour, with the details in Box 3.2 and the further explanations of the core dimensions following in the next section, provide the structure and roadmap for the development of the numeracy assessment as part of PIAAC Cycle 2.
Numeracy is an individual’s capacity to …
1. access, use and reason critically
access and assess situations mathematically (assess, identify, access and represent)
act on and use mathematics (order, count, estimate, compute, measure, graph and draw)
evaluate, critically reflect, make judgements (evaluate, reflect, justify and explain)
2. with mathematical content
quantity and number
space and shape
relationship and change
data and chance
3. represented in multiple ways
text or symbols
images of physical objects
structured information
dynamic applications
4. in order to engage in and manage the mathematical demands of a range of situations in adult life:
personal
work
societal/community.
An individual’s numerate capacity is founded on the activation of several enabling factors and processes:
context/world knowledge and familiarity
literacy skills
disposition, beliefs and attitudes
numeracy-related practices and experience.
Adults’ numeracy competence is revealed through their responses to the mathematical information or ideas that may be represented in a situation or that can be applied to the situation at hand. It is clear that numerate behaviour will involve an attempt to engage with a task and not delegate it to others or deal with it by intentionally ignoring its mathematical content. Numerate behaviour, however, depends not only on cognitive skills or knowledge bases, but also on several enabling factors and processes as listed in Box 3.2.
As outlined in the second section, including in the discussion about Numerate behaviour and practices, the PIAAC conceptualisation of numeracy operates on two levels. It relates to numeracy as a construct describing an individual’s capability to solve numeracy problems, and also to numerate behaviour and practice which is the way a person’s numeracy is manifested in the face of situations or contexts which have mathematical elements or carry information of a quantitative nature.
We argue therefore that numeracy as described in PIAAC is comprised both of cognitive elements (i.e., various knowledge bases and skills) as well as non-cognitive or semi-cognitive elements (i.e., attitudes, beliefs, habits of mind, and other dispositions) which together help to shape a person’s numerate behaviour and practices. Based on this, there are four non-cognitive or semi-cognitive enabling factors included in the elaboration of numerate behaviour:
context/world knowledge and familiarity
literacy skills
disposition, beliefs and attitudes
numeracy-related practices and experience.
Specifically, the enabling processes involve integration of mathematical knowledge and conceptual understanding with broader reasoning, problem-solving skills, and literacy skills. Further, numerate behaviour and numerate practices and autonomous engagement with numeracy tasks depend on the dispositions (beliefs, attitudes, habits of minds, etc.), and prior experiences and practices that an adult brings to each situation. These are briefly summarised below. Most of these enabling factors and processes have also been described by Kilpatrick, Swafford and Findell (2001[157]) as part of their analysis of the construct of mathematical literacy, and further examined and deemed relevance for description of adult numeracy in the analysis by Ginsburg, Manly and Schmitt (2006[29]).
It should be noted that the direct assessment via the numeracy test component in PIAAC has, as its primary emphasis, the cognitive aspects of numerate behaviour as framed in the first part of Box 3.2, namely the numeracy and mathematical knowledge and skills that underpin answering the test questions, which are mediated by written materials, without oral support, in the context of a formal assessment. The non-cognitive aspects of numerate behaviour, are addressed indirectly through other components of the PIAAC assessment, namely through the skills use questions and the comprehensive background questionnaire.
Proper interpretation of mathematical information or quantitative messages by adults depends on their ability to place messages in a context and access their world knowledge, as well as rely on their personal experiences and practices. World knowledge also supports general literacy processes and is critical to enable “sense-making” of any message. For example, adults’ ability to make sense of statistical claims or media-based graphs will depend on information they can glean from the message about the background of the study or data being discussed. When interpreting statistical claims made by journalists, advertisers and the like, context knowledge is the main determinant of the reader’s familiarity with sources for variation and error, and helps to imagine why a difference between groups can occur (as in a medical or educational experiment), or what alternative interpretations may exist for reported findings about an association or correlation between certain variables. Likewise, world knowledge is a prerequisite for enabling critical reflection about statistical messages and for understanding the implications of the reported findings.
Different people will have very different settings and applications in which they may comfortably and more confidently use and apply their mathematical knowledge, often related to their familiarity with the actual context and the mathematics that is embedded in the situation at hand. This is related to the discussion in the previous section about numeracy practices, authenticity and the embedded nature of mathematics within a numeracy context. Finding the right problem situation or setting for each individual so that they can demonstrate their understanding of mathematics concepts will be a challenge in such an assessment as PIAAC. Hence, it is important to not have contexts, especially workplace contexts, which are too technical or so uncommon that most adults faced with that stimulus and questions will not be at all familiar with the context, and would be potentially locked out from engaging with and answering the question. This is one of the key challenges in writing items for PIAAC – to make them all relatively accessible and realistic for all the respondents, but still with a wide range of difficulty and complexity.
It is clearly acknowledged in the description of numeracy and its elaborations, and then reflected in the PIAAC Complexity schema (PIAAC Numeracy Expert Group, 2009[2]; Tout et al., 2020[141]), that literacy is an integral and important aspect of numeracy. Certainly, in society and workplaces that adults occupy, tasks and challenges do not neatly divide into, or present as, discrete ‘literacy’ and ‘numeracy’ tasks. Real-world situations and demands cross those kinds of educationally defined boundaries.
The reality is that in using authentic situations as the basis for the numeracy assessment tasks where the mathematics is embedded in a real-world setting, the associated information and data can be very complicated and potentially involve a heavy reading load. In cases where “mathematical representations” involve text, one's performance on numeracy tasks will depend not only on formal mathematical or statistical knowledge but also on reading comprehension and literacy skills, reading strategies, and prior literacy experiences. For example, following a computational procedure described in text (such as the instructions for computing shipping charges or adding taxes on an order form) may require special reading strategies, as text is very concise and structured. Likewise, analysing the mathematical relationships described in words requires specific interpretive skills, as in the simple case of recognising the similarity of “the price doubled” and “the price was twice as high”, but the different meanings in “production levels were constant over the last five years” and “production levels constantly increased over the last five years.”
This creates challenges in trying to focus the assessment on the mathematics and numeracy skills and knowledge. Hence a key goal in the item development process is to make the wording of numeracy items as simple and direct as possible, in order to help minimise the reading literacy demands. Recent research that systematically compares descriptive assessment tasks with more depictive representations of problem situations through using illustrations and photos and minimising the use of words (Hoogland et al., 2016[142]; 2018[134]), gives an indication that even the use of simple supporting illustrations and images could make the contexts and questions more accessible. Lowrie and Diezmann (2009[158]) also researched the impact of supporting graphics and illustrations in numeracy test items, and argued that the design of mathematics assessment items is more likely to be a reliable indication of student performance if graphical, linguistic and contextual components are considered both in isolation and in integrated ways as essential elements of task design.
The issue of choice or disposition when engaging with and solving a numeracy problem is an important factor to consider in an adult’s use and application of mathematics in a real-world situation, and was addressed in the second section, including in the sub-section about Numerate behaviour and practices. Research literature suggests that the ways in which a person responds to a numeracy task, including overt actions as well as internal thought processes and the adoption of a critical stance, depend not only on their knowledge and skills but also potentially on their disposition and attitude towards mathematics. Negative attitudes towards mathematics, beliefs about one's mathematical skills, habits of mind, and prior experiences involving tasks with mathematical content are all key influencers on mathematics engagement and performance, alongside beliefs about mathematics and what it is for and who it is for (Geiger, Goos and Dole, 2014[125]; Goos, Geiger and Dole, 2014[124]; Lave, 1988[53]; Saxe, 1992[109]; Schliemann and Acioly, 1989[111]). Are individuals able to choose to use mathematics when it is relevant and appropriate?
This also relates to mathematics anxiety. In some cultures, some adults, including highly educated ones, decide that they are not “good with numbers” or have other sentiments or self-perceptions usually attributed to negative prior experiences they have had as pupils of mathematics. Such attitudes and beliefs stand in contrast to the desired sense of “at-homeness with numbers” (Cockcroft, 1982[44]) and can interfere with one’s motivation to develop new mathematical skills or to tackle math-related tasks, and may also affect test performance (McLeod, 1992[159]). Research about mathematics anxiety is well documented and demonstrates that it can have a significant impact on performance in mathematics [e.g., see: (Buckley, 2013[126]; Ma, 1999[127]; Tobias, 1993[128])].
In real-world contexts, adults with a negative mathematical self-concept may elect to avoid a problem with quantitative and mathematical elements, address only a portion of it, or prefer to delegate a problem, e.g., by asking a family member or a salesperson for help. Such decisions or actions can serve to reduce both mental and emotional load (Gal, 2000[27]). Yet, such actions may fall short of autonomous engagement with the mathematical demands of real-world tasks, carrying negative consequences, e.g., not being able to fully achieve one’s goals.
The discussion in the previous section about the research and issues about numerate behaviours and practices, and about the relationship between school mathematics and workplace mathematics, demonstrates that numerate behaviour and practices do not rely only on mathematical knowledge or related reasoning and problem-solving skills acquired as part of formal learning in a school context. Both attitudes and beliefs as well as numeracy-related practices and world knowledge are important enabling processes and may influence adults' ability to act in a numerate way. Therefore, scales assessing selected attitudes and beliefs about mathematics, and numeracy-related practices in work, everyday, and other settings, have been developed for PIAAC's background questionnaire. Information collected by such scales can help to explain differences in performance among adults, further informing our understanding of factors that affect skill acquisition and retention or motivation for further learning. They can be used to help explain the links between numeracy performance and covariates such as participation in a range of numeracy practices in their lives including at work, participation in further learning or employment/unemployment status.
Further, the frequency of engaging with mathematical tasks or of exposure to mathematical or statistical information or displays, whether at work, home, when shopping, or in other contexts, is of much interest. Engagements or practices in this regard can be both the result of a certain skill level, but also the cause of observed skill levels, or at a minimum a factor influencing observed skill level apart from prior formal schooling.
The above enabling factors address the issue of the non-cognitive aspect of numerate behaviour and practice which is the way a person’s numeracy is manifested in the face of situations or contexts which have mathematical elements or carry information of a quantitative nature. They are addressed indirectly in PIAAC through the other components of the PIAAC assessment, namely the skills use questions and the comprehensive background questionnaire.
The fourth section of the framework discusses the operationalisation of the construct of numeracy in PIAAC and how this is affected by many factors which determine and shape the extent to which the theoretical construct can be fully addressed by the actual collection of test items used in the direct assessment.
This sub-section elaborates on the dimensions incorporated into the definition of numeracy and the elaboration of numerate capacity, as outlined in the first part of Box 3.2. Elaborations on the original facets of the previous two cycles were based on previous research and materials documented in both the ALL and PIAAC Cycle 1 framework. Key in that work was the analysis of the components of adult numeracy by Ginsburg, Manly and Schmitt (2006[29]) which was based on an integrative review of multiple numeracy frameworks from several countries. It also benefited from the positions presented in a report of the UK’s National Research and Development Centre for Adult Literacy and Numeracy (NRDC) (2006[35]), background papers prepared for the OECD-Canada Expert workshop on numeracy (November 2006, Ottawa) and suggestions made by workshop participants. Input was also received from external reviews of early drafts of the PIAAC Cycle 1 framework, and professional perspectives of PIAAC's Cycle 1 Numeracy Expert Group.
For Cycle 2, these facets, or as they are now called, dimensions, have been further developed and substantially enhanced, mainly based on the 2017 review paper (Tout et al., 2017[1]). A significant factor to reworking the core dimensions was the comparison with the PISA 2012 mathematical literacy framework and its classifications. The comparison with PISA mathematical literacy is addressed explicitly and in detail in the fifth section: Relationship between PIAAC and PISA. Specific issues arising from this are incorporated into the discussions below.
There are four core dimensions named and described in numeracy for PIAAC Cycle 2, namely:
cognitive processes
content
representations
context.
Each of these four core dimensions are elaborated below.
This dimension is new to PIAAC Cycle 2 and replaces the previous Response facet of PIAAC and ALL. It also incorporates to some extent the first of the facets described under the category of Mathematical knowledge and conceptual understanding in the enabling processes elaboration in both ALL and PIAAC Cycle 1. This facet addressed the notion of conceptual understanding. This referred to an integrated and functional grasp of mathematical ideas (Kilpatrick, Swafford and Findell, 2001, p. 118[157]). Ginsburg, Manly and Schmitt (2006[29]) suggest that the two aspects of conceptual understanding, i.e., it being integrated and functional, frame the ability to think and act effectively as a numerate adult, and that across different numeracy frameworks in different countries, equivalent terms are used such as “meaning making,” “relationships,” “model,” and “understanding.” Conceptual understanding can help produce reasonable estimates that can help adults catch computational errors, or realise that an exact product is not necessary, but an estimate is enough for the purpose. Ginsburg, Manly and Schmitt (2006[29]) further explain that conceptual understanding permits one to be free from relying on memory for all methods and procedures, i.e., an adult can think about the meaning of the task and “construct or reconstruct” a representation that both illustrates what it means and suggests a method for solution.
The Cycle 1 framework described and elaborated how in different real-life situations, adults may have to react to a numeracy problem with different types of responses or actions. The Cycle 1 framework grouped those under three broad headings: identify, locate, or access; act upon or use; and interpret, evaluate/analyse, communicate.
The PISA 2012 to 2018 mathematical literacy framework described and used three processes—formulating, employing, interpreting and evaluating—as key components of their mathematical modelling cycle. The mathematical reasoning process has now been added as an explicit component in PISA 2021 to highlight the centrality of mathematical reasoning to solving practical problems. As described in the draft PISA 2021 framework, these reasoning skills appear to mainly focus on reasoning skills within the world of mathematics, and mathematical reasoning is seen as a separate skill or process to the three problem solving processes of formulating, employing, interpreting and evaluating. This highlights PISA’s interest in the ability of 15-year-olds to use and apply more formal mathematical skills, knowledge and representations, whereas this type of more formal mathematical knowledge is not generally assessed in PIAAC. For PISA 2021, it is acknowledged that the assessment items will be assigned to either mathematical reasoning or one of the three mathematical processes associated with real-world based mathematical problem solving. The PISA 2021 goal “is to achieve a balance that provides approximately equal weighting between the two processes that involve making a connection between the real world and the mathematical world (formulating and interpreting/evaluating) and mathematical reasoning and employing which call for students to be able to work on a mathematically formulated problem” (OECD, 2018, p. 33[123]).
Based on the views and research outlined earlier and raised in the review report, the NEG has substantially reworked the former Response facet, and replaced it with a more comprehensive description and elaboration of what is now called the dimension of Cognitive processes. This was to more explicitly describe and address the way adults have to deal with solving a problem embedded in an authentic context. The skills adults need in the 21st century cover not only a range of specific mathematical knowledge and problem solving skills, but include the ability to recognise and identify how and when to use mathematics; to be able to understand, use and apply mathematical concepts and procedures; along with strategic, reasoning and reflective skills to use when using and applying the mathematics.
This is also derived in part from the comparison with the PISA processes as part of the PISA problem solving and modelling process. Unlike a number of the other facets of numerate behaviour in PIAAC Cycle 1 and their related descriptions, this facet of responses or actions had the least in common between PISA and PIAAC. It is the view of the NEG, that for the assessment of numeracy skills of adults the mainly intra-mathematical aspect of mathematical reasoning, as added to the PISA 2021 mathematical literacy construct, needs to be embedded within the real-world problem solving aspect for PIAAC, and not assessed as a separate part of the construct. Therefore mathematical reasoning understanding is integrated into the relevant aspects of the three cognitive processes.
The revisions and enhancements to this facet or dimension also more closely met the need to address a range of factors to do with both 21st century skills and the need to be more reflective and be able to reason and think critically, and make judgements. The NEG believes that this enhanced and more explicitly defined and described Cognitive process dimension has supported the test developers to write new types of items and has greatly enhanced the item pool, helping to address a number of the concerns expressed in the review paper. It will also help develop and write the scale descriptions once the data and results are available.
Table 3.2 below compares the terms used for the cognitive process or response-related descriptions in PIAAC Cycle 1 with the three processes used in PISA 2012 and the four processes used in PISA 2021 including against the new Cognitive process of PIAAC Cycle 2.
Solving problems in real-world contexts requires a range of capabilities and cognitive processes. When engaging with a real-world problem, one of the decisions to be made is whether the use of mathematics is relevant and then if it is best way to solve a problem. If the use of mathematics is deemed appropriate, the essential features of the problem will need to be identified in order to turn the real-world situation into a mathematical problem. From this point, relevant mathematical content, procedures, processes and tools needed to solve the problem must be identified and accessed by the problem solver. Once accessed, these procedures and processes will need to be employed correctly and decisions made about the appropriate degree of accuracy required to yield a mathematical solution. The solution needs to be reflected on and evaluated against the original problem situation in terms of its reasonableness and relevance to the real-world context and a decision made about whether to accept the solution or to revisit aspects that require refinement. In cases where decisions or judgements are being made on the basis of the solution, other factors might also be considered such as social or economic consequences.
So the first core dimension described in the PIAAC definition and elaboration of numeracy is about the cognitive skills and processes required to engage with and solve the task or problem at hand. These have been named as:
access and assess situations mathematically
act on and use mathematics
evaluate, critically reflect, make judgements.
It is important to understand that these activities are not mutually exclusive of one another or that they take place in a rigidly linear manner. For example: the identification of a problem’s essential features will have consequences for the identification of relevant mathematics to be engaged; an inability to access a particular area of mathematics may result in the selection of mathematical procedures and processes that are less effective; or the evaluation of the solution against the original problem situation may indicate those features, identified as essential, were not as relevant as first thought and so backtracking through the steps of the solution is necessary. Thus, while the cognitive processes outlined in this sub-section are described separately, the activity of addressing a real-world problem via mathematical means should be considered first and foremost as a holistic process.
It will be the combination of these three processes and their components that drive the difficulty and complexity of each numeracy problem being solved and each question asked in PIAAC numeracy units and items. After the description of each cognitive process below, there are a number of key questions outlined that describe the issues and factors that will influence the complexity of each process.
Note: for the purpose of guaranteeing a spread of types of items across PIAAC that focus on or emphasise the different aspects of these cognitive processes, each item has been prioritised against one of the three processes.
When adults encounter problems within real-world contexts they must first decide if mathematics is an appropriate means to engage with the situation. Once they deem the use of mathematics will provide advantage in addressing the problem, they need to identify the essential features to be accommodated when transforming the real-world situation into a mathematical problem. This transformation requires adults to look forward and identify and access the mathematics and mathematical representation embedded in the specifics of the situation, and make decisions about how the task can be represented and solved mathematically. The direction of the thinking and reasoning in this process is going from the real world to the mathematical world.
The actions that underpin assessing situations and accessing the mathematics in order to solve a real-world problem include:
identifying the essential features of a real-world problem that can be represented mathematically
identifying and describing/defining the mathematical operation(s), processes and tools needed to solve the problem
simplifying a situation or problem in order to represent it mathematically, using appropriate representations, for example, variables, symbols, diagrams, and models
representing a problem in a different way, including organising it according to mathematical concepts and making appropriate assumptions
anticipating the real-world restrictions on the possible outcomes of decisions made while defining and representing the problem.
How is the mathematics represented and embedded within the real-world situation? Through words and language? Through numbers and symbols, diagrams, pictures, graphs and charts? How informal, formal or complex are the mathematical representations and the mathematical information?
Is a mathematical approach suitable for the presented situation – is the use of mathematics a sensible way to address the real-world problem? If so, what is the degree of transformation required of the real-world situation to move it into a mathematical problem? How implicit or explicit/obvious is it to decide on the mathematical problem solving solution? Is the question presented in an unambiguous way so that necessary mathematical processes and procedures can be identified?
What literacy skills are required to make this transformation – what are the reading demands, how much distracting information is there?
Will a decision need to be made about how well the solution generated by solving the mathematical representation of the problem matches the contexts of the original real-world situation? How complex is that decision?
Adults utilise mathematical processes, facts and procedures in order to derive results and solve real-world problems, and will need to select and use appropriate tools, including technology. For example, they may need to perform arithmetic computations; select, create, solve equations; make logical deductions from mathematical assumptions; perform symbolic manipulations; create and extract information from mathematical tables and graphs; represent and manipulate geometrical objects in 2D and 3D; and analyse data. Mathematical processes and procedures used to solve real-world problems include:
applying mathematical facts, rules and structures
performing arithmetic computations and applying routine algorithms
undertaking measurements
looking for a pattern
using symbolic, formal, and technical language and mathematical conventions
using mathematical tools, including technology
manipulating numbers, graphical, statistical and chance-based data and information, algebraic expressions and equations, geometric representations
collecting, organising, structuring and representing information
generating estimations and approximations
making and extracting information from mathematical diagrams, graphs, infographics and constructions
reviewing and reflecting upon initial or part solutions
generalise from a more complex mathematical situation to a simpler mathematical problem/situation that can be more easily solved.
How difficult and complex are the mathematical concepts, facts, processes and procedures that need to be used and applied?
What level of mathematical reasoning, arguing, manipulating and computing is required for an effective response to the problem?
How many steps and types of mathematical steps/processes are required to solve the problem? Is it one operation, action or process or does it require the integration of several steps covering more than one different operation, action or process?
Responses to real-world tasks, including any mathematical solutions, judgements, decisions or conclusions, require reasoning and critical reflection and evaluation. Any solution of a real-world problem needs to be evaluated against the original problem situation in terms of its reasonableness and relevance to the original context and a decision made about whether to accept the solution or to revise and adjust the solution—often referred to as contextual judgement. In cases where decisions or judgements are being made on the basis of the solution, other factors might also be considered such as social or economic consequences. This will require that responses include explanations and justifications for decisions, judgements and conclusions that are reasonable and make sense within the context of the original situation. Critical reflection and evaluation within real-world contexts requires:
evaluating the reasonableness of a solution or part solution to a problem. This includes consideration of the appropriateness of estimations and/or the degree of accuracy required
understanding the real-world implications of solutions generated by mathematical methods, in order to critically reflect and make judgements about how the results should be adjusted or applied
using mathematical arguments to construct, defend or challenge decisions and/or judgements
considering social norms and influences, in addition to physical constraints, when considering the validity or effectiveness of a mathematical solution to a real-world problem
reflecting on mathematical processes and arguments used and explaining and justifying results
identifying and critiquing the limitations inherent in solving some real-world problems.
How complex is it to evaluate, reflect, justify, and explain and connect the mathematical outcomes to the real-world context? Does the task require a choice from a number of provided solution options? Or does the task require an explanation to be derived or decided upon with no provided solutions?
How complex is it to justify the validity of the mathematical outcomes and evidence with the essential elements of the original real-world problem? To what extent does the task require judgement about the quality of a mathematical argument used to defend or challenge a proposition within a real-world context?
How complex is it to connect the mathematical evidence to the essential elements of the real-world problem? To what extent does the task require judgement about the appropriateness and reasonableness of a proposed result to the real-world context? To what extent does the mathematical result need to be adapted to fit in with the original real-world context? Does it require consideration of the appropriateness of estimations and/or the degree of accuracy required?
These three Cognitive processes are linked to the Numeracy Complexity Schema described further in the fourth section: Operationalisation of the PIAAC Numeracy Assessment and detailed in Tout et al. (2020[141]). It is believed that the cognitive processes will drive much of the item difficulty and that together with the descriptions and scores described in the Complexity Schema, these will help to describe performance when it comes to elaborating the different levels in PIAAC.
Mathematical information can be classified in several ways and on different levels of abstraction. One approach is to refer to fundamental “Big Ideas in mathematics” (see discussion in the second section). Steen (1990[11]), for example, identified six broad categories: Quantity, Dimension, Pattern, Shape, Uncertainty, and Change. Rutherford and Ahlgren (1990[143]) described networks of related ideas: Numbers, Shapes, Uncertainty, Summarising data, Sampling, and Reasoning. Dossey (1997[160]) categorised the mathematical behaviours of quantitative literacy as: Data representation and interpretation, Number and operation sense, Measurement, Variables and relations, Geometric shapes and spatial visualisation, and Chance. More broadly, many curriculum frameworks around the world in one way or another refer to these key areas, albeit using somewhat different terminologies and with somewhat different groupings [e.g., National Council of Teachers of Mathematics (NCTM) (2000[120])].
This dimension remains similar to PIAAC Cycle 1 and is similar to the equivalent facet in PIAAC and ALL. There are some name changes, partly to make them more consistent with the PISA mathematical literacy descriptions and labels for Content (see Table 3.3).
Four key areas of mathematical content, information and ideas are described and used in the numeracy assessment in PIAAC:
Quantity and number
Space and shape
Change and relationships
Data and chance.
For an individual item in PIAAC numeracy, these four content areas are not mutually exclusive and any item may involve one or more of these mathematical content areas. For example, a unit and item in Data and chance will necessarily also include data that will be expressed as a quantity or number, and similarly a measurement item in Space and shape will be expressed as a quantity or number. The classification of such items is based on what content area the key conceptual understanding and skill is directed at.
The notion of quantity and number is a fundamental and essential mathematical aspect of engaging with, and functioning in, our world. The Quantity and number content area involves understanding ordering, counts, place value, magnitudes, indicators, relative size and numerical trends. This will encompass aspects of quantitative reasoning, such as number sense, multiple representations of numbers, computation, mental calculation, estimation and judging the reasonableness of results. This content area requires knowing and applying integers, rational and irrational numbers, positive and negative numbers and equivalence. It also requires understanding and applying number operations, including order of operations, in a wide variety of settings.
Identify and counting the number of items shown in a photo of a set of items or object.
Calculating the cost of one can of soup, given the cost of 4.
Calculating the cost when buying 0.283 kg of cheese at a given price per kg.
Another example could be deciding whether given decimal numbers are within a given range.
The Space and shape content area encompasses a wide range of phenomena that are encountered everywhere in our visual and physical world. It includes an understanding and use of: measurement (informal and standardised) systems, measurement formulas; dimensions and units; location and direction; geometric shapes and patterns; angle properties; symmetry; transformations and 2D and 3D representations and perspectives. This content area requires understanding and interpreting measurements and scales, position and orientation, plans, models, maps and diagrams, and navigation (including understanding travel distances, speeds and times, and using tools such as Global Positioning Systems).
The identification of a shape or matching an image of a real object to the correct plan/diagram.
Reading the weight/mass of an object off an analogue scale.
Interpreting an online map in relation to travel distances, speeds and times.
Working out quantities required for a task such as wallpapering or tiling or painting given particular dimensions.
The Change and relationship content area includes the ways to describe, model and interpret mathematical relationships, quantitative patterns, and change, where they occur in the real world. Real-world variables can be based around linear and non-linear relationships. Such relationships can be represented by descriptions, picture or images, tables, graphs or formula. In the latter case it could require the understanding and use of algebraic expressions and related methods of solution. This content area requires understanding, using and applying proportional reasoning and rates of change, including the use and application of ratios. It also requires recognising, describing, and/or using a relationship between different variables derived from a real-world situation.
Comparing the different proportional discounts on a shopping item in two different sales where the discounts are displayed in different ways.
Understanding and using formulae such as for calculating interest or inflation rates, or one’s BMI (Body Mass Index).
Understanding and applying proportional reasoning to calculate values based on existing percentage or proportions of quantities/ingredients.
Understanding and applying linear growth in order to predict future growth or decline.
The Data and chance content area encompasses topics such as data collection, data displays, charts and graphs, measures of central tendency and variance, alongside understanding appropriate approaches to data collection and sampling. The representation and interpretation of data are key concepts in this category. This content area also includes understanding and knowing about chance and probability. Chance and probability encompass subjective probability, certainty and uncertainty, likelihood and unlikelihood, prediction, and decision making. For example, attaching a numerical value to the likelihood of an instance is a ubiquitous phenomenon no matter whether it has to do with the weather, the stock market, a medical prognosis or the decision to board a plane.
Interpreting and identifying particular information on a simple bar graph or pie chart.
Using an interactive online data tool and chart to interpret and analyse provided data.
Use and understand averages (mean) to calculate required targets.
Sort and interpret a set of data to test a number of opinions about the set of data.
Context is the parameter or term used in both PISA and PIAAC for naming and classifying the settings or situations where people use and apply their mathematical knowledge to solve a realistic problem. The main purpose behind the use of the chosen context categories is to ensure a mixture or blend across the different categories to help guarantee some degree of balance in the assessment, with no particular context overwhelming the others (and therefore advantaging or disadvantaging respondents with greater or lesser daily interaction with some settings/contexts).
In PIAAC Cycle 1, the contexts used were:
Everyday life
Work-related
Societal or community
Further learning.
The sets of descriptors used in both PISA and the PIAAC Cycle 1 frameworks regarding the first three contexts (Everyday life/Personal; Work-related/Occupational; Societal or Community/Societal) were highly consistent with each other. One of the review team’s recommendations was that the PISA label Personal is preferable to the PIAAC label of Everyday. “Everyday” suggests some “sameness” in what people do which is not particularly illuminating, whereas the term Personal aims to indicate that the issue at hand bears most directly just on that individual. This has been implemented.
Further Learning in PIAAC Cycle 1 was another context that the review project recommended for reconsideration. Further learning has some similarity and consistency with the term intra-mathematical that PISA refers to within its description of Scientific:
… Particular contexts might include (but are not limited to) such areas as weather or climate, ecology, medicine, space science, genetics, measurement and the world of mathematics itself. … Items that are intramathematical, where all the elements involved belong in the world of mathematics, fall within the scientific context. (OECD, 2013, p. 37[122])
This "Scientific" context of PISA has two elements to it. Some items classified as Scientific in PISA are in fact "intra-mathematical", that is, situations which are within the world of mathematics, explicitly related to knowing about formal aspects of mathematics, with no, or little, real-life connections. There were, in fact, no questions in PIAAC that are purely intra-mathematical, as there can be in PISA. There was a second set of questions in PISA that were classified as Scientific, where the situation or context related to the natural world (e.g., climate or ecology). In PIAAC Cycle 1 this context of Further learning was described as being related to adults needing to solve problems that may arise when participating in further study, whether for academic purposes or for vocational training, and was explicitly related to knowing about the more formal aspects of mathematics, including the conventions used to apply mathematical rules and principles. But the actual items could also be classified against the other three contexts. The sample PIAAC Cycle 1 item from this context that is discussed in the PIAAC reports was the item “Candles” (OECD, 2013, p. 77[161]). However, this item could also have been classified as Everyday life, Work-related, or even Societal or community.
For this, and a number of other reasons, the 2017 review paper recommended that the NEG review this fourth context of Further learning. The NEG considered this and decided that it was best to remove the classification named Further Learning. As a result, only the first three contexts continue in PIAAC Cycle 2, as these were the most relevant to adults, and that any existing Further learning items in PIAAC Cycle 1 should be reclassified against one of the other three contexts. The need to have items that were about knowing about the more formal aspects of mathematics, including the conventions used to apply mathematical rules and principles, would be covered through the inclusion of those requirements through the content knowledge area of Change and Relationship.
Hence there are some name changes, partly to make them more similar to the PISA mathematical literacy descriptions and labels for Context. The three versions of the Context labels are described in Table 3.4.
People try to manage or respond to a situation involving numeracy and mathematics because they want to satisfy a purpose or reach a goal. Three types of contexts that may require the use and application of numeracy skills are described below:
Personal
Work
Societal/community.
These are not mutually exclusive and may involve the same underlying mathematical themes. The capability of being critically reflective about the use and application of mathematics is important in the 21st century, and adults need to be able to make decisions and judgements, and defend or support arguments. The different contexts provide the different areas of their lives where adults may encounter numeracy situations and which therefore provide the purpose for engaging in, solving and being reflective about real-world problems involving mathematics.
Numeracy tasks are often encountered in personal and family life, or revolve around hobbies, sports and games, personal development and personal interests. The personal context focuses on activities for an individual and in their interactions with immediate family. Representative tasks include (but are not limited to): handling money and personal or family finances and transactions, health and well-being, activities with family and friends, shopping, personal time management, travel and holiday planning, including reading maps, and using measurements in home situations such as cooking, gardening, administering medicines, or doing home repairs.
Adults often encounter mathematical situations at work that are more specialised than those in everyday life. Today’s workplaces often require increasing levels of techno-mathematical literacy. Representative tasks include (but are not limited to): completing purchase orders, maintaining inventories, totalling receipts, calculating change, managing schedules, budgets and project resources, payroll/accounting, using spreadsheets, completing and interpreting production and control charts, managing production inputs and outputs, tracking costs and expenditures, interpreting results from technological devices, and applying formulas. Work-related tasks can also include reading plans, blueprints and workplace diagrams, having spatial awareness for best storage options and organising and packing goods, and planning the most efficient delivery journey. This context can also include making and recording measurements such as lengths, weights, temperatures, dosages, areas, volumes or other work-related measurements, and using and applying measurement ratios and formulas. Occupational contexts may relate to any level of the workforce, from unskilled work to the highest levels of professional work.
Adults need to know about quantitative data and statistics and their representations, and be able to interpret trends and the consequences of a range of activities and actions happening in the world around them at the local, national or global level. Adults need to know about and be able to understand different mathematical relationships, such as proportional reasoning, when reading and interpreting information presented by a range of community or government authorities. Adults also may take part in a range of social events or community activities, including social and political participation, organising and participating in community-based functions and fundraising. Representative tasks include (but are not limited to) understanding and interpreting financial, statistical and numerical information and graphs about public transport, crime, health, education, politics, demographics, pollution, community events, etc. This information is increasingly being presented by the media, government services, financial institutions, utilities, and by a range of community services and organisations.
The third facet of the previous elaboration in ALL and PIAAC that has been added into the actual definition and hence made more explicit is that of Representation. Although this was included in the previous elaboration, it was not part of the definition, and did not help drive item development. Under Facet 4: Representations of mathematical information, the PIAAC Cycle 1 numeracy framework stated that mathematical content/information/ideas can be represented in multiple ways: objects and pictures; numbers and mathematical symbols; formulae; diagrams and maps, graphs, tables; texts; and finally, technology-based displays. However, none of these was expanded in much detail (PIAAC Numeracy Expert Group, 2009, p. 28[2]) and although the issue of different forms of representation of information is raised, digital or dynamic formats are not addressed.
The nature of information graphics is only now being unpacked within the field of mathematics education. Diezmann and Lowrie (2008[162]), for example, have argued for the importance of becoming proficient in interpreting information graphics (e.g., graphs, tables, maps) as these are increasingly used to manage, communicate, and analyse information. Societies are becoming increasingly reliant on representing information both diagrammatically and graphically. The new, more dynamic representation of data and information needs to be addressed. It is now no longer a matter of interpreting static images, as in the existing PIAAC Cycle 1 item pool, but also how new scenarios and different problems can be posed by interpreting and manipulating dynamic representations.
The 2017 review recommended that PIAAC Cycle 2 harness the potential of technology to support a more effective and representative 21st century assessment, through greater use of different technology, media and associated representations to make the assessment more relevant to the 21st century. This is discussed further in the fourth section.
Quantitative and mathematical information in real-world situations and contexts is always represented and embedded in some format or other, whether that be in words and text, or diagrammatically or graphically, or dynamically. Mathematics, per se, does not exist in the real world by itself in its own isolated abstract form, such as 80% x €7.80 – such mathematics will be most likely embedded in an advertisement saying “20% discount” and the reader will need to read the information and decide that the solution is to take off 20% of the original price of €7.80. Hence the PIAAC framework needs to elaborate on the different ways that mathematics can be represented in the real world in a numeracy situation.
Mathematical information in a situation may be available or represented in many forms. It may appear as concrete objects to be counted (e.g., people, buildings, cars, etc.) or as pictures of such things. It may be conveyed through symbolic notation (e.g., numerals, letters, and operation or relationship signs). Sometimes, mathematical information will be conveyed by formulae, which are a model of relationships between entities or variables. Mathematical information may be encoded in visual displays such as a diagram or chart; graphs and tables may be used to display aggregate statistical or quantitative information (by displaying objects, counting data, etc.). Similarly, a map of a real entity (e.g., of a city or a project plan) may contain information that can be quantified or mathematised. Last but not least, textual elements may carry much mathematical information or affect the interpretation of mathematical (and statistical) information, as explained further below.
A person may have to extract mathematical information from various types of texts, either in prose or in documents with specific formats (such as in tax forms). Two different kinds of text may be encountered in numeracy tasks. The first involves mathematical information represented in textual form, i.e., with words or phrases that carry mathematical meaning. Examples are the use of number words (e.g., “five” instead of “5”), basic mathematical terms (e.g., fraction, multiplication, percent, average, proportion), or more complex phrases (e.g., “crime rate increased by half”) which require interpretation, or coping with double meanings (or with differences in mathematical and everyday meanings of the same terms). The second involves cases where mathematical information is expressed in regular notations or symbols (e.g., numbers, plus or minus signs, symbols for units of measure, etc.), but is surrounded by text that despite its non-mathematical nature also has to be interpreted in order to provide additional information and context. An example is a bank deposit form or interactive device (e.g., on a mobile device or an automated teller machine, ATM) with some text and instructions in which numbers describing monetary amounts are embedded, or a parking ticket specifying an amount of money that has to be paid by a certain date due to a parking violation, but also explaining penalties and further legal steps that will be enacted if the fine is not paid by a certain date.
With the 21st century digitisation of information and processes, the types of representation now explicitly include technology-based displays and visualisations on websites, in infographics, in online calculators, spreadsheets and other software and apps on mobile devices and more.
Four classifications for the representation of real-world numeracy situations are described:
Text or symbols
Images of physical objects
Structured information
Dynamic applications.
For an individual item in PIAAC numeracy, these four descriptions of different representations are not mutually exclusive and any item may involve one or more of these dimensions.
The stimulus is primarily based on running text that describes the problem situation and can include symbols and numerical information integrated into the text.
The stimulus is primarily based on photos or images of physical objects which depicts the problem situation. The image contains the crucial information to solve the problem (e.g. ruler or measuring instrument/scale, 3D objects). Sometimes some text is added to specify or narrow down the problem situation.
The stimulus is primarily based on data or information that is represented in tables, graphs/charts, maps, plans, calendars, schedules, timetables, infographics, etc. In most cases, these are computer-generated representations of data, which are becoming more ubiquitous in all news and social media, and in information from government services, financial institutions and utilities. Text will often be used to help specify and describe the information and the problem situation.
The stimulus is primarily based on interactive applications, animations, calculation applications (for instance planning and designing software, structured spreadsheets, drawing programmes, online applications and calculators such as loan calculators, currency converters, etc.), which are designed to support users to perform calculations or plan or design activities. This category could also contain: (simulations of) handheld devices and measurement instruments. Sometimes text is used to specify or narrow down the problem situation.
The operationalisation of the construct of numeracy in a large-scale assessment such as PIAAC is affected by many factors which shape the extent to which the theoretical construct can be fully addressed by the actual collection of items used in the direct assessment.
The 2017 review (Tout et al., 2017[1]) undertook a review of assessment developments, including in relation to numeracy and mathematics assessments. The review recommended that, because the numeracy description and construct in PIAAC is a multifaceted view and definition of numeracy, it requires a multimodal way of assessing the concept and construct, and because of the availability of new developments in technology and communication, new assessment developments could provide opportunities to enhance the assessment of numeracy in PIAAC Cycle 2.
The review also recognised and acknowledged up front that the existing PIAAC Cycle 1 items are based predominantly around static images and associated responses and are more like paper-based assessments transferred onto a computer, partly due to the transfer of many of the paper-based items from the previous ALL assessment to the computer-based assessment in PIAAC. As well, the platform used for PIAAC Cycle 1 was quite restrictive in terms of modalities and interactions that were available to house the stimulus and for the responses that could be automatically scored.
This section first begins with a general introduction about assessment developments in the 21st century, especially in relation to assessing adults’ mathematical skills, followed by a sub-section outlining a possible process and structure for enhancing the assessment of numeracy in PIAAC Cycle 2. This is followed by a discussion of the constraints that affect the development of the direct assessment of numeracy in PIAAC. Based on these discussions, an outline is then presented of the principles that guide the assessment of numeracy in PIAAC, including specifying the blueprint for the proportions of items against each of the core dimensions in the construct. This discussion and the consequent blueprint specifications of the test content are critical in ensuring that the direct assessment of numeracy in PIAAC Cycle 2 meets both construct and content validity requirements.
Finally, there is a brief discussion about, and further details on, a supporting scheme regarding factors that affect task complexity (or item difficulty) which is of importance both for task design as well as interpretation of results regarding numeracy in PIAAC.
There have been technologically-driven advancements in the educational measurement and assessment field in the 21st century, some of them based around the need to assess 21st century skills. There is much research about such developments [e.g., see (Bennett, 2015[163]; Geisinger, 2016[164]; Parshall et al., 2002[165]; Shute et al., 2016[166])]. Bennett (2015[163]), who has been researching and mapping educational assessment for a considerable period, describes three generations of assessment. He described first-generation technology-based testing as largely an infrastructure-building activity, laying the foundation for tests to be delivered in a new medium, where much of the testing closely resembled traditional tests. In his description of second-generation tests, he argued that qualitative change and efficiency improvement become the driving goals (Bennett, 1998[167]; 2010[168]), and where the tests use less traditional item formats, moved towards new multimodal formats and where there were attempts to measure new constructs. The driving force was often the technology. Bennett describes a third generation assessment as one of reinvention occurring on multiple fronts where these assessments were able to serve both institutional and individual-learning purposes. They are designed from both cognitive principles and theory-based domains, and where the assessments utilise “complex simulations and other interactive performance tasks that replicate important features of real environments, allow more natural interaction with computers, and assess new skills in more sophisticated ways” (Bennett, 2015, p. 372[163]). This includes the use of Augmented and Virtual Reality [e.g., see (Bower et al., 2014[169]; Sommerauer and Müller, 2014[170])].
While the review paper and the current expert group acknowledge that it is not yet possible in a large-scale international assessment such as PIAAC to implement and use the potential of Augmented and Virtual Reality, there are considerations to take on board for the future development and potential enhancements to the assessment of numeracy in PIAAC. There are many different computer-based models and options available to inform how numeracy might be more effectively assessed in future iterations of PIAAC, including in Cycle 2. The following sub-sections describe some of these possible enhancements, but also conclude with a discussion about the reality and constraints of an assessment such as PIAAC and especially the need to have all the materials and questions made available across a large number of different languages.
The literature specifically on computer-based assessment of mathematics (CBAM) and multimedia learning of mathematics [e.g., (Atkinson, 2005[171])] mostly focuses on the multimodal representation of mathematical concepts: calculating, graphs, diagrams, computer algebra systems, spreadsheets, statistical programmes, etc. However, in the computer-based assessment of numeracy another focal point is also of importance, namely the role of the representation of the problem (the situations and settings in which the mathematics is embedded). More general research on representations of real-life situations in education and assessment should also be considered [e.g., see (Schnotz, 2002[172]; Schnotz, 2005[173]; Schnotz and Bannert, 2003[174]; Schnotz and Kürschner, 2007[175]; Schnotz et al., 2010[176])], as well as research on more general multimedia learning (Mayer, 2005[177]; 2009[178]), while being aware of the cognitive load discussion (Sweller, 2005[179]; 2010[180]; van Gog, Paas and Sweller, 2010[181]).
In an analysis and review of the optional computer-based assessment of mathematics items developed for PISA 2012, the Australian Council for Educational Research (ACER) test developers created a list of features that benefited and advantaged CBAM test items over traditional paper-based assessments of mathematics:
Their appeal to students’ interactive learning styles increases the engagement of students with the task.
Items are less dependent on text and reading skills, which means students can access an item from visual cues, and then use the text to confirm the required response details.
Response modes are more flexible and less daunting. Students can easily edit a response, so they are more inclined to “have a go”.
Relevant calculations can be automated, which means answers are correct, and less time is taken. This allows items to address higher-order mathematical reasoning.
Items can assess spatial and visual skills using accurate simulations and manipulatives that are not readily available in pencil-and-paper formats.
Items can test ability to use a wider range of problem solving strategies, such as observation of patterns and trends, and of the effect of manipulations and actions.
Items can simulate computer-based processes, such as spreadsheets, drawing tools and graphing tools, and handling information in an online environment.
Systems can collect data about what the student did within an item, such as the time taken, number of clicks, processes followed and the final state (PISA Mathematics Expert Group, 2009[182]).
These are also applicable to an adult numeracy assessment such as PIAAC. A useful classification of the CBAM item types in PISA 2012 was also developed by the ACER test development team for the set of items (PISA Mathematics Expert Group, 2009[182]; Tout and Spithill, 2015[183]). These included, for example, items that were based around automatic calculation, where calculation could be automated “behind the scenes” to support assessment of deeper mathematical skills and understanding; animations, and/or manipulations; drawing, spatial, visual cues and/or responses; simulation of computer applications (e.g., using the data sorting capability of an ‘imitation’ spreadsheet); interactive graphing allowing automatic mathematical function graphing and statistical graphing; and simulation of web-based applications or contexts, with or without computer-based interactivity (e.g., buying goods online).
Another advantage of computer-based (or tablet-based) assessments is that the responses and item types available and that were used and can be automatically scored is quite extensive and can include options such as: selected-response formats (e.g., multiple choice, complex multiple choice such as true/false type questions); short numeric responses; click-on and hot-spots; drag-and-drop; pull-down menus; matching and ordering; and manipulation of images to a correct, final position and solution.
In the first cycle of the PIAAC assessment, there was a gap between the sophistication of the concept of numeracy used and the functionality of the assessment platform. As acknowledged above, the assessments that exist in the current item pool are relatively simple and one-dimensional. This analysis is corroborated in the literature [e.g., (Bennett, 2015[163])].
More sophisticated assessments utilising some of the possibilities outlined above are not necessarily aimed at more complex or higher order skills, but focus more on the multifaceted and multimodal nature of numeracy problem situations encountered in real life. To assess a sophisticated concept of numeracy there is the need for multimodal options to better represent reality, in which the respondents can show their competence (or not).
It is important to make a distinction and achieve a balance between the drive that stems strongly from development of technologies, that can be used in assessments (technology-driven) and the drive to design an assessment that is closely related to the concepts that are designed around the construct of numeracy (concept-driven).
Specifically, there is the need to frame any assessment development enhancements to PIAAC numeracy around two underpinning aspects of the PIAAC numeracy construct:
That PIAAC is based on a multifaceted concept of numeracy and has an associated multimodal assessment.
That PIAAC is an assessment of how well individuals can use their mathematical knowledge and skills to solve problems stemming from pragmatic and authentic (i.e., real-world) situations, needs or demands.
The PIAAC definition and description of numeracy falls into the category that Maguire and O’Donoghue (2002[184]) called the “integrative phase.” They classified the development of definitions and ideas about numeracy into three phases: formative, mathematical, and integrative. In this “integrative” classification, as with PIAAC, numeracy is viewed as a complex, multifaceted and sophisticated construct incorporating each individual’s mathematics, communication, cultural, social, emotional, and personal aspects in a real-world situation. Numeracy, as with mathematical literacy in PISA, is seen as a sophisticated capability requiring more than just arithmetic calculations and basic mathematics. These more integrative approaches to numeracy have become influential over the last few decades, as illustrated by projects that define numeracy instructional content standards and assessment frameworks such as in PISA, ALL, and PIAAC and national adult curriculum frameworks/standards [e.g., see (Department for Education (DfE), 2014[185]; McLean et al., 2012[77]; Quality and Qualifications Ireland (QQI), 2016[78]; Tertiary Education Commission, 2008[79])]. The assessment of such a multifaceted phenomenon therefore requires a multifaceted and multimodal set of assessment items that are authentic, as described above in relation to assessment developments and possibilities.
Hoogland and Tout (2018[186]) in looking at the pressures and challenges on Computer-based assessment of mathematics (CBAM) into the twenty-first century, argued that technology has the potential to support the assessment of higher-order thinking skills in mathematics, and also to represent authentic problems from the world around us to use and apply mathematical knowledge and skills. However, they also argued that the challenge is to not allow the technological capabilities, supported by psychometric analysis, to focus too much on assessment of lower order goals, such as the reproduction of procedural, calculation-based, knowledge and skills. These aims are consistent with the aims of the assessment of numeracy in PIAAC. Assessing a multifaceted concept based only on simple assessment tasks has two negative implications:
The capabilities of individuals to cope with complex and multifaceted mathematics problems from real life are not assessed in full when the items are too straightforward and one-dimensional.
The rollback effect of an international assessment of adult competencies on adult numeracy education practices is not to be underestimated. There is a responsibility for assessment developers such as those in PISA and PIAAC that their framework and assessment items are in sync with and reflect the complex, multifaceted constructs and concepts being assessed.
As described in the earlier sections of the framework, it is critical in the assessments of adults’ numeracy to make the situations, representations and the responses authentic and make them as similar as possible to the way adults encounter mathematics in different life contexts, and not use questions such as the typical school-based word problems described earlier. Arguably the problem of authenticity and cultural appropriateness is lessened when testing pupils in schools, such as in PISA, because test designers can use conventional mathematical terminology, formulae, symbols, and so forth; this helps school-age assessments to standardise the demands from respondents by conveying the mathematical information embodied in different situations in consistent ways regardless of the cultural context. However, testing of adults’ numeracy presents more challenges because many will not remember formal school-based notations or terminology. In countries where a sizeable proportion of the population are immigrants or speak multiple home-based languages, the gaps between mother tongues and school-based mathematical linguistic conventions may further affect performance on some numeracy tasks. Thus, attention has to be given to linguistic and cultural factors when adapting items for adult assessments.
As developments in the 21st century impact on how mathematical and numerical information is represented, the PIAAC Cycle 1 facet, Representations of mathematical information, in the PIAAC framework has been significantly updated to reflect these changes. The revised framework and definition for numeracy in PIAAC Cycle 2 and the resulting new items harness the potential of technology to support a more effective and representative 21st century assessment, for example, through greater use of visual and interactive media, such as the use of infographics, interactive websites and online calculators, spreadsheet processes, graphing and measurement tools, etc. in assessment items. However, a balance has been kept between numeracy and mathematics tasks and actions embedded in 21st century digital and technological environments with those embedded in modes that are more traditional. This balance can partly be maintained by the necessary use of the existing PIAAC Cycle 1 numeracy item pool as linking items as these were based mainly around static images and the items are more like paper-based assessments transferred onto a computer.
In order to facilitate the development of newer assessment content and delivery mechanisms, the review team developed a dimension of assessment possibilities – see Figure 3 in Tout et al. (2017, p. 31[1]), that could form a starting point for monitoring and balancing the range of PIAAC numeracy item formats and types. Hoogland and Tout (2018[186]) further developed this dimension of assessment possibilities, and argued that it could be used to reflect on, discuss and research the relevance, usefulness and effectiveness of mathematical assessment tasks, especially in relation to twenty-first century skills. Having a spread of stimuli and items developed and selected from across this spectrum would enable PIAAC numeracy to be better representative of the framework and construct, add to the issue of authenticity, and hence better assess adults’ capabilities and competencies around numeracy practices in the 21st century.
The existing PIAAC Cycle 1 numeracy items which will continue to be used as link items tend to be at the left hand, more traditional, end of the dimension of assessment possibilities as described. Therefore, the focus on new item development has been to complement the existing pool by taking on board the potential enhancements and innovations due to 21st century capabilities as described in the dimension of assessment possibilities spectrum. This has provided the ability to create a balance across the spectrum of assessment possibilities.
Based on the above, the Numeracy Expert Group set the item writing teams the task of requesting that the new item development for Cycle 2 of the PIAAC numeracy assessment build on some of the assessment developments described above and to introduce new assessment content, representations and item formats that better reflect 21st century related digital representations, stimuli and numeracy tasks and assessment responses.
Fortunately, the PIAAC Cycle 2 platform and delivery system has been able to support many such types of stimuli, items and response types, and the items are in the process of being created and implemented for use on a tablet ready for the Field Trial. For PIAAC Cycle 2 these item types, responses, representations and interactive stimuli have included:
illustrations and photos of authentic contexts/items/objects
interactive calculators and online tools
tap on area or multiple areas of the screen/image
drag-and-drop responses
drawing a graph
use of a keypad to enter responses
single and multiple selection multiple choice questions
access to an online calculator
interactive online charts and data
online map
online ruler
simple spreadsheets.
Despite the advances incorporated into the Cycle 2 assessment, there are constraints and challenges to what can be achieved in such an international assessment like PIAAC, and these need to be acknowledged. These include constraints related to the capabilities of adults aged 16-64 who will be undertaking the assessment, the practicalities and costs of developing an assessment across such a wide range of cultures and languages, and the limitations of the available computer platform.
First, the results from the Problem Solving in Technology-Rich Environments (PS-TRE) assessment in PIAAC Cycle 1 need to be considered in relation to the review and implementation of the PIAAC numeracy construct. The first cycle of the PIAAC survey provided two different pieces of information regarding the capacity of adults to manage information in technology-rich environments: the proportion of adults who had sufficient familiarity with computers to use them for PIAAC tasks, and the ability of adults with at least some basic ICT skills to solve the PS-TRE tasks.
The PIAAC Cycle 1 PS-TRE assessment results showed that there were adults with no, or extremely limited, ICT skills in all of the participating countries. The assessment found that:
From around 7% to 27% of the adult population reported having no experience in the use of computers or lacked the most elementary computer skills, such as the ability to use a mouse. In addition, there are also adults who appear to lack confidence in their ability to use computers, primarily because they use them infrequently. Of the adults undertaking the assessment, most were proficient at Level 1, which involves the use of familiar applications to solve problems that involved few steps and explicit criteria, such as sorting e-mails into pre-existing folders. (OECD, 2013, p. 98[161])
These results from the Problem Solving in Technology-Rich Environments component of PIAAC Cycle 1 with its warnings regarding the high proportion of adults with no or extremely limited ICT skills needed to be taken into account in deciding on the balance of items incorporating the new technological and digital aspects of the PIAAC numeracy framework and its associated assessment.
Second, on a pragmatic level, some of the innovations and developments arising from technologically-driven advancements in educational measurement and assessment in the 21st century needed to be carefully reviewed and considered as to the feasibility of their use and practical implementation for PIAAC Cycle 2. Some issues that needed to be considered include:
The costs—some technologies, media and tools would potentially be expensive to use and implement—both from a content development point of view (the production of videos/animations/etc.) and from a delivery and implementation perspective (conducting such assessments in people’s homes).
The time available for testing—would the use of such innovations in assessment take substantially more time for the delivery of the PIAAC assessment?
The feasibility of producing and using any animations, simulations, video or audio support in potentially 30 different languages, which would be challenging, costly and require substantial quality assurance processes of translations.
The performance of the use of any such innovations, especially the use of simulations (e.g., the use of games is highly unlikely to be relevant in such an adult assessment at this point in time), and hence the cost of the mandatory trialling and psychometric checking for performance, reliability and validity.
The aim has been to have a pragmatic and balanced set of items that meet as many of the demands as possible for the enhancement of the numeracy assessment in PIAAC Cycle 2 while taking into account the constraints outlined above, including the ICT-related capabilities of adults across the PIAAC age-range. However, it needs to be remembered that there will always be a substantial set of link items from the previous assessments and that the Field Trial can be utilised to check how any new items work compared with existing items.
It should also be noted that the change in platform for the delivery of Cycle 2—from a laptop computer requiring the use a mouse, to a tablet where respondents can use a stylus or finger—and the fact that 10 years have elapsed since Cycle 1 suggest that more participants will be able to use the platform. The Field Trial will allow the opportunity to test whether this is the case or not.
One also needs to distinguish between the conceptual framework (second section) and the assessment construct (third and fourth sections). Not all real-life numeracy tasks can necessarily be simulated well in a specific assessment. Further, the ability of an assessment to actually capture, evaluate, and score responses associated with the full spectrum of numeracy as defined in PIAAC ultimately depends on the technical aspects of that assessment. While the computer-based assessment platform chosen for PIAAC Cycle 2 offers many advantages over Cycle 1, there are still restrictions and limitations, and this has constrained the ability to develop many sophisticated interactive items or use audio or video that required translation, for example.
Firstly, the overall testing time per respondent does not allow inclusion of extended problems or lengthy simulations of complex authentic numeracy tasks, although it is recognised that ability to solve complex or extended numeracy problems is an inherent part of the numeracy competency. In order to cover all facets of the numeracy construct in the limited time available, the use of a larger number of short tasks is prescribed.
Secondly, the need to score all responses automatically limits the type of assessment tasks that can be used. The traditional divide in item format is between selected-response (sometimes called forced-choice or multiple choice) format versus a constructed-response format. Selected-response items require the choice of one or more responses from a number of response options. Responses to such questions can usually be automatically processed and scored when presented on a computer/tablet, and can include a range of interactive responses, such as tap-on, drag and drop, etc. Within constructed response items there are closed constructed-response or open constructed-response items. Closed constructed-response items provide a more structured setting for answering, and they aim to produce a response that can be automatically scored, against a scoring rubric, to be either correct or incorrect. Open constructed-response items require respondents to communicate in their own words the answers to tasks or questions, and such items require trained experts to manually code responses.
While the platform allows respondents to provide an answer in several different modes (e.g., numeric entry, tapping on an area of the screen, choosing from different options, etc.), in its present stage of development it cannot accept most types of open constructed-response or free-form items because of the huge possible diversity in how respondents may enter their answers. The limitations stem from the difficulty to automatically code (i.e., designate an answer as correct or incorrect) free-form responses in dozens of languages while accommodating various grammatical and syntactical structures, as well as overcoming typing mistakes which are naturally expected when people type text into a computer. Examples are when respondents:
write number ranges or estimates which have multiple mathematically equivalent representations, such as "a quarter", "0.25", "1/4", "1 in 4", or “around five to six”, "1.00 to 6.00"
describe their interpretation of given information such as in a simulated media statement
write justifications for their answers, or list arguments supporting their conclusions.
Specifically, tasks requiring communication-based responses, such as when adults have to explain interpretations of given information, or describe their evaluation or analysis of a situation or their thinking about that situation, are difficult to implement in the direct assessment of the skills targeted by PIAAC. Such tasks do comprise an important, inseparable part of the landscape of adult numeracy situations and are an inherent part of the conceptual framework of adult numeracy, yet few could be included in the item pool for the second cycle of PIAAC.
Consumer and shopping issues are important components of numeracy and are represented in the numeracy tasks. Since the currencies of the participating countries vary greatly, consumer-related items present a challenge to item developers and translators. It is crucial to try to keep any questions relating to money at the same time realistic and mathematically comparable. Hence, PIAAC specifies strict guidelines about how countries can change the magnitude of any monetary amounts in order to maintain comparability. These are published as part of the Translation and Adaptation Guidelines, and will be based on current exchange rates for currencies.
PISA copes with this by having all its monetary-related items set in a fictional country, Zedland, with a fictional currency of zeds and zedcents. This approach has been deemed as not suitable to be applied to PIAAC due to its need to apply to adults across a wide age and educational range, where it is believed that some adults may not relate to a very unfamiliar currency nor to fictional prices and costs.
PIAAC’s tasks are therefore designed to allow countries whose currencies are somewhat similar in value to the dollar to keep exactly the same number and change only the currency sign.
As a general practice, when we refer to a monetary value that can be written in different currencies, we note it as _45, for example. It signifies that 45 can be considered as a number of dollars, Euro, krona, guilders, pesos, or whatever the local currency is.
The following options for making changes to monetary values are listed in order, from the least to the greatest impact on the equivalence of the cognitive demand of the item.
Option 1: Change the currency sign only. Keep the numbers the same and change only the currency sign to the local currency sign. (e.g. change the US Dollar sign, $, to Euro, € or GBP, £) This will be the option of choice for the European Community countries, since the Euro and the Dollar are close in value.
Option 2: Change the numbers by multiplying or dividing them by powers of 10. When changing the currency sign does not seem to work and the object’s value seems unrealistic, the translator will have to change the numbers or amounts in the item.
The rule: If numerical values must be changed to retain the realism of an item, they can only be multiplied or divided by powers of 10 (i.e., by 10, 100, 1000, etc.). This restriction aims to keep the cognitive demands of the item (such as the nature of the mathematical steps and mental operations) similar in all countries. Consider, for example, the Raincoat item (PIAAC unit 603), priced at _80. If this price is unreasonable for any type of raincoat, the translator can choose to multiply the number 80 by 100 to be 8000, if this is a reasonable price for a raincoat in the currency of the country where this change is seen to be unavoidable. [In Hungary, for example, 1 US dollar is now equivalent to 250 HUF (Hungarian Forint). The raincoat could be priced at 8000 HUF. Yet, a change to 20000 HUF (80 x 250) is not permitted, even though it is the "true" value of the raincoat in HUF, since it significantly changes the mental operations required by the task.]
Another challenge with an international assessment such as PIAAC relates to the fact that countries can also have different measurement systems, although most countries are now solely metric, with the exception of the US, which still (mainly) use the imperial measurement system.
The following is the approach taken to creating mathematically equivalent items for metric and imperial-based measurement units. There were different solutions that could be applied, depending on how dependent the question was on understanding the measure system embedded in the context/task.
The first issue was to decide whether a conceptual understanding of the measurement system is critical to answering the question. If not, then it is possible to leave the measurement units in their original, authentic units that fitted the context. However, if a question requires "intimate" knowledge of the metric system to answer it then parallel metric and imperial units are required. For example, when an estimation of a length or height is required, as in the existing PIAAC Tree and Path questions where the respondent is expected to have sense of size in familiar contexts–the height of a tree relative to a person, and the length of a path–all shown in photographs). In these situations you could not consider using a metric unit in the US, nor an imperial unit in a metric country. People need to have a feeling for the measurement units.
Similarly, a US version of a question that included doing a conversion between metric units as part of the solution, would not work—knowledge of conversion factors could be very unfamiliar to an older generation American. The other point here is that the conversions between the various measurements in the US are not consistent and all based around powers of ten like in metrics, and many people routinely need to look them up – like how many ounces in a pound, inches in a foot, yards in a mile, fluid ounces in a pint, pints in a gallon and more.
A starting point was to research and see what is common and authentic in the US—the US does use metric units now for a number of common commodities, such as for popular soft drinks, medication doses, etc. In some cases it was therefore possible to select particular situations/objects that are measured in metric units and hence were suitable and compatible for use in questions in the US.
If a unit needed to be created in both metric and imperial, the best solution was to maintain the same dimensions and just change the units – from meters to yards, for example, or kilometers to miles. This was more easily able to occur where the units have some similarity—like cm to inches, or meters to yards. This approach has been used in link items such as the Path and the Tree.
However in some cases there was no alternative but to change the dimensions/measurements to make them both authentic. For example, this was done in the existing link items, such as the BMI formula item in ALL survey and then PIAAC. The necessity is to then keep the degree of difficulty of any arithmetical calculations as similar as possible, so the difficulty level is maintained. For example, in a possible item based on photos—in metric it was possible to have the dimensions of 8 x 12 cm, and in inches have it as 4 x 6 inches—they are authentic, similar small photo sizes and the dimensions are in the same ratio.
The PIAAC assessment design involves using a household survey methodology which assumes that overall testing time per respondent is around 60-80 minutes. In that time, study participants will be asked to complete:
A background questionnaire, which collects information on possible outcomes and antecedents of key skills, as well as on demographic and structural indicators that are needed to describe the distribution of such skills within participating countries.
A tablet tutorial and orientation to the assessment.
A short locator test, which will be used to direct respondents to the appropriate section of the direct assessment and will also provide information about the literacy and numeracy skills of adults who may not be able to continue on to the direct assessment.
A brief measure of component skills in reading and numeracy.
The direct assessment, where each respondent will take two of three domains of literacy, numeracy and adaptive problem solving.
The direct assessment will utilise an adaptive design that will optimise the match between respondent ability and the difficulty of administered items. Such a design provides more reliable information about respondents’ skills within the available testing time.
The items for the assessment are expected to enable reporting of respondents' performance in a manner similar to the one used in ALL and PIAAC Cycle 1, which scaled raw scores in the range 0-500, but focused on reporting performance on six ability levels with the following (tentative) boundaries:
Below Level 1: below 176 (lowest level)
Level 1: raw score 176 – 225
Level 2: raw score 226 – 275
Level 3: raw score 276 – 325
Level 4: raw score 326 – 375
Level 5: raw score 376 – 500 (highest level).
The assessment of numeracy, whether by paper-and-pencil tasks or computer-based, has to take into account that the practice of numeracy in everyday or work situations also involves the use of certain objects and artefacts. First is the use of calculators, either handheld or now also available on smartphones and tablets, which are now widely available to adults from all walks of life in many countries. Thus, calculators are tools which are part of the fabric of numeracy life in many cultures. Increasingly, respondents in large-scale tests are allowed, sometimes even expected, to use calculators. It follows that adults in PIAAC should be given access to a calculator as part of an assessment of numeracy skills, and they can then choose if and how to use it. An online, basic calculator will be available on the tablet-based delivery of PIAAC, and as well a handheld basic calculator will be made available and can be used if requested. There are no numeracy questions that require the use of a more sophisticated type of calculator than a basic, four function calculator.
The use of an online ruler is also presented in one unit, in both a metric and imperial (inches) system, as rulers/measuring instruments are part of contexts where adult numeracy competence is manifested. The use of other technologies, such as a computer spreadsheet, also fit the assessment of numeracy, and in PIAAC Cycle 2 some items assess this skill.
It should also be noted that it is intended that the interviewer will provide access to a pen and paper in order for respondents to make notes, write down and undertake calculations, etc.
The development of the numeracy assessment for PIAAC has been based on a number of general principles or guidelines listed below. These principles reflect the cumulative literature on large-scale assessment of mathematical skills and adult numeracy (Gal et al., 2005[28]; Gillespie, 2004[187]; Murat, 2005[188]), and various background documents and positions prepared as part of the planning of PIAAC [e.g., (Gal, 2006[153]; Gal and Tout, 2014[3]; Jones, 2006[154]; Murray, 2006[155]; PIAAC Numeracy Expert Group, 2009[2]; Tout, 2006[156]; Tout et al., 2017[1])]. This also incorporates the general ideas discussed earlier, as well as any known technical limitations in the delivery of PIAAC Cycle 2.
Some general approaches include:
Items should cover as many aspects as possible within each of the four core dimensions of the numeracy definition and elaboration. Items should require the activation of a broad range of skills and knowledge included in the construct of numeracy, as portrayed in the conceptual framework depicted in Box 3.2. The specifications and targets for the item development and spread against each of the core dimensions are spelt out in the next sub-section.
Items should aspire to maximal authenticity and cultural appropriateness. Tasks should be derived from real-life stimuli and pertain to the full range of contexts or situations (i.e., everyday life, work, societal) that can be expected to be of importance or relevant in the countries participating in PIAAC. Item content and questions should appear purposeful to respondents across cultures, although it must be acknowledged that in a large-scale assessment such as PIAAC, not all items and contexts can be personally familiar to all adults within any one country, let alone across all countries.
Items should have different response formats, to the extent feasible by the computer platform used for administering the direct assessments in PIAAC. Items should be structured to include a stimulus (e.g., a picture, drawing, visual display) and one or more questions, the answers to which the respondent communicates via the modes available within the platform. Numeric entry is limited to the set of 10 digits and common separators (, and .) or other mathematical symbols where relevant.
Items should spread over different levels of ability. Items should span the range of ability levels anticipated within PIAAC participants, from low-skilled individuals (who are of particular interest in countries where policies and educational programmes may be earmarked for low-skill populations), all the way to those with advanced competencies. The ALL and PIAAC Complexity Schema (Tout et al., 2020[141]) was used to provide an initial estimate of the spread of item difficulties in order to assist in the selection of the items for the Field Trial.
Items should vary in the degree to which the task is embedded in text. Some items should be embedded in or include relatively rich texts, while others should use little or no text. This distribution aims to reflect the different levels of text involvement in real-world numeracy tasks, as well as reduce overlap with the literacy scale.
Items should be efficient. To allow for coverage of many key facets of the numeracy competency, the inclusion of a large number of diverse stimuli and questions will be needed. However, in light of testing time constraints, the use of short tasks is necessitated, precluding items that can simulate extended problem-solving processes or that require a lengthy open-ended response.
Items should be adaptable to unit systems across participating countries. Items should be designed so that their underlying mathematical demands are as consistent as possible across countries, regarding language and mathematical conventions. For example, items should be designed so that different currency systems or different systems of measurement (metric or imperial) could be applied to the numbers or figures used. Items should retain equivalency with respect to their mathematical or cognitive demands after being translated.
Based on the definition and elaboration of numeracy described in previous sections and on the above discussions on assessment enhancements and constraints attached to delivering PIAAC Cycle 2, this sub-section specifies the blueprint for the proportions of items against each of the core dimensions in the construct. For a comparison, the specifications for each dimension is compared with the previous targets for Cycle 1 and for PISA 2012 and 2021 too.
This blueprint specifies the test content for the direct assessment of the cognitive aspects of numeracy as defined for PIAAC Cycle 2, taking on board the above constraints and limitations, and the enhanced opportunities provided for Cycle 2 compared with Cycle 1 of PIAAC. These help establish the construct and content validity requirements for the cognitive assessment aspects of numeracy, with this being confirmed and refined through the quality assurance (QA) processes and psychometric item analysis and review following the Field Trial. The QA processes include feedback from participating countries and a formal translation and review process with language experts. These QA processes pick up issues to do with the language structure and meaning of items, and also content and cultural issues. A Field Trial is undertaken with a sample of the target population in each participating country before the final assessment is implemented. Field trial data is collected and analysed psychometrically and from these detailed analyses, ‘misbehaving’ items are rejected on a number of levels including for reliability, fairness and validity. Then for the remaining successful items, any fine-tuning is undertaken, and a representative set of items are chosen based on the blueprint and placed into final forms.
This blueprint is specified against the four dimensions incorporated into the definition of numeracy and the elaboration of numerate capacity, as outlined earlier:
Numeracy is accessing, using and reasoning critically with mathematical content, information and ideas represented in multiple ways in order to engage in and manage the mathematical demands of a range of situations in adult life.
The four core dimensions named and described in numeracy for PIAAC Cycle 2, namely:
cognitive processes
content
representations
context.
For the three new Cognitive processes the spread across the dimension is very similar to PISA 2012 processes which have a similar structure, but quite different from PIAAC Cycle 1 due to the revised and different structure of this response classification. There is also an attempt to be less focused on the traditional process of doing the mathematics (Act on and use mathematics), and have a good representation in the two other processes too, which are seen by the Numeracy Expert Group as significant aspects of how adults engage with and solve a numeracy problem where mathematics is embedded in an authentic situation. This target might be difficult to achieve, but these were the aspirational targets set by the NEG.
For the four content areas the spread across the dimension was similar to Cycle 1 and similar to PISA. One difference is that PIAAC does not aim to have as many items in the more formal mathematics area of Change and relationships, which includes algebraic thinking, which is more of an interest to 15-year-olds in a school-based assessment such as PISA. For PIAAC Cycle 2 there has been a slightly higher focus placed on Data and chance. This is seen as a more common and important area that adults now have to negotiate with in the 21st century, and it is higher use and reliance on presenting numerical, quantitative and other data and related analyses in a range of ways and often in ways that are critical to people’s lives.
For the four new Representation classifications, again the NEG was aspirational in its targets and aimed to set relatively high goals for 21st century type representations, which are covered under both Structured information types of materials (infographics etc.) and also Dynamic applications which includes online interactive websites and applications alongside more standard software applications and tools. This will in the end be balanced by the existing link items from PIAAC Cycle 1 and ALL, where the representation may be more traditional and less 21st century in their style and format. It should be noted that in the previous cycle, the type of representations was not explicitly monitored in terms of its proportion across the pool of items, and it is felt that incorporating representation into the definition and the dimensions will enhance the quality of items across PIAAC numeracy.
For the three remaining contexts in PIAAC Cycle 2, the aim is to have an equal spread as per previous cycles, as also occurs in PISA.
In planning an assessment, it is of course important to be able to understand what it measures. Assessment designers assume that when engaged with the assessment items (including tasks, questions, stimuli, etc.), respondents activate cognitive processes and rely on stored knowledge and learned skills which are part of the construct being measured. Thus, differential performance levels can be accounted for by the underlying cognitive knowledge bases and other enabling processes. It follows that it is useful to have a theoretical model or set of assumptions regarding what factors cause certain tasks to be harder or more complex than others, so that the assessment results can be correctly interpreted. A model or scheme of factors affecting task complexity can also help when linking the assessment results to possible social (or educational) interventions, i.e., point to the skills that are lacking and have to be further developed in the population (Brooks, Heath and Pollard, 2005[189]).
Prior seminal work by Kirsch and Mosenthal [e.g., (Kirsch and Mosenthal, 1990[190]; Kirsch, Jungblut and Mosenthal, 1998[191])] and earlier projects have pointed to several key factors which account for task difficulty when considering arithmetic items or items involving text comprehension. These include readability, type of match, plausibility of distractors, operation specificity ('transparency'), and type of calculation and number of steps. The Kirsch and Mosenthal work has informed the design of assessment tasks for IALS and other surveys, and the interpretation of their results. In designing the ALL numeracy scale, the ALL Numeracy team attempted to advance the Kirsch and Mosenthal complexity scheme and develop tentative assumptions regarding factors which affect difficulty of multiple types of new tasks introduced to measure the numeracy construct which were beyond those encompassed by the more focused construct of quantitative literacy in IALS. Examples are items involving percents, knowledge of measurement and spatial reasoning, statistical concepts, and so forth.
The developers of the mathematical literacy scale for PISA (2006) also recognised multiple factors affecting item difficulty, such as the kind and degree of interpretation and reflection required by the problem, the kind of representation skills required, or the kind and level of mathematical skill required, e.g., single-step vs. multi-step problems, or more advanced mathematical knowledge, complex decision-making, and problem solving and modelling skills, or the kind and degree of mathematical argumentation required. Further factors that are assumed to affect difficulty both in PISA, ALL and other surveys relate to the degree of familiarity with the context, and the extent to which tasks require reproduction of known procedures and steps or present novel situations requiring non-routine and perhaps more creative responses. It should be noted that the PISA description of complexity factors seems quite compatible with that of ALL, although some of the terminology is different, and published PISA reports do not explain in detail how it was used to guide the design of specific items.
The complexity scheme for numeracy used in ALL (Gal et al., 2005[28]) has been instrumental for the item development and scale construction stages of that study, especially in that it helped to evaluate in advance if items will span different difficulty levels. Given that PIAAC's numeracy assessment is founded on the principles developed for ALL and that the PIAAC numeracy assessment scale uses linking items used in ALL, the ALL complexity scheme has been adopted and updated as an analytic tool for item development and interpretation for PIAAC as well. The details about this updated PIAAC numeracy complexity schema are provided in Tout et al. (2020[141]).
This section discusses the relationship between the PIAAC numeracy framework and the PISA mathematical literacy framework and assessment. Many of these aspects have been discussed earlier when considering the framework construct and its parameters, and the assessment blueprint. This section summarises the similarities and differences.
Note: in this section the references used are as those written and documented in the original, full PIAAC Cycle 1 numeracy framework (PIAAC Numeracy Expert Group, 2009[2]). Similarly, the references to PISA are mainly to the PISA 2012 mathematical literacy framework and its descriptions (OECD, 2013[122]). This is because in 2012, mathematical literacy was the major domain for PISA, when the relevant framework was revised and updated. The same mathematical literacy framework and assessment construct was used for the next two cycles of PISA in 2015 and 2018. For PISA 2021, mathematical literacy is again the major domain for PISA, and hence the framework and assessment construct is being updated and revised. This was happening in parallel with the development of the numeracy framework for PIAAC Cycle 2. The PIAAC Numeracy Expert Group was able to access a copy of the second draft of the PISA 2021 mathematics framework (OECD, 2018[123]) in November 2018. It was prepared by the Expert group for mathematics under the guidance of RTI International as the international contractor who led this work for the OECD. Most of the comparisons between PIAAC numeracy and PISA mathematical literacy have therefore been based on a comparison of the 2012 PISA framework and descriptions, but where possible the PIAAC NEG has also included comments and comparisons with the updated 2021 PISA mathematical literacy framework.
The following sub-sections look at the commonalities and links between PISA and PIAAC across the features and parameters of PIAAC.
Both numeracy in PIAAC and mathematical literacy in PISA use a non-school-curriculum focused approach to naming and describing the content areas covered in their assessments. The purpose behind both frameworks is describing mathematics for use and application outside of the classroom, and so the organisational structure for mathematical content knowledge is based on how mathematical phenomena are encountered in situations in the outside world. While the PISA and PIAAC frameworks were developed by independent teams, they use very similar descriptors for their content classifications, introducing and describing these in terms of the Big Ideas behind mathematics.
The two frameworks are highly consistent in terms of their descriptions and structures of the mathematical content covered in their assessments. There are very similar spreads across each content area. As discussed earlier, PIAAC has less interest in the more formal mathematics area of Change and relationships, which is more of an interest to 15-year-olds in a school-based assessment such as PISA.
The sets of context descriptors used in both frameworks regarding the first three contexts (Personal; Work/Occupational; Societal or Societal/community) are highly consistent with each other, with a similar spread of items.
The NEG reviewed the fourth context, Further learning in PIAAC Cycle 1 and after comparing it with Scientific in PISA, decided that it was best to remove the classification named Further learning which in PISA incorporated items that are considered intra-mathematical. The need to have such items in PIAAC that were about knowing about the more formal aspects of mathematics, including the conventions used to apply mathematical rules and principles, are to be covered through the inclusion of those requirements through the content knowledge area of Change and relationships.
The difference here with the PISA context classifications reflects the different interests in the more formal mathematical understandings of 15-year-olds within a school setting with those of adults out-of-school. Any other differences in the item coverage here are due to the age of the two target groups, with some of the PIAAC situations described being more relevant to adults, and some of the PISA situations being more appropriate for 15-year-olds.
This facet of the original PIAAC Cycle 1 numeracy structure was an aspect of the numeracy framework that was recommended as needing major review by the 2017 review paper, which recommended that PIAAC could potentially learn from the processes described in the PISA 2012 mathematical literacy framework. Hence for the three new Cognitive processes developed for PIAAC Cycle 2 there is a significant amount of similarity and consistency with the PISA processes of Formulate, Employ and Interpret/Evaluate. This is because the NEG took on board the intent and structure for the three PISA 2012 process cycles in their development of the new cognitive process dimension for PIAAC Cycle 2. In relation to the new mathematical reasoning process included in PISA 2021, it is the view of the NEG, that for the assessment of numeracy skills of adults the mainly intra-mathematical aspect of mathematical reasoning needs to be embedded within the real-world problem solving aspect for PIAAC, and not assessed as a separate part of the construct. Therefore mathematical reasoning and its understanding and application is integrated across the relevant aspects of the three other cognitive processes.
In their review and comparison of the two numeracy frameworks for PISA and PIAAC, Gal and Tout (2014[3]) concluded in relation to the issue of item formats that:
PISA 2012, with its more comprehensive range of item types and more interactive computer-based assessment, will enable richer and extended descriptions of sub-components of mathematical literacy compared to the information that can be generated by the numeracy assessment in PIAAC. (Gal and Tout, 2014, p. 52[3])
Furthermore, the review paper commented that much of the existing PIAAC Cycle 1 and ALL item pool was based around static images and was more like paper-based assessments transferred onto a computer, and that this also does not now seem to reflect the way numeracy tasks and actions are now situated and practised in the 21st century.
As stated earlier, the next delivery of the PIAAC Cycle 2 numeracy assessment is much more capable than Cycle 1 was in terms of allowing the use of new and more interactive, 21st century style item formats. Such items have been developed and will be trialled in the Field Trial.
Based on detailed comparisons of the two numeracy frameworks for PISA and PIAAC by the review team, by Gal and Tout (2014[3]), and from two of the 2017 review team who were familiar with both the full sets of PISA 2012 and PIAAC items and not just the publicly released items, it is apparent that both assessments describe and cover very similar territories.
On the conceptual level, numeracy and mathematical literacy are closely related constructs in terms of their core, underlying ideas. In relation to the definitions and descriptions of the constructs and what they are assessing, Gal and Tout, in their comparison of the two programmes, summarised the similarities:
Both constructs refer to the ability of individuals to cope with tasks that are likely to appear in the real world, and that contain mathematical or quantitative information, or that require mathematical or statistical skills and knowhow.
Both constructs focus on how well individuals can use their mathematical knowledge and skill to solve problems stemming from pragmatic (i.e., real-world) needs or demands, and to ‘engage’, manage, and understand various tasks in the world around them—rather than addressing decontextualised mathematical tasks.
Both PISA and PIAAC describe mathematical literacy or numeracy as not synonymous with a minimal or low level of mathematical knowledge and skills. That is, both assessments view the constructs as describing competencies lying on a continuum, i.e., individuals could be placed on a scale from low levels to high levels. (Gal and Tout, 2014, pp. 47-48[3])
They concluded that both the PIAAC and PISA frameworks, definitions and assessments have substantial conceptual similarities and also practical commonalities in their test items and design principles, as well as in the range of content areas and skills they cover (Gal and Tout, 2014[3]). However, there are some differences between the two assessments, related to the diversity in the experiential backgrounds and the distances from schooling for adults compared to children. As Gal and Tout wrote:
Because many adults may not remember more formal school-based representations or technical language, the design of PIAAC items has taken into account from the outset the need to establish authenticity while reducing the use of formal notations and ‘school-like’ appearance. (Gal and Tout, 2014, p. 39[3])
An examination of the item sets of both PISA and PIAAC shows that PISA is more interested in the ability of 15-year-olds to use and apply curriculum-based mathematical skills and knowledge embedded in a real-world situation. On the other hand, PIAAC is somewhat less focused on how respondents use formal mathematical skills when solving a real-life-type mathematical problem. For example, in some of the PISA mathematical literacy items 15-year-olds are asked to use information from a real-life situation to calculate and identify specific formal characteristics of linear equation graphs, such as the gradient and the y-intercept. This type of more formal mathematical knowledge is not assessed in PIAAC, as generally PIAAC respondents are not required to show evidence of their knowledge of the use and understanding of formal school-based mathematical notations, which are often forgotten from not having been in current or recent use.
One of the important features of both the PISA and PIAAC frameworks is the way that each has independently developed a schema that describes aspects of test items that drive item difficulty, and which indicate the mathematical proficiency of tested individuals and populations.
PIAAC does this in considerable detail in the Appendix to the framework (PIAAC Numeracy Expert Group, 2009, pp. 44-56[2]). As well as classifying test items according to the mathematical content knowledge required to complete each item, the Annex presents a detailed scheme designed to show the complexity of test items. Five ‘complexity factors’ are defined, and a scheme is presented for rating mathematical tasks according to the extent each factor is present in the test items. Examples are given to show how the rating scheme would be applied, and the assumption is that a total score across the factors for an individual item (20 score points could be generated) would be strongly related to item difficulty; and by implication, successful completion of particular items can be used as indicators of levels of mathematical proficiency.
The PISA 2012 framework specifies a set of ‘fundamental mathematical capabilities’, the activation of which is assumed to collectively provide indicators of mathematical literacy. A scheme designed to rate individual items based on the extent to which each of those capabilities is needed to respond to PISA questions has been developed as part of research activities documented by Turner, Blum and Niss (2015[192]). This research has shown that the scheme predicts the difficulty of PISA test items. Evidence of activation of the capabilities is fundamental to PISA’s descriptions of growing mathematical literacy.
The 2017 review team agreed that given the general consistency in what is being assessed by both PIAAC and PISA, and based on their review and knowledge of the two frameworks and item pools, there could be much to gain from making significant and more explicit connections and links between the PIAAC numeracy framework and the PISA mathematical literacy framework and assessment. The best way to do this would be by establishing an empirical relationship between the two scales through a mapping/linking study where adults and 15-year-olds would sit common items from across both assessments. This would make analysis and comparison across the two assessments stronger and more useful for research purposes, both within countries and internationally. It would also enable research into how items are approached differently by those in school versus those that are not, and support provision of stronger data to enable research into progress from school into adult life.
The implementation of an equivalent to the PIAAC reading components assessment in PIAAC’s numeracy assessment is outlined in this section.
The second cycle of PIAAC will include a new set of low-level items, called numeracy components, that are aiming to shed more light on the numeracy competencies of low-scoring adults (below Level 1). In analysing the first cycle of PIAAC results it was felt that information was missing to make valid inferences on what numeracy skills adults below Level 1 possessed or lacked. It should be noted that the NEG and the 2017 review recognised that there were two solutions to achieve this. The first was to write some new easier items to complement the existing three below Level 1 items, along with this second solution of developing the numeracy components.
In the review of the PIAAC Cycle 1 numeracy framework, possible building blocks for the components were investigated and arising issues and constraints were discussed (Tout et al., 2017[1]).
During the development of this numeracy framework for PIAAC Cycle 2 and the design of the second cycle numeracy items, a further investigation was undertaken to establish which kind of numeracy assessment items would be suitable for assessing some of the identified numeracy components, given the constraints of the delivery modalities.
In its first cycle, PIAAC included for the first time an assessment of ‘reading component skills’ often abbreviated to ‘reading components’ to evaluate how well individuals with low levels of proficiency master the basic building blocks of reading (Sabatini and Bruce, 2009[193]). Three types of tasks were included in Cycle 1:2
Print vocabulary, where respondents were asked to identify which one of four words matched a picture.
Sentence processing, where a single sentence was presented and respondents were asked to identify if it made sense or not by selecting “Yes” or “No”.
Paragraph comprehension, which were cloze tasks where respondents selected one of several words to make sentences within a paragraph make sense.
The delivery of the reading components assessment in the first cycle of PIAAC included a level of oral support by the test interviewer, and this is under consideration for Cycle 2. Both accuracy and timing information were captured for these items, allowing the analysis of both skill and fluency.
In Cycle 2, the reading components measure will include sentence and paragraph comprehension items and will be administered on the tablet. This will make an automated presentation of items possible, allowing respondents to better demonstrate fluency, and will also allow the collection of comparable timing information.
The overall performance in PIAAC Cycle 1 showed that 5% of adults surveyed in the first round of 24 countries were at below Level 1. When including the second round of countries, the results showed 6.7% were at below Level 1 across 33 countries. This compares with a performance in reading literacy of 3.3% across the original 24 countries, and 4.5% for the second round of 33 countries. Hence, the percentages of adults performing at the lowest level in numeracy are significantly higher when compared with literacy (OECD, 2013[161]; OECD, 2016[194]). Therefore, there is a very strong argument from the empirical data for developing an equivalent to the reading components assessment in the PIAAC numeracy assessment based on the higher numbers of adults performing at that level compared with reading.
The purpose of defining, constructing, and administering items for a numeracy components assessment have the same aims: to develop a set of “fine-grained tasks” so that “at least some of these adults would demonstrate some level of numeracy knowledge and skills”.
In numeracy, such component skills for adults have been much less researched, theorised, and examined quantitatively compared with component skills for literacy [e.g., see (Grotlüschen et al., 2016[195]; Sabatini and Bruce, 2009[193])]. Therefore, conceptualising and developing the numeracy components assessment in the second cycle of PIAAC was a challenging task. There was recognition that much more research and discussion needed to be undertaken to establish the sensible and meaningful content of such numeracy skills component for adults, the scope of those skills, and how they relate to the existing PIAAC below Level 1 items and their descriptions. However, time constraints related to the need to develop the test items within 6 months of the NEG’s first meeting in March 2018, meant that the NEG had to proceed in the best way it could.
The NEG was aware, however, that it had a unique opportunity to create an assessment that had not been developed or administered before and that would potentially provide valuable research data and insights into adults with low levels of numeracy and mathematical skills. The opportunity to utilise the PIAAC Field Trial to test how such a numeracy components assessment would work was taken on board and work proceeded to research what would work best and to develop some trial items.
A range of potential sources of content were investigated by the review team (Tout et al., 2017[1]) and consequently by the NEG, but their research time was limited. In the following years, it is the NEG’s recommendation this issue should be researched and trialled more thoroughly.
The reading components are described as fine-grained foundational reading skills which precede more complex reading skills. In numeracy, such fine-grained foundation skills are not yet clearly defined. The numeracy development of individuals starts ‘in the crib’, where newborns have their first experiences with numbers, shapes and sizes of objects and spatial orientation. The exact nature of these numeracy and mathematical foundation skills is still under-researched.
To complicate things further, in research literature, the term ‘components of numeracy’ is also used for the fundamental elements which constitute the concept of numeracy. This is a different perspective from thinking of them as the foundations or prerequisites for the development of more complex numerate skills. For instance, Ginsburg, Manly and Schmitt (2006[29]) did a comprehensive investigation in existing numeracy frameworks to discern any reoccurring aspects in a range of existing numeracy frameworks. They labelled these elements as the ‘components of numeracy’ and called them “those fundamental elements that are inherent in proficient numeracy practice” (p. 2). They listed the components as: Content, Context and Cognitive and affective. This is clearly another definition of a ‘component of numeracy’ compared with the perspective of describing and defining some assessable foundational aspects of low-level adult numeracy skills.
Another clarification is also necessary. Components of numeracy are not, as sometimes assumed in laymen’s opinions, the “basics”—knowing by rote the arithmetical operations like addition and subtraction up to 20 and multiplication and division with 1-digit numbers. The NEG views these as basic arithmetical facts in the domain of operations with decontextualised numbers, which only covers a minute part of the PIAAC content dimension. These “basic” skills are in no way basic or elementary to many of the low level performing adults in PIAAC, as these skills make use of abstract, school-based notations and conventions and lacks the key dimension of “meaningfulness”, which is an essential fundamental pillar of the numeracy framework as a whole.
So, there are two major challenges to consider in the development of a numeracy components assessment for the second cycle of PIAAC. One is the breadth and the level of the mathematical content that should be included as some of the foundational skills. Second is how the meaningfulness of the items could be maintained, for instance whether the use of real-world problems embedded in authentic situations is feasible in an assessment of the components of numeracy, or at least what considerations need to be accommodated in order for the components assessment to work and be relevant to the adult respondents undertaking the assessment.
Another major challenge for the NEG is that the items must fit within the delivery options of the whole PIAAC assessment, including issues to do with the time available and the uncertainty about the level of oral support available for respondents undertaking the Components Assessment.
The constraints imposed by the practicalities of delivering such an assessment internationally in multiple languages need to be considered, and will impact on what can be achieved. Furthermore, given the likelihood of an interaction between low numeracy and low literacy skill levels, delivery of a numeracy components assessment should take special account of the reading demands of the assessment. Other factors to consider include: the time available, which will impact on the number and range of items that can be utilised in terms of content areas and difficulty levels; the delivery and item types (oral instructions and support by administrator; online delivery; interactive or not) and more. A number of these factors are discussed below.
The NEG therefore believes that this second cycle of the PIAAC numeracy framework is only taking the first steps into gaining insights into the nature of low-level adult numerate behaviour and performance. However, these are important and very valuable first steps.
It will be essential to make the reading demands as minimal as possible for this assessment of numeracy components, while maintaining the connection to real life. The review team (Tout et al., 2017[1]) suggested to offer oral/spoken support in some form or other, either from the administrator, or if conducted on a laptop or tablet, through audio or video support. Consideration could also be given to the administrator recording oral answers for the respondent.
Another recommendation is that the stimuli should be based on photos or videos of realistic representations of real-life objects, which would help to make them accessible, more familiar and more realistic and authentic, while potentially helping to reduce reading load. Another suggestion was to use real items or objects for some test items. These could be used for tasks such as comparing, sorting, or classifying. This would make the numeracy components assessment more accessible, practical and hands-on. Additionally, or alternatively, technology could be used so that similar actions could be done on screen, such as using drag-and-drop items on a laptop or tablet using touch screen capabilities. For example, respondents could be asked to order objects representing quantities by dragging and dropping rather than writing down an ordered list.
The conclusion of the NEG is that the use of the tablet allows the use of photos and realistic representations of real-life objects, which can help make them accessible and more familiar. The existing below Level 1 item on counting the number of bottles is a good example of how this can be done using a photo and little text; in fact, even without being able to read the question text it is highly likely that respondents would be able to assume what the question was asking. If some level of oral guidance or support was also made available, then this would make such questions even more accessible.
Given the overall time constraints for the assessment, the NEG was informed that there would be a restriction in time to a maximum of approximately 3 minutes for the duration of the numeracy components assessment, as is the case for the reading components assessment. This obviously presents significant restrictions on what can be included in the components. However, it needs to be noted that this restriction in time will not be revealed to the participants to avoid unwarranted stress and failure anxiety. The NEG has argued that for the Field Trial, more time be allowed (up to a total of 5 minutes) in order to trial and test the new items. This then will provide the necessary empirical data and information for a more informed decision about how to best implement the numeracy components assessment in the Main Study.
The time restriction for the delivery means that after that time has expired the set of items will be terminated and no more new items will disappear. The respondent will not be told how many items are to be presented—they will continue until either all the items have been presented, or until the time limit expires. The interviewers will be instructed to ask the participant to keep working on during the set.
As mentioned above, the Review paper suggested to offer oral/spoken support in some form or other, either from the interviewer, or if conducted on a laptop or tablet, through audio or video support. However, the constraints on the delivery of PIAAC and the need for approved translations of any spoken texts in any video or audio files, made it unfeasible to build in any oral directions into the assessment itself. However, as mentioned earlier, consideration is being given to the possibility of some level of oral support or instructions by the test interviewer.
Hence, it was very important to address the issue of minimising the use of text, and the best way to do this was through the use of photos and realistic representations of real-life objects, which can help make items more accessible and more familiar.
One other issue is that it would seem obvious that some of these numeracy components assessment items should be based around recognising and working with money, which appears to have the advantage of being a) number-based, and b) important in most adults’ lives, and also relatively familiar. Money is, however, highly country-dependent: its very familiarity is grounded in its localisation in a particular set of relationships, financial and otherwise, and these are not necessarily consistent across countries. Monetary systems across participating countries vary significantly, and although PIAAC specifies strict guidelines about changing the magnitude of monetary amounts in order to try to keep them at the same time realistic and mathematically comparable, at this lowest level of mathematical complexity this may be difficult to achieve. The NEG believes that the number sense construct underpins an understanding of currency and working with money.
Based on the delivery constraints for the components, some of the recommended item type options which were seen to best support a numeracy components assessment, would include:
use of photos and realistic representations of real-life objects
minimise the need to read written instructions – use a simple, single stem to introduce the sets of questions/items (note that this approach will also contribute to the fluency measure as respondents will not need to spend time reading changing item instructions)
not expecting any written responses – so use “tap-on” style of responses.
As with the PIAAC reading components assessment, the aim is to better understand the numeracy and mathematical skills of adults scoring below Level 1. These will be the individuals who in previous surveys essentially could not answer any, or many, of the numeracy items correctly.
The current, lowest level in PIAAC is below Level 1, and the description of this level of numeracy performance in PIAAC is:
Tasks at this level require the respondents to carry out simple processes such as counting, sorting, performing basic arithmetic operations with whole numbers or money, or recognising common spatial representations in concrete, familiar contexts where the mathematical content is explicit with little or no text or distractors. (OECD, 2013, p. 76[161])
The existing three below Level 1 PIAAC numeracy items are:
the counting or estimating of the number of objects shown in a photo where the objects are in layers and therefore not all visible (total is under 100)
adding up three whole numbers listed in a short text (total is just over 200)
identifying the item that was packed first from four supermarket price tags, each of which includes the date packed.
Therefore, the skills that need to be assessed in a numeracy components assessment would preferably need to be at a lower level than those three questions.
In a recent review of options for developing a low level assessment of numeracy for adults in low- and middle-income countries (UNESCO, 2016[196]), the authors said:
It is therefore necessary to distinguish between people with no formal skills (those who have relatively few mental calculation skills beyond counting simple quantities and who cannot understand the meaning of written digits) and with low formal skills (those who can engage in some mental calculations using indigenous number systems or measurement techniques but know few print-based or formal numeracy symbols and systems, even if they may be able to complete very simply written math problems). (UNESCO, 2016, p. 284[196])
These issues are at the heart of the development of a numeracy components assessment.
There are existing adult numeracy standards and frameworks in different countries that have described relatively low levels of numeracy competence, and these could be used as starting points for descriptions of possible numeracy components questions and tasks. One challenge is that many such frameworks, as with PIAAC numeracy Cycle 1, do not detail or describe a level below PIAAC’s existing below Level 1.
What is common at the lowest levels of existing adult numeracy frameworks is that they describe mathematical content across a number of content areas, as with PIAAC’s four content areas of Quantity and number; Dimension and shape; Change and relationships; and Data and chance.
For example, Ireland has five areas described: Quantity and number; Data handling; pattern and relationship; Problem solving; and Shape and space (Quality and Qualifications Ireland (QQI), 2016[78]). New Zealand has three areas described: Make sense of number to solve problems; Reason statistically; and Measure and interpret shape and space (Tertiary Education Commission, 2008[79]). The Netherlands has described an entrance level for adults around four domains: Numbers, Proportions, Measurement and geometry, and Relations, stressing the concrete nature of the content with a few data, a minimum of text, rounded numbers and problems taken directly from everyday life and the work environment (Centre for Innovation of Education and Training (CINOP), 2013[197]).
As examples of what is described at the lower levels approximating below Level 1 of PIAAC or lower, Box 3.3 below includes some sample statements from a number of different national adult curriculum frameworks/standards, organised against the PIAAC content areas [excerpts from: Quality and Qualifications Ireland (QQI), (2016[78]); McLean et al. (2012[77]); Tertiary Education Commission, (2008[79])].
Recognise the relationship between numerical value and groups of objects, up to and including 10.
Recognise the language of mathematics in everyday situations using elementary language, e.g., greater than, less than, bigger than, farther than.
Solve addition and subtraction problems by counting all of the objects.
Solve addition and subtraction problems by counting on or counting back, using ones and tens.
Solve multiplication problems by skip-counting, often in conjunction with one-to-one counting and often keeping track of the repeated counts by using materials (e.g., fingers) or mental images.
Read and write personally relevant numbers, e.g., street number.
Recognise and write money as symbols (e.g., $12.50) up to $100.
Recognise and use ordinal numbers from first to tenth.
Identify key characteristics of shapes and forms, e.g., number of sides, corners and curves.
Use the language of measurement in relation to shape and form, e.g., longer, shorter, wider, narrower.
Sort and describe objects by their shape attributes.
Describe, name and interpret relative positions in space.
Compare and order objects directly, using attributes of length, area, volume and capacity, weight, angle, temperature and time intervals in order to understand the attributes.
Read digital time (not including concept of am/pm).
Identify dates in a calendar.
Recognise common time sequences; e.g., the order of the days of the week.
Identifies differences and similarities between common 2 dimensional (2D) shapes.
Make a pattern; e.g., a sequence of images, symbols or sounds with two variables (different colour, same shape, etc.).
Data and chance.
Identify the use of data in everyday life; e.g., the numbers of people who want tea/coffee.
Sort objects according to their attributes, organise data about the objects and represent data, using concrete objects or pictures.
Identify all possible outcomes in situations involving simple (single-stage) chance.
Compare information and data within highly familiar simple texts, lists, charts, diagrams and tables.
In the review of the PIAAC Cycle 1 numeracy framework (Tout et al., 2017[1]), the review team acknowledged that there is a potential issue with using national adult curriculum frameworks/standards directly, because some national adult numeracy frameworks and standards have been either developed formally to align with established, hierarchical levels in child-focused curricula or are at least built on notions of children’s learning. This can be illustrated in a number of ways, for example, by the inclusion in adult curriculum frameworks of simplistic, bounded statements such as ‘can count to 20’; by specific, school-based terminology such as the ‘place values of digits in whole numbers up to 100’; or where percentages are not named and included until higher levels of performance. Such statements do not acknowledge the empirical data that exists from PIAAC or other empirical data, as it does not match the knowledge of adults nor represent the day-to-day tasks that many adults can in fact successfully undertake, but who may nonetheless be performing at below Level 1 numeracy in PIAAC (Tout et al., 2017[1]).
Another perspective on possible content for the numeracy components is the growing body of research on number sense.
Number sense appears increasingly in literature as one of the main components of “numeracy.” Being numerate means having a certain sense of quantities and numbers and how we use numbers—orally, vocally and in writing—to represent, inform, predict, and estimate phenomenon from real life.
The term number sense was coined in the 1930s by Dantzig: “Man, even in the lower stages of development, possesses a faculty which, for want of a better name, I shall call Number Sense. This faculty permits him to recognise that something has changed in a small collection when, without his direct knowledge, an object has been removed from or added to the collection.” (Dantzig, 1934, p. 1[198]). In the 1990s the concept became more visible [e.g., (Greeno, 1991[199]; Mcintosh, Reys and Reys, 1992[18])]. McIntosh, Reys and Reys developed a framework for number sense including three components: Numbers, Operations and Computational settings, which are interconnected. According to them, number sense involves being able to use numbers, operations and their applications in different computational settings. They talk about the meaningful understanding of the Hindu-Arabic number system, the development of a sense of orderliness of the number, the multiple representations for numbers (including the idea of composition / decomposition), the understanding of mathematical properties, and the relationship between operations. For them, having “number sense” means being able to solve problems in the real world, providing suitable answers, using (or creating) effective strategies to compute, count, etc. It is not just reproducing instrumentally a certain algorithm, but being able to use the mathematical knowledge and components in a flexible manner. At the same time Dehaene (1997[200]) published his best-selling book Number Sense – How the Mind Creates Mathematics, which made a connection of number sense with the structure of our brains.
Yang, Reys and Reys (2009[17]) defined number sense as “a person’s general understanding of numbers and operations and the ability to handle daily life situations that include numbers. This ability is used to develop flexible and efficient strategies (including mental computation and estimation) to handle numerical problems” (2009, p. 384[17]). Regarding the components of number sense, these authors argue “Number sense is a complex process involving many different components of numbers, operations, and their relationships” (Yang, Reys and Reys, 2009, p. 384[17]). Among these processes, they highlight two aspects, 1) the use of benchmarks in recognising the magnitude of numbers, and 2) the knowledge on the relative effects of an operation on various numbers.
Faulkner and Cain (2009[201]) claim that “the characteristics of good number sense include: a) fluency in estimating and judging magnitude, b) ability to recognise unreasonable results, c) flexibility when mentally computing, d) ability to move among different representations and to use the most appropriate representations” (Faulkner and Cain, 2009, p. 25[201]). Cain et al. (2007[202]) described a set of components of number sense as shown in Figure 3.5, where the different components of number sense all relate to and are underpinned by language:
As stated by Thompson (1995[19]), using numbers is more than reasoning about number and more than skilled calculations. It is about making sense of the situation to which we apply numbers and calculations” (Thompson, 1995, p. 220[19]). It also involves a critical way to make decisions and solve problems in Personal, Work, and Societal/community contexts (Peters, 2012[15]).
Based on the above research about the potential content of a new numeracy components assessment, the NEG also considered the possibility of developing a spatial sense components assessment, as spatial sense was also seen as a key foundational skill for the target cohort of adults. However, mainly due to a number of the constraints on the delivery of the numeracy components discussed below, especially in relation to the time available for administering the components, alongside the greater difficulty of reducing the literacy/reading demands of items based around an assessment of spatial sense, the NEG decided to move forward with using number sense as the content base for the new numeracy components assessment.
In its broadest definition, ‘number sense’ overlaps to a great extent the ‘big ideas’ and the domains of the (inter)national frameworks mentioned before. In a smaller and more fundamental interpretation, number sense relates to the sense of quantities and the sense of how numbers represent quantities. This latter interpretation turned out to be the most suitable basis for the further development of the items that must operationalise the numeracy components.
Another significant issue that needs to be addressed in developing a numeracy components assessment is that of the embedded nature of the mathematics in real-world settings and situations and the role that this plays. This is often called context-based in contrast to non-context-based tasks or contextualised in contrast to decontextualised. Individuals acquire mathematical knowledge through both formal and informal learning, and informal learning is as valuable as formal, school-based learning. The field of ethnomathematics richly documents this issue of “street maths versus school maths” and as this components assessment will often target adults with little formal schooling but who are functioning as adults in society, this issue needs to be taken on board and addressed. For example, D’Ambrosio (1985[203]) theorised the concept of ethnomathematics. Carraher, Carraher and Schliemann’s (1985[101]) research with street children in Brazil found they could operate in quite sophisticated ways when using mathematics to survive in a commercial sense, although they had been previously adjudged as being incapable of doing mathematics in schools. This was discussed in more detail in the second section under the topic School mathematics versus everyday or workplace mathematics.
Matthijsse (2000[204]) specifically addressed the issue of how adults cope with mathematical knowledge in practical daily situations and the gap between school mathematics and its formal algorithms and the mathematics that adults use in their daily lives. He looked at the informal methods adults used in daily life, and found they were often anchored and embedded in familiar knowledge and real-life settings and situations. Although his focus was on instructional methods to use with learners, his research, like the other ethnomathematical research, indicates that this proposed PIAAC numeracy components assessment cannot be constrained by only offering non-context-based tasks with the mathematics being like formal, school-based questions. However, a significant risk, and challenge, exists with regard to cultural and the possible national specificity of particular rule of thumb or informal methods, and how these differences could be overcome in an international assessment. Given this, a low-level components assessment could aim to find out about adults’ informal/common sense ways of doing mathematics—what mental models and processes do adults use when solving a numeracy problem? In addition, can data and information be collected about the connections (or non-connections) between the school ways of doing mathematics (and the use of algorithms) versus the way adults solve such problems in everyday life?
Different people will have very different settings and applications in which they may comfortably and more confidently use their mathematical knowledge. Finding the right problem situation or setting for an individual so that they can demonstrate their understanding of mathematics concepts will be a challenge. At this more basic level of mathematical knowledge, the familiarity of the setting and situation could be critical. A potential solution could be to use a form of adaptive delivery to allow respondents to be able to select from a range of settings and situations where the same content and level of mathematics content is embedded.
In relation to the three named PIAAC numeracy contexts (Personal, Work, Societal/community), it would make sense for the numeracy components assessment to focus on the more common, generic and familiar settings and contexts which would appear to be Personal, Work, Societal/community. The three existing below Level 1 numeracy items are located within those contexts. Again, a challenge exists in how to use work-related situations, given that research shows that adults with poor formal skills are often able to function ‘perfectly well’ in particular jobs where they have learned rule of thumb or other methods that enable them to get by.
One challenge with context-based items is that where the mathematics is embedded within texts and stimuli, some of the targeted cohort will not be easily able to read, interpret and hence engage with and understand the mathematics required to be used due to their potential low level of literacy skills.
These considerations strengthen the idea of keeping items for assessing numeracy components in line with the fundamental definition of number sense, focusing on the connection with quantities in real life and the way numbers are used to represent quantities. This seems feasible without necessarily using long or complex verbal descriptions to present contexts of items or to ask the questions.
Given the constraints of level, reading demands, time, and the available representation of tasks, the NEG decided to implement a modest set of number sense items that would be the main ingredients in the landscape of relevant numeracy components. These items will ask the participants to estimate quantities from real-life pictures and furthermore estimate the relative magnitude of several numerical representations of quantities.
It was intended that the respondent would be able to quickly view the stimulus without needing to read much text at all, tap on a response and immediately be sent to the following question that would be based on the same stem, requiring no need to read anything further.
The content is limited to a fundamental perspective on number sense and more specifically to:
A set of 12 items where the respondent must select the quantity (<20) of a number of objects displayed. The representations are limited to pictures of real-life objects.
A second set of 15 items on the relative magnitude of quantities or phenomena, partly from real life and partly more decontextualised.
Based on the above decisions, the NEG:
Consulted the translating partner for PIAAC, cApStAn, to assist in identifying the best question wording that could be used that would utilise a simple, single stem and reduce the need to read each question separately.
Developed a draft of the components assessment and ran two brief pilots.
Revised the draft prior to release to countries for feedback and comment.
Reviewed and revised the draft again at the NEG meeting held in October 2018.
In discussions with cApStAn it was soon realised that when translated into other languages, what seemed simple solutions and wording in English often become complicated when translated into a range of other languages. This often depended on what the objects or images were that would be presented to respondents.
After a period of discussion with cApStAn about what wording, and what content and images would work best without creating the need to change the stem throughout, it was decided that the best solution was fourfold:
To have an introduction up front that would foreground the assessment items to follow.
Include some simple practice questions that would model what was required.
To have the two sets of items – the first using the stem How many ...? for identifying a quantity.
The second set was using the stem The biggest? for identifying the relative magnitude of different quantities/values.
It did mean that some of the NEG’s ideas re items and their images or stimuli to include could not be used, such as temperatures, charts, people etc. Some items and images were gender-sensitive and this restricted some of the options.
Two pilots of the first drafts of the Numeracy Components assessment were held, utilising access to adults participating in adult literacy and numeracy classes, and who were known to be low-performing adults. One pilot was in Belgium with 10 adults, and the other in Spain with 29 adults. The main outcomes from the pilots included:
The participating adults were positive about the experience—they were normally used to not getting mathematics/numeracy questions correct.
They liked the real-life images to quantify/count.
They could all answer all of the How Many? questions correctly, but some took a long time.
Some of the adults took up to 30 minutes to answer the sets of questions.
There were clear boundaries in knowledge with The biggest? set of questions. Difficulty started with the understanding and comparing of decimal numbers and fractions.
The results from the pilots indicated that the assessment, for the most part, worked successfully. A number of issues resulted that enabled the NEG to make further decisions and refinements to the draft assessment. These included the following:
For the most part, the content was appropriate for the target group, and the wording and presentation seemed to be accessible—some questions were able to be refined based on the observations made as the adults undertook the assessment.
The items that assessed the ability to recognise and answer some decontextualised basic sums where the adults needed to recognise the meaning of some standard arithmetical operations were confusing and not appropriate for these learners—this reinforced the beliefs of the NEG about the relevance and meaning of such types of test items for adults. They have been removed for the Field Trial.
Given the length and range of times taken, the timing and the estimate of fluency will be crucial to measure.
The number of items to be included in the Field Trial have been reduced, both due to the time taken, but also because it was felt that the extra number of items was not collecting extra data or information.
The NEG has refined the sets of questions and have developed two different forms for the Field Trial, with linking items. The NEG will select best performing items from the Field Trial for the Main Study.
For the Field Trial there will be 12 questions of this type, with a maximum time allowance of 2 minutes. Here is a mock-up of an example of the sort of item asked in this set of questions. The respondent is to tap on the matching number.
For the Field Trial there will be 15 questions of this type, with a maximum time allowance of 3 minutes. Here is a mock-up of the sort of item asked in this set of questions. The respondent is to tap on the biggest item/number of the set shown.
There will be a restriction in time for the Field Trial to a maximum of 5 minutes – 2 minutes for the set of How many? items; and 3 minutes for the set of The biggest? items. However, this restriction in time will not be revealed to the participants to avoid unwarranted stress and potential anxiety about failure. Respondents will just be told to do their best to work both accurately and quickly. The timeframe for the delivery will be limited to the number of minutes and after that time has expired the set will be terminated and no more new items will appear. The respondent will not be aware that there were unanswered items in the set, if that is the case. The time parameter will be recorded for each participant and will be analysed in relation to a measure of fluency. However, a decision how to incorporate the time issue and the estimate of fluency in the reporting will only be decided after the Field Trial data is analysed.
Given that in the delivery of the reading components assessment each part was timed in order to be able to get an estimate of speed and automaticity, this should also be made available for the numeracy components assessment. Collecting the timing information and the ability to create measures of fluency in number sense will provide the capability to look for any correlates of interest, such as with the respondent’s performance in numeracy overall and/or with particular dimensions of the numeracy assessment.
[151] Askew, M. (2008), “Mathematical discipline knowledge requirements for prospective primary teachers, and the structure and teaching approaches of programs designed to develop that knowledge”, in Sullivan, P. and T. Woods (eds.), Knowledge and Beliefs in Mathematics Teaching and Teaching Development, Sense Publishers.
[148] Askew, M. et al. (1997), Effective teachers of numeracy in primary schools: Teachers’ beliefs, practices and pupils’ learning, Paper presented at the British Educational Research Association Annual Conference, September 1997, University of York.
[171] Atkinson, R. (2005), “Multimedia Learning of Mathematics”, in The Cambridge Handbook of Multimedia Learning, Cambridge University Press, http://dx.doi.org/10.1017/cbo9780511816819.026.
[88] Australian Association of Mathematics Teachers (AAMT) and Australian Industry Group (AiGroup) (2014), Tackling the School–Industry Mathematics Divide, Commonwealth of Australia.
[45] Baker, D. and B. Street (1994), “Literacy and numeracy: Concepts and definitions”, in Husen, T. and E. Postlethwaite (eds.), Encyclopedia of Education, Pergamon Press, New York.
[163] Bennett, R. (2015), “The changing nature of educational assessment”, Review of Research in Education, Vol. 39/1, pp. 370-407, http://dx.doi.org/10.3102/0091732x14554179.
[168] Bennett, R. (2010), “Technology for large-scale assessment”, in Peterson, P., E. Baker and B. McGaw (eds.), International Encyclopedia of Education, 3rd edition, Elsevier, Oxford.
[167] Bennett, R. (1998), Reinventing assessment: Speculations on the future of large-scale educational testing, Policy Information Center, Educational Testing Service, Princeton, NJ, https://www.ets.org/research/policy_research_reports/publications/report/1998/cgln.
[20] Benn, R. (1997), Adults Count Too: Mathematics for Empowerment, National Institute of Adult Continuing Education (NIACE), Leicester.
[89] Bessot, A. and L. Ridgway (eds.) (2000), Education for Mathematics in the Workplace, Springer, New York.
[62] Binkley, M. et al. (2011), “Defining twenty-first century skills”, in Griffin, P., B. McGaw and E. Care (eds.), Assessment and Teaching of 21st Century Skills, Springer, Dordrecht, http://dx.doi.org/10.1007/978-94-007-2324-5_2.
[83] Bishop, A. (1988), Mathematical Enculturation: A Cultural Perspective in Mathematics Education, D. Reidel Publishing Co., Dordrecht.
[149] Boaler, J. and C. Humphreys (2005), Connecting Mathematical Ideas: Middle School Video Cases to Support Teaching and Learning, Heinemann, Portsmouth, NH.
[169] Bower, M. et al. (2014), “Augmented reality in education – cases, places and potentials”, Educational Media International, Vol. 51/1, pp. 1-15, http://dx.doi.org/10.1080/09523987.2014.889400.
[189] Brooks, G., K. Heath and A. Pollard (2005), Assessing adult literacy and numeracy: A review of assessment instruments, National Research and Development Centre for Adult Literacy and Numeracy, London.
[144] Bruner, J. (1960), The Process of Education, Harvard University Press, Cambridge, MA.
[90] Buckingham, E. (1997), Specific and Generic Numeracies of the Workplace, Deakin University, Melbourne.
[126] Buckley, S. (2013), Deconstructing maths anxiety: Helping students to develop a positive attitude towards learning maths (ACER Occasional Essays), Australian Council for Educational Research (ACER), Melbourne, Victoria, https://research.acer.edu.au/learning_processes/16.
[21] Bynner, J. and S. Parsons (2005), Does numeracy matter more?, National Research and Development Centre for Adult Literacy and Numeracy (NRDC), London, http://www.nrdc.org.uk/wp-content/uploads/2005/01/Does-numeracy-matter-more.pdf.
[202] Cain, C. et al. (2007), The Components of Number Sense, NC Math Foundations Training, Exceptional Children’s Division of the North Carolina Department of Public Instruction (NCDPI), Raleigh, NC.
[84] Carnevale, A., L. Gainer and A. Meltzer (1990), Workplace Basics: The Essential Skills Employers Want, Jossey Bass, San Francisco.
[57] Carpentieri, J., J. Litster and L. Frumkin (2009), Adult numeracy: A review of research, National Research and Development Centre for Adult Literacy and Numeracy (NRDC), London, England, https://www.nationalnumeracy.org.uk/sites/default/files/documents/Adult_numeracy_a_review_of_research/carpentieri_et_al_2009_bbc_adult_numeracy_a_review_of_research.pdf.
[101] Carraher, T., D. Carraher and A. Schliemann (1985), “Mathematics in the streets and in schools”, British Journal of Developmental Psychology, Vol. 3/1, pp. 21-29, http://dx.doi.org/10.1111/j.2044-835x.1985.tb00951.x.
[197] Centre for Innovation of Education and Training (CINOP) (2013), Standaarden en Eindtermen VE [Standards and Attainment Goals Adult Education], Den Bosch, https://taalenrekenenmbo.nl/app/uploads/nieuw-2.-Standaarden-en-eindtermen-ve.pdf.
[6] Charles, R. (2005), “Big ideas and understandings as the foundation for elementary and middle school mathematics”, Journal of Mathematics Education, Vol. 7/3, pp. 9-24.
[59] Chisman, F. (2011), Facing the Challenge of Numeracy in Adult Education, Council for Advancement of Adult Literacy, New York, http://www.caalusa.org/NumeracyChallenge.pdf.
[145] Clark, E. (2011), “Concepts as organizing frameworks”, Encounter, Vol. 24/3, pp. 32-44, http://www.ojs.great-ideas.org/Encounter/Clark243.pdf.
[150] Clarke, B. and D. Clarke (2004), “Using questioning to elicit and develop children’s mathematical thinking”, in Bright, G. and R. Rubenstein (eds.), Professional Development Guidebook for Perspectives on the Teaching of Mathematics, National Council of Teachers of Mathematics, Reston, VA.
[22] Coben, D. et al. (2003), Adult Numeracy: Review of Research and Related Literature, National Research and Development Centre for Adult Literacy and Numeracy (NRDC), London.
[91] Coben, D. et al. (2010), Benchmark Assessment of Numeracy for Nursing: Medication Dosage Calculation at Point of Registration, NHS Education for Scotland, Edinburgh.
[23] Coben, D., J. O’Donoghue and G. FitzSimons (eds.) (2000), Perspectives on Adults Learning Mathematics, Kluwer Academic Publishers, London.
[44] Cockcroft, W. (1982), Mathematics Counts, Her Majesty’s Stationery Office (HMSO), London, http://www.educationengland.org.uk/documents/cockcroft/cockcroft1982.html.
[24] Condelli, L. et al. (2006), A Review of the Literature in Adult Numeracy: Research and Conceptual Issues, American Institutes for Research, Washington D.C.
[25] Coulombe, S., J. Tramblay and S. Marchand (2004), Literacy scores, human capital and growth across fourteen OECD countries, Statistics Canada, Ottawa, https://www150.statcan.gc.ca/n1/pub/89-552-m/89-552-m2004011-eng.pdf.
[43] Crowther (1959), 15 to 18: A report of the Central Advisory Council of Education (England), Volume 1, Her Majesty’s Stationery Office (HSMO), London, http://www.educationengland.org.uk/documents/crowther/crowther1959-1.html.
[203] D’Ambrosio, U. (1985), “Ethnomathematics and its place in the history and pedagogy of mathematics”, For the Learning of Mathematics, Vol. 5/1, pp. 44-48.
[198] Dantzig, T. (1934), Number - The Language of Science, Macmillan Company, New York.
[200] Dehaene, S. (1997), Number Sense – How the Mind Creates Mathematics, Penguin Books, London.
[185] Department for Education (DfE) (2014), Statutory framework for the early years foundation stage: Setting the standards for learning, development and care for children from birth to five, Department for Education, London.
[162] Diezmann, C. and T. Lowrie (2008), “The role of information graphics in mathematical proficiency”, in Goos, M., R. Brown and K. Makar (eds.), Navigating Currents and Charting Directions. Proceedings of the 31st annual conference of the Mathematics Education Research Group of Australasia, Mathematics Education Research Group of Australasia Inc., Brisbane, https://merga.net.au/Public/Publications/Annual_Conference_Proceedings/2008_MERGA_CP.aspx.
[160] Dossey, J. (1997), “Defining and measuring quantitative literacy”, in Steen, L. (ed.), Why Numbers Count: Quantitative Literacy for Tomorrow’s America, College Entrance Examination Board, New York.
[119] Ernest, P. (2004), “Relevance versus utility: Some ideas on what it means to know mathematics”, in Clarke, B. et al. (eds.), International Perspectives on Learning and Teaching Mathematics, National Centre for Mathematics Education, Göteborg University.
[63] Expert Group on Future Skills Needs (Ireland) (2007), Tomorrow’s Skills: Towards a National Skills Strategy: 5th Report, Expert Group on Future Skills Needs, Dublin.
[201] Faulkner, V. and C. Cain (2009), “The components of number sense: An instructional model for teachers”, Teaching Exceptional Children, Vol. 41/5, pp. 24-30, http://dx.doi.org/10.1177/004005990904100503.
[92] FitzSimons, G. (2005), “Numeracy and Australian workplaces: Findings and implications”, Australian Senior Mathematics Journal, Vol. 19/2, pp. 27-42.
[112] FitzSimons, G. and D. Coben (2009), “Adult numeracy for work and life: Curriculum and teaching implications of recent research”, in International Handbook of Education for the Changing World of Work, Springer, Dordrecht, http://dx.doi.org/10.1007/978-1-4020-5281-1_179.
[26] Forman, S. and L. Steen (1999), Beyond Eighth Grade: Functional Mathematics for Life and Work, National Center for Research in Vocational Education, University of California, Berkeley.
[64] Foundation for Young Australians (2017), The New Basics: Big data reveals the skills young people need for the New Work Order, https://www.fya.org.au/wp-content/uploads/2016/04/The-New-Basics_Update_Web.pdf.
[74] Frankenstein, M. (1989), Relearning mathematics: A different third ‘R’ – Radical maths, Free Association Books, London.
[153] Gal, I. (2006), Assessment of adult numeracy in PIAAC: A conceptual and development framework (Unpublished manuscript prepared for OECD), University of Haifa, Haifa.
[75] Gal, I. (2002), “Adults’ Statistical Literacy: Meanings, Components, Responsibilities”, International Statistical Review / Revue Internationale de Statistique, Vol. 70/1, p. 1, http://dx.doi.org/10.2307/1403713.
[27] Gal, I. (2000), Adult numeracy development: Theory, research, practice, Hampton Press, Cresskill, N.J.
[3] Gal, I. and D. Tout (2014), “Comparison of PIAAC and PISA Frameworks for Numeracy and Mathematical Literacy”, OECD Education Working Papers, No. 102, OECD Publishing, Paris, https://dx.doi.org/10.1787/5jz3wl63cs6f-en.
[28] Gal, I. et al. (2005), “Adult numeracy and its assessment in the ALL survey: A conceptual framework and pilot results”, in Murray, S., Y. Clermont and M. Binkley (eds.), Measuring Adult Literacy and Life Skills: New Frameworks for Assessment, Ottawa: Statistics Canada.
[125] Geiger, V., M. Goos and S. Dole (2014), “Students’ perspectives on their numeracy development across the learning areas”, in Li, Y. and G. Lappan (eds.), Mathematics Curriculum in School Education, Springer, New York.
[54] Geiger, V., M. Goos and H. Forgasz (2015), “A rich interpretation of numeracy for the 21st century: A survey of the state of the field”, ZDM: The International Journal on Mathematics Education, Vol. 47/4, pp. 531-548, http://dx.doi.org/10.1007/s11858-015-0708-1.
[164] Geisinger, K. (2016), “21st century skills: What are they and how do we assess them?”, Applied Measurement in Education, Vol. 29/4, pp. 245-249, http://dx.doi.org/10.1080/08957347.2016.1209207.
[187] Gillespie, J. (2004), The “Skills for Life” national survey of adult numeracy in England. What does it tell us? What further questions does it prompt?, paper presented at ICME-10, the 10th International Congress on Mathematics Education, Copenhagen, Denmark.
[29] Ginsburg, L., M. Manly and M. Schmitt (2006), The Components of Numeracy (NCSALL Occasional Paper), Harvard Graduate School of Education, National Center for the Study of Adult Learning and Literacy, Cambridge, MA.
[124] Goos, M., V. Geiger and S. Dole (2014), “Transforming professional practice in numeracy teaching”, in Li, Y., E. Silver and S. Li (eds.), Transforming Mathematics Instruction: Multiple Approaches and Practices, Springer, Cham, http://dx.doi.org/10.1007/978-3-319-04993-9_6.
[108] Greeno, J. (2003), “Situative research relevant to standards for school mathematics”, in Kilpatrick, J., W. Martin and D. Schifter (eds.), A Research Companion to Principles and Standards for School Mathematics, National Council of Teachers of Mathematics, Reston, VA.
[199] Greeno, J. (1991), “Number sense as situated knowing in a conceptual domain”, Journal for Research in Mathematics Education, Vol. 22/3, pp. 170-218, http://dx.doi.org/10.2307/749074.
[65] Griffin, P., B. McGaw and E. Care (eds.) (2012), Assessment and Teaching of 21st Century Skills, Springer, New York.
[60] Griffiths, G. and R. Stone (2013), Teaching Adult Numeracy: Principles and Practice, Open University Press, Maidenhead.
[195] Grotlüschen, A. et al. (2016), “Adults with Low Proficiency in Literacy or Numeracy”, OECD Education Working Papers, No. 131, OECD Publishing, Paris, https://dx.doi.org/10.1787/5jm0v44bnmnx-en.
[152] Hagedorn, L. et al. (2003), Frameworks for adult numeracy education: A survey and discussion, National Literacy Secretariat, Ontario.
[132] Harris, M. (1991), Schools, Mathematics and Work, Falmer Press, London.
[134] Hoogland, K. et al. (2018), “Changing representation in contextual mathematical problems from descriptive to depictive: The effect on students’ performance”, Studies in Educational Evaluation, Vol. 58, pp. 122-131, http://dx.doi.org/10.1016/j.stueduc.2018.06.004.
[142] Hoogland, K. et al. (2016), “Representing contextual mathematical problems in descriptive or depictive form: Design of an instrument and validation of its uses”, Studies in Educational Evaluation, Vol. 50, pp. 22-32, http://dx.doi.org/10.1016/j.stueduc.2016.06.005.
[186] Hoogland, K. and D. Tout (2018), “Computer-based assessment of mathematics into the twenty-first century: pressures and tensions”, ZDM, Vol. 50/4, pp. 675-686, http://dx.doi.org/10.1007/s11858-018-0944-2.
[93] Hoyles, C. et al. (2010), Improving Mathematics at Work: The Need for Techno-Mathematical Literacies, Routledge, London and New York.
[55] Hoyles, C., R. Noss and S. Pozzi (2001), “Proportional reasoning in nursing practice”, Journal for Research in Mathematics Education, Vol. 32/1, p. 4, http://dx.doi.org/10.2307/749619.
[30] Hoyles, C. et al. (2002), Mathematical Skills in the Workplace: Final Report to the Science Technology and Mathematics Council, Institute of Education, University of London, Science, Technology and Mathematics Council, London, https://discovery.ucl.ac.uk/id/eprint/1515581/1/Hoyles2002MathematicalSkills.pdf.
[7] Hurst, C. (2014), “Big challenges and big opportunities: The power of ’Big Ideas’ to change curriculum and the culture of teacher planning”, in Anderson, J., M. Cavanagh and A. Prescott (eds.), Curriculum in Focus: Research Guided Practice. Proceedings of the 37th annual conference of the Mathematics Education Research Group of Australasia (MERGA), MERGA, Sydney, https://www.merga.net.au/Public/Public/Publications/Annual_Conference_Proceedings/2014_MERGA_CP.aspx.
[8] Hurst, C. and D. Hurrell (2014), “Developing the big ideas of number”, International Journal of Educational Studies in Mathematics, Vol. 1/2, pp. 1-18.
[31] Johnston, B. (1994), “Critical numeracy?”, Fine Print, Vol. 16/4, pp. 7-12.
[40] Johnston, B. and T. Maguire (2005), “Adult numeracy: Policy and practice in global contexts of lifelong learning”, Adult Literacy and Numeracy Australian Research Consortium (ALNARC), School of Education, Victoria University, Melbourne.
[32] Jonas, N. (2018), “Numeracy practices and numeracy skills among adults”, OECD Education Working Papers, No. 177, OECD Publishing, Paris, https://dx.doi.org/10.1787/8f19fc9f-en.
[5] Jones, G., C. Langrall and C. Thornton (2002), “Elementary students’ access to powerful mathematical ideas”, in English, L. (ed.), Handbook of International Research in Mathematics Education, Lawrence Erlbaum Associate, Mahwah, New Jersey.
[154] Jones, S. (2006), Designing Numeracy in PIAAC (Background paper prepared for the OECD-Canada Expert Technical Workshop on Numeracy, Ottawa, November 10, 2006), Ottawa.
[33] Jones, S. (1995), “The distribution of literacy”, in Literacy, Economy and Society: Results of the First International Adult Literacy Survey, OECD and Statistics Canada, OECD Publishing, Paris.
[100] Jorgensen Zevenbergen, R. (2010), “Young workers and their dispositions towards mathematics: Tensions of a mathematical habitus in the retail industry”, Educational Studies in Mathematics, Vol. 76/1, pp. 87-100, http://dx.doi.org/10.1007/s10649-010-9267-0.
[42] Karaali, G., E. Villafane-Hernandez and J. Taylor (2016), “What’s in a name? A critical review of definitions of quantitative literacy, numeracy, and quantitative reasoning”, Numeracy, Vol. 9/1, pp. 1-34, http://dx.doi.org/10.5038/1936-4660.9.1.2.
[94] Kent, P. et al. (2011), “Measurement in the workplace: The case of process improvement in manufacturing industry”, ZDM, Vol. 43/5, pp. 747-758, http://dx.doi.org/10.1007/s11858-011-0359-9.
[113] Kent, P. et al. (2007), “Characterizing the use of mathematical knowledge in boundary-crossing situations at work”, Mind, Culture, and Activity, Vol. 14/1-2, pp. 64-82, http://dx.doi.org/10.1080/10749030701307747.
[157] Kilpatrick, J., J. Swafford and B. Findell (eds.) (2001), “Mathematics Learning Study Committee, National Research Council: Conclusions and recommendations”, in Adding It Up: Helping Children Learn Mathematics, National Academies, Washington, DC.
[81] Kindler, J. et al. (1996), Certificates in General Education for Adults, Adult, Community and Further Education Board, Melbourne, Victoria.
[191] Kirsch, I., A. Jungblut and P. Mosenthal (1998), “The measurement of adult literacy”, in Murray, S., I. Kirsch and L. Jenkins (eds.), Adult Literacy in OECD Countries: Technical Report on the First International Adult Literacy Survey, National Center for Education Statistics, U.S. Department of Education, Washington, DC.
[190] Kirsch, I. and P. Mosenthal (1990), “Exploring document literacy: Variables underlying the performance of young adults”, Reading Research Quarterly, Vol. 25/1, p. 5, http://dx.doi.org/10.2307/747985.
[9] Kuntze, S. et al. (2009), Awareness of Big Ideas in Mathematics Classrooms (ABCmaths). Final report to the European Union about the EU-funded project “ABCmaths”, European Union, https://www.researchgate.net/publication/301298435_Awareness_of_Big_Ideas_in_Mathematics_Classrooms_ABCmaths_Progress_report_Fortschrittsbericht_an_die_Europaische_Union_zum_EU-geforderten_Projekt_ABCmaths.
[10] Kuntze, S. et al. (2011), “Professional knowledge related to Big Ideas in Mathematics–an empirical study with pre-service teachers”, in Pytlak, M., T. Rowland and E. Swoboda (eds.), A Study of Teaching Practices to Issues in Teacher Education. Proceedings of the 7th Congress of the European Society for Research in Mathematics Education (pp. 2717-2726), ERME, Rzeszów, http://www.cerme7.univ.rzeszow.pl/WG/17a/CERME7_WG17A_Kuntze_et_al..pdf.
[53] Lave, J. (1988), Cognition in Practice: Mind, Mathematics and Culture in Everyday Life, Cambridge University Press, New York, http://dx.doi.org/10.1017/cbo9780511609268.
[46] Lindenskov, L. and T. Wedege (2001), Numeracy as an Analytical Tool in Mathematics Education and Research, Centre for Research in Learning Mathematics, Roskilde.
[158] Lowrie, T. and C. Diezmann (2009), “National numeracy tests: A graphic tells a thousand words”, Australian Journal of Education, Vol. 53/2, pp. 141–158, https://research.acer.edu.au/aje/vol53/iss2/3.
[184] Maguire, T. and J. O’Donoghue (2002), “A grounded approach to practitioner training in Ireland: Some findings from a national survey of practitioners in adult basic education”, in Johansen, L. and T. Wedege (eds.), Numeracy for Empowerment and Democracy? Proceedings of the 8th International Conference of Adult Learning Mathematics, Roskilde University, Centre for Research in Learning Mathematics, Roskilde.
[47] Maguire, T. and J. O’Donoghue (2003), “Numeracy concept sophistication - an organizing framework, a useful thinking tool”, in Maass, J. and W. Schlöglmann (eds.), Learning Mathematics to Live and Work in our World. Proceedings of the 10th international conference on Adults Learning Mathematics (ALM-10), Strobl, Austria, ALM and Johannes Kepler Universität, Linz.
[129] Maguire, T. and A. Smith (2016), Maths Eyes- A Concept with Potential, Invited paper presented at TSG 6, ICME 13 (13th International Congress on Mathematical Education, Hamburg, 24-31 July 2016).
[146] Ma, L. (1999), Knowing and teaching elementary mathematics: Teachers’ understanding of fundamental mathematics in China and the United States, Lawrence Erlbaum, Mahwah, N.J.
[95] Marr, B. and J. Hagston (2007), Thinking Beyond Numbers: Learning Numeracy for the Future Workplace, NCVER, Adelaide, https://www.ncver.edu.au/__data/assets/file/0017/5426/nl05002.pdf.
[204] Matthijsse, W. (2000), “Adult numeracy at the elementary level: Addition and subtraction up to 100”, in Gal, I. (ed.), Adult Numeracy Development: Theory, Research, Practice. Series on Literacy: Research, Policy, and Practice, Hampton Press, Cresskill, NJ.
[127] Ma, X. (1999), “A meta-analysis of the relationship between anxiety toward mathematics and achievement in mathematics”, Journal for Research in Mathematics Education, Vol. 30/5, pp. 520-540, http://dx.doi.org/10.2307/749772.
[178] Mayer, R. (2009), Multimedia Learning (2nd edition), Cambridge University Press, New York.
[177] Mayer, R. (ed.) (2005), The Cambridge Handbook of Multimedia Learning, Cambridge University Press, http://dx.doi.org/10.1017/cbo9780511816819.
[18] Mcintosh, A., B. Reys and R. Reys (1992), “A proposed framework for examining basic number sense”, For the Learning of Mathematics, Vol. 12/3, pp. 2-8.
[77] McLean, P. et al. (2012), Australian Core Skills Framework: 5 Core Skills, 5 Levels of Performance, 3 Domains of Communication, Australian Government, Canberra.
[159] McLeod, D. (1992), “Research on affect in mathematics education: A reconceptualization”, in Grows, D. (ed.), Handbook of Research on Mathematics Teaching and Learning, Macmillan Publishing Company, New York.
[205] Moore, D. and G. Cobb (2000), “Statistics and mathematics: Tension and cooperation”, The American Mathematical Monthly, Vol. 107/7, pp. 615-630, http://dx.doi.org/10.1080/00029890.2000.12005247.
[188] Murat, F. (2005), Les compétences des adultes à l’écrit, en calcul et en compréhension orale (Report No.1044), INSEE, Paris, https://www.epsilon.insee.fr/jspui/bitstream/1/224/1/ip1044.pdf.
[34] Murnane, R., J. Willett and F. Levy (1995), “The Growing Importance of Cognitive Skills in Wage Determination”, No. 5076, National Bureau of Economic Research, Cambridge, MA, http://www.nber.org/papers/w5076.pdf.
[155] Murray, S. (2006), Reflections on the Rationale for, and Measurement of, Numeracy in PIAAC, (Background paper prepared for the OECD-Canada Expert Technical Workshop on Numeracy, Ottawa, November 10, 2006), Statistics Canada, Ottawa.
[4] Murray, S., Y. Clermont and M. Binkley (eds.) (2005), Measuring Adult Literacy and Life Skills: New frameworks for Assessment, Catalogue No. 89-552-MIE, No. 13, Statistics Canada, Ottawa.
[87] National Council of Teachers of Mathematics (NCTM) (2017), Catalyzing Change in High School Mathematics, NCTM, Reston, VA, https://www.nctm.org/uploadedFiles/Standards_and_Positions/CatalyzingChangePublicReview.pdf.
[120] National Council of Teachers of Mathematics (NCTM) (2000), Principles and Standards for School Mathematics, NCTM, Reston, VA.
[61] National Institute of Adult Continuing Education (NIACE) (2011), Numeracy Counts, NIACE Committee of Inquiry on Adult Numeracy Learning. NIACE, Leicester, https://learningandwork.org.uk/wp-content/uploads/2020/02/Numeracy-Counts.pdf.
[35] National Research and Development Centre (NRDC) (2006), Programme for the International Assessment of Adult Competencies: An Adult Numeracy Assessment Instrument for the UK, (Background paper prepared for the OECD-Canada Expert Technical Workshop on Numeracy, November 10, 2006, Ottawa). National Research and Development Centre for Adult Literacy and Numeracy, London.
[56] Noss, R., C. Hoyles and S. Pozzi (2002), “Abstraction in Expertise: A Study of Nurses’ Conceptions of Concentration”, Journal for Research in Mathematics Education, Vol. 33/3, p. 204, http://dx.doi.org/10.2307/749725.
[102] Nunes, T. (1992), “Ethnomathematics and everyday cognition”, in Grouws, D. (ed.), Handbook of Research on Mathematics Teaching and Learning, Macmillan, New York.
[103] Nunes, T., A. Schliemann and D. Carraher (1993), Street Mathematics and School Mathematics, Cambridge University Press, Cambridge.
[69] OECD (2019), “PISA 2018 Global Competence Framework”, in PISA 2018 Assessment and Analytical Framework, OECD Publishing, Paris, https://dx.doi.org/10.1787/043fc3b0-en.
[123] OECD (2018), PISA 2021 Mathematics Framework (Second Draft) EDU/PISA/GB(2018)19, Directorate for Education and Skills, Programme For International Student Assessment.
[41] OECD (2017), Building Skills for All in Australia: Policy Insights from the Survey of Adult Skills, OECD Skills Studies, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264281110-en.
[194] OECD (2016), Skills Matter: Further Results from the Survey of Adult Skills, OECD Skills Studies, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264258051-en.
[51] OECD (2016), The Survey of Adult Skills: Reader’s Companion, Second Edition, OECD Skills Studies, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264258075-en.
[122] OECD (2013), “Mathematics Framework”, in PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264190511-3-en.
[161] OECD (2013), OECD Skills Outlook 2013: First Results from the Survey of Adult Skills, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264204256-en.
[49] OECD (2005), The Definition and Selection of Key Competencies. Executive Summary., https://www.oecd.org/pisa/35070367.pdf (accessed on 31st October 2016).
[36] OECD/Statistics Canada (2005), Learning a Living: First Results of the Adult Literacy and Life Skills Survey, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264010390-en.
[14] Ontario Ministry of Education (2006), Number Sense and Numeration, Grades 4 to 6, Ontario Department of Education, Toronto, http://www.eworkshop.on.ca/edu/resources/guides/NSN_vol_1_Big_Ideas.pdf.
[137] Palm, T. (2009), “Theory of authentic task situations”, in Verschaffel, L. et al. (eds.), Words and Worlds: Modelling Verbal Descriptions of Situations, Sense Publishers, Rotterdam.
[136] Palm, T. (2008), “Impact of authenticity on sense making in word problem solving”, Educational Studies in Mathematics, Vol. 67/1, pp. 37-58, http://dx.doi.org/10.1007/s10649-007-9083-3.
[140] Palm, T. (2008), “Performance assessment and authentic assessment: A conceptual analysis of the literature”, Practical Assessment, Research, and Evaluation, Article 4, https://doi.org/10.7275/0qpc-ws45.
[135] Palm, T. (2006), “Word problems as simulations of real-world situations: a proposed framework”, For the Learning of Mathematics, Vol. 26/1, pp. 42–47.
[165] Parshall, C. et al. (2002), Practical Considerations in Computer-based Testing, Springer-Verlag, New York.
[66] Partnership for 21st Century Skills (2016), Framework for 21st Century Learning, http://www.p21.org/storage/documents/docs/P21_framework_0816.pdf.
[72] Paulos, J. (1995), A Mathematician Reads the Newspaper, BasicBooks, New York.
[71] Paulos, J. (1988), Innumeracy: Mathematical Illiteracy and its Consequences, Hill and Wang, New York.
[67] Pellegrino, J. and M. Hilton (eds.) (2012), Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century, The National Academies Press, Washington, DC, http://dx.doi.org/10.17226/13398.
[15] Peters, E. (2012), “Beyond comprehension: The role of numeracy in judgments and decisions”, Current Directions in Psychological Science, Vol. 21/1, pp. 31-35, http://dx.doi.org/10.1177/0963721411429960.
[2] PIAAC Numeracy Expert Group (2009), “PIAAC Numeracy: A Conceptual Framework”, OECD Education Working Papers, No. 35, OECD Publishing, Paris, https://dx.doi.org/10.1787/220337421165.
[182] PISA Mathematics Expert Group (2009), PISA CBAM Item Types, Australian Council for Educational research, Melbourne. (Unpublished manuscript).
[104] Presmeg, N. (2007), “The role of culture in teaching and learning mathematics”, in Lester, F. (ed.), Second Handbook of Research on Mathematics Teaching and Learning, Information Age Publishers, New York.
[86] PwC (2015), A Smart Move: Future-proofing Australia’s Workforce by Growing Skills in Science, Technology, Engineering and Maths (STEM), https://www.pwc.com.au/pdf/a-smart-move-pwc-stem-report-april-2015.pdf.
[78] Quality and Qualifications Ireland (QQI) (2016), General Learning P1GL0, http://qsearch.qqi.ie/WebPart/AwardDetails?awardCode=P1GL0.
[105] Resnick, L. (1987), “The 1987 Presidential Address: Learning in school and out”, Educational Researcher, Vol. 16/9, pp. 13-20, http://dx.doi.org/10.2307/1175725.
[106] Rogoff, B. and J. Lave (eds.) (1984), Everyday Cognition: Its Development in Social Context, Harvard University Press, Cambridge, MA.
[130] Roth, W. (2012), “The Work of Seeing Mathematically”, in Alternative Forms of Knowing (in) Mathematics, SensePublishers, Rotterdam, http://dx.doi.org/10.1007/978-94-6091-921-3_11.
[143] Rutherford, F. and A. Ahlgren (1990), Science for all Americans, Oxford University Press, New York.
[50] Rychen, D. (2004), “An overarching conceptual framework for assessing key competences in an international context. Lessons from an interdisciplinary and policy-oriented approach”, in Descy, P. and M. Tessaring (eds.), The Foundations of Evaluation and Impact Research. Third report on vocational training research in Europe: Background report, Office for Official Publications of the European Communities, Luxembourg, http://www.cedefop.europa.eu/files/BgR1_Rychen.pdf.
[193] Sabatini, J. and K. Bruce (2009), “PIAAC Reading Component: A Conceptual Framework”, OECD Education Working Papers, No. 33, OECD Publishing, Paris, https://dx.doi.org/10.1787/220367414132.
[109] Saxe, G. (1992), Culture and Cognitive Development: Studies in Mathematical Understanding, Lawrence Erlbaum Associates, Hillsdale, NJ.
[110] Saxe, G. et al. (1996), “Culture and children’s mathematical thinking”, in Sternberg, R. and T. Ben-Zeev (eds.), The Nature of Mathematical Thinking, Lawrence Erlbaum Associates, Hillsdale, NJ.
[107] Saxe, G. and M. Gearhart (eds.) (1988), Children’s Mathematics (pp. 71-88), Jossey-Bass, San Francisco.
[111] Schliemann, A. and N. Acioly (1989), “Mathematical knowledge developed at work: The contribution of practice versus the contribution of schooling”, Cognition and Instruction, Vol. 6/3, pp. 185-221, http://dx.doi.org/10.1207/s1532690xci0603_1.
[173] Schnotz, W. (2005), “An Integrated Model of Text and Picture Comprehension”, in The Cambridge Handbook of Multimedia Learning, Cambridge University Press, http://dx.doi.org/10.1017/cbo9780511816819.005.
[172] Schnotz, W. (2002), “Commentary: Towards an integrated view of learning from text and visual displays”, Educational Psychology Review, Vol. 14/1, pp. 101-120, http://dx.doi.org/10.1023/a:1013136727916.
[176] Schnotz, W. et al. (2010), “Creative thinking and problem solving with depictive and descriptive representations”, in Verschaffel, L. et al. (eds.), Use of Representations in Reasoning and Problem Solving - Analysis and Improvement, Routledge, London.
[174] Schnotz, W. and M. Bannert (2003), “Construction and interference in learning from multiple representation”, Learning and Instruction, Vol. 13/2, pp. 141-156, http://dx.doi.org/10.1016/s0959-4752(02)00017-8.
[175] Schnotz, W. and C. Kürschner (2007), “External and internal representations in the acquisition and use of knowledge: visualization effects on mental model construction”, Instructional Science, Vol. 36/3, pp. 175-190, http://dx.doi.org/10.1007/s11251-007-9029-2.
[70] Schwab, K. (2016), The Fourth Industrial Revolution: What it Means, How to Respond, http://www.weforum.org/agenda/2016/01/the-fourth-industrial-revolution-what-it-means-and-how-to-respond/.
[85] Secretary’s Commission on Achieving Necessary Skills (SCANS) (1991), What Work Requires of Schools: A SCANS Report for America 2000, U.S. Department. of Labor, Washington, DC.
[166] Shute, V. et al. (2016), “Advances in the science of assessment”, Educational Assessment, Vol. 21/1, pp. 34-59, http://dx.doi.org/10.1080/10627197.2015.1127752.
[12] Siemon, D. (2017), “Targeting ’big ideas’ in mathematics”, Teacher Magazine, http://www.teachermagazine.com.au/articles/targeting-big-ideas-in-mathematics.
[13] Siemon, D., J. Bleckly and D. Neal (2012), “Working with the Big Ideas in number and the Australian curriculum: Mathematics”, in Atweh, B. et al. (eds.), Engaging the Australian National Curriculum: Mathematics – Perspectives from the Field, http://www2.merga.net.au/sites/default/files/editor/books/1/Chapter%202%20Siemon.pdf.
[170] Sommerauer, P. and O. Müller (2014), “Augmented reality in informal learning environments: A field experiment in a mathematics exhibition”, Computers & Education, Vol. 79, pp. 59-68, http://dx.doi.org/10.1016/j.compedu.2014.07.013.
[138] Stacey, K. (2015), “The real world and the mathematical world”, in Stacey, K. and R. Turner (eds.), Assessing Mathematical Literacy: The PISA Experience, Springer, New York.
[118] Steen, L. (2004), “Data, shapes, symbols: Achieving balance in school mathematics”, in Madison, B. and L. Steen (eds.), Quantitative Literacy: Why Numeracy Matters for Schools and Colleges, Mathematical Association of America, Washington, DC, http://www.statlit.org/pdf/2003-Steen-QL-Data-Shapes-Symbols.pdf.
[11] Steen, L. (ed.) (1990), On the Shoulders of Giants: New Approaches to Numeracy, The National Academies Press, Washington, DC, http://www.nap.edu/openbook.php?isbn=0309042348.
[96] Straesser, R. (2015), “’Numeracy at work’: a discussion of terms and results from empirical studies”, ZDM Mathematics Education, Vol. 47/4, pp. 665-674, http://dx.doi.org/10.1007/s11858-015-0689-0.
[114] Straesser, R. (2003), “Mathematics at work: Adults and artefacts”, in Maasz, J. and W. Schloeglmann (eds.), Learning Mathematics to Live and Work in our World: Proceedings of the 10th International Conference on Adults Learning Mathematics (pp. 30-37), Johannes Kepler Universitat, Linz, https://www.alm-online.net/images/ALM/conferences/ALM10/proceedings/alm-03-proceedingsalm10.pdf.
[147] Sullivan, P. (2011), “Teaching mathematics: Using research-informed strategies”, Australian Education Review, Vol. 59, http://research.acer.edu.au/cgi/viewcontent.cgi?article=1022&context=aer.
[180] Sweller, J. (2010), “Element interactivity and intrinsic, extraneous, and germane cognitive load”, Educational Psychology Review, Vol. 22/2, pp. 123-138, http://dx.doi.org/10.1007/s10648-010-9128-5.
[179] Sweller, J. (2005), “Implications of cognitive load theory for multimedia learning”, in Mayer, R. (ed.), The Cambridge Handbook of Multimedia Learning, Cambridge University Press, http://dx.doi.org/10.1017/cbo9780511816819.003.
[79] Tertiary Education Commission (2008), Learning Progressions for Adult Numeracy, Wellington, https://ako.ac.nz/knowledge-centre/learning-progressions-for-adult-numeracy/.
[19] Thompson, P. (1995), “Notation, convention, and quantity in elementary mathematics”, in Sowder, J. and B. Schappelle (eds.), Providing a Foundation of Teaching Mathematics in the Middle Grades, Suny Press, Albany, NY.
[128] Tobias, S. (1993), Overcoming Math Anxiety, W. W. Norton & Company, New York.
[156] Tout, D. (2006), Review of Numeracy Component of PIAAC (Background paper prepared for the OECD Canada Expert Technical Workshop on Numeracy, Ottawa, November 10, 2006), Centre for Adult Education, Melbourne.
[1] Tout, D. et al. (2017), Review of the PIAAC Numeracy Assessment Framework: Final Report, Australian Council for Educational Research, Camberwell, https://research.acer.edu.au/transitions_misc/29.
[37] Tout, D. and I. Gal (2015), “Perspectives on numeracy: Reflections from international assessments”, ZDM, Vol. 47/4, pp. 691-706, http://dx.doi.org/10.1007/s11858-015-0672-9.
[141] Tout, D. et al. (2020), PIAAC Numeracy Task Complexity Schema: Factors that impact on item difficulty, Australian Council for Educational Research, Camberwell, http://dx.doi.org/10.37517/978-1-74286-609-3.
[38] Tout, D. and M. Schmitt (2002), “The inclusion of numeracy in adult basic education”, in Comings, J., B. Garner and C. Smith (eds.), The Annual Review of Adult Learning and Literacy: Volume 3, Jossey-Bass, San Francisco.
[183] Tout, D. and J. Spithill (2015), “The challenges and complexities of writing items to test mathematical literacy”, in Stacey, K. and R. Turner (eds.), Assessing Mathematical Literacy: The PISA Experience, Springer, New York.
[131] Turner, E. et al. (2009), “’Everything is math in the whole world’: Integrating critical and community knowledge in authentic mathematical investigations with elementary Latina/o students”, Mathematical Thinking and Learning, Vol. 11/3, pp. 136-157, http://dx.doi.org/10.1080/10986060903013382.
[192] Turner, R., W. Blum and M. Niss (2015), “Using competencies to explain mathematical item demand: A work in progress”, in Stacey, K. and R. Turner (eds.), Assessing Mathematical Literacy: The PISA Experience, Springer, New York.
[80] U.S. Department of Education (2013), College and Career Readiness Standards for Adult Education, Office of Vocational and Adult Education, US Department of Education, Washington, DC, https://lincs.ed.gov/publications/pdf/CCRStandardsAdultEd.pdf.
[196] UNESCO (2016), “Chapter 15: Literacy and Numeracy”, in Global Education Monitoring Report 2016. Education for People and Planet: Creating Sustainable Futures for All, United Nations Educational, Scientific and Cultural Organization, Paris, http://unesdoc.unesco.org/images/0024/002457/245752e.pdf.
[48] UNESCO (1997), International Standard Classification of Education: ISCED, United Nations Educational, Scientific and Cultural Organization, Paris, https://unesdoc.unesco.org/ark:/48223/pf0000146967.
[73] Utts, J. (2003), “What educated citizens should know about statistics and probability”, The American Statistician, Vol. 57/2, pp. 74-79, http://dx.doi.org/10.1198/0003130031630.
[121] van den Heuvel-Panhuizen, M. and K. Gravemeijer (1991), “Tests are not all that bad: An attempt to change the appearance of written tests in mathematics instruction at the primary school level”, in Streefland, L. (ed.), Realistic Mathematics Education in Primary School, Utrecht.
[181] van Gog, T., F. Paas and J. Sweller (2010), “Cognitive load theory: Advances in research on worked examples, animations, and cognitive load measurement”, Educational Psychology Review, Vol. 22/4, pp. 375-378, http://dx.doi.org/10.1007/s10648-010-9145-4.
[139] Verschaffel, L. et al. (2009), Words and Worlds: Modeling Verbal Descriptions of Situations, Brill | Sense, http://dx.doi.org/10.1163/9789087909383.
[82] Victorian Curriculum and Assessment Authority (VCAA) (2008), Curriculum Planning Guide: Literacy and Numeracy Skills Strand Numeracy Skills Units. First Edition, Victorian Curriculum and Assessment Authority, Victoria.
[68] Voogt, J. and N. Roblin (2012), “A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies”, Journal of Curriculum Studies, Vol. 44/3, pp. 299-321, http://dx.doi.org/10.1080/00220272.2012.668938.
[16] Wagner, D. and B. Davis (2010), “Feeling number: Grounding number sense in a sense of quantity”, Educational studies in Mathematics, Vol. 74/1, pp. 39-51.
[97] Wake, G. (2015), “Preparing for workplace numeracy: A modelling perspective”, ZDM Mathematics Education, Vol. 47/4, pp. 675-689, http://dx.doi.org/10.1007/s11858-015-0704-5.
[76] Watson, J. and R. Callingham (2003), “Statistical literacy: A complex hierarchical construct”, Statistics Education Research Journal, Vol. 2/2, pp. 3-46.
[116] Wedege, T. (2010), “People’s mathematics in working life: Why is it invisible?”, Adults Learning Mathematics, Vol. 5/1, pp. 89-97.
[115] Wedege, T. (2004), “Sociomathematics: Researching adults’ mathematics at work”, in Maasz, J. and W. Schloeglmann (eds.), Learning Mathematics to Live and Work in our World: Proceedings of the 10th International Conference on Adults Learning Mathematics in Strobl (Austria), 29th June to 2nd July 2003 (pp. 38-48), Johannes Kepler Universitat Linz.
[98] Weeks, K. et al. (2013), “Safety in Numbers 7: veni, vidi, duci: A grounded theory evaluation of nursing students’ medication dosage calculation problem-solving schemata construction”, Nurse Education in Practice, Vol. 13/2, pp. e78-e87, http://dx.doi.org/10.1016/j.nepr.2012.10.014.
[117] Williams, J. and G. Wake (2006), “Black boxes in workplace mathematics”, Educational Studies in Mathematics, Vol. 64/3, pp. 317-343, http://dx.doi.org/10.1007/s10649-006-9039-z.
[39] Willis, S. (1990), Being numerate: What counts?, Australian Council for Educational Research, Victoria.
[58] Windisch, H. (2015), “Adults with low literacy and numeracy skills: A literature review on policy intervention”, OECD Education Working Papers, No. 123, OECD Publishing, Paris, https://dx.doi.org/10.1787/5jrxnjdd3r5k-en.
[17] Yang, D., R. Reys and B. Reys (2009), “Number sense strategies used by pre-service teachers in Taiwan”, International Journal of Science and Mathematics Education, Vol. 7/2, pp. 383-403.
[133] Yasukawa, K., T. Brown and S. Black (2013), “Production workers’ literacy and numeracy practices: Using cultural-historical activity theory (CHAT) as an analytical tool”, Journal of Vocational Education & Training, Vol. 65/3, pp. 369-384, http://dx.doi.org/10.1080/13636820.2013.820214.
[52] Yasukawa, K. et al. (eds.) (2018), Numeracy as Social Practice: Global and Local Perspectives, Routledge, New York and London, http://dx.doi.org/10.4324/9781315269474.
[99] Zevenbergen, R. (2004), “Technologizing numeracy: Intergenerational differences in working mathematically in new times”, Educational Studies in Mathematics, Vol. 56/1, pp. 97-117.
← 1. The term “mathematical” is used here as inclusive of situations where statistical or probabilistic information may appear or where statistical thinking or statistical literacy are required as well. Such usage is made for brevity and convenience only. It is acknowledged that statistics is not a branch of mathematics, and that statistical reasoning and statistical literacy have unique elements, concepts and processes which are not mathematical in nature (Moore and Cobb, 2000[205]).
← 2. For more discussion and examples of the reading components tasks see OECD (2016[51]), The Survey of Adult Skills: Reader’s Companion, Second Edition.