Both the pilot and this follow-up asked computer scientists to rate AI’s capacity to answer the questions on PIAAC’s literacy and numeracy tests. AI’s likely performance on the tests was determined by looking at the majority expert opinion on each question. The use of standardised education tests enables the comparison to human capabilities, allows tracking AI progress across time and provides understandable AI measures. However, experts did not always agree in their evaluations. The study aimed at improving the methodology for eliciting expert knowledge on AI with standardised tests to address this challenge.

• According to experts, AI can answer around 80% of the PIAAC literacy questions. It can solve most of the easy questions, which typically involve locating information in short texts and identifying basic vocabulary. It can also master many of the harder questions, which require navigating across larger chunks of text to formulate responses. This evaluation rests on high consensus among experts.

• According to experts, AI can solve around two-thirds of the PIAAC numeracy test. However, there is disagreement behind this result. Some experts imagined narrow AI solutions for separate numeracy questions. Others considered general systems that can reason mathematically and process all kinds of numeracy questions similar to those in PIAAC. This led to diverging evaluations, with the latter experts giving lower ratings than the former.

• A comparison to the pilot assessment reveals considerable improvement in AI’s literacy capabilities since 2016. The expected success rate of AI in the literacy test has increased by 25 percentage points since then. This reflects the technological breakthroughs in natural language processing (NLP) in the period, related to the introduction of pre-trained language models, such as GPT.

• The discussion with experts suggested that numeracy capabilities of AI are unlikely to have changed much between 2016 and 2021. While formal mathematics underlying numeracy problems are easily automatable, extracting formal models from tasks that require general knowledge and are expressed in language and in images has received less research attention.

• Given recent technological advancements and the heavy investment and research in NLP, experts judged that AI’s literacy capabilities will continue to develop.

• More recently, large language models have been fine-tuned and applied for mathematical problems. The field has produced important benchmark tests as well as systems that perform well on these tests. These trends led experts to expect that AI will advance considerably in numeracy over the next few years.

• PIAAC assesses respondents’ proficiency in literacy and numeracy on several levels – from low (Level 1 and below) to high (Levels 4-5). Following the evaluation of experts, AI’s potential performance in literacy is close to that of adults with proficiency at Level 3. Across the OECD countries in PIAAC, on average, 90% of adults are at or below Level 3 in literacy and only 10% perform better than Level 3.

• The AI numeracy performance assessed by experts is close to that of adults at proficiency Level 2 on the easier and intermediate PIAAC questions, and similar to that of Level 3 adults on the harder questions. Across OECD countries with data, on average, 57% of adults are at or below Level 2 in numeracy, and 88% are at Level 3 or below that level.

• Despite its limitations, this study suggests that advancing AI capabilities with respect to literacy and numeracy may have important implications for employment and education. Most workers use these skills every day at work. At the same time, these skills have not improved in most countries in the last decades. By contrast, AI capabilities in literacy and numeracy are developing quickly.

• Across countries in PIAAC, on average, 59% of the workforce uses literacy skills daily at a proficiency comparable to or below that of computers. Between 27% and 44% of workers daily perform numeracy tasks at work, having numeracy proficiency below or at the level of AI. AI could affect the literacy- and numeracy-related tasks of these workers.

• Even the best-ranking countries to date cannot supply more than a quarter of their workforce with the literacy and numeracy skills needed to outperform AI. In this context, the focus of education may need to shift more towards teaching students to use AI systems to perform literacy and numeracy tasks more effectively.