This section presents the theoretical framework for the way in which PISA 2022 assesses the integration of information and communication technologies in teaching and learning (ICT). This framework provides a comprehensive strategy to document how students access and use ICT resources in and outside of school, and to identify how teachers, schools and education systems integrate ICT into pedagogical practices and learning environments. The framework allows for an exploration of how system-level factors influence schools’ and students’ experiences with ICT, how the availability and use of ICT interact with various teaching practices, and how these associations correlate with students’ performance in mathematics, reading and science, and with other outcomes, such as students’ ICT skills and well-being.
PISA 2022 Assessment and Analytical Framework
6. PISA 2022 ICT Framework
Abstract
Introduction
Why develop a framework to assess the integration of information and communication technologies in teaching and learning?
Information and communication technologies (ICT) play an increasingly important role in virtually all aspects of our daily lives. Not only is technology profoundly transforming people’s work and professional life, but it is also altering how people interact, communicate, retrieve and share information, and even how governments provide public services to citizens. ICT also significantly affect multiple facets of education. They can provide new opportunities for students to learn outside of school, and can change teachers’ pedagogical approaches and the learning experience of students in school. Moreover, education systems are increasingly embedding digital competencies in their curricula.
ICT are integrated into schools and learning in three major ways:
Students’ engagement with ICT (both in and outside of school) can affect their cognitive processes and their well-being, and eventually what they learn.
Teachers are increasingly using ICT for instruction, and administrative and communication purposes, with numerous implications for classroom management, instructional practices, pedagogical approaches and time use.
Competence in using ICT and digital literacy are being recognised as important skills that students need to acquire if they are to flourish in the digital age.
The increasing importance of digital technologies in education systems and the pressing need to equip students with digital competencies raise major policy concerns for governments: To what extent should students use ICT in and outside of school, and how should they engage with these technologies? Are ICT used for learning, for social networking or for entertainment? How and to what extent do teachers of different kinds use ICT and for what objectives? What role do ICT play in different types of pedagogical and instructional practices? Which practices work best with ICT? What are the implications of the different types of ICT use for students’ proficiency in mathematics, reading and science? Do certain types of ICT use affect students’ well-being? What should governments do to ensure that young people today and tomorrow are sufficiently skilled in the use of ICT to flourish in this digital century?
Despite the growing body of literature focusing on the relationships between students’ engagement with ICT and education outcomes, there is no consensus on the contribution ICTs make to students’ educational attainment or cognitive performance in general. Although there is little doubt that coming generations are more likely to have the ability to engage with the latest ICT, one should not take for granted that everyone will have access to ICT resources, or that they will use ICT in ways that are responsible and beneficial to them (i.e. that contribute to their personal development and well-being) and to society.
Moreover, although some studies have documented students’ access to and use of ICT resources at home and in school across countries (e.g. (Fraillon, Schulz and Ainley, 2013[1])), much remains to be investigated regarding the influence of ICT availability, quality and use on students’ academic, and social and emotional outcomes.
This framework explores these questions in the context of the Programme for International Student Assessment (PISA). In all previous rounds, starting in 2000, PISA documented various dimensions of access to and use of ICT by 15-year-old students in and outside of school. For example, in PISA 2000 students were asked whether a computer was available to them in different locations. From 2009 onwards, PISA documented the types of ICT resources available to students at home and in school separately. Depending on the PISA rounds, students’ ICT use in school or at home, and their attitudes towards ICT were also documented (OECD, 2013[2]; 2016[3]; 2017[4]).
Yet, ICT questionnaires in previous PISA studies were developed in an ad-hoc way, without a comprehensive ICT assessment framework. This resulted in a number of shortcomings. For example, questionnaires covered mainly hardware and access to the Internet while software and digital learning resources were covered to a lesser extent. The quality and accessibility of these resources were not systematically documented; and, more important, the use of ICT resources was only partially documented, with limited coverage of teachers’ pedagogical practices related to ICT.
Objectives of the PISA 2022 ICT Framework
This framework provides a comprehensive strategy to document how students access and use ICT resources in and outside of school, and to identify how teachers, schools and education systems integrate ICT into pedagogical practices and learning environments. The framework allows for an exploration of how system-level factors influence schools’ and students’ experiences with ICT, how the availability and use of ICT interact with various teaching practices, and how these associations correlate with students’ performance in mathematics, reading and science, and with other outcomes, such as students’ ICT skills and well-being.
By leveraging PISA’s wide coverage of constructs and countries, the framework contributes substantially towards filling the knowledge gap in this field. This framework will guide the development and integration of ICT-related questions into background questionnaires for the PISA 2022 cycle. As such, it will govern data collection on key dimensions of ICT availability and use in and outside of school in more than 50 countries. In addition, the questionnaire will focus on different school actors, such as students, teachers and principals, and on system-level variations in policies across countries.
A critical objective of the PISA ICT questionnaire is to answer a variety of policy questions. Using data collected through this questionnaire, countries should be able to obtain an accurate picture of their respective situation – notably through between- and within-country comparisons – in terms of access to and use of ICT resources by 15-year-old students in and outside of school. Additional important policy questions that this questionnaire aims to answer include: What are the main determinants of and obstacles to using ICT for teaching and learning in schools? How does using ICT for teaching interact with pedagogical practices, and does it relate to students’ achievement in mathematics, reading and science? What kinds of digital learning materials, professional development initiatives and teaching approaches should be supported? How the integration of ICT in schools articulates with equity issues both in terms of access to and practices with ICT resources? How do students use ICT outside of school, and is it related to their cognitive achievement and well-being?
This ICT assessment framework covers three major dimensions:
access to ICT, which encompasses availability, accessibility and quality of ICT resources with a special focus on (connected) technologies that can support learning (e.g. digital learning resources, learning management systems, etc.);
use of ICT, which covers the intensity as well as the types and modalities of ICT use by students in an informal, and possibly unsupervised, environment for learning and leisure, and in a supervised situation in the classroom, notably through teachers’ pedagogical practices with ICT;1 it also includes alternative uses of ICT by teachers to support teaching; and
students’ ICT competencies, which describe the core competency areas identified in existing assessment frameworks for “digital literacy” as well as attitudes and dispositions towards ICT use (for learning and for leisure). A self-efficacy measure is proposed to assess students’ ICT competencies.
The framework is organised in five parts: (i) overall conceptual framework; (ii) description of the system-level factors affecting both access to and use of ICT; (iii) approach to explore access to ICT resources; (iv) examine the variety of ICT uses; and (iv) conclusion by presenting PISA approaches to assess students’ outcomes and by proposing a strategy to measure students’ competencies in ICT.
Overall conceptual framework guiding the assessment of students’ interaction with ICT in PISA 2022
Overall framework
At the heart of the PISA 2022 ICT framework is the relationship between two major dimensions of ICT – access and use – and students’ outcomes (cognitive performance, well-being, and ICT attitudes and competencies). However, the framework also aims to identify how these links depends on contextual factors or background characteristics, and on existing policies and practices related to ICT. Figure 6.1 provides an overview of the underlying logical framework used to elaborate the PISA 2022 ICT framework.
Students’ use of ICT resources is conditional upon the availability, accessibility and quality of those resources. Conversely, the amount and type of ICT resources made available to students is also influenced by how and why ICT are used. Documenting access to ICT therefore aims to answer the following policy question: To what extent is student engagement with ICT determined by the availability of diverse and functional ICT resources? Consequently, the use of ICT is considered as a second step – a logical continuation. As such, the assessment of ICT use aims to answer the following question: Given the available ICT resources, how are students’ uses of different kinds of ICT related to teaching practices and to students’ cognitive performance, well-being and ICT skills?
Although this framework recognises the diverse ways in which ICT are used in school, its prime interest lies in documenting students’ use of ICT. Yet, ICT are integrated in schools without necessarily involving students’ use of them. For example, school principals can use ICT to administer and manage financial and educational resources; and teaching staff can rely on ICT to improve overall instruction, identify and monitor students’ strengths and weaknesses, or communicate with parents. Since these practices are likely to affect students’ experiences, they are covered by this framework.
Nevertheless, the structure and scope of PISA better fits a thorough examination of the use of ICT at the student level rather than at the school or teacher level. Indeed, optional questionnaires for teachers were distributed in only 17 of the countries/economies that participated in PISA in 2015; and students cannot be matched with a specific teacher, as the sampling of teachers occurs at the school level. Thus information based on teachers’ reports only contributes to between-school analyses, and student-level information is needed to capture within-school variations in students’ ICT use.
The framework also acknowledges the influence of contextual factors, and policies and practices, on both access to and use of ICT resources, and on students’ outcomes. Contextual factors include the general background characteristics of the education system, schools and students’ households. They include, for example, the level of economic development of a country; students’ grade level in secondary school; the integration of ICT literacy in the curricula; whether the school is public or private; the socio-economic and cultural background of students and parents; and even teachers’ qualifications. These elements are not specific to ICT, and overlap with the information found in the PISA background questionnaires at the student, parent, teacher, school and system levels. Although not directly related, these factors are likely to shape and constrain the degree of access to ICT resources, and how they are used. They are also likely to affect the relationship between access to and use of ICT, on the one hand, and student outcomes, on the other.
In addition, specific ICT-related policies and practices could directly influence access to and use of ICT resources. Such policies include, for example, the existence of specific funding for ICT resources in schools, principals’ attitudes towards ICT use as an instructional tool, and guidelines and support for teachers in using ICT in the classroom. These policies and practices could also be developed as a response to students’ cognitive performance or their attitudes towards ICT.
This logical framework applies to students’ engagement with ICT both in and outside of the classroom, and covers a variety of stakeholders, including students, teachers, principals and parents.2 This does not imply that students’ access to and use of ICT is conceived in an entirely similar way in both contexts. Indeed, there are crucial differences between having a teacher present as an intermediary to students’ engagement with ICT in the classroom and using ICT for leisure or learning purposes outside the classroom. For example, teachers can select relevant digital resources ex ante, explain how to use them efficiently and ensure students are focusing on the learning tasks. Yet, the specificities of ICT use in and outside of the classroom fit into the broader conceptual approach presented in Figure 6.1. The particularities of both contexts are detailed thereafter.
As mentioned earlier, the PISA 2022 ICT framework examines the relationship between students’ use of ICT and three outcomes: students’ cognitive achievement, well-being and their level of ICT competencies. The assessment of students’ cognitive achievement in mathematics, reading and science and students’ well-being is not specific to this framework; it relies on the respective frameworks dedicated to each outcome during previous PISA cycles. By contrast, the PISA 2022 ICT framework develops a specific strategy for documenting students’ ICT competencies, which are regarded as including both knowledge and skills related to ICT use and attitudes toward ICT. The framework includes ICT skills that were identified using existing ICT competence frameworks (e.g. (Fraillon et al., 2015[5]; Carretero, Vuorikari and Punie, 2017[6]; Fraillon, Schulz and Ainley, 2013[1])). The objective is to provide directions for the development of a proper assessment of ICT literacy in future PISA cycles.
However, the assessment of ICT competencies in PISA 2022 will not rely on a test, as is the case with mathematics, science and reading. Instead, it will rely on students’ self-reported attitudes and self-efficacy measures regarding ICT use for learning and leisure. Assessments of both students’ attitudes and self-efficacy will build on similar measures developed in previous PISA cycles in relation to the major domain being tested. Particular attention will be given to ensure the validity of the self-efficacy measure, which can be challenging, as seen in previous assessments. Indeed, students’ confidence in performing advanced ICT tasks is weakly associated with Computer and Information Literacy (CIL) achievement scores as measured in the International Computer and Information Literacy Study (ICILS) test (Fraillon et al., 2014[7]). Ensuring the validity of the self-efficacy measure requires the coverage of a wide array of tasks pertaining to different dimensions of digital competencies (by including technical skills as well as skills related to communication and information literacy among others).
Figure 6.1. PISA 2022 ICT conceptual framework
Source: Authors
Relationship between access to and use of ICT resources
As shown in Figure 6.1, the availability, accessibility and quality of ICT resources partly shape teachers’ and students’ practices with ICT, both in and outside of the classroom. Indeed, the total amount of ICT equipment available per student is likely to affect decisions on whether and how to use ICT resources. One could imagine that having fewer than one computer per student at school would mean that students use computers in group exercises, for example. Similarly, the ease with which ICT can be accessed during class could affect the work arrangements and frequency of use.
In addition, the quality of ICT resources for learning – encompassing dimensions as diverse as technical capacity and performance, teacher (or pedagogical) and student usability (e.g. ergonomics and ease of use), practicability and adaptability – would likely affect the range and relevance of the activities that could be conducted with the available ICT equipment. For instance, slow Internet connections would prevent students from using demanding online digital learning resources, while students working on poorly maintained computers would likely encounter software compatibility or obsolescence issues. In addition, an educational software could be accessible and engaging to students but not flexible enough to fit a teacher’s pedagogical approach or not in line with the curriculum. Access to and use of ICT outside of school for learning are vulnerable to similar constraints. However, the assessment of the quality of ICT resources for other purposes, such as leisure, has to rely on different measures, although some of the aspects mentioned above could still be relevant.
Conversely, students’ and teachers’ use of ICT can also affect decisions about the selection of and attention devoted to ICT resources. Indeed, the extent to which students use ICT resources for learning science or mathematics could guide the selection of specific software and hardware requirements, and mandate a certain level of Internet bandwidth, for example. Similarly, if teachers rely on ICT mainly for personalising the pace of students’ learning, for obtaining instant feedback or for collaborative group exercises, that could affect their school’s decision about whether to purchase a computer for each student, to invest in individualised learning software, or instead to invest in online collaborative games and Intranet installations.
Moreover, teachers’ attitudes towards the use of ICT as a tool for instruction are likely to influence the amount and types of resources they use in class and available in schools. A similar relationship applies for students’ use of ICT outside of the classroom. Indeed, students’ use of ICT for leisure could be affected by their (and their parents’) attitudes and practices. For example, parents aware of the opportunities and risks of ICT use could encourage their children to play educational or collaborative video games or simply limit the access to the game console or computer.
The strength of the inter-dependence between access to and use of ICT resources, and the relative importance of each of the effects described above, strongly depend on how responsibilities are shared across the different levels of the education system. In a centralised system, access to ICT resources may be almost entirely determined at the system level, with a limited relationship to actual ICT use at the school level. By contrast, if the school enjoys more autonomy in acquiring educational resources and encouraging teaching practices using ICT, teachers may be more involved in selecting the equipment they need to pursue their pedagogical strategies. Including ICT skills in the curriculum or adhering to security regulations may also affect schools’ and teachers’ ability to use and provide access to ICT resources. Indeed, the curriculum might favour the use of particular ICT resources while security regulations might prevent schools and teachers from conducting specific activities.
ICT use in the classroom
The conceptual framework can be further refined by describing how ICT use by students and teachers in the classroom can influence student outcomes. The aim is to postulate hypotheses that will serve as a basis for investigation. This section details three main ways that ICT use in school may be related to student outcomes (Figure 6.2).
First, students can use ICT resources to learn a traditional subject, such as mathematics, reading or science. ICT-assisted instruction can affect students’ cognitive performance (and other outcomes) through its interaction with teaching strategies and students’ engagement with learning. Teachers’ instructional practices prior to ICT integration into teaching are likely to affect how ICT is used in the classroom. Conversely, the integration of ICT can also change the use and modalities of teaching strategies, pedagogical practices and classroom arrangements (including teacher or student-centred instruction, traditional or enquiry-based teaching, and assessment and feedback practices for pedagogical purposes). Thus, the relative weight and overall combination of the different teaching practices (i.e. teacher-directed instruction, student-focused instruction, teacher support, and feedback and adaptive teaching) are expected both to guide how ICT are used and to vary with the use of ICT in the classroom.
Moreover, teachers’ use of ICT can affect subject-specific teaching strategies, such as using simulations in mathematics. Using ICT as a tool to learn a specific subject can also affect students’ engagement with learning, as manifested in the time spent receiving instruction, and students’ concentration on, efforts in and attitudes towards the subject. Teachers may introduce more ICT tools into classrooms precisely in order to attract students’ attention.
The practical use of ICT for learning also entails challenges. Teaching practices that rely on ICT require teachers to have specific skills and know-how about the learning process and classroom management with digital tools. Integrating digital resources in teaching could also initially require a substantial investment of time and effort in preparing course materials, classroom management and monitoring the success of the teaching approach. Moreover, the use of ICT for different activities could make it more challenging to maintain a quiet, peaceful and respectful classroom in which students can concentrate on academic tasks, and listen to the teacher and other students. Another related challenge is the potential misuse of ICT by students who could spend a substantial amount of their time in class on social networks, games and other distracting activities.
Second, students can benefit from specific teaching about ICT – either in a dedicated course or time period – aimed at improving ICT-related competencies. The benefits for students will depend on the availability of specific ICT educational resources. In particular, students could benefit from ICT teachers or teachers with particular qualifications for delivering ICT courses. They can also have access to learning resources specifically designed to develop ICT competencies.
In this regard, more and more education systems are integrating computational thinking (i.e. “a problem-solving methodology that expands the realm of computer science into all disciplines, providing a distinct means of analysing and developing solutions to problems that can be solved computationally”, ACM et al., 2016) into secondary education curricula, either across subjects or as a stand-alone course. This is often accompanied by in-service training for teachers, and the use of specific instructional practices and tools in the classroom, including “unplugged” activities, simulations and using coding software (Bocconi et al., 2016). Students’ activities with ICT outside of school could also be more directly related to certain ICT competencies, such as handling troubleshooting and security issues or engaging in diverse use of social media.
In addition to the expected direct effect on students’ ICT competencies and attitudes, acquiring ICT skills can make it easier for students to use ICT to learn other subjects. Specific instruction in ICT can also raise students’ awareness about the potential effects of ICT use on their well-being. But investing time and effort in teaching ICT skills could take resources away from instruction in other subjects and thus have a negative effect on students’ cognitive performance.
Third, ICT can be used to support teachers’ activities outside of the classroom. For instance, teachers can use ICT to communicate with colleagues, and to contribute to the development of overarching and transversal pedagogical practices. ICT can also be used to communicate with parents and create online communities, bringing together teachers, parents and students. Teachers can also rely on ICT to plan their courses, share materials with students, and keep track of students’ work and performance. Overall, ICT can simplify administrative tasks and processes, thereby freeing up time for more meaningful activities that could benefit students’ performance and well-being.
Figure 6.2. Detailing ICT use in school
Source: Authors
ICT use outside the classroom
Education policies can affect only part of students’ use of ICT outside of the classroom. Using ICT outside of school depends on factors as diverse as students’ socio-economic status, local Internet and broadband coverage, and the price of ICT resources. ICT activities are also affected by parents’ attitudes towards ICT use at home, the availability of digital learning resources at school, and national policies regarding the safe use of the Internet.
Unlike using ICT in the classroom, which is assumed to be solely for learning, students use ICT outside of the classroom both for learning – either for completing homework assignments or for self-motivated learning activities – and for leisure (Figure 6.3). Although these modes of ICT use are possibly connected, each has a specific relationship with cognitive performance, well-being and ICT competencies (Figure 6.3).
Students use ICT outside of the classroom in a variety of situations, and the type of equipment and the purpose of ICT use are likely to vary substantially, depending on the situation. For example, students can use ICT resources available in their school during their free time to complete their homework, use the family computer or their own device during the weekend to browse the Internet or play video games at a friend’s home after school.
An important distinction between ICT use in the classroom and outside of school is that the latter usually takes place in an unsupervised environment, although ICT use at home could be supervised, to some degree, by a household member. However, the increasing use of digital resources for home assignments and project-based learning activities tend to blur the boundaries between school and out-of-school activities. Indeed, many ICT tools offer teachers the ability to monitor students’ activities on line, whether they take place in or outside of school. As in previous PISA cycles, this framework distinguishes between students’ use of ICT during school days and during weekends, and documents whether the use of ICT takes place at home or in another location.
Students’ use of ICT resources for learning outside of the classroom may have significant consequences for their engagement and success in school. Several important aspects of students’ use of ICT for learning outside of the classroom are likely to be related to students’ outcomes.
ICT may have an effect on students’ engagement with their homework. Students who feel positive and at ease with ICT in any situation may spend more time on, invest greater effort in, and have more favourable attitudes towards homework if they can use ICT to complete their assignments. By contrast, students who have no or limited access to ICT resources and do not feel comfortable using them for learning may give up on homework more easily if it requires the use of ICT. Moreover, students who use ICT for completing homework assignments at home could be distracted by the many ways ICT can be used for leisure instead.
The effect of ICT use not only on students’ motivation but also on learning is likely to vary with the type of homework assigned and whether it was designed for ICT. In particular, the development of innovative pedagogical approaches, such as project-based learning facilitated by technology-supported collaboration tools, enquiry-based learning, the use of simulation tools or online laboratories, might facilitate learning by doing, foster students’ engagement and motivation, and help students develop problem-solving skills by putting students in various novel situations and encouraging them to adopt different perspectives (OECD, 2016[8]).
Second, using ICT for learning could help parents and teachers assess their children’s/students’ efforts and strengths, and monitor and address any problems that may arise. Eventually, this could lead to better diagnoses and greater parental engagement, both of which can have positive effects on students’ cognitive performance. Again, the effect will depend on parents’ and teachers’ attitudes and competencies regarding ICT; negative attitudes could widen the digital divide.
Third, students can also use ICT for learning in a variety of contexts unrelated to homework. Students might conduct research via the Internet to learn about specific topics of interest. They may also use digital learning resources because their parents or friends advised them that it could improve their skills in a specific subject. In addition, the boundaries between ICT use for learning and for leisure might become less clear with the development of educational gaming, and with new types of video games that include collaboration, enquiry, problem solving, and strategic components and activities. As a result, learning with ICT may more often be seen as a leisure activity for students with potential benefits for students’ cognitive performance, well-being and ICT competencies.
Nevertheless, students are more likely to use ICT outside of the classroom for leisure (European Commission, 2013[9]). When they do, students have many opportunities to develop certain cognitive processes and skills, but they also face certain risks. These risks and opportunities are not related to the use of a specific device or software. Indeed, there are both opportunities and risks no matter whether students use ICT for gaming, accessing social media, browsing the Internet for entertainment or searching for information.
By using ICT for leisure, students can develop specific ICT competencies, including technical skills, such as troubleshooting and understanding security settings, as well as the ability to access, analyse and share information, communicate and create content. There are also a variety of opportunities to develop problem-solving and collaborative skills through, for example, connected, multi-players games, and also more basic skills, such as numeracy, reading and writing. ICT use, notably through social network and digital community interactions, also offer the possibility to develop social and global competencies (OECD, 2018[10]).3 Moreover, students who do not use ICT may miss opportunities to develop social ties with other students and risk to suffer greater loneliness or exclusion (OECD, 2015[11]).
But students face some risks when using ICT for leisure. The potential lack of self-control combined with the curiosity of adolescent ICT users may lead to overuse and even addiction problems, which could have serious adverse physical, social, psychological and cognitive effects. Students’ security and safety could also be at risk, as they could be exposed to cyberbullying or to inappropriate (e.g. violent, pornographic) content (OECD, 2015[11]). Adult guardians can thus play an important role in supervising students’ use of ICT while well-designed learning experiences can encourage healthy, responsible and constructive use of ICT.
Figure 6.3. Detailing ICT use outside the classroom
Source: Authors
Country- and system-level factors related to access and use of ICT resources
Although digitalisation is progressing at an impressive rate around the globe, there are substantial differences in ICT access and quality across countries, regions and education systems (OECD, 2015[11]; Fraillon, Schulz and Ainley, 2013[1]; Korte and Hüsing, 2006[12]). For example, as shown in Figure 6.4, the number of fixed broadband subscriptions per 100 inhabitants varies significantly across OECD countries – from more than 40 per 100 inhabitants in Denmark and Switzerland, among others, to fewer than 15 per 100 inhabitants in Mexico and Türkiye. In terms of quality, more than three in four connections in Japan and Korea are fibre connections, while fewer than one in four are, on average, across most other OECD countries, and in Austria and Germany, only one in 50 are fibre connections (OECD, 2018[10]).
These country-level differences could be attributed to variations in contexts and policies. The level of economic development, the structure of a country’s economy (especially the development of ICT-enabled services to citizens and businesses), the level of investment in new technologies and the demand for ICT skills determine the availability of ICT resources. In addition, the policies governing the education system and, in particular, the weight given to integrating ICT into learning, influence the availability of ICT in schools.
Figure 6.4. Fixed broadband subscriptions per 100 inhabitants, 2017
Note: Australia: Satellite data are not available for publication.
Canada: Fixed wireless includes satellite.
France: Cable includes VDSL2 THD.
Germany: Cable includes HFC lines; fibre includes fibre lines provided by cable operators; fixed wireless includes BWA subscribers; other includes leased lines.
Israel, Luxembourg, Switzerland and the United States: Data are estimates.
United Kingdom: Terrestrial fixed wireless figures are unavailable. Satellite figures are estimates.
Source: OECD, Broadband Portal, www.oecd.org/sti/broadband/oecdbroadbandportal.htm
Since one goal of the PISA 2022 ICT Framework is to document students’ ICT experience, it is crucial to document contextual factors at the country and system levels that affect the availability of ICT resources for the population as a whole, including in the education system and for learning, in general, but also for leisure. In addition to contextual factors, national practices and policies are also likely to shape students’ access to and use of ICT resources.
Documenting country-level factors feeds into the analysis in different ways. First, country-level factors provide a useful context for understanding the dynamics of ICT availability and use in within a country; they also facilitate comparisons across countries. Second, those factors offer insights into the quantity and quality of ICT resources available to students, and can be used as a benchmark against which the data collected through PISA can be compared.
Country-level factors affect students’ access to and use of ICT resources in two major ways. They contribute to the overall availability of ICT infrastructure in the country (and at the subnational level), including the availability of ICT for learning-related purposes. This affects the availability of ICT resources at school and for learning outside the classroom. In addition, education system-level policies and practices have an impact on the access and maintenance of ICT in schools, and shape students’ and teachers’ use of ICT resources. Such system-level factors include teacher support for using ICT, qualifications requirements for teachers to use ICT, references to ICT competencies in the curriculum, guidelines regarding specific teaching practices with ICT, and security and safety regulations.
System-level factors are documented from different sources. Information related to contextual factors and the overall availability of ICT are drawn from existing surveys conducted by the OECD, including PISA system-level data collection, the Survey of Adult Skills (a product of the Programme for the International Assessment of Adult Competencies [PIAAC]), and indicators from the OECD Directorate for Science, Technology and Innovation and international organisations (e.g. the World Bank’s world development indicators, the ICT Development Index [IDI] developed by the International Telecommunications Union, etc.). Practices and policies, notably regarding the regulations of the education system, will be collected through a new PISA system-level questionnaire addressed to the countries and economies participating in PISA.
Contextual factors affecting the overall availability of ICT resources
Overall access to ICT resources
The availability of ICT resources in a country can be assessed by examining various ICT-related country-level indicators. Relevant indicators at the country level include: the availability of ICT infrastructure; the affordability of ICT resources; use of ICT by the population and government; the quality of ICT resources; inequalities in access; and the demand for ICT skills in the labour market. Several dimensions of ICT development or readiness (which refers to the propensity for countries and economies to exploit the opportunities offered by ICT) at the country level are also key indicators.
The availability of ICT infrastructure and the quality of ICT resources can be documented by collecting information on mobile network coverage, Internet bandwidth and the types of broadband connection (e.g. digital subscriber line [DSL], cable, fibre, etc.). This may be complemented by information regarding public investment in ICT, the weight of the ICT sector in the economy and the availability of the latest technologies.
Use of ICT can be documented through information on the number of mobile phone subscriptions per inhabitant, the share of individuals using the Internet, the share of households with a personal computer and Internet access, and the use of virtual social networks, among other indicators.
The distribution of ICT infrastructure and ICT use can be recorded across population subgroups (for example, gender, wealth, region, immigrant background, etc.) or through an inequality index, to shed light on inequalities in ICT access and use within a country.
The demand for ICT skills in the labour market is also an interesting measure as it indicates the extent to which developing such skills are rewarded. The share of individuals using generic ICT skills (i.e. for communicating and for searching for information) daily at work varies significantly across countries, ranging from 64% in Norway to 33% in the Slovak Republic in 2014, among the countries that participated in the Survey of Adult Skills (PIAAC) (OECD, 2016[8]).
The aforementioned IDI and the Network Readiness Index (NRI) measure the capacity of countries to leverage ICT for increased competitiveness and well-being; thus, they provide information on other dimensions of ICT, such as the political and regulatory environment, that also helps define the ICT country context.
Access to ICT resources for learning
Documenting the availability of ICT resources supporting learning would also be of great value for characterising the environment in which students and teachers use ICT both in and outside of school. Several dimensions could be covered:
Information regarding private and public expenditures on education-related ICT would contribute to the country profile in terms of ICT use for learning outside of school. This could be complemented by information regarding the existence of (and amount distributed under) programmes that provide financial support to households for buying education-related ICT equipment, which were found in a third of European countries in 2009 (EACEA, 2011[13]).
A description of the wealth (or eco-system) of available ICT resources to support learning would also be useful. One could distinguish between initiatives developed by the country or at the subnational level for all citizens, and the resources developed within the education system and for education stakeholders. In examining the purpose and type of ICT resources, it is important to consider digital learning resources and material (e.g. digital library), digital learning platforms, where learners can track their status as they progress, learning management systems and portals, which enable learners to engage in collaborative projects and communicate with peers and teachers (e.g. file sharing, virtual group rooms, chat, blog and Wiki, local knowledge portals, calendars, specific educational software and digital laboratories, etc.) and ICT-supported administrative resources mainly for administrative and teaching staff to track and monitor students and communicate with parents.
Policy environment governing ICT access and use in education
By and large, education systems acknowledge the central role of ICT in education, but their governance and regulatory frameworks differ substantially, as do their efforts and abilities to progressively adopt ICT in schools. It is critical to understand the policies, regulations and guidelines that determine the direction and evolution of ICT availability and use in an education system in order to understand the differences in ICT environments across countries. Classifying policies and practices by the extent to which they encourage investment in ICT, set rigorous or permissive rules, and provide support to various stakeholders for accessing and maintaining quality ICT resources can help describe the ICT environment students face in school.
The regulatory framework regarding the quantity and quality of ICT resources in schools
Another indicator of the ways in which ICT are integrated in education systems is the existence of a national (or subnational) strategy aimed at guiding ICT use in education. Although countries may take different approaches with regard to ICT in education, such strategies are common (all European countries have one) and therefore seem to be a promising source of information on regulations and guidelines regarding ICT in education (EACEA, 2011[13]). System-level indicators could be collected via the PISA system-level questionnaire and should cover the following dimensions:
Quantity, accessibility and quality: This includes the nature (whether it is legally binding, a measure or a guideline, for example) and the content of rules specifying the ICT resources that schools are entitled to, according to some criteria, such as the number of enrolled students. Whether the school or a higher-level institution is responsible for purchasing and maintaining ICT resources should also be documented. Accessibility also refers to the regulations regarding students’ access to ICT resources, including recommendations on time exposure to specific ICT resources, security and safety guidelines, location and disposition of ICT resources in schools and the required degree of supervision. Quality includes conventions regarding the desirable standards of the ICT equipment and the frequency with which they are maintained and renewed.
Financial resources: A good starting point from which to examine education systems’ relative positions would be to document expenditures on education-related ICT resources. Similar to the measure of cumulative expenditure per student between the ages of 6 and 15 collected in PISA, a measure of cumulative expenditure on ICT resources per student would provide a synthetic estimate of ICT access throughout 15-year-old students’ education. In addition, documenting the rules, recommendations and administrative processes guiding the allocation of funding to ICT resources (including the level at which decisions are made, the degree of autonomy schools enjoy and whether budgetary items are constrained) would provide important insights into the overall ICT education environment in schools.
Human resources: This includes the qualification requirements for teachers in terms of ICT competencies and using ICT to teach, regulations and guidelines regarding the availability of an ICT co-ordinator or support system in school, and information about the overall share of teachers with a specific ICT (for teaching) qualification. In addition, such information would cover the availability, need for and provision of different types of training, including continuing professional development training aimed at building teachers’ skills regarding ICT use for educational purposes (either in general, for a specific subject, or related to imparting ICT competencies to students), remedial training to develop ICT skills, or training targeted at principals for the deployment of ICT in schools.
ICT-related pedagogical resources: The availability of ICT resources specifically designed for education purposes is a critical component of ICT resources. In particular, the diffusion to teachers and continuous development of pedagogical resources related to ICT use for educational purposes should be documented. The availability and breadth of a central repository for pedagogical resources, the availability of human resources regarding ICT, the development of exchange platforms for teachers to share their experiences and good practices, and the existence of partnerships with researchers and developers of pedagogical resources can be documented.
Policies and guidelines framing the use of ICT resources by students and teachers in school
The governance of the education system regarding the use of ICT resources in school can vary widely across countries, and even across regions within countries. Information on the overall regulatory environment and guidelines framing teachers’ pedagogical practices with ICT and students’ use, include:
The degree to which ICT are integrated into the curriculum at different education levels. This refers to both the presence of specific ICT skills to be transmitted to students as part of the curriculum of various subjects, and reference to the use of ICT as a means of helping students acquire knowledge and skills in these subjects. Of particular interest are the inclusion of ICT as a tool for acquiring mathematical skills and the explicit reference to ICT competencies, such as computational thinking, within the mathematics curriculum.
Policies, regulations and guidelines for ICT use in the classroom. These encompass information regarding the degree of autonomy schools (and teachers) have regarding the use of ICT resources. In particular, it should document whether specific restrictions apply when using ICT resources in class, such as a requirement to obtain permission from the legal guardian or principal, the need to supervise the students, restricted access to ICT functionalities and the Internet, or limitations to the amount of time students can spend using ICT resources.
Incentives for or barriers to using ICT for teaching in class. These refer to existing financial or career incentives teachers may or may not receive for using ICT; recommendations available in official documents; and the existence of strict conditions under which ICT should not be used, notably with regard to equity issues in the classroom (e.g. restrictions on the use of ICT if some students do not have ICT resources at home or lack the basic ICT skills necessary to use them).
Evaluation and assessment of ICT competencies and ICT use for teaching. These include determining the existence of mandatory examinations of students’ ICT competencies, internal and external evaluations of teachers’ practices regarding ICT use for teaching, and specific assessments of teachers (or schools) regarding the adequate use of ICT resources.
Access to ICT resources
Over recent decades, access to ICT devices, such as computers and smartphones, has improved considerably around the world. Indeed, mobile phone subscriptions more than doubled globally between 2007 and 2017, reaching 102 subscriptions per 100 people. With a threefold increase over the same period, sub-Saharan Africa largely contributed to this progression, reaching 74 subscriptions per 100 people (World Bank, 2018[14]). In parallel, the share of individuals using the Internet more than doubled in the past decade, to 46% of the global population in 2017. Again, the evolution in sub-Saharan Africa (from 3.5% to 20%) and in low- and middle-income countries in general (from just over 11% to 39%) has been remarkable (World Bank, 2018[14]). In light of the ICT breakthrough, governments have become increasingly concerned with providing new generations access to high-quality ICT resources and minimising the “digital divide” in the population. Education systems have thus come to play an increasingly important role in promoting universal access to and the responsible use of ICT.
Governments’ concerns are legitimate. In spite of the rapid democratisation of ICT, access to ICT has not spread uniformly across all economies, regions and population groups. While nearly all 15-year-old students in OECD countries have access to the Internet (95%, on average), the proportion is much smaller in low- and middle-income countries participating in PISA, such as Brazil (84%), Thailand (71%) and Algeria, Indonesia, Peru and Viet Nam, where fewer than one in two students is connected to the Internet at home (OECD, 2015[11]). Moreover, within countries, differences between advantaged and disadvantaged students in access to computers and the Internet are even larger.4 Although these differences are small in Denmark, Finland and Hong Kong (China), where more than 99% of disadvantaged students have access to computers, in Brazil, Indonesia, Mexico, Türkiye and 8 other PISA-participating economies, less than 50% of disadvantaged students have access to computers (OECD, 2015[11]).
As ICT resources have become increasingly available, the technical standards required for accessing the latest functionalities and online resources with ease have also steadily progressed. Accounting for the differences in quality of ICT resources (in addition to their availability) would likely further increase the existing “digital divide”. Access to ICT resources for 15-year-old students should therefore include a measure of quality, as well as a notion of accessibility, which would reflect the extent to which students can access available ICT resources and whether they face certain constraints.
Previous PISA cycles mainly documented the type of ICT resources available at home or in school. This framework broadens the focus and proposes a systematic and consistent approach to measuring 15-year-old students’ access to ICT resources. Availability, accessibility and quality are documented.
The availability of ICT resources documents the presence of a specific ICT resource, which can be used either in class or during students’ free time.
The accessibility of ICT resources describes the set of elements that characterises the ease and flexibility with which ICT resources can be accessed. Therefore, it refers to existing rules, norms, configurations and arrangements guiding the access to ICT resources both in and outside of school.
The quality of ICT resources is a multi-faceted concept that refers primarily to the functionality, technical capacity and capability of ICT resources. Quality measures describe the extent to which ICT function smoothly – without flaws, delays or security issues – and are compatible with other ICT resources (hardware or software). In addition, some aspects of availability and accessibility also contribute to the definition of quality, such as the diversity of ICT resources and the quantity available per student. Moreover, dimensions such as the relevance and usability of ICT resources – notably in the context of their use by 15-year olds for learning purposes – are also important for defining quality. These correspond to the degree to which the ICT resources are relevant to the curriculum, create interest among students who can easily work with them, and can be used by for a variety of purposes and adapted to different education settings.
The approach described above is aligned with the interests of policy makers. Many ICT-related policies and programmes involve providing ICT materials to schools and students in varying quantities under different settings. In this context, documenting which ICT resources are available and how easily accessible and functional they are seems relevant from a policy perspective. For example, such information can help benchmark the provision of ICT resources for schools or help ensure optimal utilisation at the system level. Moreover, these factors may constrain or shape how students and teachers use ICT resources – which is the principal avenue for investigating the effects of integrating technologies into learning (OECD, 2015[11]; Conrads, 2017[15]; European Commission, 2013[9]; Schleicher, 2015[16]).
Findings from the literature also support the approach proposed in this framework. Research assessing the impact of policies that provide or facilitate investments in ICT in schools or for students cannot, for the most part, disentangle ICT access from use. In recent literature reviews, Bulman and Fairlie (2016[17]) and Escueta et al. (2017[18])conclude that these programmes (whether they focus on schools or students) have little or no positive effect on most academic outcomes. Thus, as Escueta et al (2017[18])highlight, “simply providing devices to students generally [does] not improve learning outcomes”.
Although it is clear that providing ICT resources to schools or students does not necessarily lead to higher learning outcomes, a closer look at the findings uncovers some promising avenues regarding the provision of ICT resources. First, the literature reveals positive effects on computer use and ICT skills acquisition in general (Fairlie, 2012[19]; Malamud and Pop-Eleches, 2011[20]). Second, research suggests that certain digital learning resources may have a positive effect on students’ cognitive achievement (Escueta et al., 2017[18]; Jackson and Makarin, 2018[21]). Third, some evidence suggests that improved accessibility to ICT resources (e.g. by decreasing the waiting time to access computers) could explain the benefits observed in some cases when providing additional ICT resources (Fairlie and London, 2012[22]). Finally, accessing ICT resources does not seem to adversely affect well-being indicators, such as social development and personal interactions (Fairlie and Kalil, 2017[23]).
In parallel, the literature also highlights the claim that providing ICT resources may be more useful when targeting post-secondary students (Escueta et al., 2017[18]). Some studies also observe a negative impact on achievement, possibly because using ICT for leisure (such as playing video games) crowds out time spent on homework assignments (Leuven et al., 2007[24]; Malamud and Pop-Eleches, 2011[20]). Improving the quality of ICT resources (in the sense of improved functionality) has not been studied extensively, but a study looking at differences in Internet speed finds no association with educational attainment (Faber, Sanchis-Guarner and Weinhardt, 2015[25]).
Overall, the evidence suggests that documenting the availability, accessibility and quality of ICT resources, with a thorough accounting of those resources dedicated to learning, could help fill a knowledge gap regarding ICT for 15-year-old students.
Availability of ICT resources
There are several constraints to mapping available ICT resources in the context of PISA. First, the surveyed ICT resources must be relevant across PISA countries and economies, consistent with previous PISA cycles, and should remain applicable over time. In addition, there is a need to cover both ICT resources for learning, in and outside the classroom, and for leisure. Moreover, in the context of this framework, the prime interest is to cover ICT resources potentially related to students’ well-being, cognitive achievement (in mathematics, reading and science) and ICT skills. In order to fulfil these requirements, the proposed approach distinguishes general ICT, which can be considered as tools, or “functional learning materials” following the terminology of Bundsgaard and Hansen (2011[26]) in the context of learning, from ICT resources specifically designed for learning or for school-related activities.5
General ICT resources
ICT resources cover an array of digital tools that can be used for different purposes, including learning and leisure. The diversity of ICT resources available to students can be documented through a two-dimensional approach distinguishing, on the one hand, between hardware and software, and, on the other hand, between ICT for general and for specific use. This framework facilitates the comparison of available ICT resources across students, schools and countries, and helps describe the diversity of those resources.
Hence, general ICT resources range from computers and access to the Internet (i.e. hardware resource for general use) to cameras (hardware resource for specific use), and also include social networking websites or applications (software for general use) and image-processing programmes (software for specific use), among other digital tools.
Overall, the choice of ICT resource identification should ensure that the most common resources are covered. Since very few ICT devices account for most of students’ use of digital resources, providing a full mapping of all available digital devices does not seem to be very valuable or feasible. Thus, the aim should not be exhaustive coverage but rather to cover key digital tools whose unavailability would clearly indicate a lack of access to ICT resources. Different sets of ICT resources could be covered depending on whether students’ access in school or at home is being investigated. This should reflect the fact that ICT resources are more often used for leisure outside the classroom for specific activities, such as video games.
ICT resources designed for learning
As mentioned above, providing resources specifically designed to enhance learning appears to be a promising avenue for ICT to benefit students’ cognitive achievement (Escueta et al., 2017[18]). Although ICT resources for learning mainly include software, classroom equipment, such as interactive whiteboards, projectors or hardware that can be used for this purpose could also be considered. ICT for learning can be diverse, ranging from online educational resources (e.g. specific YouTube channels, or Massive Online Open Courses [MOOCs]) to educational games and even online sharing platforms and intelligent tutoring systems.
The major types of ICT resources for learning can be classified as follows:
Digital content for learning, which includes online courses, digital books and multimedia resources (for the most part, it fits into “semantic learning material” in Bundsgaard and Hansen’s (2011[26]) terminology)
Communication and tracking tools, which facilitate communication among schools, parents and students (and as such could be considered as “functional learning materials”)
Virtual learning environment and intelligent tutoring systems aimed at helping students practice particular skills, which fall into the “didacticised learning materials” category as described in Bundsgaard and Hansen (2011[26]).
Documenting several attributes of the available ICT resources can help assess the ICT environment. For example, knowing whether a specific ICT resource is connected to the Internet or not, whether students can access the resource outside of school, and in addition, whether the resource was created, conceived or adapted by a teacher or a specialist, would provide information on the degree of interactivity and adaptability of available ICT.
Another important characteristic is whether digital learning resources are subject-specific (or even skills-specific) or whether they can be used for learning different subjects. The aim should be to cover, as a priority, digital learning resources relevant to supporting teaching and learning of all the domains assessed by PISA (i.e. mathematics, reading and science). In addition, it would be of interest to cover digital learning resources aimed specifically at supporting teaching and learning in the major domain assessed by PISA in a particular cycle. Hence, the focus will be on digital learning resources used to teach and learn mathematics in PISA 2022, including software designed to enhance students’ engagement with mathematics, online collaborative platforms used by students to solve specific problems or simulation software to be used by mathematics teachers. Covering digital resources and tools relying on, or aiming at developing computational thinking or programming skills (whether this subject is integrated into mathematics or not) would also be of interest. Wide coverage of these digital learning resources is particularly relevant, as Escueta et al. (2017[18]) show that educational software exhibit “enormous promise in improving learning outcomes, particularly when it comes to mathematics”.
The question of “availability” raises some concern in this context. Indeed, a number of ICT resources are freely available on line and are therefore accessible as soon as the student has access to the Internet. Consequently, documenting students’ access to online resources should focus on documenting their knowledge of where to find and how to connect to such resources, as well as whether these were made available by the school or the parents. Alternatively, the availability and actual use of these learning resources should be documented simultaneously.
Accessibility of ICT resources
The mere existence of ICT resources, even those of high quality, would be of little benefit if students and teachers could not use them appropriately. Indeed, ICT resources not only have to be available and of good quality, but students and teachers should also be able to access them when needed. Accessibility refers to the degree of availability and the flexibility with which users can reach available ICT resources. While unlimited access to ICT resources is not necessarily beneficial for students (Malamud and Pop-Eleches, 2011[20]), restricted access to ICT resources, notably in schools, can constitute a major barrier to using them (Fairlie and London, 2012[22]).
For available ICT resources, both in school and at home, access can be constrained due to issues related to ownership, the number of ICT resources per person or because of rules related to the organisation and distribution of ICT resources.
Ownership and congestion
In addition to mapping available ICT resources for students, accessibility can be documented through information on ownership. In particular, whether a specific ICT resource belongs to the student constitutes a key dimension of accessibility. Possessing a computer of one’s own would likely affect the intensity and diversity of use, for example. Supplementary information about ownership outside of school includes whether the ICT resource belongs to another family member, whether the student is the main user of the device, whether the ICT resource is lent by an organisation or whether the resource is only accessed outside the home (in a library, etc.).
The question of ownership also matters when considering ICT resources at school, as more and more schools rely on “bring your own device” strategies for the use of tablet devices and smartphones (Conrads, 2017[15]). Thus, documenting whether students own the ICT resource in school is of interest. When ICT resources belong to the school, it would be important to collect information regarding the degree of “congestion”, including the number of students per computer at the school level, the average time students have to wait to access a computer and, if possible and relevant, the number of students per computer for a specific subject (notably the major domain being investigated in PISA in a given cycle).
Regulations and norms
A variety of regulations could also affect the degree of freedom with which students can access available ICT resources. This can be the case both in and outside of school, although the types of rules are likely to differ significantly.
In school, the rules regarding access to ICT resources may be described in a “code of conduct” or equivalent that determines the type of supervision required for a student to access specific digital equipment, the type of responsibility engaged, the maximum length of time a student can use a specific ICT resource (including the Internet), whether it can be displaced or not (and used at home, for example) and the number of users per device, among other rules. All of these may be part of the administrative process that each student must follow in order to access ICT.
Another way that access to ICT resources could be limited is by the degree of technical restriction. For example, the access to ICT resources could be password-protected; teachers and students may not have access to administrative rights and therefore lack flexibility for accessing relevant learning resources.
In addition, ICT use by teachers and students may be affected by time and space constraints (e.g. classroom size, layout of tables, building design, location of ICT resources in the school). For example, grouping ICT resources in a computer lab may facilitate providing classes in ICT but may hinder the use of ICT for learning (European Commission, 2013[9]). In contrast, placing computers in the classroom can be useful for personalising teaching and learning, and for allowing teachers to use ICT resources more flexibly (Condie and Munro, 2007[27]). Time constraints could arise from queuing time to have access to ICT resources, loss of time when using ICT or simply time pressure to prepare exams and cover the curriculum (European Commission, 2013[9]). Therefore, information about the distribution and placement of ICT resources in the school are important in describing ICT access and use.
For these reasons, collecting information on ICT-related school rules and regulations would largely improve the general understanding of ICT accessibility.
Quality of ICT resources
The quality of ICT resources refers to the technical components of ICT and to the capacity of available ICT resources. Two aspects of ICT-resource quality are covered here: the functionality and maintenance of ICT resources, and the overall degree of connectivity.6
The lack of well-functioning and up-to-date ICT resources may constitute a serious obstacle to using them effectively. The European Survey of Schools ICT in Education (ESSIE) examines a set of potential obstacles affecting schools’ capacity to provide ICT teaching and learning, and shows that a shortage and inadequacy of ICT equipment are the most significant obstacles that schools face (European Commission, 2013[9]). Indeed, according to ESSIE, more than one in five general grade 11 students in Greece, Hungary, Ireland, Malta, Spain and Türkiye attend schools where the lack of bandwidth mattered significantly in providing ICT teaching and learning in 2013. Moreover, head teachers cite out-of-date and faulty computers as key problems affecting ICT teaching and learning (European Commission, 2013[9]).
The quality of available ICT resources can be documented using indicators of the level of functionality and degree of connectivity. The objective is to document the capacity of available ICT resources. However, this approach does not provide a measure of the quality of ICT resources in terms of their actual or intended use. Indeed, a laptop used mainly for basic office software applications (word processor, presentation software and spreadsheet program) does not require the same technical characteristics needed for more demanding applications, such as online gaming or running mathematical simulations. Therefore, students may have to adapt their ICT use to the limited technical capacity of available ICT resources. To overcome this limitation, an alternative approach would be to document users’ assessments of the constraints they face when using ICT resources.
Functionality and maintenance of ICT resources
To make the most of available ICT resources, users should be able to access ICT equipment in order to perform their tasks without flaws, defects, delays or security issues. This depends on a number of factors, including the resources allocated to ICT equipment, indicators of the overall modernity and capacity of the equipment, and routine maintenance practices.
Although imperfect, information regarding the resources allocated to ICT equipment in school and at home could be a good indicator of the standards of ICT. In school, documenting current-year expenditures on ICT resources separately for hardware, software, and digital learning resources would provide an interesting benchmark for quality. Ideally, information on average spending per student and per computer should be available.
Documenting information on the technical characteristics of available ICT resources may be cumbersome and may only allow for limited comparisons. Yet one could consider documenting the basic characteristics of computers, such as, for example, random access memory (RAM), storage capacity and computational power.
This should be combined with information regarding the age of ICT resources and the frequency of replacement, which are not perfect measures of quality when assessed independently, as the better the resources are maintained, the less often they should need to be replaced. Information about the resources allocated to maintaining ICT would be valuable. This would include whether the school can rely on an ICT co-ordinator, and the range of the tasks this person is responsible for, such as maintaining ICT resources, installing new software, etc.
Overall level of connectivity
The degree of connectivity is another central aspect of the quality of ICT resources. The quality of the Internet connection is the first dimension of connectivity. The extent to which available ICT resources can be and actually are connected to the Internet would provide complementary data. Indeed, the Internet is, by nature, both transversal – it is always used in combination with another ICT resource – and transformative – it expands and modifies the possibilities of the equipment with which it is associated. Moreover, as a specific ICT, the Internet also allows for the development of new ICT, such as most smartphone applications, which depend entirely on its existence and availability. Thus, students’ access to the Internet should not be documented separately – by collecting information on the type, speed and modalities of available connections – but also in relation to the set of ICT resources available. Collecting information on whether each of the available ICT resources is connected to (or enabled by) the Internet would therefore provide a detailed picture of the degree of students’ connectivity.
The overall connectivity of a school is of particular interest when describing the quality of the ICT environment. The type of Internet connection (i.e. broadband, digital subscriber line [DSL], fibre, 4G, 3G, etc.), the modalities of connection (wired or wireless), and the corresponding bandwidths available per student are likely to contribute substantially to the functionality of ICT resources and to students’ opportunities to use ICT for different purposes.
In addition, the existence of a school website, a local area network (or intranet), a specific e-mail address for teachers and students, and a virtual learning environment would be indicative of a more connected school environment.
Another dimension of quality is the extent to which ICT resources are adapted to and flexible enough to allow multiple uses. Aspects related to the compatibility of ICT resources with other ICT, and existing constraints regarding the licences and copyrights of software and digital content, would also reveal the flexibility with which ICT resources can be used.
Subjective assessment of shortages and obstacles limiting ICT use
Although indicators of the availability, accessibility and quality of ICT resources are informative, they may be difficult to compare internationally in the context of PISA. Moreover, the extent to which students and teachers are truly constrained by shortages of ICT resources depends on the potential use of those resources. To circumvent these challenges, the various dimensions of the suitability of ICT resources could be assessed through the subjective perception of users, including teachers (or principals) and students. A similar method was adopted in the ESSIE study, in which head teachers’ perceptions of obstacles to ICT use for teaching and learning were surveyed (European Commission, 2013[9]).
Following a similar approach, the users (teachers, students, principals) could therefore be asked whether they feel constrained in their capacity to use ICT for teaching and learning due to shortages of ICT resources (distinguishing among hardware, software and digital learning resources), accessibility (location of ICT resources, school regulations, etc.) or quality (Internet connection, out-of-date computers, software incompatibility, etc.).
Since the subjective assessment of the suitability of ICT resources is likely to be affected by the respective attitudes of teachers and students, a complementary approach could be to ask factual questions about users’ experiences using ICT. For example, users could be asked whether, in the past two weeks, they had decided to abandon, revise or shorten an ICT-related activity because of the absence of adequate ICT resources, because of difficulties in accessing those resources, or because of faulty, deficient or slow-functioning ICT.
Information regarding the differences between the quality of available ICT resources in and outside of school, and in comparison to the best standards could also be documented. This can capture effects related to the relative quality of ICT resources.
The use of ICT in and outside the classroom
As pointed out in recent literature reviews, merely providing ICT resources is not enough to ensure that they are used effectively to improve students’ cognitive achievement, well-being and ICT competencies ( (Bulman and Fairlie, 2016[17]; Escueta et al., 2017[18]). Although the positive impact of ICT use on student achievement remains subject to debate, there is a consensus that the specific purpose, context and pedagogical practices surrounding ICT are central to their effect on students.
This section begins by describing the various ways ICT are used by students in the classroom, with a particular focus on the pedagogical practices that were previously found to be successful in shaping cognitive achievement. It also provides a brief description of the overall school environment, which partly shapes teachers’ and students’ use of ICT in the classroom. Next, it examines students’ ICT use for learning and leisure outside the classroom and highlights important contextual factors.
Students’ use of ICT in the classroom
Teachers’ pedagogical practices and teaching strategies with ICT largely determine the extent to which their use in the classroom will result in improved cognitive achievement. Research stresses the promising potential of computer-assisted learning to bolster student achievement (Roschelle et al., 2016[28]; Pane et al., 2014[29]; Karam et al., 2016[30]; Dinarski et al., 2007[31]). Thus, using ICT for teaching and learning in the classroom does not minimise teachers’ role. On the contrary, as the primary actors for implementing the curriculum and orchestrating learning activities, teachers are likely to be even more central to learning with the adoption of ICT. Indeed, the success of using ICT for educational purposes relies heavily on teachers’ abilities to select, create and manage adequate digital resources in order to implement innovative and inclusive teaching strategies in a specific context (Redecker, 2017[32]).
Integrating ICT into teaching may lead educators to modify their approach to teaching itself, which would eventually affect students’ use of ICT for learning. Some teachers may rely more frequently on specific pedagogical approaches and teaching strategies when using ICT. For example, the ESSIE study shows that teachers using ICT also engage more often in student-centred teaching, although both student-centred and teacher-centred approaches coexist (European Commission, 2013[9]). Teachers may also explore and devise original teaching strategies specifically adapted to ICT. For example, Hennessy, Ruthven and Brindley (2005[33])show that teachers circumvent emerging constraints with ICT by combining ICT resources with other material, or by exploiting ICT possibilities to maintain students’ attention on a subject rather than on unimportant features of ICT.
Teachers’ pedagogical approaches tend to determine students’ use of ICT for learning in the classroom. ICT-based activities in the classroom constitute only one element of the overall teaching strategy. Indeed, teaching with ICT requires particularly careful planning and preparation to select, adapt and create adequate digital resources, and to determine how they are used in teaching and for assessment purposes (Redecker, 2017[32]; Trucano, 2005[34]). Moreover, teachers also support teaching activities by using ICT for communicating and collaborating with their peers, parents and students (European Commission, 2013[9]). While students can report on how digital technologies are used during class, only teachers can provide information on how they prepare to teach with ICT. Consequently, this framework relies on teachers to describe the underlying activities and conceptual approaches of ICT use in the classroom, whereas the diversity of practices and actual uses of ICT for learning are primarily documented through student-reported information.
In this framework, students’ use of ICT for learning in the classroom is assumed to be mainly determined by teachers’ choice to engage in specific pedagogical approaches. In keeping with previous PISA cycles (and due to the structure of the data collection), teaching strategies will be documented mainly from students’ reports and cover pedagogical approaches (applied with or without ICT) as well as ICT-specific practices. Teachers’ activities with teaching-related ICT outside the classroom (not observable by students), such as activities related to preparing lessons and assessments, could also be covered through the teacher questionnaire.
In parallel, the framework explores students’ use of ICT in the classroom for non-class related purposes. Ubiquitous access to the Internet among students does not only provide a boon to education, it also has its disadvantages. Permanent connectivity can lead students to engage in “distracted use” of ICT resources – constantly checking notifications and website updates, responding to friends’ messages, etc. This, in turn, might have substantial negative effects on classroom disciplinary climate and on students’ learning outcomes.
Teachers’ use of ICT for teaching
In addition to implementing pedagogical practices, teachers’ use of ICT for teaching revolves around planning teaching sessions, assessing students, and taking part in supportive communication and collaboration activities with colleagues, parents and students. These activities contribute to the overall effectiveness of ICT use for teaching and are therefore worth documenting.
Teachers spend a substantial amount of time planning and preparing teaching sessions. On average in the countries and economies that participated in the OECD Teaching and Learning International Survey (TALIS) in 2013, lower secondary teachers reported spending seven hours per week planning lessons (OECD, 2014[35]). After a preliminary period of investing time to become acquainted with new practices, ICT might actually help teachers prepare their lessons, regardless of whether ICT-based activities will be conducted during class. For example, teachers can use the Internet and other online applications to find suitable learning resources, or rely on specific software to present certain activities. In this regard, the development of ICT might significantly help teachers renew and adapt learning material and content. In fact, the preparation of teaching activities constitutes the most frequent ICT-based activity conducted by teachers in EU countries, with 30% to 45% (depending on the activity) of students taught by teachers who declare doing this every day, almost every day, or at least once a week (European Commission, 2013[9]).
In addition, teachers face specific challenges when planning to integrate ICT into teaching. Research shows that, without sufficient planning, using ICT may result in a lack of focus among students and lower overall performance (Trucano, 2005[34]). In order to plan lessons involving ICT, teachers must sort through a wealth of ICT educational resources and potentially conduct multiple, time consuming and sometimes complex activities. Indeed, teachers might have to identify, assess and select the ICT resources that best fit their learning objectives, context and pedagogical approach; sometimes, they may even have to adapt or create new digital resources. In parallel, they may also need to manage resources to share them with their students, while maintaining up-to-date knowledge regarding the potential risks involved in sensitive digital content and copyrights (Redecker, 2017[32]). Analysing the time spent on planning lessons and the various types of planning activities teachers engage in could help identify the uses of ICT that are most successful in the classroom.
Teachers also spend a non-negligible amount of time communicating and co-operating with parents and students, in addition to collaborating with teaching staff (OECD, 2014[35]). These activities may enhance the school climate and improve classroom environments (OECD, 2014[35]); they can also provide a way to share good practices for ICT use and ultimately improve student learning. In parallel, ICT may also help disseminate these practices among teachers (European Commission, 2013[9]). By contrary, ICTs could potentially contribute to spreading non-desirable teaching practices.
Indeed, with the widespread availability of computers and Internet access, European schools increasingly communicate with parents through ICT (EACEA, 2011[13]). Communicating with parents (or students) can include disseminating information on the school website and communicating via e-mails or through a dedicated online platform, such as school portals. It can also involve informing parents of their child’s progress and difficulties, encouraging parents to help monitor their child’s homework, and sharing homework assignments. Similarly, teachers can discuss innovative teaching practices with their colleagues, share and co-create digital resources, monitor students’ achievement across subjects or assess their own digital practices and engage in professional development activities.
Teachers’ beliefs about the nature of teaching and learning determine their choice of which pedagogical practices to use in the classroom (OECD, 2014[35]). Teachers may believe that active teaching strategies are more efficient and engage students more throughout the learning process. Alternately, they may think that students learn better by finding solutions on their own or that students should learn in groups. Indeed, in TALIS 2013, teachers with constructivist (i.e. student-centred forms of learning) beliefs were more likely to report that students use ICT for projects or class work (OECD, 2014[35]). Teachers also hold beliefs about whether and how to use ICT for teaching and learning. Documenting teachers’ beliefs on how teaching should be conducted with ICT would provide insights into the relationship between those beliefs and actual ICT use for learning, as reported by students.
In addition, documenting teachers’ reports of whether and how they use ICT in their teaching could be a useful complement to the information (described below) that students report. Teachers could be asked how they share instruction time between structuring practices (i.e. explicitly stating learning goals, summarising previous lessons, reviewing homework and checking student understanding), student-oriented practices (small-group work, ability grouping and student self-evaluation) and enhanced activities (working on projects, making a product, writing an essay and debating arguments) in general, and how the use of ICT may affect their approach. Documenting how ICT affect the time spent on lecturing versus drill and practice activities could also be insightful.
Students’ use of ICT for learning in the classroom
Using ICT in the classroom is likely to affect the instruction time and the curriculum to which students are exposed, as well as the teaching and learning processes they experience. These factors have been documented as important predictors of student achievement (Scherff and Piazza, 2008[36]; Schmidt and Maier, 2009[37]; OECD, 2017[4]). Analysing this relationship requires documenting the frequency and modalities of students’ use of ICT in relation to existing PISA constructs on learning processes and instruction quality.
In addition to the effect of ICT on instructional time and general learning processes, students’ learning outcomes can also be affected by the use of digital educational resources, and original teaching strategies and learning practices with ICT. Given that, existing PISA constructs on instructional quality can be complemented with ICT-specific information. Complementary data on classroom arrangements when ICT are used, and students’ opinions of teachers’ ICT competencies, can also illustrate the effectiveness of students’ use of ICT in the classroom.
Intensity and modalities of students’ use of ICT
Many studies have stressed the importance of instruction time as a determinant of student outcomes across various subjects (OECD, 2013[2]). The integration of ICT for teaching and learning can affect instruction time in many ways. Some research shows that teaching with ICT takes more time as it often requires changing the classroom layout and may require frequently altering pedagogical practices (Trucano, 2005[34]). Moreover, when using certain ICT tools, students’ attention could be drawn away from learning and they might be tempted to use the ICT resources for leisure activities (e.g. games, browsing the Internet, social media, etc.). Yet ICT-assisted instruction may also increase the overall time students spend learning if, in its absence, teachers must divide their time between group and individual instruction (Bulman and Fairlie, 2016[17]). In this scenario, the ICT resources are expected to take over instruction from the teacher in case of an interruption.7 Moreover, the modalities of integration of ICT vary across subjects which can lead to disproportionate changes in learning time across subjects.
Therefore, it is important to document not only how frequently students use ICT for learning, but also the length of time that they use ICT in each class, whether they use it continuously or recurrently, and in which classes. Moreover, students can engage with ICT for learning in different ways, which should be accounted for when assessing the intensity of ICT use. Indeed, it may be important to distinguish between situations where students use ICT on their own initiative – to take notes, for example – and learning situations where students use ICT because it is requested by the teacher. In addition, students may not be allowed to use ICT as they desire or, by contrast, might be encouraged to bring their own ICT devices to class. Classroom-level practices regarding students’ use of ICT should thus be documented.
A detailed examination of students’ ICT use could also document the time spent on, or the frequency with which students use different types of ICT resources. Both the type and the diversity of ICT resources could reveal the degree of complexity of ICT use for learning. In European schools, digital textbooks are the most frequently used resource in grade 8 (with more than 30% of students using them more than once a week), but simulations and data-logging tools are rarely used (European Commission, 2013[9]). Yet the literature emphasises that the latter are more promising for improving student achievement. Using various types of ICT resources, both individually and in combination, could also indirectly indicate more sophisticated teaching approaches and lead to higher cognitive achievement and ICT competencies. The frequency of ICT use could build on the classification of ICT resources developed to measure access to ICT.
Students’ use of ICT for non-class-related purposes constitutes an important drawback. A 2015 survey conducted in 26 states in the United States reveals that college students use a digital device around 11 times during a typical school day, on average, for non-class purposes (Mccoy, 2016[38]). Student use ICT for instant messaging and playing games (Barak, Lipson and Lerman, 2006[39]; Driver, 2002[40]), checking e-mail and watching videos (Finn and Inman, 2004[41]), and browsing the Internet and accessing social networks (Tindell and Bohlander, 2012[42]). Many studies highlight the negative effect of so-called “digital distractions” during class on students’ learning outcomes, including the ability to take notes, recall detailed information (Kuznekoff and Titsworth, 2013[43]) and comprehend lecture content (Sana, Weston and Cepeda, 2013[44]). Moreover, students’ use of ICT for non-class purposes also appears to create distractions for other students who end up with poorer learning outcomes (Tindell and Bohlander, 2012[42]). Thus, students’ constant connectivity might contribute to multitasking, which has well-documented effects on attention and the capacity to digest information (Posner, 1982[45]; Pashler, 1994[46]). These studies mainly focus on college students, and the findings may not extend to 15-year-old students who probably face more restrictions regarding ICT use during class. Documenting the types and frequency of students’ use of ICT for non-learning purposes, whether they are distracted by other students’ activities with ICT, and their attitudes toward these issues thus seems relevant to examining how such experiences affect students’ cognitive achievement and well-being.
Teaching and learning with ICT
Based on findings in the literature, the context questionnaires in PISA highlight three dimensions in assessing instructional quality: structure and classroom management, teacher support and student orientation (including scaffolding, students’ collaboration techniques, and feedback and assessment mechanisms), and cognitive activation (OECD, 2017[4]). Each of these dimensions has been found to be correlated with students’ cognitive achievement (OECD, 2013[2]; OECD, 2017[4]). Each can also be altered significantly by integrating ICT in the classroom (although probably not all to the same extent). Several features of ICT can affect the way teachers provide feedback to students, personalise instruction, develop collaborative projects and rely on group-work assignments. Indeed, findings from previous PISA cycles show that “students using ICT in mathematics class are more likely to describe their teachers as frequently using structuring practices (e.g. setting clear goals, asking questions to verify understanding), student-oriented practices (e.g. giving different work to students who have difficulties or who can advance faster, having students work in small groups), formative assessments (e.g. giving feedback on strengths and weaknesses), and cognitive activation (e.g. giving problems that require students to apply what they have learned to new contexts and/or giving problems that can be solved in several different ways)” (OECD, 2016[3]).
Although students’ ICT use in school is positively correlated with effective instructional strategies in PISA, it is not clear how students use ICT for learning and, in particular, whether ICT are used in ways that are related to quality instruction (OECD, 2015[11]). Detailed documentation of whether and how frequently the instructional processes described above actually involve ICT would help fill this knowledge gap. In particular, it could answer a central question regarding the use of ICT in the classroom: do teachers use ICT mainly as a substitute for simple instruction, or do ICT support and enhance the implementation of more complex and valuable learning activities?
While ICT can be used to support or replace more traditional teaching practices, digital learning resources can also transform them. This may occur by creating a new activity or by combining several learning processes and activities. For example, the literature underlines that computer-assisted learning based on (sophisticated) tutoring systems or educational software are more likely to improve students’ cognitive achievement (Escueta et al., 2017[18]; Bulman and Fairlie, 2016[17]). These digital learning resources often allow for combining two practices related to personalising education: proposing content and activities tailored to fit the student’s learning needs; and providing students (and sometimes teachers) with immediate feedback. Moreover, these tutoring systems also often rely on various forms of digital content, such as videos and simulation tools. The integrated nature of educational ICT resources might constitute an important dimension of their potential success. Documenting this additional aspect of ICT use in the classroom would require developing a careful classification of ICT-specific teaching and learning processes that would account for the possibility of combining practices.
Students’ use of ICT may be particularly beneficial for certain tasks or for developing specific skills in a particular subject. It would therefore be interesting to explore the subject-specific dimensions of ICT use. Documenting ICT use in relation to the domains assessed by PISA would be particularly interesting. With the increasing attention given to the acquisition of digital competencies, it would also be useful to document whether specific educational resources (e.g. specific educational software, ICT literacy curriculum, etc.) aim at fostering these competencies.
ICT may also be particularly beneficial to certain students. The development of new digital educational resources can indeed support teachers and education systems to provide inclusive education – that is to support the learning of students with disabilities and special needs in inclusive settings (EADSNE, 2013). Digital learning resources allow disabled and special needs students to take part in learning interactions (e.g. helping a student to write by dictating a text to a special software). They can facilitate communication by providing a new medium (e.g. allowing students with communication disorders to interact in a more convenient way). ICT can also enable teachers to develop personalised learning strategies for students with special needs (e.g. personalised practices and drilling exercises). Moreover, ICT can help students participate in and follow classroom activities from home (or elsewhere) when they cannot attend in person (Unesco, 2011). Enabling environments for teaching and learning with ICT
Teachers’ capacity to use ICT resources for teaching and learning depends on several contextual factors and practices, which could be referred to as the enabling environment for ICT use in school. In addition to the quality of access to ICT resources (described in the previous section), enabling factors include contextual information on students’ background, school-level policies and practices regarding the governance of ICT use for learning (notably incentives and support for teachers), and teachers’ attitudes towards and competencies in using ICT for teaching. The enabling environment partly determines whether and how teachers use ICT resources in the classroom. The adequacy of that environment can be assessed by asking teachers to report the extent to which these factors aid or impede ICT use for learning.
ICT-related practices and policies at the school level
Although numerous aspects related to ICT use in school are decided at the national level, schools often retain some leeway in organising the integration of ICT into teaching. For example, most European schools are responsible for purchasing and maintaining ICT resources in schools, as highlighted in the previous section (European Commission, 2013[9]). School-level governance regarding ICT use for teaching includes consultation mechanisms, guidelines, structures of teacher incentives, and support and practices regarding the assessment and evaluation of ICT use.
The overall school environment, including information sharing, guidance and communication among teaching staff regarding the use of ICT for teaching and learning, constitutes a first important component of school governance. All activities aimed at discussing, consulting, developing a common understanding, spreading information and even communicating guidelines on how ICT should be used for educational purposes in the school are critical to integrating ICT into the classroom. Depending on the school, such activities can differ in their degree of formality and method of delivery (i.e. oral or written ICT guidelines, official directives from education authorities or school-level statements, etc.). Documenting such aspects – for example, by asking teachers or principals whether and how frequently they engage in related activities – can build on the existing framework developed in previous PISA cycles to describe the school climate. In particular, it can explore whether teaching staff share clear norms and attitudes and have mutually supportive interactions regarding ICT use for learning (OECD, 2017[47]).
The use of assessment and evaluation for improving teaching practices constitutes another important element of school governance. The relationship between ICT and evaluation is two-way. ICT can facilitate the implementation of evaluations and assessments in school. For example, principals could rely on online surveys to receive feedback on a specific aspect of the school. At the same time, the practices related to ICT use for learning could be assessed. Both dimensions can be covered by specifying whether the current assessment and evaluation practices rely on ICT, whether new assessment practices were developed based on new ICT, and whether the use of ICT for teaching and learning has been subject to a specific evaluation.
The structure and types of incentives and support teachers can access when using ICT resources for educational purposes are also instrumental in guiding their practices with ICT. Teachers can benefit from training to improve their ICT skills and to develop ICT-specific pedagogical competences. The lack of both technical and pedagogical support was most frequently cited by grade 11 teachers in European schools (European Commission, 2013[9]). Teachers can also benefit from their school’s general support policies by having access to manuals and tutorials regarding ICT use, or by taking advantage of the presence of an ICT co-ordinator. The quality of the school environment regarding ICT use should also be assessed by documenting the types of support available to teachers for improving their skills in using ICT. In particular, supporting and assisting teachers’ engagement in collaborative teaching experiments and professional development activities promotes change in teaching practices (Ronfeldt et al., 2015[48]; Vieluf et al., 2012[49]).
Support to teachers who use ICT for teaching can also come in the form of incentive or reward programmes. For example, teachers may benefit from financial incentives, career advancement, reduced number of teaching hours, competitions that award prizes, additional training hours and additional ICT equipment for the classroom (Wastiau et al., 2013[50]). Teachers can also become more motivated through informal incentives, such as peer pressure or signals that their overall evaluation may be influenced by their use of ICT in the classroom. However, these incentives may not necessarily be effective in promoting efficient use of ICT and could even crowd out teachers’ intrinsic motivation to seize the pedagogical opportunities ICT offer. Documenting existing incentives for teachers to use ICT for learning can provide a more complete picture of the enabling environment in the school.
Teachers’ and principals’ attitudes and competencies related to ICT
Teachers’ use of ICT depends on diverse factors, such as the demographic, social, economic and cultural background of the students, the school climate (truancy, disruptive behaviour), and the level and distribution of students’ abilities at the beginning of the school year. All of these factors affect students’ learning and teachers’ pedagogical approaches. Many of these factors are already accounted for in the general PISA framework. Complementary information specifically relevant to ICT use for learning includes parents’ attitudes towards ICT, which have been shown to be significantly associated with ICT use (Brummelhuis and Binda, 2017[51]). Moreover, students’ experience and attitudes toward ICT and, in particular, towards using ICT for learning are particularly relevant. These factors are covered, in detail, in the next section.
Principals’ and teachers’ attitudes towards ICT resources, in general, as well as for teaching and learning are crucial components of the enabling environment. Indeed, in Europe, teachers’ positive opinions regarding ICT use for learning is positively correlated with actual use and experience in using ICT (Wastiau et al., 2013[50]). Principals’ opinions also matter, as the Second Information and Technology Education Study (SITES) 2006 findings suggest that ICT use by teachers is influenced by the school principal’s views about its value (Law, Plomp and Pelgrum, 2006[52]). Measures of teachers’ interest in, attitudes towards, motivation and beliefs about ICT and ICT use as a tool for instruction could be developed based on existing PISA constructs on students’ attitudes towards ICT use.
In addition, teachers’ competencies and experience with ICT also influence their engagement. Teachers’ competencies can be approximated by measures of self-efficacy and complemented with factual information about whether they obtained specific qualifications or pursued training to develop their ICT skills and learn how to teach with ICT. Some evidence suggests that new teachers have insufficient training in the pedagogical uses of ICT while more senior teachers may lack technical knowledge in using ICT for learning. Indeed, in TALIS, teachers identify “teaching with ICT” and “using new technologies in the workplace” as the second and third most important professional development needs (OECD, 2014[35]). Moreover, teachers who are more experienced in using ICT for teaching and learning build their self-confidence, which appears to foster greater development of students’ skills (OECD, 2014[35]).
Although all of the abovementioned factors contribute to the school environment regarding ICT use for learning, their relative importance is likely to differ substantially across teachers. Documenting the subjective assessment of teachers (or principals) regarding what they consider to be the main obstacles to using ICT in school could yield more insights into these issues. As discussed in the previous section about the obstacles related to the quality of ICT resources, teachers would indicate the extent to which a variety of “enabling factors” encourage or impede the use of ICT.
Students’ use of ICT outside the classroom
Over the past decade in PISA-participating countries and economies, the number of 15-year-olds with Internet access has grown, as has the amount of time spent on the Internet outside of school (OECD, 2015[11]). Time spent online increased by about 40 minutes per day between 2012 and 2015, on average across OECD countries, to reach two-and-a-half hours, on average, on weekdays and more than three hours on weekend days (OECD, 2017[47]). With the widespread availability of smartphones, many young people can go on line at virtually any moment. Indeed, a study shows that 24% of 13-17 year-olds in the United States reported going on line “almost constantly” (Pew Research Center, 2015[53]). Moreover, the ESSIE study suggests that most of the time European students spend using ICT is dedicated to leisure activities (European Commission, 2013[9]). These results are in line with the Global Web Index results, which covers over 40 countries. The results recorded an increase, between 2012 and 2016, of around 30 minutes in the average time spent on line, per day, on social media and messaging (GWI, 2017[54]).
Consequently, policy makers are expressing greater interest in understanding how students’ engagement with ICT outside the classroom relates to their well-being, cognitive achievement and acquisition of ICT skills. ICT can foster students’ engagement and motivation in learning activities outside of school. In particular, students may invest more effort in completing their home assignment when they can use ICT resources. This, in turn can be encouraged by teachers who could adapt home assignments to the possibilities offered by ICT. ICT resources (e.g. school platforms) can also be used to enhance parent and teacher supervision over students’ efforts to complete homework.
One of the important advantages of integrating ICT into the education system is bridging the divide between school and home and allowing for more continuity between the two. The increased availability of ICT resources for learning – often designed to capture students’ attention and provide an engaging interactive working environment – can also foster students’ self-motivated engagement in learning activities. Students can also develop different skills while using ICT for leisure, including problem solving and Internet-safety competencies, and also organisational, networking and communication skills, all of which can contribute to students’ cognitive achievement and well-being. However, research has also highlighted several risks associated with misuse or overuse of ICT among young people (OECD, 2017[47]; Hooft Graafland, 2018[55]).
Using ICT for learning
Since teaching and learning is not limited to formal instruction in the classroom, the PISA 2022 questionnaire framework reclassifies students’ “after-school” opportunities to learn as an integral part of education (OECD, 2018[56]). ICT can be a catalyst for learning outside the classroom, notably through their potential effect on students’ engagement with learning activities and by providing a powerful tracking and monitoring tool for teachers and parents.
Some evidence suggests that digital learning resources may affect students’ engagement with and motivation towards learning activities (Faber, Luyten and Visscher, 2017[57]; Hunsu, Adesope and Bayly, 2016[58]; Lazowski and Hulleman, 2016[59]). More specifically, digital formative-assessment tools are likely to enhance students’ motivation when the autonomy of the student is also favoured. Positive feedback can increase students’ motivation, but negative feedback may have adverse effects on motivation (Muis et al., 2015[60]).
In addition, the development of a wealth of digital learning resources, providing students with more and better learning opportunities, such as educational games, Massive Online Open Courses (MOOCs), and a variety of topic–specific media content, such as video and audio podcasts, tutorials, etc., may spark students’ interest in using ICT for learning outside the classroom. The availability of such resources for free might be particularly helpful for disadvantaged students who now face fewer barriers to e-learning and a wide range of learning content (UNICEF, 2017[61]). In addition, teachers may take advantage of students’ interest in ICT by assigning homework that requires the use of ICT. Moreover, students’ interest in ICT may encourage them to engage in informal learning activities about ICT itself such as following coding classes and participating in programming clubs.
ICT can also be used as a tool to foster communication between schools and students (or parents) and monitor students’ achievement and efforts. Research shows that behavioural interventions aimed at improving information flows between the school and parents and, in particular, encouraging parental engagement in learning activities with their children could ultimately help improve students’ education outcomes (Levine et al., 2010[62]; Senechal and LeFevre, 2002[63]). Moreover, since parental engagement is particularly poor among socio-economically disadvantaged students, strengthening school-parent communication through the use of ICT can help reduce, or at least not increase, disparities in education outcomes related to parental engagement (Escueta et al., 2017[18]). ICT may also allow teachers to track students’ completion of homework assignments, which could prompt stronger engagement with learning activities.
Using ICT for leisure
Most of the time students spend using ICT outside the classroom is dedicated to leisure activities. Over the past two decades, ICT have not only transformed how 15-year-olds learn, but also how they socialise and play (OECD, 2015[11]). Access to the Internet is now almost universal. Students use the Internet daily, and most digital activities for leisure happen on line.
The evolution of ICT practices can be observed in the PISA data. Between 2009 and 2012, e-mail and chat use have declined, likely due to the emergence of new forms of communication, such as social networking and other web-based messaging tools. Similarly, students’ use of one-player games seems to have been replaced by online collaborative games (OECD, 2015[11]). The rapid changes in students’ habits and practices regarding ICT use for leisure should be carefully documented to examine how these activities can contribute to – or impede – students’ cognitive performance and their acquisition of ICT competencies.
The extent to which ICT use for leisure is potentially related to students’ cognitive performance, ICT skills and well-being depends on the frequency, the diversity and the type of activities students engage in (van Deursen and Helsper, 2015[64]). The main challenge with documenting ICT use for leisure is to ensure enough continuity across PISA cycles and comparability across countries and economies, while expanding and transforming the coverage to capture new forms of ICT engagement. Certain activities covered in the current PISA framework are obsolete and should probably be grouped together or replaced (e.g. using e-mail and chatting on line). Conversely, some ICT resources have become more central to students’ lives and can be used in a variety of ways. These should be identified distinctly. For example, there are various ways of “participating in social networks”, with very different implications on students’ outcomes.
ICT use for leisure provides an opportunity for students to acquire ICT knowledge and skills. One way to capture the diversity of ICT use for leisure in a relatively stable and relevant manner would be to link each activity to a specific competency area identified in ICT/digital literacy frameworks. These competency areas differ across frameworks, yet they all involve accessing, evaluating and managing information; transforming, creating and sharing information; communicating; using information safely, ethically and securely; and demonstrating a general understanding of ICT use (Fraillon, Schulz and Ainley, 2013[1]; Carretero, Vuorikari and Punie, 2017[6]; Fraillon et al., 2015[5]; ICT Literacy Panel, 2002[65]). While these competency areas define ICT literacy, many are specifically relevant to cognitive achievement in mathematics, science and reading. Using information safely, ethically and securely might be particularly important for students’ well-being.
As mentioned earlier, students’ use of ICT for leisure also involves risks, and is a source of concern among parents and policy makers (Hooft Graafland, 2018[55]). Inappropriate or unsafe Internet use can expose students to harmful content or to cyberbullying. For example, the EU Kids Online survey shows that 21% of 11-16 year-olds from 25 European countries have encountered one or more websites with potentially harmful user-generated content, including hate messages (12%), pro-eating disorder sites (10%), self-harm sites (7%), pro-drug-taking sites (7%) and suicide sites (5%) (Livingstone et al., 2009[66]). In the United States, 41% of adults have been personally subjected to harassment on line; and a quarter of 18-29 year-olds have experienced mental or emotional stress as a result of online harassment (Dragiewicz et al., 2018[67]). Students also face an enormous amount of information online that might help them develop online reading skills, but can also have negative implications if the students are not able to distinguish fact from fiction and verify online sources. For instance, a recent study focusing on Italy shows that only 42% of 9-17 year-old Italians report finding it easy to check if online information is true (Mascheroni and Ólafsson, 2018[68]).
Additional risks, such as overuse of video games and compulsive use of social media, can have serious physical, social, psychological and cognitive consequences (OECD, 2017[47]; Smith et al., 2008[69]; Currie et al., 2012[70]). These risks are real and are not restricted to a minority of students. Nowadays, the vast majority of adolescents have access to ICT resources in their bedrooms. The National Sleep Foundation revealed that more than three in four 13-18 year-olds in the United States sleep with their cell phone next to their beds, and more than one in two report sending text messages in the hour before trying to go to sleep every night or almost every night (National Sleep Foundation, 2011[71]).
Therefore, in addition to the environmental factors covered above, risky behaviours and modes of ICT use should be documented. This could, for example, aim to identify compulsive and addictive use of ICT, whether students are mostly passive or active on line and whether they follow routines when they use the Internet. Information could also be collected about what students consider to be acceptable and ethical use of social media.
The environment for ICT use outside the classroom
As with ICT use in the classroom, several contextual factors, policies and practices influence ICT use outside the classroom. The quality and modalities of access to ICT resources outside the classroom, which can be described along the dimensions of availability, accessibility and quality of ICT resources (see Section 3), largely shape students’ use of ICT. Moreover, students’ use of ICT in the classroom can influence how these resources are used outside the classroom (and vice versa), notably by exploring new practices and the acquisition of ICT competencies, including awareness of the different risks related to ICT use. Schools also play an important role in providing information and training to parents regarding online safety and effective Internet use (Hooft Graafland, 2018[55]). In addition, students’ attitudes towards ICT use for learning and for leisure (covered in the next section) guide students’ use of ICT outside the classroom.
The framework refers to students’ use of ICT outside the classroom and not at home because, in some cases, students’ main access to ICT outside the classroom might not be at home but in other locations, such as in libraries, computer labs, etc. The regulatory environments in these different locations can be documented in a comparable way. Yet for reasons of comparability and simplicity, it may be more relevant to focus only on the regulatory environment at home.
Research shows that the regulatory environment can influence students’ use of ICT outside the classroom and, consequently, students’ education outcomes. For instance, in Romania, there is evidence suggesting that the negative effect on students’ grades from providing subsidies to households to purchase computers was reduced when parents provided more structure and guidance on how and when to use ICT (Malamud and Pop-Eleches, 2011[20]). This suggests exploring a set of practices that limit, to some extent, students’ liberty in using ICT. Such practices include imposing a time limit or deadline on using some ICT resources, requiring parents’ authorisation to use the ICT resources, or using parental control software. Practices related to parental guidance should also be covered.
Students’ cognitive and well-being outcomes
This framework aims to assess the relationship between students’ access to and use of ICT with three distinct outcomes: students’ cognitive achievement, students’ well-being and students’ competencies in ICT. This framework relies entirely on existing PISA frameworks to measure students’ cognitive achievement in mathematics, science and reading, as well as their well-being. It also proposes an approach to assessing students’ competencies in ICT, which are defined here in a broad sense that encompasses digital literacy as a specific domain as well as students’ attitudes and dispositions towards ICT use in various contexts. While proposing a fully-fledged assessment framework for digital (or ICT) literacy is beyond the scope of this framework, a roadmap for such a framework is suggested for future PISA assessments.
Students’ cognitive achievement and well-being
PISA’s approach to measuring students’ cognitive achievement consists “in assessing not only whether students can reproduce knowledge, but also whether they can extrapolate from what they have learned and apply their knowledge in new situations. It emphasises the mastery of processes, the understanding of concepts, and the ability to function in various types of situations” (OECD, 2017[4]). Thus, rather than assessing mathematics, science and reading per se, PISA aims at documenting mathematics literacy, science literacy and reading literacy, where literacy refers to “students’ capacity to apply knowledge and skills in key subjects, and to analyse, reason and communicate effectively as they identify, interpret and solve problems in a variety of situations” (OECD, 2017[4]). For simplicity, the following refers simply to mathematics, science and reading assessment frameworks.
This section summarises the most recent assessment framework for each domain, which is revised every nine years (three cycles) when it becomes the main domain of PISA. Thus, the assessment framework for mathematics was revised for PISA 2022, while the reading framework was revised in 2018 and science was revised in 2015. This section also presents the framework to assess adolescents’ well-being, which was developed for PISA 2018.
Assessing mathematical literacy in PISA
Preliminary work for the (to-be revised) PISA 2022 mathematics framework defines mathematical literacy as “an individual’s capacity to reason mathematically and to formulate, employ, and interpret mathematics to solve problems in a variety of real-world contexts. It includes concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals to know the role that mathematics plays in the world and to make the well-founded judgments and decisions needed by constructive, engaged and reflective 21st-century citizens” (OECD, 2018[56]). The assessment of mathematical literacy is organised around three interrelated aspects presented in detail below: mathematical reasoning and mathematical processes, the mathematical content and the context of the assessment, including its relation to 21st-century skills.
Mathematical reasoning contributes to individuals’ ability to reason logically and to present honest and convincing arguments. Mathematical reasoning comprises a few “big mathematical ideas” that can be seen as the core of mathematical literacy. These ideas include: quantity; number systems and their algebraic properties; mathematics as a system based on abstraction and symbolic representation; mathematical structure and its regularities; functional relationships between quantities; mathematical modelling as a lens onto the real world (e.g. onto the physical, biological, social, economic and behavioural sciences); and variance as the heart of statistics (OECD, 2018[56]).
Mathematical literacy is organised and structured around three major mathematical processes (or, in the terminology of PISA 2022, problem-solving processes). They describe what individuals do to connect the context of a problem with the mathematics that is entailed, thereby solving the problem:
Formulating situations mathematically refers to individuals’ ability to recognise and identify opportunities to use mathematics and to translate a problem presented in a real-world context into mathematical terms and structure.
Employing mathematical concepts, facts, procedures and reasoning corresponds to individuals’ capacity to apply mathematics to solve mathematically-formulated problems and obtain mathematical conclusions.
Interpreting, applying and evaluating mathematical outcomes focuses on individuals’ ability to reflect upon mathematical conclusions and interpret them in the context of the real-life problem. This involves translating mathematical solutions back into the context of the problem and making sense of the conclusions.
The assessment of mathematics is organised around content knowledge categories that reflect both the mathematical phenomena that underlie the general structure of mathematics as well as the major strands of typical school curricula. The content categories used in PISA 2012 and that will again be used in PISA 2022 are change and relationships, space and shape, quantity, and uncertainty and data (OECD, 2018[56]; 2013[2]). Within each of these categories, special emphasis will be given to a specific topic that reflects the type of mathematics needed to understand emerging areas of society and economy in the 21st-century. These topics are growth phenomena, geometric approximation, computer simulations, and conditional decision making.
An important feature of mathematical literacy, as defined in PISA, is that mathematics is used to solve a problem set in a real-world context and/or to help 21st-century citizens make informed decisions. A wide variety of contexts should be used to connect with a broad range of students’ interests across PISA-participating countries and economies. The following context categories will be used to develop items for PISA 2022:
personal, which focuses on one’s own activities or those of one’s peer group such as food preparation, shopping, games or personal transportation
occupational, which is centred on the world of work and includes problems such as measuring, costing and ordering material for building, accounting, quality control, design and job-related decision making
societal, which refers to problems of one’s community and includes problems related to voting systems, public transport, government and economics, among others
scientific, which relates to the application of mathematics to the natural world and issues related to science and technology including topics such as climate, ecology and medicine but also relates to the world of mathematics itself when all the elements included belong to the mathematical context.
The assessment of mathematics in PISA 2022 includes developments that are of particular interest for this framework, notably in relation to ICT literacy. In order to reflect the growing role of technology in students’ lives and to explore increasingly sought-after competencies, the mathematical literacy assessment has put more emphasis on computational thinking. In this context, computational thinking refers to formulating problems and designing their solutions in a form that can be executed by or with a computer (Cuny, Snyder and Wing, 2010[72]). In addition, the PISA 2022 assessment framework identifies critical thinking, creativity, research and inquiry, self-direction, initiative and persistence, information use, systems thinking, communication and reflection as critical 21st-century skills to be included in the assessment of mathematics (OECD, 2018[56]).
Assessing reading literacy in PISA
Starting in PISA 2018, reading literacy has been defined as “understanding, using, evaluating, reflecting on and engaging with texts in order to achieve one’s goals, to develop one’s knowledge and potential and to participate in society”. The definition has evolved over time to reflect the increasing importance of information technology in citizens’ social and work lives. Hence, reading literacy no longer predominantly focuses on the ability to understand, interpret and reflect upon single texts, but reflects a broader range of skills, including higher-level digital reading skills (OECD, 2016[3]).
Following Snow and the RAND group’s (2002) influential framework, reading is considered as the joint outcome of three components: the reader, the text and the task. Reader factors include motivation, prior knowledge and other cognitive abilities. Text factors relate to, among other things, the format, level of difficulty, type of language and number of pieces of text encountered. Task factors correspond to the requirements or reasons that motivate the reader’s engagement with the pieces of text, including time and other practical constraints, the objectives of reading (e.g. for pleasure, for deep understanding or for a cursory overview), and the complexity or number of tasks. The combination of these factors determines how readers apply reading processes in order to locate and extract information and to construct meaning from texts in order to fulfil their tasks (OECD, 2016[3]).
In order to adequately assess the many facets of reading – a pervasive and highly diverse activity – it is necessary to ensure a broad coverage of what students read, for what purposes they read and in which context they read. Thus, the PISA reading assessment relies on variation in the range of material that is read; in the reading processes, or the cognitive approach that determines how readers engage with a text; and in the reading scenarios – the range of broad contexts in which or purposes for which reading takes place.
In PISA 2018, the typology of reading processes identifies reading fluently as a process distinct from the other cognitive processes related to text comprehension (locating information, understanding, and evaluating and reflecting). Reading fluently is the ability to read accurately and automatically in order to comprehend the overall meaning of the text. Locating information includes accessing and retrieving information within a piece of text and searching for and selecting relevant texts. Following Kintsch’s definition of a “situation model” (1998[73]), two core processes contribute to the process of understanding a text: constructing a representation of the literal meaning of the text and generating an integrated text representation, which requires connecting the text with one’s prior knowledge. Finally, the evaluating and reflecting process requires readers to reflect on the content and form of the text, critically assess the quality and validity of information, and deal with contradictions and conflicts within and across texts (OECD, 2016[3]).
Assessing science literacy in PISA
The PISA 2015 assessment and analytical framework defines scientific literacy as “the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen”. A scientifically literate person is willing to engage in reasoned discourse about science and technology, which requires the competencies of explaining phenomena scientifically, evaluating and designing scientific enquiry, and interpreting data and evidence scientifically (OECD, 2017[47]).
The assessment of students’ performance in science covers four aspects: contexts, knowledge, competencies and attitudes. Contexts refer to a set of personal, local and global issues that demand some understanding of science and technology such as health and disease, natural resources and the environment. Knowledge corresponds to the understanding of the main facts, concepts and theories that form the basis of scientific knowledge; specifically, it includes content knowledge, procedural knowledge and epistemic knowledge. Competencies are the ability to explain scientific phenomena, evaluate and design scientific enquiry, and interpret data and evidence scientifically. Finally, attitudes toward science refer to students’ interest in science and technology, how they value the scientific approach and their perception and awareness of environment issues (OECD, 2017[4]).
Assessing adolescents’ well-being in PISA
Adolescents’ well-being can be defined as the quality of students’ lives and their standards of living. There seems to be a consensus that well-being is a multi-dimensional construct with both objective material components and subjective psychological facets. The PISA 2018 framework for the assessment of well-being integrates these different perspectives. In addition to students’ overall perceived quality of life or life satisfaction, the framework covers three other dimensions of well-being, each of which incorporates both objective and subjective components: self-related well-being, well-being in school environments and well-being outside of school environments (OECD, 2016[3]).
Overall life satisfaction is a core dimension of subjective well-being. Two alternative approaches are widely used to assess life satisfaction: an evaluative approach, where individuals evaluate their lives, and a life satisfaction approach, where individuals respond to questions such as “How satisfied are you with your life overall these days?” Although the two approaches are very similar, the life satisfaction approach is preferred in PISA as it is simpler to administer and less intrusive (OECD, 2015).
Self-related well-being focuses on “how fit and healthy students are and how they feel about themselves and their lives” (OECD, 2016). It is divided into three sub-dimensions:
Health, which can be documented by objective indicators such as the body mass index, physical exercise, typical sleep duration and risky behaviours; and subjective indicators including perception of and satisfaction with body image, satisfaction with sleep, and satisfaction with and perceived overall health
Education and skills, which also includes students’ perceptions of their ability to perform specific tasks and their overall confidence in their own abilities; the assessment of cognitive achievement in PISA provides objective indicators, while self-efficacy measures and questions on students’ satisfaction with their knowledge and skills provide subjective indicators
Psychological functioning, which relates to one’s sense of meaning, purpose and engagement, and is referred to as “eudaimonic well-being” in the literature (OECD, 2016[3]). The OECD guidelines on measuring subjective well-being identify three main facets of eudaimonic well-being: competence, autonomy and meaning (or purpose), and optimism.
The second dimension of well-being covered in the framework relates to students’ quality of life in their school environment. Adolescents spend a large amount of time at school and their experiences and relationships at school are strongly correlated with their perceived quality of life (OECD, 2016[3]). Two main sub-dimensions of school-related well-being are explored:
Social connections include social relationships with teachers and other students, and general patterns of students’ interactions that might foster a sense of belonging at school (OECD, 2016[3]). Objective indicators include whether students experienced bullying, while subjective indicators include student-teacher relationships, the school climate, a sense of belonging, perceived discrimination and social connectedness.
Schoolwork, which refers to students’ workload and time spent at school, also contributes substantially to students’ well-being. For example, extreme hours of schooling or an overload of homework can lead to stress and health-compromising behaviours (OECD, 2016[3]). Objective measures of the level of schoolwork include the total time students spend at school, commuting and doing homework. Subjective indicators include, for example, the emotions experienced (both positive and negative) during selected episodes associated with schoolwork.
Well-being in out-of-school environments can be broken down into three components:
Social connections outside of school, which refer mainly to students’ friendships and their relationships with their parents. Studies suggest that family relationships and friendships are the main factors that determinate self-satisfaction (Edwards and Lopez, 2006[74]; Suldo et al., 2014[75]). Social connections can be documented with objective indicators, such as time spent on activities with friends, and with subjective indictors, such as students’ perceptions of and satisfaction with their social connections (e.g. their satisfaction with their number of friends, the degree to which they feel they have fun with their friends, and the degree to which they feel that they are treated fairly by their parents) (OECD, 2016[3]).
Material living conditions are measured objectively through the PISA index of economic, social and cultural status (ESCS), which is itself derived from questions in the student questionnaire on home possessions, parents’ education and parents’ occupation. Research suggests that perceived social and economic standing could be even more crucial in determining individuals’ well-being (OECD, 2016[3]). Thus, subjective indicators about perceived poverty and perceived aspirations failure could be collected.
Leisure time, when students can engage and flourish in self-chosen activities. Objective indicators of leisure time include the total time available for such activities and the activities that students engage in. Subjective indicators could include both the positive and negative affective and emotional states of students during leisure time.
Students’ competencies in ICT: Digital literacy, and attitudes and dispositions towards ICT
As indicated by the hundreds of digital literacy initiatives around the world, ensuring that students acquire sufficient ICT competencies is becoming an increasingly important objective for policy makers and education systems (Melorose, Perroy and Careas, 2008[76]). ICT competencies are not only valuable for their capacity to support teaching and learning, but also as an independent area of focus, as those skills have become essential for participating fully in the digital age.
In line with previous PISA cycles, this framework takes a broad perspective on ICT competencies, which include the set of knowledge, understanding, attitudes, dispositions, and skills necessary to thrive in the digital age. Indeed, the attainment of knowledge and skills in a specific area is intricately intertwined with individual attitudes and dispositions towards learning. On the one hand, research shows that attitudes and dispositions are central to the learning process, and contribute to individual development and well-being ( (European Commission, 2013[9]; Almlund et al., 2011[77]; Heckman, Stixrud and Urzua, 2006[78]). On the other hand, they may also be considered as education outcomes in their own right (Bertling, Borgonovi and Almonte, 2016[79]).
Students’ attitudes and dispositions towards ICT are therefore considered as parts of ICT competencies in this framework. Indeed, students’ motivation to learn ICT-related skills, their openness to new experiences, willingness to collaborate and engage with others using ICT, and their confidence in conducting certain tasks with ICT are strong determinants of their level of proficiency with ICT and their ability to use them for learning. As teaching and learning tools, ICT can also affect students’ attitudes and dispositions towards learning in general.
Governments and policy makers are increasingly interested in assessing students’ levels of proficiency in ICT. This framework proposes a direction that the assessment of ICT literacy could take in future PISA cycles even though a complete assessment of ICT literacy is beyond its scope. Yet, this framework documents specific dimensions of students’ ICT competencies relying on measures of self-efficacy, attitudes and dispositions towards using ICT in various contexts.
Students’ digital literacy
The growing importance of students’ ICT literacy for policy makers and education systems is reflected in the frequent inclusion of a variety of ICT competencies in curricula (European Commission, 2013[9]). Interestingly, as measures of ICT competencies become more widely recognised, education systems tend to shift from teaching ICT skills in isolation towards a more horizontal approach, integrating specific ICT tasks and competencies across subjects (European Commission, 2013[9]). This highlights the cross-cutting and complex nature of ICT, which are often used as a tool to support instruction, but are also recognised as a subject of learning in themselves.
Although this framework does not provide a full-fledged assessment of ICT competencies, it proposes foundations for integrating ICT literacy as a specific domain in future PISA cycles. It relies on existing assessments of ICT literacy to identify the main methodological challenges and key competency areas that should guide the development of such assessment.
Competence framework for ICT literacy
This framework proposes measuring students’ competencies in ICT as a stand-alone discipline, independent of using ICT for enhancing the teaching and learning of specific subjects. This contrasts with assessments previously conducted in PISA, notably that of digital reading literacy, which combines the assessment of subject-specific achievement and ICT use, and therefore implicitly assumes they are inter-related (European Commission, 2013[9]). In such assessments, the ICT resource is considered as “a vehicle for students to express their discipline-specific knowledge, understanding, and skills” (European Commission, 2013[9]). By contrast, assessing ICT literacy as a specific domain recognises the importance of being able to conduct a variety of more or less complex tasks related to information processing in various digital contexts. It also facilitates comparisons across countries as the assessment is not anchored in a specific learning area or content.
The Feasibility Study for the PISA ICT Literacy Assessment defines ICT literacy as “the interest, attitude, and ability of individuals to appropriately use digital technology and communication tools to access, manage, integrate, and evaluate information, construct new knowledge, and communicate with others in order to participate effectively in society” (Lennon et al., 2003[80]). This definition shares many similarities with the approach developed in other ICT literacy assessment framework such as ICILS and Australian Curriculum, Assessment and Reporting Authority (ACARA) ICT Literacy among others (Fraillon et al., 2015[5]; 2013[1]). In particular, these definitions draw extensively upon information literacy, they assume that individuals possess the technical skills required to effectively use digital technologies, they identify similar sets of processes and they recognise ICT literacy as a requirement for individuals to fully participate in 21st-century society.
In light of the growing importance of digital businesses in the global economy and increasing work opportunities available in “big data” and “artificial intelligence”, for example, many countries show a burgeoning interest in including computational thinking, problem solving, data literacy and other 21st-century skills in their curricula. Recent assessment frameworks for ICT literacy reflect this evolution, and give more weight to computer literacy, data literacy and critical thinking. Thus, DigComp 2.1 makes explicit reference not only to information but also to data literacy, and includes problem solving as a key competency area (Carretero, Vuorikari and Punie, 2017[6]). Moreover, the International Computer and Information Literacy Study ICILS 2018 extends the computer and information literacy construct to include a new computational thinking strand (IEA, 2017[81]).
A comprehensive framework to assess ICT competencies could therefore revolve around five main competency areas, namely accessing, evaluating and managing information and data; sharing information and communicating; transforming and creating digital content; individual and collaborative problem solving in a digital context, and computational thinking; and appropriate use of ICT, which embeds knowledge and skills related to security, safety and risk awareness (Fraillon et al., 2015[5]; Fraillon, Schulz and Ainley, 2013[1]; Redecker, 2017[32]).
Competency area 1: Accessing, evaluating and managing information and data
Accessing information and data focuses on the extent to which individuals can identify the desired information, data or digital content and understand how to find and recover computer-based information from various sources, by using ICT (Fraillon et al., 2015[5]; ACARA, 2015[82]).
Evaluating information and data is an integral step in accessing information and ever more so with the development of search engines and artificial intelligence. This involves the process of filtering through multiple information sources, and assessing their relevance, integrity and usefulness (Fraillon, Schulz and Ainley, 2013[1]; ACARA, 2015[82]). As growing amounts of data, news and reports are communicated through the Internet, the process of sorting through all this information is becoming increasingly essential for users. Evaluating information successfully requires critical thinking and may include the ability to verify the credibility of various news sources, or the capacity to comprehend and isolate the necessary data for a specific task, for example.
Managing information and data refers to the ability to organise and store various types of digital information (ACARA, 2015[82]).It involves the ability to adopt and develop systems for organising and classifying information in such a way that the information can be retrieved and reused efficiently (Fraillon, Schulz and Ainley, 2013[1]). Managing information successfully requires understanding the properties of different organisational structures in relation to the way in which the information will be used eventually. This component incorporates policies and procedures for centrally managing and sharing information among different individuals, organisations and information systems.
Competency area 2: Sharing information and communicating
Sharing information and communicating refers to one’s ability to exchange information, share knowledge, and customise such communication for a specific audience, context and medium (Fraillon, Schulz and Ainley, 2013[1]; ACARA, 2015[82]). This includes detailed knowledge regarding the real and digital contexts in which information are shared and to whom and thus require awareness about the specificities of ICT-based communication platforms available, including e-mail, instant messaging and group chat, media sharing and social-networking websites, among others. Given the wide range of use of ICT for communication, effective communication would require a thorough understanding of information-based social conventions, and the ability to adapt and modify selected modes of communication for the intended recipient(s).
Competency area 3: Transforming and creating information and digital content
Transforming and creating information involves the use of ICT and ICT-based data, digital content and information to develop new information or knowledge. Successful individuals can take existing information and derive new understandings by adapting, applying, designing, inventing or authoring (Fraillon, Schulz and Ainley, 2013[1]). Individuals may transform information with ICT, either to produce or expand upon existing information, by modifying its presentation for improved understanding in specific contexts. This process often requires the ability to use ICT-based formatting, graphics and multimedia to simplify and enhance the communication of information. Information transformation and creation are also related to the quality of information, specifically with regards to how structure, layout and design are used to support overall comprehension.
Competency area 4: Problem-solving in a digital context and computational thinking
PISA 2012 defines problem solving as individuals’ capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. It includes the willingness to engage with such situations in order to achieve one’s potential as a constructive and reflective citizen (OECD, 2013[2]). In the context of ICT literacy, the focus should be on solving technical problems, identifying technical responses and solutions and creatively using digital technologies to solve a problem. The main cognitive processes involved when solving a problem individually include exploring a problem situation (e.g. observing and interacting with the situation and searching for information, limitations and obstacles) and understanding the information and relevant concepts, representing and formulating (which refers to building a coherent mental representation and hypotheses), planning and executing (which consists of setting goals, devising and strategy and carrying it out), and finally monitoring progress and reflecting on solutions (OECD, 2013[2]).
Following PISA 2015, collaborative problem-solving competency can be defined as “an individual capacity to engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge skills and efforts to reach that solution” (OECD, 2017[4]). Collaborative problem solving is particularly relevant in the context of ICT not only because digital technologies have multiplied collaborative possibilities in society and work settings but also because the digital economy is in many regards also a collaborative economy that would benefit from people’s and institutions’ abilities to collaborate. Collaborative problem solving in a digital environment includes the cognitive components of individual problem solving but requires additional cognitive, social and technical skills to ensure a shared understanding and information flow, to appropriately use digital resources in order to create and understand an appropriate team organisation, and to perform co-ordinated actions to solve the problem (OECD, 2017[4]).
According to ICILS 2018, computational thinking can be defined as the “ability to identify a problem, break it down into manageable steps, work out the important details or patterns, shape possible solutions and present these solutions in a way that a computer, human or both can understand” (IEA, 2017[81]). Although computational thinking and problem solving in a digital environment strongly overlap and share many thoughts processes, one key difference can be that computational thinking focuses on how to rely on digital and computing possibilities to solve problems and carry out solutions. Indeed, a recent study highlights that “computational thinking is a problem solving methodology that expands the realm of computer science into all disciplines, providing a distinct means of analysing and developing solutions to problems that can be solved computationally” (ACM et al., 2016[83]). According to the Computer Science Teachers Association (CSTA) and the International Society for Technology in Education, the assessment of computational thinking could focus on the following key processes (Bocconi et al., 2016[84]):
“formulating problems in a way that enables us to use a computer and other tools to help solve them
logically organizing and analysing data
representing data through abstractions such as models and simulations
automating solutions through algorithmic thinking (a series of ordered steps)
identifying, analysing and implementing possible solutions with the goal of achieving the most efficient and effective combination of steps and resources
generalising and transferring this problem-solving process to a wide variety of problems.”
Competency area 5: Appropriate use of ICT (online security, safety and risk awareness and skills)
Online safety and security issues incorporate the appropriate use of ICT across multiple contexts and platforms. Using ICT appropriately requires making critical and thorough assessments of ICT use while considering the social, legal and ethical issues in different settings (Fraillon et al., 2015[5]). With increased information sharing, students must be aware of methods for handling and protecting personal information. Basic knowledge of security, including the use of strong passwords, preventative measures against viruses and protection of private information, partially overlaps with technical ICT skills.
The interest of policy makers in assessing students’ safety and security competencies in ICT is reflected in the wide inclusion of such competencies in school curricula across education systems (EC, 2013). As ICT use in and outside of the classroom becomes more common, a wide variety of online safety issues is being included in school curricula. These online safety courses have covered a variety of topics, including safe online behaviour, privacy, cyberbullying, downloading, copyright, safe use of mobile phones, and contact with strangers (EACEA, 2011[13]). This trend emphasises the growing importance of integrating online safety and security practices with ICT instruction, in addition to understanding existing knowledge of safety and security issues regarding ICT access and use.
Students’ attitudes and dispositions towards ICT
The assessment of students’ (and potentially parents’) ICT-related attitudes and dispositions rely extensively on existing measures developed for previous PISA cycles. More precisely, it follows the PISA 2022 taxonomy, which revolves around six dimensions: attitudes; values and beliefs; task performance; emotional regulation; collaboration; and open-mindedness and engagement with others. It also includes one compound construct that draws on several aspects of the different dimensions (OECD, 2018b).
Although all dimensions listed above are not equally relevant to ICT, they are related to ICT literacy in two different ways. First, students’ attitudes, behaviours, beliefs and aspirations related to ICT are likely to be correlated with ICT literacy and students’ ability to use ICT for learning and leisure. This suggests exploring how students feel or behave when using ICT in general, but also when using ICT in specific contexts, including learning and leisure.
Second, the use of ICT for teaching and learning can alter students’ attitudes and dispositions towards learning in general or for a specific subject. This is often one of the reasons why ICT are used for learning. These two relationships between students’ ICT use, and attitudes and dispositions are investigated in the following ways:
Self-efficacy refers to students’ beliefs regarding their ability to execute a specific task or to achieve a given goal. A related construct is self-concept, which corresponds to students’ global judgement of how they perceive their abilities in relation to a particular domain. Research suggests that higher levels of ICT self-efficacy are associated with higher levels of learning outcomes (Fraillon et al., 2014[85]). In the absence of a proper assessment of ICT competencies, self-efficacy constitutes the primary source of information about students’ ICT skills. It would therefore be of great value to ask students to evaluate their abilities based on a set of tasks and situations that reflect the five competency areas mentioned above: accessing, evaluating and managing information and data, sharing information and communicating, transforming and creating digital content, problem solving and computational thinking, and knowledge, skills and behaviours related to online security, safety and risks.
Interest, enjoyment and intrinsic motivation in a particular subject are shown to be positively correlated with learners’ achievement in general. Results from the International Computer and Information Literacy Study (ICILS) 2013 suggest similar conclusions for ICT. Indeed, ICILS 2013 reveals positive associations between students’ interest and enjoyment in working with computers and ICT literacy (Fraillon et al., 2014[85]). Students’ interest and motivation should be assessed with reference to a set of tasks representing different levels and types of competences with ICT.
In parallel, research suggests that ICT use in the classroom can also affect students’ motivation and interest in learning a specific subject (Lajoie and Azevedo, 2015[86]). This suggests that students’ abilities to use ICT for learning can be measured by assessing students’ motivation and interest, when confronted with a set of ICT-based tasks, related to a given subject (i.e. mathematics, reading and science).
Emotional regulation and task performance cover aspects of students’ emotions and emotional control (i.e. their capacity to curb anxiety, handle stress, develop and maintain positive expectations, etc.), and aspects related to students’ diligence and commitment, including setting high standards, working hard and avoiding distractions (Kankaraš and Suarez-Alvarez, 2019[87]). Knowing whether students are anxious or stressed when using ICT, and whether they are committed to understanding how to conduct specific tasks with ICT in different contexts would provide insights into their abilities to use ICT, particularly for learning purposes. It would also be interesting to document whether students’ emotional control and subject-specific task performance change when using ICT resources for learning. This might require the inclusion of ICT-based tasks for a specific subject, for example.
In addition, emotional regulation and task performance could be developed to document students’ potentially risky behaviours with ICT. Notably, aspects related to students’ self-control, dependence and abilities to regulate their engagement in specific ICT activities would reveal students’ capacity to cope with addiction and overuse of ICT. This could be complemented by measures of students’ perceptions about responsible use of ICT, as well as their sense of responsibility and awareness of security with regards to digital content and practices.
Collaboration, open-mindedness and students’ engagement with others cover students’ approaches to connecting with other people and the perceived value of those connections; openness to new experiences, perspectives and eagerness to learn and experience; and enjoyment of social connections and assertiveness in voicing their own views (OECD, 2018[56]). Many of these aspects are particularly relevant to the context of ICT use for learning in the classroom. Indeed, documenting students’ approaches toward collaboration with peers to solve problems using ICT resources, or collecting information on students’ willingness to engage in enquiry-based learning activities with ICT, would provide useful information on students’ “readiness” to use ICT. Moreover, measures of students’ open-mindedness and extraversion can help document their use of ICT for leisure and, in particular, social media. For example, students’ interactions with peers on social networks, as well as their abilities to voice and consider opposing views could identify students’ social inclusion and well-being.
Metacognition refers to students’ knowledge of learning strategies for a specific subject. For example, metacognition in reading refers to students’ awareness and ability to use a variety of appropriate strategies when processing texts in a goal-oriented manner (OECD, 2009[88]). Metacognitive reading strategies have been positively associated with students’ reading proficiency (Waters and Schneider, 2010[89]; OECD, 2017[4]). When ICT serves as a mean to learn reading, science or mathematics, students face new learning strategies and practices. Thus, it seems important to document students’ awareness about the effectiveness of ICT-based learning strategies. Indeed, metacognition appears to be even more important for digital reading literacy that requires “efficient and specific self-regulated strategies” (Coiro, Julie and Dobler, Elizabeth, 2007[90]). In light of the analogies between ICT literacy and reading literacy, which are both a means to deliver instruction and an end in themselves, it seems promising to document metacognition in ICT literacy itself, focusing on students’ understanding and awareness when learning ICT skills.
References
[82] ACARA (2015), National Assessment Program – ICT Literacy Years 6 & 10 Assessment Framework 2014, Australian Curriculum and Reporting Authority (ACARA), Sydney., https://www.nap.edu.au/_resources/NAP-ICT_Assessment_Framework_2014.pdf.
[83] ACM et al. (2016), “K–12 Computer Science Framework”, Science.
[77] Almlund, M. et al. (2011), “Personality Psychology and Economics”, in Handbook of The Economics of Education, Handbook of the Economics of Education, Elsevier, https://doi.org/10.1016/b978-0-444-53444-6.00001-8.
[39] Barak, M., A. Lipson and S. Lerman (2006), “Wireless laptops as means: For promoting active learning in large lecture halls”, Journal of Research on Technology in Education, Vol. 38/3, https://doi.org/10.1080/15391523.2006.10782459.
[79] Bertling, J., F. Borgonovi and D. Almonte (2016), “Psychosocial Skills in Large-Scale Assessments: Trends, Challenges, and Policy Implications”, https://doi.org/10.1007/978-3-319-28606-8_14.
[84] Bocconi, S. et al. (2016), Developing Computational Thinking in Compulsory Education - Implications for policy and practice.
[51] Brummelhuis, T. and A. Binda (2017), “Vier in balans-monitor 2017: de hoofdlijn”, Kennisnet.
[17] Bulman, G. and R. Fairlie (2016), “Technology and Education: Computers, Software, and the Internet”, in Handbook of the Economics of Education, https://doi.org/10.1016/B978-0-444-63459-7.00005-1.
[26] Bundsgaard, J. and T. Hansen (2011), “Evaluation of learning materials: A holistic framework”, Journal of Learning Design, Vol. 4/4, https://doi.org/10.5204/jld.v4i4.87.
[6] Carretero, S., R. Vuorikari and Y. Punie (2017), DigComp 2.1: The Digital Competence Framework for Citizens. With eight proficiency levels and examples of use, EUR 28558 EN.
[90] Coiro, Julie and Dobler, Elizabeth (2007), “Exploring the online reading comprehension strategies used by sixth-grade skilled readers to search for and locate information on the Internet”, Reading Research Quarterly, Vol. 42/2.
[27] Condie, R. and B. Munro (2007), “The impact of ICT in schools - a landscape review”, Becta January.
[15] Conrads, J. (2017), Digital Education Policies in Europe and Beyond: Key Design Principles for More Effective Policies, Redecker, C., P. Kampylis, M. Bacigalupo, Y. Punie (ed.), https://doi.org/10.2760/462941.
[72] Cuny, J., L. Snyder and J. Wing (2010), Demystifying computational thinking for non-computer scientists., https://www.cs.cmu.edu/~CompThink/resources/TheLinkWing.pdf (accessed 30 July 2018).
[70] Currie, C. et al. (2012), “Social determinants of health and well-being among young pople”, Health Behaviour School-Aged Children (HSBC) Study: International Report From the 2009/2010 Survey 6.
[31] Dinarski, M. et al. (2007), Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort. Report to Congress.
[67] Dragiewicz, M. et al. (2018), “Online Harassment 2017 | Pew Research Center”, Feminist Media Studies, Vol. 18.
[40] Driver, M. (2002), “Exploring student perceptions of group interaction and class satisfaction in the web-enhanced classroom”, The Internet and Higher Education, Vol. 5/1, pp. 35-45, https://doi.org/10.1016/s1096-7516(01)00076-8.
[13] EACEA (2011), Key Data on Learning and Innovation through ICT at School in Europe 2011, https://doi.org/10.2797/61068.
[93] EADSNE (2013), Information and Communication Technology for Inclusion – Developments and Opportunities for European Countries.
[74] Edwards, L. and S. Lopez (2006), “Perceived family support, acculturation, and life satisfaction in mexican american youth: A mixed-methods exploration.”, Journal of Counseling Psychology, Vol. 53/3, pp. 279-287, https://doi.org/10.1037/0022-0167.53.3.279.
[18] Escueta, M. et al. (2017), “Education Technology: An Evidence-Based Review.”, University Education Technology.
[9] European Commission (2013), Survey of Schools: ICT in Education - Benchmarking Access, Use and Attitudes to Technology in Europe’s Schools, https://doi.org/10.2759/94499.
[25] Faber, B., R. Sanchis-Guarner and F. Weinhardt (2015), “Ict and Education: Evidence From Student Home Addresses”, NBER Lecture notes.
[57] Faber, J., H. Luyten and A. Visscher (2017), “The effects of a digital formative assessment tool on mathematics achievement and student motivation: Results of a randomized experiment”, Computers & Education, Vol. 106, pp. 83-96, https://doi.org/10.1016/j.compedu.2016.12.001.
[19] Fairlie, R. (2012), “Academic achievement, technology and race: Experimental evidence”, Economics of Education Review, Vol. 31/5, pp. 663-679, https://doi.org/10.1016/j.econedurev.2012.04.003.
[23] Fairlie, R. and A. Kalil (2017), “The effects of computers on children’s social development and school participation: Evidence from a randomized control experiment”, Economics of Education Review, Vol. 57, pp. 10-19, https://doi.org/10.1016/j.econedurev.2017.01.001.
[22] Fairlie, R. and R. London (2012), “The Effects of Home Computers on Educational Outcomes: Evidence from a Field Experiment with Community College Students”, Economic Journal, Vol. 122/561, https://doi.org/10.1111/j.1468-0297.2011.02484.x.
[41] Finn, S. and J. Inman (2004), “Digital unity and digital divide: Surveying alumni to study effects of a campus laptop initiative”, Journal of Research on Technology in Education, Vol. 36/3, https://doi.org/10.1080/15391523.2004.10782417.
[7] Fraillon, J. et al. (2014), Preparing for life in a digital age: The IEA international computer and information literacy study international report.
[85] Fraillon, J. et al. (2014), Preparing for Life in a Digital Age, Springer International Publishing, Cham, https://doi.org/10.1007/978-3-319-14222-7.
[1] Fraillon, J., W. Schulz and J. Ainley (2013), International computer and information literacy study: assessment framework..
[5] Fraillon, J. et al. (2015), National Assessment Program: ICT Literacy Years 6 & 10 Report 2014.
[54] GWI (2017), GWI Social. GlobalWebIndex’s flagship report on the latest trends in social media.
[78] Heckman, J., J. Stixrud and S. Urzua (2006), “The effects of cognitive and noncognitive abilities on labor market outcomes and social behavior”, Journal of Labor Economics, Vol. 24/3, https://doi.org/10.1086/504455.
[33] HENNESSY, S., K. RUTHVEN and S. BRINDLEY (2005), “Teacher perspectives on integrating ICT into subject teaching: commitment, constraints, caution, and change”, Journal of Curriculum Studies, Vol. 37/2, pp. 155-192, https://doi.org/10.1080/0022027032000276961.
[55] Hooft Graafland, J. (2018), “New technologies and 21st century children: Recent trends and outcomes”, OECD Education Working Papers, No. 179, OECD Publishing, Paris, https://doi.org/10.1787/e071a505-en.
[58] Hunsu, N., O. Adesope and D. Bayly (2016), “A meta-analysis of the effects of audience response systems (clicker-based technologies) on cognition and affect”, Computers & Education, Vol. 94, pp. 102-119, https://doi.org/10.1016/j.compedu.2015.11.013.
[65] ICT Literacy Panel (2002), “Digital Transformation: A Framework for ICT Literacy. A Report of the International ICT Literacy Panel”, Educational Testing.
[81] IEA (2017), What’s next for IEA’s ICILS in 2018?, The IEA’s International Computer and Information Literacy Study (ICILS) 2018, International Association for the Evaluation of Educational Achievement (IEA), Amsterdam.
[21] Jackson, K. and A. Makarin (2018), “Can online off-the-shelf lessons improve student outcomes? Evidence from a field experiment”, American Economic Journal: Economic Policy, Vol. 10/3, https://doi.org/10.1257/pol.20170211.
[87] Kankaraš, M. and J. Suarez-Alvarez (2019), “Assessment framework of the OECD Study on Social and Emotional Skills”, OECD Education Working Papers, No. 207, OECD Publishing, Paris, https://doi.org/10.1787/5007adef-en.
[30] Karam, R. et al. (2016), “Examining the implementation of technology-based blended algebra I curriculum at scale”, Educational Technology Research and Development, Vol. 65/2, pp. 399-425, https://doi.org/10.1007/s11423-016-9498-6.
[73] Kintsch, W. (1998), Comprehension: A paradigm for cognition, Cambridge University Press, Cambridge, MA.
[12] Korte, W. and T. Hüsing (2006), “Benchmarking access and use of ICT in European schools 2006: Results from Head Teacher and A Classroom Teacher Surveys in 27 European countries”, Current Developments in Technology-Assisted Education, Vol. 3/October.
[43] Kuznekoff, J. and S. Titsworth (2013), “The Impact of Mobile Phone Usage on Student Learning”, Communication Education, Vol. 62/3, pp. 233-252, https://doi.org/10.1080/03634523.2013.767917.
[86] Lajoie, S. and R. Azevedo (2015), “Teaching and Learning in Technology-Rich Environments”, in Handbook of Educational Psychology, https://doi.org/10.4324/9780203874790.ch35.
[52] Law, N., T. Plomp and W. Pelgrum (2006), “Pedagogy and ICT use in schools around the world: Findings from the IEA Sites 2006 study”, CERC Studies in Comparative Education, Vol. 23.
[94] Lazowski, R. (2015), A Meta-Analytic Tutorial and a Narrative Review on Motivation Interventions in Education.
[59] Lazowski, R. and C. Hulleman (2016), “Motivation Interventions in Education”, Review of Educational Research, Vol. 86/2, pp. 602-640, https://doi.org/10.3102/0034654315617832.
[80] Lennon, M. et al. (2003), “Feasibility Study for the PISA ICT Literacy Assessment: Report to Network A.”, Educational Testing Service.
[24] Leuven, E. et al. (2007), “The effect of extra funding for disadvantaged pupils on achievement”, Review of Economics and Statistics, Vol. 89/4, https://doi.org/10.1162/rest.89.4.721.
[62] Levine, S. et al. (2010), “What counts in the development of young children’s number knowledge?”, Developmental Psychology, Vol. 46/5, pp. 1309-1319, https://doi.org/10.1037/a0019671.
[66] Livingstone, S. et al. (2009), “The perspective of European children Risks and safety on the internet”, International Journal of Media and Cultural Politics, Vol. 6/1.
[20] Malamud, O. and C. Pop-Eleches (2011), “Home Computer Use and the Development of Human Capital *”, The Quarterly Journal of Economics, Vol. 126/2, pp. 987-1027, https://doi.org/10.1093/qje/qjr008.
[68] Mascheroni, G. and K. Ólafsson (2018), Access and use, risks and opportunities of the internet for Italian children. Preliminary findings, http://www.eukidsonline.net (accessed on 25 July 2018).
[38] Mccoy, B. (2016), “Digital Distractions in the Classroom Phase II: Student Classroom Use of Digital Devices for Non- Class Related Purposes Bernard”, Journalism and Mass Communications, Vol. 4/4.
[76] Melorose, J., R. Perroy and S. Careas (2008), COMMISSION STAFF WORKING DOCUMENT: The use of ICT to support innovation and lifelong learning for all - A report on progress.
[60] Muis, K. et al. (2015), “The effects of technology-mediated immediate feedback on kindergarten students’ attitudes, emotions, engagement and learning outcomes during literacy skills development”, Learning and Instruction, Vol. 38, pp. 1-13, https://doi.org/10.1016/j.learninstruc.2015.02.001.
[71] National Sleep Foundation (2011), 2011 Sleep in America Poll: Communications Technology in the Bedroom, https://sleepfoundation.org/sites/default/files/sleepinamericapoll/SIAP_2011_Summary_of_Findings.pdf.
[91] OECD (2019), PISA 2018 Assessment and Analytical Framework, PISA, OECD Publishing, Paris, https://doi.org/10.1787/b25efab8-en.
[56] OECD (2018), PISA 2021 Mathematics Framework (First Draft).
[10] OECD (2018), “The OECD PISA Global Competence Framework: Preparing our Youth for an Inclusive and Sustainable World.”, Oecd.
[4] OECD (2017), PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic, Financial Literacy and Collaborative Problem Solving, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264281820-en.
[47] OECD (2017), PISA 2015 Results (Volume III): Students’ Well-Being, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264273856-en.
[8] OECD (2016), “New Skills for the Digital Economy: Measuring the demand and supply of ICT skills at work”, OECD Digital Economy Papers, No. 258, OECD Publishing, Paris, https://doi.org/10.1787/5jlwnkm2fc9x-en.
[3] OECD (2016), PISA 2018 Draft Analytical Framework, http://www.oecd.org/pisa/data/PISA-2018-draft-frameworks.pdf.
[11] OECD (2015), Students, Computers and Learning: Making the Connection, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264239555-en.
[35] OECD (2014), TALIS 2013 Results: An International Perspective on Teaching and Learning, TALIS, OECD Publishing, Paris, https://doi.org/10.1787/9789264196261-en.
[2] OECD (2013), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264190511-en.
[88] OECD (2009), PISA 2009 Assessment Farmework – Key Competencies in Reading, Mathematics and Science, OECD Publishing, Paris., https://www.oecd.org/pisa/pisaproducts/44455820.pdf.
[16] OECD Publishing, P. (ed.) (2015), Schools for 21st-century learners : strong leaders, confident teachers, innovative approaches,, https://trove.nla.gov.au/work/197169112?q&versionId=215915634+252445919 (accessed on 25 July 2018).
[29] Pane, J. et al. (2014), “Effectiveness of Cognitive Tutor Algebra I at Scale”, Educational Evaluation and Policy Analysis, Vol. 36/2, pp. 127-144, https://doi.org/10.3102/0162373713507480.
[46] Pashler, H. (1994), “Dual-task interference in simple tasks: Data and theory.”, Psychological Bulletin, Vol. 116/2, pp. 220-244, https://doi.org/10.1037/0033-2909.116.2.220.
[53] Pew Research Center (2015), Teen, Social Media and Technology Overview 2015,, http://www.pewinternet.org/2015/04/09/teens-social-media-technology-2015.
[45] Posner, M. (1982), “Cumulative development of attentional theory.”, American Psychologist, Vol. 37/2, pp. 168-179, https://doi.org/10.1037/0003-066x.37.2.168.
[32] Redecker, C. (2017), European framework for the digital competence of educators: DigCompEdu, https://doi.org/10.2760/159770.
[48] Ronfeldt, M. et al. (2015), “Teacher Collaboration in Instructional Teams and Student Achievement”, American Educational Research Journal, Vol. 52/3, pp. 475-514, https://doi.org/10.3102/0002831215585562.
[28] Roschelle, J. et al. (2016), “Online Mathematics Homework Increases Student Achievement”, AERA Open, Vol. 2/4, p. 233285841667396, https://doi.org/10.1177/2332858416673968.
[44] Sana, F., T. Weston and N. Cepeda (2013), “Laptop multitasking hinders classroom learning for both users and nearby peers”, Computers & Education, Vol. 62, pp. 24-31, https://doi.org/10.1016/j.compedu.2012.10.003.
[36] Scherff, L. and C. Piazza (2008), “Why Now, More Than Ever, We Need to Talk About Opportunity to Learn”, Journal of Adolescent & Adult Literacy, Vol. 52/4, pp. 343-352, https://doi.org/10.1598/jaal.52.4.7.
[37] Schmidt, W. and A. Maier (2009), “Opportunity to Learn: Does California’s School System Measure Up?”, Handbook on Education Policy Research.
[63] Senechal, M. and J. LeFevre (2002), “Parental Involvement in the Development of Children’s Reading Skill: A Five-Year Longitudinal Study”, Child Development, Vol. 73/2, pp. 445-460, https://doi.org/10.1111/1467-8624.00417.
[69] Smith, P. et al. (2008), “Cyberbullying: its nature and impact in secondary school pupils”, Journal of Child Psychology and Psychiatry, Vol. 49/4, pp. 376-385, https://doi.org/10.1111/j.1469-7610.2007.01846.x.
[75] Suldo, S. et al. (2014), “American High School Students’ Perceptions of Determinants of Life Satisfaction”, Social Indicators Research, Vol. 118/2, https://doi.org/10.1007/s11205-013-0436-2.
[42] Tindell, D. and R. Bohlander (2012), “The Use and Abuse of Cell Phones and Text Messaging in the Classroom: A Survey of College Students”, College Teaching, Vol. 60/1, pp. 1-9, https://doi.org/10.1080/87567555.2011.604802.
[34] Trucano, M. (2005), “Knowledge Maps : Ict in Education Knowledge Maps : Ict in Education”, Education.
[92] Unesco (2011), ICTs in Education For People with Disabilities: Review of innovative practice.
[61] UNICEF (2017), State of the Worlds Children 2017 - Children in a Digital World.
[64] van Deursen, A. and E. Helsper (2015), “A nuanced understanding of Internet use and non-use among the elderly”, European Journal of Communication, Vol. 30/2, pp. 171-187, https://doi.org/10.1177/0267323115578059.
[49] Vieluf, S. et al. (2012), Teaching Practices and Pedagogical Innovations: Evidence from TALIS, TALIS, OECD Publishing, Paris, https://doi.org/10.1787/9789264123540-en.
[50] Wastiau, P. et al. (2013), “The Use of ICT in Education: a survey of schools in Europe”, European Journal of Education, Vol. 48/1, pp. 11-27, https://doi.org/10.1111/ejed.12020.
[89] Waters, H. and W. Schneider (2010), Metacognition, strategy use, and instruction, Guilford Press, New York, NY.
[14] World Bank (2018), World Development Indicators, The World Bank, https://datacatalog.worldbank.org/dataset/world-development-indicators, (accessed on 12 June 2018).
Notes
← 1. The framework distinguishes between student use of ICT resources during classroom lessons (and therefore under the supervision of at least one teacher) and ICT use outside of the classroom, which includes ICT use at home and ICT use outside of class but not at home (whether in a school computer lab, a library, or at any other location except home). For the sake of simplicity, ICT use subsumes all of the above situations in the remainder of this text, unless otherwise noted.
← 2. Although parents’ role in facilitating and shaping students’ access to and use of ICT resources is well documented, the potential absence of a parent questionnaire in PISA 2022 might restrict the ability to cover these aspects.
← 3. Global competence encompasses multiple dimensions. A globally competent student can examine local, global and intercultural issues, understand and appreciate different perspectives and world views, interact successfully and respectfully with others, and take responsible action toward sustainability and collective well-being (OECD, 2018[10]).
← 4. Advantaged/Disadvantaged students are students in the top/bottom quarter of the PISA index of economic, social and cultural status.
← 5. Bundsgaard and Hansen (2011[26]) actually refer to functional, semantic and “didacticised” learning materials. In their terms, “Functional learning materials (tools) characterised by their facilitation of learning and teaching” include black and white boards, computer applications, projectors, and mobile phones. “Semantic learning materials (texts) characterised by their meaning as constituted by signs and semantic references” correspond to film, literature, etc. Finally, “Didacticised learning materials, characterised by combining tools and text and facilitating learning and teaching, include textbooks, online teaching materials and educational games”.
← 6. Note that the discussion leans towards the quality of ICT resources for learning, but similar dimensions can be used to uncover the quality of ICT resources for leisure. Moreover, aspects related to the intrinsic relevance and suitability of ICT resources when used for teaching are not covered here.
← 7. In this case, the nature of instruction also shifts from teacher-supervised learning with ICT to unsupervised learning with ICT.