This document presents the framework for the background questionnaires for the 2018 cycle of the Programme for International Student Assessment (PISA). These questionnaires are, to a large extent, developed from ones that were used in previous cycles of PISA and, as such, permit the monitoring of trends in student-, school-, and system-level factors that may be related to student outcomes. A variety of constructs are discussed, including student background constructs, schooling constructs, and non-cognitive/metacognitive constructs. The document also makes explicit the link to reading literacy, which is the major domain of this cycle of PISA. The relevance of each of the constructs to policy issues and student outcomes is also highlighted.
PISA 2018 Assessment and Analytical Framework
7. PISA 2018 Questionnaire Framework
Abstract
Introduction
Outline of the framework
This document provides the framework for the development of the background questionnaire. It defines all major constructs, meaning what is measured by one or more items of a scale, that will be assessed in the background questionnaires of PISA 2018, including student background constructs, school-level constructs and non-cognitive and metacognitive constructs. The framework also discusses the current literature on these constructs. However, the framework focusses on the why and how of assessing constructs rather than on the relationships between the constructs, which has been addressed in many previous publications of PISA results. A wide variety of theoretical models have been proposed to link systemic or curriculum variables (e.g. instructional approaches and educational expenditure) to student achievement. However, a “yield study” such as PISA in which a set of tests is administered to a group of students only once without any follow‑up has only a limited scope for causal analysis. Therefore, the emphasis in the present framework is on identifying constructs that are of interest in either previous PISA cycles or the current literature, rather than on specifying their links.
The document is organised into two main parts: (1) defining the core content of the PISA questionnaires and elaborating its modular structure, and (2) explaining the policy issues that the questionnaires cover. Detailed references to current research are provided throughout the document.
The first part of this document links the current framework to the overarching (cross-cycle) structure of previous PISA assessments and questionnaires, as set out in the PISA 2012 and 2015 frameworks (Klieme et al., 2013[1]; Klieme and Kuger, 2014[2]; OECD, 2013, p. 168[3]). The constructs that need to be covered for monitoring trends in education are discussed in the context of research into the effectiveness of education systems. These measures have been used previously in PISA reports, as international indicators (published in Education at a Glance) and in secondary analyses.
The second part of this document explores the in-depth policy issues covered in PISA 2018. These issues are organized by modules, which comprise one or more related constructs (assessed by items or scales); for example, the module on domain-general student attitudes and behaviours is composed of various scales, such as self-related beliefs and attitudes towards school, well-being and the utilisation of information and communications technology (ICT). Additionally, this part of the document explains how the modules were implemented in the PISA 2018.
Some newly developed questions, spread across various modules, will be tested in the 2018 field trial, providing a broad set of measures that can be used in the PISA 2018 main study and/or in later cycles. Modules, constructs, questions and items will be selected for inclusion in the PISA 2018 main study based on the results from the field trial.
Defining the questionnaire core in PISA 2018
One of the major features of the implementation of PISA is the cyclical change in focus of the cognitive assessment: reading literacy was the major domain of assessment in PISA 2000 and 2009 and is so again in PISA 2018, whilst mathematics was the major domain of PISA 2003 and 2012 and science in PISA 2006 and 2015. The major domain of cognitive assessment is also the focus of domain-specific context assessment in the associated questionnaire – in other words, various reading-related constructs are assessed in the 2018 questionnaire since reading is the major domain. However, there is also a need for stability in measures administered in different waves in order to gauge and understand trends in education. Stability has to be considered at two levels: across waves of three years (various questions in the questionnaires tend to recur in every cycle) and in subject-specific constructs across waves of nine years (reading-specific constructs assessed in the 2009 wave could be reused in 2018).
The questionnaire framework first established for PISA 2012 and continued for PISA 2015 identifies core questionnaire content that should be kept comparable across cycles (OECD, 2013, p. 189[3]) to allow for the continuous monitoring of education systems and the establishment of valid and reliable trends. This includes both domain-specific and cross-domain measures that assess the conditions, processes and outcomes of education, at the level of both the student and the school.
This overarching framework, which specifies the constructs and measures in more detail and provides arguments that support the choice of core content for PISA 2018, is described below.
Outline of the content covered in the questionnaires
The green columns on the left-hand side of Figure 7.1 (student background constructs; modules 5 to 8) summarise students’ family background and the education they have received to date. The items associated with these columns are typically asked of students or parents. The constructs in the blue columns in the middle of Figure 7.1 (modules 1 to 3 and 11 to 16), refer to educational processes on different levels (system, school and classroom). Most of the questions associated with these columns are answered by schools but some may be answered by students or parents. The last grey column on the right of Figure 7.1 (modules 4, 9 and 10) asks students about various non-cognitive and metacognitive (strategy awareness) constructs. The upper half mainly deals with domain-specific (in this case, reading-related) topics, while the lower half of the figure deals with general topics not focusing on a specific domain or those domains other than reading.
Every module represents a focus of policy making, and the set of 16 modules covers a wide and comprehensive array of policy issues that are relevant across countries. Indeed, most topics treated by Sykes, Schneider, and Plank (2009[4]) and by the OECD (2015[5]) in their reviews of educational policy research are covered here. This framework first discusses non-cognitive and metacognitive constructs, followed by student background constructs, teaching and learning constructs, and finally school policy and governance constructs.
PISA treats the mandatory core questionnaires (school questionnaire and student questionnaire) separately from the optional questionnaires, which countries may opt out of. The framework attempts to make the connections among the questionnaires as transparent as possible. It also describes what is conceptually covered in each questionnaire, which constructs are examined at the student and at the school levels, and who responds to each individual question.
Reading as the major domain
A new reading framework has been developed for PISA 2018. While this new framework shares many similarities with the 2000 and 2009 frameworks, it has been reconceptualised to address the main differences between print and online reading (Afflerbach and Cho, 2010[6]). In online reading, the text is not given: the reader has to build his/her own text, choosing which paths to follow and which ones to dismiss, in a context where the reader is offered many more options and opportunities in which to get lost. The PISA 2018 framework for reading literacy aims to address the additional complexities linked to online reading comprehension as defined by Coiro and Dobler (2007[7]), such as additional sources of prior knowledge (knowledge about search engines, website structures), a higher incidence of multilevel forward inferential reasoning (predicting what is behind a link), and new dimensions of self-regulated reading (the integration of physical actions such as clicking and scrolling down with cognitive processes such as predicting, assessing and evaluating the quality of information). All the reading-related constructs and questions should therefore cover both print and online reading.
Moreover, the new reading framework highlights the importance of metacognitive task management processes, such as setting goals and plans, monitoring and regulation.
Global competence
PISA 2018 introduces the new domain of global competence. This domain is seen as being critically important because our learning, working and living environments are becoming more global, interconnected and interdependent. Young people will encounter, actively engage with and help shape these environments, no matter where they are born, attend school, work or live. They therefore need to leave school equipped with the necessary knowledge, skills and attitudes that will enable them to learn, work and live in a globalised world, and with the tools to further develop these attributes as they move through life. In particular, this involves a knowledge of and interest in engaging with the world around them; a growing confidence and a spirit of curiosity, adventure, flexibility and resilience; and the communication and interaction skills necessary to make the most of the opportunities and challenges that fast-changing, interconnected and interdependent environments bring. Classrooms and schools should foster the value of and embrace the diversity of peoples, languages and cultures. Schools should also encourage intercultural sensitivity and help students move away from ethnocentric world views and beyond tolerance to acceptance, respect and appreciation. Students can be given the chance to engage in experiences that facilitate international and intercultural relationships, exchanges and conversations and should then reflect upon what they have learned from such experiences (Bennett, 1993[8]; Sinicrope, Norris and Watanabe, 2007[9]).
Several authors have signalled that because of the increasing speed of change in society (such as developments in information and communications technology (ICT) and the rise of cross-border working), schools need to adapt their curricula to account more precisely for what students will need in their future lives (Fisch and McLeod, 2009[10]). Furthermore, Anderson (2008[11]) points out that the knowledge and skills required to prosper in the 21st century go far beyond the traditional literacies. He identifies knowledge construction, adaptability, finding, organising and retrieving information, information management, critical thinking and teamwork as the skills demanded by modern societies. Meanwhile, Binkley et al. (2012[12]) maintain that achieving competence in 21st century skills, such as global competence, requires specific knowledge, skills, attitudes, values and ethics.
Constructs to be covered in the questionnaires
Since PISA began in 2000, the background questionnaires have served two interrelated purposes. The first purpose has been to provide a context through which to interpret scores from the cognitive assessment (both within and across education systems). The second purpose has been to provide reliable and valid non‑cognitive outcomes, which can inform policy and research in their own right. Over the six cycles of PISA to date, new non‑cognitive outcomes have emerged for both domain-specific and cross-domain features of education. The background questionnaire has also tracked developments in psychometric theory and survey research methodology so as to provide increasingly reliable and valid measures of non‑cognitive constructs that are not sensitive to cultural differences in response style. These developments have taken place while maintaining the ability to report trends across PISA cycles.
PISA 2018 is the seventh cycle of PISA and the third cycle where reading is the major domain of assessment. In addition, PISA 2018 will also introduce a new domain – global competence – while diminishing the distinctions among the major and minor domains. These goals will require additional questionnaire delivery time, and may therefore prompt a change in how the questionnaires are developed and designed.
This section of the questionnaire framework presents the constructs for PISA 2018 and is organised around: (1) non‑cognitive and metacognitive constructs; (2) student background constructs; (3) teaching and learning constructs; and (4) school policies and governance constructs.
Non‑cognitive and metacognitive constructs
PISA measures and documents the outcomes of education attained at the age of 15. Educating a student means fostering his or her individual development as a unique, self-determined, knowledgeable person who gradually gains in his or her ability to participate in society. As each PISA cycle is a cross-sectional study, it does not capture developmental processes in the same way that longitudinal studies can; rather, PISA serves as a snapshot of students’ developmental status at the age of 15. This includes achievement in cognitive areas such as reading, mathematics and science, all of which are measured in PISA; however, other outcomes are also important. Success in school – and in life – also depends on being committed to learning, respecting and understanding others, being motivated to learn and being able to regulate one’s own behaviour. These constructs can be perceived as prerequisites to learning, but they may themselves also be judged as goals of education, as elaborated in the OECD project Defining and Selecting Key Competencies (DeSeCo) (Rychen and Salganik, 2003[13]). Educational research has shown that non‑cognitive factors are very important for individual development as well as for success in life and well-being, and thus have an impact on individuals and society alike (Almlund et al., 2011[14]; Heckman, Stixrud and Urzua, 2006[15]).
Given the increasing importance of non‑cognitive outcomes, PISA complements the assessment of cognitive, learning-related behaviour (e.g. self-regulation, strategies and invested time) with non‑cognitive and metacognitive outcomes, such as attitudes, beliefs, motivation and aspirations, as measured primarily in the student questionnaire (but also in the school questionnaire). These outcomes may be of a general nature, such as the achievement motivation and well-being of students and the drop-out rates of schools, or related to the domains of the cognitive assessment, such as reading engagement, interest in mathematics or enjoyment of science. Domain-specific non‑cognitive outcomes are also mentioned in the respective definitions of literacy, so this array of constructs serves as a link between the cognitive frameworks and the questionnaire framework.
Student background
In order to understand learning outcomes, educational trajectories and equity issues within and across countries, one must take into account family background variables, such as socio-economic status (SES) and ethnic background.
PISA has become well known for its detailed, theory-based assessment of family background, SES and immigration background. Much effort has gone into the definition and operationalisation of individual student background indicators, leading to the establishment of a composite indicator for economic, social and cultural status, known as the ESCS (Willms, 2006[16]). The components of this indicator need to be assessed in as stable a way as possible across PISA cycles.
Furthermore, PISA gathers retrospective and prospective information about educational pathways. In recent years, researchers and the public debate in many countries have stressed the importance of early childhood education (Blau and Currie, 2006[17]; Cunha et al., 2006[18]). Therefore, PISA intends to collect at least some information on students’ participation in primary and pre-primary education, bearing in mind that, for the most part, this would be solicited from 15‑year‑olds, which could challenge the validity of the reports.
Beyond individual student background, the social, ethnic and academic composition of the school has an impact on students’ learning processes and outcomes. Therefore, PISA aggregates student data at the school level to characterise schools’ background factors. These are used in combination with structural factors, such as the location and size of a school.
Teaching and learning
School-based instruction is the core process of formal education. Therefore, policy makers need information on the teaching and learning that takes place in schools. To increase the explanatory power of the study, the assessment of teaching and learning will focus on the major domain of assessment, which in 2018 is reading, as well as on the innovative domain for 2018, global competence. Research on education effectiveness identifies the following core factors as possibly affecting students’ reading literacy: teachers’ qualifications, teaching practices, classroom climate, learning time and learning opportunities provided both within and outside of school (Creemers and Kyriakides, 2008[19]; Scheerens and Bosker, 1997[20]). Teaching practices are comprised of three basic dimensions (Klieme, Pauli and Reusser, 2009[21]): (i) structure and classroom management; (ii) teacher support; and (iii) cognitive challenge.
One challenge in addressing teacher- and teaching-related factors in PISA is that sampling is performed by age rather than by grade or class. Another challenge is linked to the reading domain itself. When students are 15 years old, reading is no longer taught as a standalone subject in the same way that mathematics and science are. However, reading literacy is still improved by teaching practices, and reading strategies are taught or learned through not only language arts and literature courses in the test language, but also through foreign language courses and social and natural science courses, known in their entirety as “content literacy” (McKenna and Robinson, 1990[22]; Shanahan and Shanahan, 2008[23]). While questions about teaching and learning mathematics and science can be, to a great extent, limited to solely mathematics and science lessons, there is clear evidence that rich and valuable information about reading (especially online reading) cannot be obtained solely from test language instruction lessons. Indeed, one of the most striking differences between countries in their reading curriculum is their emphasis on and time dedicated to content literacy, including the teaching reading in other subjects, (Lafontaine et al., 2017[24]). Consequently, any teacher questionnaire implemented in PISA 2018 investigating the teaching of reading literacy should be administered to a sample of teachers across domains, rather than only to test language teachers.
School policies and governance
Policy makers have only a limited direct impact on teaching and learning processes. Instead, much of their impact takes place via their influence on school-level factors that directly affect schools, and thus indirectly affect student learning. It is hence important to gather information on these school-level factors. As with teacher and teaching variables (Barile et al., 2012[25]), research has shown that “essential supports” at the level of the school promote school effectiveness (Bryk et al., 2010[26]; Chapman et al., 2011[27]). These essential supports comprise professional capacity with a focus on professional development; a well-organised curriculum; leadership and school management; parental involvement; an ambitious but nurturing school climate (clear norms and shared values, high achievement expectations, truthful and mutually supportive interactions between stakeholders); and the use of assessment and evaluation for improvement. These factors will be addressed within the PISA questionnaires as cross-domain processes on the school level. In addition, the questionnaires will cover school-level support for teaching the major domain, such as the provision of libraries, ICT equipment and a school curriculum for reading literacy, including multimodal aspects of reading in a digital era.
The PISA 2018 questionnaires also need to address issues related to governance at the system level (Hanushek and Woessmann, 2010[28]; Woessmann et al., 2007[29]). Student allocation, selection and evaluation are the basic processes that policy makers and/or school administrators use to control school quality and to monitor and foster school improvement. Some of this information can be obtained from other sources (as documented in OECD’s Education at a Glance), while other information can be assessed through the PISA school questionnaire.
Previous use of contextual data from PISA: Measures that have been and will be important for analysis and reporting
Comparability and trend items
In previous cycles, statistical analyses – in particular, exploratory and confirmatory factor analyses – were conducted to address whether identical underlying constructs were measured across all participating countries and whether scores could be compared across countries (OECD, 2012[30]; OECD, 2014[31]). In other words, these analyses attempted to determine whether constructs and scores were invariant across countries. Invariance issues will continue to be important issue in the analysis of questionnaire data.
An important asset of the PISA study is its use of trend items, or those that have been used in at least one previous round. As many trend items as possible will be retained in the 2018 questionnaire to compare 2018 data with those from previous rounds and to thereby conduct trend analyses.
The PISA 2009 report
The OECD combines PISA data from both the cognitive assessments and the questionnaires to create its various reports, and the use of this data in previous cycles helped decide which variables should be included in the 2018 study. This section describes the background variables used in the PISA 2009 Initial Report, so chosen as it was the last cycle during which reading was the major domain (as it is in 2018). In addition to student achievement, non‑cognitive outcomes, such as student engagement, cognitive strategies and metacognitive strategies, were studied in detail, and the impact of background variables and classroom-, school- and system-level factors was reported. Most of these were gathered through the student and school questionnaires. In more detail, the six volumes describing the PISA 2009 results used the following questionnaire data:
Volume I: What Students Know and Can Do: Student Performance in Reading, Mathematics and Science
Student background: gender
Volume II: Overcoming Social Background: Equity in Learning Opportunities and Outcomes
Student background: economic, social and cultural status (ESCS), gender, immigration status, language spoken at home, age of arrival, country of origin
Support for students assessed through parent questionnaire: parental support (at the beginning of primary education and at age 15), pre-primary education (attendance, quality)
Volume III: Learning to Learn: Student Engagement, Strategies, and Practices
Student background: ESCS, gender, immigration status, language spoken at home
Outcomes: enjoyment of reading, time and material used for reading, metacognition (awareness of strategies), self-reported use of reading strategies (memorisation, elaboration, control)
Volume IV: What Makes a School Successful? Resources, Policies, and Practices
Student background: socio-economic status
Student-reported factors: learning time (previous education, learning time at school, enrichment/remedial education, after-school lessons), teacher-student relationships, disciplinary climate, teacher’s stimulation of reading engagement
School- and system-level factors (as reported by the principal): type of school (public/private), number of programmes, class size, educational resources (e.g. ICT, library), school responsibility for assessment and curriculum and for resource allocation, extra-curricular activities available, age of school entry, grade repetition, school admittance/grouping/transfer policies, assessment practices/purposes, use of achievement data, school accountability, methods for monitoring teachers, teacher and student behaviour, parent involvement and expectations, leadership, school climate
Volume V: Learning Trends: Changes in Student Performance since 2000
Trends in student background variables: socio-economic status, immigration status, language spoken at home
Trends in non‑cognitive outcomes and schooling constructs: reading attitudes and practices (reading for pleasure, diversity of texts read, reading engagement, reading fiction), school climate indicators (teachers-student relationships, disciplinary climate)
Volume VI: Students On Line. Digital Technologies and Performance.
ICT familiarity (optional questionnaire): access to ICT at home and at school, use of ICT at home and at school, students’ attitudes towards and self-confidence in using computers, self-confidence in performing ICT tasks and activities, navigation indices extracted from log-file data (number of pages visited, number of relevant pages visited)
In PISA 2000, in addition to the main international report, an in-depth thematic report was dedicated to reading (Kirsch et al., 2002[32]).
As will be outlined below in more detail, most measures that were described in the PISA 2009 Initial Report are included among the 2018 instruments, thereby ensuring the opportunity to compare findings between 2009 and 2018.
Research publications
Numerous scientific research papers using PISA data can be found in the literature. Many papers discuss non‑cognitive, domain-specific outcomes: re‑scaling the questionnaire items, studying the structure of indices based on questionnaire items within and across countries, analysing outcomes across subgroups and across countries, examining the impact of student and family background and identifying and explaining school-level factors.
Coverage of policy issues in PISA 2018
A balance needed to be struck between the need for trend items and the need for new or changed constructs in PISA 2018. Where possible and sensible, constructs and modules were carried forward intact or with only minor changes. If measures were outdated, redundant, or did not comply with psychometric criteria (e.g. due to low internal consistency), they were recommended for deletion. Finally, two types of constructs were added: (1) those found throughout the research literature that have not previously been covered and (2) those relevant to the new domain of global competence. (Existing constructs were also extended to cover global competence or the new dimensions in the reading framework – specifically, online reading).
This section is divided into four subsections that group the modules into the larger constructs of: (1) assessing non‑cognitive and metacognitive constructs; (2) assessing student background; (3) assessing teaching and learning processes; and (4) assessing school policies and governance. The subsection on assessing non‑cognitive and metacognitive constructs contains the largest number of changes for 2018, as it includes the new domain of global competence in addition to reading-specific variables.
Assessing non‑cognitive and metacognitive constructs
This subsection summarises the conceptual foundations for modules 4 (reading-related outcomes: attitudes, motivation, attitudes and strategies), 9 (dispositional and school-focused variables), and 10 (dispositions for global competence, Figure 7.1).
PISA has traditionally considered only the results from cognitive achievement tests as student outcomes. Students’ motivations, attitudes, beliefs and behaviours were seen to be important precursors and predictors of scholastic performance, educational attainment and labour market success. However, non‑cognitive outcomes are increasingly considered to be important in their own right from the standpoint of both educational policy and labour market policy, because they are instrumental for personal growth, individual success, long-term achievement and society as a whole (Marsh et al., 2006[33]). Research has demonstrated the considerable power of non‑cognitive outcomes for success in secondary education, higher education and the workforce (Heckman, Stixrud and Urzua, 2006[15]; Lindqvist and Vestman, 2011[34]; Poropat, 2009[35]; Richardson, Abraham and Bond, 2012[36]; Shiner et al., 2007[37]). By collecting information on non‑cognitive outcomes, PISA can investigate the complex relationships (e.g. moderation or mediation) between non‑cognitive outcomes and achievement at the individual, school and country levels.
Previous PISA cycles have focused on domain-specific student attitudes and behaviours, such as interest in and motivation towards reading and mathematics; mathematics self-concept and mathematics anxiety; or knowledge of reading strategies (metacognition). Most of these attitudes and behaviours display robust relationships with student proficiency scores. Domain-specific student attitudes and behaviours are once again included in PISA, particularly, in module 4 (reading-related outcomes). In addition, the current framework includes a broader set of non‑cognitive student factors, which will increase the policy relevance of the PISA 2018 database.
The questions in this section cover students’ achievement-relevant dispositions, school-focused variables, and reading- and global competence-specific variables. The questionnaire thus adopts a hierarchical approach, investigating constructs that are specific to a domain and then constructs that cut across domains in order to understand and explain student achievement, engagement and behaviour (Elliot and Thrash, 2001[38]).
Reading-related outcomes (module 4)
Reading motivation, engagement and practices
Reading motivation, engagement and practices have been shown to be strongly linked with reading proficiency (Becker, McElvany and Kortenbruck, 2010[39]; Guthrie et al., 1999[40]; Klauda and Guthrie, 2015[41]; Mol and Bus, 2011[42]; Morgan and Fuchs, 2007[43]; Pfost, Dörfler and Artelt, 2013[44]; Schaffner, Philipp and Schiefele, 2016[45]; Schiefele et al., 2012[46]). In PISA 2000 and 2009, reading engagement (i.e. interest, intrinsic motivation, avoidance and practices) was a factor with one of the strongest relationships with reading proficiency; indeed, reading engagement was more strongly associated with reading proficiency than socio-economic status (OECD, 2010[47]; Kirsch et al., 2002[32]). It was shown that a high level of engagement compensated, to some extent, for a poor socio-economic background. It was also found that if boys were equally engaged in reading as girls, the gender gap would be reduced by two thirds (OECD, 2010[48]). In other studies, reading engagement has been shown to explain reading achievement more than any other variable besides previous reading achievement (Guthrie and Wigfield, 2000[49]). Thus, motivation and engagement are powerful variables, and are therefore possible levers on which one can act in order to enhance reading proficiency and reduce gaps between groups of students.
In the past, the components of motivation that PISA mainly targeted, particularly when reading was the major domain, were interest and intrinsic motivation. Other motivational constructs, such as self-efficacy and self-concept, were investigated when mathematics and science were the major domain. Here, self-efficacy (Bandura, 1997[50]; Ferla, Valcke and Cai, 2009[51]) refers to an individual’s perceived capacity of doing specific tasks while self-concept is a general measure of an individual’s own perceived abilities in a domain such as reading, mathematics or science, (Marsh and Craven, 1997[52]). Positive self-concept and self-efficacy are highly related to motivation, learning behaviour, general expectations for the future and performance (Baker and Wigfield, 1999[53]; Marsh and Craven, 2006[54]; Morgan and Fuchs, 2007[43]; Retelsdorf, Köller and Möller, 2011[55]; Solheim, 2011[56]; OECD, 2007[57]; OECD, 2016[58]) OECD, 2007. As a result, these constructs are measured in the PISA 2018 student questionnaire. Following Chapman and Tunmer’s recommendations (1995[59]), the instruments should cover not only students’ perceptions of their own competence in reading, but also perceptions of their difficulty with reading. Indeed, Klauda and Guthrie (2015[41]) have provided evidence that perceived difficulty with reading is a stronger predictor of reading achievement than self-efficacy. Similarly, they confirmed that students’ avoidance of reading and their devaluation of reading (the belief that reading is not useful) are negatively correlated with growth in engagement and motivation indicators among grade 7 pupils, even if students show positive self-efficacy and engagement in reading tasks.
PISA has now made the identification of students who perceive themselves to be struggling readers a higher priority through emphasizing the importance of basic components of reading such as fluency, and through extending the lower end of the reading scale since PISA 2009.
Metacognition
Like engagement, metacognition is significantly correlated with reading proficiency and is responsive to teaching and learning (Artelt, Schiefele and Schneider, 2001[60]; Brown, Palincsar and Armbruster, 2004[61]). The prominent metacognitive reading strategies include setting reading goals, adapting one’s reading strategies depending on these goals, knowing how to summarise a piece of text or remember essential information, monitoring comprehension and knowing how to repair comprehension problems. The new PISA 2018 reading literacy framework acknowledges the paramount importance played by these reading task management processes. They are now an integral part of the model of reading processing that organises the reading literacy framework (see Figure 2.2 in the reading literacy framework).
Explicit or formal instruction of metacognitive strategies leads to an improvement in understanding text and using information (National Reading Panel, 2000[62]). That is, when readers are given cognitive and metacognitive strategy instruction, they make more significant gains on measures of reading comprehension than students only trained with conventional instruction procedures (Baker and Carter-Beall, 2009[63]; Dole, Nokes and Drits, 2009[64]; Pressley, Graham and Harris, 2006[65]; Pressley et al., 1989[66]; Rosenshine and Meister, 1994[67]; Rosenshine, Meister and Chapman, 1996[68]; Waters and Schneider, 2010[69]).
PISA 2009 assessed students’ metacognitive strategies by asking them how useful they thought two reading strategies – summarizing a piece of text and understanding and memorizing a piece of text – were in order to solve a reading task. Correlations of these two metacognitive strategies with reading performance were robust, with median correlations across OECD countries of, respectively, 0.46 and 0.39 (Artelt and Schneider, 2015[70]).
The growing importance of digital reading literacy in PISA 2018 makes the need to assess metacognition even more important. Coiro and Dobler (2007[7]) pointed out that in online reading, efficient and specific self-regulated strategies (such as selecting the most relevant links and pathways and avoiding distracting information) are crucial to facilitating reading goals and plans. A new questionnaire item will focus on another important aspect of online reading, namely assessing the quality and the credibility of sources.
Dispositional and school-focused variables (module 9)
Complementing the reading-related outcomes are dispositions towards achievement and school-focused variables common across domains. Dispositional variables are the personality-based context in which students approach or avoid learning; they are the result of a lifetime of socialisation from parents, teachers, coaches and one’s cultural surroundings, and they capture how behaviour is energised over time. School-focused variables, or how students view and approach school, are influenced by both students’ disposition and the particular situation in which they find themselves. These dispositional and school-focused variables are the best predictors of both the aforementioned domain-specific variables (e.g. reading-related outcomes) and achievement outcomes. Together, dispositional and school-focused variables provide important information on students’ attitudes towards learning and achievement.
The following paragraphs provide an overview of some of the target dispositional and school-focused variables. These variables are relevant to all domains and focus on the non‑cognitive components important to learning. The dispositional variables include the achievement motives of competitiveness, work mastery, and the fear of failure; incremental mind-set; perseverance; subjective well-being; and ICT motivation and practices. The school-focused variables include learning beliefs and attitudes towards school, and achievement goals.
Dispositional variables
Achievement motives - Competitiveness, Work mastery, and Fear of Failure: Achieving motivation1, as assessed in the 2015 field trial, represented a combination of competitiveness and extrinsic motivation. However, the questionnaire expert group has replaced the construct of achieving motivation in PISA 2018 with the constructs of competitiveness and work mastery, which is how achievement motivation theorists have conceptualized their central construct over the past four decades (Helmreich et al., 1978[71]; Elliot and McGregor, 2001[72]). Here, competitiveness is defined as the dispositional desire to outperform others, while work mastery is defined as the dispositional desire to work hard to master tasks. Research shows that these two components of approach-oriented achievement motivation are linked to different sets of antecedents and consequences; so, when assessing achievement motivation, it is important to measure these constructs separately (Baranik, Barron and Finney, 2007[73]; Murayama and Elliot, 2012[74]; Spence and Helmreich, 1983[75]).
The questionnaire expert group has replaced the 2015 construct of general test anxiety with the construct of fear of failure. Test anxiety is worry about potential failure at the task- or domain-specific level of analysis (Hembree, 1988[76]). Fear of failure, however, is the more general tendency to self-protectively avoid potential mistakes and failures because they are experienced as shameful, which may be more predictive of cognitive achievement in real-life situations than test anxiety. Research has shown that fear of failure leads students to be self-protective and to avoid challenging situations and opportunities that are essential for learning and development (Covington, 1992[77]; Heckhausen, 1975[78]; Kaye, Conroy and Fifer, 2008[79]).
The optimal learner is high in work mastery and low in fear of failure. Competitiveness alone can be problematic, but the confluence of high competitiveness and work mastery appears to be beneficial. The positive and negative implications of competitiveness is a hot topic in the achievement literature, and data on this variable both within and across countries should prove valuable and garner considerable attention.
Incremental mind-set: Students with an incremental mind-set believe that ability is changeable rather than fixed, which is another core characteristic of an optimal learner. Having an incremental mind-set is related to perseverance, and has been found to be positively correlated with work mastery while negatively correlated with performance avoidance. This mind-set has also been shown to be correlated with both persistence in the face of failure and performance attainment (Aronson, Fried and Good, 2002[80]; Blackwell, Trzesniewski and Dweck, 2007[81]; Dweck, 2007[82]).
Perseverance: This construct was included in the 2012 student background questionnaire but was not included in the 2015 wave because of time constraints. Despite the six‑year gap, examining perseverance is still valuable given that it has been shown to be an important predictor of achievement (Duckworth et al., 2007[83]; Ţuţu and Constantin, 2012[84]). Many different labels are used in the current literature for this construct, including “persistence” and “grit”. The optimal learner is high in perseverance.
Subjective well-being: Subjective well-being can be defined as “good mental states, including all of the various evaluations, positive and negative, that people make of their lives and the affective reactions of people to their experiences” (OECD, 2013, p. 10[3]; OECD, 2011[85]; OECD, 2013[86]; OECD, 2017[87]). This definition encompasses three elements: life evaluation – one’s reflective assessment of one’s life (including the “general life satisfaction” question: “Overall, how satisfied are you with your life as a whole these days?”); affect – one’s emotional state, typically at a particular point of time; and eudaemonia – a sense of meaning and purpose in life, which can increase one’s sense of belonging. The recent growing interest from researchers and policy makers in this construct has resulted in recommendations to statistical agencies to “incorporate questions on subjective well-being in their standard surveys to capture people’s life evaluations, hedonic experiences and life priorities” (Stiglitz, Sen and Fitoussi, 2009, p. 216[88]). The OECD (OECD, 2013[86]) has responded to this charge in providing guidelines on measuring subjective well-being. The QEG has included information on all three elements, life evaluation, affect, and eudaemonia, for PISA 2018. The optimal learner has a positive life evaluation, frequent positive affect and infrequent negative affect, and strong eudaemonia.
ICT motivation and practices: Module 9 also covers information and communications technology (ICT). ICT-related behaviours and motivational attributes can be regarded as domain-general student outcomes because ICT plays a role across all educational domains. Following the OECD’s DeSeCo project and the 21st Century Skills Initiative, the optimal learner exhibits general skills related to information, media and technology above and beyond the traditional core subjects (OECD, 2005[89]; Partnership for 21st Century Skills, 2008[90]). The ICT familiarity questionnaire of PISA 2018 will assess students’ interest in ICT, use of ICT, perceived competence and autonomy in using ICT, and the use of social media.
School-focused variables
Learning beliefs and attitudes towards school: Beliefs about one’s own success or failure in school learning have been shown to be strong predictors for further effort and success, including for test scores in student assessments (Opdenakker and Van Damme, 2000[91]; Rumberger and Palardy, 2005[92]). PISA 2018 investigates several factors last examined in 2012, including students’ school self-efficacy, their evaluation of their experience in school, and their attitudes toward school. The optimal learner has strong school self-efficacy and a positive inclination toward school.
Achievement goals: One important characteristic of optimal learners is that they are focused on improvement in the classroom; in other words, they pursue mastery-approach goals. Students who adopt mastery-approach goals have been shown to engage in deep learning, to persist upon failure and to show high levels of intrinsic motivation (Hulleman et al., 2010[93]; Kaplan and Maehr, 2007[94]; Middleton and Perks, 2014[95]). An equally important characteristic of an optimal learner is that he/she does not strive to avoid performing worse than other students, or in other words, he/she does not pursue performance-avoidance goals. Students who adopt such goals have been shown to engage in shallow learning, to give up in the face of failure, and to display low levels of both performance and intrinsic motivation (Hulleman et al., 2010[93]; Rawsthorne and Elliot, 1999[96]; Van Yperen, Blaga and Postmes, 2014[97]). In short, achievement goals – both mastery-approach and performance-avoiding goals – are key predictors of the two central outcomes indicative of sustainable student success: performance attainment (which shows that short-term learning has taken place) and intrinsic motivation (which shows that the motivation for continued, long-term learning is in place). Focusing on both the presence of the positive (mastery-approach goals) and the absence of the negative (performance-avoidance goals) is important, as both are essential for optimal learning to take place (Elliot, 2005[98]).
Dispositions for global competence (module 10)
Global competence is the new domain in PISA 2018. This domain is critically important because learning, working and living environments are becoming more global, interconnected and interdependent. Young people will encounter, actively engage with and help shape those environments during their lifetime no matter where they are born, educated, work or live. It is therefore important that students leave school equipped with the necessary knowledge, skills and attitudes that will enable them to learn, work and live in a globalised world.
In PISA 2018, global competence is defined as the capability and disposition to act and interact appropriately and effectively, both individually and collaboratively, when participating in an interconnected, interdependent and diverse world. The domain of global competence is comprised of the following four dimensions:
Communication and relationship management refers to the willingness and capability to adapt one’s communication and behaviour in order to interact appropriately and effectively with others holding diverse perspectives and in different contexts.
Knowledge of2 and interest in global developments, challenges and trends refers to a learner’s interest in and knowledge of cultures, major issues, events and phenomena in the world, as well as the learner’s ability to understand their global significance and their implications for adapting appropriately and effectively to learning, working, and living with others holding diverse perspectives and in different contexts.
Openness and flexibility refers to being receptive to and understanding of new ideas, people and situations, as well as different perspectives and practices. It also refers to the ability to seek out and understand new and different perspectives and experiences and to appropriately and effectively adapt one’s thinking, behaviour and actions to learning, working and living situations that involve others holding diverse perspectives and in different contexts.
Emotional strength and resilience refers to the ability to deal appropriately with the ambiguity, changes, and challenges that different perspectives and experiences can present and to have the resilience to maintain one’s identity and/or to develop personally despite or as a result of encountering different perspectives and experiences.
Generally, the questionnaire items related to global competence (construct-related and contextual information to inform the interpretation of outcomes) will focus on the two dimensions of openness and flexibility and emotional strength and resilience. Approximately ten questions will be asked for each of the following four scales: openness, flexibility, emotional strength and resilience. The questions for each scale will primarily be behaviour-based (e.g. “When I meet people who are different from me, I am interested in learning more about them”; “I like to eat in a variety of ethnic restaurants”), will be located in simple contexts (e.g. “When I travel abroad …”; “When I am at school …”) and will be presented in a consistent format whenever possible (e.g. using a Likert scale).
Topics for the global competence questionnaire items could include (most measures comprise both construct and contextual components):
Languages:
Languages spoken at home/understood by the student and by his/her mother/father/siblings
Languages taught in school; number of languages spoken/understood by the student
Number of languages to which the student has access or with which the student has experience (actual, virtual)
Global developments/challenges/trends:
The student’s engagement with others about global events/issues (e.g. on line chat);
Extent of exposure to/awareness of global developments/challenges/trends (e.g. via the news or other media)
Migration/movement of student:
The student’s own background; the student’s experience living abroad
The possibility of the student working/studying abroad in the future
Demographics in his/her community
Student interaction with or exposure to people from other countries/cultures; student’s travel experience
Student’s degree of curiosity/motivation to travel
Existing questionnaire items (i.e. from the school and teacher questionnaires) could also be used or extended to inform the measures of global competence. For example:
School climate:
Student demographics (at the school level)
School philosophy/values/policies related to global competence
Support given to students for language and/or culturally-based reasons (e.g. such items could be revised to focus on global competence-related challenges)
Global competence in the curriculum
Professional development: how/the degree to which teachers are prepared to manage multiculturalism and/or facilitate global competence (e.g. global trends, international events)
The availability of resources related to global competence
Finally, the following items could be added to the teacher questionnaire in order to measure global competence:
Teacher demographics/background (e.g. cultures represented)
Number of teachers who have taught in another country
Number of teachers who hold a certification from another country
Exposure/experience/engagement with international and/or global events
ICT literacy
Awareness of and support for school policies that relate to/support global competence
Teacher practices related to global competence, including cross-curricular practices
Most existing measures in the area of global competence have been developed for older learners (e.g. college students or adult employees) rather than for the 15‑year‑old students assessed in PISA 2018. Nonetheless, various measures for 15‑year‑old students can be derived from the extant literature (Deardorff, 2009[99]).
Assessing student background
This subsection covers module 6 (student SES, family and home background), module 7 (ethnicity and migration), and module 8 (educational pathways in early childhood). These topics require careful revisiting every cycle because they contain the basic information needed to calculate the index of economic, social and cultural status (ESCS), the proxy that PISA uses for socio-economic status.
Student SES, family and home background (module 6): PISA 2018 keeps questions regarding socio-economic status and other background variables basically unchanged in order to be able to observe trends in social, cultural and economic indicators. However, some minor changes have become necessary. For example, extensive developments in ICT mean that items that were once only sometimes found in students’ homes (e.g. laptops or tablets) are now commonplace and are therefore less discriminatory as a marker of socio-economic status. Thus, the measures of home possessions will be updated to ensure better coverage of within- and cross-country variation of home possessions. These changes are expected not to have an impact on the important trend measures in this module.
Migration and culture (module 7): Many nations are home to several subpopulations with different languages and cultures. International migration perpetuates this diversity. On average across OECD countries, first and second generation immigrant students composed 12.5% of the student population in 2015, up from 9.4% in 2006 (OECD, 2016[58]). At the same time, students from ethnic minority groups and immigrant students often face particular challenges. In a number of education systems, immigrant students perform at significantly lower levels than their native peers in key school subjects (Stanat and Christensen, 2006[100]; OECD, 2016[58]), and both groups are often faced with overt or covert discrimination with potentially detrimental consequences for their psychological development and well-being. Thus, providing students from different linguistic and cultural backgrounds with equal opportunities is often considered one of the central challenges for education systems in the 21st century (OECD, 2010[47]).
PISA 2015 put a special focus on diversity-related aspects of the school climate. A new question developed for the PISA 2015 field trial asked students about their membership in a group that they believed to be discriminated against in their country. If they identified themselves as belonging to such a group, they were then asked whether they felt treated in a respectful and fair manner by their teachers and equal to their classmates, a factor that has been shown to be related to educational outcomes (Fisher, Wallace and Fenton, 2000[101]; Wong, Eccles and Sameroff, 2003[102]). Another new question, implemented in the optional parent questionnaire, assessed perceived barriers to parental involvement. Additionally, teachers and principals are asked about diversity-related assumptions among teachers in their school. The wording of the question is based on research on how multiculturalism is supported (van de Vijver, Breugelmans and Schalk-Soekar, 2008[103]). Additionally, PISA 2015 examined aspects of multicultural education practices and the extent to which multicultural educational practices were implemented in different schools. Altogether, findings from this module may help researchers better understand educational inequalities and can suggest ways to address these inequalities. However, these items have proved problematic and will need to be reworked if they are to be successful in measuring this important concept in 2018. Given the relevance of this module for global competence, such a reworking (presumably with more emphasis on opportunities for intercultural encounters) is worthwhile.
Educational pathways in early childhood (module 8): Children already have varying levels of ability in their language, pre-reading and early numeracy skills by the time they enter primary school, and these differences are often maintained throughout life. Promoting school readiness and better adjustment to school is hypothesised to be an efficient means of raising the achievement levels of all children, but especially of those children who experience a lack of parental support or who grow up in disadvantaged circumstances. It has been argued that investing in early education programmes will have large long-term monetary and non‑monetary benefits (Heckman, 2006[104]).
According to UNESCO (2006[105]), Early Childhood Care and Education (ECCE) programmes are “programmes that, in addition to providing children with care, offer a structured and purposeful set of learning activities either in a formal institution (pre‑primary or ISCED 0) or as part of a non‑formal child development programme” (p. 348). PISA will also use this definition, as opposed to inquiring only about students’ experiences in ISCED 0; most literature on early childcare – such as brain research, studies on domain-specific development and support, evaluation studies of model programmes, and longitudinal large-scale studies – do the same.
Yet not all ECCE programmes lead to long-term benefits. For example, the British EPPE study found short-term effects showing that pre‑school attendance was beneficial for cognitive and socio-emotional development, in particular for children from disadvantaged backgrounds. However, in the long term, only those children who attended a high-quality pre‑school centre showed lasting beneficial effects (Sammons et al., 2009[106]; Sylva et al., 2011[107]); cf. also (Valenti and Tracey, 2009[108]). A certain degree of intensity in terms of hours per week/months also seems to be a precondition for the beneficial effects of ECCE attendance (Logan et al., 2011[109]; Sylva et al., 2011[110]).
Thus, asking about early education experience in PISA only makes sense if specific aspects of quantity, quality and curriculum can be retrieved retrospectively, which is highly unlikely when the questions are asked of students (Fivush and Hudson, 1990[111]; Markowitsch and Welzer, 2009[112]). As a consequence, PISA 2018, whilst keeping a short question on ISCED 0 attendance in the student questionnaire, will continue to administer a series of questions on students’ ECCE attendance in the parent questionnaire as parents are expected to be a more reliable source of information. Those countries administering the optional parent questionnaire will thus gain information on basic characteristics of the ECCE arrangements in their countries and the reasons for attending or not attending ECCE.
Assessing teaching and learning processes
This subsection summarises the conceptual foundations for module 1 (teacher qualifications and professional development), module 2 (teaching practices for reading), module 5 (out-of-school reading experience), and module 11 (learning time and curriculum).
Teaching and learning are at the heart of education. Most cognitive and non‑cognitive, curricular and extra-curricular goals of school education are achieved or impeded by the way students and teachers interact in classrooms. Whilst teaching is the core process in schools, the curriculum determines its content and professional teachers are the force who implement the curriculum, orchestrate learning activities and thus arrange for quality learning time.
There is ample evidence that teaching and learning activities are very good predictors of student ability and PISA therefore needs to examine these activities if it is to inform educational policy making at the system and the school level. Module 2 describes reading education by broad sets of teaching and learning activities. In addition, this module investigates general dimensions of teaching quality, such as the structure of instruction, classroom management and support, and cognitive activation, as they are applied in reading education. Next, module 11 covers learning time, including non‑mandatory, additional instruction within and out of school, as well as the coherence, focus and rigour of the reading curriculum. Module 5 examines students’ engagement with reading outside of school. Finally, the initial education, beliefs and professional development of the teaching force will be described in module 1.
Teaching practices for reading (module 2) and learning time and curriculum (module 11)
Teaching practices and classroom support for reading growth and engagement in PISA
Research on reading shows that classroom practices, such as the direct teaching of reading strategies (Pressley, 2000[113]; Rosenshine and Meister, 1997[114]; Waters and Schneider, 2010[69]) and teachers’ scaffolding and support for autonomy, competence and ownership (Guthrie, Klauda and Ho, 2013[115]; Guthrie, Wigfield and You, 2012[116]) are powerful ways of improving students’ reading proficiency, awareness of strategies (metacognition) and engagement (motivation) in reading.
Hence, it is important that the 2018 questionnaire address teaching practices around reading literacy. Two broad theories inform the selection of the most relevant constructs:
Practices that support reading engagement and motivation relate to self-determination theory (Deci and Ryan, 1985[117]; Reeve, 2012[118]; Vansteenkiste, Lens and Deci, 2006[119]). The pathway to students’ self-determination in reading depends on support from “significant others” in their lives. A favourable family and school context leads to greater confidence in one’s reading ability and expanded autonomy in directing one’s own reading activities. This, in turn, results in intrinsically motivated and self-determined reading and finally to greater reading proficiency. The teacher is a “significant other” for reading literacy. Numerous studies show that teachers who improve students’ sense of ownership and competence enable them to become active and competent readers. By contrast, teachers who neglect these instructional practices impede students’ efforts to become autonomous, resulting in students who disengage from reading and fail to progress in reading achievement (Guthrie, 2008[120]).
Practices that enhance reading skills and metacognitive strategies are based on direct instruction of reading strategies (Pressley, 2000[113]), and also correspond to the “cognitive activation” facet of the tridimensional model of quality of teaching (Klieme, Pauli and Reusser, 2009[21]), namely the model on which the selection of teaching constructs for PISA 2012 and 2015 has been built. According to this model, classroom management processes, teacher support and cognitive activation independently predict growth in students’ mathematics, science and language skills; teacher support is the main predictor of motivation growth; and cognitive activation is the main predictor of cognitive gains at different grades (Klieme, Steinert and Hochweber, 2010[121]).
However, when students are 15 years old, reading is no longer taught as a subject matter in the same way that mathematics and science are. It is therefore a challenge to tailor questions capturing the classroom practices that students are exposed to and their opportunities to develop their reading skills. Reading is indeed not a part of test language lessons, but also of, for example, social science, natural science, foreign language, civic education and ICT lessons. There is even more uncertainty in where the new skills and processes related to digital reading are taught: they are learned mostly in non‑formal contexts outside of school in some countries, taught in specific courses in others, and considered to be “transversal” and not taught at all in specific courses but incorporated throughout the curriculum in yet other countries.
There is an ongoing debate in the field of reading research between scholars supporting the view that “every teacher is a reading teacher” (“generic” or “intermediate” reading literacy) and advocates of the “content” or “disciplinary” literacy view, who argue that reading texts dealing with, for example, mathematics, chemistry or history require distinct reading skills linked to the domain that should be taught by content matter teachers (Shanahan and Shanahan, 2008[23]). Taking into account time limitations and the fact that teaching practices were not considered a priority for PISA 2018, the questionnaire will ask the students only a small number of questions about their experience and exposure to “generic” or “intermediate” literacy teaching practices in their classes in general, as took place in PISA 2009 (Lafontaine et al., 2015[122]).
Out-of-school experience (module 5)
In previous PISA cycles when reading was the major domain (2000 and 2009), reading practices were measured as self-reported frequencies of reading activities with diverse content in various media. The initial list of possible content included fiction and non‑fiction books, comics, newspapers, magazines and e‑mails. In 2009, new items about online reading practices were included. Some of these new items focused on social online practices (blogs, forums, e‑mails), while others focused on searching for information (searching for news online, for information in order to learn about a topic, or for practical information). While students’ online practices related to searching for information explained a significant, yet small, proportion of the variance of their online digital reading abilities, their social practices on line had no significant relationship with digital reading in most countries that participated in PISA 2009 (OECD, 2011[123]; Naumann, 2015[124]).
The list of online reading practices will be extended in PISA 2018 to take into account recent and emerging reading media (e.g. e‑books, social networks). However, despite the growing importance of reading on digital devices, it should be kept in mind that traditional forms of reading (especially books) are still the most influential means for developing students’ reading comprehension and vocabulary (Pfost, Dörfler and Artelt, 2013[44]).
As in the reading cognitive assessment, it is important to keep some of the 2009 items measuring reading practices in order to measure trends. This allowed PISA to say, for instance, that 15‑year‑olds read less in 2009 than in 2000, with a stronger decline among males. However, the same question might take on different meanings in 2009 and in 2018 because reading itself has changed: the term “book”, for instance, might now refer to both printed books and e‑books. The reading practices reported by students could change drastically depending on how questions are worded.
Due to the limitations of self-reported measures based on Likert scales (Allen, Cipielewski and Stanovich, 1992[125]; Pfost, Dörfler and Artelt, 2013[44]), alternative ways of capturing reading practices (forced-choice, behavioural or situational questions) will be tested during the field trial. If some of these prove to be more stable, more culturally invariant and better correlates of reading ability, they could replace or complement some of the trend items and potentially become new trend items themselves.
Teacher qualifications and professional development (module 1)
Many studies have demonstrated a clear influence of teacher-related factors on student learning and outcomes, and there has been increased focus in recent years on teacher-related policies. In addition to teachers’ professional behaviour within the classroom, the age distribution and educational level of the teaching force, teachers’ initial education and qualifications, their individual beliefs and competencies, and their professional practices on the school level (such as collaboration and professional development), and how these factors are related to student outcomes, are core topics in education policy. Basic information on these topics will be available from the PISA 2018 school as well as the optional teacher questionnaire.
Assessing school policies and governance
This subsection summarises the conceptual foundations for module 3 (school-level learning environment for reading) and modules 12‑18.
School policies and approaches to educational governance
During the last two decades, research on educational effectiveness has largely been concerned with the impact of school-level factors on students’ learning. Studies show that such school factors are related to student progress. It has been asserted that the school environment can influence the behaviour of teachers and students and thus – mostly indirectly – their success in teaching and learning. Both “soft” factors, such as school climate and parental involvement, and “hard” factors, such as school management activities and allocation policies, vary within and across countries and are related to student outcomes.
School-level learning environment for reading (module 3): Conceptually, this module overlaps to a considerable degree with other modules dealing with school-level factors, such as module 11 (learning time and curriculum), module 14 (school context and resources), and module 16 (assessment, evaluation, and accountability). The questionnaire expert group recommends that some questions in the school questionnaire focus directly on the status of reading education in the school and available resources dedicated to reading. Accordingly, a question has been developed to investigate the overall value of reading within the school (“Is reading education a shared priority for stakeholders?”), along with questions on resources available that are primarily dedicated to reading education: the size of the teaching staff, resources such as libraries and digital learning devices, and potentially cooperation with external partners.
School climate (module 12): The school climate encompasses a school’s shared norms and values, the quality of the interpersonal relationships within the school, and its general atmosphere. The general consensus about both the mission of the school and the value of education, shared by school leaders, staff and parents, affects the norms of student peer groups and facilitates learning (Opdenakker and Van Damme, 2000[91]; Rumberger and Palardy, 2005[92]). In addition, an orderly learning atmosphere maximises the use of learning time. By contrast, disrespectfulness and an unruly environment are counterproductive for teachers and students alike and distract from the school’s educational mission. As in previous PISA assessments, school climate will be assessed in both the student questionnaire (such as through questions on student-teacher relationships and achievement pressure) and the school questionnaire (such as through questions on teacher morale and behaviours that could affect school climate).
A number of items related to student-teacher relationships, as identified by Caldwell and Bradley (1984[126]) in their “home inventory”, were added to the PISA 2015 questionnaires: emotional warmth; guidance and boundaries; stimulation/scaffolding and stability. Students report on their relationship with teachers and parents, whilst school principals and parents (in the optional questionnaires) are asked parallel items. All in all, an interesting picture of social relationships between students, school actors and parents could emerge in countries that administer both the parent and the school questionnaires. Finally, two scales in the student questionnaire that cover more problematic, often hidden aspects of school climate should be continued: bullying by peers and unfair treatment by teachers.
Parental involvement (module 13): Over the past years, the involvement of parents in educational processes has gained importance in the educational debate and relevance for educational policy. PISA has collected information on parental involvement in education since 2006, when the parent questionnaire was administered for the first time, directly addressing the parents of PISA students. In PISA 2015, specific aspects of parental involvement were added to all questionnaires, not only the parent questionnaire, focusing on (a) parent-school communication and collaboration and (b) parental support for learning; these scales will be continued for 2018. Although aspects of parental involvement can be found in other modules, the majority of items and topics regarding parental involvement are included in the parent questionnaire that will be administered as an international option.
The parent questionnaire will also ask questions specifically related to reading literacy, in particular regarding parental support during the development of their child’s early literacy skills (at ISCED 0), and regarding parents’ own interest in and motivation for reading. There is extensive evidence that parental support before the formal instruction of reading at school, such as the joint reading of books or playing language games, is critical to the development of early or “emergent” literacy (Neuman and Dickinson, 2003[127]). However, there is still much to be discovered about the relationship between parental support and adolescents’ reading motivation and practices (Klauda, 2009[128]).
School context and resources (module 14): Information on school type (public vs. private) and class size has always been included in the school questionnaire. In addition to these trend questions, the PISA 2015 field trial expanded this module, discriminating between types of private schools (religious/denominational, not-for-profit and for-profit) and eliciting more advanced information on ICT use. All PISA cycles to date have included a question on the degree to which a school experiences problems due to a lack of resources or poor-quality resources. PISA 2018 now inquires about these issues in one set of coherent questions in the school questionnaire.
Allocation, selection, choice and grade repetition (module 15): The way students are channelled into educational pathways, schools, tracks or courses is a core issue of educational governance known as stratification, streaming or tracking. Selection and allocation procedures are also important aspects of school organisation. The learning environment in highly selective schools may differ from that in more comprehensive schools. For all of these reasons, questions on allocation, selection, choice and grade repetition answered by school administrators and parents have been retained from previous PISA questionnaires.
PISA 2015 asked students whether they had ever repeated a grade. Many longitudinal studies have demonstrated grade retention to have a negative relationship with individual careers and outcomes (Ou and Reynolds, 2010[129]; Griffith et al., 2010[130]), student behaviour, and well-being (Crothers et al., 2010[131]). Grade repetition is less common in secondary schools compared to primary schools, but the negative effects of late retention seem to be larger (Ou and Reynolds, 2010[129]). Greene and Winter (2009[132]) showed that once a test-based retention policy had been installed, those who were exempted from the policy performed worse. Babcock and Bedard (2011[133]) showed that a large number of students being retained could have a positive effect on the cohort (i.e. on all students, including those who were promoted). Kloosterman and De Graaf (2010[134]) argued that in highly tracked systems, such as in some European countries, grade repetition might serve as a preferred alternative to moving into a lower track; indeed, they found evidence that this strategy is preferred for low-performing students with higher SES. Thus, changing grade repetition policies might be a viable low‑cost intervention (Binder, 2009[135]). Therefore, it is worthwhile to take a closer, comparative look at grade retention policies and their relationship with student outcomes (for both students who do and do not repeat a grade). PISA 2018 will explore grade-repetition related question.
Assessment, evaluation, and accountability (module 16)
Assessing students and evaluating schools is a common practice in most countries (Ozga, 2012[136]). Since the 1980s, policy instruments, such as performance standards, standard-based assessment, annual reports on student progress and school inspections, have been promoted and implemented across education systems. Reporting and sharing data from assessments and evaluations with different stakeholders provides multiple opportunities for monitoring, feedback and improvement. In recent years, there has been a growing interest in the use of assessment and evaluation results for quality management and improvement (OECD, 2010, p. 76[47]); formative assessments, also known as assessment for learning, have been one of the dominant movements in this domain (Baird et al., 2014[137]; Black, 2015[138]; Hattie, 2009[139]). Accountability systems based on these instruments are increasingly common in OECD countries (Rosenkvist, 2010[140]; Scheerens, 2002, p. 36[141]).
Previous PISA cycles have covered aspects of assessment, evaluation and accountability in the school questionnaire by identifying a variety of purposes for the assessment of students. School leaders have been asked: whether they use test results to make comparisons with other schools at the district or national level; or to improve teacher instruction by asking students for written feedback on lessons, teachers or resources. Relevant research on school evaluation and student assessment is summarised below to provide the rationale for questionnaire development in this module in PISA 2018.
Evaluation: The evaluation of schools is used as a means of assuring transparency, making judgements about systems, programmes, educational resources and processes, and guiding school development (Faubert, 2009[142]). In PISA 2018, the term evaluation will be used for processes at the school and system level, as was done in PISA 2015.
Evaluation can be either external or internal (Berkenmeyer and Müller, 2010[143]). In an external evaluation, the process is controlled and headed by an external body and the school does not define the areas that are judged. On the other hand, an internal evaluation is part of a process controlled by the school, where the school defines the areas that are judged; the evaluation may be conducted by members of the school (self-evaluation) or by persons/institutions commissioned by the school. Different evaluation practices generally co‑exist and benefit from each other (Ryan, Chandler and Samuels, 2007[144]). External evaluations can expand the scope of internal evaluation and validate the results from an internal evaluation. Internal evaluations can improve the interpretation and increase the utilisation of results from an external evaluation. However, the improvement of schools seems to be more likely when an internal evaluation is undertaken. Moreover, country and school-specific context factors may influence the implementation of evaluation results as well as their effects on schools. In many countries, the individual evaluation of teachers and principals, as separate from a school-wide evaluation, is also common (Faubert, 2009[142]; Santiago and Benavides, 2009[145]); they are treated here as a separate type of evaluation.
Assessment: Many countries have implemented national standards to assess students’ learning outcomes. Together with formative assessment practices, these summative assessment systems influence the way teachers teach and students learn. In particular, formative assessment practices can enhance students’ achievement (Black and Wiliam, 1998[146]). However, there is large variation in the implementation of formative assessment practices, as reported in recent studies in Canada, Norway, Scotland, Singapore, Sweden and the United States among others (Wylie and Lyon, 2015[147]; DeLuca et al., 2015[148]; Jonsson, Lundahl and Holmgren, 2015[149]; Hayward, 2015[150]; Ratnam-Lim and Tan, 2015[151]; Hopfenbeck and Stobart, 2015[152]). PISA 2018 therefore aims to assess both the formative and summative aspects of student assessment through questions in both the student and school questionnaires.
Dealing with response bias
The analysis of response bias plays an important role in the analysis of PISA data. Two types of approaches have been implemented to handle bias: adapted instruments (using novel constructs or response formats) and advanced statistical modelling. The field trial will enable experimentation with different question formats (e.g. using interactive features of the computer-based administration system), and with new content that could be subject to response bias (e.g., measures of students’ physical and emotional well-being). The field trial will assess the psychometric properties of these instruments and whether new measures are can be compared across different education systems. A subset of measures will be selected for the main study based on findings from the field trial.
Well-known examples of design measures to correct for bias include overclaiming, anchoring vignettes and cognitive interviews to examine response styles. Overclaiming is a procedure in which students are asked about their knowledge of a number of concepts, some of which do not exist. Students who indicate a high knowledge of these non‑existent concepts show strong response styles, which is somewhat akin to social desirability. Statistical correction for overclaiming had an impact on cross-national differences in scores. For example, correlations between motivation and performance are often computed at the individual and country levels as a test of bias correction procedures. These correlations are usually positive and significant yet small at the individual level, but strong and negative at the country level (Marsh and Hau, 2004[153]). It has been argued that cross-national differences in response styles are responsible for this change in correlation after aggregation. Correcting for overclaiming led to a sizable reduction of the negative correlation between motivation and performance at the country level (Kyllonen and Bertling, 2014[154]).
Anchoring vignettes present descriptions of hypothetical persons, usually with very high, medium and very low levels of a target construct, before assessing the target construct itself (King et al., 2004[155]). Students are asked how they would rate the motivation of these hypothetical persons on a scale. The response on the target item, how the students would rate their own reading motivation, is then re‑scaled using the anchor scores of the three hypothetical persons. The results of the use of anchoring vignettes have been mixed. Very promising results were obtained in the 2012 field trial, where the country-level correlations between motivation and achievement were rather close to the individual-level correlations. However, the 2015 field trial yielded a more complex and less supportive set of results. Other problems with anchoring vignettes are the additional testing time they require and the relatively high reading load of the items.
Various statistical procedures to correct for cross-national bias have been proposed. One example is the correction for response styles, such as acquiescence, extremity and social desirability, in analyses of covariance, using response style indices as covariates (He and Van de Vijver, 2016[156]; Van de Vijver and He, 2014[157]). Such procedures do not tend to have much impact on the negative correlation between motivation and achievement at the country level (see above). Another example of such a procedure is the use of propensity score matching (Buckley, 2009[158]), which attempts to increase the comparability of samples obtained in different countries by matching them through relevant background characteristics, such as socio-economic status. Although still frequently used, there are indications that the negative correlation between motivation and achievement at country level is not strongly affected by propensity matching. A final example is the statistical modelling of a response style factor in a confirmatory factor analysis (Billiet and McClendon, 2000[159]). The applicability of this procedure may be limited, however, as it can only be used in balanced scales where some items are formulated in a positive direction and others formulated in the opposite direction (such as in measures of extroversion, where some items assess extroversion and some introversion). PISA background scales do not use such a balanced approach.
It can be concluded that there is no simple way to eliminate cross-cultural bias in the PISA 2018 background questionnaires. It is advisable to use cognitive interviewing to avoid response styles and other problematic aspects as much as possible, and to use data from the field trial to further investigate the feasibility of different statistical approaches (including the standard tests of invariance using confirmatory factor analysis). However, despite the aspects of the questionnaire design outlined above and the statistical analyses proposed, it may not be possible to eradicate all sources of bias in the PISA 2018 background questionnaire.
In summary, the field trial will test the psychometric properties of the new scales and question formats. Given the inconclusiveness about how to deal with cross-cultural bias and response styles, it is proposed to employ procedures already used in the past (e.g. confirmatory factor analysis and IRT modelling of response styles) instead of using the field trial for further development of these issues.
References
[6] Afflerbach, P. and B. Cho (2010), “Determining and Describing Reading Strategies: Internet and Traditional Forms of Reading”, in Waters, H. and W. Schneider (eds.), Metacognition, Strategy Use and Instruction, Guilford Press, New York.
[125] Allen, L., J. Cipielewski and K. Stanovich (1992), “Multiple indicators of children’s reading habits and attitudes: Construct validity and cognitive correlates”, Journal of Educational Psychology, Vol. 84/4, pp. 489-503, http://dx.doi.org/10.1037/0022-0663.84.4.489.
[14] Almlund, M. et al. (2011), “Personality Psychology and Economics”, IZA Discussion Papers, No. 5500, Institute for the Study of Labour (IZA), Bonn, http://ftp.iza.org/dp5500.pdf.
[11] Anderson, R. (2008), “Implications of the Information and Knowledge Society for Education”, in Voogt, J. and G. Knezek (eds.), International Handbook of Information Technology in Primary and Secondary Education, Springer, New York.
[80] Aronson, J., C. Fried and C. Good (2002), “Reducing the Effects of Stereotype Threat on African American College Students by Shaping Theories of Intelligence”, Journal of Experimental Social Psychology, Vol. 38/2, pp. 113-125, http://dx.doi.org/10.1006/jesp.2001.1491.
[60] Artelt, C., U. Schiefele and W. Schneider (2001), “Predictors of reading literacy”, European Journal of Psychology of Education, Vol. XVI/3, pp. 363-383, http://dx.doi.org/10.1007/BF03173188.
[70] Artelt, C. and W. Schneider (2015), “Cross-Country Generalizability of the Role of Metacognitive Knowledge in Students’ Strategy Use and Reading Competence”, Teachers College Record, Vol. 117/1, pp. 1-32, https://www.tcrecord.org/content.asp?contentid=17695.
[133] Babcock, P. and K. Bedard (2011), “The Wages of Failure: New Evidence on School Retention and Long-Run Outcomes”, Education Finance and Policy, Vol. 6/3, pp. 293-322, http://dx.doi.org/10.1162/EDFP_a_00037.
[137] Baird, J. et al. (2014), Assessment and Learning: State of the Field Review, Knowledge Center for Education, Oslo, https://www.forskningsradet.no/servlet/Satellite?c=Rapport&cid=1253996755700&lang=en&pagename=kunnskapssenter%2FHovedsidemal.
[63] Baker, L. and L. Carter-Beall (2009), “Metacognitive Processes and Reading Comprehension”, in Israel, S. and G. Duffy (eds.), Handbook of Research on Reading Comprehension, Routledge, New York.
[53] Baker, L. and A. Wigfield (1999), “Dimensions of Children’s Motivation for Reading and Their Relations to Reading Activity and Reading Achievement”, Reading Research Quarterly, Vol. 34/4, pp. 452–477, http://dx.doi.org/10.1598/RRQ.34.4.4.
[50] Bandura, A. (1997), Self-Efficacy: The Exercise of Control, W. H. Freeman, New York.
[73] Baranik, L., K. Barron and S. Finney (2007), “Measuring Goal Orientation in a Work Domain: Construct Validity Evidence for the 2×2 Framework”, Educational and Psychological Measurement, Vol. 67/4, pp. 697-718, http://dx.doi.org/10.1177/0013164406292090.
[25] Barile, J. et al. (2012), “Teacher-Student Relationship Climate and School Outcomes: Implications for Educational Policy Initiatives”, Journal of Youth and Adolescence, Vol. 41, pp. 256–267, http://dx.doi.org/10.1007/s10964-011-9652-8.
[39] Becker, M., N. McElvany and M. Kortenbruck (2010), “Intrinsic and Extrinsic Reading Motivation as Predictors of Reading Literacy: A Longitudinal Study”, Journal of Educational Psychology, Vol. 102/4, pp. 773-785, http://dx.doi.org/10.1037/a0020084.
[8] Bennett, J. (1993), “Towards ethnorelativism: A developmental model of intercultural sensitivity”, in Paige, M. (ed.), Education for the intercultural experience, Intercultural Press, Yarmouth, ME, https://www.amazon.com/Education-Intercultural-Experience-Paige/dp/1877864250.
[143] Berkenmeyer, N. and S. Müller (2010), “Schulinterne Evaluation - nur ein Instrument zur Selbststeuerung von Schulen?”, in Altrichter, H. and K. Maag Merki (eds.), Handbuch Neue Steuerung im Schulsystem, VS Verlag für Sozialwissenschaften, Wiesbaden.
[159] Billiet, J. and M. McClendon (2000), “Modeling Acquiescence in Measurement Models for two Balanced Sets of Items”, Structural Equation Modeling, Vol. 7/4, pp. 608-628, http://dx.doi.org/10.1207/S15328007SEM0704_5.
[135] Binder, M. (2009), “Why Are Some Low-Income Countries Better at Providing Secondary Education?”, Comparative Education Review, Vol. 53/4, pp. 513-534, http://www.journals.uchicago.edu/t-and-c.
[12] Binkley, M. et al. (2012), “Defining Twenty-First Century Skills”, in Griffin, P., B. McGaw and E. Care (eds.), Assessment and Teaching of 21st Century Skills, Springer, New York.
[138] Black, P. (2015), “Formative Assessment – An Optimistic but Incomplete Vision”, Assessment in Education: Principles, Policy and Practice, Vol. 22/1, pp. 161-177, http://dx.doi.org/10.1080/0969594X.2014.999643.
[146] Black, P. and D. Wiliam (1998), “Assessment and Classroom Learning”, Assessment in Education: Principles, Policy & Practice, Vol. 5/1, pp. 7-74, http://dx.doi.org/10.1080/0969595980050102.
[81] Blackwell, L., K. Trzesniewski and C. Dweck (2007), “Implicit Theories of Intelligence Predict Achievement Across an Adolescent Transition: A Longitudinal Study and an Intervention”, Child Development, Vol. 78/1, pp. 246 – 263, http://dx.doi.org/10.1111/j.1467-8624.2007.00995.x.
[17] Blau, D. and J. Currie (2006), “Pre-School, Day Care, and After-School Care: Who’s Minding the Kids?”, in Hanushek, E. and F. Welch (eds.), Handbook of the Economics of Education, Elsevier, Amsterdam, http://dx.doi.org/10.1016/S1574-0692(06)02020-4.
[61] Brown, A., A. Palincsar and B. Armbruster (2004), “Instructing Comprehension-Fostering Activities in Interactive Learning Situations”, in Ruddell, R. and N. Unrau (eds.), Theoretical Models and Processes of Reading, International Reading Association, Newark.
[26] Bryk, A. et al. (2010), Organizing Schools for Improvement: Lessons from Chicago, University of Chicago Press, Chicago.
[158] Buckley, J. (2009), Cross-National Response Styles in International Educational Assessments: Evidence from PISA 2006, New York University, New York, https://edsurveys.rti.org/PISA/documents/Buckley_PISAresponsestyle.pdf.
[126] Caldwell, B. and R. Bradley (1984), Home Observation for Measurement of the Environment: Administration Manual, Revised Edition, Unpublished manuscript. University of Arkansas, Little Rock.
[27] Chapman, C. et al. (2011), School Effectiveness and Improvement Research, Policy and Practice: Challenging the Orthodoxy?, Routledge, Abingdon.
[59] Chapman, J. and W. Tunmer (1995), “Development of Young Children’s Reading Self-Concepts: An Examination of Emerging Subcomponents and Their Relationship With Reading Achievement”, Journal of Educational Psychology, Vol. 87/1, pp. 154-167, http://dx.doi.org/10.1037/0022-0663.87.1.154.
[7] Coiro, J. and E. Dobler (2007), “Exploring the Online Reading Comprehension Strategies Used by Sixth-Grade Skilled Readers to Search for and Locate Information on the Internet”, Reading Research Quarterly, Vol. 42/2, pp. 214-257, http://dx.doi.org/10.1598/RRQ.42.2.2.
[77] Covington, M. (1992), Making the Grade A Self-Worth Perspective on Motivation and School Reform, Cambridge University Press, New York, http://dx.doi.org/10.1017/CBO9781139173582.
[19] Creemers, B. and L. Kyriakides (2008), The Dynamics of Educational Effectiveness: A Contribution to Policy, Practice and Theory in Contemporary Schools, Routledge, Abingdon.
[131] Crothers, L. et al. (2010), “A Preliminary Study of Bully and Victim Behavior in Old-For-Grade Students: Another Potential Hidden Cost of Grade Retention or Delayed School Entry”, Journal of Applied School Psychology, Vol. 26/4, pp. 327-338, http://dx.doi.org/10.1080/15377903.2010.518843.
[18] Cunha, F. et al. (2006), “Interpreting the Evidence on Life Cycle Skill Formation”, in Handbook of the Economics of Education, Volume 2, Elsevier, Amsterdam.
[99] Deardorff, D. (2009), The SAGE Handbook of Intercultural Competence, Sage Publications, Thousand Oaks.
[117] Deci, E. and R. Ryan (1985), Intrinsic Motivation and Self-Determination in Human Behavior, Plenum Press, New York.
[148] DeLuca, C. et al. (2015), “Instructional Rounds as a Professional Learning Model for Systemic Implementation of Assessment for Learning”, Assessment in Education: Principles, Policy and Practice, Vol. 22/1, pp. 122-139, http://dx.doi.org/10.1080/0969594X.2014.967168.
[64] Dole, J., J. Nokes and D. Drits (2009), “Cognitive Strategy Instruction”, in Israel, S. and G. Duffy (eds.), Handbook of Research on Reading Comprehension, Routledge, New York.
[83] Duckworth, A. et al. (2007), “Grit: Perseverance and Passion for Long-Term Goals”, Journal of Personality and Social Psychology, Vol. 92/6, pp. 1087-1101, http://dx.doi.org/10.1037/0022-3514.92.6.1087.
[82] Dweck, C. (2007), Mindset: The New Psychology of Success, Ballantine Books, New York.
[98] Elliot, A. (2005), “A Conceptual History of the Achievement Goal Construct”, in Elliot, A. and C. Dweck (eds.), Handbook of Competence and Motivation, The Guilford Press, New York.
[72] Elliot, A. and H. McGregor (2001), “A 2×2 Achievement Goal Framework”, Journal of Personality and Social Psychology, Vol. 80/3, pp. 501-519, http://dx.doi.org/10.1037/0022-3514.80.3.501.
[38] Elliot, A. and T. Thrash (2001), “Achievement goals and the hierarchical model of achievement motivation”, Educational Psychology Review, Vol. 13/2, pp. 139-156, http://link.springer.com/article/10.1023/A:1009057102306.
[142] Faubert, V. (2009), “School Evaluation: Current Practices in OECD Countries and a Literature Review OECD Education Working Paper No. 42”, OECD Education Working Papers, OECD, Paris, http://www.oecd.org/edu/workingpapers.
[51] Ferla, J., M. Valcke and Y. Cai (2009), “Academic Self-Efficacy and Academic Self-concept: Reconsidering Structural Relationships”, Learning and Individual Differences, Vol. 19/4, pp. 499-505, http://dx.doi.org/10.1016/j.lindif.2009.05.004.
[10] Fisch, K. and S. McLeod (2009), Did you know? 3.0 – 2009 Edition, http://www.youtube.com/watch?v=PHmwZ96_Gos (accessed on 15 May 2009).
[101] Fisher, C., S. Wallace and R. Fenton (2000), “Discrimination Distress During Adolescence”, Journal of Youth and Adolescence, Vol. 29/6, pp. 679–695, http://dx.doi.org/10.1023/A:1026455906512.
[111] Fivush, R. and J. Hudson (eds.) (1990), Knowing and Remembering in Young Children, Cambridge University Press, New York.
[132] Greene, J. and M. Winters (2009), “The Effects of Exemptions to Florida’s Test-based Promotion Policy: Who is Retained?. Who Benefits Academically?”, Economics of Education Review, Vol. 28/1, pp. 135-142, http://dx.doi.org/10.1016/j.econedurev.2008.02.002.
[130] Griffith, C. et al. (2010), “Grade Retention of Students During Grades K–8 Predicts Reading Achievement and Progress During Secondary Schooling”, Reading & Writing Quarterly, Vol. 26/1, pp. 51-66, http://dx.doi.org/10.1080/10573560903396967.
[120] Guthrie, J. (2008), “Reading Motivation and Engagement in Middle and High School: Appraisal and Intervention”, in Guthrie, J. (ed.), Engaging Adolescents in Reading, Corwin, Thousand Oaks.
[115] Guthrie, J., S. Klauda and A. Ho (2013), “Modeling the Relationships Among Reading Instruction, Motivation, Engagement, and Achievement for Adolescents”, Reading Research Quarterly, Vol. 48/1, pp. 9–26, http://dx.doi.org/10.1002/rrq.035.
[49] Guthrie, J. and A. Wigfield (2000), “Engagement and Motivation in Reading”, in Kamil, M. et al. (eds.), Handbook of Reading Research, Volume 3, Lawrence Erlbaum Associates, Mahwah, NJ.
[40] Guthrie, J. et al. (1999), “Motivational and Cognitive Predictors of Text Comprehension and Reading Amount”, Scientific Studies of Reading, Vol. 3/3, pp. 231-256, http://dx.doi.org/10.1207/s1532799xssr0303_3.
[116] Guthrie, J., A. Wigfield and W. You (2012), “Instructional Contexts for Engagement and Achievement in Reading”, in Christenson, S., A. Reschly and C. Wylie (eds.), Handbook of Research on Student Engagement, Springer Science, New York, http://dx.doi.org/10.1007/978-1-4614-2018-7.
[28] Hanushek, E. and L. Woessmann (2010), “The Economics of International Differences in Educational Achievement”, in Hanushek, E., S. Machin and L. Woessmann (eds.), Handbook of the Economics of Education, Volume 3, Elsevier, Amsterdam.
[139] Hattie, J. (2009), Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement, Routledge, New York.
[150] Hayward, L. (2015), “Assessment is Learning: The Preposition Vanishes”, Assessment in Education: Principles, Policy and Practice, Vol. 22/1, pp. 27-43, http://dx.doi.org/10.1080/0969594X.2014.984656.
[78] Heckhausen, H. (1975), “Fear of Failure as a Self-Reinforcing Motive System”, in Sarason, I. and C. Spielberger (eds.), Stress and anxiety, Hemisphere, Washington, DC.
[104] Heckman, J. (2006), “Skill Formation and the Economics of Investing in Disadvantaged Children”, Science, Vol. 312, pp. 1900-1902, http://dx.doi.org/10.1126/science.1128898.
[15] Heckman, J., J. Stixrud and S. Urzua (2006), “The Effects of Cognitive and Noncognitive Abilities on Labor Market Outcomes and Social Behavior”, Journal of Labor Economics, Vol. 24/3, pp. 411-482, http://dx.doi.org/10.1086/504455.
[156] He, J. and F. Van de Vijver (2016), “The Motivation-Achievement Paradox in International Educational Achievement Tests: Toward a Better Understanding”, in King, R. and A. Bernardo (eds.), The Psychology of Asian Learners: A Festschrift in honor of David Watkins, Springer, Singapore, http://dx.doi.org/10.1007/978-981-287-576-1_16.
[71] Helmreich, R. et al. (1978), “Achievement Motivation and Scientific Attainment”, Personality and Social Psychology Bulletin, Vol. 4/2, pp. 222-226, http://dx.doi.org/10.1177/014616727800400209.
[76] Hembree, R. (1988), “Correlates, Causes, Effects, and Treatment of Test Anxiety”, Review of Educational Research, Vol. 58/1, pp. 47-77, http://dx.doi.org/10.3102/00346543058001047.
[152] Hopfenbeck, T. and G. Stobart (2015), “Large-Scale Implementation of Assessment for Learning”, Assessment in Education: Principles, Policy and Practice, Vol. 22/1, pp. 1-2, http://dx.doi.org/10.1080/0969594X.2014.1001566.
[93] Hulleman, C. et al. (2010), “A Meta-Analytic Review of Achievement Goal Measures: Different Labels for the Same Constructs or Different Constructs with Similar Labels?”, Psychological Bulletin, Vol. 136/3, pp. 422-449, http://dx.doi.org/10.1037/a0018947.
[149] Jonsson, A., C. Lundahl and A. Holmgren (2015), “Evaluating a Large-Scale Implementation of Assessment for Learning in Sweden”, Assessment in Education: Principles, Policy and Practice, Vol. 22/1, pp. 104-121, http://dx.doi.org/10.1080/0969594X.2014.970612.
[94] Kaplan, A. and M. Maehr (2007), “The Contributions and Prospects of Goal Orientation Theory”, Educational Psychology Review, Vol. 19/2, pp. 141–184, http://dx.doi.org/10.1007/s10648-006-9012-5.
[79] Kaye, M., D. Conroy and A. Fifer (2008), “Individual Differences in Incompetence Avoidance”, Journal of Sport & Exercise Psychology, Vol. 30/1, pp. 110-132.
[155] King, G. et al. (2004), “Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research”, American Political Science Review, Vol. 98/1, pp. 191-207, http://dx.doi.org/10.1017/S000305540400108X.
[32] Kirsch, I. et al. (2002), Reading for Change Performance and Engagement Across Countries: Results From PISA 2000, OECD, Paris.
[128] Klauda, S. (2009), “The Role of Parents in Adolescents’ Reading Motivation and Activity”, Educational Psychology Review, Vol. 21/4, pp. 325-363, http://dx.doi.org/10.1007/s10648-009-9112-0.
[41] Klauda, S. and J. Guthrie (2015), “Comparing Relations of Motivation, Engagement, and Achievement Among Struggling and Advanced Adolescent Readers”, Reading and Writing, Vol. 28/2, pp. 239–269, http://dx.doi.org/10.1007/s11145-014-9523-2.
[1] Klieme, E. et al. (2013), “PISA 2012 Context Questionnaires Framework”, in PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, OECD, Paris.
[2] Klieme, E. and S. Kuger (2014), PISA 2015 Draft Questionnaire Framework, OECD, Paris, http://www.oecd.org/pisa/pisaproducts/PISA-2015-draft-questionnaire-framework.pdf.
[21] Klieme, E., C. Pauli and K. Reusser (2009), “The Pythagoras Study. Investigating Effects of Teaching and Learning in Swiss and German Mathematics Classrooms”, in Janík, T. (ed.), The Power of Video Studies in Investigating Teaching and Learning in the Classroom, Waxmann, Münster.
[121] Klieme, E., B. Steinert and J. Hochweber (2010), “Zur Bedeutung der Schulqualität für Unterricht und Lernergebnisse”, in Bos, W., E. Klieme and O. Köller (eds.), Schulische Lerngelegenheiten und Kompetenzentwicklung, Waxmann, Münster.
[134] Kloosterman, R. and P. de Graaf (2010), “Non-Promotion or Enrolment in a Lower Track? The Influence of Social Background on Choices in Secondary Education for Three Cohorts of Dutch Pupils”, Oxford Review of Education, Vol. 36/3, pp. 363-384, https://www.jstor.org/stable/25699590.
[154] Kyllonen, P. and J. Bertling (2014), “Innovative Questionnaire Assessment Methods to Increase Cross-Country Comparability”, in Rutkowski, L., M. von Davier and D. Rutkowski (eds.), Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis, CRC Press, Boca Raton.
[122] Lafontaine, D. et al. (2015), “Equity in Opportunity-To-Learn and Achievement in Reading: A Secondary Analysis of PISA 2009 Data”, Studies in Educational Evaluation, Vol. 47, pp. 1-11, http://dx.doi.org/10.1016/j.stueduc.2015.05.001.
[24] Lafontaine, D. et al. (2017), “Pratiques d’enseignement et compétences en lecture des élèves : qu’évaluent les enquêtes internationales et que peuvent en apprendre les enseignants ?”, in Bianco, M. and L. Lima (eds.), Comment enseigner la compréhension en lecture?, Hatier, Paris.
[34] Lindqvist, E. and R. Vestman (2011), “The Labor Market Returns to Cognitive and Noncognitive Ability: Evidence from the Swedish Enlistment”, American Economic Journal: Applied Economics, Vol. 3, pp. 101-128, http://dx.doi.org/10.1257/app.3.1.101.
[109] Logan, J. et al. (2011), “Children’s Attendance Rates and Quality of Teacher-Child Interactions in At-Risk Preschool Classrooms: Contribution to Children’s Expressive Language Growth”, Child and Youth Care Forum, Vol. 40, pp. 457-477, http://dx.doi.org/10.1007/s10566-011-9142-x.
[112] Markowitsch, H. and H. Welzer (2009), The Development of Autobiographical Memory, Psychology Press, Hove.
[54] Marsh, H. and R. Craven (2006), “Reciprocal Effects of Self-Concept and Performance From a Multidimensional Perspective Beyond Seductive Pleasure and Unidimensional Perspectives”, Perspectives on Psychological Science, Vol. 1/2, p. 133, http://dx.doi.org/10.1111/j.1745-6916.2006.00010.x.
[52] Marsh, H. and R. Craven (1997), “Academic Self-Concept: Beyond the Dustbowl”, in Phye, G. (ed.), Handbook of Classroom Assessment: Learning, Achievement, and Adjustment, Academic Press, Orlando, FL.
[153] Marsh, H. and K. Hau (2004), “Explaining Paradoxical Relations Between Academic Self-Concepts and Achievements: Cross-Cultural Generalizability of the Internal/External Frame of Reference Predictions Across 26 Countries”, Journal of Educational Psychology, Vol. 96/1, pp. 56-67, http://dx.doi.org/10.1037/0022-0663.96.1.56.
[33] Marsh, H. et al. (2006), “OECD’s Brief Self-Report Measure of Educational Psychology’s Most Useful Affective Constructs: Cross-Cultural, Psychometric Comparisons Across 25 Countries”, International Journal of Testing, Vol. 6/4, pp. 311-360, http://dx.doi.org/10.1207/s15327574ijt0604_1.
[22] McKenna, M. and R. Robinson (1990), “Content Literacy: A Definition and Implications”, Journal of Reading, Vol. 34/3, pp. 184-186.
[95] Middleton, M. and K. Perks (2014), Motivation to Learn: Transforming Classroom Culture to Support Student Achievement, Sage Publishing, Thousand Oaks, http://dx.doi.org/10.4135/9781483394480.
[42] Mol, S. and A. Bus (2011), “To Read or not to Read: A Meta-Analysis of Print Exposure from Infancy to Early Adulthood”, Psychological Bulletin, Vol. 137/2, pp. 267-296, http://dx.doi.org/10.1037/a0021890.
[43] Morgan, P. and D. Fuchs (2007), “Is There a Bidirectional Relationship Between Children’s Reading Skills and Reading Motivation?”, Exceptional Children, Vol. 73/2, pp. 165-183, http://dx.doi.org/10.1177/001440290707300203.
[74] Murayama, K. and A. Elliot (2012), “The Competition–Performance Relation: A Meta-Analytic Review and Test of the Opposing Processes Model of Competition and Performance”, Psychological Bulletin, Vol. 138/6, pp. 1035-1070, http://dx.doi.org/10.1037/a0028324.
[62] National Reading Panel (2000), Report of the National Reading Panel: Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction: Report of the Subgroups, National Institutes of Health.
[124] Naumann, J. (2015), “A model of online reading engagement: Linking engagement, navigation, and performance in digital reading”, Computers in Human Behavior, Vol. 53, pp. 263–277, http://dx.doi.org/10.1016/j.chb.2015.06.051.
[127] Neuman, S. and D. Dickinson (2003), Handbook of Early Literacy Research, Volume 1, Guilford Press, New York.
[87] OECD (2017), PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic, Financial Literacy and Collaborative Problem Solving, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264281820-en.
[58] OECD (2016), PISA 2015 Results (Volume I): Excellence and Equity in Education, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264266490-en.
[5] OECD (2015), Education Policy Outlook 2015: Making Reforms Happen, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264225442-en.
[31] OECD (2014), PISA 2012 Technical Report, OECD, Paris.
[86] OECD (2013), OECD Guidelines on Measuring Subjective Well-being, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264191655-en.
[3] OECD (2013), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264190511-en.
[30] OECD (2012), PISA 2009 Technical Report, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264167872-en.
[85] OECD (2011), How’s Life?: Measuring Well-being, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264121164-en.
[123] OECD (2011), PISA 2009 Results: Students On Line: Digital Technologies and Performance (Volume VI), PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264112995-en.
[48] OECD (2010), PISA 2009 Results: Learning to Learn: Student Engagement, Strategies and Practices (Volume III), PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264083943-en.
[47] OECD (2010), PISA 2009 Results: What Students Know and Can Do: Student Performance in Reading, Mathematics and Science (Volume I), PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264091450-en.
[57] OECD (2007), PISA 2006: Science Competencies for Tomorrow’s World: Volume 1: Analysis, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264040014-en.
[89] OECD (2005), The Definition and Selection of Key Competencies: Executive Summary, OECD, Paris.
[91] Opdenakker, M. and J. Van Damme (2000), “Effects of Schools, Teaching Staff and Classes on Achievement and Well-Being in Secondary Education: Similarities and Differences Between School Outcomes”, School Effectiveness and School Improvement, Vol. 11/2, pp. 165-196, http://dx.doi.org/10.1076/0924-3453(200006)11:2;1-q;ft165.
[129] Ou, S. and A. Reynolds (2010), “Grade Retention, Postsecondary Education, and Public Aid Receipt”, Educational Evaluation and Policy Analysis, Vol. 32/1, pp. 215–229, http://dx.doi.org/10.3102/0162373709354334.
[136] Ozga, J. (2012), “Assessing PISA”, European Educational Research Journal, Vol. 11/2, pp. 166-171, http://dx.doi.org/10.2304/eerj.2012.11.2.166.
[90] Partnership for 21st Century Skills (2008), 21st Century Skills, Education & Competitiveness: A Resource and Policy Guide, Partnership for 21st Century Skills, Tucson, AZ, http://www.21stcenturyskills.org.
[44] Pfost, M., T. Dörfler and C. Artelt (2013), “Students’ Extracurricular Reading Behavior and the Development of Vocabulary and Reading Comprehension”, Learning and Individual Differences, Vol. 26, pp. 89-102, http://dx.doi.org/10.1016/j.lindif.2013.04.008.
[35] Poropat, A. (2009), “A Meta-Analysis of the Five-Factor Model of Personality and Academic Performance”, Psychological Bulletin, Vol. 135/2, pp. 322–338, http://dx.doi.org/10.1037/a0014996.
[113] Pressley, M. (2000), “What Should Comprehension Instruction be the Instruction Of?”, in Kamil, M. et al. (eds.), Handbook of reading research, Volume III, Lawrence Erlbaum Associates, Mahwah, NJ.
[65] Pressley, M., S. Graham and K. Harris (2006), “The State of Educational Intervention Research as Viewed Through the Lens of Literacy Intervention”, British Journal of Educational Psychology, Vol. 76, pp. 1-19, http://dx.doi.org/10.1348/000709905X66035.
[66] Pressley, M. et al. (1989), “Strategies That Improve Children’s Memory and Comprehension of Text”, The Elementary School Journal, Vol. 90/1, pp. 3-32, http://dx.doi.org/10.1086/461599.
[151] Ratnam-Lim, C. and K. Tan (2015), “Large-Scale Implementation of Formative Assessment Practices in an Examination-Oriented Culture”, Assessment in Education: Principles, Policy and Practice, Vol. 22/1, pp. 61-78, http://dx.doi.org/10.1080/0969594X.2014.1001319.
[96] Rawsthorne, L. and A. Elliot (1999), “Achievement Goals and Intrinsic Motivation: A Meta-Analytic Review”, Personality and Social Psychology Review, Vol. 3/4, pp. 326-344, http://dx.doi.org/10.1207/s15327957pspr0304_3.
[118] Reeve, J. (2012), “A Self-Determination Theory Perspective on Student Engagement”, in Christenson, S., A. Reschly and C. Wylie (eds.), Handbook of Research on Student Engagement, Springer, New York.
[55] Retelsdorf, J., O. Köller and J. Möller (2011), “On the Effects of Motivation on Reading Performance Growth in Secondary School”, Learning and Instruction, Vol. 21/4, pp. 550-559, http://dx.doi.org/10.1016/j.learninstruc.2010.11.001.
[36] Richardson, M., C. Abraham and R. Bond (2012), “Psychological Correlates of University Students’ Academic Performance: A Systematic Review and Meta-Analysis”, Psychological Bulletin, Vol. 138/2, pp. 353-387, http://dx.doi.org/10.1037/a0026838.
[140] Rosenkvist, M. (2010), “Using Student Test Results for Accountability and Improvement: A Literature Review”, OECD Education Working Papers, No. 54, OECD Publishing, Paris, https://dx.doi.org/10.1787/5km4htwzbv30-en.
[114] Rosenshine, B. and C. Meister (1997), “Cognitive strategy instruction in reading”, in Stahl, S. and D. Hayes (eds.), Instructional Models in Reading, Lawrence Erlbaum Associates, Mahwah, NJ.
[67] Rosenshine, B. and C. Meister (1994), “Reciprocal Teaching: A Review of the Research”, Review of Educational Research, Vol. 64/4, pp. 479-530, http://dx.doi.org/10.3102/00346543064004479.
[68] Rosenshine, B., C. Meister and S. Chapman (1996), “Teaching Students to Generate Questions: A Review of the Intervention Studies”, Review of Educational Research, Vol. 66/2, pp. 181-221, http://dx.doi.org/10.3102/00346543066002181.
[92] Rumberger, R. and G. Palardy (2005), “Test Scores, Dropout Rates, and Transfer Rates as Alternative Indicators of High School Performance”, American Educational Research Journal, Vol. 42/1, pp. 3-42, http://dx.doi.org/10.3102/00028312042001003.
[144] Ryan, K., M. Chandler and M. Samuels (2007), “What Should School-Based Evaluation Look Like?”, Studies in Educational Evaluation, Vol. 33/3-4, pp. 197-212, http://dx.doi.org/10.1016/j.stueduc.2007.07.001.
[13] Rychen, D. and L. Salganik (2003), Key Competencies for a Successful Life and a Well-Functioning Society.
[106] Sammons, P. et al. (2009), “Children’s Cognitive Attainment and Progress in English Primary Schools During Key Stage 2: Investigating the potential continuing influences of pre-school education”, in Roßbach, H. and H. Blossfeld (eds.), Frühpädagogische Förderung in Institutionen, VS Verlag für Sozialwissenschaften, Wiesbaden, http://dx.doi.org/doi.org/10.1007/978-3-531-91452-7_12.
[145] Santiago, P. and F. Benavides (2009), Teacher Evaluation A Conceptual Framework and Examples of Country Practices, OECD, Paris.
[45] Schaffner, E., M. Philipp and U. Schiefele (2016), “Reciprocal Effects Between Intrinsic Reading Motivation and Reading Competence? A Cross-lagged Panel Model for Academic Track and Nonacademic Track Students”, Journal of Research in Reading, Vol. 39/1, pp. 19–36, http://dx.doi.org/10.1111/1467-9817.12027.
[141] Scheerens, J. (2002), “School Self-Evaluation: Origins, Definition, Approaches, Methods and Implementation”, in Nevo, D. (ed.), School-Based Evaluation: An International Perspective, JAI Press, Amsterdam.
[20] Scheerens, J. and R. Bosker (1997), The Foundations of Educational Effectiveness, Pergamon, Oxford.
[46] Schiefele, U. et al. (2012), “Dimensions of Reading Motivation and Their Relation to Reading Behavior and Competence”, Reading Research Quarterly, Vol. 47/4, pp. 427–463, http://dx.doi.org/10.1002/RRQ.030.
[23] Shanahan, T. and C. Shanahan (2008), “Teaching Disciplinary Literacy to Adolescents: Rethinking Content- Area Literacy”, Harvard Educational Review, Vol. 78/1, pp. 40-59, http://dx.doi.org/10.17763/haer.78.1.v62444321p602101.
[37] Shiner, R. et al. (2007), “The Power of Personality: The Comparative Validity of Personality Traits, Socioeconomic Status, and Cognitive Ability for Predicting Important Life Outcomes”, Perspectives on Psychological Science, Vol. 2/4, pp. 313-345, http://dx.doi.org/10.1111/j.1745-6916.2007.00047.x.
[9] Sinicrope, C., J. Norris and Y. Watanabe (2007), “Understanding and Assessing Intercultural Competence: A Summary of Theory, Research, and Practice”, Second Language Studies, Vol. 26/1, pp. 1-58.
[56] Solheim, O. (2011), “The Impact of Reading Self-efficacy and Task Value on Reading Comprehension Scores in Different Item Formats”, Reading Psychology, Vol. 32/1, pp. 1-27, http://dx.doi.org/10.1080/02702710903256601.
[75] Spence, J. and R. Helmreich (1983), “Achievement-Related Motives and Behavior”, in Spence, J. (ed.), Achievement and Achievement Motives: Psychological and sociological Approaches, Freeman, San Francisco, CA.
[100] Stanat, P. and G. Christensen (2006), Where Immigrant Students Succeed - A Comparative Review of Performance and Engagement in PISA 2003, OECD, Paris.
[88] Stiglitz, J., A. Sen and J. Fitoussi (2009), Report by the Commission on the Measurement of Economic Performance and Social Progress, European Commission, Brussels, http://www.stiglitz-sen-fitoussi.fr.
[4] Sykes Gary, B. Schneider and D. Plank (eds.) (2009), Handbook of Education Policy Research, Routledge, New York.
[107] Sylva, K. et al. (2011), “Pre-School Quality and Educational Outcomes at Age 11: Low Quality Has Little Benefit”, Journal of Early Childhood Research, Vol. 9/2, pp. 109–124, http://dx.doi.org/10.1177/1476718X10387900.
[110] Sylva, K. et al. (2011), “Effects of Early Child-Care on Cognition, Language, and Task-Related Behaviours At 18 Months: An English Study”, British Journal of Developmental Psychology, Vol. 29, pp. 18-45, http://dx.doi.org/10.1348/026151010X533229.
[84] Ţuţu, A. and T. Constantin (2012), “Understanding Job Performance Through Persistence and Job Competency”, Procedia - Social and Behavioral Sciences, Vol. 33, pp. 612-616, http://dx.doi.org/10.1016/j.sbspro.2012.01.194.
[105] UNESCO (2006), Education For All Global Monitoring Report 2007: Strong Foundations, UNESCO, Paris, http://dx.doi.org/10.1007/978-94-007-0753-5_101111.
[108] Valenti, J. and D. Tracey (2009), “Full-Day, Half-Day, and No Preschool”, Education and Urban Society, Vol. 41/6, pp. 695-711, http://dx.doi.org/10.1177/0013124509336060.
[103] van de Vijver, F., S. Breugelmans and S. Schalk-Soekar (2008), “Multiculturalism: Construct Validity and Sstability”, International Journal of Intercultural Relations, Vol. 32/2, pp. 93-104, http://dx.doi.org/10.1016/j.ijintrel.2007.11.001.
[157] Van de Vijver, F. and J. He (2014), “Report on Social Desirability, Midpoint and Extreme Responding in TALIS 2013”, OECD Education Working Papers, No. 107, OECD Publishing, Paris, https://dx.doi.org/10.1787/5jxswcfwt76h-en.
[97] Van Yperen, N., M. Blaga and T. Postmes (2014), “A Meta-Analysis of Self-Reported Achievement Goals and Nonself-Report Performance Across Three Achievement Domains (Work, Sports, and Education)”, PLoS ONE, Vol. 9/4, pp. 1-16, http://dx.doi.org/10.1371/journal.pone.0093594.
[119] Vansteenkiste, M., W. Lens and E. Deci (2006), “Intrinsic Versus Extrinsic Goal Contents in Self-Determination Theory: Another Look at the Quality of Academic Motivation”, Educational Psychologist, Vol. 41/1, pp. 19-31, http://dx.doi.org/10.1207/s15326985ep4101_4.
[69] Waters, H. and W. Schneider (eds.) (2010), Metacognition, Strategy Use, and Instruction, Guilford Press, New York.
[16] Willms, J. (2006), Learning Divides: Ten Policy Questions About the Performance and Equity of Schools and Schooling Systems, UNESCO Institute of Statistics, Montreal, http://www.uis.unesco.org.
[29] Woessmann, L. et al. (2007), “School Accountability, Autonomy, Choice, and the Level of Student Achievement: International Evidence from PISA 2003”, OECD Education Working Papers, No. 13, OECD Publishing, Paris, https://dx.doi.org/10.1787/246402531617.
[102] Wong, C., J. Eccles and A. Sameroff (2003), “The Influence of Ethnic Discrimination and Ethnic Identification on African American Adolescents’ School and Socioemotional Adjustment”, Journal of Personality, Vol. 71/6, pp. 1197-232, http://dx.doi.org/10.1111/1467-6494.7106012.
[147] Wylie, E. and C. Lyon (2015), “The Fidelity of Formative Assessment Implementation: Issues of Breadth and Quality”, Assessment in Education: Principles, Policy and Practice, Vol. 22/1, pp. 140–160., http://dx.doi.org/10.1080/0969594X.2014.990416.