This chapter describes the framework and core content for the PISA-VET contextual questionnaires. The chapter presents the content and the aims of the instruments for learners in the targeted occupational areas. It also describes the teacher, VET institution, and employer/trainer questionnaires that are used for the assessment.
PISA Vocational Education and Training (VET)
Assessment and Analytical Framework
PISA
Abstract
As noted in Chapter 1, the focus of the PISA-VET contextual questionnaires is on understanding how measures of student performance at the end of their initial VET programmes relate to practices of educational institution-based learning and work-based learning as well as to other factors from learners’ economic, social, and cultural contexts. The questionnaires of PISA-VET include these aspects; they also cover a broader set of well-being outcomes and of risk and protective factors, accounting for differences in life experiences of the participants in the study. Four questionnaires are proposed: one for the learner, one for the teacher, one for the VET institution, and one for the work-based learning provider.
This chapter begins with a presentation of the core elements of the contextual framework for PISA-VET. It introduces the principles of educational prosperity and life course, and discusses them in the context of VET, with a focus on quality, equality and equity, and inclusion. It also explains how the PISA-VET contextual assessment extends or reinterprets these principles.
In subsequent sections, the chapter presents the structure and organisation of core and additional contents of the PISA-VET contextual instruments. The key informants are introduced, and the chapter explains the approach taken for triangulation of information. The chapter also addresses the rationale for single informants on some aspects of VET.
Finally, the chapter elaborates on how the contextual framework of PISA-VET allows for analysing equity and equality with the information obtained from the four questionnaires.
The chapter also provides guidance to prepare the questionnaires for future development or for pilot testing. The questionnaires are included at Annex B of this framework document.
PISA-VET aims at conducting a comparative international study of learners near the end of their initial VET programmes at the upper-secondary, post-secondary non-tertiary education level, and/or short-cycle tertiary education level (ISCED levels 3, 4, and 5). Across countries and occupational areas, students are in programmes that are predominantly institution-based, as well as in programmes that have a large work-based learning component. Student age varies, both because programmes are provided at different education levels and because, in some countries, they attract not only young learners but also adults - which is an important departure from PISA and from its age-cohort approach.
PISA-VET has a broader range of measurable learner outcomes than PISA, a more diverse student population, and two distinct places to learn and to develop personal and occupational identity: the educational institution, as well as the workplace, (also referred to as the training company1). In line with the PISA framework, three questionnaires are addressed to students, to the principals or administrators of the VET institution, and to teachers in the selected occupational areas respectively. A short questionnaire was designed for the trainer or the person most knowledgeable about the student’s training in the work-based learning environment.
The questionnaires describe students and their learning experience in context, in their physical and social spaces of work and study. The questionnaires establish a connection between the student occupational-specific skills and the employability skills, and the core features of VET institution-based and work-based environments. These core features are: material and human resources, teaching and training practices, the prevalent climate and beliefs, the social interactions, and the organizational processes. The contextual questionnaires allow an extensive data collection on student outcomes by tracing students’ previous education and life experiences that promoted participation, progression, and outcomes in their current VET programme. The questionnaires also include measures of students’ health and well-being, safety, and cognitive and personal engagement.
The framework of the International VET contextual assessment incorporates core and unique elements of the Educational Prosperity model (Willms, 2015[1]) developed for the 2018-19 PISA for Development (PISA-D). To orient the focus of PISA-VET towards the green transition and green skills, the contextual assessment refers to the PISA 2025 Science Framework (OECDa, 2023[2]) particularly in recognizing the greater agency of learners in the face of climate change and knowledge acquisition to respond to it. To orient the focus of PISA-VET towards the digital transition and digital skills, the contextual assessment refers to the PISA 2025 Learning in the Digital World assessment framework (OECDb, 2023[3]). Additionally, the assessment framework borrows and reflects on lessons from international studies on VET and from national assessments and research studies conducted in the participating countries or in the targeted occupational areas.
The framework of the International VET contextual assessment builds on PISA 2022 and especially on the main components of the Questionnaire Framework for PISA 2022 (ETS/OECD, 2019[4]). It shares the same key areas of assessments of students’ outcomes and personal background, of school-based teaching and learning processes, and of school policies and educational governance. Along with these code dimensions, new areas were included in the PISA-VET framework to capture the unique features of work-based learning and training and of the additional stakeholders involved in training and teaching VET students.
The PISA-VET’s overarching framework tasks the contextual assessment’s instruments, or questionnaires, to inform policy makers on how to track and to promote the success of all learners. These instruments measure holistically the success of learners with multiple outcomes. They collect data on the foundational resources and processes that support them, and detailed information on learner-level demographic characteristics. More detail on the diversity of VET systems will be captured through the system-level questionnaire presented in Chapter 9.
The data collected with the contextual instruments can help assess whether good performance of educational systems is obtained at the expense of equality and equity of outcomes for different learner sub-groups.
The framework of the PISA-VET contextual assessment aims at embedding the experience of studying and working in initial VET programmes within a broader description of learners’ life trajectories and their skill development for employment in the labour market.
The framework of the contextual assessment of PISA-VET therefore adopts a life-course approach to assessing learners’ outcomes, and it draws on the Educational Prosperity model (Willms, 2015[1]), which guided the construction of the PISA for Development (PISA-D) contextual assessment. The PISA-D contextual framework was designed to better describe a more diverse population of young adults and their complex lives, in and out-of-school. Several measures introduced in the PISA 2022 questionnaire have embraced the principals and the approaches pioneered in PISA-D. The term “prosperity” used by Willms (2015[1]) and adopted in the framework documents of PISA-D, refers to the condition of experiencing success or thriving. Educational systems and society are committed to deploy key resources and processes to ensure the full development of young people’s cognitive skills, employability skills, and well-being. Educational prosperity indicates the achievement of this goal.
Participation, progression, and academic outcomes at all levels of education build on past educational tracks and depend on successful personal transitions. (Ainsworth and Roscigno, 2005[5]; Alba and Lavin, 1981[6]; Bozick and DeLuca, 2005[7]; Dougherty, 1987[8]; Entwisle and Alexander, 1993[9]). This means that youth enter, progress, and achieve academic and personal goals in secondary and post-secondary education according to stratification mechanisms, by building on prior achievement, resulting from the sifting and sorting processes embedded in educational, work, and social institutions. Health and socioeconomic advantage experienced early in the life course continue to impact youth development, with ripple effects on their outcomes at various ages, from childhood to adulthood. Cumulative advantage and institutional selective mechanisms impact students’ access to quality of education, to acquisition of skills and maintenance of well-being and engagement.
A life course approach explains the yield of learning as well as the timely entry in postsecondary or its premature dropout (NCES, 2005). The interplay of personal transitions, educational and social resources, and institutional mechanisms leads to the construction and negotiation of educational careers that extend into midlife (Astone et al., 2000[10]; Boudett, Murnane and Willett, 2000[11]).
As described in Chapter 9, VET programmes serve different purposes. Most VET programmes target the development of medium-level vocational skills that are immediately relevant for the labour market but can also serve as pathways into further learning. Some programmes equip learners with higher-level vocational skills and lead to postsecondary or tertiary qualifications, supporting entrepreneurial activities, and functioning as innovation hubs for companies. Other programmes focus on lower-level vocational skills and provide job-relevant skills to students at risk of dropping out of school or to those who have already dropped out. These different purposes of VET programmes produce great variability in programme characteristics as well as in student characteristics. One of the main challenges for PISA-VET is portraying while comparing how vocational programmes are organised and delivered across and within countries, and by occupational areas.
To embrace this diversity and to establish some terms of comparison, the contextual framework of PISA-VET focuses on the collection of information on learners’ educational and occupational trajectories and on their personal characteristics to describe learners’ success outcomes, including participation, progression, and personal and educational achievements. The framework guides the collection of data on these outcomes and relates them to the structure of VET programmes, the organisation and the main practices in the venues for learning, and to the resources for learning available and used by teachers, trainers, and learners.
The framework of PISA-VET defines learner success as both a holistic and a cumulative concept. The success of learners enrolled in VET comprises of job-specific learning and skills, employability skills, learners’ health and well-being including work safety, progression and engagement in the programme, aspirations and attitudes towards VET and development of occupational and organisational identity. These outcomes vary by sociodemographic characteristics and by educational and work background. The success outcomes are also associated with learners’ previous experiences, and with current practices, interactions, and resources available in their social, educational, and occupations environments.
The PISA-VET questionnaires collect answers from learners, teachers, workplace trainers, and educational administrators (largely VET institution leaders or principals, or programme administrators) to understand presence, availability and access of key resources and processes that constitute foundations for success.
Following the Educational Prosperity approach of PISA-D, four criteria guided the selection of foundational resources, relationships and processes measured by the PISA-VET questionnaires (OECD, 2018[12]). Foundational resources must be universal; in early learning stages as in the current time of assessment, these foundational factors have been found to be necessary conditions for youth to thrive. Additionally, to be selected, foundational resources must be potent, proximal, and pervasive factors. A potent factor has a strong correlation with a success outcome; quality of instruction, for example, is the single most potent driver of learning. A proximal factor is directly connected to an outcome and related to it. For instance, quality of instruction is also proximal, as it is directly connected to one or more learning outcomes, including attendance and academic achievement, as well as cognitive engagement. But teachers’ opportunity for professional development, which is often looked at as a key driver of learning, is not proximal: in fact, it can be mediated by other “closer” factors, like teaching skills and effective training practices. Finally, a pervasive factor is a driver of several success outcomes. Again, quality of instruction is strongly and directly related to learning, it promotes greater engagement and protects from absenteeism or dropout.
Questionnaire respondents (referred to also as “informants”) are selected to provide a mix of unique perspectives and combined views of outcomes, processes, and resources. The term “triangulation” is used to describe the approach of combining data collection on the same general area of assessment from different informants, using different instruments. For example, learners are the direct and only informants on their educational and work career, but they offer information on training arrangements and school processes that is combined with the view of workplace trainers and teachers. Again, learners describe the relationships and the experiences of inclusion at school and in the workplace, while institution leaders add information on policies and school-wide approaches to support inclusion and safety for all learners.
Following OECD’s Education at a Glance (2011[13]) and the PISA-D contextual framework (2018), the PISA-VET contextual framework defines equality and equity as separate concepts and measures them with a consistent approach.
Inequality is shown by differences among sub-populations in the distribution of their educational outcomes. The measure of equity involves an assessment of fairness based on the observed differences among sub-populations in their access to the resources, relationships and processes that affect learners’ outcomes (OECD, 2011[13]; Willms, 2011[14]). Equality is therefore measured by the differences among groups in the distribution of learners’ success outcomes, which in the case of PISA-VET are the following: occupational specific learning outcomes, employability skills, learners’ educational planned path and educational aspirations; engagement in work and learning; occupational and organisational identity, health and well-being, and work safety.
Equity refers to the possibility for all groups of VET learners to experience success in the form of these outcomes, by having fair access to the foundational resources, relationships, and processes for success. These are the quality of instruction and of training in the school and workplace; inclusive environments, effective usage of institution-based and work-based time; the family’s and the parents’ or the friends’ support; material and human resources; the community support; and a fluid collaboration and effective coordination between VET institution and workplace. Unfair access to the foundational factors increases inequalities in outcomes. For example, for VET learners to use obsolete equipment can impact several success outcomes, including gaining up-to date and relevant occupational skills and learn and work safely.
Equity is a normative concept; the PISA-VET contextual framework assesses it in relative terms – by comparing the levels of inequality in outcomes and in access to the foundations of success to those achieved by other countries, in comparable circumstances (OECD, 2018[12]).
Source: Modified from (OECD, 2018[15]) based on (Willms et al., 2012[16]).
Equality in occupational-specific learning outcomes for learners from different socioeconomic backgrounds is assessed by examining the relationship between their performance in occupational-specific tasks and their socioeconomic status (SES), while equity in the same outcomes is assessed by also examining the relationship between SES and the foundation factors that support and promote job-specific learning and skills.
With reference to Bowman (2004[17]), equity in VET can be effectively assessed and ensured by focusing on policy-specific “priority groups” and by monitoring when and where they experience barriers in the training cycle, from participating in training, progressing through elected VET studies, to achieving the outcomes sought from the training.
The life-course approach at the core of PISA-VET’s instruments guided the construction of questions to nuance the definition of priority groups, to assess accurately equality and equity, and to inform on specific policy changes and interventions that can alleviate or resolve inequities.
The equity priority groups may differ slightly by country and occupational areas, but existing research and national reports point consistently to five types of students to prioritise, identify and to monitor in their access, progression, and acquisition of skills. Those are women, of various age groups; students from lower socioeconomic background; students whose first language is not the language of the country of assessment; migrants; learners with lower levels of proficiency in literacy and numeracy and with a history of school dropout; and students with disability.
The instruments are self-completed questionnaires. These questionnaires may be completed at the time of the testing of the sampled students or at a later date on-line. They include: a student questionnaire; a teacher questionnaire, administered to most of the teachers in the selected occupational areas in each school (or college or programme); and a VET institution questionnaire, administered to the VET institution leader/principal. The VET institution questionnaire can, where necessary, include programme-specific items for each of the occupational areas. Its function is to collect data on the institution as a whole as well as on the specific VET programmes in the selected occupational areas administered in and by the institution.
The questionnaires targeted to the institutions and the teachers will address questions at three different administrative and contextual levels. Some questions will be specific to different occupational area programmes within an institution. Other questions will be about the general institutional context. Finally, questions will be about the context and the activity of the single programmes. As an additional source of data, trainers in the work-based learning environment that are most knowledgeable about the participating learner’s training receive from the learner in training a one-page questionnaire to fill out and return by pre-stamped mail or scanned attachment to an email. The trainer/employer questionnaire will only be administered in countries and programmes that have a substantial work-based component.
Figure 2 maps out the distribution of questions by respondent, for each of the constructs and areas defined by the contextual framework. A dot represents a question. The questions can be drawn from OECD studies; they can be new questions for PISA-VET, or they can be adaptations from questions in other international or national assessments. Several core measures are created with questions drawn directly from various cycles of PISA, including PISA-D, and from PIAAC. The new questions are crafted originally, based on the findings from national and international research studies.
The criteria followed for selecting questions and for developing new items included:
a) Relevance for the VET system and coherence with the goals of the system, which are:
the development of the individual’s potential occupational mobility, self-regulation, and autonomy
the safeguarding of human resources in a society
the warranty of social participation and equal opportunity.
b) Balanced representation of the institutional and individual context factors that are recognised by the international literature and research studies as fundamental for individuals’ competence development.
c) Parsimonious and country-invariant systemic and instructional characteristics and their influences on the development and use of competencies as well as in the promotion of learners’ success.
d) Pervasive in their ability to capture interactions between individual and social factors.
e) When relevant, traditionally and successfully used in other International Large-Scale Assessments, like PISA and PIAAC.
In their initial form, pre-Pilot Phase, the questionnaires are inclusive and comprehensive. Annex B presents them in draft form. The questionnaires are structured along a provisional sequence of relevant topics and related questions. The questions and items presented are an attempt to operationalise the conceptual framework, irrespective of specific occupational areas or a country’s unique features. Because they are very broadly worded, the questions are formatted to accommodate additional items or for conducting adaptations of terminology by country and by occupational area. The questionnaires in Annex B of this document include adaptable terms contained between the symbols < >, as well as blank items for <occupational specific> or <country specific> new items. Further adaptation, specific to programmes within educational institutions will apply to terms like “classroom”, “lessons”, and even “teachers”.
In the Pilot Phase, the learner questionnaire’s content could be tested by designing alternative forms or questionnaire booklets. Each form or booklet would have a common core of questions, and a unique subset of questions. A rotated design in the administration of these forms or booklets contains the length of the questionnaire and the time of response while still allows collecting information on the effectiveness and responses to all the questions drafted in the pre-Pilot Phase version. As noted above, the Pilot Phase would explore the options for students to complete the questionnaire on-line in order to manage the length of the actual assessment time.
VET institution, trainer, and teacher questionnaires are shorter than the learner questionnaire; also, the respondents may be given more time to fill them out. Hence, they could be administered as they are. The Pilot Phase testing may still involve the design and comparison of different layouts and sequences of the same questions.
For the Pilot Phase, and prior to their administration, questionnaires should be finalised by PISA-VET contractors, in collaboration with the OECD and national project managers. To maintain comparability between countries and occupational areas, coordination and adaptation are symbiotic, and instrumental for effective pilot testing. National project managers should coordinate with the PISA-VET contractor and the OECD on the national adaptation of the questions of all the questionnaires. Adaptation involves translation and contextualisation, but also customisation of the country-specific and occupation-specific items that are offered for some of the questions.
The contractors of PISA-VET, with feedback from national project managers, will develop pivots and filters to conduct an initial screening of students and teachers and optimise the sequence of questions that they will answer. The VET institution questionnaire can follow the same approach, if deemed necessary.
The Pilot Phase will also provide the opportunity to ascertain the response rate for the trainer/employer one-page questionnaire, which is designed for an agile and quick completion by the trainer in the work-based learning environment most knowledgeable about the learner. It is necessary to specify further the guidelines for the choice of the person most knowledgeable in the work-based learning environment.
In the PISA study, the student questionnaire begins strategically with questions on students’ demographic and socioeconomic characteristics. This information is instrumental for the construction of the plausible values of the test scores. In the PISA-VET study, if plausible values were to be constructed for VET learner, structuring the flow of questions in the learner questionnaire to ensure full and effective collection of their sociodemographic profile would be necessary.
The PISA-VET contextual framework includes the measurement of learners’ characteristics and background, their success outcomes, the foundational factors that support their success, and additional contextual elements that describe the features of the VET institution and the workplace. The content of these elements is discussed below.
Learner success includes job-specific learning outcomes and employability skills, as described in Chapters 2-6 and Chapter 7 of this framework document respectively and assessed by the cognitive instruments. The contextual assessment focuses on learners’ educational and work-related progression and plans, occupational and educational engagement, health and well-being, and work safety.
The framework and the measures for job-specific learning outcomes and employability skills are described in Chapters 2-6 and Chapter 7 of this framework document, respectively.
The PISA-VET framework adopts a life course approach to measure and examine educational attainment and work-related expectations. The life course perspective offers a long view of lives (Elder, 1994[18]), and helps positioning orientations and attainment at the time of data collection as embedded in ongoing trajectories. The learners’ trajectories are shaped by systems of resources and promotion as well as by social stratification’s mechanisms that impact their lives. The combination of responsibilities and resources at various stages of learners’ lives affect their plans, especially school and work decisions, which depend on the costs of attending, the perceived probability of success, and the utility of completing a degree (Astone et al., 2000[10]; Johnson and Reynolds, 2013[19]).
Learners’ motives for enrolling in VET are typically diverse and their expectations much wider than just gaining job-related skills. VET is also seen as a pathway to further study, either in the VET or academic sector. The learner questionnaire collects detailed information on the trajectories of the learners and allows matching them with learners’ current experience and progress in their VET programmes. The learner questionnaire informs on learners’ intentions and plans to continue and complete the programme, as well as on occupational expectations and plans for education and employment. All the other questionnaires, with exception of the trainer/employer questionnaire, collect additional information on learner absenteeism and truancy.
The PISA studies have examined for more than two decades students‘ interest and motivation in reading, mathematics and science, and their engagement in activities related to these subjects. PISA has also considered engagement more broadly, as students’attitudes towards schooling and their participation in school activities (Willms, 2003[20]). Student engagement is more than a predictor of students’ achievement, both in their school setting and in the workplace.
Student engagement is an outcome that leads to lifelong learning and the likelihood of becoming a productive member of society (OECD, 2018[12]). Like PISA, the PISA-VET learner questionnaire includes measures of institutional engagement, providing information on general attitudes towards VET institution, workplace and learning outcomes, as well as attitudes towards learning activities, both in the educational institution and in the workplace.
The concept of well-being refers to the quality of people’s life. Diener (2006[21]) defines subjective well-being as “an umbrella term for the different valuations people make regarding their lives, the events happening to them, their bodies and minds, and the circumstances in which they live”. Previous cycles of PISA defined well-being beyond students’subjective appraisal of their life quality: “Students’well-being refers to the psychological, cognitive, social and physical functioning and capabilities that students need to live a happy and fulfilling life” (OECD, 2018[12]).
Students‘ well-being incorporates five domains: emotional, physical, social, cognitive, and spiritual well-being. The health and well-being module of PISA-VET focuses on the first two of these domains, while social and cognitive well-being are included in other areas of assessment. Spiritual well-being is not included in the framework. As in PISA, the PISA-VET learner questionnaire asks about general life satisfaction and includes a non-clinical measure of anxiety and depression. Additionally, it asks learners questions about their overall physical health during the past year.
Social well-being pertains to learners’ sense of belonging and their connectedness to others. In PISA-VET, it is measured by questions on attitudes towards VET institutions and the workplace, on learning and inclusive environments focusing, and on connectedness to others, including peers, teachers and trainers or mentors. The development of VET identity can also be seen as key indicator of learners’ well-being.
Health and safety are key components within the programmes delivered by the VET sector, particularly given that competent application of work-specific skills and knowledge includes applying them in a safe manner.
The learner questionnaire of PISA-VET collects information from the learners about the risks and exposure to hazards in the workplace, and the frequency of such exposure. Additionally, the questionnaire includes a question on learners’ awareness of occupational health and safety (e.g. hazards, the rights and responsibilities of both employees and employers).
As discussed earlier in this chapter, the foundations for learners’ success are necessary conditions for success at each stage of the learners’ educational and occupational trajectory. A large body of research provides evidence of the effects of each factor on student outcomes. The foundations selected in PISA-VET are inclusive environments, quality instruction, work-based and school-based learning time, human and material resources, family, partners and friends’ support, and social partner engagement.
Inclusion is a “process of addressing and responding to the diversity of needs of all learners through increasing participation in learning, cultures and communities, and reducing exclusion within and from education” (UNESCO, 2005[22]). Inclusion involves modifications in content, approaches, structures, and strategies, with a shared vision that a system must provide quality learning to all students. UNESCO’s policy guidelines (2009[23]) provide a schema for measuring aspects of inclusion relevant to teachers’, trainers’ and institution leaders’ attitudes and values. Inclusive environments are places in which learners across the categorical boundaries of disability, social class, gender, age, ethnicity, national origin, sexual orientation, and religion can succeed.
Inclusive learning and work environments guarantee and promote a complete learning and training experience, including opportunities to learn, engage in the social life of the programme or the workplace, and feel accepted by their peers, mentors, trainers, and teachers.
PISA-VET collects information on inclusion from learners, teachers, trainers, and VET institution leaders. As in PISA, PISA-VET asks learners to report on their sense of belonging at school. It explores the climate in the VET institution with questions about the safety of their VET institution and in their work-based learning environment, whether they feel safe, accepted, and supported. In the teacher questionnaire, PISA-VET asks teachers about their attitudes and practices towards teaching learners with low literacy levels. Like in PISA, in the PISA-VET institution leaders report on institutional or programme specific policies concerning admission and management of instruction for learners with different skills and proficiency. Additionally, leaders or administrators report on the degree of diversity of the educational environment.
Quality of instruction is the most important driver of student’s learning, both in the educational environment and in the workplace. The literature on general education stresses that effective teachers are goal-oriented, and this is evident in virtually all the contemporary models of effective instruction (Coe et al., 2014[24]).
Theoretical studies in vocational institutions and practical training at workplaces should be considered complementary; they provide different kinds of opportunities for learning (Aarkrog, 2005[25]). Close collaboration between vocational institutions and workplaces creates the ideal setting to provide workplace learning for VET students. Learners require opportunities to recontextualise their theoretical and practical knowledge in new contexts to create new knowledge and practices (Griffiths and Guile, 2003). Collaboration between learners, workplaces, and vocational institutions benefits learning (Savoie-Zajc. and Dolbec, 2003[26]; Virtanen and Tynjälä, 2008[27]; Virtanen, Tynjälä and Eteläpelto, 2014[28]).
PISA-VET recognises that the provision of guidance and quality of instruction is the responsibility of several stakeholders that constitute the educational and-work based community. All four questionnaires designed in PISA-VET involve questions on support and mentorship by teachers and trainers; pedagogical methods; and learning venue cooperation. Quality instruction manifests in learning sessions the educational programme and in continuous learning in the work-based learning or training place.
Guidance is described as a process during which more experienced workers, or trainers and teachers, and novice learners work together. Working together involves experts monitoring, providing help if needed. It also involves providing explanations or information, categorisation, and transferring of explicit and tacit knowledge. Finally, guidance manifests as reflection, through conversations, discussions, and feedback. Additional practices in the training and workplace involve scaffolding and fading; observation and demonstration; and independent work and experimentation (Evanciew and Rojewski, 1999[29]; Filliettaz, 2011[30]; Koskela and Palukka, 2011[31]; Onnismaa, 2008[32]; Tanggaard, 2005[33]).
Following Billett (2002[34]; 2002[35]), PISA-VET definition of quality of instruction incorporates school-based and work-based pedagogical practices on three interdependent areas of guided engagement with work activities. The first area includes everyday participation and the organising of access to knowledge through observing and listening, but also by engaging in tasks of increasing responsibility, accountability and understanding the goals of the required performance. The second area comprises direct guidance and intentional learning strategies that are directed towards developing and promoting values, procedures, and understandings. Guided learning, especially at work, includes the use of modelling, coaching, and scaffolding as well as other techniques to develop understanding and to engage learners in learning for themselves. The third area of guided learning focuses on extending the adaptability of learners’ knowledge to new situations and circumstances. The use of questioning, problem solving, dialogues and group discussions aim at assisting learners to assess the scope and the limits of their knowledge and the possibilities of its transfer to new situations.
The process of guidance clearly involves teachers and trainers, especially in planning and evaluating learners’ workplace learning periods. Discussions between learners and teachers are vital pedagogical elements of workplace learning that help to integrate school learning and workplace learning. Disconnects between goals and didactical practices in school and in the workplace lead to conflicts and learning loss.
These elements are collected from the learner, the teacher, and the trainer/employer questionnaires, with attention to complementarity and triangulation of information from the three sources of information and stakeholders.
Learner related or work-related factors can hinder the frequency and the intensity of access to quality of instruction. Time for learning and time management, in their educational institution and in the workplace are foundations for students’ success.
The PISA-VET questionnaires involve questions on schedule disruption, on learning sessions interruptions and on aspects of classroom management. Additionally, questions on learners’ and teachers’ voluntary or unplanned absenteeism combined with questions on flaws in the management of work-based learning opportunities and effective educational-institution and work-based learning time can help evaluate the actual and effective learning time available, in the VET institution and in the workplace.
Drawing from the approach implemented in PISA-D, the PISA-VET teacher, trainer/employer and VET institution questionnaires adopt a schema set out by Murillo and Roman (2011[36]), which distinguishes between basic services, didactic facilities, and didactic resources:
Basic services at the VET institution include minimal and more advanced physical infrastructure, resources, and services such as potable water, water fountains, bathrooms, electricity, heating and cooling systems, and other infrastructural features.
Didactic facilities refer to places other than the classroom for teaching and learning. These include, for example, training rooms and laboratories/workshops, equipment rooms, simulation rooms, computer rooms, libraries, study rooms and other facilities dedicated to teaching and studying.
Didactic resources can include from very basic materials, such as textbooks, manuals, or whiteboards, through to computers or tablets in the VET institution, laptop computers for learners and teachers, and tools like virtual reality, simulators and other advanced equipment.
Whereas PISA asks principals about their perceptions of school resources (lack of or inadequate physical infrastructure and educational material) and collects information on the availability of information and communication technology resources and internet connectivity, PISA-VET, following PISA-D, designed questions for the VET institution leaders that focus on the availability and conditions of specific elements of school infrastructure and facilities as well as the availability of some basic didactic resources.
Moreover, teachers from the five selected occupational areas respond in their questionnaire to questions on the availability of didactic facilities and resources and on their use of didactic facilities and resources in daily pedagogical practices. Trainers from the work-based learning environment are also asked whether in their workplace they can access dedicated spaces, resources, and equipment to support them in their practice with VET learners.
Human resources are assessed primarily by the teacher and the VET institution questionnaires, which collect information on staffing and qualification as well as occupational specific experience of teachers and staff in the school. Questions on teachers’ sociodemographic characteristics help picturing the teaching body in the educational institution and in the programmes (see “context factors” in the item map). Likewise, the trainer/employer questionnaire offers additional information on the occupational profile of the trainer.
The learner questionnaire for PISA-VET collects information on learners’ family structure and living arrangements, and inquires specifically about the support, approval, and expectations that family members offer them during their education and work. Perseverance and investment in a VET programme can also depend on the resources and the emotional support as well as on the approval that learners perceive and effectively receive from their family members.
Social partnership is a working relationship between trade unions and employers, aiming at improving the prosperity of the company and its employees. Employers and trade unions shape VET and adult learning policies by informing and intervening on training needs, priorities, and delivery of effective training (OECD, 2019[37]). The engagement of social partners ensures that the skillsets embodied in vocational qualifications reflect occupational needs, that the mix of training provision between different occupations matches the mix of demand for jobs of different types, that programmes reflect the broader needs of workers, and that opportunities for work-based learning are of high quality (OECD, 2023[38]). The involvement of employers and worker representatives is relevant at multiple levels in VET: the national, sectoral, regional, local as well as the individual VET institution level. The influence of social partners can be just consultative, or alternatively can involve full decision-making (OECD, 2023[38]). Typically, social partners’ role is stronger in apprenticeships than in school-based VET.
PISA-VET asks questions on engagement with social partners to the VET institution leader. It also relies on the system-level data collection for evaluating the involvement of social partners in VET systems at a more aggregate level.
PISA-VET focuses on the following measures pertaining to learners’ backgrounds that are particularly relevant to identify subgroups for equality and equity analyses: age, gender, socio-economic status, living arrangements, the language spoken at home and the language of instruction in the VET institution and at work, urban/rural status, the previous educational and work experience, their immigrant status, and disability.
Differently from PISA, where all students are age 15 at the time of assessment, PISA-VET assesses learners of different ages, from age 16 up to adulthood, enrolled in various levels of vocational education across secondary and tertiary education.
Like in some national versions of the PISA student questionnaire, the PISA-VET questionnaire captures data about gender identity. As discussed in Chapter 1, the selected occupational areas for PISA-VET cover programmes that have predominantly male learners, some that have mostly female learners, and some that have a more mixed gender enrolment pattern. More and better data on outcomes by gender could contribute to breaking gender stereotypes in VET.
Socio-economic status (SES) refers to the position of a person or family in a hierarchical social structure, based on their access to, or control over, wealth, prestige, and power (Dutton and Levine, 1989[39]; Mueller and Parcel, 1981[40]).
Research studies repeatedly show the presence of a socioeconomic gradient that predicts people’s life outcomes, especially health, education, and future occupation and revenues. Drawing on a life course perspective, PISA-VET recognises the role of SES in enabling learners to maintain career and educational expectations (e.g. (Alexander, Bozick and Entwisle, 2008[41]); Bozick et al, 2010). Socioeconomic status differentiates other aspects of the life course pathways of young people, especially the earlier timing and transitioning into adult roles. Youth from lower socioeconomic status marry, become parents and work full-time at earlier ages than their more advantaged counterparts (Dariotis et al., 2011[42]; Sandefur, Eggerling-Beck and Park, 2005[43]). Lower socioeconomic status young adults may hold more volatile expectations or more frequently drop their plans through assuming adult roles that may conflict with that of students. To measure individual socioeconomic status, PISA-VET uses the measure of SES in PISA, called the index of economic, social, and cultural status (ESCS). The ESCS provides a measure of SES by aggregating information on household possessions (e.g. a room to study, books or electronic devices), and parental predictors for SES such as parental instruction and occupation, combined into an internationally comparable measure.
PISA 2022 has also introduced a question about food security, which PISA-VET maintained.
The PISA-VET student questionnaire collects information on learners’ family structure, allowing to distinguish between those who live withing their parental social unit and those who have formed their family or live alone.
Life course transitions are increasingly demographically dense (Rindfuss, 1991[44]), with multiple transitions occurring in a short amount of time. Employment is the most common social role combined with schooling, and many young adults participate in the two realms simultaneously (Cooksey and Rindfuss, 2001[45]). PISA-VET captures the density and variability of VET learners’ life course trajectories by collecting detailed information on previous educational and occupational experiences. Combined with family and adulthood roles, these can explain how and how successfully learners pursue further educational and career steps.
PISA-VET also asks learners when previous education and occupational experiences took place, and what barriers or personal factors motivated them to change their paths.
Learners’first language, which often is the language still spoken at home or with the family and close friends, may differ from the language of instruction in the VET institution or the language used in the workplace.
Researchers have identified that there is a complex array of factors that impact on the education and training experience of students from a non-country language speaking background, with language proficiency the most important one. VET learners who are born in non-country language speaking countries or who speak a language other than the language of instruction at home can come from a wide range of economic and social backgrounds.
Despite the recognition of this intragroup heterogeneity and the variability of resources and support that the students can access, learners with lower levels of proficiency in the language of instruction and literacy more generally are more likely to experience poor outcomes from training programmes and reduced access to higher-level courses (Robertson and Barrera, 1999[46]). Inadequate basic skills, which referred to both inadequate country language proficiency and insufficient previous education and training, challenge the day-to-day learning progress in the education institution and workplace setting.
Combined with the immigrant experience and lack of local work-experience, low language proficiency is also associated with greater struggles to finding a training venue or job. Low levels of language proficiency have been found to limit awareness of training opportunities, such as apprenticeships and traineeships in non-country language speaking background communities, leaving careers advice programmes in education institutions as the main avenue for distribution of information about and promotion of interest in vocational education and training, and specifically apprenticeships and traineeships (O’Loughlin and Watson, 1997[47]; Robertson and Barrera, 1999[46]; Watson and Pope, 2000[48]).
PISA-VET therefore asks learners “What language do you speak at home most of the time?”, “When did you begin learning the language of instruction?" and “What language did your first learn to read?”
In addition, teachers are asked about their usage of other languages to interact with learners who have low proficiency in the language of instruction.
The school questionnaire includes a variable pertaining to the size of the community, which can be used to determine the VET institution’s rural status. Living in a rural area versus a larger community is sometimes confounded with other student level demographic factors and the analyses will enable to discern this.
The common factors for all migrants are those related to the actual act of migration, of moving countries, separation from family and friends and the task of establishing themselves in a new country. The length of time a VET learner born in another country has been living in the country of assessment has an impact on language proficiency, development of community support networks, stress levels associated with the migration experience, awareness of VET options and potential outcomes, and confidence in accessing support services offered by various providers. Residential location will also have an impact.
The measure of immigrant status follows a long-standing approach used in PISA which is based on questions to students about where they and their parents were born. PISA-VET asks learners if their parents live outside of the country for more than six months of the year.
PISA-VET includes a self-reported measure pertaining to disability. Students indicate the nature of their disability, and specifically whether they have impairment, disabilities, or difficulties concerning vision, mobility, behaviours, or learning.
The VET institution and teacher context questionnaires gather data on other teacher, VET institution and system-level background variables that are expected to help explain student outcomes but are not included in any of the previously mentioned modules.
Like PISA, PISA-VET asks teachers about their age and sex, qualification, employment status, educational background, years of experience and professional development activities. Additionally, it asks teachers about their experience and training in the occupational area assessed, on their multiple teaching jobs or on their work in other jobs, related or not to the occupational area. Like in PISA, in PISA-VET teachers are asked about their job satisfaction.
The teacher questionnaire collects information on pre-service training, SES, and health and well-being. To meet policy requests directly, PISA-VET also needs to address issues related to governance at the system level. Principals of VET institutions in PISA-VET are asked numerous questions on resources and management, including type of school (public vs. private, distinguishing between types of private schools), number of students, average class or programme size, VET institution management and funding, as well as how many full- and part-time teachers work at their VET institution or in the programme. Unique to PISA-VET is a module of questions in the VET institution questionnaire that investigates the relationship with the industry and the workplace, including aspects of coordination, collaboration, and governance. A small number of questions in the VET institution questionnaire capture the efforts to comply with the principles of the green transition, in both the school management and the teaching of students (ETF, 2022[49]).
References
[25] Aarkrog, V. (2005), “Learning in the workplace and the significance of school‐based education: a study of learning in a Danish vocational education and training programme”, International Journal of Lifelong Education, Vol. 24/2, pp. 137-147, https://doi.org/10.1080/02601370500056268.
[5] Ainsworth, J. and V. Roscigno (2005), “Stratification, School-Work Linkages and Vocational Education”, Social Forces, Vol. 84/1, pp. 257-284, https://doi.org/10.1353/sof.2005.0087.
[6] Alba, R. and D. Lavin (1981), “Community Colleges and Tracking in Higher Education”, Sociology of Education, Vol. 54/4, pp. 223-37.
[41] Alexander, K., R. Bozick and D. Entwisle (2008), “Warming Up, Cooling Out, or Holding Steady? Persistence and Change in Educational Expectations After High School”, Sociology of Education, Vol. 81/4, pp. 371-396, https://doi.org/10.1177/003804070808100403.
[10] Astone, N. et al. (2000), “School Reentry in Early Adulthood: The Case of Inner-City African Americans”, Sociology of Education, Vol. 73/3, pp. 133-154, https://doi.org/10.2307/2673213.
[34] Billett, S. (2002), “Toward a Workplace Pedagogy: Guidance, Participation, and Engagement”, Adult Education Quarterly, Vol. 53/1, pp. 27-43, https://doi.org/10.1177/074171302237202.
[35] Billett, S. (2002), “Workplace pedagogic practices: co-participation and learning”, British Journal of Educational Studies, Vol. 50/4, pp. 457–481.
[11] Boudett, K., R. Murnane and J. Willett (2000), “‘Second-Chance’ Strategies for Women Who Drop Out of School”, “Second-Chance” Strategies for Women Who Drop Out of School.
[17] Bowman, K. (2004), Equity in Vocational Education and Training. Research Readings, National Centre for Vocational Education Research (NCVER), Adelaide, https://files.eric.ed.gov/fulltext/ED493980.pdf.
[7] Bozick, R. and S. DeLuca (2005), “Better Late Than Never? Delayed Enrollment in the High School to College Transition”, Social Forces, Vol. 84/1, pp. 531-554, https://doi.org/10.1353/sof.2005.0089.
[39] Bunker, J., D. Gomby and B. Kehrer (eds.) (1989), Overview, methodological critique, and reformulation, The Henry J. Kaiser Family Foundation, Menlo Park, CA.
[24] Coe, R. et al. (2014), What Makes Great Teaching? Review of the Underpinning Research, Centre for Evaluation and Monitoring, Durham University, Durham.
[45] Cooksey, E. and R. Rindfuss (2001), “Patterns of work and schooling in young adulthood”, Sociological Forum, Vol. 16/4, pp. 731-755, https://doi.org/10.1023/A:1012842230505/METRICS.
[42] Dariotis, J. et al. (2011), “Pathways of Early Fatherhood, Marriage, and Employment: A Latent Class Growth Analysis”, Demography, Vol. 48/2, pp. 593-623, https://doi.org/10.1007/s13524-011-0022-7.
[21] Diener, E. (2006), “Guidelines for national indicators of subjective well-being and ill-being”, Applied Research in Quality of Life, Vol. 1, pp. 151-157, Springer, New York.
[8] Dougherty, K. (1987), “The Effects of Community Colleges: Aid or Hindrance to Socioeconomic Attainment?”, Sociology of Education, Vol. 60/1, pp. ,86-103.
[18] Elder, G. (1994), “Time, Human Agency, and Social Change: Perspectives on the Life Course”, Social Psychology Quarterly, Vol. 57/1, pp. 4-15, https://doi.org/10.2307/2786971.
[9] Entwisle, D. and K. Alexander (1993), “Entry into school: The beginning school transition and educational stratification in the United States”, Annual Review of Sociology, Vol. 19, pp. 401–423.
[49] ETF (2022), ENESAT22 - ETF Self-assessment Tool for Excellence..
[4] ETS/OECD (2019), PISA 2021 Context Questionnaires Framework (SECOND DRAFT, Preliminary Version for Sr. Advisors).
[29] Evanciew, C. and J. Rojewski (1999), “Skill and knowledge acquisition in the workplace: a case study of mentor-apprentice relationships in youth apprenticeship programs”, Journal of Independent Teacher Education, Vol. 36/2, pp. 24–53.
[30] Filliettaz, L. (2011), “Collective guidance at work: a resource for apprentices?”, Journal of Vocational Education & Training, Vol. 63/3, pp. 485-504, https://doi.org/10.1080/13636820.2011.580359.
[19] Johnson, M. and J. Reynolds (2013), “Educational expectation trajectories and attainment in the transition to adulthood”, Social Science Research, Vol. 42/3, pp. 818-835, https://doi.org/10.1016/j.ssresearch.2012.12.003.
[31] Koskela, I. and H. Palukka (2011), “Trainer interventions as instructional strategies in air traffic control training”, Journal of Workplace Learning, Vol. 23/5, pp. 293-314, https://doi.org/10.1108/13665621111141902.
[40] Mueller, C. and T. Parcel (1981), “Measures of Socioeconomic Status: Alternatives and Recommendations”, Child Development, Vol. 52/1, pp. 13-30, https://doi.org/10.2307/1129211.
[36] Murillo, F. and M. Román (2011), “School infrastructure and resources do matter: analysis of the incidence of school resources on the performance of Latin American students”, School Effectiveness and School Improvement, Vol. 22/1, pp. 29-50, https://doi.org/10.1080/09243453.2010.543538.
[47] O’Loughlin, T. and I. Watson (1997), Loyalty is a one-way street: NESB immigrants and long-term unemployment, Australian Centre for Industrial Relations, Research and Training, Sydney.
[38] OECD (2023), Building Future-Ready Vocational Education and Training Systems, OECD Reviews of Vocational Education and Training, OECD Publishing, Paris, https://doi.org/10.1787/28551a79-en.
[37] OECD (2019), Getting Skills Right: Making adult learning work in social partnership, OECD, https://www.oecd.org/employment/emp/adult-learning-work-in-social-partnership-2019.pdf.
[15] OECD (2018), PISA for Development Assessment and Analytical Framework: Reading, Mathematics and Science, OECD Publishing, Paris.
[12] OECD (2018), PISA for Development Assessment and Analytical Framework: Reading, Mathematics and Science, OECD, http://doi.org/10.1787/9789264305274-en.
[13] OECD (2011), Education at a Glance 2011: OECD Indicators, OECD Publishing, Paris.
[2] OECDa (2023), PISA 2025 SCIENCE FRAMEWORK (DRAFT), OECD, https://pisa-framework.oecd.org/science-2025/assets/docs/PISA_2025_Science_Framework.pdf.
[3] OECDb (2023), PISA 2025 LEARNING IN THE DIGITAL WORLD FRAMEWORK (SECOND DRAFT), https://www.oecd.org/media/oecdorg/satellitesites/pisa/PISA%202025%20Learning%20in%20the%20Digital%20World%20Assessment%20Framework%20-%20Second%20Draft.pdf.
[32] Onnismaa, J. (2008), “Age, experience, and learning on the job: crossing the boundaries between training and workplace”, Journal of Employment Counseling, Vol. 45/2, pp. 79–90.
[44] Rindfuss, R. (1991), “The Young Adult Years: Diversity, Structural Change, and Fertility”, Demography, Vol. 28/4, pp. 493-512, https://doi.org/10.2307/2061419.
[46] Robertson, R. and S. Barrera (1999), “Workable solutions: final report for Australian National Training Authority”, pp. 55, ANTA, Brisbane.
[43] Sandefur, G., J. Eggerling-Beck and H. Park (2005), “Off to a Good Start? Postsecondary education and early adult life”, in Settersten, R., F. Furstenberg and R. Rumbaut (eds.), On the Frontier of Adulthood: Theory, Research, and Public Policy, University of Chicago Press, https://doi.org/10.7208/chicago/9780226748924.003.0009.
[26] Savoie-Zajc., L. and A. Dolbec (2003), “Co-operative education in the pulp and paper sector in Quebec”, Journal of Workplace Learning, Vol. 15/3, pp. 114–122.
[33] Tanggaard, L. (2005), “Collaborative teaching and learning in the workplace”, Journal of Vocational Education & Training, Vol. 57/1, pp. 109-122, https://doi.org/10.1080/13636820500200278.
[23] UNESCO (2009), Policy Guidelines on Inclusion in Education, UNESCO, Paris.
[22] UNESCO (2005), Guidelines for Inclusion: Ensuring Access to Education for All, UNESCO, Paris.
[27] Virtanen, A. and P. Tynjälä (2008), “Students’ experiences of workplace learning in Finnish VET”, European Journal of Vocational Training, Vol. 44/2, pp. 200–213.
[28] Virtanen, A., P. Tynjälä and A. Eteläpelto (2014), “Factors promoting vocational students’ learning at work: study on student experiences”, Journal of Education and Work, Vol. 27/1, pp. 43–70.
[48] Watson, L. and B. Pope (2000), “Equity in Australian education and training: An examination of access and outcomes data across the sectors”, paper presented at the Australian Vocational Education and Training Research Association Conference, March 23–24, Canberra.
[1] Willms, J. (2015), Educational Prosperity, The Learning Bar Inc., Fredericton, Canada.
[14] Willms, J. (2011), “An analysis plan on educational equality and equity: Recommendations for the OECD Education at a Glance”, paper prepared for the OECD NESLI INES Network for the collection and adjudication of system-level descriptive information on educational structures, policies, and practices (NESLI), UNB-CRISP, Fredericton.
[20] Willms, J. (2003), Student Engagement at School: A Sense of Belonging and Participation: Results from PISA 2000, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264018938-en.
[16] Willms, J. et al. (2012), Assessing Educational Equality and Equity with Large‐Scale Assessment Data: Brazil as a Case Study, Inter‐American Development Bank, Washington, DC.
Bergseng, B., E. Degler and S. Lüthi (2019), Overview: Unlocking the potential of migrants through vocational education and training in Germany, OECD Publishing, Paris.
Jeon, S. (2019), “Vocational Education and Training in Sweden”, OECD Reviews of Vocational Education and Training, OECD Publishing, Paris.
Kis, V. (2020), “Improving evidence on VET: Comparative data and indicators”, OECD Social, Employment and Migration Working Papers 250, OECD Publishing, Paris.
OECD (2019), Getting Skills Right: Making adult learning work in social partnership, OECD Publishing, Paris.
← 1. In some countries, VET learners also spend significant time in intercompany training centres – a setting that could be considered as sitting between the VET institution and the workplace/employer. These types of training centres fall outside of the scope of the Development Phase of PISA-VET. However, the use of such facilities will be explored in the system-level data collection.