PISA Vocational Education and Training (VET)

1. What is PISA-VET?

Abstract

This chapter provides an overview of PISA-VET and explains how the initiative was developed in response to interest from several countries. It describes the purpose of the assessment and its scope, the guiding principles, the target population, as well as the test design for assessing professional skills and employability skills across different systems and the policy implications of this. The chapter also highlights the key features of the initiative, describes the collaborative nature of PISA-VET and sets out the timeline for its Development Phase and the preparation of its Pilot Phase.

How the initiative was developed

Interest in vocational education and training (VET) is increasing. It is high on the policy agenda in most OECD countries and the European Union (EU), as a vehicle to engage learners in learning and facilitate effective school-to-work transitions (see Annex A for a glossary that provides definitions of the main concepts and terms used in this document). It can contribute to developing the skills needed for greener and more digital economies and societies, both by equipping young learners in initial VET programmes with the right skills and by providing opportunities for upskilling and reskilling to adults.

The OECD’s Education Policy Committee (EDPC) first discussed a proposed PISA (Programme for International Student Assessment)-type assessment focused on vocational skills in 2014 but countries decided to defer work on it since the paper-based assessment formats available at that time were not adequate to the task.

During 2020, in response to interest from several countries and considering advances in assessment technology and methodologies, the OECD PISA Secretariat, in collaboration with the OECD’s Centre for Skills (SKC), prepared a new proposal for an international assessment of professional skills of learners undertaking programmes of vocational education and training. The proposal is to leverage recent innovations in assessment technology to assess and compare learners in initial VET programmes (i.e. those programmes offering a first VET qualification, regardless of learner background) in different countries, through the assessment of the professional skills/outcomes of individuals and their learning environments. This will eventually lead to the assessment of the outcomes of VET activities.

The proposal was discussed among interested OECD member countries in a series of meetings during 2021 and into 2022 in which the questions that guided the design of the assessment were formulated. These questions included:

• How to compare achievement in VET across countries and within specific occupations?

• How does achievement in VET relate to differences in the design and delivery of VET programmes (between and within countries)?

• How to relate achievement in VET to per learner expenditures?

• Which VET programmes and occupational areas lend themselves most readily to international comparison?

The OECD Secretariat established a PISA-VET Expert Group (EG) in September 2022 to support the technical work involved in the project. This Group works closely with the OECD on the development of the initiative, including this framework document. The EG is chaired by a representative of Germany and is made up of more than 100 specialists in VET and assessment that are drawn mainly from the participating countries but also includes experts from partner organisations, experts that have been identified by the OECD, together with representatives of employers and trade unions. The OECD formed sub-groups of the EG for each chapter of the framework (see Annex C of this document for details of all the experts that have contributed to the framework) and each of these was led by a lead expert.

A zero draft of the framework was prepared in May 2023, and many VET and assessment experts in the participating countries (known as the Extended Expert Group, or EEG for short) beyond those involved in the EG reviewed and commented on this. At the same time, a wide range of stakeholders, including representatives of employers and trade unions, as well as the OECD’s network of VET and skills experts also reviewed and commented on the zero draft.

The OECD and the EG took account of the feedback on the zero draft as they prepared the current version of the framework, this document. This framework will be used as the basis for the instrument development and methodological study scheduled to take place in 2024-2025. This framework will be revised and updated during 2025 in the light of the instrument development and the results of the methodological study, to clarify any remaining issues regarding the feasibility and manageability of the assessment, test design, learner performance reporting and the likely costs to countries of participating in a large-scale assessment of VET.

Organisation of the framework

This framework is intended to guide the work of the instrument developers, help policymakers understand the purpose and the appropriate use of the assessment results, and inform researchers and educators about methodological choices. It:

• Defines the target occupational areas.

• Specifies the competencies to be tested.

• Provides orientations for task design.

• Indicates appropriate data analysis and reporting approaches, including how to use the proposed proficiency scale.

In line with the recommendations of the OECD’s education committees and the Working Party on the PISA-VET (WP) and the EG, the assessment framework follows the design of the PISA assessment and analytical frameworks and is organised as follows:

• Introductory Chapter (Chapter 1 of this document) – an overview of PISA-VET and explanation of how the initiative was developed; a description of the purpose of the assessment and its scope, the guiding principles, the target population, as well as the test design for assessing professional skills and employability skills across different systems and the policy implications of this.

• One chapter on each of the five selected Occupational Areas (Chapters 2-6 of this document) – a definition of each of the occupational areas and the domains associated with each of these, description of the contexts, processes, underlying capabilities and knowledge content involved in the domains, organising the domains, reference points (such as the occupational profiles of the European Skills, Competences, Qualifications and Occupations (ESCO), the European Qualification Framework (EQF) levels and equivalents from other regions), description of competencies, definitions of the constructs to be measured, assessing the domain, the types of tasks to be used in the assessment, including sample items and task characteristics, how it is proposed that performance in the domain will be measured, including reporting proficiency.

• Chapter on Employability Skills (Chapter 7 of this document) - definition of the domains, description of the processes involved in the domains, organising the domains, description of competencies, definitions of the constructs to be measured, assessing the domains, the types of tasks to be used in the assessment, including sample items and task characteristics, how performance in the domains will be measured, reporting proficiency.

• Chapter on Background Questionnaires (learner, institution, teacher and work-based trainer/employer – Chapter 8 of this document) – describes the framework and core content for the PISA-VET contextual questionnaires. The Chapter presents the content and the aims of the instruments for learners in the targeted occupational areas. It also describes the teacher, VET institution, and employer/trainer questionnaires that are used for the assessment.

• Chapter on System Level Data (Chapter 9 of this document) – description of the contextual variables of VET systems that PISA-VET will use to analyse and interpret the results of the assessment, presentation of the system level questionnaire to be used to collect the data, including highlighting system level data currently collected for the OECD’s Education at a Glance, description of the data collection methodology.

• Annexes

  • Glossary of Terms - Annex A of this document

  • Background questionnaires (learner, teacher, institution, and work-based trainer/employer) – Annex B of this document

  • Details of experts contributing to the framework - Annex C of this document

Purpose of the assessment

Vocational education and training systems vary widely across the world. They have their own government systems and unique frameworks which are in general underpinned by specific legal regulations with often shared ministerial responsibilities, including regional government bodies and diverse funding schemes. They may assign particular roles and responsibilities to relevant stakeholders like companies and social partners and exploit strategic initiatives and programmes to respond to training and labour market developments.

The updated International Standard Classification of Education (ISCED) allocates in a comprehensive approach the different educational programmes and pathways to its hierarchical taxonomy with the aim to globally providing statistical data. In Europe and beyond, the paradigm shift to learning outcomes has become established as a result of the European Qualifications Framework (EQF). The learning outcomes approach basically expresses that the total amount of training hours spent is less important than what VET graduates are able to perform.

PISA-VET does not and could not aim to ultimately produce an international ranking of complex VET systems. It rather targets the cross-border assessment of typical professional skills of VET students the outcomes of which provide VET policy makers and relevant stakeholders on an operational level with meaningful data to inform their national decision-making when seeking to modernise or improve programmes, content, learning settings or concepts according to their national or regional circumstances and objectives. In this regard, the PISA-VET project supports countries in strengthening their skills systems by developing, piloting, and implementing an internationally standardised assessment of outcomes of initial VET programmes.

The aim of the initiative is to enhance comparative policy insights on improving relevance, quality, equity, and effectiveness of initial VET programmes. The initiative will accomplish this by delivering a comprehensive dataset on selected VET learning outcomes and related background factors. This dataset will go beyond participation and employment rates and provide internationally comparable data on the performance of selected VET programmes within systems in key occupational areas in the context of a broader policy environment. In doing so, the initiative will inform decision making, help to further develop the quality of VET provision, provide evidence for competency development of VET learners, facilitate peer learning opportunities between countries, and help to raise the image and promote the profile of VET.

PISA-VET will achieve its aim by collecting and analysing data on learner performance in the tests of professional skills outlined in Chapters 2-6 and the tests of employability skills described in Chapter 7 alongside the collection and analysis of contextual data and system level data presented in Chapters 8 and 9 respectively. Thus, through the collection and analysis of this data PISA-VET will provide policy insights on a range of outcomes and related factors, including:

• Comparative skill levels of learners across countries within specific VET programmes and programme types1 and differences between learners with distinct characteristics.

• Distinct or similar features of VET programmes in selected occupational areas across and within countries and how they are related to outcomes.

• System-level features, such as resources invested in VET and regulations around VET teachers and trainers, and how these are related to outcomes.

To develop meaningful policy insights from the differences that will be observed between countries and programmes, these variations must be contextualised to account for contrasts in programme design and learner background. PISA-VET will provide the means for this contextualisation through the collection of the necessary background data and system level data.

This assessment for vocational education and training is inspired by PISA and follows the PISA model in respect of its framework and approach to assessment. However, the exercise will reflect the distinctive features of vocational education and training when it comes to developing the test methodology and test items. The test design is meant to cover in a coherent way both domain-specific as well as cross-cutting competencies by focusing on selected professional activities that are typical for the work and business process in the selected occupational areas.

As with PISA, the PISA-VET test is designed to provide an assessment of performance at the system (or country) or programme level. Similarly, it is not designed to produce scores for individual learners, and it is not, therefore, necessary for each learner to receive the same set of test items. Thus, PISA-VET adopts an efficient design in which the full set of test material, covering all aspects of the framework, is distributed over many test forms. This procedure enables the OECD to obtain a much greater coverage of the content at a test length that fosters learner engagement and institutional participation, resulting in higher data quality.

This initiative builds on a range of national and industry-specific efforts to measure professional skills, but it will be the first time to apply these methods at scale and across countries. The initiative has the potential to pay huge dividends in terms of improving the labour market relevance, quality and social status of vocational education and training. To achieve these benefits, the OECD, international experts, and participating countries must confirm the validity of the metrics and measurement and the international comparability of results that will come from this assessment. To address this, the OECD has established a step-by-step approach to the development of the assessment framework and the instruments with multiple opportunities for countries to review progress and decide on subsequent work.

Guiding principles

The purpose of PISA-VET will be achieved ultimately through the continuous and regular administration of the survey following a successful Development Phase and Pilot Phase. PISA-VET will be developed and implemented in accordance with the following principles.

PISA-VET principles for 2023-2030:

(i) PISA-VET is policy-oriented and meets the needs of educational policy making and implementation for enhancing teaching and learning in selected VET programmes.

(ii) PISA-VET will be a regularly administered system-level assessment of learning outcomes, well-being and learning environments of learners following selected initial VET programmes.

(iii) PISA-VET measures knowledge, skills, attitudes and values of learners and their capacity to apply these competencies creatively and effectively in work situations and real life.

(iv) PISA-VET provides valid, comparable, and reliable data on selected initial VET programmes across a wide range of countries and over time.

(v) PISA-VET is a collaborative and innovative effort to develop forward-looking assessments.

The scope of the assessment

The purpose of PISA-VET determines its scope, target population and design. With the aim of providing internationally comparative information on the professional skills of learners near the end of their training, the scope of the assessment is sufficient to measure the “yield” of VET programmes, or what skills learners have and can apply in their chosen occupational areas to real-work contexts. The assessment emphasises the mastery of processes, the understanding of concepts, and the application of knowledge and functioning in various work situations and contexts in the selected occupational areas.

The occupational areas and employability skills are treated as domains or subject areas for assessment purposes. The domains covered by the assessment are first defined and then presented and analysed in the chapters of this framework in terms of four inter-related aspects:

• Contexts – the domain specific context or situation in which the tasks and problems arise or are set.

• Processes – the mental strategies, approaches, purposes, concepts, and reasoning that the learners are required to engage with and to employ, formulate or interpret to be successful in the domain.

• Underlying capabilities – the set of fundamental domain specific capabilities that underpins each of the processes.

• Content knowledge – the learners’ knowledge and understanding of the domain content – and the ability to apply that knowledge to contextualised tasks and problems.

The scope and target population for PISA-VET was agreed by the EDPC and PGB, following proposals and recommendations from the PISA-VET Expert Group (EG) and Working Party (WP)2.

Occupational areas

Following discussions with VET and assessment experts from OECD countries and beyond, five occupational areas have been selected as the focus of the assessment’s Development Phase, as follows:

• Automotive Technician occupations

• Electrician occupations

• Business and Administration occupations

• Health Care occupations

• Tourism and Hospitality occupations.

Some of these occupational areas are broad in nature, encompassing several types of occupations, while others have a narrow focus on a particular occupation or set of occupations. As such, where needed, the target population within these occupational areas focuses on one or a set of more narrowly defined occupations.

The occupational areas have been narrowed down during the development of this framework and as the target population has been defined (as shown in the relevant sections below). This is particularly the case for Business and Administration, Health Care and Tourism and Hospitality, which cover a broad range of occupations and associated VET programmes. In these three occupational areas the assessment will focus on the occupations illustrated in Table 1.1 below. The focus of the assessment in each of the five occupational areas is summarised in the later sections of this chapter and is elaborated in more detail in the respective chapters for each of the occupational areas.

Table 1.1. Narrowing of occupational areas for the assessment

Occupational Area	Assessment focus
Business and Administration	Business and administration job family in commercial workplaces
Health Care	Healthcare/nursing assistant
Tourism and Hospitality	Hotel receptionist

When deciding on the selection of occupational areas, a set of selection criteria were used. The occupational areas have been chosen because each of them comprises fields that are of significant economic importance now and are likely to remain so well into the future while demanding professional skills that are sufficiently comparable at the international level. In addition, the choice was also guided by the existence of learner cohorts of sufficient size in participating countries to support reliable, valid, and relevant conclusions for the labour market. A further key consideration was whether there are sufficient resources already available to support measurement of competencies within these occupational areas.

Consideration was also given to achieving a balanced mix of types of occupations and the gender of learners. This mix of occupational areas achieves a balance between those occupational areas requiring mostly Science, Technology, Engineering and Mathematics (STEM)-related competencies and those requiring mostly competencies outside of STEM fields. Gender dimensions are a crucial feature of the assessment and, taken together, the five occupational areas selected for the Development Phase represent an acceptable gender balance in terms of learner population. Two of the occupational areas (electrician and automotive technician) are typically male dominated, one of the occupational areas (health care) is female dominated and two (business and administration and tourism and hospitality) represents both male and female learners equally, while the more narrowly focused occupation of hotel receptionist is female dominated. The assessment at the same time integrates a gender equality perspective into all aspects of the assessment material to promote a less gender-stereotyped view of the occupational areas and associated VET programmes.

In addition, the digital and the green transitions are critical dimensions for VET3. The green transition is necessary to mitigate the consequences of climate change and environmental degradation. The digital transition is an ongoing process that continues to transform the way we live. The green and digital transitions together have the potential to further transform dominant practices in the economy and in society. The five selected occupational areas are each impacted by these transitions, albeit at varying degrees. Therefore, the two transitions are treated as crosscutting issues in the framework for each occupational area. They are also reflected in the framework for employability skills and in the background questionnaires.

While each of the occupational areas selected for the assessment are expected to continue and remain important for the economy and for VET over the longer term, it is important to recognise that the assessment is taking place in a rapidly changing world. All the occupational areas selected for the assessment are affected by rapidly changing contexts and technologies. In the case of automotive technicians, for example, there is a move away from petrol and diesel driven engines towards alternative fuels and electric powered vehicles. These changes are reflected in the framework and the instruments in the same way that they are being reflected in the VET systems that are the focus of the assessment.

Employability Skills

Success in the world of work requires transversal competencies in addition to job-specific knowledge and skills. In a changing world of work, employers increasingly seek employees who adapt easily and can apply and transfer their skills and knowledge to new occupational contexts. This is especially the case in the context of the green and digital transformations which are changing the world of work in rapid and profound ways. It is thus important to supplement the assessment of occupation-specific skills with information on a broader set of cognitive and social and emotional competencies – referred to as employability skills in the contexts of this framework document.

While certain employability skills are important across the occupational areas, others are more relevant and potentially occupation-specific in some areas than in others. The assessment therefore includes a set of “core” or “foundational” employability skills which are considered relevant for all VET learners irrespective of their occupational focus.

Following lengthy consultation with VET and assessment experts, the following “core” employability skills were selected for the Development Phase:

• Collaboration, as defined by the OECD’s Survey of Social and Emotional Skills (SSES).

• Literacy, as defined by the OECD’s Survey of Adult Skills (PIAAC).

• Problem Solving, as included in the OECD’s PIAAC Cycle 2 (adaptive problem solving).

• Task performance (responsibility, self-control, and persistence), as defined by the OECD’s SSES.

When deciding on the selection of employability skills, a set of selection criteria were used. Foremost among these criteria was the condition that only employability skills that had existing international frameworks and instruments could be considered for selection. From a shortlist of employability skills that met this fundamental criterion, the EG selected the four “core” employability skills for the assessment based on their clear understanding of what employers are looking for from newly trained professionals in this regard and their analysis of the foundational skills required by learners to participate effectively in the selected occupational areas.

While various other employability skills met the fundamental criteria for selection and can be considered of importance across occupational areas and hence potentially interesting additions to the “core” set of employability skills, the number of abilities to include in the core had to be kept low due to practical constraints (see below). However, the focus on these “core” employability skills has not precluded the possibility of assessing other abilities that are transversal in nature but show different complexity, characteristics, and shape from a certain occupational area to another (e.g. ICT literacy in business and administration, communication in healthcare, numeracy in automotive technician). The framework therefore includes a broader set of employability skills as elaborated in later chapters of this document, in particular Chapter 7.

In addition, the framework allows for assessment of the core employability skills within the occupational contexts in addition to the PIAAC Literacy and PIACC problem solving tests to reflect the specialised nature of these skills for certain occupations (e.g. literacy skills needed to read patient care plans in the case of healthcare assistants). The further work on measuring these occupation-specific employability skills carried out in the Development Phase will aim at establishing a common understanding, especially regarding these other abilities outlined at Table 7.3 in Chapter 7. Indeed, Table 7.3 in Chapter 7 might serve as a basis for such a common understanding. These distinctions between a core set of foundational employability skills measured using existing PIAAC and SSES instruments and the skills that are applied to each of the occupational areas are elaborated further in the remainder of this Chapter and in the respective Occupational Area and Employability Skills Chapters included in this framework document.

Given the target population for PISA-VET of learners close to the end of their initial VET programmes of learning (see below), it will be a challenge to isolate the skills they have acquired from their VET programme from those they had developed before entry to these programmes – and this represents a particular challenge for employability skills, which can be developed over the lifetime through various learning experiences. At this stage, it is not envisaged that there will be an additional assessment of employability skills of VET entrants in the participating countries. As in PISA, PISA-VET will measure the accumulated learning of the participating learners, not just the skills and knowledge they have acquired in their initial VET programmes. The analysis of contextual data will help to identify those factors that are most closely related to learner performance, for example by taking into account the age of the learners and their pathway into the VET programme. It is therefore possible that the policy implications of certain findings from the assessment may go beyond the VET sector. The two opening phases of the assessment, the Development Phase and the Pilot Phase, will seek to account for differences in these initial conditions when comparing professional skills across countries and to determine the outcomes of VET programmes.

The target population

The target population for the assessment generally are learners nearing the end of initial formal VET programmes in the five occupational areas at the upper-secondary and post-secondary non-tertiary education levels (International Standard Classification of Education (ISCED) Levels 3 and 4). The population of learners enrolled in the sampled VET programmes will be outlined for the assessment based on their programme comparability in terms of learning outcomes (levelled across levels 3-4 of the European Qualifications Framework, EQF, or equivalent for those countries not using EQF). This includes learners in school-based programmes and in programmes that combine school-based and work-based learning (e.g. apprenticeships).

The learners sampled for this programme will be assessed in the last six months of their programmes, as the aim of the assessment is to measure professional skills of learners when they are close to gaining their first VET qualification in readiness to enter their target occupation.

While no age range is specified for the target population, it is likely that most learners sampled for this assessment will have commenced their training programmes at age 16 or 17 and would be aged 18-20 at the time of taking the test. In some countries, however, the learners are likely to be older. The OECD has consulted the participating countries on the target population and these have confirmed the feasibility of targeting these learners and ensuring representativeness and cross-national comparability.

In recognition that for some systems adults are a significant and increasing proportion of the VET student population, adult learners returning to VET after beginning working life, may be included in the assessment, but they will be kept separate for analytical purposes as an additional sample if of interest to participating countries – given their different educational backgrounds, work experiences and life skills.

The precise target population in terms of VET programmes for each occupational area has been narrowed down during the development of the framework. The focus will be on a set of VET programmes or qualifications within the selected occupational areas that are comparable in terms of learning outcomes. The target populations for each of the five occupational areas has been defined as presented in Table 1.2 below.

Table 1.2. Occupational areas and target populations

Occupational Area	Target population
Automotive Technician	Students, apprentices and trainees training to be automotive technicians in the last six months of their initial VET programmes corresponding to ISCED levels 3-4 and EQF levels 3-4, or equivalent
Electrician	Students, apprentices and trainees training to be electricians in the last six months of their initial VET programmes corresponding to ISCED levels 3-4 and EQF levels 3-4, or equivalent.
Business and administration job family	Students, apprentices and trainees training for employment in the business and administration job family in the last six months of their initial VET programmes corresponding to ISCED levels 3-4 and EQF levels 3-4, or equivalent.
Healthcare/nursing assistant	Students, apprentices and trainees training to be healthcare/nursing assistants in the last 6 months of their initial VET programmes corresponding to ISCED levels 3-4 and EQF levels 3-4, or equivalent.
Hotel receptionist	Students, apprentices and trainees training to be hotel receptionists in the last 6 months of their initial VET programmes corresponding to ISCED levels 3-4 and EQF level 4, or equivalent.

The PISA-VET tests

The PISA-VET instrument is a computer-based test designed to be taken by groups of learners on the premises of their VET institution or in their work-based learning environment if this is the main place of training (for example, in the case of apprentices). Given the constraints for the practical implementation of the assessment and the resources likely to be available, the assessment is designed to take place within a three-to-four-hour test window. This window includes time for:

• Introductions, setting up and guidance for the test-takers.

• Approximately 35 minutes for completing the learner background questionnaire discussed later in this chapter (and in more detail in Chapter 8).

• 10 minutes for the non-cognitive skills assessed through a self-report discussed later in this chapter (and in more detail in Chapter 7).

• Between two-three hours for the cognitive test, including any pause time needed for test-takers to adjust to using the simulation environment of the test and time for changing test location, for example, moving from a testing room to a workshop or workplace scenario for demonstration tasks.

The VET assessment experts have found that VET test takers generally endure longer test durations than those participating in classical literacy, numeracy or theoretical tests, such as PISA and PIAAC. Therefore, given the target population for PISA-VET, it could be assumed these test takers are likely to be more motivated and tolerate a longer testing time than PIAAC (one hour) and PISA (two hours) test takers. In addition, the test design envisages more pauses between tasks than is found in PIAAC and PISA to promote student engagement, increase test-taker motivation, reduce test fatigue and increase response rates.

The initial test design presented in this framework allocates two-hours-thirty minutes of the total test window for the cognitive tests, including the cognitive test of core employability skills and occupation-specific skills. However, there is scope to increase the length of time for the cognitive test to, say, three or even four hours depending upon the resources available and confirmation during the Development Phase and Pilot Phase that test taker motivation and test fatigue are not significant challenges.

On the basis of a two-hours-thirty-minute cognitive test window with two pauses, each test form distributed to learners will comprise one 15-minute cluster of test material, one 30-minute cluster of material and one 90-minute cluster of material as follows:

• PIAAC Literacy (15 minutes)

• PIAAC Problem Solving (30 minutes)

• One of the five occupational domains (90 minutes).

It is important to note that if the test time is increased by, say, one hour, then it is most likely that the test forms would comprise two 30-minute clusters of literacy and problem-solving test material from PIAAC and one two-hour cluster of occupation-specific skills material.

The testing material will comprise five kinds of tasks, namely:

• Reading and answering questions on digital texts (e.g. texts containing hypertext and navigation features, such as scrolling or clicking on links) as well as traditional print-based texts (PIAAC literacy).

• Solving scenario-based every day and working-life problems (PIAAC problem solving).

• Scenario-based questions of occupation-specific knowledge and skills.

• Digital simulations of occupation-specific professional skills at the workplace and, where possible,

• Live or recorded demonstrations of occupation-specific professional skills with expert judges4.

These types of tasks are most suitable for an assessment in a VET context and also provide a promising testbed for innovation in PISA and other large-scale assessments. The first two of these tasks are well established in PISA and PIAAC as well as other international large-scale assessments. The third task is also well established in international large-scale assessments, but not in respect of occupation-specific knowledge and skills.

PISA-VET will be the first attempt to incorporate digital simulations and live or recorded demonstrations of occupation-specific professional skills with human judges in an international large-scale assessment. These two tasks are summarised as follows:

• A simulation is a recreation of a real-world situation or task, which is designed to explore key elements of that situation. Simulations are ‘‘interactive events’’ in which ‘‘the environment … is simulated ... but the behaviour is real.’ (Jones, 1995, p. 7[1])’. Simulation offers a simplified representation or imitation of an object or process which may not be directly accessible due to issues of scale, time, risk, or complexity. A controlled set of challenges can be offered, both to train learners, and subsequently to assess their skills. Literature has emerged on the use of technology, recognising both its potential and its limitations (for example in simulating the capacity to address interpersonal challenges). Simulation technology may also facilitate standardisation in assessment, so that candidates face the same, or similar challenges in a final assessment (Nyström and Ahn, 2020, p. 6[2]). These kinds of tasks have been created most successfully in the VET sector by ASCOT + which is a research and transfer initiative of the German Federal Ministry of Education and Research (BMBF). ASCOT stands for “Technology-based Assessment of Skills and Competences in VET”. The “+” denotes transfer into training and examination practice of the results which emerged from the predecessor initiative ASCOT (ASCOT, 2011–2015). ASCOT + has created computer-based learning and assessment instruments which can be used to promote and measure the competencies of trainees in industrial and technical occupations, commercial occupations and healthcare occupations.

• Live demonstrations of occupation-specific professional skills with expert judges have been created and developed by WorldSkills International over a period of sixty years. WorldSkills has built a movement aimed at changing the lives of young people through skills. It has 85 member organisations that have global reach for the purpose of raising the profile and recognition of skilled people, and showing how important skills are in achieving economic growth and personal success. Through competitions, projects, conferences and campaigns WorldSkills has become a global hub for skills excellence and development as well as skills assessment. WorldSkills International has confirmed its willingness to work with the OECD and its Contractor on the development of instruments for PISA-VET that may be based on its demonstration tasks with human judges.

Simulation items have the potential to offer a more engaging experience for learners as they are capable of emulating real-world (and thus more relevant) settings through complex interactions and the use of multiple media, such as animations and sounds. Albeit complex to develop, simulation-based items have the potential to assess skills that are challenging to assess in the abovementioned formats, such as decision-making and situational judgement. Another challenge with simulations is grading – subject matter and assessment experts will be involved in the development of (automatic) marking rubrics and how to collect data on learner strategies to potentialize the reporting of learner outcomes. Experiences with the development of the PISA 2025 innovative domain, which relies on this item format and grading will also be leveraged.

Demonstrations of professional skills assessed by expert judges present even further potential to assess learner skills with even more flexibility, given the possibility of interactions between learners and judges and the nuanced assessment of learner performance. Furthermore, demonstrations can be central for the validity and reception of results as results are directly connected to trained professionals in the field of occupation being assessed. Benefits notwithstanding, demonstrations are challenging to scale in large-scale settings, due to the human resources required, internationally comparable training protocols to be developed and statistical models to be employed, thus demanding their careful addition to the assessment’s test design.

Within the 90 minutes of testing for the occupational areas there will be variation regarding the amount of testing time devoted to scenario-based questions, digital simulations and live or recorded demonstrations. For example, automotive technicians may devote more time to digital simulations, while electricians might spend more time on live or recorded demonstrations with expert judges. The total test material is summarised in the table 1.3 below.

During the Development Phase particular attention will be given to the efficacy of the three types of tasks. In particular, data from the methodological study will be analysed to determine the extent to which learner performance on the knowledge-based tasks and digital simulation tasks are a predictor of performance on the live demonstration tasks. If the first two types of tasks prove to be accurate and strong predictors of performance on the live demonstration tasks the role of the latter in the assessment will be reviewed during the Pilot Phase.

Table 1.3. Total test material by clusters (two-hour test)

Domains	No. of 15-minutes Clusters
PIAAC Literacy	2
Domains	No. of 30-minutes Clusters
PIAAC Problem Solving	2
Domains	No. of 90-minutes Clusters
Automotive Technicians	2
Electricians	2
Business and administration job family	2
Healthcare/nursing assistant	2
Hotel receptionist	2

While in the Development Phase a smaller scale administration of the test is anticipated and will only include the five selected occupational area forms, in the Pilot Phase each test form (employability skills and occupational areas) will be completed by a representative sample of eligible learners from each occupational area to allow for estimations of proficiency on all items by learners in each country/economy and in relevant subgroups within a country/economy, such as males and females, or learners from different social and economic backgrounds.

The test design includes items from each of the domains of literacy, problem solving and the occupational area domains. At this stage, traditional psychometric Classic Test Theory (CTT) and Item Response Theory (IRT) analyses are planned to guide the test design. The test design will feature different test booklets within a rotating block design for each occupational area, including the occupational domain together with literacy and problem solving in order to assure adequate framework coverage. These different test booklets will be designed with common items that work as anchors to build multiple linkages across each domain. In this way, the results from the different booklets will scale together for each domain.

With the initial test design, each booklet allocated to learners comprises one 15-minute cluster of test material and one 30-minute cluster of material for the Employability Skills and one 90-minute cluster for one of the five occupational areas, as illustrated in Table 1.4 below. In total, learners spend 135 minutes on all domains plus two pauses amounting to 15 minutes. Final decisions on test design and analysis will be made in light of findings at the end of the Development Phase and prior to the commencement of the Pilot Phase. In connection with test design, it is important to note that PISA-VET is designed to provide an assessment of performance at the system (or programme) level. Thus, PISA-VET adopts an efficient design in which the full set of test material, covering all aspects of the framework, is distributed over a number of test forms (2x2x2), see Table 1.4. This procedure enables the OECD to obtain a much greater coverage of the framework within feasible survey operational constraints while fostering learner engagement and school/VET institution/work-based learning environment participation.

Table 1.4. Total test material (forms) by domains (135 minutes test)

Test booklet	No.	Domains (15 minutes each)	Domains (30 minutes each)	Domains (90 minutes each)
Automotive Technicians	8	Literacy	Problem Solving	Automotive Technicians
Electricians	8	Literacy	Problem Solving	Electricians
Business and administration job family	8	Literacy	Problem Solving	Business and administration job family
Healthcare/nursing assistant	8	Literacy	Problem Solving	Healthcare/nursing assistant
Hotel receptionist	8	120 minutes	Literacy Problem Solving	Hotel receptionist

An overview of what is assessed in each domain

Box 1.1. Definitions of the domains

Automotive Technicians: the competencies and content knowledge required to perform the duties of an automotive technician employed in a workshop, including servicing, overhauling and troubleshooting light vehicles (passenger cars, light commercial vehicles) of various manufacturers.

Electricians: the competencies and content knowledge required to perform the tasks of installing, commissioning, and maintaining and adjusting electrical systems and components in accordance with regulations and professional standards in a safe way.

Business and administration job family: the competencies and content knowledge required to perform typical tasks and activities related to the management and organisation of business enterprises, including record, organise, store, compute and retrieve information, clerical duties in connection with money-handling operations, travel arrangements, requests for information, and appointment, financial accounting and transaction matters, selling and buying financial instruments, specialised secretarial tasks, and enforcing or applying relevant government rules.

Healthcare/nursing assistant: the competencies and content knowledge needed to provide direct personal care and assistance with activities of daily living to patients and residents in a variety of healthcare settings, such as hospitals, clinics, and residential nursing care facilities together with the capacity to implement established care plans and practices under the direct supervision of medical, nursing, or other health professionals or associate professionals.

Hotel receptionist: the competencies and content knowledge required to perform the necessary tasks and procedures related to the arrival, entry, stay, and departure of guests and customers at hospitality establishments, including the planning and organising of hotel receptionist activities.

Literacy: The ability to access, understand, evaluate, and reflect on written texts to achieve one's goals, to develop one's knowledge and potential and to participate in society (OECD, 2021[3]).

Problem Solving: The capacity to achieve one’s goals in a dynamic situation, in which a method for solution is not immediately available. It requires engaging in cognitive and metacognitive processes to define the problem, search for information, and apply a solution in a variety of information environments and contexts (OECD, 2021[3]).

Box 1.1 above presents definitions of the seven domains assessed in PISA-VET, two of which (literacy and problem solving) are the same as in PIAAC. The definitions for the occupational areas all emphasise functional knowledge and skills that allow one to participate fully in the occupations selected for the assessment. Such participation requires more than just being able to carry out tasks imposed externally by, for example, an employer; it also means being able to participate in decision making. The more complex tasks in the assessment require learners to reflect on and evaluate material, not just to answer questions that have one correct answer. A summary of what is assessed in each domain is presented below.

Automotive technicians

Automotive Technicians (Chapter 2) focuses on the two main capabilities that constitute the proficiency of automotive technicians, namely:

• Investigation capability is defined as an automotive technician’s ability to gather information about light vehicles in workshops, to draw conclusions from the information obtained and to justify the actions taken and results achieved.

• Rectification capability is defined as the ability to establish or re-establish desired states of light vehicles to remove damage, correct malfunctions, maintain functionality, meet manufacturers' specifications, or fulfil customers' wishes.

PISA-VET assesses learners’ performance in the domain of automotive technicians in the workshop context and with a focus on the following core processes or task areas of automotive technicians that are grouped under the two capabilities of investigate and rectify.

Investigate includes the following processes:

• Inspect, which are activities to determine if there are any undesired states of vehicles. It comprises (1) gathering information about the vehicle systems, and (2) drawing conclusions from this information.

• Diagnose aims to identify the cause(s) of a vehicle malfunction and suggest actions to eliminate it. Actions to eliminate malfunctions are “(dis)assemble and replace,” “rebuild and restore,” and “adjust”.

Rectifying includes the following processes:

• Replace and (Dis)assemble includes all actions in which some part, component, subsystem, or operating fluid (e.g. engine oil, coolant, etc.) is removed from a vehicle and either reattached or replaced. There are various reasons for such actions.

• Restore and Rebuild refers to corrective actions that reinstate the function and/or specified state of a broken, damaged, malfunctioning, or excessively worn part, component, or subsystem without replacing it.

• Adjust refers to actions in which automotive technicians change the setting of a car's part, component, or subsystem to specified values.

To complete these processes, skills and knowledge in various content areas are required. Automotive technicians deal with light vehicles, and these consist of several systems, including drive unit, powertrain, undercarriage, assistance, and chassis. These systems, as well as combinations of these systems, provide the knowledge content for this occupational area. Automotive technicians need to know the relevant facts and concepts regarding light vehicles systems as well as the principles that apply to them.

Electricians

The domain of electricians (Chapter 3) is comprised of the set of competencies and content knowledge required to perform the tasks of installing, commissioning, and maintaining and adjusting electrical systems and components in accordance with regulations and professional standards in a safe way.

PISA-VET assesses learners’ performance in the domain of electricians in three contexts: commercial and public sector, domestic, and industrial. In these contexts, there is a focus on the following core processes or task areas of electricians:

• assess and diagnose

• plan and design

• assemble and install

• repair and maintain

• inspect, commission, test and report.

To complete these processes, skills and knowledge in various content areas are required. Electricians require knowledge of electrical theory and fundamentals (including mathematics and science); norms, standards & regulations; materials, components, and tools; electrical systems and applications; configuration and control. Electricians also require underlying capabilities related to the processes and content areas, including:

• Identifying and resolving electrical problems - the ability to identify, diagnose and resolve electrical problems in dynamic situations.

• Responding to customers and colleagues - the ability to impart or exchange information with customers and colleagues, both verbally and “in writing.”

• Collaborating on electrical tasks, problems, projects - the ability to work with other professionals on electrical projects and problems.

• Using digital and communications technology for daily electrical tasks and problems - the ability to use digital technology, tools and networks within daily work routines.

Business and administration job family

Business and Administration (Chapter 4) focuses on a core of four capabilities that constitute the proficiency required by a family of occupations involved in the management and organisation of business enterprises, namely, the capability to:

• retrieve situationally adequate knowledge stocks: Retrieving business and administration data and other relevant information.

• describe system coherences and to act within them: Drawing conclusions from business and administration data and other relevant information.

• think and act purposefully in specific situations: Making appropriate decisions/proposals for business and administration work products.

• independently control processing procedures: Establishing appropriate, quality-assuring working practices for business and administration workplaces as well as promoting their own professionalism.

PISA-VET assesses learners’ performance in the domain of business and administration in the context of operational value chains in business and administration. The operational value chain is a framework used to describe the various processes and activities involved in producing and delivering goods and services to customers by business enterprises. Within this contextual framework, the assessment focuses on the following core processes or task areas of business and administration:

• identifying and understanding business and administration information

• analysing business and administration information and situations

• applying business and administration knowledge and skills

• evaluating and reflecting on business and administration issues.

To complete these processes, skills and knowledge in various content areas are required. These content areas include knowledge of:

• The significance of relevant operational indicators in business administration and how to interpret and present data appropriately.

• Appropriate procedures and tools for correspondence and communication, of cross-cultural differences, and of dealing with diverse customers and stakeholders.

• Methods, forms, and areas of application of project management and of the use of concrete work strategies.

Healthcare/nursing assistant

The domain of healthcare/nursing assistant (Chapter 5) is comprised of the set of competencies and content knowledge required to perform the tasks of providing direct personal care and assistance with activities of daily living to patients and residents in a variety of healthcare settings.

PISA-VET assesses learners’ performance in the domain of healthcare/nursing assistant in five contexts: hospitals (inpatient or outpatient facilities, specialist or general); clinics for specific purposes and conditions (e.g. psychiatric, geriatric); residential nursing care; hospices for terminal care; clients’ and patients’ own homes. In these contexts, there is a focus on the following core processes or task areas of healthcare/nursing assistant:

• Working in professional teams.

• Identifying needs and collaborate in healthcare planning.

• Supporting and enhancing clients’ quality of life.

• Providing and supporting treatment and medical processes.

• Reviewing and evaluating care.

To complete these processes, skills and knowledge in various content areas are required. Healthcare/nursing assistants require knowledge of:

• Multi-professional collaboration.

• Communication and relationship building.

• Medical and related sciences.

• Client/patient care, service, and assistance.

• Administrative and legal frameworks.

Healthcare/nursing assistants also require underlying capabilities related to the processes and content areas, including: collaboration with others to achieve optimal healthcare; open-mindedness throughout the healthcare process; engagement with others to achieve therapeutic outcomes; communication as a therapeutic instrument; emotional regulation to achieve optimal healthcare; access to and use of evidence and information to support healthcare solutions; management of self, time, tasks, and workspace to optimise health outcomes; coordination of mind, feelings, and movement for dextrous, respectful healthcare; contribution to healthcare solutions; and reflective practice for healthcare.

Hotel receptionist

Hotel receptionists (Chapter 6) focuses on the main capabilities that constitute the proficiency of hotel receptionists, namely:

• Hospitality service orientation

• Hotel receptionist interpersonal and communication skills

• Collaboration with other hotel staff

• Problem Solving in the context of hotel reception

• Task performance in the context of hotel reception

• Critical thinking in hospitality service

• Literacy in hospitality service

• ICT Literacy for hotel reception.

PISA-VET assesses learners’ performance in the domain of hotel receptionists in the hotel context and with a focus on the following core processes or task areas:

• check-in and check-out

• reservations

• management of complaints

• touristic information.

To complete these processes, skills and knowledge in various content areas are required. Hotel receptionists need to know about:

• hotel reservation procedures

• checking-in procedures

• administration and back-office procedures

• checking-out procedures

• promotion of the local area.

Hotel receptionists need to know the relevant facts and concepts regarding these procedures as well as the principles that apply to them.

Literacy

PIAAC defines literacy (Chapter 7) as the ability to "access, understand, evaluate and reflect on written texts to achieve one's goals, to develop one's knowledge and potential and to participate in society" (OECD, 2021, p. 186[3]). In this, literacy refers to one's ability to comprehend and use written sign systems. Literacy encompasses a range of skills, from decoding written words and sentences to comprehending, interpreting, and evaluating complex texts.

The definition of literacy in PISA-VET is strongly rooted in the literacy framework of PIAAC. PISA-VET assesses learners’ performance in literacy as follows:

• The assessment will include units developed within the PIAAC literacy framework designed to measure foundational skills.

• The assessment will be based on existing and validated PIAAC units, using national versions for participating countries and economies whenever available.

• Three main contexts were defined in PIAAC literacy (“work and occupation,” “personal use” and “social and civic contexts”); if feasible, units should be chosen from the context “work and occupation.”

• Linking with a domain-embedded assessment within occupations in addition to the core generic assessment of literacy is an option.

Problem-solving

The definition of problem solving (Chapter 7) in PISA-VET is strongly rooted in the adaptive problem-solving framework of PIAAC Cycle 2. PIAAC Cycle 2 defines problem solving as “the capacity to achieve one’s goals in a dynamic situation, in which a method for solution is not immediately available. It requires engaging in cognitive and metacognitive processes to define the problem, search for information, and apply a solution in a variety of information environments and contexts” (OECD, 2021[3]).

PISA-VET assesses learners’ performance in problem solving as follows:

• Their assessment will include units developed within the PIAAC Cycle 2 adaptive problem solving as foundation.

• The assessment will be based on existing and validated PIAAC units, using national versions for participating countries and economies whenever available.

• Three main contexts were defined in PIAAC adaptive problem solving; if feasible, units should be chosen from the context “work”.

• Linking with a domain-embedded assessment within occupations in addition to the core generic assessment of problem solving is an option.

Reporting learner performance in PISA-VET

At this stage, it is proposed that the results from PISA-VET will be reported using numeric scales and proficiency levels. However, modern digital assessments offer a host of other options, and some two-dimensional and multi-dimensional metrics (as discussed in Chapter 3 in particular) may be better suited for the purpose of PISA-VET. These options will be explored during the Development Phase. Based on the PISA experience with a numeric scale, it is envisaged currently that the participating average score for all five occupational area domains will be set at 500 with a standard deviation of 100, which means that two-thirds of learners across participating countries should score between 400 and 600 points. These scores represent degrees of proficiency in a particular domain. The PIAAC literacy scale and the PIAAC problem solving scale will be used for the literacy and problem-solving tests that are based on existing PIAAC items. No link across the PIAAC and occupational scales is planned. Each of the learners taking PISA-VET will therefore receive three separate scores as follows:

• PIAAC Literacy

• PIAAC Problem Solving.

• Occupational area (one of the five occupational areas).

The proposed reporting of performance in PISA-VET in the seven domains (PIAAC literacy, PIAAC problem solving and the five occupational areas) is discussed in each of the occupational area and employability skills chapters. As in PISA, the proficiency levels proposed in each occupational domain are descriptors of learner proficiency constructed through scale anchoring methods. However, as noted above, other options for reporting, such as multi-dimensional metrics, will be considered during the second half of the Development Phase.

Occupational areas

At this stage, the proficiency levels that are anticipated for each occupational area are based on notional cutpoints. In some cases (automotive technicians and electricians), these cutpoints refer to actual standards for performance while in others (business and administration job family, healthcare/nursing assistant and hotel receptionist) the cutpoints refer to a continuum from “novice” to “expert.” Given that there is no pre-existing definition of proficiency levels in the five domains, the final decisions on reporting, especially the number of cutpoints and the bandwidth for proficiency levels will be determined ex post the Development Phase, based on the reliability of the scale and the number and difficulty of tasks included in the instruments developed for the assessment,

In the meantime, the notional proficiency levels and cutpoints proposed for each occupational area are summarised in Table 1.5.

Table 1.5. Summary descriptions of proposed performance reporting

Occupational area	Components/sub-components scales	Anticipated proficiency levels and cutpoints
Automotive technicians	Domain of automotive technician scale.	Three proficiency levels that will be based on a numeric scale. A common automotive technician’s scale will be a real composite (compound) measure made up of the reports on two competencies: investigation and rectification.
Electricians	Domain of professional electrician scale.	Six proficiency levels that are defined based on numeric scale and described in terms of typical task requirements and learners’ performance.
Business and administration job family	Domain of business and administration scale.	Five proficiency levels that will be based on a numeric scale. A common business and administration scale will be a real composite (compound) measure made up of the reports on two competencies: occupation-specific and occupation related.
Healthcare/nursing assistant	Domain of professional healthcare/nursing assistant scale	Six proficiency levels that are defined based on numeric scale and described in terms of typical task requirements and learners’ performance.
Hotel receptionist	Domain of professional hotel receptionist scale	Six proficiency levels that are defined based on numeric scale and described in terms of typical task requirements and learners’ performance.

Literacy

The proficiency scales used to report PIAAC results describe what learners typically know and can do at given levels of proficiency. The numeric scales are divided into proficiency levels that carry a descriptive purpose and aid the intepretation and understanding of the scales. PISA-VET will employ the PIAAC literacy scale as described in the PIAAC reports. As the PIAAC literacy framework was updated for Cycle 2, this scale will be updated before reporting. For information, in PIAAC Cycle 1, there were five proficiency levels leading to six categories: proficiency at level 5, proficiency at level 4, proficiency at level 3, proficiency at level 2, proficiency at level 1, proficiency below level 1. Considering that the assessment time in PISA-VET will be limited, the level of detail might be lower as compared to the original PIAAC scale, for instance, maybe only three levels of proficiency (for instance, low, medium, high) can be distinguished in PISA-VET.

Problem Solving

PISA-VET will employ the PIAAC adaptive problem solving as described in the PIAAC framework. As of now, there is only a general description of different proficiency levels in the PIAAC adaptive problem-solving scale from Cycle 2 (for details, consult PIAAC Cycle 2 assessment framework: (OECD, 2021[3])). These are based specifically on behavioural descriptions that were offered in the PIAAC Cycle 2 framework for low and for high scorers on the three task dimensions, i.e. when (a) confronted with different problem configurations, (b) confronted with different dynamics in a situation, and (c) confronted with different features of the environment. These might also be used for PISA-VET, but – as for literacy - the level of detail might be lower.

Employability Skills (non-cognitive)

In addition to the reports on learner performance noted above, PISA-VET also covers two employability skills that are non-cognitive, namely: task performance and collaboration (as described in Chapter 7).

The definitions of task performance and collaboration in PISA-VET are strongly rooted in the OECD’s survey on social and emotional skills (SSES) framework. The OECD defines “task performance” as a range of constructs that describe the ability to be self-controlled, responsible to others, hardworking, motivated to achieve, honest, orderly, persistent and rule-abiding. In short, it refers to the skills that enable individuals to get things done, as required and on time. The OECD defines “collaboration” as a range of constructs that describe the ability to understand, feel and express concern for others’ well-being, manage interpersonal conflict and maintain positive relationships and beliefs about others (trust). In short, it refers to the skills that enable individuals to have a good relationship with other people and work successfully together in various contexts.

The assessments of task performance and collaboration will be based on self-reports that will be part of the background questionnaire. The assessments will be based on existing and validated items from the OECD-convened SSES; depending on the specific wording of the items, they might need to be adapted to the target population, which will be, on average, older than the SSES population. Linking with a domain-embedded assessment within occupations in addition to the core generic assessments of task performance and collaboration will not be explored during the Development Phase (mainly due to the different formats and the difficulty in assessing task performance within occupational contexts).

Proficiency levels were not developed for task performance and collaboration in SSES, although reporting scales were. For the recently completed Round 1 of SSES data collection, psychometric scales were developed using the assessment items for each skill. The reference was value fixed at 500 and standard deviation set to 100. The scale for each of the two sub-skills of task performance (e.g. persistence, self-control) features two meaningful poles. Taking the “self-control” scale, as an example, respondents towards the highly self-controlled pole, reported themselves as more inclined to be careful with tasks, to think before speaking, and to postpone fun until they are finished with work. Likewise, the scale for each process of collaboration (i.e. empathy, trust, co-operation) included in SSES features two meaningful poles. Taking “empathy,” for example, respondents towards the highly empathetic pole, reported themselves as more inclined to consider others’ wellbeing and their perspectives.

The PISA-VET contextual questionnaires

The focus of the PISA-VET contextual questionnaires (Chapter 8) is on understanding how measures of performance in initial VET programmes are related to various aspects of VET institution, classroom, and work-based practice as well as other related factors, such as economic, social, and cultural context. The PISA-VET questionnaires include these aspects and cover a set of well-being outcomes.

The contextual framework for PISA-VET

The PISA-VET contextual framework with its four questionnaires (see Annex B) provides policy makers with information on: outcomes that measure holistically learners’ success; foundational resources and processes that support learners’ outcomes; and learner-level demographic characteristics that allow assessing whether performance of VET systems is obtained at the expense of equality and equity of outcomes for different learner sub-groups.

The framework describes processes and resources that support two major roles of VET: building platforms of skills and competencies for young commencing workers and promoting or enriching skills for established workers. Learners’ success is defined as acquired skills and competencies; engagement, persistence in the programme and educational and training attainment; and finally, health and well-being. The framework elects three areas of foundational resources and processes within the VET institution and the work-based learning places that support and promote learners’ outcomes. The three areas consistently appear in educational research as directly connected to learners’ outcomes. Specifically, a large body of applied research findings identified them as proximal to learners’ outcomes; as pervasive in their relationships with more than one outcome; and as having potent effects on learners’ success. The three areas include resources and processes pertaining to effective learning time, in the VET institution and in work-based learning; quality of teaching and instructional/training practices; and quality and suitability of human and material resources for education and training.

These resources and processes, when effectively deployed and fully accessed by learners, support their active participation to formal education and training; they sustain learners’ progression through elected VET studies. Finally, they promote the full achievement of the outcomes sought from the training. Additionally, the framework acknowledges the permeability of the VET system to the labour market and describes exchanges between the educational programmes and the social partners, as well as the broader community.

The framework places great emphasis on equality and equity, with equality referring to differences among sub-populations in the distribution of their VET outcomes and equity referring to differences among sub-populations in their access to the resources and VET processes that affect VET outcomes.

The four questionnaires developed under this framework explore 15 modules of content. These modules measure the seven Learner Success Outcomes, the nine Foundations for Success, and the ten demographic factors relevant to assessing equality and equity that are shown in Table 1.6 below. In addition, the questionnaires include several teachers, trainer, VET institution, employer, and system-level background measures that provide context for the Learner Success Outcomes. Chapter 8 presents the PISA-VET contextual framework in detail and the questionnaires are included at Annex B.

Table 1.6. Modules assessed in PISA-VET

VET programme/learning outcomes	Foundations for Success	Demographic factors for assessing equality and equity
Job-specific learning outcomes (the five occupational areas). Employability skills (collaboration, literacy, problem solving and task performance). Students’ educational and work-related progression and plans. Learner engagement in school and work. Health and well-being. Work safety: knowledge on safety and experience of injuries.	Inclusive environments. Quality of instruction: Instructor collaboration and mentorship Peer mentorship and support Work-related and general learning tasks Pedagogical methods Learning venue co-operation Education institution-based and work-based learning time. Material and human resources. Family, friends’ and partners’ support for learners. Social partners’ engagement.	Age. Gender. Socio-economic status. Family structure and living arrangements. Previous education and work career. Language spoken at home and language of instruction. Urban/rural status. Immigrant status. Disability.

The questionnaires

The questionnaires for learners, teachers of the occupational areas covered by the assessment, and the principals/leaders of VET institutions, as well as the questionnaire for employers (the establishments responsible for the work-based learning) have been developed in accordance with the contextual framework. These questionnaires take about 35 minutes for the learners to complete, about 30 minutes each for teachers and the institution leaders/principals, and 10-15 minutes for the person most knowledgeable about the respondent’s training at the work-based training site. The responses to the questionnaires are analysed with the assessment results to provide at once a broader and more nuanced picture of learner, VET institution, and programme/system performance. These questionnaires seek information about:

• Learners and their family backgrounds, including their economic, social, and cultural capital, and the language they speak at home versus the language of instruction.

• Aspects of learners’ lives, such as their level of educational attainment, their health and well-being, and their engagement with school.

• Aspects of learning, including quality of instruction, inclusive environments, learning time, school material resources and family and community support.

• Contexts of learning, including teacher, VET institution, employer, and system-level information as well as impacts of the green and digital transformation.

System level data

The contextual information collected through the learner, institution, teacher, and trainer/employer questionnaires comprises only a part of the information available to PISA-VET. System-level data (Chapter 9) describing various features of VET systems will be used in PISA-VET analysis and reports. This system-level data includes information on the place of VET in national skills systems, pathways into and from VET (e.g. stratification and selection, access to higher levels), the profile of VET learners, the delivery of VET (e.g. provider institutions, the use of work-based learning), skills targeted by VET (e.g. targeted fields of study), teachers and trainers (e.g. teacher training and salaries), finance and employment outcomes. While some of the system-level data may cover the overall VET system, certain elements will focus on the occupational areas or selected programmes within those areas – given that regulations and practices might differ between programmes.

The system-level questionnaire (Chapter 9) is an OECD tool designed to complement existing data, providing further contextual information on VET systems to underpin the interpretation and analysis of the results of PISA-VET. The system-level questionnaire to be used in the PISA-VET implementation consists of a set of worksheets that capture data on system-level features on VET. Available data on all the indicators have been reviewed for the data collection countries, identifying the status of system-level data collection and availability in terms of quality and completeness, in effort to design a system-level questionnaire that focuses on existing data gaps.

A collaborative project

PISA-VET is a highly collaborative effort carried by the OECD Secretariat, twelve participating countries, national experts and experts hired by the OECD together with the support of several institutional partners. From the outset of the project, the OECD has engaged the participation of the key international agencies and programmes concerned with VET learner assessment and improving the quality of VET. These partners include the United Nations Educational, Scientific and Cultural Organisation (UNESCO), the European Centre for the Development of Vocational Training (CEDEFOP), the European Commission, the European Training Foundation (ETF), International Labour Organisation (ILO), VALID-8 network testing company, World Bank and WorldSkills International. Experts from ASCOT+, a research and transfer initiative of the German Federal Ministry of Education and Research, were also closely involved.5 Representatives of these institutions and initiatives have been consulted on all aspects of project design and development and they have also contributed to the preparation of this framework document.

The OECD also sought to have a strong involvement of employers in the design of the initiative, particularly through the auspices of Business at OECD (BIAC). Employers are key, as they know the skills the labour market needs so that the offer and content of VET programmes can be closely aligned to those needs. Employers are often also an important provider of VET, especially in countries with a large apprenticeship sector, and research consistently shows that it is the work-based component of VET that makes it effective. Given the essential role of businesses, corporations, and other employers in VET, it is vitally important that the PISA-VET initiative benefits from their inputs and partnership. The initiative is currently in its Development Phase and the employers’ understanding of skills needs, their expectations of VET graduates and their experiences of training VET learners will provide crucial information to help the OECD to design a relevant assessment and to ensure the validity and credibility of the assessment. In addition, the OECD has sought the engagement of trade unions in the initiative (mostly representing teachers’ unions), particularly through the auspices of the Trade Union Advisory Committee (TUAC) to the OECD.

Implementation of PISA-VET

PISA-VET is being developed and implemented in three phases over the course of 2023 to 2030.

1. The Development Phase from 2023-2025 would include the following activities:

• Draft the assessment framework, including the identification of occupational areas and the skills to be assessed within those areas.

• Undertake a consultation with the business sector in the field of selected occupational areas to ensure the labour market relevance of the assessment.

• Select the most suitable target population.

• Develop a set of measurement instruments (scenario-based units, digital simulations of workplace situations, and/or demonstration tasks).

• Undertake a methodological study comparing the reliability, validity, and costs of the instruments developed for assessing the five selected occupational areas in four data collection countries (Australia, Germany, Portugal and the United Arab Emirates).

• Complete an operational document that describes the second phase of instrument development and implementation, including target population, test design, reporting and sampling design, costs of participation and recruit countries to participate in the Pilot Phase.

2. The Pilot Phase from 2025-2029, subject to the agreement of the EDPC and PGB and following the successful completion of the Development Phase and the recruitment of a minimum number of countries that can commit resources to undertake further instrument development and data collection, would include the following activities:

• The further development and validation of the assessment instruments developed during the Development Phase.

• The implementation of the data collection (occupational areas, employability skills, contextual data and system level data) in participating countries (field trial and main survey).

• The analysis of the data and the preparation of an international report, including confirmation of the feasibility, manageability and affordability of the assessment together with recommendations for the implementation of a large-scale PISA-VET.

At the end of the Pilot Phase, it should be possible to confirm how much of the VET sector can be covered by PISA-VET and the extent to which results can be informative for occupational areas not included in the assessment. It should also be possible at that stage to confirm the limits of the approach and whether there are any VET sectors that cannot be compared internationally.

The Large-scale Implementation Phase from 2029-2032, subject to a successful implementation of the Pilot Phase, countries could decide to continue with an international assessment of VET on a regular basis as per PISA to collect timely data on the quality and effectiveness of VET. Such regular data collection would allow observations of trends, which provide a valuable tool for measuring the impact of reforms. If there was sufficient interest and funding, it would also be interesting and help to establish the validity of the assessment to do a follow-up study a few months after the Pilot Phase, to see how the participating learners integrated into the labour market or further education. Future rounds of data collection could also expand the exercise to a wider range of occupation areas and skills, possibly reflecting emerging occupations and skills (e.g. green skills). It would be possible to envisage a rolling survey with several different occupational areas (as prioritised by members) being included in each cycle. It would be important to repeat assessments in occupational areas every few years to enable assessment of change in skills levels. Additional countries could be recruited over time, including emerging economies with a particular interest in expanding or strengthening their VET systems.

References

[1] Jones, K. (1995), Simulations: a Handbook for Teachers and Trainers, Routledge, https://doi.org/10.4324/9780203761151.

[2] Nyström, S. and S. Ahn (2020), “Simulation-based training in VET through the lens of a sociomaterial perspective”, Nordic Journal of Vocational Education and Training, Vol. 10/1, https://doi.org/10.3384/njvet.2242-458x.201011.

[3] OECD (2021), “PIAAC Cycle 2 assessment framework: Adaptive problem solving”, in The Assessment Frameworks for Cycle 2 of the Programme for the International Assessment of Adult Competencies, OECD Publishing, Paris, https://doi.org/10.1787/3a14db8b-en.

Notes

← 1.  VET can be accomplished through many different programmes, and at varying points in someone's career. The VET programme types that are the focus of PISA-VET include upper secondary and post-secondary VET programmes that are either mostly school-based (i.e. with the training mostly being delivered in VET institutions) or have a large work-based learning component (i.e. with a substantial proportion of the training taking place on the premises of employers, like for example in apprenticeship programmes).  

← 2.  See the acronyms and abbreviations section of this document for the explanations of these bodies.

← 3.  Digital transition refers to the economic and societal effects of digitisation and digitalisation. Digitisation is the conversion of analogue data and processes into a machine-readable format. The green transition means a shift towards economically sustainable growth and an economy that is not based on fossil fuels and overconsumption of natural resources. A sustainable economy relies on low-carbon solutions that promote the circular economy and biodiversity.

← 4.  The Development Phase will consider whether live or recorded demonstrations of tasks are most suitable for PISA-VET. If live demonstrations are included, it is possible that there will be a heterogeneity of marking (despite instructions) which could impact on the assessment results.

← 5.  ASCOT stands for “Technology-based Assessment of Skills and Competences in VET”. The “+” denotes transfer into training and examination practice of the results which emerged from the predecessor initiative ASCOT (2011–2015).