This chapter defines employability skills as assessed by PISA-VET and elaborates the framework for employability skills that has been designed for the Development Phase of the project, including descriptions of the competencies and constructs to be assessed. It presents and explains the dimensions reflected in PISA-VET’s employability skills domains and how these will be assessed and provides several sample items with descriptions of task characteristics. The chapter also discusses how student performance in the employability skills is measured and reported against proficiency levels and scales.
PISA Vocational Education and Training (VET)
Assessment and Analytical Framework
PISA
Abstract
PISA-VET must reflect today’s workplaces in their complexity and dynamic nature. As mentioned in Chapter 1, the changing world of work in the 21st century is being profoundly shaped by several megatrends, including the Fourth Industrial Revolution, the advent of Artificial Intelligence, and the green and digital transformation. These trends are fundamentally alerting the nature of work and employment and shifting the global economy, society as well as consumer and employee demands (Frey and Osborne, 2017[1]; McClelland, 2020[2]; World Bank, 2018[3]). Therefore, labour markets are constantly evolving, requiring individuals to engage in lifelong learning and master new situations within their occupation or even across different occupations. The prerequisite for achieving this mastery, even in job market situations that we may not yet know will exist a decade from now, is through possessing so-called employability, i.e. a set of transversal and foundational skills that are fundamental for success in the workplace and for navigating a changing labour market. The significance of such employability skills is paramount for the workplace but extends even further: they are equally valuable for leading a successful and fulfilling social and civic life.
The topic of employability skills needed for the future of work has been of interest to policy makers and practitioners alike for several years, and the role of employability skills for students and adults in successful participation in modern society has been receiving increasing attention in several international surveys. PISA is one such study that has looked at the importance of employability skills in student populations. The most recent PISA report (OECD, 2019[4]) found that students who possess strong employability skills, such as problem solving, critical thinking, and communication, are more likely to perform well in all three subject areas and exhibit higher levels of well-being. PIAAC is another OECD study that has examined the importance of employability skills in the context of the working adult population. PIAAC assesses the proficiency of adults in key areas, such as literacy, numeracy, and problem solving in technology-rich environments. The most recent PIAAC report (OECD, 2019[5]) found that adults with stronger employability skills are more likely to have higher levels of employment, earn higher wages, and have better health outcomes. Furthermore, the OECD’s Study on Social and Emotional Skills (SSES) (OECD, 2021[6]) identified five key employability skills that help individuals to be successful in the workplace, namely task performance, emotional regulation, collaboration, open-mindedness, and engaging with others. Overall, these studies not only emphasise the relevance of employability skills for academic and career achievements but also highlight the individual benefits for personal growth and overall well-being associated with these skills in both student and adult populations.
PISA-VET includes employability skills derived from these OECD frameworks. These employability skills can be defined as a broad set of skills and abilities that are essential for success in any occupational field (a more detailed definition can be found in the next section). Without a strong foundation in these skills, individuals may struggle to acquire specialised skills or adapt to changing job requirements and will, thus, not be well-prepared to master the workplaces of the future. Therefore, developing employability skills through VET programmes and integrating them nationwide across occupational areas is critical for ensuring employability and success in the modern workforce as well as individual well-being. The assessment of employability skills in PISA-VET strives to provide valuable insights into their development within VET programmes, predict future employability, and facilitate comparisons both within specific occupations and across diverse contexts.
The purpose of this chapter is to define employability skills as understood by PISA-VET and highlight their relevance in the modern workforce. It also aims to elaborate the framework for employability skills that has been designed for the Development Phase of the project and explain the selected domains and the proficiency levels of employability skills in PISA-VET. Finally, a mode for the assessment of employability skills is proposed.
Employability skills are a set of competencies that are relevant across occupational areas for success in the workforce and beyond. Employability skills are characterised by the following hallmarks: a) they are necessary for an individual’s full integration and participation in the labour market, education, and training, as well as in social and civic life; b) they are highly transferable to other contexts; and c) they are learnable, and thus subject to the influence of policy (OECD, 2019[4]). In other words, these skills enable individuals to adapt to new technologies and rapidly changing workplaces (OECD, 2019[5]) and prepare individuals for future learning and help them master situations at work that may not even be relevant today but will be in the future. They are crucial for individuals to navigate the rapidly changing labour market and succeed in an increasingly complex and dynamic world of work. Numerous studies have demonstrated the importance of employability skills in education and the workforce. For instance, PISA and PIAAC have shown that individuals who possess strong employability skills are more likely to perform well academically, have better job prospects, and earn higher wages. Research studies have also found that employability skills are positively associated with job performance, job satisfaction, and career advancement (OECD, 2013[7]; 2019[4]; 2019[5]), which in turn contribute to individuals’ integration into society, resilience, personal growth, and well-being (Vanhercke, De Cuyper and De Witte, 2016[8]).
There are many different terminologies in use for what this chapter calls “employability” skills, and the term “employability skills” has been used synonymously to foundational skills, transversal skills, transferable skills, cross-domain skills, generic skills, core skills, key competencies, soft skills, and 21st-century skills. For example, PISA defines key competencies as the set of knowledge, skills, attitudes, and values needed for individuals to succeed in the 21st century. Similarly, PIAAC emphasises the importance of transferable skills for success in a constantly evolving labour market. Albeit in slightly different terminology, the OECD's PISA (OECD, 2019[4]) and PIAAC (OECD, 2013[7]) studies regard employability skills as a combination of cognitive and non-cognitive skills, including problem solving, critical thinking, communication, collaboration, literacy, numeracy, information and communication technology (ICT) literacy, and more. These skills are not limited to specific school subjects, occupations, or industries and can be applied across a variety of contexts. In their report “Enhancing Employability” for the G20 Employment Working Group, the OECD, the ILO (International Labour Organization), and the World Bank (OECD, ILO and The World Bank, 2016, p. 3[9]) define transferable skills as “skills that can be used in most occupations – e.g. problem solving, team working, etc. – including core skills such as literacy and numeracy, which are essential in all occupations and required for learning new skills.” Furthermore, these ‘skills that are relevant to labour market needs and transferable to different sectors and technologies’ (p. 7) are also referred to as adaptability and are supposed to enhance the employability of employees and job seekers. Like the report “Enhancing Employability”, the focus of PISA-VET is the improvement of individuals’ employability and their capacity to successfully seek and maintain employment in the future. The term “employability skills” is chosen to best reflect the context of PISA-VET, and essentially includes skills that are not occupation-specific but relevant across occupations and sectors (albeit at different levels/intensity).
For success in the workforce, employability skills are essential, both by themselves and in close interplay with occupation-specific skills. However, a key question is whether these employability skills should be regarded as domain-general (i.e. independent of occupation-specific knowledge and competencies) or domain-embedded (i.e. including occupation-specific knowledge and competencies). One main advantage of the understanding of employability skills as domain-general is that it not only enables comparability across different occupations but also allows for the measurement of a general capacity to navigate even in workplaces that do not yet exist. However, completing work tasks and solving complex problems in a particular domain always require domain-specific knowledge and skills. Therefore, employability skills are always, at least to some extent, intertwined with occupation-specific skills (Humburg and Velden, 2013[10]). PISA-VET considers both considerations and includes a generic assessment part of employability skills and a domain-embedded assessment part to capture the full breadth of both the generic and the domain-nature of employability skills as will be outlined in more specific terms throughout this chapter. While there are highly domain-specific approaches that are represented within the context of occupational skills, the primary emphasis here deliberately focuses on the overarching, non-occupation-specific aspects of employability skills and their integration within occupation-specific contexts.
The topic of employability skills is not only limited to whether they are domain-general or domain-embedded. An associated aspect under discussion is the time frame for defining these skills, which can be either short-term (aimed at being employable now) or long-term (aimed at becoming or staying employable). However, on the basis of responses from more than 900 employers in nine different European countries, Humburg et al. (2013[10]) conclude that “the skills that are needed to ensure short-term employability are no different from the skills that are needed to increase employability in the long run.”
The core employability skills selected for PISA-VET cut across occupational areas and are essential for all the selected occupations, while also enabling international comparability. As such, four core employability skills are identified for PISA-VET: literacy, problem solving, task performance, and collaboration. Other employability skills may also be important for certain occupations, such as communication skills in healthcare and tourism and hospitality, and numeracy skills for electricians, and these have been integrated within the frameworks for the occupational area assessments (see previous chapters). The four core employability skills were identified as the most relevant by policy makers and stakeholders from participating countries, based on extensive existing knowledge of their relevance from research and policy making (as discussed in Chapter 1). Moreover, test constraints were considered when selecting the employability skills to be included in PISA-VET, as only a small core of the most relevant employability skills could be assessed in the time available. The selected employability skills represent a diverse set of skills that tap into both cognitive and non-cognitive aspects. They are closely aligned with existing well-validated scales from PISA, PIAAC, and SSES. Detailed definitions of these skills will be presented in the section on assessment.
The four employability skills chosen for PISA-VET, literacy, problem solving, task performance, and collaboration, have been extensively studied in previous OECD assessments. PISA-VET therefore benefits from the in-depth conceptual work already completed on these assessments. To this end, the approach taken towards the employability skills included in PISA-VET is to use the existing OECD frameworks for organising the selected domains. This involves PIAAC Cycle 1 (and Cycle 2) for literacy, PIAAC Cycle 2 for problem solving, and the SSES for both task performance and collaboration.
PISA-VET will utilise these existing frameworks and adapt them to the VET context to provide the conceptual background of the four core employability skills. In the sections below information on each of the four core employability skills is presented, including their definition, relevance to VET, (sub-)processes involved in the employability skill, proficiency levels, relations to other relevant frameworks, and its occupation-specific modulation. Acknowledging that employability skills beyond literacy, problem solving, task performance, and collaboration can also be of paramount importance for specific occupations, a later section of this chapter will briefly discuss a set of further skills that are not included as direct assessments in PISA-VET. These further skills were discussed for potential inclusion in assessment but had to be excluded, mainly due to concerns related to testing time and potential low data validity from test taker fatigue.
This section aims at summarising the literacy framework, as defined by PIAAC and its relevance to the area of VET as a foundational employability skill. It is based to a large extent on the PIAAC Cycle 2 Literacy Framework (OECD, 2021[6]) that was developed by the members of the PIAAC Cycle 2 Literacy Expert Group. Literacy skills are fundamental for individuals to participate fully in society, access opportunities, pursue education and employment, communicate effectively, and continue learning and growing throughout their lives. Literacy serves as a gateway to knowledge, empowerment, and personal development, contributing to both individual success and societal progress.
PIAAC defines literacy as the ability to "access, understand, evaluate and reflect on written texts to achieve one's goals, to develop one's knowledge and potential and to participate in society" (OECD, 2021[6]). In its most literal sense, literacy refers to one's ability to comprehend and use written sign systems. Literacy encompasses a range of skills, from decoding written words and sentences to comprehending, interpreting, and evaluating complex texts. However, it does not involve the production of text (writing). Information on the skills of adults with low proficiency levels is provided by assessing reading components that cover text vocabulary, sentence comprehension, and passage fluency.
PIAAC views key competencies as "general" in the sense of being relevant to all members of the working population and across all fields of economic and social activity (OECD, 2021[6]). It further recognises that "while the economic and social importance of 'specific' competencies (skills related to specific rather than general-use technologies, discipline-specific or occupation-specific skills) is not denied, they are intentionally defined to be outside the scope of key competency frameworks" and thus not included in the PIAAC literacy framework (in PISA-VET, both aspects are considered for the assessment as outlined in a later section of this chapter).
Literacy provides the foundation for effective participation in society and literate individuals can make use of a broad diversity of written materials in the service of a wide range of activities and are knowledgeable of the cultural standards of their communities of practice. Thus, literacy is a necessary, but not sufficient, skill for performance in occupational areas where written communication dominates education and training and continuing education. As an employability skill in the PISA-VET framework, it allows individuals to progress towards applying literacy within their occupational areas of interest.
For instance, an automotive technician will need to read through technical manuals and understand the requirements and processes in the maintenance or repair of a car. An electrician will need to read and understand installation requirements, wiring diagrams, and operating instructions. Business and administration professionals will need to comprehend written information and graphical representations, such as evaluating delivery conditions based on international standards. Healthcare workers will habitually engage in reading scientific publications or medical procedures to stay updated on relevant topics and unfamiliar diseases. In tourism and hospitality, literacy is also a strong requirement, as employees will need to both converse with customers in oral and written form, and understand documentation, procedures or regulations that are applicable – for example, reading and understanding documentation relating to checking-out procedures and guest’s departure. This intimate relationship between literacy as an employability skill and its role in the various occupational areas is further illustrated below.
In addition to being directly relevant for their target occupations, literacy skills are also crucial to VET learners to adapt to the constant change that they experience at the workplace. As highlighted above, literacy skills are important prerequisites for engaging in further learning. Evidence shows that adults who hold a VET qualification are strongly exposed to automation in their jobs and are therefore likely to need to adapt and update their skills (OECD, 2020[11]). Without solid literacy skills they risk not being able to adjust to new working methods and technologies or not being flexible enough to change sectors or occupation – creating a risk that they leave the labour market or end up in low-quality jobs.
It needs to be acknowledged that literacy skills, as with the other employability skills, are not only developed through the VET programmes that the learner participates in. As mentioned in Chapter 1, employability skills are developed in a variety of settings - including settings outside of the education and training system - and are accumulated over the life course. In particular for literacy skills, early stages of education are important. The literacy skills of the learners targeted by PISA-VET are likely to reflect to a large extent the skills they developed even before entering their VET programme. As such, the assessment will not derive conclusions about how well VET does in equipping learners with literacy skills, but rather about how well VET together with the earlier stages of education is at ensuring that all learners have the literacy foundations needed in today’s labour market.
The literacy domain in PIAAC encompasses cognitive processes, content, and social context. These dimensions are also helpful in defining the proficiency levels for literacy.
Cognitive processes involve accessing text, understanding its meaning, and evaluating its quality. Accessing text includes identifying relevant texts and locating information within them. Understanding involves comprehending written words, integrating text information with prior knowledge, and handling multiple texts with inconsistent or conflicting information. Evaluating entails critically assessing the accuracy, soundness, and task relevance of the information, considering both the content and the source.
Content refers to the texts, tools, knowledge, and cognitive challenges that authors use to convey ideas. It includes different text types (description, narration, exposition, etc.), text formats (continuous or non-continuous), text organization (amount of information and density), and text sources (single or multiple texts).
Social contexts represent situations where reading activities are normally situated and may serve various purposes, from personal to professional and civic with professional contexts being the one that PISA-VET will be looking at. These refer to the different situations in which individuals must read: i) work and occupation, ii) personal use, and iii) social and civic context. For PISA-VET, the focus will be on situations in work and occupation.
Proficiency scales, also known as reporting scales, are important components of large-scale international assessments, enabling comparisons across countries and providing insights into student performance. In addition to defining the numerical range of the proficiency scale, it is also possible to define the scale by describing the competencies typical of students at points along the scale. Thus, the described proficiency scales describe what students typically know and can do at given levels of proficiency.
Table 7.1 shows the PIAAC literacy described scale from Cycle 1. Across the 32 participating Cycle 1 OECD countries, the average literacy score was 266.2 (Proficiency Level 2) with a standard deviation of 47 points. As the PIAAC literacy framework was updated for Cycle 2, this scale will be updated for reporting. Thus, the scale below is shown for reference only and PISA-VET will rely on Cycle 2.
The PIAAC Cycle 2 Literacy Framework (OECD, 2021[6]) represents an evolution of the role of literacy in the adult population (16–65 year-olds) that started with the International Adult Literacy Study (IALS; 1994-1998). The framework has undergone several iterations, including the Adult Literacy and Life Skills Survey (ALL; 2003-2008), the Survey of Adult Skills Cycle 1 (PIAAC; 2008-2018), and the ongoing implementation of the Survey of Adult Skills Cycle 2. The PIAAC Cycle 2 framework summarises and organises the evolution of the literacy construct in four areas (p. 24):
A reduction of the number of separate domains of literacy assessed: IALS (OECD/Statistics Canada, 2000[13]) measured literacy through separated scales of prose, document, and qualitative literacy. This aspect remained in ALL (Murray, Clermont and Binkley, 2005[14]) when quantitative literacy was replaced by numeracy. Quantitative literacy in IALS had a sole focus on functional, arithmetic calculation embedded in printed materials. In comparison, numeracy is a broader measure, covering a wider breadth of mathematical skills, purposes, and content. Furthermore, it depends less heavily on reading skills (Tout, 2020[15]). PIAAC Cycle 1 (OECD, 2012[12]) introduced literacy as a single, global construct, which remained in PIAAC Cycle 2 (OECD, 2021[6]).
An expansion of the range of text types covered in the assessment: PIAAC eliminated the differentiation between the reading of prose and document texts and expanded the range of text to include digital and electronic texts.
An increasing emphasis placed on evaluation and evaluating metacognition as cognitive strategies required for effective reading: IALS/ALL emphasised "matching" strategies (locating, integrating, and generating understanding) with PIAAC Cycle 1, expanding it towards evaluation and reflection and PIAAC Cycle 2 expanding evaluation further into the evaluation of the accuracy, soundness, and task relevance of a text concerning both its source and content.
The disentangling of the description and specification of cognitive strategies from questions of task difficulty: IALS/ALL frameworks combined cognitive strategies and the factors affecting task difficulty, while PIAAC views these independently. It conceived task difficulty as being driven by the stimulus text(s) features, the formulation of the question/task description and the interaction of the text and question/task description.
Additionally, there has been an increasing interest in providing more detailed information about adults with poor literacy skills. In response, PIAAC Cycle 1 introduced the assessment of reading components – the basic set of decoding skills essential for extracting meaning from written texts: knowledge of vocabulary (word recognition), the ability to process meaning at the level of the sentence, and fluency in reading passages of text – with PIAAC Cycle 2 assessing only the sentence meaning and passage fluency dimensions (OECD, 2021[6]).
The previous section mentioned that the PIAAC framework recognises 'specific' competencies but intentionally leaves them out of the key competency frameworks. This aligns with the proposed PISA-VET theoretical skills continuum (Figure 7.1). In this continuum, employability skills function as a prerequisite for individuals as they proceed from securing foundational level competency towards applying literacy in one of PISA-VET’s five occupational areas and then onwards to demonstrate occupation-specific skills in one of these occupational areas. A key issue for PISA-VET is the point along this continuum at which the skills being measured are no longer the application of employability skills in the occupational area but are rather their direct application as an occupation-specific skill. This aspect and how a generic assessment of literacy as done in PIAAC as well as an extension of this as literacy applied in occupational domains within PISA-VET is discussed further in a later section of this chapter.
Both as a generic employability skill and as a skill that is occupationally embedded, literacy is related to the five vocational domains covered in PISA-VET. The diagnostic skill of automotive technicians cannot be separated from their abilities to understand what the client wants and to consult the relevant technical repository, in order to understand how a specific technical ensemble is diagnosed, repaired, or maintained. An automotive technician needs to diagnose (which is impossible without understanding the target of the diagnosis, i.e. being able to read through the relevant technical documentation), repair (which is not possible without continuously consulting technical documentation, especially in non-routine cases), maintain or inspect (e.g. write off for service) mechanical parts or vehicles. All these processes and activities require literacy, especially related to technical literature.
Electricians may need to assess and diagnose technical equipment - for which they need to read and understand installation requirements and technical guidance documents, as well as wiring diagrams and operating instructions. They also need to assemble and install and to inspect, test and maintain electrical equipment - for which they need to read and understand data generated by testing equipment, and need to interpret associated reports, as well as read and understand inspection and commissioning procedures. In general, to perform these activities, electricians need to read and understand manufacturer’s instructions, wiring diagrams and layout drawings, to read and understand outputs of the various instruments they work with, and to read and understand (in order to follow them) industry procedures, regulations, or rules with respect to functionality and safety.
Employees working in business and administration need to use literacy in identifying needs for action (e.g. identifying the requirements for a new supplier by processing written information from internal and external business communication and retrieving information from databases), process quantitative and qualitative data (e.g. applying algorithms such as calculating bid prices and interpreting written information and graphical representations regarding delivery conditions based on the comparison with international standards), and communicate with internal and external stakeholders (e.g. understanding and evaluating written and oral communication from a customer).
Employees working in health care need literacy in activities and processes such as assessing needs and plan healthcare (e.g. documentation of past illness and related to planning intervention for a specific illness need to be consulted), supporting and enhancing clients’ quality of life (e.g. clients may need to be counselled and supported, and understanding their written case history, as well as their specific personal circumstances will require literacy), providing and supporting treatment and interventions (e.g. preparing for a case conference to assist the assessment of patients’ needs).
Finally, literacy will come into play in tourism and hospitality settings, including in activities such as checking in (e.g. reading a reservation document and understanding hotel’s policy and procedure for issue room keys to guests and providing directions to allocated rooms and information about hotel services and facilities) or checking out (e.g. reading and understanding documentation relating to checking-out procedures and guest’s departure) a customer. Table 7.4. presents examples of occupation-specific modulations of literacy to highlight the relevance of this employability skill within the five occupational areas under consideration in PISA-VET.
Literacy, as defined by the PIAAC framework, is a foundational or employability skill essential for participation in all aspects of society. More importantly, it serves as the basis for developing additional skills within occupational areas. In the context of PISA-VET, higher literacy levels are most likely linked with better use of these skills within occupational settings and more successful engagement in upskilling or reskilling.
This section aims at defining problem solving, based on existing assessment frameworks: PIAAC Cycles 1 (“problem solving in technology rich environments”; (OECD, 2009[16])) and 2 (“adaptive problem solving”; (OECD, 2021[6]); PISA 2003 (“mathematical problem solving”; (OECD, 2003[17])) and 2012 (“creative problem solving”; (OECD, 2013[18])). Most of this summary is driven by the latest conceptualisation of problem solving, i.e. “adaptive problem solving”, as defined by the PIAAC Cycle 2 (OECD, 2021[6]). This broad inclusion of problem solving across several large-scale assessments is driven by the notion that problem solving skills are essential in the personal, social, and professional lives of adults. They enable individuals to approach complex situations with confidence, creativity, and resilience, allowing them to overcome obstacles and achieve desired outcomes in both personal and professional domains. This section explains the relevance of the concept to the area of VET as an employability skill.
PIAAC Cycle 2 defines problem solving as “the capacity to achieve one’s goals in a dynamic situation, in which a method for solution is not immediately available. It requires engaging in cognitive and metacognitive processes to define the problem, search for information, and apply a solution in a variety of information environments and contexts” (OECD, 2021[6]). In its most basic sense, problem solving is centered around contexts that demand non-routine solutions, and some difficulties for the one facing them, irrespective of any content-specific domain and across occupations.
The latest PIAAC Cycle 2 approach emphasises the following components of this definition:
Capacity: Problem solving is a complex proficiency, and its application to a problem is a goal-directed activity that may result in various degrees of success or performance in handling the problem.
Adaptivity: Problem solving takes place in complex environments that force the process to be a dynamic and not a static sequence of pre-set steps, thus forcing problem solvers to remain open, to monitor the problem environment, and to adapt their approach constantly.
Cognition and metacognition: Problem solving involves cognitive performance to organize and integrate information into a mental model, as well as metacognition for self-monitoring and self-reflection on progress. These two components are interconnected and challenging to separate.
Process: Problem solving is a three-stage process containing problem identification (i.e. “problem finding”), actions to bridge the gap between the initial point to the desired goal (i.e. “problem shaping”), and the actual performance of actions until a satisfactory outcome is achieved. These stages can occur in different orders or even simultaneously.
Environment: Problem solving is typically embedded in information-rich physical, social, and digital environments, emphasizing the particular importance of digital literacy. Digital environments especially are critical in this respect. PIAAC Cycle 1 has already recognized problem solving in technology-rich environments, highlighting the integration of digital skills in today’s problem solving scenarios. Consequently, competent problem solvers are expected to be able to deal with problem solving situations that are digitally embedded.
Problem solving skills are essential in the personal, social, and professional lives of adults. Problem solving has been linked with positive individual-level and societal level outcomes, as it describes at its core “the ability to quickly and flexibly adapt to new circumstances, learn throughout life, and turn knowledge into action” (OECD, 2021[6]). Problem solving skills have been found in modern economies to be most important for a worker to be successful (OECD, 2015[19]).
PIAAC views problem solving as a general key competency, i.e. it emphasises the empirically underscored assumption that this competency is relevant to all members of the working population and across all fields of economic and social activity (OECD, 2021[6]). Problem solving is thus ostensibly defined in contrast (and opposition) with specific competencies, i.e. skills that are discipline-specific or occupation-specific. The importance of such specific competencies is not denied, but on the contrary emphasised; however, they were not included in the PIAAC framework: they are intentionally defined to be outside the scope of key competency frameworks (OECD, 2021[6]). This fundamental notion of how problem solving is defined is rooted in its pivotal role in current and future employability across diverse contexts. In PISA-VET, both generic and occupationally embedded aspects of problem solving are considered for the assessment as outlined in a later section of this chapter. In this, problem solving is considered to be foundational and essential for effective participation in society and the labour market but is not sufficient for performance in specific professional settings. In such specific professional settings, aside from more specialised skills, problem solving itself may be challenged by the context: Specialized information may mark the problem environment, and specialised reasoning may mark the stages of the problem solving process.
Some examples of how problem solving is inherently bound into the focal occupational areas follow here. An automotive technician may need to identify the causes of automotive malfunction and derive appropriate repair actions, such as removing a rusted bolt with a snapped-off head. An electrician will need to identify and diagnose electrical breakdowns and make informed recommendations to customers or clients. For employees in business and administration roles, typical office work will encompass significant non-routine problems, exceptions, errors, and innovative tasks that include problem solving, especially now that many routine work processes have been automated or outsourced. In healthcare, the essence of intervention is that the problem needs to be understood beforehand – diagnosing the situation and deciding on the appropriate course of action may often be literally the difference between life and death. In tourism and hospitality, adhering to established procedures while adapting services to specific situations and customer needs defines excellency. This relationship between problem solving as an employability skill and success in occupational areas is further illustrated in the section on occupation-specific modulation.
In fact, in a labour market that is exposed to digitalisation and automation, including in the five occupational areas under consideration in PISA-VET, problem solving skills become increasingly important. Complex problem solving is identified as one of the skills that are least exposed to automation (Lassébie and Quintini, 2022[20]) that is, with a low probability that technology will be able to replace humans in carrying out tasks that require extensive problem solving skills. As such, problem solving skills are key for VET learners who are or will be working in an increasingly digital labour market.
The problem solving domain in PIAAC is organised along a set of task dimensions (problem configuration, dynamics of the situation, features of the environment, information environment, and problem contexts) and two large groups of processes: cognitive and meta-cognitive. This section focuses on these two overarching processes and describes the specific processes in each domain.
Cognitive processes are those that individuals must bring into play in order to construct a mental model of the state of affairs described in the problem and then to apply that mental model in real life. These cognitive processes are applied to the three stages of the problem solving process:
1. Defining (Problem definition: Mental model construction)
Selecting, organising, and integrating information into the mental model: Constructing a mental representation of the problem space (initial state, goal state, operators).
Retrieving relevant background information: Accessing memory to retrieve background knowledge (note: assessment tasks should be designed to avoid necessity of this process).
Externalising internal problem representation: Creating an external representation (e.g. drawing, table) that illustrates the problem solver’s mental model of the problem.
2. Searching (Search solution: Identifying effective operators)
Searching for operators in the mind and environment: Locating information about available action options that might be suited to solve the problem.
Evaluating operators based on problem constraints: Determining which of the action options will be best to reach the goal while considering all possible constraints.
3. Application (Apply solution: Applying plans and executing operators)
Applying plans and executing operators: Implementing the selected operator(s) to solve the problem.
Metacognitive processes are those that become more important to the extent that problems are more complex and difficult to comprehend, that the problems change, and that progress towards the solution becomes more difficult. The metacognitive processes are also applied to the three stages of the problem solving process:
1. Defining (Problem definition: Mental model construction)
Goal setting: Deciding upon what the to-be-achieved state is about (cannot be considered in large-scale assessments because allowing problem solvers to set their own goals would yield too many degrees of freedom).
Monitoring problem comprehension: Supervising whether one’s mental model of the problem matches the current state of affairs.
2. Searching (Search solution: Identifying effective operators)
Evaluating operators based on executability: Determining which of the action options will be best to reach the goal while considering all possible constraints.
3. Application (Apply solution: Applying plans and executing operators)
Monitoring progress: Determining whether executing operators achieves the desired outcome.
Regulating application of operators: Modifying the selection of operators in case the problem configuration has changed (cf. monitoring problem comprehension) or impasses have been noted (cf. monitoring progress).
Reflection: Deliberating about one’s own capabilities to solve problems with the goal of abstracting knowledge from it that can be applied in the future (cannot be considered in a large-scale assessment context because it requires repeated confrontation with similar problem solving instances).
Table 7.2 shows the general descriptions in the PIAAC adaptive problem solving scale from Cycle 2. These are based specifically on behaviourally descriptions that were offered in the PIAAC Cycle 2 framework for low and for high scorers on the three task dimensions, that is, when confronted with different (a) problem configurations, (b) dynamics in a situation, and (c) features of the environment. Table 7.2 illustrates the nexus of task dimensions, cognitive, and metacognitive processes, and proficiency levels.
Note: APS is Adaptive Problem Solving.
Source: PIAAC Cycle 2 assessment framework; (OECD, 2021[21]): 167.
Different frameworks have given slightly different definitions to problem solving, emphasising one or another component. The first paradigm was analytical problem solving in an earlier version of PISA (OECD, 2003[17]). PISA 2012 (OECD, 2013[18]) linked problem solving to creativity, emphasising creative and critical thinking abilities. PIAAC Cycle 1 (OECD, 2009[16]) focused on problem solving in technology-rich environments, considering the complexity introduced by technology. The fourth and current approach, championed by PIAAC Cycle 2 (OECD, 2021[6]), is central to understanding problem solving in PISA-VET. It introduces innovations such as the adaptive component (the problem changes during the process and an adaptive reaction to this is needed) and includes both metacognitive and cognitive skills in its definition. PIAAC Cycle 2 includes problem solving items in various activity contexts, with the "work" context being particularly relevant for the current endeavour. When selecting items from existing assessments, the focus will be on sampling PIAAC items from the work context owing to this compatibility.
The previous section mentioned that the PIAAC frameworks of both Cycle 1 and Cycle 2 recognise the importance of distinct competencies (some of which are associated with specific professional contexts) but intentionally leaves them out of the key competency framework. This approach is aligned with the proposed PISA-VET continuum of employability skills, in which these skills are considered transversal skills that function as prerequisites for individuals as they proceed from foundational level competency towards more sophisticated and profession-specific competency that are manifested and demonstrated as occupational-specific skills in one of the occupational areas of PISA-VET. Research regarding the five occupational domains covered by this project (see previous chapters) is overwhelming in terms of acknowledging the importance of problem solving as a prerequisite of job performance and of staying competitive on the job market throughout life.
For automotive technicians, their diagnostic skills have been closely linked to their problems solving skills and are a central point in the selection, training, and appraisal of mechanics (Abele, 2018). An automotive technician needs to diagnose (e.g. a faulty engine performance), repair (e.g. outdated equipment with no spare parts), or inspect (e.g. write off for service) mechanical parts or vehicles. Likewise, electricians, like all STEM-related professions, are deeply in need of problem solving skills, often for even routine and daily operations (Neubert et al., 2016[22]). They need to assess and diagnose (e.g. the breakdown of a large piece of electrical equipment in a factory), they need to assemble and install (e.g. a 'new to the market' piece of electrical equipment) and to inspect, test and report (e.g. a newly installed new piece of electrical equipment that shows a malfunction) technical equipment.
In business and administration, problem solving has been central both as managerial competency (van Aken and Berends, 2018[23]) and implicitly as well as explicitly as it is sometimes associated with Ackoff’s formulation of the central concept of “problem mess” (1981[24]), becoming part of the “problem solving cycle” as an approach to management. Similarly, Jonassen (2000[25]) points out that problems in real life are usually “meta problems” which consist of a bundle of nested problems of various kinds. An employee in such a role may need to identify needs for action (e.g. identify information gaps), process quantitative and qualitative data (e.g. apply the calculation scheme of manufacturing costs), or communicate effectively with internal or external stakeholders in loaded situations (e.g. recognise potential or actual conflicts and work towards their resolution; (Rausch and Wuttke, 2016[26])).
In health care, the importance of problem solving has been acknowledged very early on, and continues to be considered crucial for specialist performance and inclusion in modern approaches to the design of healthcare systems and physician training (Lorusso, Lee and Worden, 2021[27]). Employees in the health care industry may need to assess the needs of clients (e.g. analyse the physical and mental changes of a patient), plan a healthcare process (e.g. propose new coping strategies for a client in a therapeutic context), or monitor treatment and interventions (e.g. monitor the vital signs of a person who has undergone a difficult operation and who is still in a critical phase).
In tourism and hospitality, problem solving is considered a critical core skill by both experienced managers and students in hospitality (Dimmock, Breen and Walo, 2003[28]). Employees working in the hospitality industry may need to understand the personal needs of customers (e.g. converse with customers that are not fluent in your language and understand specific dietary requirements that they have), or manage complaints (e.g. understand and solve effectively complaints about the lodging arrangements of a customer, that have not been communicated in advance and for which the hotel is not prepared). To demonstrate the relevance of problem solving in a specific occupation, Table 7.4. displays examples of occupation-specific modulations of this employability skill across the occupational areas.
Problem solving, as defined in the PIAAC Cycle 2 framework, is an employability skill that is foundational for participation in all aspects of work. While an essentially general skill, it may have job-specific components that are saturated in knowledge, reasoning, and decision-making typical for specific job domains or environments as acknowledged in an earlier section of this chapter. Thus, it may both contain and, more importantly, serve as the basis for developing additional occupation-specific skills within occupational areas. In the context of PISA-VET, higher levels on problem solving may be linked to better proficiency in the use of problem solving skills in specific professional settings, and in easier development of other skills.
This section aims at summarising the task performance framework, as defined by the OECD's SSES and its relevance to the area of VET as a sub-set of employability skills. Developing and nurturing task performance skills supports individuals in leading fulfilling lives, forming meaningful connections, and thriving in various personal and professional contexts. This section on task performance was primarily extracted from the SSES Assessment Framework and Conceptual Framework (Kankaraš and Suarez-Alvarez, 2019[29]; Roberts et al., 2009[30]), which were developed by members of the SSES team. It derives from the international report on SSES Round 1 results (OECD, 2021[21]).
In SSES, task performance was derived from the Big Five model of personality. Task performance corresponds to the dimension of conscientiousness. It is defined as a set of social and emotional skills, which are “individual capacities that can be manifested in consistent patterns of thoughts, feelings, and behaviours” (OECD, 2021[6]). These skills enable individuals to effectively manage and develop their emotions, thoughts, tasks, and relationships. Task performance refers to a range of constructs that describe the ability to be self-controlled, responsible to others, hardworking, motivated to achieve, honest, orderly, persistent and rule-abiding (Kankaraš and Suarez-Alvarez, 2019[29]). In short, it refers to the skills that enable individuals to get things done, as required and on time.
The skills that are part of the SSES, including task performance and its dimensions, were chosen for their relevance to individuals across all aspects of life, whether professional, personal, or social. Empirical evidence supports the hypothesis that task performance is highly relevant across different work-related criteria such as job performance and trainings proficiency and across different occupations (Barrick and Mount, 1991[31]). Dudley et al. (2006[32]) found that responsibility, understood as one’s ability to honour commitments and be reliable, was significantly linked to several job performance criteria. It was positively linked to task performance (contributing to production of good or service) and organisational citizenship (pro-social workplace behaviour) and inversely linked to counter-productivity.
Consequently, task performance is relevant to the future of work as well as its demands today because it encompasses personal characteristics like being planful, careful, and hard working. Technology is replacing routine jobs, but, at the same time, is creating new opportunities that require individuals who can manage complex social interactions and non-routine tasks (OECD, 2019[5]). In fact, the capability to focus one’s attention on the task at hand is crucial in a lot of occupations and work settings, including scenarios like remote work, which has been accelerated adoption during and post the COVID-19 pandemic. For example, as an automotive technician, conscientiously working through a service checklist, despite high time pressure, noise, and other distractions is important to ensure a car’s repair and maintenance. Similarly, management assistants need to concentrate on the current task avoiding distractions from other activities when dealing with information-rich problems (Rausch and Wuttke, 2016[26]). For a peripatetic healthcare assistant, who provides support and assistance to patients in various healthcare settings, it is important to carefully plan the next week’s route for visits taking on responsibility towards the clients’ needs, but also on medical needs, logistical issues, and own role-associated boundaries. Well-rounded VET learners should therefore have solid task performance skills to be effective in their target occupation – in line with employers’ expectations. Research also suggests that these skills are malleable and susceptible to interventions (Durlak et al., 2011[33]; Sklad et al., 2012[34]; Taylor et al., 2017[35]), and can therefore be developed through education and training – including VET. Social and emotional skills are also more malleable at later stages in life than cognitive skills, which is especially relevant to VET programmes, as they typically cater to older populations, as opposed to general education (Cunha, Heckman and Schennach, 2010[36]; Cunha and Heckman, 2007[37]).
As aforementioned, the SSES domain of task performance derives from the domain of conscientiousness from the Big Five model of personality and skills. SSES adapted the general Big Five framework to create a framework of skills, which are individual capacities that are learnable and malleable and thus, susceptible to program and policy interventions (OECD, 2021[21]). In SSES, task performance is divided into four dimensions, which compose more contextualised manifestations of broad domains. These narrower skills are more descriptive, specific, and accurate, and thus easier to measure:
Self-control: The ability to resist distractions, delay gratification and maintain concentration. Someone with high self-control postpones fun activities until important tasks are completed and thinks before they act, such as finishing notes before chatting to colleagues. Someone with low self-control is prone to say things before thinking and engages in impulsive behaviour, such as an employee who blurts out opinions in a meeting.
Responsibility: The ability to honour commitments and be punctual and reliable. Someone with high responsibility arrives on time for appointments and gets tasks done promptly, such as ensuring all emails are addressed before going home. Someone with low levels of responsibility, for example, does not follow through on agreements.
Persistence: The ability to persevere in tasks and activities until completion. Someone who is highly persistent finishes projects or work once started. Someone who has low persistence gives up easily when confronted with obstacles. This dimension is often assessed differently by age. In adult inventories of the Big Five, being persistent in the face of challenges is sometimes included under achievement motivation.
Achievement motivation: The drive to set high standards for oneself and work hard to meet them. Someone with high achievement motivation enjoys reaching a high level of mastery in some activity, is highly productive and aspires to excellence, such as an entrepreneur who works long hours and perseveres against setbacks to make their business profitable. Someone with low motivation lacks interest in reaching mastery in any activity, including professional competencies.
Proficiency levels were not developed for SSES, although reporting scales were. For Round 1 of SSES (the only completed round to date), psychometric scales were developed using the assessment items for each skill. The reference was value fixed at 500 representing a centre of the scale characterized by individuals with either only mid-point or perfectly balanced responses (i.e. not leaning on one direction or another towards the poles defined below) and standard deviation set to 100.
The scale for each dimension (i.e. self-control, responsibility, persistence, and achievement motivation) features two meaningful poles. Individuals placed at one end of the scale have more of the attributes and qualities that define the pole to which they are closest and fewer of the attributes defining the pole from which they are farthest. Taking self-control out of the four dimensions of task performance, for example, respondents towards the highly self-controlled pole, reported themselves as more inclined to be careful with tasks, think before speaking, and postpone fun until they are finished with work. Respondents at the opposite end of the spectrum (uninhibited, less self-control) more often reported acting impulsively or speaking without thinking (SSES uses exclusively self-reports to assess skills).
Utilising proficiency levels is particularly beneficial when applied to performance-based measures since they enhance clarity and consistency by offering a clear and standardized framework for assessment and interpretation. However, there are several reasons for avoiding proficiency levels in social and emotional skills such as task performance. In fact, “proficiency levels” for social and emotional skills create problematic, universally normative measures for context-embedded and culturally relative capacities by implying an order, or hierarchy, to assessed behavioural patterns. Additionally, skills that may be useful in one context (e.g. high levels of self-control in fixing a delicate mechanical device or administering medication), may be less useful in others (e.g. coming up with new business ventures or taking creative risks may benefit from less restraint). This aspect of context-embedding is elaborated in the section below on the occupation-specific embedding of task performance. Measures of task performance provide countries and stakeholders with important insights into how test takers manifest task performance dimensions and how these link to contextual factors and outcomes. However, assigning universal proficiency levels may obscure the complexity and context-specific value of these skills.
The SSES task-performance dimensions were derived from a range of existing child and adult taxonomies of the Big Five domains and facets. Task performance in general corresponds to the domain of conscientiousness. However, there are many different conceptualisations of its narrower dimensions. For SSES, taxonomies that were consistently identified and cross-culturally replicated were considered for possible inclusion. Seven dimension-level taxonomies for children and adolescents were used, along dimension scales from several well-known adult personality inventories. These seven taxonomies were examined for conceptual overlaps, similarities and differences and then served as a starting point of the development of a common framework. In addition to this, PISA 2021 Context Questionnaire Framework (OECD, 2019[38]) can be utilised as an additional point of reference for item adaptation. Within the PISA framework, task-performance items related to SSES were adapted, featuring extended item numbers across several task performance dimensions. These adapted items were seamlessly integrated into the background questionnaire and demonstrated strong psychometric properties.
In addition to these, task performance relates to other OECD frameworks on non-academic skills, such as PISA’s innovative domains and Education 2030’s Concept Notes. Outside the OECD, task performance overlaps with domains of many skills frameworks, including for social and emotional learning and employability. For example, achievement motivation and persistence both relate to Duckworth’s (Duckworth, 2007) much-used construct of Grit, describing sustained interest and perseverance toward long-term goals. The ExploreSEL platform created by the Ecological Approaches to Social and Emotional Learning (EASEL) Lab at Harvard identifies six common domains across 43 different frameworks for non-academic skills, including SSES. Task performance overlaps with all 42 other frameworks in certain dimensions of Cognitive, Values, Social and Identity (EASEL Lab., 2021[39]).
Task performance is a central skill to successfully perform at the workplace. All four facets of self-control, responsibility, persistence, and achievement motivation play an important role in different occupational settings. Concentrating on complex tasks, e.g. performing calculations while being frequently interrupted by phone calls or colleagues’ questions is challenging. Staying focused and handling those distractions reflects a strong degree of self-control.
In a working situation, for instance, when a colleague has not yet successfully solved a specific problem or an error is detected in an order, taking responsibility means not to ignore the problem or trying to delegate it to someone else. In contrast, a responsible person stays calm and remains focused on the relevant problem searching for additional information to solve it. Similarly, continuing with a task although unexpected challenges like a delivery of incorrect materials has occurred, reflects a high level of persistence. To demonstrate the relevance of task performance and the other three core employability skills in a specific occupation, Table 7.4. displays examples of occupation-specific modulations of the four employability skills.
Task performance is a social and emotional skill with high relevance for the workplace across occupations consisting of the subdimensions self-control, persistence, responsibility, and achievement motivation. It has been included in previous OECD-led international assessments and describes an important employability skill in PISA-VET.
This section summarises the Collaboration framework, as defined by the OECD's SSES and its relevance to the area of VET as a sub-set of employability skills. Collaboration skills facilitate effective teamwork, innovation, problem solving, and communication. They contribute to personal growth, enhance relationships, and improve outcomes in various contexts, ultimately leading to individual and collective success. This section was extracted from the SSES Assessment Framework and Conceptual Framework (Kankaraš and Suarez-Alvarez, 2019[29]), which were developed by members of the SSES team. It derives from the international report on SSES Round 1 results (OECD, 2021[21]).
In SSES, collaboration was derived from the Big Five model of personality. Collaboration corresponds to the dimension of agreeableness. It is defined as a set of social and emotional skills, which are “individual capacities that can be manifested in consistent patterns of thoughts, feelings, and behaviours” (OECD, 2021[21]). These skills enable individuals to effectively manage and develop their emotions, thoughts, tasks and relationships (as already outlined in the previous section). Collaboration refers to a range of constructs that describe the ability to understand, feel, and express concern for others well-being, manage interpersonal conflict, and maintain positive relationships and beliefs about others (trust) (Kankaraš and Suarez-Alvarez, 2019[29]; Soto and John, 2017[40]). In short, it refers to the skills that enable individuals to get along with other people and work successfully together in various contexts and situational settings.
In SSES, collaboration was chosen for its relevance to school children and individuals across all aspects of life, whether professional, personal, or social. Collaboration is also particularly relevant for VET due to its links to job performance outcomes, especially how individuals relate to colleagues in the workplace. Sackett and Walmsley’s meta-analysis found that agreeableness (the corollary to collaboration) predicted levels of organisational citizenship—pro-social workplace behaviours like supporting co-workers—and inversely predicted counter-productivity, such as absenteeism (Sackett and Walmsley, 2014[41]). Research also suggests that social and emotional skills such as collaboration are malleable and susceptible to interventions (Durlak et al., 2011[33]; Sklad et al., 2012[34]; Taylor et al., 2017[35]). Importantly, these skills are also more malleable at later stages in life than cognitive skills, which is especially relevant to VET programmes, as they typically cater to older populations, as opposed to general education (Cunha, Heckman and Schennach, 2010[36]; Cunha and Heckman, 2007[37]).
Developing the ability to collaborate is essential for VET learners, as many VET jobs – and the labour market more broadly – involve tasks that require workers to work and interact with others. Many typical VET jobs involve collaborating with workers in other roles and at different levels in a workplace hierarchy, as well as with clients/users. For example, healthcare/nursing assistants typically work closely with other healthcare professionals, while also interacting closely with patients to anticipate and understand (empathising) their needs. Electricians on construction sites will need to discuss with the client to understand wishes and constraints and need to work in tandem with architects and other construction workers. As such, collaboration capabilities are key, and VET learners can develop those in school settings as well as during work placements. Collaboration is relevant in many ways to today’s changing economies and societies. Diversifying societies, automation and the decline of traditional social networks are creating new pressures but also opportunities for those who can manage complex social interactions and non-routine tasks (OECD, 2019[5]). Strong collaboration and interpersonal skills, such as empathising and managing conflict, are increasingly key capacities for the future of work.
The SSES domain of collaboration derives from the domain of agreeableness from the Big Five model of personality and skills (Chernyshenko, Kankaraš and Drasgow, 2018[42]). SSES adapted this framework to create its skills framework, focusing on capacities that are learnable and malleable and thus, susceptible to program and policy interventions (OECD, 2021[21]). In SSES, collaboration is divided into three dimensions:
Empathy: The ability to understand and care about others and their well-being. In VET contexts, someone with high empathy might be a nurse who knows how to console an upset patient or a manager who sympathizes with the needs of her employees. Someone with low empathy tends to misinterpret or disregard others’ feelings, such as a manager who ignores his employees’ wellbeing at work.
Trust: The ability to assume that others generally have good intentions and forgive those who have done wrong. Someone with high trust tends to lend things to others and avoids being judgmental or harsh, such as trusting the good intentions of an inexperienced colleague who makes mistakes. Someone with low levels of trust is secretive and suspicious in relation to other people, such as a manager who does not delegate tasks because they do not think their colleagues can handle them.
Co-operation: The ability to live in harmony with others, compromise, and value group cohesion. Someone with high levels of co-operation finds it easy to get along with people and respects group decisions, such as an engineer who can compromise on their plans when their teammates propose an alternative vision. Someone with low levels is prone to arguments or conflicts and does not tend to compromise, such as a mechanic who finds fault with suggestions from co-workers and gets into arguments over projects.
As mentioned above in the section on task performance, proficiency levels were not developed for SSES, although reporting scales were developed. For Round 1 of SSES (the only completed round to date), psychometric scales were developed using the assessment items for each skill. The reference value was fixed at 500 representing a centre of the scale characterised by individuals with either only mid-point or perfectly balanced responses (i.e. not leaning on one direction or another towards the poles defined below) and standard deviation set to 100.
The scale for each process of collaboration (i.e. empathy, trust, and co-operation) included in SSES features two meaningful poles. Individuals placed at one end of the scale have more of the attributes and qualities that define the pole to which they are closest and fewer of the attributes defining the pole from which they are farthest. Taking “empathy”, for example, respondents towards the highly empathetic pole, reported themselves as more inclined to consider others’ wellbeing and their perspectives. Respondents at the opposite end of the spectrum (less empathetic) more often reported disregarding others’ feelings when making decisions.
As discussed in the earlier section on task performance, there are several reasons for avoiding proficiency levels in social and emotional skills such as collaboration. “Proficiency levels” for social and emotional skills create potentially problematic measures for context-embedded and culturally relative capacities. Skills that may be useful in one context (e.g. high levels of co-operation in healthcare where one often coordinates between healthcare assistants, nursing professionals, and medical doctors) may be less useful in others (e.g. a manager who must make tough but necessary staffing decisions despite short-term employee disapproval). This aspect of context-embedding is elaborated on in the occupation-specific embedding of collaboration.
The SSES collaboration dimensions were derived from existing taxonomies of the Big Five domains and facets. Collaboration in general corresponds to the domain of agreeableness. However, there are many different conceptualisations of its narrower dimensions. For SSES, taxonomies that were consistently identified and cross-culturally replicated were considered for possible inclusion.
Collaboration relates to other OECD frameworks on non-academic skills, such as the PISA’s collaborative problem solving assessment in 2015 (OECD, 2017[43]; 2017[44]), the PIAAC Cycle 2 and the PISA 2022 and 2025 questionnaires. In the PIAAC Cycle 2, the assessment of Social and Emotional Skills is based on an extra short 15-item version of the Big Five Inventory BFI-2-XS (Soto and John, 2017[40]), an instrument based on self-reports composed of three items per domain. Countries had the option of administering a 30-item version (BFI-2-S) with six items per domain. The assessment distinguishes five broad skill domains: conscientiousness (i.e. task performance), emotional stability, extraversion, agreeableness (i.e. collaboration), and openness to experience. The inclusion of this module helps analysts understand the interaction between cognitive and social-emotional skills.
Also related to collaboration is the concept of collaborative problem solving as defined in PISA 2015: “the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution, and pooling their knowledge, skills and efforts to reach that solution” ( (OECD, 2017[43]): 47). The framework encompasses four problem solving processes and three collaboration processes, ensuring comprehensive coverage of the underlying theoretical concept (OECD, 2017[44]; 2017[43]). The PISA 2022 and 2025 questionnaires expanded the assessment to include additional aspects such as co-operation, perseverance, self-control, curiosity, empathy, trust, perspective taking, assertiveness, stress resistance, and emotional control, using self-reported instruments. More specifically, within the PISA 2021 Context Questionnaire Framework (OECD, 2019[38]), collaboration items related to SSES were adapted, featuring extended item numbers across several dimensions of collaboration. These adapted items were seamlessly integrated into the background questionnaire and demonstrated strong psychometric properties. Thus, this PISA-VET framework can serve as an additional point of reference for item adaptation.
Outside OECD-conveyed studies, collaboration overlaps with domains of many skills frameworks, including for social and emotional learning and employability. The ExploreSEL platform created by the Ecological Approaches to Social and Emotional Learning (EASEL) Lab at Harvard identifies six common domains across 43 different frameworks for non-academic skills, including SSES (EASEL Lab, 2021). Collaboration overlaps with 41 other frameworks in the domains of Social, Emotion and Values.
Collaboration and its sub-domains are relevant to almost all occupational areas in VET, at multiple levels. Collaboration in VET contexts involves working with others within one’s own team, within an organisation, or with clients/users. In many situations, all three aspects are used simultaneously. For example, in tourism and hospitality, a hotel receptionist is at the core of communication between different operations of the hotel. They must not only cooperate with various departments but doing so well involves understanding their role, procedures, and limitations to optimise workflow and trusting their abilities. In business and administration, a member of a sales department must be able to cooperate with internal teams, trust them to manage their responsibilities, but also empathise with customers to anticipate needs and develop opportunities. Further examples of the specific modulation of collaboration across occupational areas is found in Table 7.4.
Collaboration refers to a range of constructs that describe the ability to understand, feel and express concern for others, manage interpersonal conflict and maintain positive relationships and trust. It is essential for VET learners, as many VET jobs and the labour market generally involve tasks that require workers to interact with others and work together to achieve shared goals. Derived from the OECD’s SSES, collaboration has three dimensions: empathy, trust, and co-operation. Collaboration is based on the domain of agreeableness in the Big Five model of personality. It directly relates to the PIAAC assessment of Social and Emotional Skills and overlaps with PISA’s domain of “Collaborative problem solving.”
Upon entering the labour market, individuals graduating from a VET programme need a set of skills to find and retain a suitable job. In this endeavour, the four employability skills outlined in previous sections of this chapter, cover a broad array of these skills, but they do not account for the full set of employability skills. Further cognitive and social and emotional skills such as numeracy, ICT literacy, and financial literacy skills, along with social and emotional proficiencies like emotional regulation, self-efficacy, and communication skills, contribute to a comprehensive skill set necessary for a successful transition from formal education to the workplace.
Table 7.3 presents definitions and references to existing frameworks of some further relevant employability skills such as numeracy. Numeracy is traditionally regarded as a key cognitive skill in addition to literacy for succeeding in the labour market. Furthermore, ICT literacy plays an increasingly important role in professional and private life. Recently, financial literacy skills have emerged as a new construct, reflecting the demands of global trends in modern society. Emotional regulation, self-efficacy, and communication skills are social-emotional proficiencies complementing cognitive skills. Emotional regulation and self-efficacy are both important when dealing with uncertainty and challenging situations.
Given time constraints and potential test taker fatigue, it is not feasible to include all relevant employability skills in PISA-VET, and the selected employability skills will provide unique and comprehensive insights into key employability skills in the VET population. In fact, adding additional employability skills to PISA-VET might yield limited new information. Taking numeracy as an illustrative example, findings from the PIAAC suggest a strong correlation between literacy and numeracy skills, potentially limiting the incremental value gained from assessing both skills concurrently (Hanushek et al., 2015). Therefore, it might suffice to incorporate numeracy assessments within occupational domains (e.g. for electricians) where its significance can be directly acknowledged within the assessment of occupation-specific skills. Considering all these factors, the set of four employability skills is likely to be the upper limit for inclusion in PISA-VET. Although the skills presented in Table 7.3 are unlikely to be included as separate scales in the Development Phase of PISA-VET, they do exhibit relevance from a conceptual perspective and are listed here also under the consideration of their relevance within selected occupations.
Note: F = Frequency: how often this ES is encountered; C = criticality: how essential this ES is; +++ = high relevance, ++ = middle relevance, + = low relevance.
This section outlines the approach taken to apply the elements of the frameworks described in previous sub-sections to PISA-VET. Utilising the existing frameworks discussed earlier in this chapter, the first sub-section of this section elaborates on the role of generic and occupationally embedded assessments within PISA-VET and how they might be combined. Later sub-sections outline the specific strategies for documenting and measuring students’ competencies in each of the four domains selected for the core employability skills.
The assessment of employability skills within PISA-VET should serve the purpose of predicting future employability and facilitating within-occupation and cross-context comparisons, including comparisons across occupations and different international VET systems. In addition, it should allow for comparisons with existing international educational large-scale studies such as PISA and PIAAC. In doing so, a two-pillar framework for the assessment of employability skills, as illustrated in Figure 7.3, is proposed to capture the full breadth of both the generic and the domain-nature of employability skills. More specifically, a generic assessment is independent of occupational knowledge and competencies as outlined in the frameworks in the previous section, whereas an occupationally embedded assessment includes knowledge and competencies of the respective occupational field. On the one hand, it is argued that due to the increasingly rapid technological development, specific knowledge becomes outdated quickly and hence must be acquired at the time of demand. A generic understanding of employability skills as depicted in an earlier section of this chapter are then beneficial prerequisites for the efficient acquisition and successful application of knowledge within concrete work tasks in the occupational domains. On the other hand, it is argued that the requirements for the successful completion of any work task differ between professional domains. As a result, employability skills are, to some extent, modulated differently across occupational fields as conceptually outlined throughout this chapter. An occupationally embedded assessment of employability skills respects these differences by including domain knowledge from the respective professional fields in the assessment. Thus, it is recommended to employ an assessment of employability skills within PISA-VET that accounts for both components. This approach will be outlined and justified in this section and is displayed in Figure 7.3.
Within existing OECD studies, skills are usually defined in a generic manner and targeted broadly towards ‘... effective functioning in everyday life’ and ‘... to participate in society’ (Schleicher and Tamassia, 2000[47]). Accordingly, in the existing PISA and PIAAC instruments, the influence of domain knowledge has been largely excluded to achieve generic and broadly applicable assessments. Effects of prior knowledge would be considered as a confounding variable. A generic assessment of employability skills entails the following advantages and disadvantages: One obvious advantage is the possibility to use existing assessment instruments, which reduces the development effort required and allows relying on well-validated scales. It facilitates a more nuanced contextualisation of assessment findings, a noteworthy strength that synergistically reinforces both aspects. Furthermore, the results are more generalisable and directly comparable to the results of existing international studies such as PISA and PIAAC contributing a straightforward and natural link to international policy making. Consistent assessment across occupations allows for comparing these occupational groups within PISA-VET. Finally, interpreting generic skills as outcomes of formal education and using these skills as predictors of occupation-specific skills is another advantage. However, it can be argued that this approach may not be as relevant for actual employability as domain knowledge and skills. Furthermore, a less context-dependent assessment of generic skills may be less relevant – and thus less engaging – for test-takers, potentially resulting in lower test motivation and, in consequence, lower validity of the data. Nonetheless, the insights grained from test motivation in PISA, PIAAC, and SSES may be useful, and various studies based on data from these assessments show that data is valid and allows for meaningful conclusions. As an additional measure, it may be worth exploring the evaluation of generic employability skills before delving into occupation-specific skills, when test engagement remains particularly high. Some advantages and disadvantages of a generic assessment of employability skills are depicted in Table 7.5.
Assessment of employability skills as a set of occupationally-embedded skills (Winther and Achtenhagen, 2009[48]; Klotz, Winther and Festner, 2015[49]) can be considered a “bridge” between occupation-specific skills that build on domain knowledge and experience and generic employability skills that are not specifically linked to a professional context. These occupationally modulated employability skills (see later section) offer basic approaches to solving fewer specific tasks in the professional field and they facilitate the acquisition of more specific professional skills at later stages. Accordingly, an occupationally embedded assessment of employability skills requires some basic knowledge of and experience in the occupational field. For instance, problem solving can be defined as a generic skill and assessed based on tasks in which context matters little. On the other hand, problem solving can be seen as domain-linked and accordingly assessed with tasks embedded into a professional field, not only in terms of context but also in terms of the occupation-specific skills that are needed in addition to problem solving. The latter approach acknowledges that, for example, a competent problem solver in automotive technicians is not necessarily a good problem solver in health care, and vice versa.
An occupationally embedded assessment of employability skills entails the following advantages and disadvantages: One of the advantages is a high level of construct and face validity because the construct is closer to what makes people employable in a specific field. Candidates are familiar with the context reflecting real life situations, which could lead to higher test motivation. Additionally, it is possible to develop the occupational simulation tasks first and identify the employability skills within them, providing a more practical approach to the assessment. However, one of the disadvantages is that the results are not comparable across occupational fields. Furthermore, a fully embedded assessment is challenging with regard to psychometrics and requires high costs for developing and/or adapting measurement instruments. Also, occupationally embedded assessments of employability skills are limited to currently available contexts and situations and may say little about future employability, which is at the core of PISA-VET. To ensure that the occupation-specific nature of employability skills and the unique way that they work in concert is adequately represented in the measurement, it is imperative that the development of occupationally embedded employability skills incorporates a clear and well-defined reference to the frameworks of employability skills and occupation-specific skills taken into account the respective frameworks corroborated throughout this volume and as displayed in Figure 7.3. Some advantages and disadvantages of an occupationally embedded assessment of employability skills are depicted in Table 7.6.
The assessment of employability skills is of fundamental importance for PISA-VET, and an optimal balance between generic and occupationally embedded is key for a valid interpretation of employability skills. After contrasting a generic and an occupationally embedded assessment of employability skills, it is recommended to combine the two approaches, in particular for literacy and problem solving, which have, in addition to their established generic component, an occupationally embedded dimension to them. The approach that is recommended is depicted in Figure 7.3. It is recommended for all four employability skills to implement a stand-alone generic assessment on the backdrop of existing frameworks and items: Literacy on the basis of PIAAC Cycle 2, problem solving on the basis of PIAAC Cycle 2, and task performance and collaboration on the basis of SSES (and with additional consideration of PISA). In addition to this, an occupationally embedded assessment with simulations that combine the respective occupation-specific skill and the employability skill (Figure 7.3) is recommended for the two performance assessments of literacy and problem solving. Such an approach might be less relevant for task performance and collaboration as they are considered trait-like characteristics with little variation across contexts and situations. For them, the occupationally embedded assessment is considered optional, and the core will be the generic assessment across contexts and situations.
Thus, two mutually complementing pillars are recommended for PISA-VET for the assessment of employability skills: Pillar 1 with a generic and stand-alone assessment of four employability skills and Pillar 2 with an occupationally embedded assessment of employability skills with a focus on literacy and problem solving. PISA-VET is completed by a third pillar that is composed of the occupation-specific skills assessment as outlined in Chapters 2 to 6 without or with only minimal involvement of employability skills (not depicted in Figure 7.3). Further information on the specific assessment of each employability skill is provided in the next section.
This section summarises the strategy for measuring the four different employability skills that will be part of PISA-VET: literacy, problem solving, task performance, and collaboration. Both, the generic (Pillar 1) and the occupationally embedded component (Pillar 2) are considered.
The OECD had led on the development of assessments of literacy among both schooled and adult populations in the context of the PISA and PIAAC assessments. Building on such experience, the VET assessment of literacy will be rooted in the conceptual framework of PIAAC that is outlined in an earlier section of this chapter. The assessment of literacy in PISA-VET will contain a generic and an occupationally embedded component as outlined earlier in this chapter and displayed in Figure 7.3. This reflects the fact that literacy has both a generic component as a transversal skill applicable across vocational contexts and, at the same time, an occupation-specific modulation, for instance, due to the specific nature of the type of texts encountered in a particular occupation.
The generic component of the literacy assessment in PISA-VET will be composed of items from the existing PIAAC literacy assessment within general professional contexts (as defined in PIAAC) and without specific occupational embedding (Pillar 1 in Figure 7.3). To ensure a meaningful assessment within the given time constraints and to reduce the burden for individual respondents, it is recommended to implement a scoping assessment of literacy with a minimum testing time of 15 minutes derived from existing PIAAC items. This will allow normative comparisons across different groups of respondents, but the criterion-referenced comparisons will not be as detailed as in the original PIAAC study. There, the scale contained six proficiency levels (see earlier section of this chapter). For PISA-VET with reduced testing time, a shorter assessment that allows separation into three proficiency levels (e.g. low, medium, and high) is a likely and feasible option. That is, estimates of overall levels of literacy will be obtained for sampled students in each participating country, and proficiency levels are defined in accordance with the PIAAC framework albeit at a lower level of granularity. Nevertheless, it will be worthwhile to explore the feasibility of employing a 30-minute testing duration for this performance-based assessment in a later phase (e.g. the Piloting Phase) of PISA-VET. An example of a generic literacy item from PIAAC is described and displayed in the Appendix of this chapter.
An occupationally-embedded component of the literacy assessment in PISA-VET might be composed of newly developed items and would account for occupation-specific modulations of literacy, for instance through embedding literacy items in a context of health care or by presenting a certain type of occupationally-relevant text that would not be part of a generic assessment such as a manual for automotive technicians (Table 7.4). Thus, this component combines literacy as generic employability skill with occupation-specific skills (Pillar 2 in Figure 7.3). An assessment time of approx. 15 minutes for the occupationally embedded literacy assessment is recommended. Importantly, this embedded assessment improves content validity, but does not allow for comparisons across occupations as items will be developed in an occupation-specific manner. To ensure the validity of the measure and enhance the interpretability of the results, the development of occupationally embedded literacy items necessitates a clear and precise alignment with the assessment frameworks of both employability skills (this chapter) and occupation-specific skills (Chapters 2 to 6). Achieving a high level of proficiency in occupationally embedded literacy skills demands not only robust generic literacy skills but also profound knowledge and skills within specific occupations. In terms of scaling, it is expected that literacy items in Pillar 2 should load on both employability skills (that are generically assessed in Pillar 1) and occupation-specific skills (that are generically assessed in Pillar 3). Scores derived from this assessment component will not contribute to a separate scale, but they might contribute to both the generic literacy scale and the occupation-specific skills scale in terms of better content coverage and better psychometric quality. Thus, the aim of the occupationally embedded component is to increase the validity of both the generic literacy assessment (complemented through occupationally embedded literacy) and the occupation-specific skill assessment (complemented through literacy within occupation-specific modulations). An example of an occupationally embedded item related to literacy (in the example: functional reading) is described and displayed in the Appendix.
The most relevant existing assessment of problem solving for the VET context is that of adaptive problem solving as proposed for PIAAC Cycle 2. Adaptive problem solving in PIAAC Cycle 2 underlines the relevance of both cognitive and metacognitive processes when facing new problems and stresses the digital information context of problem solving closely aligned with 21st century workplaces. Utilising this extensive experience, the VET assessment of problem solving will be rooted in the conceptual framework of adaptive problem solving proposed for PIAAC Cycle 2. This framework is outlined both in general terms and with its VET-specific modulation as described earlier in this chapter. Comparable to literacy, the assessment of problem solving in PISA-VET will contain a generic and an occupationally embedded component as outlined previously and displayed in Figure 7.3. This demonstrates that, just as other employability skills, problem solving exhibits a component that pertains across all vocational contexts, but, at the same time, exposes occupation-specific modulations, for instance, due to the specific tools that might be available for problem solving within a specific occupation.
The generic component of the problem solving assessment in PISA-VET will be composed of items from the existing PIAAC Cycle 2 problem solving assessment selecting items mainly from the work context and across different information environments. Accounting for the digital nature of most problem solving activities, the focus should be digital information contexts. For the generic component, there will be no occupation-specific embedding (Pillar 1 in Figure 7.3). Considering the juxtaposition of time constraints in PISA-VET and the paramount role of problem solving in digital information environments for current and future employability, an assessment of problem solving with a testing time of approx. 30 minutes derived from existing PIAAC Cycle 2 items is recommended. This will allow normative and criterion-based comparisons across different groups of respondents even though the level of measurement precision will be somewhat lower as compared to PIAAC Cycle 2, where more testing time is allocated to problem solving. A separation into four proficiency levels (below level 1, level 1, level 2 and level 3) as outlined earlier in this chapter is proposed. However, it is important to emphasise that the suggested testing duration of 30 minutes should be viewed as a minimum requirement because problem solving often entails intricate scenarios, and it is essential to encompass the digital aspects within these assessments. Both components necessitate a test duration of at least 30 minutes. An example of a generic problem solving item from PIAAC Cycle 2 is described and displayed in the Appendix.
The occupationally embedded component of the problem solving assessment in PISA-VET that is recommended will be composed of newly developed items and will account for occupation-specific modulations of problem solving, which can be manifold and will need to be elaborated alongside the occupation-specific skills frameworks. Such modulations might include identifying the reasons for and repairing an electrical breakdown for electricians or dealing with a previously not encountered medical emergency in health care. Thus, the occupation-specific component embeds problem solving as generic employability skill in occupation-specific skills (Pillar 2 in Figure 7.3). An assessment time of approx. 15 minutes for the occupationally embedded problem solving assessment is recommended. Just like for literacy, this embedded assessment of problem solving is expected to improve content validity, but it will not allow for comparisons across occupations as items will be developed in an occupation-specific manner. To ensure the validity of the measure and enhance the interpretability of the results, the development of occupationally embedded problem solving items necessitates a clear and precise alignment with the assessment frameworks of employability skills (this chapter) and occupation-specific skills (Chapters 2 to 6). Achieving a high level of proficiency in occupationally embedded problem solving skills demands not only robust generic problem solving skills but also profound knowledge and skills within specific occupations. In terms of scaling, it is expected that problem solving items in Pillar 2 should load on both employability skills (that are generically assessed in Pillar 1) and occupation-specific skills (that are generically assessed in Pillar 3). Scores derived from this assessment component will not contribute to a separate scale, but they will contribute to both the problem solving scale and the occupation-specific skills scale in terms of better content coverage and better psychometric quality. Thus, these items might have, psychometrically speaking, a loading on the generic problem solving skill scale and the occupation-specific skill scale. Two examples of an occupationally embedded item related to problem solving (in the examples complex reception competence and problem solving in business and administration in the form of an office simulation) are described in the Appendix.
The OECD has overseen the development of several international assessments of social and emotional skills, most notably in SSES (as an additional note and as outlined above: PISA can be considered as a relevant source of information in addition to SSES), where, amongst others, task performance and collaboration were the focus of the international assessment. Building on this strong level of experience, the assessment of task performance and collaboration in PISA-VET will be constructed around the conceptual framework for task performance and collaboration developed in SSES outlined earlier in this chapter.
Unlike the cognitively focused assessments of literacy and problem solving, the need for developing a modulation of task performance and collaboration across occupational areas is low. While social and emotional skills always show themselves in specific situations, they are generally considered trait-like characteristics that are largely stable across domains. In addition to this, task performance and collaboration are usually assessed via self-reports, which can lead to bias but is a highly efficient way of measuring complex skills. Thus, the assessment of task performance and collaboration in PISA-VET will mainly be composed of self-report items and a generic component in line with the approach taken in SSES and as displayed in Figure 7.3.
The generic assessment of task performance and collaboration in PISA-VET will be composed of items from the existing SSES assessment. Given the efficiency and brevity of the SSES scales, it is recommended that the full self-report scales from SSES are included in the background questionnaire of PISA-VET. This will include sub-scales of self-control, responsibility, persistence, and achievement motivation for task performance and sub-scales of empathy, trust, and co-operation for collaboration. Of note, some adaptations in the wording might be necessary to make the item content sufficiently relevant for the target population in the VET context but these changes should be kept to an absolute minimum. In SSES (Kankaraš and Suarez-Alvarez, 2019[29]), eight items were used to assess each sub-scale of task performance and collaboration, with each sub-scale taking approximately 1 minute to complete. Therefore, it is expected that approx.7 minutes will be required in total for task performance and collaboration. It is further recommended maintaining the established 5-point Likert scale that was employed in SSES. Proficiency levels were not developed for SSES, although reporting scales were. We recommend the same approach for PISA-VET allowing for comparisons across different groups of respondents, but without defining a distinct set of qualitatively different proficiency levels. An example of a self-report item of task performance and collaboration is given in the Appendix. PISA 2021 Context Questionnaire Framework (OECD, 2019[38]) has taken a comparable approach by incorporating SSES task-performance and collaboration items into its background questionnaire. Nevertheless, the results indicate that certain scales (e.g. self-control, a facet of task performance; trust, a facet of collaboration in PISA) did not exhibit satisfactory psychometric properties across diverse countries. In contrast, scales with a more extensive items (e.g. persistence/perseverance, a facet of task performance; cooperation and empathy, facets of collaboration in PISA) displayed sound psychometric properties. Hence, it is prudent to consider the international PISA experience when incorporating SSES items into PISA-VET.
While the generic assessment of task performance and collaboration is deemed sufficient in the PISA-VET context, there might be compelling reasons to complement the assessment with additional measures of task performance and collaboration that are occupationally embedded. For instance, when in a specific health care situation, a particular way of teamwork is essential for patient care and the relevant aspects of collaboration are not sufficiently represented in the generic assessment of collaboration. Assuming this kind of compelling reason for expanding the assessment of task performance and collaboration, a similar approach as described for literacy and problem solving could be viable to account for the occupation-specific modulation of social and emotional skills. In this case, new self-report items will need to be developed. A separate mode of assessment for these skills is not recommended (e.g. there are performance-based assessments of collaboration, but we do not recommend switching the mode of assessment away from self-reports). However, different response formats are acceptable, for instance when the assessment of task performance and collaboration is embedded within an occupation-specific skill assessment. Similar constraints as for literacy and problem solving apply in the sense that the occupationally embedded component will not be reported as a separate scale but will complement the generic scale of task performance and collaboration as well as the occupation-specific skill scale. Along these lines, occupationally embedded self-report items of task performance and collaboration can be either included in the background questionnaire or in the occupation-specific skill assessment of PISA-VET. Two examples of occupationally embedded self-report items related to collaboration and task performance (in the examples communication strategies and self-efficacy) are described and displayed in the Appendix.
Employability skills are not only vital for success in the workplace but also for personal growth, well-being, and mental health of the entire VET student. These skills enable them to navigate the challenges and opportunities of 21st century labour markets and lead fulfilling lives. In the context of PISA-VET, four core employability skills that are all foundational in their nature and that are considered vital for the selected occupational areas have been selected for inclusion in PISA-VET: literacy, problem solving, task performance, and collaboration. They cover a broad array of cognitive competencies and social and emotion skills and have previously been included in international assessments such as PISA and PIAAC.
Drawing on this experience, a dedicated three-pillar assessment framework is proposed (Figure 7.3). This framework includes a generic and standalone assessment of each of the four employability skills (Pillar 1), an occupationally embedded assessment of employability skills that primarily focuses on literacy and problem solving (Pillar 2), and an occupation-specific skills assessment that only minimally (or not at all) taps into employability skills (Pillar 3). This way, the assessment accounts for both the generic (i.e. foundational) nature and the occupation-specific modulation of employability skills, allowing for cross-occupation and international comparisons and, at the same time, acknowledging of and allowing for occupational specifics.
On a technical level, the proposed assessment framework allows for both norm-referenced (i.e. comparisons across occupations, across countries, and across other groups of respondents) and criterion-referenced (i.e. through the development of dedicated proficiency levels) assessments. Complex item formats embedded into computer-delivered simulations can facilitate detailed process examination beyond final performance data, enhancing the accuracy and reliability of the results. The use of realistic assessment environments with real-world contexts further facilitates face validity and relevance of PISA-VET. While not explicitly addressed in this chapter, emerging methodologies such as artificial intelligence and machine learning may hold the potential to assist in data analysis and to provide insights on the underlying processes of how VET students engage, utilise, and develop their employability skills.
Overall, the proposed assessment of employability skills is essential for understanding whether individuals are equipped with the necessary competencies to succeed in the labour market, both as we know it today but also as labour markets continue to evolve into the future. By assessing employability skills through a comprehensive and multi-pillar framework, PISA-VET will provide valuable insights into the strengths and weaknesses of education and training systems on an international level, supporting policy and practice in a rapidly evolving global economy.
References
[53] Abele, S. (2017), “Diagnostic Problem-Solving Process in Professional Contexts: Theory and Empirical Investigation in the Context of Car Mechatronics Using Computer-Generated Log-Files”, Vocations and Learning, Vol. 11/1, pp. 133-159, https://doi.org/10.1007/s12186-017-9183-x.
[24] Ackoff, R. (1981), “On the use of models in corporate planning”, Strategic Management Journal, Vol. 2/4, pp. 353-359, https://doi.org/10.1002/smj.4250020404.
[51] Balkenhol, A. (2016), Lesen in beruflichen Handlungskontexten: Anforderungen, Prozesse und Diagnostik, Technische Universität Darmstadt, https://tuprints.ulb.tu-darmstadt.de/id/eprint/5209.
[31] Barrick, M. and M. Mount (1991), “THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META‐ANALYSIS”, Personnel Psychology, Vol. 44/1, pp. 1-26, https://doi.org/10.1111/j.1744-6570.1991.tb00688.x.
[56] Bennett, E. et al. (2003), “Assessing Complex Problem Solving Performances”, Assessment in Education: Principles, Policy & Practice, Vol. 10/3, pp. 347-359, https://doi.org/10.1080/0969594032000148181.
[42] Chernyshenko, O., M. Kankaraš and F. Drasgow (2018), “Social and emotional skills for student success and well-being: Conceptual framework for the OECD study on social and emotional skills”, OECD Education Working Papers, No. 173, OECD Publishing, Paris, https://doi.org/10.1787/db1d8e59-en.
[37] Cunha, F. and J. Heckman (2007), “The Technology of Skill Formation”, American Economic Review, Vol. 97/2, pp. 31-47, https://doi.org/10.1257/aer.97.2.31.
[36] Cunha, F., J. Heckman and S. Schennach (2010), “Estimating the Technology of Cognitive and Noncognitive Skill Formation”, Econometrica, Vol. 78/3, pp. 883-931, https://doi.org/10.3982/ecta6551.
[28] Dimmock, K., H. Breen and M. Walo (2003), “Management Competencies: An Australian Assessment of Tourism and Hospitality Students”, Journal of Management & Organization, Vol. 9/1, pp. 12-26, https://doi.org/10.5172/jmo.2003.9.1.12.
[62] Drechsel, B., C. Carstensen and M. Prenzel (2011), “The Role of Content and Context in PISA Interest Scales: A study of the embedded interest items in the PISA 2006 science assessment”, International Journal of Science Education, Vol. 33/1, pp. 73-95, https://doi.org/10.1080/09500693.2010.518646.
[32] Dudley, N. et al. (2006), “A meta-analytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits.”, Journal of Applied Psychology, Vol. 91/1, pp. 40-57, https://doi.org/10.1037/0021-9010.91.1.40.
[33] Durlak, J. et al. (2011), “The Impact of Enhancing Students’ Social and Emotional Learning: A Meta‐Analysis of School‐Based Universal Interventions”, Child Development, Vol. 82/1, pp. 405-432, https://doi.org/10.1111/j.1467-8624.2010.01564.x.
[39] EASEL Lab. (2021), Explore SEL, http://exploresel.gse.harvard.edu/compare-terms/ (accessed on 26 January 2024).
[1] Frey, C. and M. Osborne (2017), “The future of employment: How susceptible are jobs to computerisation?”, Technological Forecasting and Social Change, Vol. 114, pp. 254-280, https://doi.org/10.1016/j.techfore.2016.08.019.
[10] Humburg, M. and R. Velden (2013), The Employability of Higher Education Graduates: The Employers’ Perspective, European Commission, https://data.europa.eu/doi/10.2766/54258.
[25] Jonassen, D. (2000), “Toward a design theory of problem solving”, Educational Technology Research and Development, Vol. 48/4, pp. 63-85, https://doi.org/10.1007/bf02300500.
[29] Kankaraš, M. and J. Suarez-Alvarez (2019), “Assessment framework of the OECD Study on Social and Emotional Skills”, OECD Education Working Papers, No. 207, OECD Publishing, Paris, https://doi.org/10.1787/5007adef-en.
[49] Klotz, V., E. Winther and D. Festner (2015), “Modeling the Development of Vocational Competence: a Psychometric Model for Economic Domains”, Vocations and Learning, Vol. 8/3, pp. 247-268, https://doi.org/10.1007/S12186-015-9139-Y/FIGURES/3.
[20] Lassébie, J. and G. Quintini (2022), “What skills and abilities can automation technologies replicate and what does it mean for workers? New evidence”, OECD Social, Employment and Migration Working Papers 282.
[27] Lorusso, L., J. Lee and E. Worden (2021), “Design Thinking for Healthcare: Transliterating the Creative Problem-Solving Method Into Architectural Practice”, HERD: Health Environments Research & Design Journal, Vol. 14/2, pp. 16-29, https://doi.org/10.1177/1937586721994228.
[2] McClelland, C. (2020), The impact of artificial intelligence–widespread job losses, IoT for all, https://www.iotforall.com/impact-of-artificial-intelligence-job-losses.
[58] Monnier, M. et al. (2016), “Occupation-specific social competences in vocational education and training (VET): the example of a technology-based assessment”, Empirical Research in Vocational Education and Training 8, pp. 1-18, https://doi.org/10.1186/s40461-016-0036-x.
[14] Murray, T., Y. Clermont and M. Binkley (2005), Measuring Adult Literacy and Life Skills: New Frameworks for Assessment, Statistics Canada.
[46] National Research Council (2012), Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century, National Academies Press, Washington, D.C., https://doi.org/10.17226/13398.
[22] Neubert, J. et al. (2016), “Complex Problem-Solving in a Changing World: Bridging Domain-Specific and Transversal Competence Demands in Vocational Education”, in Mulder, M. (ed.), Competence-based Vocational and Professional Education, Springer International Publishing, Cham, https://doi.org/10.1007/978-3-319-41713-4_44.
[52] Norwig, K. et al. (2021), “Tools to tap into the content of human minds”, Zeitschrift für Berufs- und Wirtschaftspädagogik, Vol. 117/4, pp. 658-693, https://doi.org/10.25162/zbw-2021-0025.
[21] OECD (2021), Beyond Academic Learning: First Results from the Survey of Social and Emotional Skills, OECD Publishing, Paris, https://doi.org/10.1787/92a11084-en.
[6] OECD (2021), The Assessment Frameworks for Cycle 2 of the Programme for the International Assessment of Adult Competencies, OECD Skills Studies, OECD Publishing, Paris, https://doi.org/10.1787/4bc2342d-en.
[11] OECD (2020), OECD Employment Outlook 2020: Worker Security and the COVID-19 Crisis, OECD Publishing, Paris, https://doi.org/10.1787/1686c758-en.
[4] OECD (2019), PISA 2018 Results (Volume II): Where All Students Can Succeed, PISA, OECD Publishing, Paris, https://doi.org/10.1787/b5fd1b8f-en.
[38] OECD (2019), PISA 2021 financial literacy analytical and assessment framework, OECD Publishing, Paris, https://www.oecd.org/pisa/sitedocument/PISA-2021-Financial-Literacy-Framework.pdf.
[5] OECD (2019), Skills Matter: Additional Results from the Survey of Adult Skills, OECD Skills Studies, OECD Publishing, Paris, https://doi.org/10.1787/1f029d8f-en.
[43] OECD (2017), PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic, Financial Literacy and Collaborative Problem Solving, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264281820-en.
[44] OECD (2017), PISA 2015 Results (Volume V): Collaborative Problem Solving, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264285521-en.
[19] OECD (2015), Education at a Glance 2015: OECD Indicators, OECD Publishing, Paris, https://doi.org/10.1787/eag-2015-en.
[7] OECD (2013), OECD Skills Outlook 2013: First Results from the Survey of Adult Skills, OECD Publishing, Paris, https://doi.org/10.1787/9789264204256-en.
[18] OECD (2013), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264190511-en.
[12] OECD (2012), Literacy, Numeracy and Problem Solving in Technology-Rich Environments: Framework for the OECD Survey of Adult Skills, OECD Publishing, Paris, https://doi.org/10.1787/9789264128859-en.
[16] OECD (2009), “PIAAC Problem Solving in Technology-Rich Environments: A Conceptual Framework”, OECD Education Working Papers, No. 36, OECD Publishing, Paris, https://doi.org/10.1787/220262483674.
[17] OECD (2003), The Pisa 2003 Assessment Framework: Mathematics, Reading, Science and Problem Solving Knowledge and Skills, OECD Publishing, Paris.
[9] OECD, ILO and The World Bank (2016), Enhancing employability: Report prepared for the G20 Employment Working Group, https://www.oecd.org/employment/emp/Enhancing-Employability-G20-Report-2016.pdf.
[13] OECD/Statistics Canada (2000), Literacy in the Information Age: Final Report of the International Adult Literacy Survey, OECD Publishing, Paris, https://doi.org/10.1787/9789264181762-en.
[45] Petrides, K. and A. Furnham (2001), “Trait emotional intelligence: psychometric investigation with reference to established trait taxonomies”, European Journal of Personality, Vol. 15/6, pp. 425-448, https://doi.org/10.1002/per.416.
[55] Rausch, A. et al. (2021), “Die web-basierte Bürosimulation LUCA – Funktionen, Einsatzmöglichkeiten und Forschungsausblick”, Zeitschrift für Berufs- und Wirtschaftspädagogik, Vol. 117/3, pp. 372-394, https://doi.org/10.25162/zbw-2021-0017.
[61] Rausch, A., K. Kögler and J. Seifried (2019), “Validation of Embedded Experience Sampling (EES) for Measuring Non-cognitive Facets of Problem-Solving Competence in Scenario-Based Assessments”, Frontiers in Psychology, Vol. 10, https://doi.org/10.3389/fpsyg.2019.01200.
[54] Rausch, A. et al. (2016), “Reliability and validity of a computer-based assessment of cognitive and non-cognitive facets of problem-solving competence in the business domain”, Empirical Research in Vocational Education and Training, Vol. 8/1, https://doi.org/10.1186/s40461-016-0035-y.
[26] Rausch, A. and E. Wuttke (2016), “Development of a Multi-Faceted Model of Domain-Specific Problem-Solving Competence and its Acceptance by Different Stakeholders in the Business Domain”, Unterrichtswissenschaft, Vol. 44/2, pp. 164-189, https://www.researchgate.net/publication/299746973_Development_of_a_Multi-Faceted_Model_of_Domain-Specific_Problem-Solving_Competence_and_its_Acceptance_by_Different_Stakeholders_in_the_Business_Domain (accessed on 11 February 2024).
[30] Roberts, B. et al. (2009), Conscientiousness, Handbook of Individual Differences in Social Behavior, pp. 369–381, The Guilford Press.
[60] Rosenberg, M. (2010), Gewaltfreie Kommunikation: Aufrichtig und einfühlsam miteinander sprechen; neue Wege in der Mediation und im Umgang mit Konflikten, Junfermann Verlag.
[41] Sackett, P. and P. Walmsley (2014), “Which Personality Attributes Are Most Important in the Workplace?”, Perspectives on Psychological Science, Vol. 9/5, pp. 538-551, https://doi.org/10.1177/1745691614543972.
[47] Schleicher, A. and C. Tamassia (2000), Measuring Student Knowledge and Skills: The PISA 2000 Assessment of Reading, Mathematical and Scientific Literacy, Education and Skills, ERIC.
[59] Schulz von Thun, F. (2019), Klarkommen mit sich selbst und anderen: Kommunikation und soziale Kompetenz: Reden, Aufsätze, Dialoge (6. Auflage), Rowohlt-Taschenbuch-Verlag.
[57] Seifried, J. et al. (2020), The computer-based assessment of domain-specific problem-solving competence—A three-step scoring procedure, Informa UK Limited, https://doi.org/10.1080/2331186x.2020.1719571.
[34] Sklad, M. et al. (2012), “Effectiveness of school‐based universal social, emotional, and behavioral programs: Do they enhance students’ development in the area of skill, behavior, and adjustment?”, Psychology in the Schools, Vol. 49/9, pp. 892-909, https://doi.org/10.1002/pits.21641.
[40] Soto, C. and O. John (2017), “The next Big Five Inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power.”, Journal of Personality and Social Psychology, Vol. 113/1, pp. 117-143, https://doi.org/10.1037/pspp0000096.
[35] Taylor, R. et al. (2017), “Promoting Positive Youth Development Through School‐Based Social and Emotional Learning Interventions: A Meta‐Analysis of Follow‐Up Effects”, Child Development, Vol. 88/4, pp. 1156-1171, https://doi.org/10.1111/cdev.12864.
[15] Tout, D. (2020), “Evolution of adult numeracy from quantitative literacy to numeracy: Lessons learned from international assessments”, International Review of Education, Vol. 66/2-3, pp. 183-209, https://doi.org/10.1007/s11159-020-09831-4.
[23] van Aken, J. and H. Berends (2018), Problem Solving in Organizations, Cambridge University Press, https://doi.org/10.1017/9781108236164.
[8] Vanhercke, D., N. De Cuyper and H. De Witte (2016), “Perceived employability and well-being: An overview”, Psihologia Resurselor Umane, Vol. 14/1, pp. 8–18.
[48] Winther, E. and F. Achtenhagen (2009), “Measurement of vocational competencies - A contribution to an international large-scale assessment on vocational education and training”, Empirical Research in Vocational Education and Training, Vol. 1/1, https://doi.org/10.1007/bf03546481.
[3] World Bank (2018), World Development Report 2019: The Changing Nature of Work, Washington, DC: World Bank, https://doi.org/10.1596/978-1-4648-1328-3.
[50] Ziegler, B. et al. (2012), “Diagnostik „funktionaler Lesekompetenz “”, Bwp@ Berufs-Und Wirtschaftspädagogik-Online, Vol. 22, pp. 1–19, https://www.bwpat.de/ausgabe22/ziegler_etal_bwpat22.pdf.
Greiff, S., & Borgonovi, F. (2022). Teaching of 21st century skills needs to be informed by psychological research. Nature Reviews Psychology, 1(6), 314–315.
Lassébie, J. and G. Quintini (2022), “What skills and abilities can automation technologies replicate and what does it mean for workers?: New evidence”, OECD Social, Employment and Migration Working Papers, No. 282, OECD Publishing, Paris, https://doi.org/10.1787/646aad77-en.
OECD (2020). OECD Employment Outlook 2020: Worker Security and the COVID-19 Crisis, https://doi.org/10.1787/1686c758-en.
This annex presents examples of published items from existing employability skill assessments that can be employed as generic assessments (Pillar 1 in Figure 7.3) along the lines of the recommendations contained in the chapter. One example each of a literacy item from PIAAC, of a PIAAC Cycle 2 problem solving item, of a task performance item a collaboration item from SSES is presented.
Only three literacy sample tasks were publicly released from PIAAC Cycle 1, and all three tasks were from the personal context. The example presented below, “Preschool Rules”, is of an average difficulty literacy task representing the following aspects of the literacy domain: i) cognitive processes: access and identify, ii) context: personal, and iii) medium: print-based text. There are also tasks available from a professional context, but these have not been released to the public, and may be a better choice for inclusion in PISA-VET. Tasks are delivered through computer-based assessment, with respondents being asked to highlight words and phrases that contain the answer to the task.
The difficulty of literacy tasks depends on three sets of factors, namely a) characteristics of the text(s); b) characteristics of the question; and c) the specific interaction between a question and a text (or set of texts). In addition, some of these factors affect the difficulty of the task regardless of the specific cognitive demands involved, whereas other factors are specific to a certain type of task demand (OECD, 2021[6]).
Source: PIAAC Cycle 1 assessment framework; (OECD, 2012[12]).
The item in the figure below is an example task presented in the PIAAC Cycle 2 in the Adaptive Problem Solving framework (OECD, 2021[6]). In this item, test takers are provided with a stock market simulation, in which they begin with initial stock in five companies, and a small disposable sum of cash that they can invest. They can sell stock for cash or buy new stock with cash. Stock prices vary on a day-by-day basis. A new “day” comes on screen every 60 seconds, with new information about the evolution of the five companies. A short history, i.e. the last few days in each company’s evolution are displayed on the screen. The pattern of change for some of the companies is transparent, i.e. future change is predictable. In this first item, problem solvers need to decide, based on the past evolution history of each company, where to invest their money. They need to sell the stock they do not need, and buy stock in the more promising companies, to maximise the value of their portfolio.
This item assesses the overall level of adaptive problem solving (it necessarily requests from the test-taker to go through all the phases of the problem solving process, and to use both cognitive and meta-cognitive skills). It is an example of a criterion- and norm-referenced, objective, digital and adaptive assessment, focusing on final outcomes for adaptive problem solving that is considered an important employability skill in PISA-VET.
Source: PIAAC Cycle 2 assessment framework; OECD, 2021.
Task performance derives from the SSES, where it is divided into four sub-domains that are each assessed separately: self-control, responsibility, persistence, and achievement motivation. Examples of items for each sub-domain are provided below in Table A1. These examples all come from SSES Round 1, except for “achievement motivation”, which comes from Round 2, because in Round 2 it was assessed as a separate rather than composite skill as in Round 1.
SSES uses generic, self-report assessments that focus on processes and behavioural patterns and use a norm-referenced perspective. Students are asked about their general tendencies and their observable behaviours or reactions. These items were developed for both computer-based and paper-pencil-based administration.
Collaboration derives from the SSES, where it is divided into three sub-domains that are each assessed separately: empathy, co-operation, and trust. Examples of items for each sub-domain are provided below in Table A2. These examples all come from SSES Round 1. SSES uses generic, self-report assessments that focus on processes and behavioural patterns and use a norm-referenced perspective. Students are asked about their general tendencies and their observable behaviours or reactions. These items were developed for both computer-based and paper-pencil-based administration.
In this annex, examples of occupationally embedded assessments of employability skills are presented. They pertain to slightly different skills as those that will be included in PISA-VET but are supposed to give an impression of the combination of a transversal employability skill and a specific occupational context (pillar 2 in Figure 7.3): Occupationally embedded assessments related to literacy, occupationally embedded assessments related to problem solving, and occupationally embedded assessments related to social and emotional skills.
Functional reading competence was defined and measured within the context of the German “Technology-oriented measurement of competences in vocational education” (ASCOT) programme funded by the German Ministry of Education and Research (BMBF). Ziegler et al. (2012[50]) developed an assessment of “functional reading competence”, which is defined as reading with the purpose of completing a vocational or a daily task and therefore translating the reading content into an action. Functional reading implies searching for specific information, for example in instruction leaflets, information brochures, (legal) regulations, tables, registers, charts etc., to take an action. It is closely related to the concept of literacy as understood in PISA-VET but embedded into an occupational context and requires occupation-specific skills.
In fact, depending on their prior knowledge, individuals sometimes only must select the relevant information from the given text, e.g. the application rate of a medication. In less familiar contexts, individuals first have to construct a mental model from the information. This means they must develop a concrete idea of the relevant process or task or adjust their knowledge based on new information. The construct of functional reading competence comprises descriptional representations (continuous texts), depictional representations (lists, tables, charts, forms etc.) and mixed representations. Further, the assessment differentiates three cognitive demands: identifying single information, integrating different information, and generating a mental model. The assessment aimed at measuring reading competence in an occupationally embedded context, but simultaneously aimed at being suitable for the assessment across different occupational areas addressed in the ASCOT initiative (car mechatronics, electricians, health care and business and administration). An example item is illustrated below (Balkenhol, 2016[51]). The participant is required to interpret the chart displaying the evolution of revenues and order numbers of North America, Europe, and Asia from the first quarter of 2003 to the second quarter of 2004. It is crucial to note that the diagram depicts two distinct aspects within the same coordinate system. Correctly associating the lines and bars with their respective statements is essential. The diagram allows for multiple statements about regional development, some of which lack detail.
To keep testing time to a minimum, the assessment is constructed as a computer-based adaptive test with multiple-choice items. Some of the items are taken or adapted from Programme for International Student Assessment (PISA), International Adult Literacy Study (IALS), or Adult Literacy and Life Skills Survey (ALL) that rather follow the generic assessment approach. However, most items were newly developed displaying vocational situations (e.g. order for office supplies, sales, job announcements). The construct of functional reading competence is derived using a one-dimensional Rasch model. In addition, Balkenhol distinguishes between four proficiency levels (Balkenhol, 2016[51]).
Source: Balkenhol, 2016.
Diagnostic skill in car mechatronics, which is closely related to the concept of problem solving, was defined and measured within the research programme ASCOT+ (the successor of the above-mentioned programme ASCOT) by Norwig et al. (2021[52]). Generally, the competence to diagnose car malfunctions is key in the occupation of car mechatronics (or automotive technicians) although research reveals that a huge proportion of students are not able to perform car-related problem solving tasks appropriately. Abele (2017[53]) distinguishes four subprocesses in the context of diagnosing car-related malfunctions: First, information regarding the car is collected. Second, hypotheses considering the cause of the malfunction are generated. Third, ideas how to test these hypotheses are derived and carried out. Finally, the results of the testing are evaluated. In this, the measured competence is closely related to the concept of problem solving but is strongly embedded into the field of car mechatronics and requires occupation-specific skills.
Since detecting malfunctions in motor vehicles is a complex task, electronic assistant systems are applied to help employees to identify those malfunctions. The electronic system provides information on how to proceed and suggestions on possible hypotheses and measurements. The output contains short texts loaded with technical terms, technical diagrams, and figures like electronic circuit diagrams (see Figure A4). Consequently, VET students need to learn how to deal with the available information, how to connect them with their domain-specific knowledge, how to build up a mental model and how to proceed in problem solving (this corresponds to the functional reading competence described by (Ziegler et al., 2012[50]). The authors regard this diagnostic skill as a specific form of problem solving and expect this competence to be crucial for detecting malfunctions in motor vehicles.
Source: Güzel et al., in preparation
As another example from problem solving, domain-specific problem solving in the context of business administration was measured within office simulations that were also developed within the ASCOT initiative (Rausch et al., 2016[54]). Rausch and Wuttke distinguish four subprocesses of the cognitive dimension of problem solving: identifying needs for action and information gaps, processing information, coming to well-founded decisions, and communicating decisions appropriately (Rausch and Wuttke, 2016[26]). Test takers must process complex and authentic business-related scenarios within an office simulation. The latest version, LUCA Office Simulation, is a browser-based learning and assessment platform available in German and English (https://luca-office.de). The LUCA Office Simulation provides test takers with typical software tools such as a document viewer, an e-mail client, a spreadsheet software, an enterprise resource planning software and so forth (see below screenshot). Within this open-ended environment, test takers have to solve authentic work scenarios over periods of 20 to 40 minutes. LUCA Office Simulation comes with additional modules to create and modify scenarios, administer assessments, and score results based on customizable scoring rubrics (Rausch et al., 2021[55]). Competence scores for the above subprocesses of problem solving are ideally derived in a three-step process according to Bennett et al. (2003[56]): (1) fine-grained scoring of problem solving behaviour, (2) aggregating the resulting behavioural patterns into partial credits, and (3) and calibrating a multidimensional Item Response Theory (IRT) model (Seifried et al., 2020[57]).
Source: (Rausch et al., 2021[55]).
A set of social and emotional skills were addressed and measured with a situational judgment test within the ASCOT initiative by Monnier et al. (2016[58]) for prospective medical assistants. The project aimed at measuring social competences particularly for this occupation considering occupation-specific situations and demands. The three key social and emotional competences of communication strategies, emotional regulation and perspective coordination were addressed due to a requirement analysis based on curriculum analyses and expert interviews. Those are related to task performance and collaboration, which will be included in PISA-VET, but present a somewhat different focus. They are included here to show how social and emotional skills could be assessed in an occupationally embedded way.
More specifically, competent communication strategies are important for medical assistants to handle complex communication with clients/patients balancing their needs as well as other restrictions (e.g. needs of an emergency patient) towards a good solution for both parties. Communication models like Schulz von Thun’s communication square (Schulz von Thun, 2019[59]) and Rosenberg’s Nonviolent Communication (Rosenberg, 2010[60]) were considered as a basis for appropriate communication. For example, establishing a common understanding by repeating what happened and which possibilities exist (sharing one’s knowledge) is a useful strategy.
The below figure provides a screenshot of a video presenting a typical scenario from the computer-based situational judgment test in which a client/patient is angry. Based on this video students must read a couple of (mostly) multiple-choice items and select how they would behave in this situation.
Source: Monnier et al., 2014
More directly related to the employability skills considered for PISA-VET, task performance could also be measured using Embedded Experience Sampling (EES), which was developed and validated as an occupationally-embedded assessment in the field of business administration (Rausch, Kögler and Seifried, 2019[61]). EES allows for an integrated measurement of non-cognitive facets of competences within simulation-based assessments. Test-takers are requested to pause their work on business-related scenarios at certain times and spontaneously answer short items (EES items) regarding their actual experience of the scenario. A similar approach was applied in PISA 2006 as an ‘embedded science interest assessment’ (Drechsel, Carstensen and Prenzel, 2011[62]). However, EES items are embedded in an EES event that resembles typical social interactions with non-player characters and hence, are part of authentic problem scenarios (Rausch, Kögler and Seifried, 2019[61]). The figure below shows screenshots of the LUCA Office Simulation. The items refer to situational interest and situational self-efficacy, but further items related to other social and emotional skills could easily be developed and implemented.
