This chapter defines “business and administration” as assessed by PISA-VET and elaborates on the framework for business and administration that has been designed for the Development Phase of the project, including descriptions of the competencies and constructs to be assessed. It presents and explains the processes, content knowledge and contexts and provides several sample items with descriptions of task characteristics. The chapter also discusses how performance in the business and administration area is measured and reported against proficiency levels and scales.
PISA Vocational Education and Training (VET)
Assessment and Analytical Framework
PISA
Abstract
Vocational education and training for business and administration occupations is a critical success factor for national economies: business and administration is an important cluster (and often the hub) in the business world and is present in every sector, region/country (Dummert, 2021[1]; Hoidn and Šťastný, 2021[2]). Many professions in this cluster are high in demand and short in supply. Overall, there are three main reasons for selecting this occupational area for PISA-VET related to the important role of the business and administration domain in the economy and the VET landscape, namely:
a. The domain is critical to the still ongoing trend toward a service economy.
b. Commercial clerks are trained and in demand all over the world; in some countries, commercial training is the most chosen training option in VET.
c. The occupational field is at the centre of several developments: automation (including artificial intelligence) and the digital transition, new forms of work, outsourcing/offshoring.
The corporate world can be seen as both the driver and the driven of digitalisation. On the one hand, it drives digitalisation processes to optimise corporate and work processes and achieve competitive advantages. On the other hand, the sometimes-disruptive developments of digitalisation present companies with major challenges in adapting their business models. The domain of business and administration can be regarded as one of the disciplines that are intensively dealing with the digitalisation of the working world and often assumes a pioneering role here. For example, companies are increasingly relying on digital work processes and tools such as collaboration tools, cloud software and artificial intelligence to optimize their workflows and make them more efficient. The use of big data and business analytics is also becoming increasingly important to make decisions based on data.
The domain of business and administration encompasses a wide range of activities related to the management and organisation of business enterprises. Therefore, the variety of occupations within the occupational area is huge. Specifically, the business and administration occupational area covers more narrow subareas such as Accounting and Finance, Marketing and Sales, Human Resources, Operations and Strategic Management and Information Technology. Rather than choosing to focus on one subarea, it is proposed to look at the entire area of business and administration but focus on core cross-cutting aspects.
In the International Standard Classification of Occupations (ISCO-08), the occupational area of business and administration covers major group 4 “Clerical support workers”, as well as group 33 “Business and administration associate professionals”. Clerical support workers record, organise, store, compute and retrieve information, and perform a number of clerical duties in connection with money-handling operations, travel arrangements, requests for information, and appointments. Business and administration associate professionals perform mostly technical tasks connected with the practical application of knowledge relating to financial accounting and transaction matters, mathematical calculations, human resource development, selling and buying financial instruments, specialised secretarial tasks, and enforcing or applying relevant government rules. Also included are workers who provide business services such as customs clearance, conference planning, job placements, buying and selling real estate or bulk commodities, and serving as agents for performers such as athletes and artists.
The proposed approach focuses on what young adults can do in their chosen occupational fields of business and administration near the end of their training programmes, not just what they know. Most countries focus on one of up to four VET programmes in the field of business and administration - the target sample will be able to be drawn from students participating in these programmes. Almost all the training programmes are located at ISCED level 3 and consist mostly of a combination of training on the job or practical training, and training in a VET institution (e.g. PISA-VET: A Feasibility-Study) (Baethge, 2006[3]). For the measurement of competency in the occupational field of business and administration, this chapter, therefore, focuses on ISCED level 3 (OECD/Eurostat/UNESCO Institute for Statistics, 2015[4]): upper secondary vocational education. With this in mind, the target population must meet the requirement of having already completed at least an ISCED level 2 qualification. Due to changes in educational biographies, heterogeneity can be observed when entering initial vocational training regarding the various commercial occupational fields, so an age group between 16 and 19 years can be assumed. With regard to the entry qualification at the start of initial training, differences can also be expected with regard to different occupational fields so that a qualification level EQF3 and EQF4 must be assumed. In terms of the EQF, the focus is primarily on level 4.
Selected programmes in data collection countries of PISA-VET
The possibilities of international comparability have already been examined within the framework of a feasibility study (Baethge and Arends, 2009[5]). For this purpose, selected test contents were subjected to international validation to check whether they can be transferred within and between different countries: It turns out that there are a variety of occupational tasks that are relevant in different countries. For the economic field, for example, a high degree of agreement can be found in the preparation of invoices, reports and annual financial statements.
Based on different international VET standards, legal and economic systems, it is of particular relevance to identify connected domain-specific elements and to narrow down the scope of the domain. To establish international comparability, the focus of an international VET-assessment must therefore be on the typical work situations of the trainees and their domain-specific requirements.
The chapter is based on the following key aspects, which are differentiated in the business and administration area as follows:
A PISA-VET must be based on a common understanding of the goals of VET in a specific occupational area. For the area of business and administration three central goals are defined: (1) to ensure individual mobility within the professional field, (2) to expand human resources in a society, and (3) to foster social participation. These goals are reference points for the definition of competencies in business and administration; they make it clear that professional competencies are the starting point for employability and integration.
A PISA-VET does not claim to be representative of all facets of a specific occupational field; in business and administration, the focus is on measuring competencies in specific fields of action in workplaces that can be considered typical for a broad professional area.
A PISA-VET records the competencies required for successful participation in the world of work in the 21st century. Against this background, in business and administration, professional competencies and employability are not discrete constructs, but measurable constructs on a continuum between occupation-related or cross-occupational and occupation-specific competencies to act appropriately in the workplace (Winther, 2010[6]). Occupation-specific competencies denote young adults’ abilities to successfully apply their knowledge and experience to authentic occupational situations in selected contexts that characterize the area of business and administration; occupation-related competencies refer to young adults’ abilities to be successful in the broader professional field and to develop professional identities (Klotz, 2015[7]; Winther and Klotz, 2016[8]).
In vocational education and training, the measurement of competencies is primarily discussed in terms of the domain-specific and domain-related content to which competencies relate (Winther, 2010[6]). A widespread opinion is that competencies should be derived from the description of concrete occupational activities. This would mean that the instruments of competency measurement would be oriented to and developed from the professional activity. In this case, the test items would be taken from a specific occupational context of action, which shows the characteristics of a domain (Wigdor and Green Jr., 1991[9]). This section, therefore, outlines the contexts in which learners of business and administration will be employed, the processes they will engage in and the underlying capabilities and knowledge content they will acquire in their training and apply in these contexts.
To be able to make statements about competency in the occupational area of business and administration, the focus of this chapter is not on a specific occupation, but on specific occupational activities, typically performed in administrative as well as business workplaces or other types of organisations with administrative areas. The work activities in business and administration are widespread but each can be affiliated at some stage of the operational value chain. The operational value chain is a framework used to describe the various processes and activities involved in producing and delivering goods and services to customers. It is a model that helps organisations understand how different parts of their operations are interconnected and how they contribute to creating value for customers (Porter, 1985[10]). Therefore, activities are categorised along operational value chains: (Porter, 1985[10]; Rüegg-Stürm and Grand, 2020[11])
Management activities – knowledge and action strategies regarding operational and management and the day-to-day operations; used to prepare decision-making aids for management.
Business activities – knowledge and action strategies regarding customer processes, production processes, and the distribution of goods; cover the quantifiable process of service creation and its marketing.
Support activities – knowledge and action strategies regarding business principles, business communication, and the preparation and presentation of relevant business information; address issues of corporate management and organization.
Operational value chains can be defined both in terms of internal company relationships (e.g. coordination between departments) and in terms of their market-oriented external image (e.g. relationships with customers, suppliers, and lenders). Specific occupational scopes of activity can be assigned, see Table 4.2. Occupational scopes of activities in this context can be defined as a range of activities that are affiliated with the same category in the operational value chain. By means of the occupational scopes, the occupational area of business and administration can be operationalised as a job family rather than a specific occupation. Within the job family approach, the focus is on such competencies that are necessary to be able to perform similar activities in different occupations of the area of business and administration. This implies basic competencies in the occupational field as well as their professional-content-related intersection with more specific occupations.
In particular, the megatrends of digitalization and the ecological transformation of the economy have the potential to significantly change (commercial) professions. Based on the megatrend of digitalisation, a change in work contexts can be expected in the future. The use of artificial intelligence and the automation of activities will mean that simple routine and administrative activities will become less important in day-to-day work (Frey and Osbourne, 2017). In view of the increasing importance of the use of big data in the operational work context, the handling of large volumes of data and their adequate processing will gain significance. Depending on the occupational field within the domain, it can be expected that support in decision-making processes and increased work on projects in agile team settings will become more important (Jordanski, 2019[12]; Seeber and Seifried, 2019[13]). With regard to the increasing implementation of environmental, social, and corporate governance (ESG), it can be expected that decisions for investments such as the establishment of new production and supply chains as well as corporate reporting will experience a change in values regarding the inclusion of sustainability-related assessment standards. In this way, the sustainability of products will become an important advertising and image factor that can make a significant contribution to a company's success (Williams et al., 2017[14]).
Orientation to operational value chains provides a good approach to this because changes in the economy are very quickly reflected in the value chains. In addition, the value chains can be used to identify contexts that can be considered relevant for education and training in the domain. The cognitive processes for mastering the context-related challenges as well as the content knowledge required for this can be defined.
The occupational scope of activities can be modelled via contexts. The contexts differentiate challenges at the workplaces and delimit relevant activities in different occupational situations from one another. When modelling contexts, two aspects are particularly crucial. First, the contexts describe specific occupational activities that can be summarised categorically. Three contexts are distinguished (see Figure 4.1):
1. Working with (business) data – This category describes occupational situations that relate to working with relevant operational indicators or finding and presenting appropriate representations of data.
2. Working with and through communication – This category describes occupational situations that focus on appropriate correspondence or customer-facing skills.
3. Working in and for administrative projects – This category describes occupational situations that can be related to operational project management and administrative work practices.
Source: author’s own representation
The linking element of the three contexts is working in a digital working environment. This refers to the use of digital tools for data preparation, communication, project administration and project management. An increasingly digital work environment represents a key contextual factor for a variety of professions within the domain. This can include cloud-based tools for storing, sharing, and collaborating on documents and files, communication tools such as instant messaging, video conferencing, and email to facilitate communication and collaboration among employees and project management tools to help teams track tasks, deadlines, and progress. The extent to which work environments are digitalised depends to a large extent on the industry under consideration and the size of the company. Nevertheless, the complete disappearance of face-to-face workplaces is not in the cards, so face-to-face communication with both colleagues and customers will continue to be important.
The three context categories – working with (business) data, working with and through communication, and working in and for administrative projects – are almost evenly contained in the test design (as shown in Table 4.2). This nearly balanced distribution allows no single context type to dominate so that learners are given tasks that cover a wide range of situations they expect to encounter in their work. The goal is to simulate workplace conditions that are as authentic as possible. At commercial workplaces, communicative and administrative tasks are increasing. Responsible for this are changed business models in which collaboration becomes necessary. In this context, commercial employees are often mediators between the individual departments. Due to the digitalisation of business and work processes, the relevance of dealing with big data is steadily growing and will continue to gain importance in the future. Therefore, the stronger focus on the items in this context area is forward-looking.
At the centre of an assessment construction for vocational education and training in the field of business and administration are competencies at business and administrative workplaces. That includes knowledge and skills as well as characteristic ways of acting in occupation-specific as well as occupation-related situations. Business and administrative competencies should be understood as learnable and thus measurable constructs, which are demonstrated by the fact that professional challenges can be mastered through the adequate and understanding-guided use of knowledge and skills. Therefore, business and administrative competencies include both the ability and potential to perform typical tasks in administrative and business workplaces based on a systemic understanding of operational and administrative sub-processes. An individual's competency includes occupation-specific and occupation-related competency dimensions. Occupation-specific competencies include recognising, analysing, using, and evaluating business principles, data, and processes to successfully apply individuals´ knowledge and experience to authentic occupational situations and contexts; occupation-related competencies refer to individuals' abilities to succeed in the broader occupational field by using transversal competencies like numeracy and digital and ICT literacy skills appropriate to an business and administration occupational situation (OECD, 1996[15]; 2012[16]; 2015[17]; 2019[18]). This implies a mature approach to business communication and to working tools and techniques for generating general templates, organising professional collaboration, and creating value, for themselves, for the company and for the community. Both occupation-specific and occupation-related competencies help individuals make informed judgments and decisions in the occupational domain (Greiff et al., 2014).
In the following, the question must be answered which cognitive processes and resources are necessary to cope with an occupational situation. In the context of the construct representation, the cognitive processes and resources are chosen as the starting point for the description of occupational situations. Hypotheses about the solution process are consequently translated into hypotheses about the contents of the demanding situations and their difficulty parameters to arrive at a detailed description of the measurement construct (Pellegrino, Chudowsky and Glaser, 2001[19]). For this purpose, a sufficiently precise idea must be developed about which processes take place during the processing and solving of occupational situations. Therefore, the process category describes the underlying cognitive processes required to complete the assessment. The cognitive processes describe the abilities of professionals to recognise, understand, apply, analyse, and evaluate occupation-relevant concepts of the business domain.
For the business and administration area, four process categories are often differentiated (see Figure 4.2; e.g. (Rausch et al., 2021[20])):
Source: author’s own representation
1. Identifying and understanding business and administration information – This process addresses the search for workplace-related information and the recognition of its relevance. Information in the business domain is mainly presented in text form, such as emails from customers or supervisors, business letters, invoices, or presentations from Enterprise Resource Planning (ERP) systems. The process category also includes the recognition and understanding of technical terms and different types and sources of information within the domain.
2. Analysing business and administration information and situations – This process describes the ability to analyse information and recognise relationships between different professional contexts in relation to a specific problem situation. This includes, for example, criterion-supported analysis and interpretation of key performance indicators in relation to different professional contexts, analysing the impact of pricing strategies, or analysing marketing metrics and interpreting them in relation to product sales. The process of analysis also includes understanding the underlying analytical assumptions and their implications.
3. Applying business and administration knowledge and skills – This process focuses on acting in professional situations by applying professional conceptual knowledge in conjunction with professional action knowledge. The testee realises a typical work product. This can include, for example, communicating and corresponding adequately in one's own mother tongue or in a foreign language to finalize a contract conclusion or to inform different stakeholders appropriately. A more complex use case could be the development of a solution or the formulation of recommendations for action for a job-specific problem that requires the use of mathematical and linguistic skills and/or the use of digital work tools such as Office/ERP/CRM.
4. Evaluating and reflecting on business and administration issues – This process refers to the use of knowledge and skills that goes beyond a specific professional situation. This may involve critically evaluating the pros and cons of a career decision both in terms of one's own point of view and in relation to third parties. It includes cognitive skills such as reasoning, evaluating, and generalising. It also includes critical reflection on one's own professional behaviour during the work process and work outcomes and formulating changes in behaviour.
The operationalisation of the process category is based on the taxonomy of Marzano and Kendall (Marzano, Kendall, 2007[21]) without assuming a hierarchy. The order of the processes rather reflects a typical sequencing for processing an occupational action situation. It is considered that in professional actions an interaction of the described processes can take place. Regarding assessment, tasks are evaluated according to which process category is most important for solving the task. Therefore, the process category is used as a tool to characterise workplace requirements and to show what competencies an apprentice/trainee should have regarding an occupational situation to be able to cope with this situation.
With recourse to cognitive models, the four identified subcategories of the cognitive system represent differentiable processing and action processes that can be assigned to different levels of complexity: (1) the activation and situation-specific retrieval of knowledge and skills, (2) the elaboration of knowledge and skills and thus the representation of comprehension processes, (3) the inferential interpretation of knowledge and skills with regard to the plausibility of a situation-specific solution, and (4) the utilisation of knowledge and skills, for example, in the context of transfer tasks.
Even though recent studies show that the process categories in the test items are not determinants of difficulty, a distribution is aimed that places an emphasis on the processes of analysis and application. This considers the fact that at workplaces in the field of business and administration, these processes also dominate. The same applies to the corresponding skills of the trainees. The ability to act within system contexts and that of goal-oriented action in concrete situations predominate in practice over those of isolated knowledge acquisition and independent further processing.
The distribution proposed here provides the plan for selecting items based on practical relevance and on the importance to the workplace. Item selection is based on item characteristics related to a range of framework aspects - including the process and context categories described. In summary, this is reflected in the underlying definition of vocational competency in the domain of business and administration:
Vocational competency is the potential to act adequately in occupational situations; it is expressed in the capability to cope with variously challenging situations at administrative as well as business workplaces by incorporating both occupation-specific as well as occupation-related knowledge, skills and strategies.
Regarding the requirement profile, occupational situations can vary, so a situation-specific explication of the higher-level capability of coping with differently complex and challenging occupational situations is necessary. The situation-specific explication is carried out via business and administration processes and capabilities which are typically in the focus of action in a situation-specific manner. Accordingly, four fundamental business and administration capabilities underlying the cognitive processes can be usefully identified:
1. The capability to retrieve situationally adequate knowledge stocks: Retrieving business and administration data and other relevant information.
2. The capability to describe system coherences and to act within them: Drawing conclusions from business and administration data and other relevant information.
3. The capability to think and act purposefully in specific situations: Making appropriate decisions/proposals for business and administration work products.
4. The capability to independently control processing procedures: Establishing appropriate, quality-assuring working practices for business and administration workplaces as well as promoting their own professionalism.
These four capabilities must be integrated to cope with various challenging situations at administrative as well as business workplaces. Processes and capabilities meet in the test situations. The processes describe the requirements of the situation; the abilities describe the possibilities of the person. The interaction of process and capability is consequently based on what is considered a typical situation in administrative as well as business workplaces. Consider for example the following occupational situation. This situation is an authentic and typical work situation; it integrates all four business and administration capabilities:
You are employed as an assistant to the head of the sales department of a household appliance manufacturer. Due to geopolitical events, there are difficulties in prospecting the future revenues of our newly released product, a smart vacuum cleaner robot. We want to sell our product nationally as well as in the United States. Our sales department calculated that with the revenues of both markets, our product should have a positive cash flow within three years. Your supervisor asks you to evaluate the cash flow calculation based on recent events and to develop a justified proposal when we can expect a positive cash flow at the earliest and should we release our product on both markets?
This occupational situation refers specifically to the context: working with business data. How this occupational situation can be differentiated via the processes and which capabilities underlie it is shown in the following table. The table should be read as follows: The complex occupational situation is divided into process categories - these are the last four columns of the table. It becomes clear that each occupational situation - if it is authentic and complex - addresses several process categories (identify, analyze, apply, evaluate). The process categories describe the cognitive requirements that lie in complex, occupational situations and thus in individual processes. These requirements are empirically well separable construction criteria for test task. The last four rows of the table indicate which capabilities are needed to master the process categories of an authentic and complex occupational situation. The capabilities must meet the requirements at the process level to be able to deal appropriately with the challenges of the occupational situation. In this table, the two aspects of competence measurement come together: the demand of the situation and the ability of the person.
Content knowledge is interpreted in a broad sense as action-relevant knowledge which is necessary to be able to decide, against the background of occupational complexity, how to deal with the authentic occupational situation, considering operational main and side effects (e.g. being able to name which information is necessary for an economic decision, such as the selection of a supplier). Content knowledge is the flagship of the test instrument; it determines the breadth and depth of the construct being measured.
The following content knowledge categories are suggested:
1. Content knowledge of the significance of relevant operational indicators in business administration and how to interpret and present data appropriately – this content knowledge is linked to the occupational context 1: working with business data.
2. Content knowledge of appropriate procedures and tools for correspondence and communication, of cross-cultural differences, and of dealing with diverse customers and stakeholders – this content knowledge is linked to the occupational context 2: working with and through communication.
3. Content knowledge of methods, forms, and areas of application of project management and of the use of concrete work strategies – this content knowledge is linked to the occupational context 3: working in and for administrative projects.
Each of these content knowledge categories incorporates knowledge of how and when to use digital work tools, what information-seeking strategies are useful, and how to manage data.
Source: author’s own representation
Regarding the measurement of vocational competency, it is demanded that the contents of the measurement be as complex as possible to capture an understanding of the dynamics of economic reality. Against this background, static linearised views of business realities are abandoned, and the system-oriented approach of business and administration is pursued. System orientation means emphasizing the interrelationships between individual topics, concepts, and instruments to integrate the individual content knowledge into a meaningful whole that can depict specific, dynamic relationships.
The system perspective of business administration is to be supplemented by process orientation: All value-added activities of a company and the necessary organisational and administrative work are performed in processes that can be characterised by a certain factual and temporal logic in the execution of specific tasks. Consequently, cross-functional business processes ask about the sequence, the temporal course, the priorities of subtasks as well as the use of people, material goods and the use of information (operational process organization).
An almost balanced distribution of content knowledge categories is sought in the assessment. Analogous to the distribution of context categories, we consider a stronger content focus on the adequate handling of data for analysis and decision-making to be forward-looking in view of the increasing digitization of corporate and work processes. Mapping the system perspective requires a well-balanced distribution of items across content categories.
To effectively understand and solve contextualised problems involving the content categories requires drawing upon a variety of business and administration concepts, procedures, facts, and tools at an appropriate level of depth and sophistication. As an assessment of vocational competency in the field of business and administration, PISA-VET strives to assess the levels and types of professional skills in the domain that are appropriate for VET learners on a trajectory to become skilled professionals able to make well-founded judgments and decisions at the workplace (Winther et al., 2016). It is also the case that PISA-VET, while not designed or intended to be a curriculum-driven assessment, strives to reflect the business and administration area that students have likely had the opportunity to learn by the time they are approaching the end of their initial VET programmes.
The field of business and administration is primarily characterised by its service orientation and the provision of services. The proposed occupational contexts as well as the associated content-related knowledge categories explicitly take this into account. It is recommended that the focus of an international comparison be placed on occupational tasks that deal in different ways with aspects of service provision and are based on different work processes (Holtsch et al., 2016[22]). Therefore, all knowledge categories are based on real work and business processes. Knowledge must be activated to master occupational challenges in authentic situations. The idea behind using authentic challenges as an assessment style is to observe authentic behaviour in authentic test environments as key indicators of vocational competency in real-life settings (Sangmeister et al., 2018[23]). In addition, and consistently geared to the requirements of working life in the business and administration area, content knowledge categories are highlighted that stimulate adequate work in corporate domains with their specific economic contexts. Items in this category are referred to as occupation-specific challenges. Each occupation-specific challenge refers to a systemic view of business data and processes regarding economic decision-making processes.
The focus of the assessment is on knowledge stocks and skills that can be assumed to be needed for an individual's participation in working life. The requirements for graduates of vocational education and training are primarily oriented toward the workplace and, beyond that, the labour market. This implicitly includes personal development, as this is the only way to ensure that graduates can adapt to changing working and living conditions.
Consequently, the following questions are central to the assessment: (1) How can vocational competency be defined, and which operationalisations are useful? Two aspects are of central importance here: on the one hand, the domain and thus both the contexts and the associated contents of professional competency. On the other hand, the person is a characteristic carrier of professional competency. Unlike comparable international large-scale assessments (Third International Mathematics and Science Study (TIMSS); cf. (Baumert, Bos and Lehmann, 2000[24]) the focus is not on curricular congruence but on functional aspects of education. Closely related to this are (2) the questions of what radius of action, what scope for action and decision-making graduates of vocational education have at their workplaces and on the labour market, and thus of typical contexts of requirements by means of which "competent" behaviour is demonstrated, as well as (3) the question of test situations and test items that adequately depict vocational behaviour and thus make it accessible to a competency measurement. (4) The evaluation and analysis are linked back to the requirement profiles of the occupations and the labour market and thus along the action parameters of the employees.
In the preceding chapters, questions (1) and (2) were answered. Based on the value chains and the associated occupational scopes, the context, the processes, and the content for the domain have been identified (job family approach). The domain as well as the occupational actions have thus been comprehensively described and explained. Questions (3) and (4) are addressed below.
Question (3) clarifies which options are available for recording vocational competency in relation to workplace activities. In general, two procedures can be distinguished in this context, which can have different characteristics (cf. (Rauner et al., 2009[25])):
The external and/or self-assessment of work behaviour by questionnaire-based or rater-based procedures as well as.
Simulation-based procedures in which work behaviour can be observed and diagnosed through knowledge and performance measurements.
The feasibility study on the implementation of an international comparative large-scale assessment in vocational education and training (Baethge, 2006[3]) relies on simulation-based procedures and thus follows the current findings and recommendations of major international comparative studies. Procedures of external and self-assessment can usefully complement simulation-based assessments, especially for validation.
A simulation-based method of vocational competency measurement is proposed for an international assessment in the field of business and administration. Simulations can be used to visualise value chains and the associated professional scopes of action and contexts. Simulation-based test platforms can be used to represent the authentic occupational tasks to be mastered by trainees, including the action and communication processes required to master them. Computer simulation ensures that:
1. concrete processes can be observed in typical contexts, and
2. content knowledge can be tied to work products that are available as standardized results for analysing trainees' capabilities.
A simulation-based approach to a vocational assessment should be modelled in an occupationally authentic setting to represent occupational reality. Authenticity is an essential prerequisite for measuring occupational competency (Deutscher and Winther, 2018[26]). Authentic contexts serve to shape the space of action (e.g. working with and through communication, such as formulating apologies for delivery difficulties), which can be characterised by various artefacts of professional action:
1. Requirements and tools of work (e.g. office software, ERP) that determine the scope of professional action.
2. Patterns of work (e.g. professional reasoning strategies) to determine the value of a professional action.
3. Ways of working (e.g. communication, judgment) to describe the importance of cross-cutting competencies within a professional action.
The artefacts are design anchors for constructing an authentic context. They determine how well professional activities as well as values and attitudes are represented in the test design (Figure 4.6). Furthermore, authentic situations consider the situational nature of competencies and thus increase test validity. Artefacts are used to construct multi-channel and interactive test stimuli given by videos, animations, or authentic tools.
Design anchor’s structure occupation complexity. They help construct authentic test environments that address both the action aspect and the value aspect of occupational competencies. The starting point for this is an accurate and authentic description of the workplace and the tasks that an employee/worker must perform. The action aspect concerns concrete professional challenges that must be dealt with and solved in such a way that an economically sensible decision can be made (e.g. handling purchasing and sales, data preparation, and invoicing). Values and attitudes describe transversally how employees/workers behave in the professional context (e.g. professional reasoning strategies, professional communication, and judgment).
Source: author’s own representation
Actions and value attitudes depend on how understanding-guided professional tasks can be processed and which concrete possibilities of doing/acting exist in the process of the action itself. Together with the artefacts (requirements and tools of working, patterns of work, and ways of working) sufficient information with workplace relevance is available in this way. This provides the basis for an authentic test environment. Considering the job family approach, an authentic test environment consists of several test units: Test units with which those competencies can be assessed that are necessary to be able to perform similar activities in different occupations in the field of business and administration and test units that relate to sector specifics. The design anchors make it very easy to link sector-specific content to the basic vocational competencies.
The simulation-based approach using digital test platforms will be complemented with elements from more traditional knowledge-based tests. Moreover, due consideration should be given to the potential benefits of integrating a live demonstration (or video recorded) in the assessment. Given the nature of the business and administration occupational area and its processes, the value added of such live demonstrations are likely to be limited compared to what can be done using digital test platforms.
Vocational competency in the domain of business and administration is defined as primarily activity-related competency in situations of occupational demand, which can be described in terms of mastering a typical work process. Based on a specific work process, it is thus possible to include many aspects of vocational competency – to the extent that they are necessary for mastering the work process – without having to assume that the dimensions are empirically separable. For the international assessment, vocational competencies are measured as occupation-specific competency across occupational contexts and, by definition, this includes occupation-related competency as well.
Vocational competency in the business and administration domain refers to the interplay of occupation-related and occupation-specific dimensions of competency, based on which trainees have the capability to make entrepreneurial and broader administrative decisions in authentic occupational situations and to validate these decisions to expand their own knowledge and action potential. The distinction between occupation-specific and occupation-related competency is based on a differentiated structure of vocational competency that can be related both to the occupational contexts and to the various cognitive processes with which typical occupational content is mastered (see Figure 4.5).
This considers the fact that competency in a domain is (1) both context-based as well as content-related and (2) not a one-dimensional cognitive disposition and that, consequently, it is not sufficient for competency acquisition and development to focus only on one cognitive structure (Kilpatrick, Swafford. and Findell, 2001[27]). This model assumption allows the competency areas to be analysed separately without violating the structure of the model.
Source: author’s own representation
For test development, the competency structure model of vocational competency has the following implications:
The starting point for test development are authentic occupational contexts that are based on the reality of workplaces. In these contexts, professional action takes place and professional content knowledge is used to master the occupational challenge.
The degree of demand of the occupational contexts is to be selected in such a way that their processing is differently challenging and addresses different processes.
The contexts are to be constructed in such a way that two different competency structures can be mapped with them: occupation-specific and occupation-related competency.
Assumptions are to be formulated as to what effects a chosen test format has on the response behaviour of the trainees in the test and to what extent conclusions can be drawn about competency structures and capabilities via the response behaviour shown. When choosing a simulation-based test format, it is implicitly assumed that by simulating authentic work contexts and processes, work actions and professional behaviour can be depicted better than would be possible with test formats on paper and pencil (criterion of authenticity). This is supported above all by the fact that the use of simulations eliminates the need for a verbal description of the processing steps in an authentic occupational situation and that both the "handling of concepts" and the "handling of technical equipment" can be measured simultaneously in the complexity typical of real work processes. The aim is to develop occupational contexts in such a way that it becomes possible to relate the tasks to be processed to the underlying work processes and, at the same time, to obtain information about the cognitive processing steps during the solution process (criterion of activity-specific validity). In other words: The design steps enable the modelling of tasks that represent occupational contexts and at the same time allow statements about the cognitive requirements during their processing as psychometrics.
Criterion of authenticity: It must be ensured that the test formats used to present the items can be considered "authentic". Following Doyle (2000[28]), the following dimensions of authenticity are considered in the test formats:
Content authenticity: it involves real data, instruments, and procedures of a particular field/occupation.
Situational authenticity: This refers to identification in the test through a high degree of proximity to the workplace and around experiencing a "community of practice".
Criterion of activity-specific validity: Internationally, there are different cuts for occupations for which activity clusters can be identified in an international comparison, which can be used as an activity-related validity criterion. This makes it possible to identify a workplace and concrete work contexts and tasks. Value chains and occupational scopes can thus be recorded in a valid manner.
For the test format, this means that an authentic description and recording of the action execution in the contexts must be possible. It must therefore be possible to assess the following via the respective test format:
authentic work processes (e.g. negotiations, work distribution)
authentic work orders (e.g. the concrete processing of an order), and
authentic work results (e.g. determination of a delivery date).
Therefore, the test uses different types of response formats that can be implemented in simulations: e.g. open constructed-response, closed constructed-response and selected-response tasks (simple and complex multiple-choice tasks).
Simulations can be used to depict the authentic occupational contexts to be worked on by the trainees, including the action and communication processes required for processing, as well as the content knowledge to be used. In the following, a fictitious user interface and individual test items are used to illustrate how simulations can be used to measure vocational competencies in the business and administration domain. It is shown how the criterion of authenticity and the criterion of activity-related validity are met. The development of a simulation-based test environment requires modelling steps that translate notions of reality with respect to the ideas and goals of testing (criteria of authenticity and validity). The following construction and design criteria are characteristic (see Figure 4.6)
Note: Videographic representations of authentic occupational contexts are currently only available in German. Short stimuli are given to increase identification with the task at hand. Two perspectives are provided: perspective 1 describes interactions between colleagues; perspective 2 involves the trainee directly (for example, by means of video conferencing).
Source: author’s own representation
Workplace simulations contain general introductions to the company structure (including the company history) as well as all-general compilations on the company's business situation (text-based and graphical analyses of business figures as well as presentations of the balance sheet, income statement and cash flow statement). This information can be used to classify the value chains and the associated occupational scopes.
The test situation itself is linked to animated administrative and business workplaces (desk with access to all relevant information and (business) data as well as embedded office software and digital communication tools). This allows the various authentic occupational contexts to be modelled.
Videographic representation of the authentic professional context including the description of the action aspect as well as the value aspect define the item stem (stimulus).
Both artefacts of professional activities as well as products of professional action ensure authenticity. The artefacts in the context of stimulus presentation and the products as authentic outcomes of task/item completion.
Using the example of the occupational context “working with business data”, example test items are presented below. The occupational contexts are constructed in such a way that they could each be used as an independent test environment.
Initial situation presented by video: Even in difficult market situations, a customer-oriented company tries to pass on price advantages directly to customers. To this end, individual price lists are drawn up for the most important customers. This approach has proved successful to date; in some cases, customers have increased their order quantities in response to price reductions, and customer satisfaction has improved. The price lists were published last week. First orders are to be processed.
Assess to the following tools of work: The trainees are provided with a simulated ERP system and additional information on the company's customers and products as well as on warehousing and production planning. This information is necessary to fully confirm the sales order.
Products/results requested to assess vocational competency: A complex work order is simulated that contains the following requirements, which can be formulated into six test items:
Create the order confirmation using the information available from the ERP system.
Determine the valid price for the sales order and determine its deviation from the list price.
Notify production that there will be delivery delays. Communicate this to the customer via e-mail.
Based on the information from production, determine the expected production and shipping times.
Give a reasoned judgment. Are the delivery delays within the normal range or do they deviate significantly?
What measures can be taken to reduce delivery delays? Draft a justified proposal.
The six items illustrate that both occupation-specific and occupation-related skills are required to answer them. The items also show that different processes are addressed by them and that they refer to different content knowledge. To assess the capability to cope with differently challenging situations in administrative and business workplaces, these and other similar sequences must be solved.
The model most intuitively applicable to vocational competency measurement is the Partial Credit Model (PCM) (Masters, 1982[29]). The term partial credit refers to the fact that it is possible not to receive full credit on an item, but to receive partial credit. The PCM has the advantage over other item response theory (IRT) models in that the distance between the points awarded can vary. This advantage is countered by a demanding coding. It will have to be examined whether a suitable modelling of complex simulation-based test designs is also possible on the basis of the generalised partial credit model (GPCM).
One way of scoring is to evaluate the quality and the reference area of an economic decision. The following figure shows an example of this.
Source: author’s own representation
For international comparison, the assessment is constructed based on a competency structure model. It has a two-dimensional structure. The test contains items regarding a between-item multidimensionality and some item constellations that represent the interaction between occupation-specific and occupation-related competency in terms of within-item multidimensionality. Therefore, confirmatory multidimensional IRT models are used for the analysis.
PISA generally works with one-dimensional scales that are subdivided into grades/levels of competency. However, a VET assessment in the domain of business and administration is intended to record professional behaviour in authentic occupational contexts, which conceptually and empirically entails occupation-specific skills and task performance, on the one hand, and occupation-related skills and working methods on the other hand. Against this background, it is suggested to use a two-dimensional model for the development of tasks and the empirical modelling of competency scores. Both scales can be divided into different competency levels. This grading makes it possible to relate well the task difficulties and the competency levels of the study participants in an international comparison.
The competency dimensions are empirically distinguishable but highly correlated. Reporting on a common scale is possible; the focus of reporting will be on the different levels of proficiency. How many proficiency levels can be differentiated is an empirical question and depends on the test design and the breadth of test content. It is proposed to construct the test along coherent causal relationships between workplace requirements (occupation-specific), labour market requirements (occupation-related), and cognitive psychological assumptions to heuristically locate the test items on a one-dimensional scale (Wilson and Draney, 2004[30]). Five levels of proficiency (0 to 4) are proposed, which can be interpreted well in terms of content. Level 0 (off-task) describes a missing answer and/or provides no evidence for the economic decision made. Level 4 (advanced), on the other hand, goes significantly beyond the daily work routine, e.g. by providing additional evidence or suggesting solutions that reveal complex economic contextual knowledge.
The levels of competency described are deliberately kept very general. To what extent they emerge via the measurement model is an empirical question. Existing studies confirm the graduation proposals - here then with a specific look at the underlying measurement construct and with a stronger differentiation regarding the field of economics and administration (Deutscher and Winther, 2018[26]).
References
[3] Baethge, M. (2006), PISA-VET: A Feasibility-Study, Franz Steiner Verlag.
[5] Baethge, M. and L. Arends (2009), “Die Machbarkeit eines internationalen Large- Scale-Assessment in der beruflichen Bildung: Feasibility Study VET-LSA. Eine komparative Analyse von Ausbildungsinhalten und Berufsprofilen in acht Europäischen Ländern.”, Zeitschrift für Berufs- und Wirtschaftspädagogik, Vol. 105/4, pp. 492-520, https://doi.org/10.25162/zbw-2009-0036.
[24] Baumert, Bos and Lehmann (2000), TIMSS/III: Dritte Internationale Mathematik- und Naturwissenschaftsstudie — Mathematische und naturwissenschaftliche Bildung am Ende der Schullaufbahn, Leske u. Budrich, https://doi.org/10.1007/978-3-322-83411-9.
[26] Deutscher, V. and E. Winther (2018), “Instructional sensitivity in vocational education”, Learning and Instruction, Vol. 53, pp. 21-33, https://doi.org/10.1016/j.learninstruc.2017.07.004.
[28] Doyle, W. (2000), Authenticity, Paper presented at the annual meeting of the American Educational Research Association (AERA), New Orleans.
[1] Dummert, S. (2021), “Employment prospects after completing vocational training in Germany from 2008-2014: a comprehensive analysis”, Journal of Vocational Education & Training, Vol. 73/3, pp. 367-391, https://doi.org/10.1080/13636820.2020.1715467.
[2] Hoidn, S. and V. Šťastný (2021), “Labour Market Success of Initial Vocational Education and Training Graduates: A Comparative Study of Three Education Systems in Central Europe”, Journal of Vocational Education & Training, Vol. 75/4, pp. 629-653, https://doi.org/10.1080/13636820.2021.1931946.
[22] Holtsch, D. et al. (2016), “Challenges of a cross-national computer-based test adaptation”, Empirical Research in Vocational Education and Training, Vol. 8/1, https://doi.org/10.1186/s40461-016-0043-y.
[12] Jordanski, G. (2019), “Kompetenzen für die digitalisierte Arbeit von morgen – Industriekaufleute”, In K. Wilbers (Hrsg.), Digitale Transformation kaufmännischer Bildung: Ausbildung in Industrie und Handel hinterfragt, pp. (S. 119–146) epubli GmbH, Berlin.
[31] Jürgen Baumert, W. (ed.) (2000), TIMSS/III Dritte Internationale Mathematik- und Naturwissenschaftsstudie — Mathematische und naturwissenschaftliche Bildung am Ende der Schullaufbahn, VS Verlag für Sozialwissenschaften Wiesbaden, https://doi.org/10.1007/978-3-322-83411-9.
[27] Kilpatrick, J., J. Swafford. and B. Findell (eds.) (2001), Adding It Up: Helping Children Learn Mathematics, Mathematics Learning Study Committee, National Research Council, Washington, D.C., https://doi.org/10.17226/9822.
[7] Klotz, V. (2015), Diagnostik beruflicher Kompetenzentwicklung: Eine wirtschaftsdidaktische Modellierung für die kaufmännische Domäne, Springer Fachmedien Wiesbaden; Springer Gabler.
[21] Marzano, Kendall (2007), The new taxonomy of educational objectives (2nd ed.), Corwin Press, California.
[29] Masters, G. (1982), “A rasch model for partial credit scoring”, Psychometrika, Vol. 47/2, pp. 149-174, https://doi.org/10.1007/bf02296272.
[18] OECD (2019), The Survey of Adult Skills : Reader’s Companion, Third Edition, OECD Skills Studies, OECD Publishing, Paris, https://doi.org/10.1787/f70238c7-en.
[17] OECD (2015), Adults, Computers and Problem Solving: What’s the Problem?, OECD Skills Studies, OECD Publishing, Paris, https://doi.org/10.1787/9789264236844-en.
[16] OECD (2012), Literacy, Numeracy and Problem Solving in Technology-Rich Environments: Framework for the OECD Survey of Adult Skills, OECD Publishing, Paris, https://doi.org/10.1787/9789264128859-en.
[15] OECD (1996), Literacy, Economy and Society: Results of the First International Adult Literacy Survey, OECD Publishing, Paris, https://www.digizeitschriften.de/de/dms/met/?ppn=ppn509215866_0049&dmdid=dmdlog61.
[4] OECD/Eurostat/UNESCO Institute for Statistics (2015), ISCED 2011 Operational Manual: Guidelines for Classifying National Education Programmes and Related Qualifications, OECD Publishing, Paris, https://doi.org/10.1787/9789264228368-en.
[19] Pellegrino, J., N. Chudowsky and R. Glaser (2001), Knowing What Students Know, National Academies Press, Washington, D.C., https://doi.org/10.17226/10019.
[10] Porter, M. (1985), Competitive Advantage: Creating and Sustaining Superior Performance, Free Press.
[25] Rauner, F. et al. (2009), Messen beruflicher Kompetenzen: Band I: Grundlagen und Konzeption des KOMET-Projektes, LIT Verlag.
[20] Rausch, A. et al. (2021), “Die web-basierte Bürosimulation LUCA – Funktionen, Einsatzmöglichkeiten und Forschungsausblick”, Zeitschrift für Berufs- und Wirtschaftspädagogik, Vol. 117/3, pp. 372-394, https://doi.org/10.25162/zbw-2021-0017.
[11] Rüegg-Stürm, J. and S. Grand (2020), Das St. Galler Management-Modell: Management in einer komplexen Welt (2., überarbeitete Auflage), Haupt Verlag, https://doi.org/10.36198/9783838554990.
[23] Sangmeister, J. et al. (2018), “Designing Competence Assessment in VET for a Digital Future”, in Ifenthaler, D. (ed.), Digital Workplace Learning (S. 65–92), Springer International Publishing, Cham, https://doi.org/10.1007/978-3-319-46215-8_5.
[13] Seeber, S. and J. Seifried (2019), “Herausforderungen und Entwicklungsperspektiven der beruflichen Bildung unter veränderten Rahmenbedingungen, Challenges and development prospects for vocational education and training in times of changing socioeconomic and technological conditions”, Zeitschrift für Erziehungswissenschaft, Vol. 22/3, pp. 485-508, https://doi.org/10.1007/s11618-019-00876-2.
[9] Wigdor, A. and B. Green Jr. (1991), Performance Assessment for the Workplace. Technical issues, 1(2), National Academies Press, Washington, D.C., https://doi.org/10.17226/1862.
[14] Williams, A. et al. (2017), “Systems thinking: A review of sustainability management research”, Journal of Cleaner Production, Vol. 148, pp. 866-881, https://doi.org/10.1016/j.jclepro.2017.02.002.
[30] Wilson, M. and K. Draney (2004), “Some Links Between Large‐Scale and Classroom Assessments: The Case of the BEAR Assessment System”, Yearbook of the National Society for the Study of Education, Vol. 103/2, pp. 132-154, https://doi.org/10.1111/j.1744-7984.2004.tb00051.x.
[6] Winther, E. (2010), Kompetenzmessung in der beruflichen Bildung, wbv Publikation, Bielefeld, https://doi.org/10.3278/6004148w.
[8] Winther, E. and V. Klotz (2016), “Berufliche Kompetenzniveaumodellierung : Eine Blaupause für die kaufmännische Domäne”, Unterrichtswissenschaft, Vol. 44/2, pp. 131-146.
Klotz, V., Winther, E. and Festner, D. (2015), “Modeling the Development of Vocational Competence: A Psychometric Model for Economic Domains”, Vocations and Learning, Vol. 8(3), pp. 247–268, https://doi.org/10.1007/s12186-015-9139-y.
Baumert, J., Bos, W. and Lehmann, R (2000), TIMSS/III – Dritte Internationale Mathematik und Naturwissenschaftsstudie – Mathematische und naturwissenschaftliche Bildung am Ende der Schullaufbahn; Band 1: Mathematische und naturwissenschaftliche Grundbildung am Ende der Pflichtschulzeit; Band 2: Mathematische und physikalische Kompetenzen am Ende der gymnasialen Oberstufe. Opladen: Leske + Budrich. [Rezension], in Tertium comparationis 6 (2000) 2, pp. 198-205, https://doi.org/10.25656/01:3366.
Wiggins, G. (1990), The Case for Authentic Assessment, https://doi.org/10.7275/ffb1-mm19.
Wilbers, K. (2019), “Kaufmännische Digitalkompetenzen als Ausgangspunkt der digitalen Transformation beruflicher Bildung”, in K. Wilbers (eds), Digitale Transformation kaufmännischer Bildung. Ausbildung in Industrie und Handel hinterfragt, pp. 11–72, epubli GmbH.