AI and the Future of Skills, Volume 1

9. Competence assessment in German vocational education and training

Britta Rüschoff
FOM University of Applied Sciences for Economics and Management

Abstract

This chapter reviews the methods of competence assessment in German vocational education and training (VET) and discusses their suitability for assessing the capabilities of artificial intelligence (AI) and robotics. It describes how vocational competences are defined, the instruments used to assess them and how VET examinations are developed and administered. In addition, it discusses the validity and reliability of assessment instruments in VET, providing concrete examples of assessments. Finally, the chapter indicates the advantages of using VET tests for assessing AI capabilities and concludes with several considerations for applying VET examinations on machines.

Introduction

Developments in artificial intelligence (AI) and robotics will have a substantial impact on future working environments. Technologies will partially or completely take over tasks previously performed by humans. Such developments will change the professional skills and competences required of future generations of workers.

Which professional tasks are likely to be partially or entirely performed by AI or robotics? How can AI and robotics be trained to perform these tasks? What methods and tools can be used to assess their performance, particularly in comparison with humans?

One approach to these questions is to review methods commonly used to train and assess competences in humans. These methods could then inform the training and assessment of AI and robotics. This chapter provides an overview of how competences are assessed in vocational education and training (VET) in Germany. It looks at how skills and competences are defined; methods and tools in use; and examples of tasks used in VET. Finally, it reflects on the suitability of these methods and tools to assess the capabilities of AI and robotics and offers recommendations.

Vocational education and training in Germany

Germany has 325 occupations that are state-recognised or deemed state-recognised under the Vocational Training Act (BBiG) or the Crafts Code (HwO) (BIBB, 2020[1]). Some occupations, such as in the medical field, are regulated in special laws (e.g. Nursing Act, Geriatric Care Act). Most vocational training programmes completed in Germany are based on the dual model (Baumgarten, 2020[2]). In these cases, the apprentices alternate between the vocational school and their respective training company with which they have made a contract for their training. In dual VET, framework curricula (Rahmenlehrpläne) regulate school-based vocational education. They provide a detailed overview over the learning fields and learning outcomes for each VET-occupation. In-company training is regulated by the Vocational Training Act (BMBF, 2019[3]) and the respective training regulations (Ausbildungsordnungen) for an occupation. Both the framework curricula and the training regulations are available for public inspection through the Standing Conference of the Ministers of Education and Cultural Affairs (KMK).

This chapter refers primarily to dual VET since it is the most common training model in Germany.1

Definitions of competence in German vocational education and training

Empirical and normative definitions of competence

The objective of VET is the acquisition of vocational competences. However, these vary on areas of application, and there is no uniform definition in VET (Koeppen et al., 2008[4]; Zlatkin-Troitschanskaia and Seidel, 2011[5]; Rüschoff, 2019[6]).This section briefly discusses definitions of vocational competences in different contexts to provide a framework for the illustrations of VET in Germany. A rough distinction can be made between empirical and normative definitions of competence.

Empirical definitions are applied particularly when competences are to be measured. In empirical contexts, vocational competences are frequently described as “context-specific cognitive performance dispositions, which in functional terms relate to situations and requirements in certain domains” (Klieme and Leutner, 2006[7]). Hence, they are context-specific dispositions for action (Weinert, 2001[8]; Klieme and Leutner, 2006[7]; Rüschoff, 2019[6]). Such approaches usually aim at developing standardised assessment instruments. These would capture individual facets or sub‑dimensions of vocational competence. In so doing, they draw quantifying conclusions about the respective (sub-) competences of an individual, e.g. about specific knowledge or skills (Nickolaus, 2018[9]).

Normative definitions of competence are used mostly in political and legislative debates when it comes to general educational objectives. Normative definitions may go well beyond the immediate work context. Thus, they might also consider non‑occupational (e.g. societal) contexts as possible areas to apply vocational competences (KMK, 1996, 2007, 2011, 2017[10]; Roth, 1971[11]). Such comprehensive definitions are almost impossible to capture empirically due to their extensive scope. However, they are not generally designed with the aim of empirical measurement. Rather, they serve as an orientation framework for policy making and the description of educational goals such as by the Standing Conference of the Ministers of Education and Cultural Affairs of the Länder in the Federal Republic of Germany (KMK, 1996, 2007, 2011, 2017[10]), the German Vocational Training Act (BMBF, 2019[3]), and the German Qualifications Framework for Lifelong Learning (DQR, 2011[12]; BMBF, 2013[13]).

This difference between normative and empirical definitions has an important implication. While normative approaches often describe rather broad competence dimensions, empirical approaches will break these dimensions down into sub-dimensions to make them accessible for measurement. This distinction becomes relevant in the sections that follow that present both the normative frameworks of vocational competence and examples of different instruments for assessing competence.

Dimensions of competence in vocational education and training

At the policy level, the dimensions and levels of competence in German VET are largely based on the German Qualifications Framework (DQR, 2011[12]; BMBF, 2013[13]). This, in turn, is based on the European Qualifications Framework (EQF) (Cedefop, 2020[14]).

The EQF offers a shared European reference framework to facilitate the recognition of qualifications across different countries and educational systems. It covers qualifications at all educational levels and in all education and training sub-systems. In so doing, it provides a comprehensive overview of the qualifications in the countries involved in its implementation.

When applied in individual countries, the EQF is implemented in National Qualifications Frameworks that consider country‑specific contexts and characteristics (BMBF, 2013[13]). The German Qualifications Framework, for example, presents a general taxonomy of competence dimensions that applies at all educational levels. It distinguishes between Professional Competence (Knowledge and Skills) and Personal Competence (Social Competence and Autonomy). It further distinguishes between eight educational levels – from pre-vocational training at Level 1 to a doctoral degree at Level 8 (BMBF, 2013[13]; DQR, 2011[12]).

The general competence dimensions remain the same at all educational levels. However, the required learning outcomes increase. Thus, the complexity with which graduates at each level need to master the competences also increases. German VET is located either at Level 3 for occupations that require two years of training or at Level 4 for occupations that require 3-3.5 years of training (DQR, 2011[12]); (BMBF, 2013[13]).

Table 9.1 depicts the overarching competence dimensions along with the required mastery of these competences at Levels 3 and 4. Table 9.2 depicts an example of how these competence dimensions are applied to Industrial Electricians at Level 3 of the German Qualifications Framework. It also describes Industrial Electricians’ learning outcomes for each competence dimension (BMBF, 2013[13]).

Table 9.1. Competence requirements of the German Qualifications Framework at Levels 3 and 4

Level	Requirements	Professional competence		Personal competence		Education (example)
		Knowledge Depth and breadth	Skill Instrumental and systemic skills; judgement	Social competence Team/ leadership skills, involvement, communication	Autonomy Autonomous responsibility; reflectiveness and learning competence
3	Being able to autonomously perform tasks that remain structured in some areas.	Having extended general or professional knowledge.	Having cognitive and practical skills for planning and processing tasks and evaluating them according to largely pre-determined criteria; transferring methods/results.	Working in a group and occasionally offering support; helping shape the working environment; presenting processes and results to the appropriate recipients.	Working autonomously and in less familiar contexts; appraising own actions and actions of others; requesting and selecting learning guidance.	VET (2 years)
4	Being able to autonomously plan and process tasks in fields that are subject to change.	Having deeper general or professional knowledge.	Having a broad spectrum of cognitive and practical skills to autonomously perform tasks and problem solving and to evaluate processes and results; considering alternative courses of action and effects with neighbouring areas; transferring methods/solutions.	Helping shape the work in a group and offer ongoing support; justifying processes and results; providing comprehensive communication on facts and circumstances.	Setting own learning and work objectives; reflecting on and assessing these objectives and taking responsibility for them.	VET (3-3.5 years)

Source: Adapted from the German Qualifications Framework (DQR, 2011[12]), pp. 7-15.

Table 9.2. Exemplary competence dimensions for Industrial Electricians (excerpt)

Level	Professional competence		Personal competence
	Knowledge Depth and breadth	Skill Instrumental and systemic skills; judgement	Social competence Team/Leadership skills, involvement, communication	Autonomy Autonomous responsibility; reflectiveness and learning competence
3	Industrial electricians have an understanding of the interaction between mathematical and natural science contents and safety, economic and business administration, and environmental aspects. They have acquired extensive specialist knowledge particularly in electrical engineering, installation techniques, information technology, measuring and control technology. Industrial electricians are in possession of extended specialist knowledge allowing them to: analyse electronic systems and test functions plan and implement electrical installations analyse and adapt control systems; roll out IT systems (…).”	Industrial electricians are in possession of cognitive and practical skills which enable them to process and connect mechanical components and equipment and to analyse electrical systems and test functions. They can assess work results and provide transfers of methods and solutions. They ... process, assemble and connect mechanical components and electrical manufacturing resources measure and analyse electrical functions and systems assess the safety of electrical installations and manufacturing resources install and configure IT systems (…).”	Industrial electricians are able to work in a team, provide mutual support, communicate correctly in technical language, help shape the learning and working environment, and present processes and results in a manner geared to their target group. They ... work predominantly in a team and communicate, in their professional activity, using correct technical language both in house and externally with other people apply work, time and learning planning methods plan tasks within the team and co‑ordinate them research, procure and evaluate information are able to present facts, draw up minutes, and use German and English technical terms (…).”	Industrial electricians are able to work independently. Particularly when dealing with current-carrying components they act responsibly and carefully. They... plan work processes and subsidiary tasks, taking into account economic and scheduling guidelines calculate and evaluate material and labour costs, record services performed apply customary requirements and take advantage of training opportunities recognise their own training requirements and take advantage of training opportunities (…).”

Source: German EQF-Referencing Report (BMBF, 2013[13]).

Applications of competence dimensions in practice

Comprehensive vocational competence dimensions, such as those in the German Qualifications Framework, must be broken into smaller sub-competences, both general and occupation-specific, to make them accessible for empirical assessment. General competences encompass a broad spectrum of basic skills related to mathematics, reading and writing or general problem solving, among others (Weinert, 2001[8]; Klotz and Winther, 2016[15]). Occupation‑specific competences are pertinent to a specific occupation or occupational area, for example, specific rules, principles and action patterns (Klotz and Winther, 2016[15]).

Rüschoff (2019[6]) conducted a systematic review on the methods of competence assessment in German VET. This included both instruments used within VET examinations and those developed for research. The results show that most available methods and tools, roughly 60% of the instruments covered in the review, are focused on the assessment of occupation-specific professional competences. These include commercial knowledge and skills in industrial clerks (Klotz and Winther, 2015[16]) or problem-solving skills (e.g. error analysis/troubleshooting) in car mechatronics (Abele et al., 2016[17]; Gschwendtner, Abele and Nickolaus, 2009[18]). Significantly fewer instruments, about 24%, were focused on general competences. Only 9% measured personal competences such as social or communicative competences (Rüschoff, 2019[6]).

Even if the professional competences and skills are clearly in the forefront, personal competences are equally relevant for most professions. The relative scarcity of instruments to assess social and communicative competences, then, does not indicate these competences are less important. Since these instruments are usually aligned with the framework curricula and training regulations, they typically emphasise professional aspects more than social and communicative aspects.

Yet, social and communicative skills will likely become more important for most professions, as employees will focus on more complex activities that require interdisciplinary exchange. This is especially the case with greater automation for routine tasks. The ability to communicate in a goal‑oriented manner with professionals from a variety of backgrounds will hence be an important skill in the future.

Examinations in German vocational education and training

Structure and grading of examinations

Final examinations in (dual) VET are regulated by the Vocational Training Act or the Crafts Code (BBiG, §37 – §50/ §31-§40a HwO) and organised by the chambers for the respective occupations. The chambers appoint examination boards (Prüfungsausschüsse) and conduct the examinations. Examination boards consist of at least three people who are qualified experts in the profession. They must also include representatives of employers and employees (in equal numbers) and at least one vocational schoolteacher. The examination board may also obtain qualified expert opinions from third parties (e.g. vocational schools).

The learning fields covered in the examinations are aligned with the framework curricula and training regulations for an occupation. Such an alignment and the explicit inclusion of occupational experts ensure that assessed competences and the occupational scenarios in the examinations correspond to industry practices and reflect workplace conditions.

Examinations are structured in different examination areas. They must cover between three to five areas with “economics and social studies” being obligatory. Each area defines competences that apprentices must demonstrate and specifies certain instruments to assess them. Examinations are commonly designed to yield a total of 100 points across all tasks and examination areas. A very good performance or grade 1.0 in the German system is awarded 100 points (BIBB, 2020[19]).

Aggregated results for all final examinations held by the Chamber of Industry and Commerce (roughly 300 000 annually) are available in their examination statistics. These statistics contain information on the examination areas of the individual occupations, the pass rate of the examinations and the distribution of grades and average scores for individual examination areas (IHK, 2021[20]).

Development of examination tasks

The examination tasks are developed by one of several entities:

• the examination board

• a task development committee put in place for a certain profession by the responsible chambers

• a committee appointed by a chamber for a specific occupation in a federal state or region to develop the examination tasks, often together with a (supra-regional) development office for examination tasks

• development offices for examination questions, for example the PAL (www.stuttgart.ihk24.de/pal) for industrial and technical professions, the ZPA (www.ihk-zpa.de) or AkA (www.ihk-aka.de) for commercial professions and the ZFA (www.zfamedien.de) for professions in the printing and media industry.

Development offices develop the examination tasks and carry out the examinations for a variety of different occupations. For example, the PAL develops examination tasks for 133 of the currently 325 recognised occupations in German VET (PAL, 2020[21]).

The development offices usually hold the copyright and the exploitation rights to the examination tasks. For examination tasks still in use, secrecy regulations are strict. Examination tasks stemming from previous years can in some cases be acquired for inspection or exam preparation from the development offices.

Types of examination tasks

Table 9.3 provides an overview of the different types of tasks. Tasks can be written, oral or practical, and either closed (e.g. written multiple choice) or open (e.g. practical work assignments). The respective examination board decides which tasks are used; not all types of tasks are used in every examination. However, certain tasks are related in that the content of one requires the completion of a previous one. This is because many examinations are case-related. They have different tasks but refer to the same underlying work-related scenario and the results build on one another.

The case-related approach makes it possible to depict complete action sequences. This, in turn, permits a more representative picture of the apprentices’ competences and skills in authentic work settings. For instance, apprentices might first complete a practical work assignment and then present and discuss the results. Table 9.4 depicts common and obligatory combinations of tasks.

Table 9.3. Types of assessment tasks in VET

Instruments	Description of the instruments	Evaluated dimension(s)
Practical tasks
Product sample	Apprentices craft a product typical for their profession, e.g. a wooden product.	The final product
Work sample	Apprentices carry out a single professional activity (e.g. the provision of a service).	Work procedure
Work task	Apprentices carry out a complex professional activity.	Final result of the task Work procedure
Work assignment	Apprentices carry out a typical assignment for their future profession (either in the company in which they receive their training or at the site of a customer).	Work procedure and/or Final result of the assignment
Oral tasks
Case-related expert discussion	Case-related expert discussions are based on a practical task that has either been completed beforehand or is described in detail as part of the examination. Apprentices may receive documents related to this practical task to prepare themselves and to be used during the discussion.	Understanding of backgrounds and contexts Methodical approach Quality of solutions Communication skills
Assignment-related expert discussion	Assignment-related expert discussions are based on a professional assignment (e.g. a work sample) that has been completed beforehand and discuss procedures, problem solving and solutions.	Understanding of backgrounds and contexts Methodical approach Quality of solutions
Situational expert discussion	Situational expert discussions are based on a work sample or a work task and discuss apprentices’ procedures, problem solving and solutions. Situational expert discussions are held during the completion of work samples or work tasks.	Understanding of backgrounds and contexts Methodical approach Quality of solutions
Conversation simulations	Conversation simulations are a type of role play in which apprentices engage in conversations that are typical for their future professions. This can, for example, be conversations with customers or patients.	Understanding of backgrounds and contexts Methodical approach Quality of solutions Communication skills Customer orientation
Presentations	Apprentices hold a presentation on, for example, a previously completed professional work task, a work assignment or a typical professional situation and respond to questions regarding their presentation.	Methodical approach Communication skills Formal criteria of the presentation
Written tasks
Written tasks	Tasks to be completed in writing are typical professional activities for a certain profession. This might cover drafting a business letter in commercial professions or a circuit diagram in technical professions.	Demonstration of professional knowledge Understanding of backgrounds and contexts Methodical approach Quality of solutions
Documenting with practice‑based documents	Documentations with practice-based documents are based on a previously completed professional work task, a created product or an assignment and documents the execution of this task in, for example, reports, work plans, operating instructions, etc.	Practice-based documentations are drawn on to support the evaluation of the work and/or the results

Source: Recommendation of the Main Committee of the Federal Institute for Vocational Education and Training (BIBB) on the structure and design of training regulations and examination requirements (BIBB, 2013[22]).

Table 9.4. Possible and obligatory combinations of tasks

Types of tasks	Can be combined with…	Must be combined with…
Practical tasks
Product sample	Documenting with practice-based documents or Presentation or Assignment-related expert discussion
Work sample	Documenting with practice-based documents or Assignment-related expert discussion or Situational expert discussion
Work task	Written tasks or Documenting with practice-based documents or Presentation or Assignment-related expert discussion or Situational expert discussion
Work assignment	Presentation	Documenting with practice-based documents Assignment-related expert discussion
Oral tasks
Case-related expert discussion
Assignment-related expert discussion		Product sample or Work sample or Work task or Work assignment
Situational expert discussion		Work sample or Work task
Conversation simulations
Presentations		Product sample or Work sample or Work task or Work assignment
Written tasks
Written tasks	Documenting with practice-based documents or Situational expert discussion or Work sample or Work task or Work assignment
Documenting with practice-based documents		Product sample or Work sample or Work task or Work assignment

Source: Recommendation of the Main Committee of the Federal Institute for Vocational Education and Training (BIBB) on the structure and design of training regulations and examination requirements (BIBB, 2013[22]).

Psychometric properties of assessment instruments in vocational education and training

Regarding the psychometric quality of these examination tasks, one of the advantages lies in their validity. The validity of an assessment refers to the extent to which a method or instrument can depict the intended target construct (e.g. vocational competence). This can be established in a variety of ways. For example, content validity addresses how well the tasks of a test cover the range of all possible tasks. Hence, it addresses how well the test represents the underlying construct to be assessed. The examinations aim to ensure content validity in two ways. First, they align examinations to the framework curricula and training regulations. Second, they draw on experts in the field develop and select tasks to reflect the behaviour and knowledge needed by apprentices in their occupation.

A recent systematic review provides a comprehensive overview of the psychometric properties of the methods and instruments developed or in use in German VET (Rüschoff, 2019[6]), both in the context of VET examinations and in research contexts. Most instruments included in the review ensured the validity of the instruments through establishment of their content validity. They did this, for instance, by aligning the instruments with curricula and training regulations and drawing on expert ratings from the industry.

Roughly 22% of the instruments were further subjected to a construct validation, primarily through one of two methods. On the one hand, their convergent validity was assessed (i.e. the relationship of the instrument to other instruments measuring the same construct). On the other, their discriminant validity was assessed (i.e. ensuring the developed instrument does not correlate [strongly] with validated instruments measuring different constructs). The review identified a scarcity of results on the predictive validity of the instruments. It is unclear whether findings regarding the predictive validity are entirely lacking or just not publicly available. Most instruments in the review showed good or acceptable reliability (Rüschoff, 2019[6]).

An analysis of the psychometric properties of the final examinations in commercial professions in German VET studied the final examinations of 1768 apprentices (Klotz and Winther, 2012[23]; Winther and Klotz, 2013[24]). The results suggest the examinations are clearly aligned with the requirements of an occupation. They also align with the practical training apprentices received before their examination.

However, the relative weights of the examination areas sometimes deviate. For instance, a large part of the curriculum in commercial professions refers to the goods and services domain (roughly half of the curriculum and one-third of the practical training). Yet, only about 20% of the examination relates to this content (Winther, 2011[25]; Winther and Klotz, 2013[26]).

The analyses also indicated that reliabilities of the tasks used in commercial final examinations are sufficiently high. However, reliabilities differed by apprentices’ ability levels. The examinations were especially reliable for test takers with average ability levels (with scores centred around the mean). They were somewhat less reliable for apprentices with either very high or very low ability levels (Winther and Klotz, 2013[26]). More detailed information on the psychometric properties of the examinations for individual professions is available to the respective chambers and is considered when tasks are to be reused.

Exemplary examination tasks for Advanced Manufacturing Technicians

Overall structure of the examination

The following examples provide an overview of the tasks used in an examination of Advanced Manufacturing Technicians at Level 4 of the German Qualifications Framework. The examples show excerpts of an examination held in the summer of 2017. All displayed tasks stem from the same examination. The tasks displayed were developed in Germany by the PAL (PAL, 2020[27]) and translated by the German American Chamber of Commerce of the Midwest (GACCMidwest, 2020[28]). To facilitate communication with an international readership, only the English translation is shown here.

The examination has a practical part, an oral part and a written part. As is common, the examination is case-related, meaning the different parts revolve around a typical professional scenario. In this case, they involve the construction and use of a disk separator. When completing the tasks, apprentices may use a mechanical and metal trades handbook, a collection of formulas, drawing tools and a non-programmable calculator with no communication with third parties. They are further provided with a set of technical drawings. Figure 9.1 depicts one of seven technical drawings that were part of the examination materials.

Figure 9.1. Example of a technical drawing forming part of the examination materials

© IHK Region Stuttgart und GACC Midwest, alle Rechte vorbehalten/all rights reserved.

Practical tasks

In the practical component of the examination, apprentices demonstrate professional skills through the completion of a sequence of interrelated tasks (e.g. producing a functional module of a disk separator). Apprentices are provided with a functional description of the module to be produced, the setup requirements and the necessary drawings. They have 6.5 hours to complete the assignment. Their performance is evaluated according to pre-determined evaluation criteria on every sub-task. Figure 9.2 shows an excerpt of the assignment as presented to apprentices during the examination.

Oral tasks

The oral part of the examination consists of situational discussions. During the practical assignment (i.e. while producing a functional module of a disk separator), apprentices will be asked questions related to the completion of the assignment. They are expected to provide short answers that show they can explain professional issues and hold situational expert discussions.

Figure 9.2. Excerpt of a work assignment for Advanced Manufacturing Technicians

© IHK Region Stuttgart und GACC Midwest, alle Rechte vorbehalten/all rights reserved.

Written tasks

The written part of the examination has two parts. Part A consists of 23 multiple-choice questions of which 6 are mandatory and 3 are optional. Apprentices can decide which three questions they wish to skip. Part B consists of eight short open questions that all need to be answered. Both parts A and B refer to the technical drawings provided beforehand (see Figure 9.1). Apprentices have 90 minutes to complete the two parts. Figure 9.3 provides examples of multiple-choice questions in part A. Figure 9.4 provides examples of short open questions in part B.

Figure 9.3. Examples of multiple-choice questions used in examinations for Advanced Manufacturing Technicians

© IHK Region Stuttgart und GACC Midwest, alle Rechte vorbehalten/all rights reserved.

Figure 9.4. Example of a short open question used in examinations for Advanced Manufacturing Technicians

© IHK Region Stuttgart und GACC Midwest, alle Rechte vorbehalten/all rights reserved.

Evaluation and scoring of the results

The practical assignment is scored in three areas:

• performance (testing the function of the model, control technology, visual inspection of the mechanical module and dimensional inspection)

• inspection (based on a test protocol)

• the oral situational discussion (which is deemed part of the work assignment as it is held during the assignment).

Each of the three areas can yield a maximum of 100 points (with points already awarded and weighted in intermediate steps). In the final evaluation of the work assignment, a maximum of 100 points can be attained. The breakdown of points awarded is 85% for the performance phase; 10% for the test protocol; and 5% for the discussion phase. In the written part of the examination, each multiple-choice question yields 1 point, resulting in a maximum number of 20 points for part A (20 of 23 multiple-choice questions must be answered). Part B consists of eight short-answer questions that must all be answered. Each short‑answer question yields 10 points, resulting in a maximum number of 80 points for part B. The divisors of parts A and B are 0.4 and 1.6, respectively (i.e. a maximum of 20/0.4 = 50 points for part A and a maximum of 80/1.6 = 50 points for part B). This gives parts A and B each 50 points or a weighting of 50% each. Together, apprentices can get 100 points for the written part of the examination. The practical and written parts of the examination each count for half of the final grade.

Suitability of contemporary vocational education and training tests to assess capabilities of artificial intelligence and robotics

Advantages of performance-based assessments

The above review of the psychometric properties of the examination tasks in VET suggests these tasks are suitable for assessing vocational competences in humans. Are they equally suitable to assess vocational competences in machines? Addressing this question requires first understanding what the tasks reviewed above do and do not measure and how they differ from other assessment methods and instruments.

The tasks presented above are action-oriented and inherently practical, which are advantages. With work-related tasks, examiners can observe performance on the same behaviours (or very similar behaviours) that apprentices will need to demonstrate in their subsequent professions. Performance on these tasks is thus likely a direct predictor of later job performance.

Although these tasks draw on observable task-related behaviour (e.g. crafting a certain product), they indicate more than task performance. The examinations aim to use action-oriented tasks to draw inferences on an underlying characteristic of the apprentices such as the competences (e.g. knowledge or skills) that enable apprentices to exert this task-related behaviour. Yet, in action-oriented examinations, the tasks themselves already have considerable meaning. They are not only abstract indicators of the underlying construct but also demonstrate the target behaviour. This distinguishes the action-oriented tasks used in VET from other tests seeking to assess underlying characteristics or constructs such as tests of general mental ability. Tests of general mental ability are likewise used to predict job performance and occupational attainments in humans (Schmidt and Hunter, 2004[29]; Bertua, Anderson and Salgado, 2005[30]). However, they do not draw on work-related tasks to make inferences about future professional performance. Rather, they aim to assess general mental abilities that are expected to manifest in various domains, including future job performance. For example, individuals who score high on a general mental ability test have proven to be able to solve complex test items (observable behaviour). This observable behaviour is theoretically related to the concept of general mental ability. This, in turn, is assumed to relate to performance on other complex problem-solving tasks in real-life situations.

Advantages of performance-based assessments in artificial intelligence and robotics

The results that hold for humans may not apply in the assessment of the capabilities of AI and robotics. Questions in a general mental ability test are only indicators of general intelligence. The resulting score, then, is not an end in itself. For example, individuals will likely attain a high score on a general mental ability test if they memorise the answers first. However, this score will not measure underlying general mental ability. Rather, it will represent their ability to memorise answers to a test. As the result would no longer measure general intelligence, it would also not predict outcomes beyond the immediate test. Hence, it would be unrelated to outcomes usually associated with higher general mental ability.

In humans, the practice described above would be considered cheating on a test. When training an AI, algorithms are trained by knowing the answers in advance, at least for specialised/weak AI. Through repeated exposures to certain tasks or challenges, the AI is trained to solve tasks with increasing accuracy and efficiency. As a result, the AI’s performance will be a measure of its ability to learn the answers (or answering logics) to this type of test rather than of the presumed underlying characteristics (e.g. mental ability). Like an individual learning the answers to a test by heart, the resulting score will not be a measure of the intended underlying construct. Therefore, it will most likely no longer predict other outcomes typically associated with general mental ability because such measures gain their predictive value through links with other complex problem-solving skills.

In humans, research has consistently demonstrated these links. However, to successfully apply their general mental ability to other problem-solving tasks, humans likely also draw on other sources of information. These could be basic knowledge about material objects and their properties, events and their effects, or beliefs and desires – what computer science calls “common sense knowledge” (McCarthy, 1989[31]; Miller, 2019[32]). Whereas it can be assumed that humans possess this common sense knowledge, this is not (yet) the case in AI. Consequently, an AI trained on such tests might only be able to solve the questions on this type of test without allowing for inferences about their (job) performance beyond the tests.

The tasks used in VET assess apprentices’ underlying competences through authentic professional tasks and requirements. In professional contexts, such work-related tasks are widely recognised as diagnostic instruments and are used successfully for assessment and selection (Hunter and Hunter, 1984[33]; Roth, Bobko and McFarland, 2005[34]; Lievens and Patterson, 2011[35]). Although practical work-related tasks also intend to infer underlying competences from task performance, the tasks themselves are directly related to vocation-specific requirements. In this way, they assess not only underlying competences but also vocation-specific task performance.

Thus, training an AI to learn and reproduce specific procedures to complete a professional task or to solve a work-related problem would train it to perform vocation‑specific tasks efficiently. However, performance on one task cannot or can hardly be generalised to performance on a different task or an unknown context. The generalisability of measures of this type of task performance are thus narrow. Still, it would be certain that a person (or AI) can successfully complete this specific task.

Conclusions

Instruments used in German VET offer several advantages that could be useful for training and assessing the capabilities of AI and robotics:

• Reference to labour market requirements

Measurements used in VET are aligned with the framework curricula and training regulations, and fall under the responsibility of the respective chambers. They are thus developed together with industry and reflect authentic labour market requirements.

• Performance-based measurements

Measurements used in VET are performance-based. Performance-based measures capture competences through concrete professional behaviours. The assessed target behaviour has immediate relevance to exercise of the profession.

• Interpretability

Competence assessments draw strongly on psychometric procedures and aim to capture underlying competences. However, assessing performance on concrete vocational tasks relies far less on theoretical assumptions regarding underlying psychological constructs than other assessment approaches. Thus, in performance-based assessments, a “successfully completed” task is meaningful in itself and there is no need to draw inference on any underlying competences that enabled its completion.

• Ongoing development process and topicality

The respective chambers constantly develop and evaluate examination tasks through ongoing examinations. Hence, they exist for a large number of different occupations.

In sum, training and assessing AI and robotics based on concrete work-related tasks might yield scientifically interesting results. Moreover, due to the strong practical orientation of the tasks, it can provide relevant insights for the industry when anticipating the capabilities of AI and robotics. Industry can then change the educational and skill requirements of their future workforce accordingly.

Recommendations

When seeking to select tasks to train and assess the capabilities of AI and robotics, the following points should be considered in this order of sequence:

• Choose occupations that are subject to similar regulatory structures

In German VET, there are different responsible authorities and regulations for different types of occupations and training. It is advisable to start with occupations that fall under similar legislation, so that the requirements for the occupations and the examination structures used are largely comparable. The majority of occupations are regulated by the Vocational Training Act (BBiG) or the Crafts Code (HwO).

• Choose occupational areas that represent a broad spectrum of the working world

Choose both industrial-technical and commercial professions, as well as craft trades and possibly health professions to represent as broad a spectrum of the working world as possible. Within these areas, a variety of occupations should be chosen that require different competences.

• Choose common occupations

The objective is to explore the future role of AI and robotics in the labour market. The most meaningful and relevant information will therefore be obtained by looking at occupations prevalent in today’s labour market. In addition, access to testing instruments will be easier for more common occupations than for less common ones. Statistics on the number of apprentices per occupation and year are publicly available in Germany (BIBB, 2019[36]).

Against the background of representing a broad spectrum of the working world, other occupations with more diverse competence requirements can be selected if there is too much overlap between the most common occupations.

• Cover different competence domains

Initially, one competence domain (e.g. skills or knowledge) can be chosen as a starting point. Ultimately, all relevant domains (i.e. skills, knowledge, social competences and autonomy) should be covered to give a comprehensive picture of AI and robotics capabilities.

• Consider different types of tasks

All types of tasks (i.e. practical tasks, written tasks and oral tasks) should be covered to gain a comprehensive picture of the capabilities of AI and robotics. Initially, only one type of task could be chosen as a starting point. The different types of tasks in the assessment of relevant competences are complementary. Consequently, all types of tasks should ultimately be considered to provide a comprehensive impression of the capabilities of AI and robotics.

• Select individual tasks to be used

As the last step, individual tasks to be used must be selected. The chambers, or the bodies commissioned by the chambers to develop the examination tasks, hold the rights to the examination materials. Therefore, materials should be obtained through the chambers.

As some occupations stemming from the same occupational areas are also somewhat similar, they may overlap in their test instruments (e.g. in different sales occupations or different types of mechanic occupations). In such cases, tasks used across occupations within a field should be prioritised.

References

[17] Abele, S. et al. (2016), “Berufsfachliche Kompetenzen von Kfz-Mechatronikern – Messverfahren, Kompetenzdimensionen und erzielte Leistungen (KOKO Kfz) [“Occupational Competencies of Mechatronics Technicians – Measurement, Dimensions and Achievement Goals”]”, Technologiebasierte Kompetenzmessung in der beruflichen Bildung: Ergebnisse aus der BMBF-Förderinitiative ASCOT, Bertelsmann.

[2] Baumgarten, C. (2020), Vocational Education and Training: Overview of the Professional Qualification Opportunites in Germany, German Office for International Cooperation in Vocational Education and Training, Bonn, https://www.govet.international/dokumente/pdf/vocational_education_and_training_web.pdf.

[30] Bertua, C., N. Anderson and J. Salgado (2005), “The predictive validity of cognitive ability tests: A UK meta-analysis”, Journal of Occupational and Organizational Psychology, Vol. 3, pp. 387-409, http://dx.doi.org/10.1348/096317905X26994.

[19] BIBB (2020), Richtlinie des Hauptausschusses des Bundesinstituts für Berufsbildung: Musterprüfungsordnung für die Durchführung von Abschluss- und Umschulungsprüfungen, ["Guidelines of the Committee of the Federal Institute for Vocational Education and Training: Recommended Regulations for Final Exams and Exams for Entering Retraining"], Bundesanzeiger Amtlicher Teil (BAnz AT 21.12.2020 S3), https://www.bibb.de/dokumente/pdf/HA120.pdf.

[1] BIBB (2020), Verzeichnis der anerkannten Ausbildungsberufe 2020, ["Directory of Recognised Occupations in Vocational Education and Training"], https://www.bibb.de/veroeffentlichungen/de/publication/show/16754.

[36] BIBB (2019), Rangliste 2019 der Ausbildungsberufe nach Anzahl der Neuabschlüsse, ["Ranking of Apprenticeships sccording to the Number of New Degrees 2019"], https://www.bibb.de/de/103962.php.

[22] BIBB (2013), Empfehlung des Hauptausschusses des Bundesinstituts für Berufsbildung (BIBB) zur Struktur und Gestaltung von Ausbildungsordnungen (HA 158), ["Recommendations of the Committee of the Federal Institute for Vocational Education and Training for the Structure and Design and Training Regulation Frameworks"], Bundesanzeiger Amtlicher Teil (BAnz AT 13.01.2014 S1), http://www.bibb.de/de/32327.htm.

[3] BMBF (2019), Berufsbildungsgesetz - BBiG (The new Vocational Training Act), Bundesministerium für Bildung und Forschung/Federal Ministry of Education and Research, Bonn, https://www.bmbf.de/upload_filestore/pub/The_new_Vocational_Training_Act.pdf.

[37] BMBF (2019), Report on Vocational Education and Training 2019, Bundesministerium für Bildung und Forschung/Federal Ministry of Education and Research, Bonn, https://www.bmbf.de/upload_filestore/pub/Berufsbildungsbericht_2019_englisch.pdf.

[13] BMBF (2013), Deutscher Qualifikationsrahmen für lebenslanges Lernen [German EQR Referencing Report], Bundesministerium für Bildung und Forschung/Federal Ministry of Education and Research, Bonn, https://www.dqr.de/media/content/German_EQF_Referencing_Report.pdf (accessed on 24 August 2020).

[14] Cedefop (2020), “European qualifications framework. Initial vocational education and training: Focus on qualifications at levels 3 and 4”, Research Paper, No. 77, Publications Office of the European Union, Luxembourg, http://data.europa.eu/doi/10.2801/114528.

[12] DQR (2011), Original Title in German [The German Qualifications Framework for Lifelong Learning adopted by the “German Qualifications Framework Working Group“], Willkommen auf der Internetseite zum Deutschen Qualifikationsrahmen, http://www.dqr.de/media/content/The_German_Qualifications_Framework_for_Lifelong_Learning.pdf (accessed on 24 August 2020).

[28] GACCMidwest (2020), “German American Chamber of Commerce in the Midwest”, webpage, https://www.gaccmidwest.org/ (accessed on 8 September 2020).

[18] Gschwendtner, T., S. Abele and R. Nickolaus (2009), “Computersimulierte Arbeitsproben: Eine Validierungsstudie am Beispiel der Fehlerdiagnoseleistungen von Kfz-Mechatronikern [Computer-simulated Work Samples: A Validation Study using the Example of Failure Detection by Mechatronics Technicians]”, Zeitschrift für Berufs- und Wirtschaftspädagogik, Vol. 4, pp. 557–578.

[33] Hunter, J. and R. Hunter (1984), “Validity and utility of alternative predictors of job performance”, Psychological Bulletin, Vol. 96/1, pp. 72–98, https://doi.org/10.1037/0033-2909.96.1.72.

[20] IHK (2021), Prüfungsstatistik der Industrie- und Handelskammer, http://pes.ihk.de/Berufsauswahl.cfm.

[7] Klieme, E. and D. Leutner (2006), “Kompetenzmodelle zur Erfassung individueller Lernergebnisse und zur Bilanzierung von Bildungsprozessen. Beschreibung eines neu eingerichteten Schwerpunktprogramms der DFG”, Zeitschrift für Pädagogik, ["Competence Models for Measuring Individual Learning Outcomes and Educational Processes. Description of a Programme of the German Research Foundation"], pp. 876–903.

[15] Klotz, V. and E. Winther (2016), “Zur Entwicklung domänenverbundenerund domänenspezifischer Kompetenz im Ausbildungsverlauf: Eine Analyse für die kaufmännische Domäne”, Zeitschrift für Erzeihungswissenschaft, ["On the Development of Domain-related and Domain-specific Competences in Vocational Education. Analysis in the Commercial Domain"], pp. 765-782, http://dx.doi.org/10.1007/s11618-016-0687-1.

[16] Klotz, V. and E. Winther (2015), “Kaufmännische Kompetenz im Ausbildungsverlauf – Befunde einer pseudolängsschnittlichen Studie [“Commercial Competence in the Course of Vocational Training - Findings of a Pseudo-longitudinal Study”]”, Empirische Pädagogik, Vol. 29/1, pp. 61-83.

[23] Klotz, V. and E. Winther (2012), “Kompetenzmessung in der kaufmännischen Berufsausbildung: Zwischen Prozessorientierung und Fachbezug: Eine Analyse der aktuellen Prüfungspraxis”, Berufs- und Wirtschaftspädagogik, ["Competence Measurement in Vocational Training in Commerce. Between Process Orientation and Field Relevance: Analysis of Current Examination Procedures"], pp. 1-16, http://www.bwpat.de/ausgabe22/klotz_winther_bwpat22.pd.

[10] KMK (1996, 2007, 2011, 2017), “Handreichung für die Erarbeitung von Rahmenlehrplänen der KMK für den berufsbezogenen Unterricht in der Berufsschule und ihre Abstimmung mit Ausbildungsordnungen des Bundes für anerkannte Ausbildungsberufe”, Ständige Konferenz der Kultusminister der Länder in der Bundesrepublik Deutschland, ["Handout for the Development of Framework Curricula of the KMK for Job-related Teaching in Vocational Schools and their Coordination with the Federal Training Regulations for Recognised Training Occupations"].

[4] Koeppen, K. et al. (2008), “Current issues in competence modeling and assessment”, Zeitschrift Für Psychologie, Vol. 2, pp. 61-73, http://dx.doi.org/10.1027/0044-3409.216.2.61.

[35] Lievens, F. and F. Patterson (2011), “The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection”, Journal of Applied Psychology, Vol. 96/5, pp. 927-940, https://doi.org/10.1037/a0023496.

[31] McCarthy, J. (1989), “Artificial intelligence, logic and formalizing common sense”, in Thomason, R. (ed.), Philosophical Logic and Artificial Intelligence, Springer, Dordrecht, https://doi.org/10.1007/978-94-009-2448-2_6.

[32] Miller, T. (2019), “Explanation in artificial intelligence: Insights from the social sciences”, Artificial Intelligence, Vol. 267, pp. 1-38, http://dx.doi.org/doi:10.1016/j.artint.2018.07.007.

[9] Nickolaus, R. (2018), “Kompetenzmodellierung in der beruflichen Bildung – eine Zwischenbilanz (Competency modeling in vocational training - an interim balance)”, in Schlicht, J. and U. Moschner (eds.), Berufliche Bildung an der Grenze zwischen Wirtschaft und Pädagogik, Springer VS, https://doi.org/10.1007/978-3-658-18548-0_14.

[21] PAL (2020), Jahresbericht 2019 [“Annual Report 2019”], Prüfungsaufgaben- und Lehrmittelentwicklungsstelle der IHK Region Stuttgart, https://www.stuttgart.ihk24.de/blueprint/servlet/resource/blob/4391022/cd49a986c985f428a9716ea48bb06c08/pal-2019-jahresbericht-data.pdf.

[27] PAL (2020), “PAL-Prüfungsbücher [“PAL-Examinations”]”, webpage, https://www.stuttgart.ihk24.de/pal/publikationen/buecher/pal-pruefungsbuecher-liste-3811836 (accessed on 8 September 2020).

[11] Roth, H. (1971), Pädagogische Anthropologie – Band 2. Entwicklung und Erziehung, ["Educational Anthropology - Volume 2. Development and Nurture"], Hermann Schroedel Verlag, Braunschweig.

[34] Roth, P., P. Bobko and L. McFarland (2005), “A meta-analytic analysis of work sample test validity: Updating and integrating some classic literature”, Personnel Psychology, Vol. 58/4, pp. 1009-1037, http://dx.doi.org/10.1111/j.1744-6570.2005.00714.x.

[6] Rüschoff, B. (2019), Methoden der Kompetenzerfassung in der beruflichen Erstausbildung in Deutschland: Eine systematische Überblicksstudie, ["Competency Assessment Methods in Initial Vocational Training in Germany: A Systematic Overview Study"], Bundesinstitut für Berufsbildung, City.

[29] Schmidt, F. and J. Hunter (2004), “General mental ability in the world of work: Occupational attainment”, Journal of Personality and Social Psychology, Vol. 1, pp. 162–173, http://dx.doi.org/10.1037/0022-3514.86.1.162.

[8] Weinert, F. (2001), Vergleichende Leistungsmessung in Schulen – eine umstrittene Selbstverständlichkeit [“Comparative Performance Measurement in Schools - a Controversial Matter of Course”], Weinert, F.E. (ed.), Publisher, City.

[25] Winther, E. (2011), “Kompetenzorientierte Assessments in der beruflichen Bildung–Am Beispiel der Ausbildung vonIndustriekaufleuten [“Competence-oriented Assessments in Vocational Training - Using the Example of the Training of Industrial Clerks”]”, Zeitschrift für Berufs- und Wirtschaftspädagogik, Vol. 107/1, pp. 33–54.

[24] Winther, E. and V. Klotz (2013), Measurement of vocational competences: An analysis of the structure and reliability of current assessment practices in economic domains, http://www.ervet-journal.com/content/5/1/2.

[26] Winther, E. and V. Klotz (2013), “Measurement of vocational competences: An analysis of the structure and reliability of current assessment practices in economic domains”, Empirical Research in Vocational Education & Training, Vol. 5/2, http://www.ervet-journal.com/content/5/1/2.

[5] Zlatkin-Troitschanskaia, O. and J. Seidel (2011), “Kompetenz und ihre Erfassung – das neue ’Theorie-Empirie Problem’ der empirischen Bildungsforschung? [“Competences and their Assessment – a New Problem between Theory and Empiricism in Empirical Education Research”]”, in Zlatkin-Troitschanskaia, O. and J. Seidel (eds.), Stationen empirischer Bildungsforschung: Traditionslinien und Perspektiven, Springer, VS Verlag für Sozialwissenschaften.

Note

← 1.  Other forms of training include full-time school-based training. For a more detailed overview of the overall German VET system, see the German Federal Ministry of Education and Research (BMBF, 2019[37]) or the German Office for International Cooperation in Vocational Education and Training (GOVET; www.govet.international).