Increasing Adult Learning Participation

2. Lessons from successful reforms

Abstract

While much has been written about the need for reform, it is less clear how adult learning participation can be increased in practice. Many good ideas struggle to translate into real change on the ground, as they get stuck in the reality of policy implementation. To understand what makes a successful adult learning reform, this chapter synthesises lessons from countries that have increased learning participation in the past 15 years and the reforms they implemented. It investigates what other countries can learn from the nature of the reforms, how they were designed, funded and implemented, as well as how policy learning took place.

Introduction

Perfect policies do not exist, yet some policies are more successful than others in achieving desired outcomes. These policies are often characterised by a common set of good practices throughout the policy cycle. This includes a well-conceived policy design with the involvement of key stakeholders and strong implementation mechanisms that are subject to robust quality assurance. It also includes the set-up of regular monitoring and evaluation mechanisms, insights of which are used to improve the policy over time. These guiding principles hold true for policies seeking to increase participation in adult learning.

This chapter presents results of a comparative analysis of the seventeen adult learning reforms in Austria, Estonia, Hungary, Italy, the Netherlands and Singapore, selected based on the process outlined in Chapter 1. It draws out cross-country patterns relating to: i) the content of reforms; ii) funding of reforms, iii) reform implementation; and iv) policy learning. Based on the analyses it highlights lessons for other countries seeking to increase adult learning participation.

Learning from the content of reforms

As discussed in Chapter 1, the reforms included in this study cover a wide variety of measures, with diverse aims and objectives. At the most general level, they include measures that could reasonably have contributed to the observed increase in adult learning participation in the six countries. Some patterns can be identified with regards to the type of training each reform focuses on, i.e. formal, non-formal and or informal, its target group, and the mechanisms by which it aims to increase learning participation.

Type of training

As discussed in Box 1.1 of Chapter 1, learning activities are generally classified in three types, i.e. formal, non-formal or informal learning. This section discusses to which extent the selected policies address the different types of training.

As a set, the reforms selected in each country typically address both formal and non-formal education and training. No country focuses on one type of training across all selected reforms. For example, SkillsFuture Credit (Singapore) can be used for any type of training, whether formal (basic education, VET, or higher education) or non-formal (e.g.  SkillsFuture Series). In Austria, the focus of the Expansion of Training-related ALMPs lies on training provided by the PES, which is both formal and non-formal in nature, whereas much of the Initiative for Adult Education focuses specifically on formal second-chance education. In Hungary, the reform to obtain a Free Second Vocational Degree focuses on increasing participation in formal VET, while the Basic Skill Courses are non-formal.

In the vast majority of selected countries, none of the policies under review directly target informal learning. The Hungarian Open Learning Centres are an exception, in the sense that the reform stimulates informal learning by allowing people to use computers freely in the centres to surf the net, or practice and learn on their own (even when there are no classes). Some other reforms under review indirectly incentivise participation in informal learning by formally recognising it. For instance, the Adult Education Centres in Italy provide a system for the recognition of informal learning that is then counted towards the acquisition of formal (basic) qualifications. The Dutch Training Vouchers and Sector Plans can be used to finance formal recognition of prior learning (including, but not limited to, informal learning) through ‘experience certificates’, which were introduced in 2005 as a way to obtain formal recognition of prior learning.

There are two reasons that may explain why the reforms under review typically focus on non-formal and formal learning, but not informal learning. First, the policies were selected based on their potential contribution to the observed increase in participation, i.e. formal/non-formal education and training. Second, participation in (non-)formal learning is better documented than informal learning, for example through existing surveys on adult learning, which makes it easier for policy makers to target and monitor these types of learning. This notwithstanding, it is important to stress the importance of informal learning and its recognition, as it constitutes the vast part of learning in adult age and in the workplace in particular (Fialho, Quintini and Vandeweyer, 2019[1]).

Target groups

Policy-makers seeking to increase adult participation in education and training must consider if this is best achieved by targeting the entire adult population, or by focusing on more specific under-represented groups. Table 2.1 provides an overview of which groups are targeted by the selected reforms, namely all adults, the employed, unemployed, the low skilled or low qualified, or other more specific target groups.

Table 2.1. Target groups of the reforms under review

Country	Reform	Target groups
		All adults*	Employed adults	Unemployed adults	Low-skilled / low qualified	Other
AUT	Expansion of ALMPs
	Initiative for Adult Education
	Paid Educational Leave
EST	Expansion of ALMPs		from 2017		.
	Lifelong-Learning Strategy				from 2014	Adults with outdated skills and qualifications
	State-Commissioned Short Courses			from 2010	from 2011
HUN	Free Second Vocational Degree
	Basic Skill Courses					Public workers (2013‑2015)
	Open Learning Centres				2009‑2015	Disadvantaged adults (incl. those at risk of losing job, young or older adults, Roma)
ITA	Adult Education Centres
	Training Funds
NLD	Network Training					Unemployed aged 55‑63 (2013) Unemployed aged 50‑63 (2014‑2017)
	Training Vouchers
	Sector Plans
SGP	SkillsFuture Credit	(incl. retirees)				Singaporeans aged 25 and above
	SkillsFuture Mid-Career Enhanced Subsidy					Singaporeans aged 40+
	SkillsFuture Series	(incl. retirees)				Singaporeans

Source: OECD elaboration based on expert interviews and literature review.

Note: * Includes those who are inactive on the labour market (e.g. homemakers or students). Target groups do not include retirees, unless stated otherwise.

The target groups of the selected reforms vary from very specific to very broad. While the Sector Plans in the Netherlands, for example, focus on 50+ year-old unemployed workers which represent about 1% of the 25‑64 year-old population (see Chapter 3), other measures aim to engage the entire adult population including retirees (e.g. SkillsFuture Credit in Singapore).

However, the majority of reforms have a relatively broad target group, at least when they are first introduced, which includes either the entire (working) population, or adults with low skills or low qualification levels (see Chapter 3 for a discussion on the target group sizes in each country). However, these programmes often remain open to anyone who is willing to participate, irrespective of skills or educational level. For example, although the Hungarian Open Learning Centres target adults in disadvantaged situations (e.g. adults with low skills or the unemployed), in practice, they are open to anyone.

Where reforms target specific sub-groups of the population, they typically focus on the unemployed. It should be noted that targeting the unemployed implies that the size of the target group fluctuates with the business cycle. Several policies introduced a focus on a more specific target group at a later stage of the policy implementation.

Mechanisms that increase learning participation

There are many reasons why adults do not participate in education and training, including shortage of time (either work- or family-related), lack of financial resources, lack of prerequisites, lack of employer support, and that the training is provided at an inconvenient time or place (OECD, 2019[2]). Data from the 2016 Adult Education Survey show that people may face multiple participation barriers at the same time. Shortage of time, a lack of financial resources, or family reasons are the most important reason for non-participation, according to AES data. Using data from the Survey of Adult Skills (PIAAC), Figure 2.1 shows that these three reasons are also the most important reasons for non-participation in OECD countries. On average in the six countries under review, the share of individuals who did not participate in training due to a work‑related shortage of time is slightly higher compared to the OECD average (34% vs. 29%).

Analysing the barriers to training participation the policies address, i.e. the mechanism by which the policy aims to increase adult learning participation, is an important step in understanding how the selected policies may affect education and training participation.

Figure 2.1. Barriers to participation in adult learning

Note: Average of OECD countries participating in PIAAC.

Source: PIAAC data (2012, 2015, 2018).

Table 2.2 shows which of these participation barriers are addressed by the policies under review. It is interesting to note that, within each of the selected countries, the combination of the selected reforms address several participation barriers at the same time. For example, the Dutch Training Vouchers directly target the barrier of a lack of financial resources by covering 100% of the training costs up to EUR 1 000. Additionally, they indirectly address the barriers of shortage of time and lack of prerequisites, because the vouchers are intended to be used for relatively short courses and can be used to fund the recognition of prior learning since 2014.

This stresses the need for comprehensive approaches when tackling adult learning participation, because it may be the combination of reforms, rather than each reform in isolation, that contributed to the increase of adult learning participation in a country.

Table 2.2. Participation barriers addressed by the reforms under review

Country	Reform	Shortage of time*	Lack of financial resources	Lack of prerequisites	Lack of employer support	Other barriers addressed
AUT	Expansion of ALMPs					Not being part of the target group
	Initiative for Adult Education
	Paid Educational Leave
EST	Expansion of ALMPs					Not being part of the target group
	Lifelong-Learning Strategy					Not knowing which training to follow; Could not find any training of interest
	State-Commissioned Short Courses					Not being part of the target group; could not find any training of interest
HUN	Free Second Vocational Degree					Not being part of the target group; could not find any training of interest
	Basic Skill Courses					Could not find any training of interest
	Open Learning Centres					Could not find any training of interest
ITA	Adult Education Centres
	Training Funds
NLD	Network Training					Not being part of the target group; Could not find any training of interest
	Training Vouchers					Not being part of the target group
	Sector Plans
SGP	SkillsFuture Credit					Not being part of the target group
	SkillsFuture Mid-Career Enhanced Subsidy
	SkillsFuture Series					Could not find any training of interest

Source: OECD elaboration based on expert interviews and literature review. * This category also includes the barrier of an inconvenient time or place.

Shortage of time

Not having enough time to participate in training (either due to work or family reasons) is the most frequently named reason for non-participation in learning; in total 44% of the adult population gives this as the most important reason for non-participation (Figure 2.1). Reforms that can address this barrier are the introduction or expansion of statutory training leave, short/modular courses or part-time education, as well as flexible study options, such as online and evening classes. Although “shortage of time” and “inconvenient time or place” are independent barriers to training participation, the policies that address them can be similar. For example, by providing paid education and training leave, the barrier of lack of time is addressed directly. Yet, this also improves the opportunities for people to participate in training that takes place during working hours or further away, i.e. a training which might otherwise be considered to be given at an inconvenient time or place.

Almost all successful reforms included in this review address the issue of shortage of time in one way or another. Austria’s Paid Educational Leave reform is the only policy included in this review where paid educational leave was expanded substantially. On the other hand, many of the policies under review introduced new short-time courses, while others expanded existing policies (typically funding of training costs) to include short courses. For instance, Austria’s Training-Related ALMPs expanded the training and subsistence subsidies to cover short courses, whereas Estonia’s State-Commissioned Short Courses introduced new short vocational courses, as did Hungary’s Open Learning Centres (with a focus on low‑skilled adults) and Singapore’s SkillsFuture Series (focussing on emerging skills ). For some other reforms, introducing or expanding short/modular courses, part-time education, or online and evening classes were a secondary policy objective. The Austrian Paid Educational Leave, the Dutch Training Vouchers and Sector Plans, as well as many of the SkillsFuture initiatives can be used for any type of training, including short, part-time or online courses.

Lack of financial resources

Besides a shortage of time, lack of financial resources can be an important barrier to participation, particularly for low-wage or low-skilled workers (OECD, 2019[3]). This barrier can be tackled by covering the direct and indirect costs of training participation, partly or in full.

Almost every reform under review addresses the barrier of lack of financial resources. Estonia (ALMPs), the Netherlands (Training Vouchers) and Singapore (SkillsFuture Credit) introduced training vouchers to reduce the financial burden of participation. Austria (Expansion of training-related ALMPs), Hungary (Basic Skill Courses) and Singapore (SkillsFuture Mid-Career Enhanced Subsidy) introduced and expanded subsidies for training, as well as subsidies to cover sustenance costs or travel costs.

In addition to individuals, employers may also face financial barriers in providing training to their workers. They face direct costs, either to organise training provision in house or paying (part of) training costs for their employees to attend training outside the workplace. Employers also face indirect costs due to decreased productivity during the training and the risk that workers leave their job after the training. Employers’ financial barriers may be addressed through subsidies to cover training costs, which is the case in the Dutch Sector Plans. Training levies, such as the Italian Training Funds, are a funding mechanism by which employers pay a (compulsory or voluntary) contribution to a pooled fund out of which training is financed. This can have a redistributive effect since most funds allow redistributing funding from larger to smaller enterprises.

Lack of prerequisites

Adults are sometimes not able to access certain training programmes because they either lack the minimum required skills or lack the qualification to show that they possess those skills. For those adults who actually lack basic (native) language skills or numeracy skills to follow the course, providing opportunities to participate in basic skill courses (particularly at the primary and lower secondary level) can address this barrier. Examples of selected policies that explicitly address the barrier of a lack of prerequisites are Austria’s Initiative for Adult Education, Hungary’s Basic Skill Courses, and Italy’s Adult Education Centres.

For adults who lack formal qualifications, but have acquired relevant skills non- or informally, skill recognition is an important policy. Estonia’s Life Long Learning Strategy, Italy’s Adult Education Centres (since the adjustments in 2011), the Dutch Training Vouchers (since the adjustments in 2014), and the Dutch Sector Plans explicitly address this by allowing adults to use training vouchers, for example, to take part in a formal recognition process.

Lack of employer support

According to the 2016 Adult Education Survey, a lack of employer support is a common barrier to participation for working adults, although rarely the most important barrier. Only two of the reforms under review explicitly address this barrier. In Italy, employers are incentivised to engage in training activities by earmarking future training costs through training levies (Training Funds). In the Netherlands, employers are actively engaged in the development of Sector Plans to enhance their local labour market, e.g. through retraining and upskilling their current or future employees.

Other barriers

Individuals may be willing to participate in learning activities, yet they do not register for training because they do not know which training they can or should follow in order to change job or adapt to new requirements in their own job. It is therefore important to provide people with sufficient and easily accessible information about training opportunities, to actively bring this information to the attention of particularly vulnerable groups that are unlikely to participate, and provide them with quality guidance and support (OECD, 2019[2]). The Estonian Lifelong-Learning Strategy, for example, highlights guidance and support to adults who would like to return to formal education, and introduces a national qualifications framework that provides insight into which skills and qualifications are relevant for which occupations.

Some groups will not participate in training due to the fact that they are not eligible to participate in it. Expanding the target group for training programmes is therefore a mechanism to increase participation. Several of the selected reforms expanded the target group in the time period under observation, but the expansion itself is typically not the primary aim of the policy. Exceptions are the Expansions of Training-Related ALMPs in Austria and Estonia, and the Free Second Vocational Degree in Hungary, which expanded the eligibility to obtain a free second VET degree to all adults. Examples of policies for which expansion of the target group happened without it being the primary policy aim are the Network training and Training vouchers (the Netherlands), for which the target group was adjusted from 55+ year-old to 50+ year-old unemployed workers.

Finally, some people may not participate because there is currently no training on offer that fits their needs and preferences. Introducing entirely new training programmes, or revising existing ones to tailor them to emerging needs, may therefore increase participation. The Estonian Lifelong-Learning Strategy and the Singaporean SkillsFuture Series introduced new training programmes for in-demand skills. New basic skill programmes aimed at adults with low skills (typically without a primary or lower-secondary degree) were introduced in Hungary (Basic Skill Courses, Open Learning Centres). Finally, the Network Training in the Netherlands introduced a new job-search training programme for unemployed older workers.

Lessons learnt – content

• There is no magic bullet for increasing adults’ participation in education or training. There is significant variation across countries when it comes to the types of reforms implemented.

• Most reforms are addressed to individuals by expanding their training options, or by improving incentives to participate in existing training programmes. Far fewer reforms are aimed at increasing training provision by employers.

• Most countries adopt a comprehensive approach to increasing learning participation; combining policies that address multiple barriers to training participation, cover several types of training and engage multiple target groups. Most countries introduced both untargeted measures aimed at increasing the participation of large shares of the adult population, as well as policies with a more specific and narrow target group, such as the low-skilled or older unemployed individuals.

Learning from the policy development process

Reforms often occur under the influence of urgency, pressed by events. Careful and considered reform design, however, is key for reform effectiveness. This section describes aspects of this design process, which led to the deployment of successful reforms in the countries included in this study. It analyses the existence of ex-ante needs assessments and the actors involved in the conceptualisation of the reform.

Assessing the need for reform

All reform processes are designed to meet a policy target. Each country, however, can follow a different path in the definition of its adult learning policy objectives and priorities. Countries can motivate their action by political priorities, to amend a previous policy, or as the necessary consequence of an analysis of the skill system in their country. The latter can take the form of ad-hoc exercises to assess the existing skill gaps in the population, or rely on more broad-scoped analysis of the labour market and education trends in the country.

Policy- and political priorities

Some of the reforms considered in this study were motivated by the political conviction that adult learning would support competitiveness and growth, without an ad-hoc empirical analysis of individuals’ skill needs. In the Netherlands, Sector Plans were conceived to sustain employability of workers during the financial crisis, and therefore make regional and sectoral labour markets more resilient to economic shocks. Similarly, the expansion of ALMPs in Estonia did not leverage any precise assessment of skill needs. Instead, it targeted unemployed individuals who did not meet the labour market demands in terms of qualifications and skills, as well as employed individuals who stood to benefit from retraining to stay employed or to change jobs.

Among the reforms that fall under the scope of this analysis, some were motivated by the need to fill a policy vacuum, or remedy inconsistencies in the adult learning policy landscape. Such is the case, for instance, for the Italian Training Funds. Before their introduction, the main instrument to support training for employed workers in the country was a voucher for individual training activities, the take-up of which was limited and conditional on employers’ agreement. In Hungary, making the second (and first) vocational degree free irrespective of age, eliminated disparities in access to secondary VET education above or below a certain age, and avoided the automatic transfer of those above the age limit to a different educational offer (Evening Schools). In Austria, the features of Initiative for Adult Education have been continuously adapted to reflect changes in other laws that could be of relevance for education and training. For example, the programme content was adapted when the law introduced new requirements to obtain a lower secondary degree.

Leveraging the analysis of labour market features

The policy impetus for adult learning reforms relied, in many cases, on an analysis of labour market trends, the level of skills in the adult population or other empirical considerations. The design of the Italian Adult Education Centres stemmed from two key insights: the high proportions of the population lacking a secondary education degree², and the low literacy and numeracy levels emerging from the results of PIAAC. Similarly, in the Netherlands, a 2011 report (Raad voor Werk en Inkomen, 2011[4]) found that one of the reasons why older individuals stayed longer in unemployment was their disadvantage in job searching, and in particular their lack of network of relations or willingness to activate it. Based on this report and an analysis of the dynamics of unemployment by age category, the country developed the Network Training program, which requires individuals older than 50 years of age to take 20 hours of training to enhance their employability. The training enables individuals to get to know their abilities and interests, learn how to use their network to find jobs and ultimately improve their job search skills. The same applies for the Training Voucher program, which provides older and unemployed individuals with a monetary incentive to take job-oriented training, thus raising their probability of finding a job.

In Estonia, the development of the Lifelong-Learning Strategy was based on previous analysis conducted by the Ministry of Education and Research, i.e. the Estonian Education Strategy 2012‑2020 (Ministry of Education Estonia, 2012[5]). The analysis included research evidence on education trends in Estonia and international comparisons. Also in Estonia, evidence of low learning participation, including from PIAAC analysis, supported the introduction of State-Commissioned Short Courses. In Hungary, the introduction of Open Learning Centres and the Basic Skill Courses leveraged evidence suggesting that the labour market was characterised by: i) a large share of the population not in the condition to succeed in the labour market, due to low education achievement and low skills; and ii) significant labour shortages. Both domestic and international recommendations, including the OECD Economic Survey for Hungary, urged the country to improve skills levels as a top priority. The Open Learning Centres leveraged information accumulated by NGOs working in direct contact with individuals in need of retraining.

Skill anticipation and assessment exercises

Among the policies considered in this study, few perform an economic needs analysis ahead of the policy design. An exception is the Initiative for Adult Education in Austria, which conducted an analysis of the supply and potential demand for upskilling and reskilling opportunities. In Estonia, each round of State Commissioned Short Courses follows the conclusions of a needs analysis performed by the Ministry of Education and Research, based on OSKA, the system for analysis and skills prognosis of future labour market skills. The focus of the analysis changed from competences to sectoral employment demands, after the evaluation of the European Social Fund 2009‑2014 programming period concluded that the development process of the provided courses did not sufficiently rely on evidence (Haaristo and Nestor, 2014[6]). In Singapore, SkillsFuture policies were informed by the desire to stimulate economic and productivity growth, but also informed by an assessment of the demand for skills through Industry Transformation Mappings (ITMs). This translated into strong support for lifelong learning in multiple fields, based on the recognition that a more educated workforce makes companies more productive and competitive.¹

Some of the examined policies and programmes continuously incorporate new skill anticipation and assessment information to keep the policy up to date. This is particularly important in the context of changing skill demand in the labour market and the resulting need to update the training offers. Some of the policies considered internalise the skill anticipation exercise, while others “outsource” it to an external body.

Policies that internalise skill anticipation set out who is responsible for incorporating the information and how that is linked to decision-making. For example, in the Netherlands skills assessment and anticipation has to be included in the Sector Plans prepared by the stakeholders. Based on such information, the Ministry of Social Affairs and Employment decides which sectoral plan to co-fund. In Italy, Training Funds both have to incorporate skill information and make training decisions. There is some evidence that this setup was not entirely successful as the Funds have devoted resources to developing skills which were not in demand (e.g. manufacturing and production), while failing to support training in skills in severe shortage (e.g. computer and electronics) (OECD, 2019[7]).

In Estonia, conversely, counsellors can only approve ALMP training in fields that are in line with the skill anticipation and assessment exercises. These fields are identified by OSKA. This is an example of policy where information on skills is produced by a different institution than the one which takes decisions, although conditional on a well-defined set of rules.

Piloting

Lastly, in few selected cases the initial design of the policy was tested in a dedicated pilot, or a trial period during which the new policy was implemented in a limited geographical area or for a number of recipients. This offered the opportunity to assess the potential outcomes of the reform, without major disbursements, and eventually amend the policy design, if deemed necessary. In Hungary, a first phase of the Open Learning Centres involved 10 centres and 1 000 learners, and was funded by the EEA and Norway grants. In Estonia, different aspects of Training-Related ALMPs reforms were first experimented with using funds from the European Structural Funds, then made permanent and financed through EUIF when deemed successful. In Italy, a pilot was carried out in 2014/15, whereby nine Adult Education Centres were created and monitored in their functioning for a year. The results of this experimental year are not publicly available, but the pilot was considered successful, according to stakeholders.

Stakeholder involvement

Adult learning policies are typically not the purview of one level of the administration or one particular ministry. On the one hand, this reflects the complex nature of this policy area, which lies at the crossing of educational, labour and industrial policies; on the other hand, it follows from the specificities of the governance system in the country, which can attribute competency over skill policies at the central government only or to a broader set of stakeholders. Therefore, several levels of government participate in the design of the policies of interest for the present study. Other stakeholders regularly involved include social partners, representatives of teachers and school administrators, or experts in the civil society. Table 2.3 provides a summary of the stakeholders involved in the conceptualisation of the policies selected for this study.

Table 2.3. Stakeholder involvement in the design of reforms under review

Country	Reform	Actors involved
		National public admin	Regional public admin	PES	Social partners	Learning providers	NGOs	Individual employers	Civil society
AUT	Expansion of ALMPs
	Initiative for Adult Education
	Paid Educational Leave
EST	Expansion of ALMPs
	Lifelong-Learning Strategy
	State-Commissioned Short Courses
HUN	Free Second Vocational Degree
	Basic Skill Courses
	Open Learning Centres
ITA	Adult Education Centres
	Training Funds
NLD	Network Training
	Training Vouchers
	Sector Plans
SGP	Skillsfuture Credit
	SkillsFuture Mid-Career Enhanced Subsidy
	SkillsFuture Series

Note: Lighter coloured boxes indicate stakeholders, which only had a marginal involvement in the design phase.

Source: OECD elaboration based on expert interviews and literature review.

The central role of the national government in the design of learning policies emerges clearly from Table 2.3: in all policies considered was the central government the key player, even where regional or local governments were also involved (Initiative for Adult Education in Austria; Adult Education Centres in Italy; the Dutch Sector Plans). Only in the case of the Austrian Initiative for Adult Education did representatives of the federal states have a stronger role to play, as they outnumbered representatives from the federation 9 to 4 in the reform’s Steering Committee. This likely reflects the sharing of competences in a federal state.1

The centralised administration is usually represented by the Ministry of Education (Initiative for Adult Education in Austria, the Lifelong-learning Strategy in Estonia, Adult Education Centres in Italy, SkillsFuture Mid-Career Enhanced Subsidy in Singapore), of Employment, Labour and Social Affairs (the Expansion of ALMPs in Austria, the Training Funds in Italy, the Network Training, Training Vouchers and Sector Plans in the Netherlands), of Innovation and Technology (Free Second Vocational Degrees in Hungary) or of Finance and/or Economic Activities (Estonian State-commissioned Short Courses), and by the central branches of the PES (several policies in Austria, Estonia, and the Netherlands).

As far as other stakeholders are concerned, Table 2.3 highlights that employers’ and workers’ associations were frequently engaged in the reform design. They are often directly involved in the provision of training, which enables them to understand reform needs in the country. In the case of the Dutch Sector Plans and the Italian Training Funds, such involvement extended to co-funding of training by employers and/or unions. At the very least, they are likely to be the first to understand how changes in production affect skills demand, and therefore the new training needs of employed workers.

In a few of the analysed reforms, training providers and representatives from teachers and administrators of education institutions were also consulted in the design of the policy. These reforms most frequently focused on second-chance education opportunities (the Austrian Initiative for Adult Education, the Italian Adult Education Centres) or the design of a Lifelong-Learning Strategy (for Estonia). Similar reforms in Hungary, instead, did not consult learning providers in the policy design.

Forms of coordination among stakeholders

In most analysed cases, the initiative to reform adult learning policies is taken at the central level. In all policies considered, however, the central government did not conceive the details of the reforms on its own, but rather sought to coordinate with other stakeholders affected by the policy. Reforms in Hungary represented exceptions in this respect: The reform of Free Second Vocational Degrees and the establishment of the Basic Skill Courses were carried out by the government with very limited stakeholder involvement, according to the experts interviewed. Further, the design of the Open Learning Centres did not involve the national government, but rather an alliance of ten non-profit civil sector organisations. In a similar bottom-up way, social partners initiated the creation of the Dutch Sector Plans in the majority of cases.

A coordinated approach ensures that key players in the adult learning space can provide advice and feedback on the reform design, which in turn can further translate into greater buy-in in the policy. In some of the reforms considered, establishing a cooperation among key players was a goal in itself, such as in the case of the Austrian Initiative for Adult Education, or a key value added, as for the Dutch Sector Programs. The downsides of such approach are related to the costs of coordinating parties with different and sometimes competing interests, which can reduce the efficiency of the decision-making process. To a certain extent, embedding cooperation among stakeholders not only in the design but also in the implementation phase of the reform can limit strategic considerations, insofar as stakeholders have to repeatedly interact, and can be “punished” by others for uncooperative behaviours. Some form of cooperation in the implementation phase takes place in several of the analysed policies (e.g. Austrian and Estonian reforms of ALMPs, the Singaporean SkillsFuture initiatives, see the implementation section of this chapter below).

Another possible strategy to minimise the downsides of cooperation is to differentiate the tasks attributed to different stakeholders in a multi-stage process. In most analysed cases, for instance, decision-making power broadly stayed with the central or regional administration, while other stakeholders mostly had an advisory role. Coordination could take the form of a direct exchange between central government and other stakeholder organisations (e.g. the Dutch Sector Plans), or the creation of a body congregating and giving a precise role to stakeholders (usually, a Steering or Supervisory Committee). The mission and scope of authority of these bodies varied broadly. They could imply the first-hand design of the policy (e.g. the SkillsFuture initiatives in Singapore, or the Dutch Training Vouchers), an advisory role (e.g. the Estonian Adult Education Council, or the Technical Inter-institutional Committee for the Italian Adult Education Centres), or a supervisory role (e.g. the Supervisory Board for the Estonian EUIF). The tasks of these congregating bodies in some cases further extend to implementation and monitoring (see the implementation section of this chapter below).

Good practice would also consider monitoring stakeholders’ satisfaction in the governance of the policy design, as in the case of the Dutch Sector Plans. Lastly, efficiency should not be achieved at the cost of a merely formal co-optation process. A mechanism of stakeholder involvement should be formalised only if there exists a political space to follow-up on the consultation of stakeholders, or to engage in the ensuing negotiations.

Lessons learnt – development

• Many of the reforms included in this review were motivated by the political will to support and regulate adult learning overall, no matter what individuals’ geographical, occupational or demographic characteristics were. In a few cases, reforms filled a policy vacuum, or remedied inconsistencies in the adult learning policy landscape.

• In the majority of cases, the reform design relied on an analysis of skill, education or labour market trends and the status of the education and training systems. Many policies stayed relevant in a context of changing skill needs by establishing mechanisms to regularly incorporate updated skill anticipation and assessment information. There is some evidence that separating skill information generation from the decision-making leads to better outcomes

• The reform impetus mostly came from the central administration, but other stakeholders were usually involved in the policy design. Social partners were most frequently involved, in light of their proximity to the demand of training.

• Most reforms under review take a coordinated approach to policy design. Stakeholders were involved in the co-creation of the policy, or were consulted in the form of advisory or supervisory groups. A coordinated approach requires reflection on the mechanisms applied to ensure the effectiveness of the decision-making process.

Learning from the financing of reforms

Sufficient funding for the implementation of adult education and training measures is a necessary condition for their successful implementation. Yet, there is surprisingly little knowledge about funding sources and funding levels in different countries. The latest comprehensive cross-country evidence of funding for adult learning dates back to 2013, with some of the data within the study as old as 2008 (FiBS/DIE, 2013[8]). More recent data is available for specific subsections of the adult learning system, such as Active Labour Market Policies (OECD, 2019[2]). The vast majority of research in this area has focused on financial incentives or instruments to pay for adult learning, rather than the funding itself (Andriescu et al., 2019[9]; OECD, 2017[10]; OECD, 2019[11]).

This sub-chapter summarises how the reforms in this review were funded and at what levels of overall, as well as per participant, investments were made in the context of the reforms. It contributes to a better understanding of the type and amount of funding needed to increase adult learning participation.

Sources of funding

The adult education and training policies included in this study are financed through four key sources of revenue: i) taxes; ii) social insurance contributions; iii) earmarked training levies; and iv) the European Social Fund (ESF). Countries typically use two or more funding sources to implement measures. Funding sources used for different measures very much reflect the institutional set-up in each country.

Table 2.4. Funding sources of reforms under review

Country	Reform	Funding source
		Tax funding	Social insurance funding	Training levy funding	ESF funding	Other
AUT	Expansion of ALMPs
	Initiative for Adult Education
	Paid Educational Leave
EST	Expansion of ALMPs
	Lifelong-Learning Strategy
	State-Commissioned Short Courses
HUN	Free Second Vocational Degree
	Basic Skill Courses
	Open Learning Centres
ITA	Adult Education Centres
	Training Funds
NLD	Network Training
	Training Vouchers
	Sector Plans
SGP	SkillsFuture Credit
	SkillsFuture Mid-Career Enhanced Subsidy
	SkillsFuture Series

Source: OECD elaboration based on expert interviews and literature review.

Tax-financed adult learning policies make use of the revenue generated by the general taxation regime of a country. As tax payment is mandatory for most, this funding source has the advantage of being available for general use – unlike social security contributions (see below). Governments can allocate tax revenues freely to adult learning policies, depending on the needs of the labour market, even on an ad-hoc basis. Of the reforms included in this study, only the Dutch Network Training and Training Vouchers are exclusively funded through tax revenues, in this case allocated by the Ministry of Social Affairs and Employment. The vast majority of reforms are partly funded by tax revenues. In Singapore, for example, all SkillsFuture initiatives are financed by government funds (i.e. funds of the Ministry of Education, the Lifelong Learning Endowment Fund, the National Productivity Fund), as well as by funds raised through a Skill Development Levy on employers, and the tripartite SkillsFuture Jubilee Fund, which is financed through donations from employers and unions that are matched by the government. Several reforms are funded through tax revenues at different levels of government. Between 2012 and 2014, the Austrian Initiative for Adult Education was 50% funded by funds of the Federal Ministry of Education and 50% by funds of the Federal States. From 2015 onwards, it has also made use of ESF funding. Along the same lines, the Italian Provincial Centres for Adult Education are co-founded by the Ministry of Education, University of Research, which funds staff costs, while municipalities and providers are responsible for funding school infrastructure.

Social insurance contributions, more precisely contributions to the unemployment insurance, are used to finance education and training in the area of Active Labour Market Policies. Using social security contributions pools individual risk and channels resources towards those most in need of support. On the downside, the use of social insurance contributions for education and training typically comes with eligibility rules, i.e. training is only available to individuals who have contributed to the unemployment insurance fund for a specific time. This limits the flexibility of use for this funding. The expansion of ALMPs in Austria and Estonia included in this study has primarily been financed through an increased use of social insurance contributions. In Austria, biggest source of funding for ALMP in quantitative terms are contributions from the unemployment insurance, with smaller amounts of tax- and ESF-funds (Bock-Schappelwein et al., 2014[12]). Social insurance contributions are charged at a rate of 3% of gross wage to employees and 3% of payroll to employers. In Estonia, ALMPs are exclusively funded through the employers’ share of the unemployment insurance premium, which amounts to 1.4% of gross payroll, as well as ESF funding (Cedefop, 2019[13]).

Training levies are collected from employers as a share of payroll and – in contrast to general social security contributions – are earmarked for training measures. Levy contributions are then pooled across companies and/or sectors and can be accessed by individual companies to finance training. Mandated by law or collective agreements, they are an effective way to encourage firms to set-aside resources for future training (OECD, 2019[2]). Italian companies, for example, pay up to 0.3% of payroll to one of nineteen Training Funds. Companies can then request funding from Funds to finance training plans in line with the strategic priorities of the Funds. Some of the revenue generated through training levies is attributed to other government provision, for example to support welfare measures for redundant workers (Cassa Integrazione). The Singaporean Skills Development Levy deducts a similarly high amount of payroll than in Italy (0.25%), with a minimum of SGD 2.0 and a maximum of SGD 11.25 per employee and month. The levy covers around 1/3 of the costs of all SkillsFuture Initiatives. These payroll levies are small compared to training levies in the Netherlands earmark up to 2% of payroll for training purposes, depending on the sector. The revenue generated is used to finance 50% of the Sector Plans.

Funding from the European Social Fund is a key funding source for some of the reforms included in this study, most importantly Estonia and Hungary. Both countries benefit from a low ESF co-financing rate of 15%. It is notable that both countries joined the EU in the time-period of observation in this study (1 May 2004) and that investment into adult learning has exponentially increased since then – with the help of ESF funding. While the availability of these resources can act as a catalyst for investment, it is questionable if investments will be sustainable in the longer run, when ESF-related support phase out. Models that take‑over the investment into regular tax or social security-based funding are essential for sustainability. Promising practice can be observed in Estonia, which has strategically used project-based ESF funding to pilot adult learning initiatives. Once proven successful, these were funded by national funding sources.

Levels of funding

While information on funding levels is available for the majority of policies included in this review, data is not available in a unified format across countries. Table 2.5 presents data on the estimated yearly funding for the reforms between 2002 and 2017, or the time-period relevant to the specific policy. In cases where there is substantial variation in funding over time, the information is presented in bands. In Austria, for example, training-related ALMPs were funded with EUR 160 million per year in 2002, but increased regularly to reach 510 million per year in 2016. To improve comparability of the data, per participant expenditure of the reforms are calculated. The figures need to be interpreted with care, as they are based on back-of-the-envelope calculations designed to give a general idea of the generosity of the funding.

Table 2.5. Estimated yearly total funding and estimated per participant funding of reforms under review

Country	Reform	Annual funding 2002‑2017	Annual number of participants	Average annual funding per participant
AUT	Expansion of ALMPs	EUR 160 million (2002) – 510 million (2016)	120 500 (2002) – 219 000 (2014)	EUR 1400 – 2600
	Initiative for Adult Education	EUR 17 million (2012‑2014) – 25 million (2015‑2017)	7 000 (2012‑2014) – 10 000 (2015‑2017)	EUR 2400‑2600
	Paid Educational Leave	EUR 6 million (2002) – 185 million (2016)	1 500 (2002) – 18 000 (2016)	EUR 4 000 – 12 000 (full-time) EUR 3 500 – 5 500 (part-time)
EST	Expansion of ALMPs	EUR 3 million (2003) – 16 million (2017)	7 000 (2008) – 55 000 (2012)	EUR 320 - 760
	Lifelong-Learning Strategy	n/a	n/a	n/a
	State-Commissioned Short Courses	EUR 1 million (2007‑2009) – 2 million (2009‑2014)	8 000 (2007‑2014)	EUR 160 (2007‑2009) – 240 (2009‑2014)
HUN	Free Second Vocational Degree	EUR 20‑25 million	20 000	EUR 1130
	Basic Skill Courses	EUR 36 million	46 000	EUR 780
	Open Learning Centres	EUR 1.5 million	2 000	EUR 720
ITA	Adult Education Centres	n/a	183 000 (2015) – 229 000 (2016)	n/a
	Training Funds	EUR 469 million (2016)	1 560 000	EUR 301 (2016)
NLD	Network Training	EUR 12.3 million	41 000	EUR 300
	Training Vouchers	EUR 5.5 million	6 000	EUR 880
	Sector Plans	EUR 52 million	39 000	EUR 1 256
SGP	SkillsFuture Credit	n/a	143 000	SGD 125 (EUR 83)
	SkillsFuture Mid-Career Enhanced Subsidy	n/a	42 500	n/a
	SkillsFuture Series	n/a	7 500	n/a

Source: OECD elaboration based on expert interviews and literature review.

The level of financial investment in the policies under review varies substantially. Based on the data available, the largest amounts of funding per year are dedicated to the Italian Training Funds (EUR 470 million) and Training-Related ALMP in Austria (EUR 510 million in 2016). These two reforms are also attracting the largest number of participants. By contrast, the smallest overall funding is received by the State-Commissioned Short Courses in Estonia (EUR 1‑2 million per year, depending on funding period) and the Open Learning Centres in Hungary (EUR 1.5 million per year). Both involve less than 10 000 individuals in training measures per year.

Looking at per-participant investments paints a slightly different picture. The Austrian Paid Educational Leave had the highest cost per participant at around EUR 12 000 in 2016. These high costs can be explained by the fact that the measure covers (part of) individual wages while the worker is in full-time training. Programmes that only cover direct costs of education and training are much less expensive. They range from around EUR 200 per participant for State-Commissioned Short Courses in Estonia to EUR 2 500 per participant in Training-related ALMPs in Austria. Differences in per participant funding likely reflect differences in the duration and intensity of adult learning provision, as well as differences in purchasing power between countries. Per-participant expenditure can also vary over time, particularly when linked to cyclical factors such as the incidence of unemployment and the duration of unemployment spells. For example, per participant expenditure on training-related ALMPs in Estonia was the highest during the height of the crisis, reaching a height of EUR 800, compared to an average of EUR 400 previously.

Lessons learnt – funding

• The vast majority of reforms are at least partially tax-funded, while only some reforms are funded through social security contributions or levies. This may be because general taxation provides greatest possible flexibility for government stakeholders and can benefit all, not only those paying into social security or levy schemes.

• The ESF is a significant funding source for many reforms, most notably those implemented in Estonia and Hungary. Both countries joined the European Union during the time covered by this study and benefited from a favourable co-financing rate of 15%. The increase in learning participation in both countries can be partially attributed to the availability of this additional funding. This reliance on ESF-funding can have important implications for the sustainability of the reforms beyond the ESF funding cycle. Estonia has established good practice by trialling new measures using ESF-funding and transferring them to tax or social security based funding, when proven successful.

• Successful adult learning reforms do not have to come with a high price tag. Per-participant investment ranges from EUR 200 to 2 500 for policies that only cover the direct cost of delivering some kind of education and training course for adults. The sole measure compensating individuals for foregone wages comes with a higher price tag: per participant costs of the Austrian paid education leave amount up to EUR 12 000.

Learning from the implementation of reforms

Even when well designed, many adult learning policies struggle to translate into real change on the ground. Reasons for this vary, including a lack of capacity of the state bureaucracy, weak governance mechanisms, limited buy-in from key stakeholders and inadequate delivery structures. To avoid these issues, there is increasing interest by governments on how to strengthen and support implementation in the area of adult learning.

This subchapter reviews how the successful reforms included in this study have addressed implementation issues, namely the governance and delivery of the policies, and lessons to be learnt from the issues encountered.

Governance

Effective decision-making structures and processes are the basis for successful implementation of adult learning policies. One can distinguish between three basic types of governance: i) top-down governance led by the responsible levels of government; ii) network governance, which also involves non-government actors such as social partners or civil society organisations; and iii) governance through market mechanisms.

Top-down approaches can be an efficient way to govern adult learning policies, but may lack buy-in from relevant non-state actors. There are few policies included in this review that are exclusively governed in this way. Examples include the Adult Education Centres (CPIAs) in Italy, which fall under the responsibility of the Ministry of Education, University and Research (MIUR). Local administrative levels (Italian regions) have limited input into the operation of the Centres and only get involved in rare cases when they want Centres to deliver courses that respond to local needs. The 2012 law creating CPIAs mandated the creation of ‘Lifelong learning territorial networks’ that bring together other local stakeholders involved in lifelong learning, yet this has not yet been implemented. Similarly, the Hungarian introduction of a Free Second Vocational Degree has been implemented under the exclusive responsibility of the Ministry of Innovation and Technology.

Box 2.1. Governance case study – SkillsFuture in Singapore

In Singapore, the Future Economy Council (formerly SkillsFuture Council) has the mission to make Singapore’s economy ready for the future and drive growth. The Council oversees the implementation of all SkillsFuture measures, including SkillsFuture Credit, SkillsFuture Mid-career Enhanced Subsidy and SkillsFuture Series. It is currently chaired by the Deputy Prime Minister who is also the Minister of Finance, and brings together high-level stakeholders from across the skill development system. The council encompasses different government ministers (Trade and Industry, Communication and Information, Social and Family Development, Education, Manpower, National Development), representatives of the National Trades Union Congress, the Chamber of Commerce & Industry, the Singaporean Business Federation and the National Employers Federation. Unusually by international comparison, it also includes education providers (Universities, Polytechnics, Technology Institutes) and a large number of individual employers (e.g. McKinsey & Company, Straits Construction) in the governance of its skill development system.

Source: www.skillsfuture.sg, https://www.futureeconomy.sg/about/the-future-economy-council/, Ministry of Finance Singapore (n.d.[14]), Scope of Future Economy Council. Answer to parliamentary question, https://www.mof.gov.sg/Newsroom/Parliamentary-Replies/Scope-of-Future-Economy-Council.

Network approaches to governance ensure that the views of stakeholders affected by a policy are represented and to get their buy-in for implementation. On the downside, they increase coordination costs and may decrease efficiency of decision-making. In the case of adult learning policies, key stakeholders include trade unions, employer organisations, individual employers, learning providers, civil society organisations and learners themselves (or their representatives). Most policies included in this study are governed through some kind of networked approach, most often in the form of supervisory boards composed of key actors. In Estonia, for example, representatives of ministries, social partners, universities and individual employers make up the Steering Group of the Lifelong-Learning Strategy.

Social partners play a key role in networked approaches, while other stakeholders are less frequently involved in governance in practice (Table 2.3). Learning providers in particular, who are responsible for the implementation of policies, are frequently not included in decision-making processes. Notable exceptions are the Estonian Lifelong-Learning Strategy, the Steering Group of which includes a university representative, and the SkillsFuture Singapore measures, which include representatives of different educational institutions (see Box 2.1). Similarly, governance structures involving multiple levels of government are rare amongst the policies under review, which is relevant as regional administrations are often key players in the adult learning policy area. Only the Austrian Initiative for Adult Education is governed with the strong involvement of Austrian federal states. Its steering group includes nine representatives of the Austrian federal states and only four representatives of the federal government. There is limited involvement of NGOs and no involvement of civil society organisations in the governance of policies included in this review.

National level research highlights that the lack of involvement of all relevant stakeholders hinders the effective implementation of policies. In Estonia, for example, the evaluation of State-Commissioned Short Courses 2009‑2014 finds the lack of involvement of some relevant bodies a challenge and recommends that: i) government partners should be involved in funding decisions to achieve greater buy-in; and ii) training providers should be involved in priority setting to harness their knowledge on regional labour market needs and on the target group (Haaristo and Nestor, 2014[6]).

Market mechanisms make use of competition between learning providers to drive supply and demand, within the bounds of government regulation. In theory, this governance approach has the advantage of producing efficient solutions. However, education and training markets may have limited effects in practice (Waslander, Pater and van der Weide, 2010[15]). Few policies included in this review rely on market-mechanisms to governance. These include voucher-based adult learning provision, namely the Dutch Training Vouchers, Training Vouchers implemented in the context of Estonian ALMPs and the Singaporean SkillsFuture Credit.

Table 2.6. Governance of reforms under review

Country	Reform	Type of governance	Actors involved
			National ministries	Regional govern.	PES	Social partners	Learning providers	NGOs	Individual employers
AUT	Expansion of ALMPs	Network
	Initiative for Adult Education	Network
	Paid Educational Leave	Network
EST	Expansion of ALMPs	Network / Market
	Lifelong-Learning Strategy	Network
	State-Commissioned Short Courses	Network
HUN	Free Second Vocational Degree	Top-down
	Basic Skill Courses	Top-down
	Open Learning Centres	Network
ITA	Adult Education Centres	Top-down
	Training Funds	Network
NLD	Network Training	Network
	Training Vouchers	Market
	Sector Plans	Network
SGP	SkillsFuture Credit	Market
	SkillsFuture Mid-Career Enhanced Subsidy	Network
	SkillsFuture Series	Network

Note: In Hungary, national ministries have only been a governance partner of the learning centres since the latest ESF funding cycle, lighter coloured box indicates limited role.

Source: OECD elaboration based on expert interviews and literature review.

Delivery structures

Adult learning policies need robust delivery structures to reach potential learners. Some of the policies included in this review are directly ‘delivered’ to learners, e.g. the Dutch Network Training, and hence have more straightforward delivery structures. Other implementation mechanisms are more complex. The Paid Educational Leave Policy in Austria, for example, is solely a means of funding participation in learning rather than delivering provision itself. While the measure itself is ‘delivered’ by the Austrian PES, the training that individuals take-part in using the Paid Educational Leave is ‘delivered’ by public and private learning providers.

Type of providers

Government or quasi-governmental learning providers are involved in the delivery of many of the policies under review. Often these are learning providers who are already delivering education and training programmes, but who receive additional funding to implement the reforms. They are perceived as reliable providers, with limited incentives to ‘game the system’, according to some of the stakeholders interviewed. In Hungary, for example, the reform that made the pursuit of a second vocational degree free is implemented through publicly-funded 44 VET centres (Szakképzési Centrum). Similarly, in Estonia, State-Commissioned Short Courses are implemented by public vocational and higher professional education institutions. In Italy, the Adult Education Centres that started delivering adult education courses in 2014/2015 are public institutions under the responsibility of the Ministry of Education, University and Research.

Private learning providers are involved in the delivery of policies in all countries, with the exception of Hungary. This includes cases where delivery chains are more complex, for example where the PES implement the policy by providing funding, which can then be used towards courses at private learning providers. Examples include the Paid Educational Leave in Austria and the Dutch Training Vouchers. In most cases where private learning providers are involved, these undergo a registration, certification, accreditation or licensing process. Providers of Estonian Training-Related ALMPs, for example, must be registered in the Estonian Education information system EHIS, deliver courses in line with specified curricula and engage in monthly reporting to the Estonian Unemployment Insurance Fund. A notable exception is training delivered in the context of the Italian Training Funds. Interviewed stakeholders raised concern about the lack of quality assurance of training measures implemented by providers of non-formal learning.

Table 2.7. Providers of reforms under review

Country	Reform	Implementing actors					Details
		State actors	Social partners	NGOs	Companies	Private providers
AUT	Expansion of ALMPs						Accredited learning providers
	Initiative for Adult Education						Accredited learning providers
	Paid Educational Leave						PES; final delivery through public and private learning providers
EST	Expansion of ALMPs						Licensed learning providers
	Lifelong-Learning Strategy						Final delivery through public and private learning providers
	State-Commissioned Short Courses						Vocational + higher professional education institutions
HUN	Free Second Vocational Degree						Publicly-funded VET centres
	Basic Skill Courses						Türr István Training and Research Centres (public)
	Open Learning Centres						Network of NGOs
ITA	Adult Education Centres						Publicly-run CPIA centres
	Training Funds
NLD	Network Training						Delivery through PES
	Training Vouchers						PES; final delivery through private training providers
	Sector Plans						Final delivery through private training providers
SGP	SkillsFuture Credit						Accredited public and private learning providers
	SkillsFuture Mid-Career Enhanced Subsidy
	SkillsFuture Series

Note: lighter coloured box indicates a limited role

Source: OECD elaboration based on expert interviews and literature review

Social partners have limited involvement in the delivery of the reforms under review, despite their significant involvement in governance issues. An exception is Austria, where social-partner-run training institutions are one of many providers delivering training-related ALMPs. The Estonian Lifelong-Learning Strategy is also implemented with the involvement of social partners, yet, being a strategy, it is not delivered to learners directly. NGOs are sometimes entrusted with the delivery of adult learning, especially when the policy aims to engage adults with low skills or other specific target groups. The Hungarian Open Learning Centres are run by an alliance of ten non-profit civil sector organisations (Szövetség az Életen Át Tartó Tanulásért), which deliver low-threshold learning opportunities in around 50 locations across the country. Similarly, accredited NGOs deliver parts of the Austrian Initiative for Adult Education, which enables adults with low skills to obtain basic competences.

In the policies under review, individual companies are rarely involved in the direct delivery of adult learning. An exception is the training delivered through the Italian Training Funds, 60% of which is implemented directly by contributing employers.

Procurement and contracting

Providers are tasked with the delivery of adult education and training through a process of public procurement and contracting. This process can take part in three key ways: governments either: i) directly approach providers to negotiate the scope of provision; ii) draw on calls for proposals to gauge provider interests and select providers to be funded; or iii) give individuals the opportunity to directly purchase education and training with the help of training vouchers and similar funding mechanisms.

Direct negotiations about the scope of the provision take place in settings where there is a single obvious provider or provider network that is already delivering similar education and training services. Providers in these settings typically receive a negotiated lump-sum payment for the provision of education and training services and have to observe compliance and reporting requirements. This process has the advantage of having a high degree of administrative efficiency, due to the absence of complicated procurement rules (see below). However, it leaves the public administration with limited control over provider selection and can be fiscally inefficient. Only few of the reforms under review rely on this kind of arrangement. Publicly funded VET centres in Hungary, for example, obtain increased funding as part of their regular grant funding to deliver Free Second Vocational Degrees. The same is true for Adult Education Centres in Italy that deliver basic adult education courses.

Calls for proposals are the most common approach to procuring education and training services in the reforms under review. The process involves providers writing proposals according to specified criteria to apply for the delivery of education and training services. The degree of competitiveness of calls for proposals can vary across countries. In many cases, only accredited or certified providers are invited to apply. From a public administration perspective, calls for proposals increase competition between providers, which can lead to higher quality provision and more efficient allocation of resources. On the downside, calls for proposals are often associated with high administrative costs, both for the public administration and providers. They also lead to planning uncertainty, when providers receive short-term funding and need to reapply at regular intervals to sustain operations. In Estonia, for example, the Ministry of Education launched bi-yearly calls for proposals for its state-commissioned courses between 2009 and 2014. This has since then been reduced to yearly calls, due to the high administrative burden. The PES and relevant ministries draft calls for proposals according to priorities informed by SAA information and data on the performance of courses in the previous funding period. Applications by learning providers are reviewed by a panel of experts. While in most cases, calls are launched to select learning providers, Training Funds in Italy also use public calls to engage enterprises. Enterprises can submit a training plan in line with specified criteria to access funding from collective accounts. After an evaluation by the Fund, they can be awarded funding that covers 60‑65% of the costs outlined in the training plan. The remainder is co-financed by companies.

Box 2.2. Procurement case study – Initiative for Adult Education Austria

The Austrian Initiative for Adult Education (Initiative Erwachsenenbildung) procures basic skills provision and second-chance education from a large variety of public, private and third-sector learning providers. Providers are selected following regular calls for proposals by the Federal Ministry of Education or direct application to the regional education authorities.

Only accredited providers can apply for funding. Accreditation can be obtained by providing evidence that minimum requirements in three key dimensions are met: i) Institutional framework conditions, e.g. organisational structure, infrastructure, application processes; ii) Quality of the course offer, including pedagogic concept, counselling offer and existence of tailored outreach activities; iii) Qualifications of teachers, trainers and counselling staff, including a concept for their continuing professional development.

The most recent evaluation of the initiative attests the high quality of the course offer. However, it also highlights issues resulting from the high standards for the adult learning workforce: recruitment and retention of qualified staff proves challenging given the limited availability of individuals meeting the high standards

Source: Steuerungsgruppe Initiative Erwachsenenbildung (2019[16]), Programmplanungsdokument Initiative Erwachsenenbildung. Länder-Bund-Initiative zur Förderung, www.initiative-erwachsenenbildung.at; Länder-Bund-ExpertInnengruppe and „Initiative Erwachsenenbildung“ (2011[17]), Programmplanungsdokument "Initiative Erwachsenenbildung"; Steiner et al. (2017[18]), Evaluation der Initiative Erwachsenenbildung.

Market-based mechanisms tie funding for learning providers to the decisions of individual learners. They are often implemented using individual training vouchers or credits to purchase training from education and training providers. Funding for learning providers follows the individual. This procurement method has the advantage of increasing individual choice, competition between providers and hence quality and fiscal efficiency in theory. In practice, it needs strong support structures that help individuals make informed choices about their training to function well. From a provider perspective, it should be implemented in combination with a level of core funding to ensure planning certainty. Three of the reforms under review make use of these market-based mechanisms. This includes the Dutch Training Vouchers, which are available to individuals above the age of 50+ (previously 55+) and can be used to part-fund job-oriented education and training of at most one-year duration. The vouchers initially had a value of EUR 750, which was later increased to EUR 1 000. Similarly, SkillsFuture Credit in Singapore gives every Singaporean aged 25 and above a training credit of SGD 500. This credit can be used to part-fund participation in education and training courses delivered by government agencies. In practice, individuals pay net fees of training participation (after deduction of the credit), while credit payments are directly disbursed to training providers. The credit is currently a one-time subsidy, but the credit may be topped up at regular intervals. Finally, Estonia introduced training vouchers (Training card) for the unemployed in 2009 and later expanded the scheme to the employed population in 2017. Individuals can receive training vouchers to pay for training delivered by certified training institutions. Voucher-funded training has a maximum duration (2 years for the unemployed, 3 years for the employed) and funding is capped at EUR 2 500 per person.

Lessons learnt – implementation

• The vast majority ofreforms under review are governed by networks of different adult learning stakeholders. These include social partners and public employment services in many cases. Individual employers, learning providers and regional ministries are less frequently involved in the reforms under review.

• A wide range of public and private providers deliver adult learning reforms on the round. They typically have to meet specific quality standards or criteria to be eligible for funding.

• Procurement of providers often takes place through calls for proposals, although there are some examples of more market-based mechanisms where individuals make choices about their own training through training vouchers.

Learning from the further adaptation of reforms

A key pattern emerging from the analysed policies is that almost all of them went though some changes over time compared to the initial design. These alterations ranged from changing the target group, through adapting features of the programme, to altering the delivery process. Given that such adjustments were made to almost all policies under review, their continuous adaptation may have contributed to their success in increasing adult learning participation.

Policy changes

Adapting policies based on lessons learnt from implementation or due to changing circumstances can be an important success factor. Adjustments can help overcome low take-up by identifying bottlenecks or improving effectiveness. Changes ranged from shifting the target group through changing features of the programme (eligibility criteria, benefit/subsidy generosity, provision), to changing the implementation processes. For example, the target group of the Hungarian Basic Skill Courses were adapted after the first year of operation. Due to the low voluntary take-up of adults with low qualifications, it was linked to the public work programme, where such adults are overrepresented. Similarly, in the Netherlands, the value of Training Vouchers was increased one year after the launch of the programme to increase their attractiveness as the initial amount seemed insufficient.

Some reforms were themselves intended to correct shortcomings of previous policies. In Estonia, for example, the development of the Lifelong-Learning Strategy was based on a comprehensive stakeholder consultation because a previous attempt to design a strategy lacked broad political support. The introduction of the Italian Adult Education Centres was the result of extensive ex-ante consultation with stakeholders to amend the shortcomings of the previous policy setting and improve alignment with the “Upskilling Pathways” recommendation (European Council, 2016[19]). Similarly, one of the reasons behind modifying the VET legislation in Hungary and making a Second Vocational Degree Free of Charge was to correct some inconsistencies of the former regulation.

Policy learning

Policy learning took place based on information generated through three different approaches; monitoring progress, evaluating results and sharing experience (Table 2.8). Monitoring is defined as the regular collection of data with the purpose of understanding if policy delivery is going according to the plan. Evaluations go further than monitoring activities in that they draw conclusions about the relevance, effectiveness and sometimes efficiency of a particular policy. They ensure that the outcomes of the programmes and policies are in line with its theoretical aims and objectives. Data collected during the monitoring phase can be the basis of evaluations. Another way of accumulating learning is by bringing together providers of adult learning to exchange practices. This is possible only in set-ups with decentralised delivery.

Table 2.8. Different ways of policy learning in reforms under review

Country	Reform	Monitoring progress	Evaluating results	Sharing experiences
AUT	Expansion of ALMPs
	Initiative for Adult Education
	Paid Educational Leave
EST	Expansion of ALMPs
	Lifelong-Learning Strategy
	State-Commissioned Short Courses
HUN	Free Second Vocational Degree
	Basic Skill Courses
	Open Learning Centres
ITA	Adult Education Centres
	Training Funds		*
NLD	Network Training
	Training Vouchers
	Sector Plans
SGP	SkillsFuture Credit
	SkillsFuture Mid-Career Enhanced Subsidy
	SkillsFuture Series

Note * Evaluation if target group was reached is planned, but not implemented yet.

Source: OECD elaboration based on expert interviews and literature review.

Monitoring progress

Almost all of the examined initiatives are monitored, however the frequency and extent of reporting varies substantially. Monitoring plays an essential role in verifying if implementation is going according to plan and can provide the basis for adjusting policy. In the Netherlands, for example, take-up of the Network Training was lower than expected in the first year after initiation, so the eligibility criteria was reduced from 55+ to 50+ one year after the introduction of the policy.

Looking across countries, monitoring activities are weaker in the Hungarian and Italian policies, while the Estonia practices seem the most developed. Regarding the Basic Skill Courses in Hungary, monitoring was limited to reporting the number of participants once a year. In Italy, monitoring reports of the Adult Education Centres were produced twice over the examined period, due to ad-hoc requests by the Ministry of Labour (MIUR). In Estonia, on the other hand, the monitoring activities are more regular for the Lifelong-Learning Strategy; a steering committee was formed to continuously follow the implementation (Ministry of Education and Research, 2014[20]).

The monitoring activity is typically carried out by the main implementing agency of the reform such as the head office of the Austrian Initiative for Adult Education or the PES (EUIF) in Estonia. In Italy, the Training Funds are required to monitor part of their own activities (financed through collective accounts) in a more decentralised way. There are two exceptions, where an independent body carries out the monitoring activity. In the Netherlands a research institute (SEO Economisch Onderzoek) monitors the progress of the Sector Plans in the form of six ‘Quick Scans’, and the outcomes are evaluated as well, in a final evaluation. In Italy, the Adult Education Centres are monitored on an ad-hoc basis by the National Agency for the Development of Schools Autonomy.

Due to EU guidelines, programmes using ESF funds were typically monitored more closely. Domestic bodies responsible for distributing the ESF funds were also often involved. In Hungary, for example, the Finance Ministry monitors training delivery and the finances of the Open Learning Centres in addition to activities of the head office. Related to the Austrian Initiative for Adult Education extra questions were added regarding childcare and benefits to the data collection forms to comply with data collection requirements. However, ESF monitoring requirements were met to different extents across countries. For example, a data is scarce related to Hungarian ESF funded projects, data quality is often unreliable and monitoring is very limited according to the official report on ESF funded projects (Századvég and E&Y, 2016[21]).

In some cases, the data collection is not done by the same organisation that carried out the monitoring activities. This is the case, when multiple providers are involved in delivery. In Italy, monitoring information regarding the Adult Education Centres is collected through territorial institutions. In Hungary, coordinators of the Open Learning Centres are responsible for providing data, while the overall monitoring activities are carried out by regional hubs and the head office. Similarly Singaporean Continuous Education and Training (CET) Centres offering courses in the SkillsFuture Series are required to record their trainees' training and placement activities, while SkillsFuture Singapore conducts the overall monitoring.

Take-up was always monitored and, in many cases, additional data was collected regarding demographic characteristics or labour market status of the participants. For example, the Initiative for Adult Education in Austria requires participants to fill out a form and provide socio-economic information (regarding their age, citizenship, native language, highest educational attainment level) and current situation (employment status and benefits received). Fewer monitoring activities focus on the delivery or the process. In Italy, monitoring of Training Funds focuses on whether the commissioned courses have actually taken place. In the Netherlands, the SEO periodically asked about the satisfaction of the different stakeholders involved with the Sector Plans, i.e. employers, regional government, trade unions.

Evaluating results

Incorporating learning from evaluations can ensure that policies remain or become increasingly successful over time. For example, in the case of the Paid Education Leave in Austria, a part-time option was introduced in 2013 as evaluation evidence suggested that long, full-time absences of employees are more likely to have negative effects on labour market outcomes (more details in Chapter 4).

More than half of the selected policies were evaluated. Some countries seem to have stronger evaluation cultures than others: In Austria and Estonia, all of the selected successful reforms were thoroughly evaluated. In the Netherlands and Italy, some of the policies were evaluated, although a systematic approach was missing. Meanwhile there are no (publicly available) evaluations of the SkillsFuture initiatives in Singapore, or the Hungarian initiatives apart from a non-representative academic study focusing on subjective outcomes. As some policies were implemented relatively recently, this could change over time.

A variety of actors commission evaluations, including national or regional governments, social partners or the implementing organisations themselves. For example, the Evaluation of the Estonian LLL strategy was initiated by the Ministry of Education and Research. Social partners were active in this regard especially in Austria. The Austrian Social partners and the Trade Unions Federation in Tirol commissioned two evaluations related to the Paid Education Leave. Evaluations of individual Sector Plans in the Netherlands and certain Training Funds in Italy were initiated in a more bottom-up way by the stakeholders involved. The body that commissions the evaluation is not always the one paying for it. For example, although not typical practice in Austria, one assessment of the Paid Education Leave in Austria was funded by the Austrian National Bank and employers amongst others.

The organisation that conducts the evaluation is often different from the implementing or commissioning entities. In Estonia, independent research consultancies carry out a large share of the evaluation work. Previously, evaluations have also been conducted in-house, e.g. by EUIF research staff. In Austria, evaluations are often implemented by non-profit independent research institutes, while in the Netherlands by universities.

Since conducting evaluations is a demanding task, they are often carried out only for a specific aspect of the reform, for example certain regions, time-periods or providers. In the Netherlands and Italy, only some of the Sector Plans and Training Funds have been evaluated. A 2006 evaluation of the Paid Education Leave in Austria only examined the measure in the Federal State of Tirol. Evaluations are typically ad-hoc, only in Estonia were selected ALMP measures analysed on a yearly basis.

Evaluations can be categorised into outcome, impact and process evaluations. Outcome evaluations typically look at changes in the outcomes or participants, compared to their position prior to participation in the measure. Impact evaluations construct a counterfactual to compare outcomes to what would have happened in the absence of the measure. Meanwhile process evaluations assess how far the policy was implemented in line with the initial plan and why there were deviations. Cost-benefit analysis is a useful complement to evaluations as it allows for efficiency considerations. It compares the outcomes of a programme to its costs by expressing both in monetary terms and discounting them to a present value. (Table 2.9).

Table 2.9. Types of evaluations conducted of reforms under review

Country	Reform	Type of evaluation
		Outcome	Impact	Process	Cost-benefit
AUT	Expansion of ALMPs
	Initiative for Adult Education
	Paid Educational Leave
EST	Expansion of ALMPs
	Lifelong-Learning Strategy
	State-Commissioned Short Courses
HUN	Free Second Vocational Degree
	Basic Skill Courses
	Open Learning Centres
ITA	Adult Education Centres
	Training Funds
NLD	Network Training
	Training Vouchers
	Sector Plans
SGP	SkillsFuture Credit
	SkillsFuture Mid-Career Enhanced Subsidy
	SkillsFuture Series

Note: Lighter coloured box if evaluation if target group was reached is planned, but not implemented yet.

Source: OECD elaboration based on expert interviews and literature review.

Outcome evaluations typically examine change in employment status or earnings after taking part in training or self-reported satisfaction data. For example to evaluate one of the Austrian training-related ALMPs, former participants were asked during telephone interviews about their employment status and job quality a certain time after finishing the programme (e.g. 3 months or one year).

A wide range of methods are used to evaluate the impact of adult learning programmes. According to international good practice, the most rigorous evaluation method would be to conduct a randomised experiment. The impact evaluation of the Network Training in the Netherlands was based on a randomised control trial that assessed the probability of exiting from unemployment as well as job quality in the new job (Groot and Klaauw, 2016[22]). However, this type of research is often not possible due to data limitations and because it can be costly to build in a rigorous research set-up from the outset. Instead, quasi-experimental methods were used to evaluate the expansion of training-related ALMP measures in both Estonia and Austria. These included propensity score matching (Lauringson et al., 2011[23]; Lutz, 2005[24]), instrumental variable estimation (Winter-Ebmer, 2001[25]) and difference‑in‑difference analysis (Hausegger, 2005[26]).

Process evaluations are carried out to improve or increase efficiency of the delivery. These are based mostly on qualitative research methods - in contrast to the two previous types - due to the questions they intend to answer. In Estonia, the evaluation of the Lifelong-Learning Strategy currently examines whether appropriate measures have been introduced to achieve the strategy’s objectives (Ministry of Education and Research, 2014[20]). Certain evaluations uncover issues with the project design. For example evaluations related to the Paid Education Leave in Austria analysed reasons for low take-up from both the employee and the employer perspective (see Box 2.3) (Bock-Schappelwein, Huemer and Pöschl, 2006[27]; Kernbeiß, Lehner and Wagner-Pinter, 2006[28]).

Box 2.3. Learning from evaluations case study – Paid Educational Leave in Austria

Since its introduction in 1998, the Paid Educational Leave (Bildungskarenz) has gone through several changes based on evidence from evaluations. Due to identifying and addressing bottlenecks, the measure has developed from attracting less than 2 000 participants to reaching over 15 000 people every year.

Evaluations uncovered that the benefit amount was insufficient and the minimum duration of the training was too long. Employees found that the low financial benefits did not appropriately compensate for the loss of income and they worried about the impact of a taking time out on their career opportunities. In January 2008, the financial support was increased from a flat-rate tariff to the level of the unemployment benefit.

Introduction of the part-time leave was also a response to evidence that long, full-time absences of employees are more likely to have negative effects on their labour market outcomes. A 2011 evaluation of the measure across Austria drew attention to the need for introducing a measure supporting individuals taking part in part-time learning. Based on this, in July 2013 the part-time Bildungsteilzeit was introduced.

Source: Bock-Schappelwein, Huemer and Pöschl (2006[27]), WIFO-Weißbuch: Mehr Beschäftigung durch Wachstum auf Basis von Innovation und Qualifikation. Teilstudie 9: Aus- und Weiterbildung als Voraussetzung für Innovation; Kernbeiß, Lehner and Wagner-Pinter (2006[28]), Bildungskarenz in Tirol. Inanspruchnahme, Zielgruppe und die Auswirkungen auf die Berufslaufbahn, http://www.ak-tirol.com/pictures/d40/Endbericht20060330.pdf; Lassnigg et al. (2011[29]), Evaluierung der Bildungskarenz.

Cost-benefit analysis was only conducted for certain ALMP measures. In Estonia, this type of analysis was used to assess whether money spent on labour market training had positive returns for the society. The study found that every EUR 1 invested in training lead to EUR 4 benefit for society after two years (Lauringson et al., 2011[23]). In Austria, the method was used to assess if longer or shorter programmes were more effective. Holl et al. (2013[30]) estimated that in the eleven years following the training the net benefit of long training measures (lasting from six months to one year) was seven times higher than that of shorter training courses. Although no cost-benefit analysis was performed related to the Austrian Paid Educational Leave, interviewed experts suggest that this would have not yielded a positive outcome, as cost per participant was high and evidence of positive wage and employment effects of the policy scarce.

Box 2.4. Cost-benefit analyses in the broader context

Cost-benefit analysis can be an important instrument to evaluate training policies, decide on the best allocation of public resources, or facilitate the choice between different alternative policy designs. However, they require extensive sensitivity checks due to the assumptions made during the estimation and calculation of monetary values.

In many cost-benefit analyses, the magnitude and sometimes the sign of the net present value of benefits relies on the assumptions researchers are asked to make. Scant information is typically available about the costs of a given program, although less so for its direct component (i.e. the cost of running it) than its indirect one. The indirect ones include the costs associated with the use of distortionary taxes, and the cost of collecting the extra dollar in tax revenues as it is generated by an individual switching from unemployment to employment. The opportunity cost of wages foregone by the participant because of training could be accounted for, too.

The estimation of the benefits of training is also non-trivial, even conditional on finding a statistically significant impact of the program on participants’ wages and probability of employment. The timing of impact (short- vs long-term) is an important factor, for instance (Andersson et al., 2018[31]). The out-of-sample estimation of long-term gains of the policy further requires assumptions on the individual’s potential career path and wage evolution. Moreover, the participants’ employment choices and future earning can be affected by other labour-market or societal outcomes of training programs, that were not measured or were impossible to measure: health, crime, child care, or even the value of leisure Lastly, cost-benefit analyses hardly account for the general equilibrium effects of the training program, which are discussed in Chapter 4.

Source: Barnow and Smith (2015[32]), Employment and Training Programs, https://doi.org/10.3386/w21659; Greenberg and Robins (2008[33]), Incorporating nonmarket time into benefit-cost analyses of social programs: An application to the self-sufficiency project, https://doi.org/10.1016/j.jpubeco.2007.09.011.

Experience sharing

Bringing together education providers to share their experiences related to implementation can improve delivery. When there are multiple providers local differences can contribute to identifying best practices that can be mainstreamed. Problems can also be overcome more effectively when building on previous experiences.

Experience sharing is not observed frequently in the policies under review. There is some degree of experience sharing regarding the Austrian Initiative for Adult Education and the Italian Adult Education centres. For example, the head office of the Initiative for Adult Education organises all-day events for training providers at three different locations across the country (Wien, Graz, Salzburg). Representatives answer in depth questions regarding training delivery, data collection and even accounting. In Italy, teachers and administrators of the Adult Education Centres can participate in similar sessions. During these events informal peer-learning also takes place as the providers meet and interact with each other.

Other setups allow for more active and decentralised peer learning. In the Dutch Sector Plans, the parties involved in implementation participate in ad-hoc meetings, with the explicit aim of learning through sharing experiences. In Hungary, meetings are organised for the coordinators of the learning centres multiple times a year to raise issues and network (see Box 2.4). The 2012 law creating the Adult Education Centres in Italy mandated the creation of “lifelong learning territorial networks” to create linkages among all stakeholders involved in lifelong learning, including local administrations, social partners, universities and the non-profit sector, but this is not yet operational.

Box 2.5. Learning from experience sharing case study – Open Learning Centres in Hungary

As the programme is delivered at 52 different locations across the country, information flow and coordination is a challenge. To avoid it becoming an issue, the head office (founding NGO) of the programme decided to adapt the organisational structure and to establish a management information system at the time of up-scaling the programme.

A number of regional hubs were introduced, each of which are in daily contact with 6‑8 individual centres. These hubs support the centres and feed information back to the head office. Additionally, there are periodical meetings with the presence of education experts for both local and regional coordinators. During these meetings issues and experiences are shared and good practices are recorded. For example coordinators of the centres did not feel that they are competent to advise and guide individuals towards the appropriate learning opportunities. To overcome this problem, the head office decided to organise training courses on interviewing techniques for all of the coordinators.

Establishing a management information system was part of the ESF funding proposal. The aim was to increase transparency and accumulate information and experiences. For example if a coordinator wants to organise a training in a certain topic, s/he can research if someone else had organised a similar event, who held it and how it went. The coordinators of the learning centres are responsible for feeding timely data in the system related their own learning activities.

A few of the policies or programmes use digital tools to improve information flow between different providers. An online Q&A is available regarding the Austrian Initiative for Adult Education, prepared by the head office. In Hungary, the Open Learning Centres have an online information system that intends to collect, store and make accessible knowledge accumulated by the training centres.

Information generated through these mechanisms does not only benefit providers, but also the policy makers or the head office. According to stakeholders interviewed, this was the case in the Open Learning Centres in Hungary and the Sector Plans in the Netherlands, which led to an improved overall policy design by overcoming barriers and bottlenecks. For example, in the Netherlands, the project manager of the Sector Plans was present at the stakeholder meetings. The Ministry of Social Affairs learned that the application procedure for the sectoral plans was too bureaucratic and time consuming. Following this, the procedures was modified to decrease administrative burden and increase take-up.

Lessons learnt – policy learning

• The vast majority of reforms under review were altered compared to their initial design. Adapting policies based on lessons learnt from implementation or due to changing circumstances can be an important factor for the success of policies. It can lead to addressing barriers to take-up, removing bottlenecks or improving effectiveness.

• Policy learning took place based on information generated through monitoring progress, evaluating results or sharing experience. It is worth incorporating such mechanisms into the design of policies or programmes early on.

• Cost-benefit analyses are rare among the selected successful policies. Although they can be extremely helpful, for example, in deciding between policy alternatives, their results depend largely on a number of methodological assumptions.

References

[31] Andersson, F. et al. (2018), Does Federally-Funded Job Training Work? Nonexperimental Estimates of WIA Training Impacts Using Longitudinal Data on Workers and Firms, http://www.census.gov/ces (accessed on 26 November 2019).

[9] Andriescu, M. et al. (2019), Adult learning policy and provision in the Member States of the EU, Publications Office of the European Union, Luxembourg, http://dx.doi.org/10.2767/82188.

[32] Barnow, B. and J. Smith (2015), Employment and Training Programs, National Bureau of Economic Research, Cambridge, MA, https://doi.org/10.3386/w21659.

[12] Bock-Schappelwein, J. et al. (2014), Aktive und passive Arbeitsmarktpolitik in Österreich und Deutschland. Aufkommen und Verwendung der Mittel im Vergleich, Wifo, Wien, http://www.forschungsnetzwerk.at/downloadpub/AMS_akt_pass_AT_DE_13032014-ENDBERICHT_und_Deckblatt.pdf (accessed on 7 October 2019).

[27] Bock-Schappelwein, J., U. Huemer and A. Pöschl (2006), WIFO-Weißbuch: Mehr Beschäftigung durch Wachstum auf Basis von Innovation und Qualifikation. Teilstudie 9: Aus- und Weiterbildung als Voraussetzung für Innovation, WIFO, Wien.

[13] Cedefop (2019), Labour market training measures provided by the Estonian Unemployment Insurance Fund, https://www.cedefop.europa.eu/en/tools/matching-skills/all-instruments/labour-market-training-measures-provided-estonian-unemployment-insurance-fund (accessed on 7 October 2019).

[19] European Council (2016), “Council Recommendation of 19 December 2016 on Upskilling Pathways: New Opportunities for Adults”, Official Journal of the European Union, Vol. C 484/1.

[1] Fialho, P., G. Quintini and M. Vandeweyer (2019), “Returns to different forms of job related training: Factoring in informal learning”, OECD Social, Employment and Migration Working Papers, No. 231, OECD Publishing, Paris, https://dx.doi.org/10.1787/b21807e9-en.

[8] FiBS/DIE (2013), Developing the adult learning sector. Annex to the final report, FiBS/DIE, Berlin, http://lll.mon.bg/uploaded_files/financingannex_en.pdf (accessed on 30 July 2019).

[33] Greenberg, D. and P. Robins (2008), “Incorporating nonmarket time into benefit-cost analyses of social programs: An application to the self-sufficiency project”, Journal of Public Economics, Vol. 92/3-4, pp. 766-794, https://doi.org/10.1016/j.jpubeco.2007.09.011.

[22] Groot, D. and V. Klaauw (2016), A randomized experiment on improving job search skills of older unemployed workers.

[6] Haaristo, H. and M. Nestor (2014), Programmi "Täiskasvanute tööalane koolitus ja arendustegevused" riikliku koolitustellimuse protsessi ülevaade ja eksperthinnang [Evaluation of the adult training programme and development activities], Praxis, Tallin, http://www.praxis.ee (accessed on 6 July 2019).

[26] Hausegger, T. (2005), Evaluation des Projektes „Outsourcing von Beratungs- und Betreuungsleistungen“, Prospect Research & Solution, Wien.

[30] Holl, J. et al. (2013), “Die Langzeitwirkung von Qualifikationsmaßnahmen des Arbeitsmarktservice”, Sozialpolitische Studienreihe, Vol. 14, http://www.ssoar.info (accessed on 16 July 2019).

[28] Kernbeiß, G., U. Lehner and M. Wagner-Pinter (2006), Bildungskarenz in Tirol. Inanspruchnahme, Zielgruppe und die Auswirkungen auf die Berufslaufbahn., Evaluation for the AK Tirol, Innsbruck, http://www.ak-tirol.com/pictures/d40/Endbericht20060330.pdf.

[17] Länder-Bund-ExpertInnengruppe and „Initiative Erwachsenenbildung“ (2011), Programmplanungsdokument “Initiative Erwachsenenbildung”.

[29] Lassnigg, L. et al. (2011), Evaluierung der Bildungskarenz.

[23] Lauringson, A. et al. (2011), Impact Evaluation of Labour Market Training: the Estonian Unemployment Insurance Fund.

[24] Lutz, H. (2005), Evaluierung der österreichischen Arbeitsmarktförderung 2000-2003, WIFO, Wien, https://www.wifo.ac.at/jart/prj3/wifo/resources/person_dokument/person_dokument.jart?publikationsid=28498&mime_type=application/pdf.

[20] Ministry of Education and Research (2014), The Estonian Lifelong Learning Strategy 2020, https://www.hm.ee/sites/default/files/estonian_lifelong_strategy.pdf.

[5] Ministry of Education Estonia (2012), The Five Challenges of Estonian Education. Estonian Education Strategy 2012-2020, Ministry of Education, Tallinn, https://www.kogu.ee/wp-content/uploads/2013/09/Haridusstrateegia-ENG-spreads-appendix.pdf (accessed on 8 July 2019).

[14] Ministry of Finance Singapore (n.d.), Scope of Future Economy Council. Answer to parliamentary question., https://www.mof.gov.sg/Newsroom/Parliamentary-Replies/Scope-of-Future-Economy-Council (accessed on 9 October 2019).

[7] OECD (2019), Adult Learning in Italy: What Role for Training Funds ?, Getting Skills Right, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264311978-en.

[3] OECD (2019), Getting Skills Right: Engaging Low-Skilled Adults in Learning, http://www.oecd.org/employment/emp/engaging-low-skilled-adults-2019.pdf (accessed on 9 October 2019).

[2] OECD (2019), Getting Skills Right: Future-Ready Adult Learning Systems, Getting Skills Right, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264311756-en.

[11] OECD (2019), Individual Learning Accounts : Panacea or Pandora’s Box?, OECD Publishing, Paris, https://dx.doi.org/10.1787/203b21a8-en.

[10] OECD (2017), Financial Incentives for Steering Education and Training, Getting Skills Right, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264272415-en.

[4] Raad voor Werk en Inkomen (2011), G(oud)! kansen creëren voor werkloze ouderen, https://www.kennisplatformwerkeninkomen.nl/documenten/rapporten/2011/10/01/goud-kansen-creeren-voor-werkloze-ouderen (accessed on 2 December 2019).

[18] Steiner, M. et al. (2017), Evaluation der Initiative Erwachsenenbildung, Institut für Höhere Studien, Wien.

[16] Steuerungsgruppe Initiative Erwachsenenbildung (2019), Programmplanungsdokument Initiative Erwachsenenbildung. Länder-Bund-Initiative zur Förderung grundlegender Bildungsabschlüsse für Erwachsene inklusive Basisbildung, http://www.initiative-erwachsenenbildung.at.

[21] Századvég and E&Y (2016), A Társadalmi Megújulás Operatív Program átfogó ex post értékelése.

[15] Waslander, S., C. Pater and M. van der Weide (2010), “Markets in Education: An Analytical Review of Empirical Research on Market Mechanisms in Education”, OECD Education Working Papers, No. 52, OECD Publishing, Paris, https://dx.doi.org/10.1787/5km4pskmkr27-en.

[25] Winter-Ebmer, R. (2001), “Evaluating an Innovative Redundancy-Retraining Project: The Austrian Steel Foundation”, IZA Discussion Papers No. 277, http://hdl.handle.net/10419/21129.

Note

← 1.  Austria is the only federal state included in this study.