With the help of a cross-ministry research instrument, the Ministry of Economic Affairs and Employment has been able to meet most of its needs for research on active labour market policies (ALMPs), despite its somewhat modest and fragmented internal resources. Counterfactual impact evaluations have been conducted for most of the key ALMPs over the past years and the operative needs for policy design have been largely covered, although the evidence has been at times dismissed by policy makers referring to its inconclusiveness. A rich set of data are available to be used remotely and securely for ALMP evaluation via Statistics Finland. Nevertheless, the use of data specific to ALMPs suffers due to outdated IT infrastructure supporting ALMP provision and costly access. Moreover, accessing data beyond what is readily available in Statistics Finland can be cumbersome and delay research projects and policy making significantly.
Evaluation of Active Labour Market Policies in Finland
3. Assessment of the system of active labour market policy evaluation in Finland
Abstract
3.1. Introduction
This chapter assesses the system Finland has put in place to evaluate active labour market policies (ALMPs). Knowing what measures and services work, for whom and in what context, is crucial to scale up effective policies as needed and adapt or terminate inefficient policies, especially in the context of limited budgets. Generating evidence and designing policies based on evidence is not only important for specific labour market services (such as counselling services) or measures (training, employment incentives), but also across the tools, processes and approaches used to provide ALMPs. For example, many countries have significantly modernised their IT infrastructure over the past few years to provide ALMPs (OECD, 2022[1]), as well as service delivery models (OECD (2021[2]; 2021[3])) in addition to adjusting the design of specific measures (OECD, 2021[4]), which all need to be evaluated to support continuous improvement. Also changes in the institutional set-ups of ALMP provision need to be evaluated to determine the right path forward (Lauringson and Lüske, 2021[5]). Hence, this chapter refers to ALMP evaluations as evaluations of any dimension of a system to provide ALMPs.
This chapter discusses first the strategy and processes of evidence‑based policy making for ALMP provision, focusing particularly on the plans, resources and activities of the Ministry of Economic Affairs and Employment (TEM), which is the ministry responsible for ALMPs. Second, the chapter discusses the implementation of ALMP evaluations, such as designing, procuring and steering research projects. The last section of the chapter discusses the data available to conduct ALMP evaluations, particularly the rich data that are available for research via Statistics Finland.
3.2. Strategy and processes of evidence‑based policy making
Evidence‑based policy making needs to be systematic and involve the whole cycle of designing policies together with the monitoring and evaluation frameworks, implementing policies, evaluating policies to generate evidence, disseminating evidence, and adjusting policies based on evidence (Figure 3.1). In addition, the evidence generation process itself needs to be assessed and adjusted then accordingly. The cycle of evidence‑based policy making can be implemented successfully only if researchers, policy makers policy implementers and other relevant stakeholders work together.
Figure 3.1. Evidence‑based policy making is a systematic co‑operation process
Note: TEM – Ministry of Economic Affairs and Employment.
This section reviews the strategy to evaluate ALMPs in Finland and the resources allocated to implement the strategy, as well as the practices to disseminate evidence on ALMPs and how this evidence is used for policy making. The aspects of implementing ALMP evaluation are discussed in the following two sections.
3.2.1. Fragmentation in research activities and a missing longer-term strategy can lead to gaps in evidence
TEM is responsible for ALMP design and the organisation of ALMP implementation in Finland. To design ALMPs and their delivery in an evidence‑based manner, the analysis and research activities of TEM need to collect and generate sufficient and relevant evidence.
TEM uses a decentralised model for its analysis and research activities, which can lead to people with more diverse skills and expertise conducting these tasks but might also cause fragmentation in knowledge generation. Tasks related to policy analysis are scattered across policy areas and teams. Furthermore, analysis is mostly a part of tasks for a staff member (often only 10‑20% of working time), rather than a full-time job. This enables people who are experts in specific policies, also to engage in the related analysis. Nonetheless, it is difficult to ensure in such set-up systematic knowledge generation, avoid duplication in analysis and ensure that data and knowledge sharing are streamlined for maximum efficiency. To lower the risk of fragmentation, TEM has established a co‑ordination group for policy analysis, involving staff from its different units.
Due to the decentralised model for analysis and the low number of staff dedicated to analysis and research, TEM has decided to no longer prepare and implement a long-term research strategy. Currently, TEM’s annual research plan is the strategic document on which evaluation and research activities are based, above all concerning research to be contracted out. Each TEM division makes its proposal for the annual research plan. The Permanent Secretary of TEM decides on the budget allocation across the divisions, optionally after the management team of TEM has discussed the research plan (on high level rather than individual projects). Hence, both the research side and the policy makers are involved in developing the annual research plan. The decentralised model for analysis further supports developing an annual research plan proposing projects that would be analytically feasible and politically relevant.
TEM should consider re‑establishing its longer-term strategic view (i.e. several years ahead) on research and evaluation activities to be able to manage and develop its analytical capacity, and above all, ensure continues improvement in evidence‑based policy making. In addition to specific research projects in the annual plan, the longer-term strategy should outline the objectives and priorities of research activities longer term, and the milestones to reach the objectives, considering both internal capacity and contracting out. At the moment, a substantial part of policy analysis takes place internally and ad‑hoc, not planned even in the annual plan for analysis. Such analytical tasks are undertaken when new amendments to laws are drawn or quick inputs for policy making are needed. On the one hand, this ensures flexibility for the analytical capacity to meet the needs of policy making, contributing to evidence‑based policy design. On the other hand, a lack of a more strategic view on evidence generation can be hindering the potential for continuous improvement, might not ensure these activities receive the attention and priority they need, and leave gaps in evidence to inform policy making.
Nevertheless, TEM has used some approaches to research systematically over the years, regardless of not being fixed in its strategic documents. Above all, one underlying principle for labour market research has been to contract major policy evaluations out to external researchers. Internal capacity for analysis is reserved mostly for operational needs, such as quick insights on labour market situation or simpler ex-ante policy evaluations. It is assumed that externally conducted research ensures more objectivity and credibility among the stakeholders of the ALMP system, the policy makers and the society more generally. Following this principle, all counterfactual impact evaluations of ALMPs have been in the past years conducted by external researchers.
Encouraging external research on ALMPs is indeed important to have a large evidence base for policy making, taking an advantage of a larger set of skills and knowledge available on the market and bringing more transparency, quality and credibility in evidence‑based policy making. Nevertheless, good understanding of ALMP evaluation is needed internally in TEM to ensure good inputs to design the procurement of research, be able to communicate with the external researchers as equals, control research quality and feed research results into policy design. The best way to acquire and maintain such knowledge and skills for TEM staff would be to also conduct such evaluation activities in some scale in‑house. For example, the Employment and Social Development Canada (ESDC)1 fulfilling a somewhat similar role as TEM in Finland, has developed a high capacity to conduct systematic ALMP evaluations in-house (OECD, 2022[6]). However, as no data are shared by the ESDC with external researchers and no research is out-sourced, it can raise issues of the credibility of evidence (see Box 3.1).
Box 3.1. The system to conduct ALMP impact evaluations in Canada
The Canadian Ministry of Employment, ESDC, internally produces high-quality counterfactual impact evaluation of its ALMPs. ESDC has invested over the years to build an internal evaluation directorate to deliver counterfactual impact evaluation of its ALMPs. It conducts evaluations using its linked administrative data and quasi‑experimental techniques such as propensity score matching and difference‑in-difference analysis. Removing the use of survey-based data and external contractors for delivery of the analysis facilitated much reduced analytical production times and cost savings of around CAD 1 million per annum.
Canada has an institutional set-up in the delivery of its ALMPs that offers potential insights for Finland when the latter introduces its 2024 reform to increase responsibility for municipalities for delivery of ALMPs. Canada discharges the responsibility for delivery of the majority of its ALMPs to its 13 provinces and territories. Rolling cycles of ALMP evaluations are primarily managed through a Joint Evaluation Committee, comprising federal and provincial officials from the 12 participating regions. This sets the objectives for each cycle of evaluation and monitors progress of the analysis. The committee ensures that decisions are made collectively and contributes to a common vision for progress. The analysis is then conducted on behalf of the participating regions via the central evaluation function in ESDC. The result of this co‑ordinated approach is that ESDC has been able to progressively build high-quality evidence on the effectiveness of its ALMPs across Canada, both at national and regional level, and publish all of its evaluation work on the Government of Canada website.
There are data management features within the process of evaluation that offer insights for Finland. The transfer of ALMP data from provinces and territories to the federal government allows a streamlined and efficient centralised evaluation, which minimises any additional work by individual provinces and territories. In addition, detailed cost-benefit analyses allow comparison of ALMPs with very different cost and benefit structures. Despite the lack of individual-level participant costs, estimates are formed using aggregate costs and participant numbers. Taxes, benefits and government financing implications are incorporated into this analysis, and soon also health data.
One of the drawbacks of the current approach is that not all the necessary data for ALMP evaluation are available to external researchers. This means that analysis cannot be externally corroborated, and it is not currently possible to benefit from potential innovations and additional evidence generation that these researchers could provide.
Source: OECD (2022[6]), Assessing Canada’s System of Impact Evaluation of Active Labour Market Policies, https://doi.org/10.1787/27dfbd5f-en; Government of Canada (2022[7]), Evaluation reports, https://www.canada.ca/en/employment-social-development/corporate/reports/evaluations.html.
3.2.2. Resources for knowledge generation on ALMPs are not sufficient in the Ministry of Economic Affairs and Employment
In Finland, ALMP impact evaluations are not only contracted out by TEM, but also via a cross-ministerial initiative, as well as occasionally conducted by the Ministry of Finance (the latter is not responsible for conducting ALMP evaluations per se, but needs to evaluate policy impacts on public finance, co‑operating with other ministries as necessary). The differences in the capacities of these organisations to conduct research raises however a question, whether TEM can be indeed the driver of policy making in the field of ALMPs. This section discusses the capacity and resources to conduct research on ALMPs in each of the three organisations separately.
TEM lacks the necessary internal capacity to conduct ALMP evaluations
TEM currently employs only one full-time researcher (the Research Director of Employment and Well-Functioning Markets Department) focusing on ALMPs among other topics. Furthermore, a shift in the concepts and terminology of the Finnish public sector has taken place since the 1990s, which aims at more administrative job titles and tasks in the ministries. Hence, rather than researchers, people in charge of analytical activities have the job titles of senior specialists and Ministerial advisers.2 The Research Director of Employment and Well-Functioning Markets Department is the key staff member to design research projects for contracting out to external researchers and conduct some analysis internally in TEM.
Although some ministries in Finland employ economists who have the capacity to conduct credible policy evaluations, TEM employs only few economists. Firstly, this is the result of the TEM’s hiring policy that aims to have people who could fit in its decentralised model for analysis and simultaneously contribute to the perceived core tasks of the ministry, i.e. policy design and the preparation of changes in legislation. Thus, most of the workforce in TEM has a rather homogenous qualification and skill set, above all related to legal affairs. Secondly, the lack of economists is the result of a tight budget to cover internal needs for analysis, and wage levels that are not competitive on the market. In these conditions in the past years, it has been difficult for TEM to hire and maintain staff with sufficient knowledge, skills and experience to conduct analysis using more advanced econometric methods, such as counterfactual impact evaluations of ALMPs. As TEM has only few researchers and economists, there is also only a limited number of staff that is able to design research projects for contracting out and steer these projects in partnership with external researchers. A limited number of staff for analytical tasks means that some of the needs for evidence are not met, as the staff are able to address only the more urgent needs for policy analysis that do not require intensive data analytics. Rather than conducting data analysis, the analysists might propose proxies for policy impacts based on previous related Finnish and international research, which is often the only feasible solution considering the tight resources. TEM also has a shortage of related skills, such as data scientists to support the preparation of data for data analytics (e.g. defining data needs for the IT developers that TEM works with, making the data available for analysts across TEM).
To overcome the challenges of low resources for internal analysis, the analysts in TEM have taken steps to increase the analytical capacity internally. In general, the staff do not have time and resources to individually build up their capacity for more advanced analysis and be current with the latest tools and methods of policy evaluation. Thus, the staff more involved and advanced in policy analysis, occasionally organise internal training for their colleagues, particularly before a joint analysis is about to start. Internal training on interpreting research results and research terminology is conducted sometimes for staff that need to use the evidence in policy design, while not being data analysts themselves. External researchers are invited sometimes to participate and present in TEM seminars. Nonetheless, to fully overcome the capacity challenges, TEM needs to additionally allocate more resources for analysis and research, and preferably hire some economists who could support these functions full time.
The budget of TEM for external research is flexible, but insufficient
The annual research plan is the main strategic document in TEM that defines the research to be contracted out and the allocated budget across policy fields. After the allocations have been assigned by the Permanent Secretary of TEM, each division can decide which of their projects and how to implement.
The total budget of TEM to contract out research projects has been around EUR 1.5 million over the past few years (for comparison, the operating expenditures of TEM in 2022 are EUR 38.2 million (Ministry of Economic Affairs and Employment, 2022[8])) and the topic of labour market policy is just one of the many topics covered. Furthermore, a large share of the research budget goes on annual reviews, such as annual labour market reviews and barometers. Other projects financed from the research budget are mostly small-scale projects aiming to feed into the preparation of specific changes in legislation. This leaves only a low budget to outsource actual research, and even less to outsource ALMP evaluations.
The research budget has some flexibility, and divisions can make changes within their budgets in case policy priorities change. The co‑ordination group of research activities that involves representatives from the different divisions of TEM oversees the management of the research budget. In case needs emerge for major research projects, some re‑allocation of research budget between the divisions is possible.
The cross-ministry joint research instrument is critical for strategic needs for knowledge, but does not help with systematic ALMP evaluations
Since 2014, Finland implements a cross-ministry research instrument – the joint analysis, assessment and research activities (VN TEAS) – co‑ordinated by the government (the Prime Minister’s Office) (Prime Minister’s Office, n.d.[9]). This instrument aims to cover strategic research needs that are relevant across policy fields, ensuring strong horizontal knowledge base to support decision-making and the implementation of the Government’s Programme. The government working group for the co‑ordination of research, foresight and assessment activities (TEA Working Group) consisting of representatives across the ministries in Finland, makes a proposal for the VN TEAS annual plan to the Prime Minister’s Office (Prime Minister’s Office, n.d.[10]).
The VN TEAS 2022 plan touches upon labour market issues within its nine main research topics, particularly within “Finland built on trust and labour market equality”, “Fair, equal and inclusive Finland” and “Finland that promotes competence, education, culture and innovation” (Prime Minister's Office, 2021[11]). Moreover, the 2022 plan foresees meeting additional demands for research needs based (“Other information needs” as a tenth topic left to be defined over the year).
The budget to implement the 2022 plan of the VN TEAS is EUR 9 million (Prime Minister's Office, 2021[11]) and the volume of the budget has been similar during the past years. This allows to conduct substantial research projects, reaching occasionally even close to half a million euros, which would be difficult to achieve for a single ministry. It can be particularly helpful for smaller ministries that do not have a dedicated research budget. While the short projects funded via the VN TEAS last for a few months, the projects can be often longer, even up to three years. Due to the budgets and timelines, the research funded by the VN TEAS enable to conduct projects needing intensive data analytics, including policy evaluations requiring advanced econometric methods.
The research organisations to conduct the projects of the VN TEAS are selected through a public procurement process (open call). The procurement process aims at transparency and quality using the following selection criteria for tenders: relevance, usability, project quality, expertise of the project implementers, adequacy of resources, communication and information management (Prime Minister's Office, 2021[11]). Often several research organisations work together on a project (in collaboration with the TEA Working Group and the ministries responsible for the specific topic), making it possible to involve a more varied expertise and increase research quality. The strategic importance of the projects, the possibilities to use advanced data analysis and the funding scheme make the projects appealing for the research community, enabling the VN TEAS to select the best research organisations in Finland to conduct the analyses. To further ensure the research quality, the VN TEAS has implemented a peer review process (research results are reviewed by another researcher) in some of its latest projects.
In practice, the research topics and projects to be included in the annual plan of the VN TEAS follow mostly a bottom-up approach. The research needs are generally identified by ministries individually and proposed to the TEA Working Group. In some cases, the research ideas have been developed in open co‑operation of several ministries. All proposals are assessed by the TEA Working Group and can be included in the annual plan only if several ministries consider the research need relevant. As the TEA Working Group comprises both researchers (research directors) and high-level policy designers, the projects included in the annual plan have a high potential to feed well into the cycle of evidence‑based policy making.
The project assessment process by the TEA Working Group ensures that only strategic research needs of cross-ministry relevance get funded via the VN TEAS. There is no certainty of acceptance or the budget allocation in the proposal stage as the TEA Working Group gets many more proposals for research needs than the budget enables to cover. Hence, TEM aims to make its proposals usually beyond its own specific needs, tying in related research questions from other policy fields and get other ministries interested in these projects. TEM has been successful in getting some of its research needs covered within the VN TEAS on several occasions, including regarding ALMP impact evaluations, thus not needing to cover this part of evidence generation within TEM’s annual research plan and internal research budget.
Nevertheless, the VN TEAS might benefit the research needs of some ministries better than others, due to both political priorities, as well as specificities of some policy fields. In any case, the VN TEAS cannot be the only or even the main instrument to fund systematic ALMP evaluations due to its nature. Yet, it can be instrumental in generating evidence in case of more major reforms of ALMPs that could potentially affect other policy areas, such as education, social and health policy.
The Ministry of Finance helps to overcome some gaps in evidence, but does not necessarily meet the strategic needs of other ministries
The Ministry of Finance has a higher capacity to conduct policy evaluations than many other ministries in Finland. Contrary to TEM, the Ministry of Finance employs a substantial number of economists, some of whom have the skills to conduct counterfactual impact evaluations of policies, such as using matching methods or regression discontinuity design. Developing such a team has taken place over several years, as similarly to TEM, it has been difficult to hire experienced researchers at the prevailing public sector wages.
As the Ministry of Finance has higher analytical capacity, it has been recovering some of the other ministries’ research needs. Above all, the Ministry of Finance conducts policy evaluations in the fields that it finds politically relevant, that affect the state budget and where evidence generated by the responsible ministry or external researchers is not considered to be sufficient. In this context, the Ministry of Finance has recently conducted an impact evaluation of self-motivated training provided to jobseekers (the results have been shared with other ministries but have not made publicly available).
While research conducted by the Ministry of Finance can overcome some gaps in evidence generation in the framework of limited research budget in TEM, this mechanism does not ensure that evidence generation on ALMPs meets the strategic needs of TEM. As TEM is the policy designer in the field of ALMPs, it needs to assign sufficient resources for related knowledge generation to be able to drive labour policy and do so based on evidence. Research on ALMPs initiated by other organisations needs to be taken into account in policy making, but this channel cannot be assumed to cover TEM’s strategic needs for research.
3.2.3. Despite systematic dissemination, evidence does not always feed into policy design
This section discusses the dissemination and communication of evidence, as well as the challenges for evidence to reach policy makers.
Research results are published systematically
In addition to publishing the main research reports on the TEM website, the research and analysis results on labour policy issues are systematically published in the dedicated publication series of TEM – Finnish Labour Review (Työpoliittinen aikakauskirja).3 The periodical is published four times a year, comprising articles written by TEM internal analysts and economists, as well as external researchers (including research that has not been funded by TEM). The articles discuss the issues related to ALMP evaluation and changes in the ALMP system in Finland from the ex-ante and ex-post perspective, among other topics.
In addition, TEM has a publication series for analyses across TEM policy fields, TEM analyses (TEM-analyyseja).4 This series publishes the results of these analyses and research projects that are conducted internally by TEM. Many of the analyses in this series are regular reviews of specific topics, such as labour market forecasts (for example Alatalo et al. (2022[12])) and overviews on job creation (for example Räisänen and Ylikännö (2021[13])). One of the annual reviews provides an overview of employment rates three and six months after participation in different key ALMPs (Tuomaala, 2021[14]). This review labels some of the ALMPs, somewhat misleadingly to be better than others, by presenting only gross effects and not (net) counterfactual effects. It should be communicated more clearly in this report, including in its abstract and executive summary, that participants in business start-up subsidy and work-related rehabilitation are likely to have very different employment prospects, which does not directly allow to compare the gross effects on employment (or even net effects unless the groups of participants in different measures are balanced between them).
Both the Finnish Labour Review and TEM analyses series publish extensive statistical overviews of key labour market statistics. Labour market statistics, including the statistics on ALMPs are also published in the Annual Statistical Bulletins of TEM (Tilastotiedote‑vuosijulkaisut).5 All these statistics are published in PDF format. The monthly Employment Bulletin6 provides somewhat higher user-friendliness and interactivity and covers the key statistics of ALMPs. In addition, these statistics are available via the database of Statistics Finland,7 which allows the data user to customise and download the data of interest.
The research conducted by the VN TEAS has its dedicated publication series managed by the Prime Minister’s Office directly – Publication series of the Government’s analysis and research activities (Valtioneuvoston selvitys- ja tutkimustoiminnan julkaisusarja) – accompanied by shorter Policy Briefs.8 In this series, several counterfactual impact evaluations of ALMPs have been published over the years. These include the evaluation of jobseeker counselling (Valtakari et al., 2019[15]), wage subsidies (Asplund et al., 2018[16]), training (Alasalmi et al., 2022[17]) and two evaluations across several key ALMPs (training, wage subsidies, work-related rehabilitation by Aho et al. (2018[18]); and wage subsidies, training and business start-up subsidies by Alasalmi et al. (2019[19])).
Communication around publication is less systematic
The TEM communication department supports TEM analysts in disseminating their newest reports to media via press conferences and press releases. The press conferences are occasionally organised to disseminate the key results that could be of wider public interest.
There are no systematic dissemination channels between analysts and policy makers in TEM. The initiative to learn about the new research results comes occasionally from the minister and key policy designers in TEM, in which case they are briefed about the research outcomes in a meeting or seminar, which can be supported with a short policy brief. A difference in the qualification background between economists and lawyers drafting legal amendments in TEM has been one of the hurdles in communicating the results. However, some dissemination between analysts and policy designers takes place indirectly due to the decentralised model for analysis, as the analysis tasks are often only a small fraction of tasks for a staff member and the majority of tasks are more related to policy design.
To support evidence‑based policy design and implementation, analysts in TEM need to take initiative to disseminate the analysis results more systematically to policy makers, policy implementers and the broader public. The content and channel of communication need to be defined based on the specific audience. For example, the communication to policy designers might need to focus less on the evaluation methods, but more on policy design elements needed to be changed. Communication to policy implementers might need to take a form of guidelines for employment counsellors.
Policy making concerning ALMPs is not thoroughly evidence‑based
The systems to develop the annual research plan in TEM and the annual plan of the VN TEAS ensure that the undertaken research projects have high political relevance and that their results are needed for policy making (see Sections 3.2.1 and 0). Thus, it is common that projects to generate evidence are initiated before major changes in ALMP design or in the ALMP system more generally. Furthermore, piloting and experiments in the field of ALMPs are relatively more common in Finland than in many other OECD countries, enabling to collect credible evidence.
However, the evidence gained via the research projects of TEM and the VN TEAS are not systematically taken into consideration in policy design. The main reason behind this is the political ambition or need to make changes faster than the evidence can be generated, partly affected by election cycles. For example, before the major institutional reform currently taking place in the ALMP system in Finland was approved in September 2021 (see Chapter 2), a pilot to test the new set-up was started. Nevertheless, the decision to go on with the reform was taken before the evaluation results of the pilot became available (the pilot only started in March 2021 and was planned to end in June 2023). The decision was based on ex-ante evaluations of the reform (drafted by the Ministry of Finance in co‑operation with TEM) and evidence from other countries (above all Denmark), but not the pilot that Finland had carefully planned and designed, and even allocated additional funding for in the midst of the COVID‑19 pandemic challenges (Ministry of Economic Affairs and Employment, 2020[20]). Furthermore, the decision was potentially affected by the ambition of the more capable municipalities to take on additional responsibilities.
In addition, there can be other factors that contribute to a somewhat weak link between evidence and policy making, such as:
Insufficient dissemination of the research results of TEM among policy makers as discussed in the previous subsection. The research results are communicated to the policy makers in case of their initiative, but not systematically vice versa. The TEA Working Group aims to disseminate the research results of the VN TEAS and facilitate the use of this evidence in policy design among its key tasks. Nonetheless, there is no mechanism to ensure that evidence generated in the VN TEAS is taken into account in policy design.
The evidence is often dismissed by policy designers referring to its inconclusiveness due to the methodology, such as quasi‑experimental design of ALMP impact evaluations.
Some of the evidence has been dismissed due to unexpected research results, not coinciding with the expectations and assumptions of the policy makers.
Gaps in evidence generation due to a missing longer-term strategic view (3.2.1) and limited resources for systematic ALMP evaluation (0).
As a result, major reforms can be initiated without sufficient ex-ante knowledge about the upcoming changes or ex-post evidence on the previous reforms.
To continuously improve the quality of draft laws and particularly the impact assessments in government proposals, the Finnish Council of Regulatory Impact Analysis (FCRIA) was established in 2016 (Prime Minister's Office, n.d.[21]). The FCRIA issues around 30 to 50 statements each year concerning draft laws and draft amendments. Although located in the Prime Minister’s Office, the FCRIA is an independent body and chooses which legal proposals to assess. Above all, this council assesses whether the impact evaluations of the proposed legal changes (generally ex-ante evaluations) are appropriate and makes proposals to improve them. The statements issued by the FCRIA are not binding for the ministries, but by making these publicly available, these potentially do encourage the ministries to consider improving the evaluations in the draft laws. Nonetheless, the FCRIA does not conduct impact evaluations itself or assesses whether indeed (all) relevant evidence has been taken into account in the legal proposal.
To further improve the link between evidence and policy design, Finland could consider extending the tasks of the FCRIA to monitor that all relevant evidence available has been indeed used as inputs in the draft legal proposals. As a key priority, the FCRIA could check whether above all the research results of the VN TEAS have been considered, as these are ought to be the key strategic research projects. Furthermore, the FCRIA and the VN TEAS could co‑operate in exchanging information on on-going research projects and draft legislation being assessed to avoid a situation where a major legal change is taking place while the evidence generation is still in progress.
3.3. Implementation of research projects
This section discusses the procurement process in TEM to outsource research projects, the practices to design and steer research projects, and the types of ALMP evaluation implemented via these processes.
3.3.1. Procurement process to outsource research is well-established and transparent
The key parts of the calls for tenders for research issued by TEM are the research questions, the expected cost and the timeline. Sometimes other research components are established as well, such as which data are expected to use. The procurement process to outsource research in TEM follows the national procurement legislation. The exact process depends on the foreseen cost of the research project, as smaller projects costing below EUR 60 000 can follow somewhat simpler process of a small-scale procurement and projects exceeding the thresholds set by the EU need to follow the EU procurement process. Generally, the TEM research projects fall under the regular or small-scale procurement processes.
TEM focuses on quality components in its assessment criteria in procurement, selecting the most economically advantageous tender. The price component weighs usually about 10‑20%, while the key component is the research plan. The assessment criteria also often include components like competency of the research team and the communication and dissemination plan.
The calls for proposals receive usually in between two to five applicants and joint applications. Regarding research on ALMPs the applications usually involve universities (e.g. Tampere University), research institutes (e.g. ETLA Economic Research, VATT Institute for Economic Research, the Labour Institute for Economic Research LABORE, PTT Economic Research), consulting firms in case of more qualitative research, as well as lately some research and consulting organisations from other countries (such as Copenhagen Economics).
The assessments of tenders are considered to be very transparent by the applicants (research organisations). Thus, there have been almost no cases of contesting the assessment results over the years. The assessment process involves several people from TEM who need to assess each tender by the evaluation criteria and provide a written assessment statement. The official in charge of the assessment collects and compiles the different written assessments.
3.3.2. Designing research projects is an internal process in TEM
The process to develop the description of the research project for contracting out takes place internally in TEM. First, the divisions define their needs and research questions. Then, the final formulation and description of expected methodology are finalised, led by the TEM official assigned to be in charge of the research project. External researchers are not involved in the design of the project descriptions for procurement as the process needs to be transparent and fair for any potential bidder. Hence, TEM rather organises more general research seminars to exchange ideas with external researchers on ALMP analysis and evaluation, rather than looks for explicit ideas and proposals from them for specific projects.
However, also research and analysis projects to be conducted internally are designed and conducted only internally. In these cases, the design and methodology could be discussed more explicitly and openly with external researchers, for example within the TEM research seminars. Furthermore, discussions with external researchers could be particularly important when designing major trials and pilots even if the evaluation would be later contracted out. It is critical to design trials and pilots correctly to generate credible evidence. The external researchers might be interested in contributing to these discussions without remuneration (their incentive could be to simply have a say in politically relevant topics and have a better awareness of policy developments) and the information on the design could be shared in the later procurement documents to ensure equal opportunities for the external researchers.
Developing the research projects within the VN TEAS for procurement involves generally more stakeholders. Above all, staff from different ministries are involved to design projects that satisfy the cross-ministry needs for evidence (although generally one of the ministries initiates a project with its research questions, and other ministries are asked to join the project). As project design involves researchers from different ministries and policy fields, it helps to compensate somewhat for not involving external researchers in designing research projects.
3.3.3. Outsourced research projects sometimes underestimate the timeline to generate credible evidence
According to external researchers, the greatest issue with the design of research projects in the field of ALMPs, as well as with research projects in Finland more generally, is the foreseen project timeline. As policy makers expect the evidence quickly, the timeline in the procurement documents sometimes underestimates the time it takes to generate credible evidence. The analysts in TEM need to create more awareness among the policy makers on the feasible timelines of research projects, such that the procurement documents would not foresee ex-ante evaluations of changes in legislation where the agreements among the policy makers on policy design have not been reached yet; or ex-post impact evaluations of policies that have been in force only recently and where the impact could not be identified yet.
A particularly critical issue with the timelines for the research projects concerns the timeline expected to access the relevant data. While rich data are available via Statistics Finland, getting access to these can take a while (see Section 3.4). The timeline is even longer in case additional data need to be linked with the data in Statistics Finland. Furthermore, the datasets in Statistics Finland regarding ALMPs and the labour market have a considerable time lag, which makes them generally unfit to be used for evaluating recent policy changes. These circumstances mean that either the research projects cannot be evaluated within the foreseen timeline or there are only a few research organisations that are better placed to deliver the evaluation faster as they have a contract on the remote use of data with Statistics Finland on permanent terms (e.g. the University of Helsinki; although the scope of data access needs to be identified again for each specific project). This in turn puts the potential bidders to research tenders in unequal position.
As a solution to the problems with foreseen timelines to conduct research, TEM extends the project timelines when the researchers make it evident that the proposed timelines are not feasible. In addition to improving the planning of research projects and raising awareness on the feasible timelines among policy makers, Statistics Finland and administrative registers need to continue their efforts to shorten time lags in data availability (see Section 3.4).
3.3.4. Good project steering practices in TEM are the main tools to ensure sound methodology and quality in evidence
As TEM aims to select the most economically advantageous tender when outsourcing research, the key component to be assessed is the proposed research plan, and particularly the proposed methodology to answer the research questions. If necessary, additional details on the methodology are requested from the tenderers to ensure that sufficient information would be available to select the most appropriate approach. A contract is signed with the successful tenderer, which makes the accepted research plan and research methodology binding for the project.
A dedicated steering group is set up for each contracted research project in TEM, involving relevant experts from policy design as well as analysis. The steering group is guiding and monitoring the research projects, ensuring that the external researchers have all of the relevant information available and that the generated evidence is sound for policy making. The steering group also monitors that the project is following the accepted research plan and methodology, although minor changes can be enabled if necessary (e.g. some data cannot be accessed). The TEA Working Group is co‑ordinating and monitoring the implementation of the research projects of the VN TEAS.
In a few more recent research projects, the VN TEAS has started to explore peer reviews as an additional tool for quality control. In such a case, external researchers others than those who conducted the research project, review the methodology used and provide their opinion on the credibility of the research results. On the one hand, this approach can increase policy designers’ confidence to use the generated evidence in policy making. On the other hand, it can create additional concerns, such as how to select the peer reviewers (whether researchers who also made a tender to conduct the research could be later peer reviewers or whether also peer reviewing needs to be procured) and whether peer reviewing should be remunerated increasing slightly the project costs.
TEM aims to ensure research quality via sound procurement process and project management. Furthermore, the research results are published by TEM and researchers are encouraged to publish the results in academic journals. The transparency of results also acts as a tool for quality control as these are available for the research community to publicly discuss and criticise. In addition, the academic journals generally also apply peer reviewing before publishing the papers. Hence, the academic community is safeguarding the level of research.
The ESDC in Canada uses external peer reviewers for quality assurance. The ESDC consults independent academic experts on methodology while conducting research, as well as asks them to review the research reports (OECD, 2022[6]). However, contrary to TEM, the ESDC conducts all ALMP evaluations exclusively in-house, so peer reviewing concerns the research done by the analysts in the ESDC, not by other external researchers. In addition, the reports of ESDC are written in English/French, which enables them to engage peer reviewers from a wide pool of academic researchers internationally. In the case of TEM, it generally might not make sense to translate the research reports fully into English as the main aim is to support policy making in Finland and translation would further increase project costs.
TEM could consider peer review process for the research projects it conducts internally. In addition to discussing methodologies and results generally, TEM could invite external researchers to review the research reports written by TEM analysts in more detail.9 Moreover, if this approach could be used, TEM could consider conducing ALMP impact evaluations internally as these are at the moment fully contracted out due to objectivity concerns. In case peer reviewing by external researchers would be systematically applied, the research objectivity criteria could be still met if some of the research would be conducted internally. Internal analysis with external peer reviewing could be, for example, used when the evidence needs to be generated quickly, as TEM has then better control over data access, as well as the timeline more generally. This approach could also help TEM to have a better control over the research cost, enabling potentially remunerate peer reviewing within the research budget. Nonetheless, conducting ALMP evaluations internally might need building the capacity first for this kind of research, above all by hiring more staff with appropriate skills and knowledge (see Section 0).
3.3.5. Impact evaluations of ALMPs will gain more credibility by applying the experimental design more often and adding cost-benefit analyses
A considerable volume of evidence on ALMPs has been generated via the different funding mechanisms over the past years. Some evidence on the effectiveness of most of the key ALMPs (jobseeker counselling, training, wage subsidies, work-related rehabilitation, business start-up subsidies) and a meta‑analysis of Finnish and international labour market policy evaluations are publicly available via the research reports of the VN TEAS (Aho et al., 2018[18]; Alasalmi et al., 2019[19]; Valtakari et al., 2019[15]; Asplund et al., 2018[16]; Alasalmi et al., 2022[17]; Alasalmi et al., 2020[22]).A few additional counterfactual impact evaluations of ALMPs (above all training) have been conducted by the Ministry of Finance, although they are not publicly available. The research and analysis conducted and outsourced by TEM provides inputs for policy making via ex-ante evaluations and descriptive analysis of ALMP measures and services, as well as reforms and digital tools in the Finnish ALMP system.
The impact evaluations of ALMPs use counterfactual evaluation methodology,10 combining quasi‑experimental11 and experimental designs. The impact evaluations of ALMP measures and services conducted so far are using mostly quasi‑experimental evaluation methodology, almost exclusively propensity score matching. Experimental design has been used to trial issues like the institutional and organisational set-up of ALMP provision and job-search training.
As more evidence from ex-post evaluations has become available, the quality of ex-ante evaluations has increased. Ex-ante evaluations use increasingly the previous impact evaluations as inputs, aiming to predict the changes in policy effectiveness due to changes in policy design (legal changes). In addition to the results of previous impact evaluations in Finland, evidence from other countries is used, particularly if evidence from Finland is not available.
Although Finland is already using experimental design for ALMP impact evaluation more than many other OECD countries, applying pilots and trials could be used even more to generate credible evidence. At the moment, evidence generated using quasi‑experimental design is often criticised or even dismissed by policy makers as being inconclusive. Testing and evaluating new ALMP designs, (digital) tools for employment services and business models for ALMP provision could be piloted and trialled more systematically before nation-wide rollout to fine‑tune the design and be confident in the effectiveness, as well as have more credible inputs for future ex-ante evaluations. Particularly the randomised controlled trial design (RCT)12 could be the evaluation design to be aimed at, as it is considered to be the gold standard of impact evaluation. The results of a well-designed and correctly implemented RCT are more credible, because unlike quasi‑experimental design, these results are not exposed to selection bias (OECD, 2020[23]).13
While impact evaluations have been conducted about most of the key ALMPs in Finland, there is scope for conducting systematically cost-benefit analyses to demonstrate the value added of different ALMPs more explicitly. Cost-benefit analysis builds on counterfactual impact evaluation, examining the impact of ALMPs in relation to the costs of implementing the ALMP and, if possible, the opportunity costs for participants (e.g. foregone earnings) as well as indirect costs on non-participants (e.g. negative externalities). Cost-benefit analysis demonstrates whether the funding invested in ALMPs could generate benefits for the society exceeding the investments. For example, the ESDC in Canada (OECD, 2022[6]) and the Department for Work and Pensions in the United Kingdom (2014[24]) complement their ALMP impact evaluations systematically with cost-benefit analyses. In Finland, there has been one research project conducted under the VN TEAS (Alasalmi et al., 2019[19]) that in addition to evaluating the impact of different ALMPs, also aims to estimate the direct and indirect costs of unemployment, although not conducting a full cost-benefit analysis to assess the value added of ALMP provision. Another more recent research project outsourced by the Audit Committee of the parliament conducts a cost-benefit analysis of a range of key ALMPs using aggregate expenditures on ALMPs to derive estimations (Alasalmi et al., 2022[25]). Conducting cost-benefit analyses is hindered in Finland due to the difficulties to access comprehensive cost data on services, measures and benefits for research purposes as these data are scattered across many different registers and so far not made available on individual level via Statistics Finland. The register owners, researchers and Statistics Finland need to define the set of cost data relevant for cost-benefit analyses of ALMPs and invest in making these available together with other datasets on ALMPs and jobseekers for researchers (see the next section for further discussions on data availability for research purposes).
3.4. Data used for ALMP evaluation
This section discusses the quality and accessibility of administrative data relevant for conducting ALMP impact evaluations. Administrative data are generally preferred data source to conduct impact evaluations as these can have high coverage of population, be continuously collected over time, suffer less from measurement errors and misreporting and be used cost-effectively (OECD, 2020[26]). The impact evaluations of ALMPs conducted in Finland so far rely on the administrative data sources. Nevertheless, it might be necessary to link administrative data with survey data in some evaluations to address any potential selection bias (for example due to differences in motivation between ALMP participants and other jobseekers) or measure outcomes for which administrative data are not easily available (such as mental health or well-being).
The main organisation facilitating access to administrative data for research purposes in Finland is Statistics Finland.14 However, a wider range of stakeholders play a role in generating administrative data and sharing these with Statistics Finland. The data flows concerning administrative data on ALMPs, jobseekers and job mediation are depicted in Figure 3.2. Staff in TE Offices (Employment and Economic Development Offices) implement ALMPs and insert the relevant data within this process using the IT infrastructure provided for them by the KEHA Centre (the Development and Administrative Centre for ELY Centres and TE Offices). The IT infrastructure provided by the KEHA Centre is also relevant for ELY centres (Centres for Economic Development, Transport and the Environment) in steering TE Offices. KEHA Centre is also responsible for making queries in the ALMP database to share data for research purposes with TEM, with Statistics Finland via TEM, as well as with researchers directly.
The next subsections discuss the role of these different stakeholders in data exchange in detail while the role of these stakeholders in the ALMP provision more generally is discussed in Chapter 2. The specific issues regarding the suitability of the data available via Statistics Finland to conduct ALMP impact evaluations is discussed in Chapter 4, as well as in detail in the technical report accompanying the current report (OECD, 2023[27]).
Figure 3.2. Administrative data on ALMPs for research purposes are shared mainly by Statistics Finland
Note: TE Offices – Employment and Economic Development Offices. ELY Centres – Centres for Economic Development, Transport and the Environment. KEHA Centre – the Development and Administrative Centre for TE Offices and ELY Centres. TEM – Ministry of Economic Affairs and Employment. See the discussion on the institutional set-up of ALMP provision in Finland in Chapter 2.
3.4.1. The availability and quality of data on ALMPs and jobseekers are limited due to outdated IT infrastructure
This section discusses the data collection on ALMPs, data used for ALMP statistics by TEM, and the data transfer process to Statistics Finland for research purposes.
The IT infrastructure supporting ALMP provision and data collection is largely outdated
Since 2016, the KEHA Centre is responsible for providing the IT infrastructure for the system of ALMP provision, while previously it was the responsibility of TEM. The IT infrastructure needs to support TE Offices to implement ALMPs, and jobseekers and employers to interact with TE Offices, and collect data within the service provision process. The IT infrastructure also needs to support ELY Centres to steer TE Offices and manage the provision of ALMPs regionally.
The operational IT system for ALMP provision (URA register) is largely a legacy system, dating back to 1997. The development of the new operational IT system started in 2017, but so far only the new job mediation platform for employers and jobseekers “Job Market Finland” was adopted in 2021. Nonetheless, this new platform has already become an example of a good practice for other countries as it includes AI tools to support users in refining and streamlining vacancy descriptions and jobseekers (OECD, 2022[1]). Job Market Finland analyses the data provided by employers and jobseekers, as well as taxonomies of tasks and skill needs linked to each occupation to recommend suitable jobs (see Box 3.2).
Box 3.2. Job Market Finland is a modern digital tool to match jobseekers and vacancies
Job Market Finland is a sophisticated digital tool aiming to match jobseekers and vacancies effectively, combining a set of AI algorithms to overcome discrepancies in the descriptions of job postings and jobseeker profiles. The “competence recommender” within the tool helps jobseekers to create their profiles, generating their set of competencies based on different possible inputs, such as keywords, CVs, motivation letters or job advertisements.
Above all, the tool uses the European Skills, Competences, Qualifications and Occupations classification (ESCO), which provides a description of about 3 000 occupations and the skills and competencies associated with these occupations. Second, the tool includes detailed descriptions of about 600 occupations. The AI algorithms enable to look for suitable vacancies by occupation, competencies, skills and work preferences, regardless of the exact wording used for job search, in the profile or the vacancy. The competency-based search enables the jobseeker to discover suitable occupations and careers that they might not realise to look for otherwise.
Job Market Finland includes also a so-called “relevance model” to pre‑process vacancies and extract only those words for matching that indeed describe the job, rather than the general description and objectives of the company. The relevance model has been trained to extract the relevant content in Finnish, Swedish and English, so that the model is able to process vacancies in any of these languages.
Although the Job Market Finland platform provides also general information for jobseekers, such as information on available training and education schemes, the digital tool does not currently enable to map skill gaps or suggest relevant training. Finland could consider using the data collected via the matching processes of Job Market Finland to detect those occupations that have higher employment potential for the specific profiles of jobseekers. The jobseeker could get recommendations on occupations to focus on, skills needed to be acquired to improve job finding prospects, as well as training and education to acquire the respective skills. Furthermore, such mapping of individual skill gaps and training needs could educate referrals to the training programmes within the ALMP system in Finland, such as the labour market training and self-motivated education discussed more specifically in the next chapters of this report.
The data generated within Labour Market Finland could also be used for generating labour market information, particularly to assess and anticipate skill needs in Finland. Such analyses would complement well the qualitative Occupational Barometer (see Chapter 2), as well as the analytical exercises conducted by TEM using the PES vacancy data on occupations, the data in Statistics Finland or survey data.
Source: Ministry of Economic Affairs and Employment (2022[28]), Työpoliittinen aikakauskirja 4/2022, https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/164474/TAK_4_2022.pdf; Niittylä (2021[29]), Using AI for jobseeking, https://tyomarkkinatori.fi/en/blogit/vieraskyna-niittyla-tekoalyn-hyodyntaminen-tyopaikkojen-haussa; Niittylä (2020[30]), Utilising artificial intelligence in the application process for ESCO competence, https://tyomarkkinatori.fi/en/blogit/vieraskyna-niittyla-tekoalyn-hyodyntaminen-esco-osaamisten‑haussa; Vänskä (2020[31]), The Good AI, https://kokeile.tyomarkkinatori.fi/en/blogit/hyva-tekoaly.
However, the staff in TE Offices and ELY Centres still use the legacy IT system to register jobseekers and provide ALMPs, which can affect the quality of the data used in ALMP impact evaluations, in addition to not supporting well the ALMP provision in general. According to the employment counsellors from TE offices from different parts of Finland interviewed for the current assessment, the URA system supports basic customer services, but has many shortcomings. For example, some data fields that could be structured (classifications and code lists) are implemented as free text, not supporting well the work of employment counsellors or statistics and research based on these data. Some fields are mandatory to fill, although employment counsellors have rarely information on these issues, resulting in incorrect data in the database. For example, jobseeker destination is often marked to be unknown, although in reality the destination is known to be employment, but only the mandatory details on employment contract are unknown. This issue also reveals that the URA system is not sufficiently exchanging data with other administrative registers in Finland to fully support services provision, as well as data accuracy. In addition, at times policy design has been changed quicker than it has been possible to adjust the URA system to continuously support service provision and data collection.
The data analytics tools linked to the operational IT system are currently still in their infancy and need to be developed further together with the overall modernisation of the IT infrastructure in the ALMP system. The KEHA Centre has started to build some pre‑defined reports for internal monitoring using QlikView (a Business Intelligence tool) and plans to change it to Power BI over some years. Considerable investments are needed to develop a data analytics system that provides monitoring reports and dashboards for TE Offices and employment counsellors to manage their work and caseloads, for ELY Centres to steer TE Offices and regional ALMP provision, for TEM to govern the overall ALMP system, and potentially also for the public to support interactive knowledge dissemination on ALMPs.
The outdated IT infrastructure, including the data analytics tools, also hinders the data availability for statistics, analysis and research on ALMPs. The KEHA Centre has the possibility to flexibly query any data from the operational database of URA system for research needs (following an application for the data, assessment and permission), and occasionally ad-hoc queries have been made in practice. Anonymised data and grouped data could be then shared by the KEHA Centre directly, while data to be pseudonymised and linked with other data sources could be made available for the researchers via Statistics Finland. Nevertheless, the dataset that is regularly shared with Statistics Finland (via TEM) and is the core dataset for ALMP evaluation, is inflexible and has remained the same over the past years regardless of changes in policy design.
The modernisation of the IT infrastructure for ALMP provision is further complicated by the on-going reform
The replacement of the URA system has been cumbersome due to challenges in allocating funding for the IT developments and finding experts to drive the modernisation. The KEHA Centre outsources a large share of the IT development, while 13 staff members are engaged with IT developments in-house. When the KEHA Centre was created in 2015 and responsibilities were transferred to it from TEM over 2015‑16, not all resources and staff that were planned to be shifted together with responsibilities were in practice shifted. Also, not all tasks regarding the administration and support of TE Offices and ELY Centres were transferred, potentially making the governance model and change processes more complicated. The budget for the KEHA Centre is assigned yearly without longer-term commitments, which complicates recruitment and signing contracts extending over one year.
A sudden decision on changing the institutional set-up of ALMP provision in September 2021 has further delayed the IT modernisation process, as the exact responsibilities of each stakeholder need to be assigned before the IT solutions to support these roles can be defined. The government still aims to have a central operational IT system and the data analytics platform (Data Warehouse or Data Lake together with Business Intelligence tools) to support data availability for monitoring and evaluation, as well as evidence‑based policy making more generally. Yet, some municipalities might want to set up their own IT platforms that have simply interfaces with the national IT infrastructure, while most of the municipalities are not likely to have such capacity and need to rely solely on the national IT infrastructure. Hence, before the developments of the national IT infrastructure can be adjusted to the new institutional set-up, the government and TEM need to be clear on which responsibilities will be transferred to municipalities, what will be their scope of freedom regarding their operating models and business processes in implementing ALMPs, and what kind of support needs to be provided by the central level (e.g. via the KEHA Centre or its successor in the future).
As TEM is governing the KEHA Centre, it needs to drive and enable the process of modernisation of the IT infrastructure to ensure well-performing ALMP provision, as well as data availability for evidence generation. First, TEM needs to be in a systematic dialogue with the KEHA Centre in preparing for the reform and developing the IT infrastructure to meet the needs of the new set-up. The operational IT system needs to maximise its support to its users, i.e. employment counsellors. Hence, the planning, development and testing needs to involve employment counsellors (currently in the TE Offices, but to be transferred to the municipalities), as well as the municipalities more generally. The plans need to be discussed not only with those municipalities that are eager to go through the reform, have been part of the piloting of the new system and tend to have a higher capacity, but also with those for which the reform might be challenging and thus might have different needs for support. Second, in addition to the careful design of data exchange and integration of IT infrastructure between the core stakeholders of ALMP provision, data exchanges with other administrative registers need to be strengthened to support employment counsellors, jobseekers and employers, as well as to ensure data accuracy. Data already available in other administrative registers should not be collected again but received automatically, implementing so the “once‑only” data collection principle.15 The implementation of such data exchanges assumes a careful analysis of data needs and amendments in regulations to provide a solid legal basis for the relevant data exchanges. Third, TEM (and the government) needs to find sufficient and sustainable funding model for the KEHA Centre to enable it to carry out its responsibilities, particularly in terms of projects and developments that have longer than one year horizon.
TEM receives monthly datasets from the KEHA Centre for statistics and analysis
An automatic data exchange is set up to provide data from the URA system to TEM. TEM receives a pre‑defined set of individual level data each month on jobseekers, ALMPs and vacancies already a few days after the end of month to produce timely statistics based on these data. The data files are uploaded in the internal statistical system of TEM to facilitate producing standard reports quickly.
A mixture of automatic and manual quality checks takes place before data transfer to TEM, after uploading data in the TEM system and before publishing the statistics based on the data. Errors in data transfer and uploading do happen sometimes, in which case the quality checks enable to identify them and re‑run the processes. Data quality can be affected also by incorrect data inputs (see the discussion in the beginning of this section) but are generally harder to identify by the data quality checks in the KEHA Centre and TEM. Hence, the approach by TEM is to ensure the correct data inputs simply via guidelines and training for TE Offices, which however might not be sufficient in the context of an outdated IT infrastructure.
The statistics produced on URA data is currently limited to the inflexible and static pre‑defined dataset that is regularly shared with TEM. The outdated IT infrastructure of the URA system does not enable to easily adjust the regular data exchange, and thus statistics by TEM does not capture well the changes in ALMP design during the past years.
Along with developing a new operational IT system for ALMP provision, setting up a new data transfer system is planned as well. The new IT infrastructure is expected to be fully in place within around five years, with some modules and data transfer from them being ready sooner. The current plan foresees setting up a Data Warehouse in KEHA Centre, and TEM also receiving the data inputs from the same Data Warehouse. The Data Warehouse would include data from those municipalities that use the central operational IT system for service provision, as well as those that use their own, but still deliver the data to the central Data Warehouse. Regardless of having data from all municipalities, an additional challenge for TEM in terms of statistics (and potentially for analysis and research) will be how to classify and compare municipality-specific ALMPs.
For research purposes, ALMP data need be linked with other datasets
The individual level dataset that TEM receives from the KEHA Centre, is further shared with Statistics Finland that uses the data for national statistics and can make these available for researchers (see also next subsection). As the data on ALMPs and jobseekers are safely available via Statistics Finland, TEM refers the researchers there for data needs rather than sharing data directly with researchers. In case researchers’ data needs go beyond the dataset shared by KEHA Centre with TEM, they are referred to request data from the KEHA Centre directly, which can make ad hoc data queries to the URA system. Nonetheless, most of the research projects (and particularly ALMP evaluations) need additional data from other registers in any case, such as data on benefits, employment, education or other socio‑economic characteristics. Thus, ALMP data for research purposes are increasingly being used via Statistics Finland, rather than shared by TEM, the KEHA Centre or in co‑operation with another administrative register directly.
Also, TEM itself uses data from Statistics Finland to be able to link ALMP data with additional datasets (although also direct ad hoc data exchanges with other registers have been sometimes practiced in the past). Linking ALMP data with employment data is of particular importance as the jobseeker destination data after ALMP participation are essentially missing in the URA system. To facilitate ALMP monitoring, TEM has outsourced statistics on labour market status after ALMP participation (ALMP gross impact) fully to Statistics Finland, receiving aggregate ready-made statistics on ALMP participants entering employment, education and other destinations.
3.4.2. Researchers can access rich data securely via Statistics Finland
This section discusses how Statistics Finland makes data and metadata available for research, including the legal basis for data sharing, the technology to ensure data protection and the process to link data across registers.
Only Statistics Finland and Findata have a legal basis to link and share data for research
The amendments of the Statistics Act16 in 2013 made it possible for Statistics Finland to share their data remotely for research purposes in a pseudonymised form. This has significantly widened the data availability for researchers as the uniquely pseudonymised format (contrary to the anonymised format used previously) enables researchers access full datasets from Statistics Finland and combine different datasets according to the research needs. At the same time, the data can be used securely, as these never leave the databases of Statistics Finland. Nonetheless, the Statistics Act gives Statistics Finland the right to collect data for statistics and to share these data for research purposes, but not explicitly collect data for research purposes.
As the current Statistics Act only mentions sharing data for research purposes and does not explain in more detail how Statistics Finland supports research activities, the Ministry of Finance that governs Statistics Finland has initiated further amendments in the Statistics Act. The new regulation would list research services (above all providing data for research) explicitly among the tasks of Statistics Finland, in addition to producing statistics. This amendment is important as research services of Statistics Finland have become a very important part of its activities. For example, at the end of 2020 there were in total 1 590 valid user licences to access data for research in Statistics Finland (Statistics Finland, n.d.[32]).
The only other organisation that has the right to collect, link and share administrative data for secondary use (statistics, analysis, research) in Finland is Findata. The legal foundation giving Findata the authority to share data for research purposes rests in the Act on the Secondary Use of Health and Social Data of 2019. Findata provides secure access to data on social and health care services, such as data from the Social Insurance Institution of Finland (Kela), the Finnish Institute for Health and Welfare (THL), the Finnish Centre for Pensions and the electronic health record system (Kanta) (Magazanik, 2022[33]), thus having a different data coverage and purpose than Statistics Finland. The data in Findata are not the core datasets relevant for ALMP evaluation but could be used as supporting data to define credible counterfactual or measure additional outcomes of policies. In this case, the data of Findata could be used via Statistics Finland.
Accessing data in Statistics Finland can be costly
Currently, Statistics Finland does not receive any funding from the Ministry of Finance for research services, i.e. the processes to share data for research purposes. In the past years, these services have received only some project-based funding, such as via Academy of Finland, the Finnish Innovation Fund Sitra and the sister organisation Findata under the Ministry of Social Affairs and Health. The amendments in the Statistics Act could provide Statistics Finland with a stronger case to also receive funding for its research services.
As long as the research services are not funded from the state budget, the researchers accessing data are covering the associated costs themselves. However, this funding has not covered in the past all related costs. In addition, as also the other tasks of Statistics Finland are not sufficiently funded, it is not possible to cross-subsidise the research services from its other funding. This has resulted in shortage of staff to carry out the research services and long waiting times for researchers to access the data (see the discussion in the next subsection).
To overcome the funding issues, Statistics Finland increased significantly its prices to access data for research in the beginning of 2022 from already a high level. Already before the increase, a yearly access to data to carry out a policy impact evaluation could have easily costed around EUR 10 000 a year for a small group of researchers. From 2022 onwards, the research services have been further modernised (such as the possibility to access newer data during the contract with Statistics Finland, but then the contract is binding for at least two years), but at higher prices than before. This means that a significantly higher share of budgets for research projects needs to be allocated to data access. In addition, ALMP impact evaluations would be possible only essentially within publicly procured research projects, and not for example for purely academic reasons.
The research services of Statistics Finland are expensive particularly due to the IT infrastructure to enable secure remote access to data. Depending on the required computing power of the virtual machine and the number of users, the yearly fees in 2022 range from EUR 3 000 to close to EUR 7 000 (or even more if five or more researchers in the project). The fees for data depend on which specific datasets are needed for the project and whether these need to be updated during the project duration. As the main datasets are split to many smaller datasets and charged individually, the costs for the data can well match the costs of renting the virtual machine. The fees for contracting and licencing are added to the total cost of data access.
When Statistics Finland opened remote access to research data in 2013, it initially used its own servers, which was a cheaper solution, but was not sufficient for researchers due to too low computing power. In that set up, one researcher running a model needing more computing power, could block the work of all other researchers.
The current solution of remote access to the data in Statistics Finland uses the servers of the CSC, which is an organisation providing IT services for research purposes and is owned by the Ministry of Education and Culture and the universities in Finland (CSC, n.d.[34]). This solution enables the researchers choose the computing power needed for their research projects and the capacity is not affected anymore by how many other researchers are simultaneously accessing data from Statistics Finland in other projects. Yet, this more convenient solution comes at a higher cost, as these services of the CSC are to be covered by Statistics Finland, which in turn bills the researchers.
Statistics Finland has well-established, transparent and secure processes for data sharing, although granting access can take time
Statistics Finland has set up a standard procedure for researchers to apply and access data, which is communicated on its website in detail for transparency and clarity (Statistics Finland, n.d.[32]). In the data application, researchers need to provide information on the intended use of the data, the research plan, pledges of secrecy and applicant’s details, as well as other documents depending on the specificities of the application (e.g. licences from other authorities are needed if additional data need to be linked with the data in Statistics Finland).
The processing of the data request depends on the extent and complexity of the data to be used, such as whether the application must be approved by the Board of Statistical Ethics. While the less complicated applications could be theoretically processed in a couple of weeks, the processing time has tended to stretch over a few months in most cases during the past years. The waiting times for data access have taken even around one year in case additional data from administrative registers have been requested to be linked with the data already in Statistics Finland. The research services have become very popular among the researchers, while the resources to process this volume of data requests have not kept up in Statistics Finland. Although this might change somewhat with the increased prices of research services. Nonetheless, Finland needs to consider funding the research services of Statistics Finland sustainably for example from the state budget to cut the data application processing times and cap research data prices to support evidence generation across policy fields and encourage research not only tied to a current political agenda.
Once the application to access the data is approved and datasets prepared, the researcher can access the data remotely and securely. The researcher can access the remote access system of Statistics Finland called FIONA from their local workstation and use the data via various software programmes like STATA, R, Python, SAS and SPSS (Statistics Finland, n.d.[35]). However, the results of the data analysis can be transferred from the system only after a review by Statistics Finland to ensure data protection (e.g. no individual level data can be displayed among the results).
A strong authentication is enforced for researchers to access FIONA, while offering a choice of identity authentication systems. In addition to authentication systems specific to Finland (State Treasury identification service, Haka, Suomi.fi), an international authentication service by EduGAIN17 is enabled to facilitate international researchers to use data in Statistics Finland.
A rich set of administrative data are collected by Statistics Finland
Contrary to quite a few other OECD countries, Finland uses a unique identification number (the social security number) for all its residents across administrative registers, enabling to link data accurately across registers. The same holds for URA data on jobseekers. The exceptional cases are recent migrants that might receive some services before they have got their social security number.
Statistics Finland has a long experience of linking data across administrative registers. Finland was the second country in the world to conduct population census based on register data in 1990, preceded by a test census in 1987. This required mapping different registers and data available in them and identifying data relevant for statistics and evidence more generally. Hence, data exchange between Statistics Finland and various registers in Finland has been established often many years ago, although continuous work takes place to keep the data exchange up to date when the IT infrastructure, collected data and policies change.
Statistics Finland has an elaborate Data Warehouse to manage data for statistics and from which data for research purposes are shared. About 97% of these data are administrative data from different registers, while only 3% are survey data. Register data are generally preferred to survey data as these are covering the whole population and are more efficient to collect.
The core data shared for research purposes are data in the Data Warehouse of Statistics Finland received via interfaces to different registers. Researchers can apply to registers for additional data to be sent to Statistics Finland for research purposes, which can be then used together with the data in Statistics Finland (after pseudonymisation) but deleted after the end of the project. The registers can also share additional pre‑defined datasets with Statistics Finland on their initiative to meet the research needs, that Statistics Finland can keep in their Data Warehouse although not used for statistics and make available for researchers when requested. If a research project needs only data available in Findata, a data application would have to be made directly to Findata and used via their IT infrastructure. However, if additional data from Statistics Finland are needed, then the respective data from Findata are transferred to the system of Statistics Finland to enable linking the different datasets.
As a result, the scope of data available in Statistics Finland can support well ALMP impact evaluations. Firstly, the datasets include the original pre‑defined datasets on jobseekers, ALMPs and vacancies from URA system, as well as individual level indicators calculated for statistics based on URA data, that can on some occasions further facilitate conducting evaluations. Secondly, the datasets include rich data on socio‑economic characteristics of the population to construct a counterfactual for the evaluation (e.g. family composition, household data), as well as observe the effects of ALMPs on different outcomes (using employment, education and firm data). All these datasets are updated annually in Statistics Finland. Further discussion on the coverage and format of the data in Statistics Finland for ALMP impact evaluations is in Chapter 4 and in the technical report accompanying the current review (OECD, 2023[27]).
Nevertheless, the data are generally not fit to evaluate very recent policy changes. First, many datasets are shared with Statistics Finland only once a year. Second, it takes some time for the registers to share their data with Statistics Finland (some time needed for data accuracy as some registers are more prone to retrospective changes than others, some time needed for data to be extracted, transformed and loaded, etc.). Third, the data shared by the registers with Statistics Finland go through a thorough quality check and pseudonymisation. And fourth, Statistics Finland is not currently sharing the data for research purposes before they have published the official statistics based on these data. As a result, as of October 2022, the data on ALMPs and jobseekers are available up to 2021, qualification data up to 2020 and employment data up to 2019. An impact evaluation to be conducted in the end of 2022 could thus not easily evaluate ALMPs provided after 2018, and the time lag is even longer if looking at long-term effects would be aimed at. In addition, the currently available datasets do not enable conducting credible cost-benefit analyses of ALMPs, as Statistics Finland does not have comprehensive data on the costs of services, measures and benefits for jobseekers (see Section 3.3.5). Further modernisation of the IT infrastructure across the public sector in Finland is necessary to ensure better data quality and coverage in the registers, prepare operational data better for data analytics and establish more timely and frequent automatic data exchanges. Better financing of Statistics Finland would enable it to shorten data preparation periods and produce more timely statistics, simultaneously shortening the time lags of research data as a side effect.
Elaborate metadata are available, but support to researchers could go further
Statistics Finland collects metadata systematically together with data from the administrative registers. It uses a metadata system called Metsy that is based on the Generic Statistical Information Model (GSIM, an international reference framework for statistical information) (Kaukonen, 2019[36]).18 The metadata are, however, at the moment only rather technical details and not descriptions of the data. For, example the metadata do not address changes in time series data due to changes in policy design, taxes, currency or legislation more generally. Or, at times no definitions for missing values or zero values and their differences in practice. Also, metadata regarding ALMP data from the URA system are not at times complete and clear for researchers.
While Statistics Finland has worked on improving the metadata over the years, the metadata quality is still limited due to what is shared with Statistics Finland by the registers, as well as due to the shortage of resources of Statistics Finland to improve metadata quality. The Ministry of Finance is aiming to address this challenge by establishing the criteria for metadata that administrative registers should follow.
Statistics Finland publishes the metadata on the data available for research in its Taika metadata catalogue19 that has been developed in co‑operation with registers and researchers and in use since 2016 (Statistics Finland, 2016[37]). First, this catalogue enables researchers to define their data request as the catalogue displays which variables are available and their time coverage. Second, the Taika catalogue contains variable descriptions to help researchers during the data analysis. In addition to elaborate definitions in Finnish, most metadata on core datasets are also available in English to support international researchers.
In case of technical questions on research data, researchers can contact a dedicated email account in Statistics Finland. Statistics Finland can then check if there could be errors with certain variables, or changes in metadata, but cannot provide advice on data use or methodologies and policy changes originating from the underlying administrative register. Statistics Finland backlogs the questions to investigate the technical aspects further in case of recurrent issues.
Further descriptions of data and information on addressing certain issues in the data are also not collected and shared by any other organisation, regardless of different researchers facing continuously the same issues. For example, regarding ALMP evaluation, all researchers need to address mismatches between the original files from the URA system and the indicators calculated by Statistics Finland, consider changes in ALMP time series data due to changes in policy design and address issues concerning matching flow and stock data (see more on the issues in Chapter 4 and OECD (2023[27]). Each researcher group has to then partly address the same issues again, map the policy changes, develop the same coding to prepare data for analysis, although these steps have been already previously conducted by other researchers. This approach is inefficient and can mean that the ministry outsourcing the research needs to essentially pay several times for the same work.
In the outsourced research projects, TEM should request that the researchers share with TEM also the codes used to conduct the research and publish these together with the reports discussing the results. First, this would bring down the costs of research projects as some of the work does not need to be repeated again (enable TEM to get more evidence within the same budget). Second, publishing the codes would serve as an additional quality assurance, as the exact methodology would be more transparent for the research community and any questionable steps in the methodology could be identified easier.
3.5. Conclusion
TEM has been able to generate sound evidence on key ALMPs by using its internal capacity, outsourcing research projects and co‑operating with other ministries. The limited resources are steered to cover the priority research needs, the good practices to procure and steer research projects ensure quality in evidence and the generated evidence is systematically disseminated. Conducting research on ALMPs is well facilitated by Statistics Finland, which is able to share rich data linked across many administrative registers securely with researchers. As a driver of policy design in the field of ALMPs, TEM needs to ensure that it has sufficient evidence for policy design and invest in its internal research capacity, outsourced research projects and the IT infrastructure to collect data on ALMP provision accordingly.
References
[18] Aho, S. et al. (2018), “Työvoimapalvelujen kohdistuminen ja niihin osallistuvien työllistyminen”, Valtioneuvoston selvitys-ja tutkimustoiminnan julkaisusarja 19/2018.
[19] Alasalmi, J. et al. (2019), “Työttömyyden laajat kustannukset yhteiskunnalle”, Valtioneuvoston selvitysja tutkimustoiminnan julkaisusarja 16/2019.
[17] Alasalmi, J. et al. (2022), “Ammatillisen työvoimakoulutuksen toimivuuden ja vaikutusten arviointi”, Valtioneuvoston selvitysja tutkimustoiminnan julkaisusarja.
[25] Alasalmi, J. et al. (2022), Aktiivisen työvoimapolitiikan palveluiden kustannusvaikuttavuus, https://www.eduskunta.fi/FI/naineduskuntatoimii/julkaisut/Documents/TrVJ_2_2022_tyovoimapolitiikan_palvelut.pdf.
[22] Alasalmi, J. et al. (2020), Työpolitiikka ja työllisyysaste : tutkimukseen perustuvia johtopäätöksiä, http://urn.fi/URN:ISBN:978-952-287-951-6.
[12] Alatalo, J., E. Mähönen and M. Ylikännö (2022), Työ- ja elinkeinoministeriön lyhyen aikavälin työmarkkinaennuste, kevät 2022, Työ- ja elinkeinoministeriö, https://tem.fi/julkaisu?pubid=URN:ISBN:978-952-327-658-1.
[16] Asplund, R. et al. (2018), “Palkkatuen vaikuttavuus – palkkatukijärjestelmän ja sen uudistuksien arviointi”, VN:n selvitys- ja tutkimustoiminnan julkaisusarja, http://urn.fi/URN:ISBN: 978-952-28.
[34] CSC (n.d.), CSC yhtiönä, https://www.csc.fi/csc (accessed on 30 September 2022).
[7] Government of Canada (2022), Evaluation reports, https://www.canada.ca/en/employment-social-development/corporate/reports/evaluations.html.
[36] Kaukonen, E. (2019), Identifier Service developed in NSO Finland including GSIM-based information objects. ModernStats World Workshop 2019, https://unece.org/fileadmin/DAM/stats/documents/ece/ces/ge.58/2019/mtg2/MWW2019_MarketPlace_Finland_Kaukonen.pdf.
[5] Lauringson, A. and M. Lüske (2021), “Institutional set-up of active labour market policy provision in OECD and EU countries: Organisational set-up, regulation and capacity”, OECD Social, Employment and Migration Working Papers, No. 262, OECD Publishing, Paris, https://doi.org/10.1787/9f2cbaa5-en.
[39] Linnerud, J. (2020), Introduction to GSIM. ModernStats World Virtual Workshop 2020, https://unece.org/fileadmin/DAM/stats/documents/ece/ces/ge.58/2020/mtg2/MWW2020_GSIM_Introduction.pdf.
[33] Magazanik, L. (2022), “Supporting Health Innovation With Fair Information Practice Principles: Key issues emerging from the OECD-Israel Workshop of 19-20 January 2021”, https://www.oecd.org/health/OECD-Israel-Health-Data-Governance-Workshop-Report.pdf.
[8] Ministry of Economic Affairs and Employment (2022), Budjetti, https://tem.fi/budjetti (accessed on September 2022).
[28] Ministry of Economic Affairs and Employment (2022), “Työpoliittinen aikakauskirja, Finnish Labour Review, 4/2022”, https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/164474/TAK_4_2022.pdf.
[20] Ministry of Economic Affairs and Employment (2020), Minister of Employment Haatainen: Government spending limit discussion and supplementary budget support the processing of applications and multi-professional services, https://tem.fi/en/-/tyoministeri-haatainen-kehysriihesta-ja-lisatalousarviosta-tukea-tyottomyyshakemusten-kasittelyyn-ja-monialaiseen-palveluun.
[29] Niittylä, H. (2021), Using AI for jobseeking, Job Market Finland - Current issues, https://tyomarkkinatori.fi/en/blogit/vieraskyna-niittyla-tekoalyn-hyodyntaminen-tyopaikkojen-haussa.
[30] Niittylä, H. (2020), Utilising artificial intelligence in the application process for ESCO competence, Job Market Finland - Current issues, https://tyomarkkinatori.fi/en/blogit/vieraskyna-niittyla-tekoalyn-hyodyntaminen-esco-osaamisten-haussa.
[27] OECD (2023), “Technical Report: Evaluation of active labour market policies in Finland”, OECD, Paris, https://www.oecd.org/els/emp/FITechnicalReport2023.pdf.
[6] OECD (2022), Assessing Canada’s System of Impact Evaluation of Active Labour Market Policies, Connecting People with Jobs, OECD Publishing, Paris, https://doi.org/10.1787/27dfbd5f-en.
[1] OECD (2022), “Harnessing digitalisation in Public Employment Services to connect people with jobs”, OECD, Paris, https://oe.cd/digitalPES.
[4] OECD (2021), “Designing active labour market policies for the recovery”, OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing, Paris, https://doi.org/10.1787/79c833cf-en.
[2] OECD (2021), OECD Employment Outlook 2021: Navigating the COVID-19 Crisis and Recovery, OECD Publishing, Paris, https://doi.org/10.1787/5a700c4b-en.
[3] OECD (2021), “Scaling up policies that connect people with jobs in the recovery from COVID-19”, OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing, Paris, https://doi.org/10.1787/a91d2087-en.
[26] OECD (2020), “Impact evaluation of labour market policies through the use of linked administrative data”, OECD, Paris, https://www.oecd.org/els/emp/Impact_evaluation_of_LMP.pdf.
[23] OECD (2020), “Impact Evaluations Framework for the Spanish Ministry of Labour and Social Economy and Ministry of Inclusion, Social Security and Migrations”, OECD, Paris, https://www.oecd.org/els/emp/Impact_Evaluations_Framework.pdf.
[24] OECD (2014), Connecting People with Jobs: Activation Policies in the United Kingdom, Connecting People with Jobs, OECD Publishing, Paris, https://doi.org/10.1787/9789264217188-en.
[10] Prime Minister’s Office (n.d.), Government working group for the coordination of research, foresight and assessment activities, https://tietokayttoon.fi/en/government-working-group-for-the-coordination-of-research-foresight-and-assessment-activities (accessed on 30 September 2022).
[9] Prime Minister’s Office (n.d.), What we do, https://tietokayttoon.fi/en/what-we-do (accessed on 30 September 2022).
[11] Prime Minister’s Office (2021), Government adopts 2022 plan for analysis, assessment and research to support decision-making, https://vnk.fi/en/-/government-adopts-2022-plan-for-analysis-assessment-and-research-to-support-decision-making.
[21] Prime Minister’s Office (n.d.), Finnish Council of Regulatory Impact Analysis, https://vnk.fi/en/council-of-regulatory-impact-analysis (accessed on 30 September 2022).
[13] Räisänen, H. and M. Ylikännö (2021), Minne uudet työpaikat syntyivät vuonna 2020?, Työ- ja elinkeinoministeriö, https://tem.fi/julkaisu?pubid=URN:ISBN:978-952-327-816-5.
[38] Ruuskanen, O. and M. Obstbaum (2022), TEM:n oman analyysitoiminnan ulkoinen arviointi.
[37] Statistics Finland (2016), Taika research data catalogue opened, https://www.tilastokeskus.fi/uutinen/taika-research-data-catalogue-opened.
[35] Statistics Finland (n.d.), 4 Application process and agreement practices. Remote Access to Research Data, https://guides.stat.fi/remote-access-to-research-data/application-process-and-agreement-practices#s-lg-box-15690950 (accessed on 30 September 2022).
[32] Statistics Finland (n.d.), Introduction. Remote Access to Research Data, https://guides.stat.fi/remote-access-to-research-data (accessed on 30 September 2022).
[14] Tuomaala, M. (2021), Aktiivisilta työvoimapoliittisilta palveluilta sijoittuminen vuonna 2019, Työ- ja elinkeinoministeriö, https://tem.fi/julkaisu?pubid=URN:ISBN:978-952-327-800-4.
[15] Valtakari, M. et al. (2019), “Työttömien määräaikaishaastattelujen arviointi”, VN:n selvitys- ja tutkimustoiminnan julkaisusarja, http://urn.fi/URN:ISBN:978-952-287-696-6.
[31] Vänskä, M. (2020), The Good AI, Job Market Finland - Current issues, https://kokeile.tyomarkkinatori.fi/en/blogit/hyva-tekoaly.
Notes
← 1. The Employment and Social Development Canada is a department of the Government of Canada responsible for social programs and the labour market, including the evaluation of the related policies.
← 2. There used to be a research department also in TEM, but since the change in paradigm, it is no longer the case.
← 9. A qualitative assessment of analysis and research activities of TEM conducted by external evaluators in April to October 2022 has been a good initiative for TEM to understand better the scope for improvement in evidence generation (Ruuskanen and Obstbaum, 2022[38]). Nevertheless, a regular systematic process of peer reviews would enable timely feedback on the research methodologies and interpretation of research results.
← 10. A method to assess whether a policy produces the effects expected by the policy makers (i.e. the policy outcomes). The method involves comparing the expected outcomes for two groups i) those, who benefitted from a policy or programme (the “treatment group”), with ii) those, who did not benefit from the policy, but are otherwise similar to the treatment group (the “comparison/control group”). The comparison group provides information on “what would have happened to the participants in a policy in case they had not participated in it”.
← 11. Impact evaluation methods that use a counterfactual, but are not based on randomised assignment of the intervention, are called “quasi‑experimental methods”.
← 12. The “gold standard” in counterfactual impact evaluations is often considered to be an RCT, in which participation and non-participation in the intervention is allocated randomly and the outcomes of these two (or more) groups are measured. Randomising participation in the intervention minimises the chances that there are systematic differences between participants and non-participants, which are not related to participation in the intervention.
← 13. Selection bias occurs when the reasons for which an individual participates in an intervention are correlated with the (potential) outcomes this individual would observe under participation or non-participation. Ensuring that the estimated impact is free of selection bias is one of the major objectives and challenges for any impact evaluation.
← 14. In addition, an authority called Findata shares social and health care data for research purposes (see https://findata.fi/en/). However, this initiative does not include ALMP and labour market data and is thus not discussed extensively in this report.
← 15. The (cross-border) “once‑only” data collection principle is for example encouraged by the European Commission in its Single Digital Gateway Regulation to support Digital Single Market, as well as in the eGovernment Action Plan. Once‑only principle aims at more efficient administrative processes for both the government and the citizens and at a higher data accuracy, as any data relevant for a public sector organisation should be collected only once and consequently shared with others securely if needed for service provision.
← 18. GSIM is a reference framework of internationally agreed definitions, attributes and relationships that describe the pieces of information used in the production of official statistics and enables generic descriptions of the definition, management and use of data and metadata throughout the statistical production process (Linnerud, 2020[39]).