This chapter presents the results of the evaluation of the Finnish Education Evaluation Centre (FINEEC). It begins with an overview of perceptions of FINEEC, highlighting its reputation as an impartial and credible institution. The chapter then identifies nine key areas for improvement, organised around the themes of FINEEC’s strategy and scope of work, operations and contributions to policy making. For each of the challenges identified, strategies and actions for moving forward are highlighted.
Finnish Education Evaluation Centre (FINEEC)
OECD Centre for Skills Evaluations
OECD Skills Studies
Abstract
Based on the information gathered during the interviews with stakeholders in Finland, the Finnish Education Evaluation Centre (FINEEC) is seen as an impartial, relevant and credible institution that provides regular information on the status of the Finnish education system through the evaluations and assessments it conducts.
This assessment is supported by findings from the latest FINEEC stakeholder survey conducted in 2022, which received responses from 312 participants and achieved a 23% response rate. The survey aimed to assess FINEEC’s success in evaluation activities, usage of evaluations, its reputation among stakeholders, the effectiveness of stakeholder co‑operation and communication, and considerations for the upcoming evaluation plan. Results indicated that 89% of respondents consider FINEEC’s reputation to be at least “fairly good”, reflecting a 9% improvement compared to the same measurement in 2018 and a 5%. improvement to 2020.
Its positive reputation is largely attributable to FINEEC’s acknowledged expertise, strong stakeholder engagement practices and perceived impartiality. These three elements could be considered part of the main added value of FINEEC. Stakeholders’ positive assessments of FINEEC’s expertise are based on its longstanding experience in conducting evidence-based evaluations and national assessments. Positive perceptions of its stakeholder engagement performance are attributed to its employment of enhancement-led evaluation methods, which involve consulting with stakeholders at various stages of the evaluation process, promoting awareness of the strengths and weaknesses of the organisation (or target) being evaluated, and supporting change in the organisation. Lastly, FINEEC’s perceived impartiality has been founded largely on its operational and financial autonomy, which protects it from potential third-party interests. Notwithstanding the generally positive perceptions about the work and attributes of FINEEC, there remains room for improvement.
It is important to note that this evaluation aims to identify potential areas for improvement within FINEEC’s scope of evaluations, as well as their quality and relevance to policy making, rather than examining every aspect of its operations. For this reason, the findings presented below do not cover the numerous areas where FINEEC excels and for which it is internationally recognised.
The remainder of this report presents nine key areas for improvement, organised into three broad categories to provide a structured overview. These categories are FINEEC’s strategy and scope of work, FINEEC’s operations, and FINEEC’s contribution to policy making.
Additionally, international examples are included to show how other countries have addressed similar challenges and to serve as inspiration for future policy reforms. However, these examples are not intended to be interpreted as recommendations; they are provided to underscore the possibility for improvement and stimulate thinking of enhancements in FINEEC’s approach to evaluation.
While FINEEC’s evaluative role is recognised and valued by many stakeholders, there is a lack of consensus on the role FINEEC should play in proposing or developing solutions to the problems identified in its reports. This disagreement centres around whether FINEEC should actively propose policy recommendations or concentrate on providing the data and analysis to facilitate a broader dialogue on possible solutions. The mandate of FINEEC does not explicitly require it to provide recommendations, which leaves its role open to interpretation This lack of agreement on FINEEC’s role in the policy-making process creates ambiguity that can detract from the use of its core evaluation findings. There is also disagreement about what constitutes sufficiently robust evidence to underpin its recommendations, a topic that will be explored in depth in other sections (see Key Finding 7).
There is a divergence in opinions about FINEEC’s role in proposing specific policy solutions to the problems illuminated by its evaluations. While FINEEC staff believe they are responsible for making recommendations and contributing to developing solutions to problems in the education system, its stakeholders generally do not expect FINEEC to fulfil this role.
This disagreement is rooted in two main factors. First, FINEEC’s recommendations are often perceived as overly broad and not well grounded in evidence (see Key Finding 7), and questions have been raised about their practical applicability. Second, the prevailing view among stakeholders is that the Finnish National Agency for Education (EDUFI) should be the primary entity responsible for proposing solutions, with FINEEC’s role being more concentrated on conducting evaluations and presenting information and analysis to stimulate discussion about potential solutions.
For example, education providers argue that FINEEC’s primary role should be to assess the education system, with the aim of generating quality information on students’ trends performance, as well as on the education system’s strengths and challenges. FINEEC, according to the results of the latest stakeholder survey, fulfils this role quite well: 91% report that FINEEC produces information on current and relevant issues through the evaluations it conducts, and 82% report that FINEEC evaluations contribute to the development of the Finnish education system.
In addition, education providers recognise FINEEC’s role in informing public discussion on issues associated with Finland’s education system, a role that, from their perspective, could be strengthened in the future. This latter perspective is supported by the FINEEC stakeholder survey results, with only 57% of respondents reporting that they “somewhat agree” or “completely agree” with the statement “FINEEC participates actively in the societal debate on education”.
Furthermore, education providers noted that FINEEC should leverage its reputation, expertise and the trust it has garnered to promote and inform evidence-based dialogue and decisions. One of the approaches FINEEC already employs to do this is issuing statements, which have mostly been made in response to parliamentary requests [see, for example (FINEEC, 2023[1]; FINEEC, 2023[2]; FINEEC, 2023[3])]. FINEEC’s increased engagement in public discussion would not only strengthen the quality of policy dialogue but could also lead to more effective and evidence-based educational policies, ultimately enhancing the Finnish education system.
From EDUFI’s perspective, FINEEC’s added value lies in its impartiality as well as its competence and experience in conducting evaluations on strategic and/or long-term issues, where the assessment of learning outcomes plays a fundamental role. For this reason, in EDUFI’s view, FINEEC could prioritise its limited resources on these activities.
For its part, the OKM has noted that the methodological approaches used by FINEEC do not provide a sufficient basis for developing concrete policy recommendations, primarily because the methods and analysis undertaken by FINEEC typically do not produce causal evidence (Key Finding 7 .). Furthermore, the OKM holds that FINEEC is not expected to deliver policy recommendations but that if it decides to do so, these recommendations must be based on causal evidence, which is not always currently the case.
Like the OKM, education providers and EDUFI representatives did not see FINEEC as the organisation responsible for generating policy recommendations or proposing solutions to the problems identified in their evaluations.
A review of a sample of FINEEC’s reports by the OECD corroborates the perception that policy recommendations are not sufficiently grounded in evidence (see Annex F).
For example, the report Equality and Participation in Education - An Overview of National Evaluations, published in 2021 (FINEEC, 2021[4]), makes several recommendations for dismantling inequality- producing structures and ensuring continuous learning opportunities for all socio-economic backgrounds. While these recommendations are broadly consistent with the need for equality in education identified by FINEEC, they are not directly supported by the findings of the evaluation or by external literature.
Further evidence can also be found in the set of recommendations presented by FINEEC in the 2022 report, Partnering with Working Life - Evaluation of Workplace Education and Training and Working Life Co-operation in Vocational Education and Training (FINEEC, 2022[5]). These recommendations emphasise improving the effectiveness of workplace education and promoting closer co-operation between vocational education and training (VET) providers and employers. While the recommendations align with the report’s overall conclusions, highlighting the need to better integrate VET with working life, they lack support from specific empirical data or external research.
Similarly, in a 2022 report on vocational qualifications in electrical engineering and automation technology (FINEEC, 2022[6]), FINEEC presented recommendations to improve vocational education practices. The report advocated for increased participation of workplace representatives in assessments and a dedicated workplace instructor for each student, enhanced teacher-instructor collaboration, specialised training for instructors, and teacher work placements. These proposals appear to be driven by an understanding of educational needs, but limited empirical support or detailed analysis is provided for these specific recommendations.
While FINEEC has made recent efforts to make the recommendations more concrete and targeted, for example, by making distinct recommendations for each implicated party [see, for example, FINEEC (2022[5])], most stakeholders consulted still considered its recommendations inadequate as a basis for setting policy. Some education providers argued that it may be hard for FINEEC to generate useful recommendations for education providers as their contexts and needs are typically unique; therefore, it is challenging to generate recommendations that are applicable and useful to all.
Among the stakeholders consulted, there was a broad consensus about the roles of EDUFI and the OKM. Education providers interviewed see EDUFI as a crucial partner in proposing solutions and directly supporting schools in implementing educational programmes. This perspective on EDUFI’s role is consistent with its own vision for itself. For its part, stakeholders see the OKM as being responsible for developing legislation and providing resources for the functioning and improvement of the education system.
Clarity about the role of FINEEC in policy making will be important to ensure that scarce resources are used effectively and efficiently. There is a need, therefore, for key actors in the education sector to engage in a frank discussion about FINEEC’s role in this regard and reach a formal consensus.
One possible way forward is for FINEEC to concentrate its resources on its evaluation role. By limiting its role in developing policy recommendations, FINEEC could allocate more resources to enhancing its existing evaluation practices. This strategic shift would not only free up resources to devote to evaluations themselves but would also allow more resources to be devoted to disseminating the organisation’s findings, thereby strengthening its contribution to the public discourse and fostering a broader consensus on educational evaluation policies (see Box 3.1 to see how providing access to evidence can contribute to building consensus on educational evaluation policies). For example, this could be achieved through more frequent publications of statements and ensuring that all FINEEC’s publications receive wider dissemination (see Box 3.6 later in this chapter; it includes an example of an effective dissemination strategy from Switzerland).
A significant challenge for policy makers lies in the transition from identifying necessary changes to successfully implementing them. Implementing educational evaluation policies is a complex process involving diverse stakeholders with distinct interests, necessitating informed debates and capacity building. The resistance to reform could stem from stakeholders’ inadequate information about proposed policy changes, their impact and the potential individual or group outcomes. This lack of awareness may be exacerbated by the unpreparedness of public opinion for certain reforms, leading to a lack of social acceptance for innovative policies, especially in the context of an underdeveloped culture and limited tradition of evaluation in education. It underscores the need to promote research and make the evidence supporting policy proposals accessible to relevant stakeholders. The goal is to raise awareness of issues, foster a national debate and disseminate evidence on the effectiveness and impact of various policy alternatives, ultimately aiming to build a consensus on educational evaluation policies.
Source: OECD (2013[7]), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, https://doi.org/10.1787/9789264190658-en.
Having FINEEC specialise more narrowly in the conduct of evaluations is consistent with the practices of a number of other countries, including the United States, where research and evaluation tasks in the education sector are allocated to distinct entities, which may or may not be under the umbrella of a single organisation (Box 3.2 presents the role of the various public centres focused on advancing education research in the United States). This division or roles allows each entity to focus on a narrower set of functions, facilitating the development of specific competencies and experience that, other things being equal, can translate into better quality. When these entities are grouped under the responsibility of the same organisation, collaboration can be facilitated, and resources can be used more efficiently. This example illustrates possible ways of dividing the roles related to evaluation and research within the education sector among different stakeholders and underscores the importance of these stakeholders working together and complementing each other’s efforts.
The US Institute of Education Sciences comprises four centres and ten Regional Educational Laboratories, each with distinct roles in advancing education research in the United States:
National Center for Education Research (NCER): NCER supports research to address significant education problems, aiming to enhance education quality, boost academic achievement, reduce gaps and improve access to post-secondary education. NCER’s research spans diverse areas, including reading and writing, mathematics and science education, teacher quality and leadership. Its tasks include identifying impactful programmes and policies, developing new interventions (e.g. curricula, teacher professional development programmes), evaluating the efficacy and effectiveness of specific interventions, and developing and validating evaluations, thereby contributing to the understanding of teaching and learning.
National Center for Education Statistics (NCES): As the primary statistical agency, NCES collects, compiles and analyses comprehensive statistical information on American education.
National Center for Education Evaluation and Regional Assistance (NCEE): Specialising in impact studies, NCEE evaluates federally funded education programmes, particularly in reading, mathematics and science, providing crucial insights for education policy and practice.
National Center for Special Education Research (NCSER): NCSER advances practices for teaching children with disabilities through research grants, supporting interventions and validating measures.
Regional Educational Laboratories (RELs): The Institute of Education Sciences operates ten RELs, which collaborate with educators and policy makers to generate and apply evidence, aiming to enhance learner outcomes. Their work is intended to be change-oriented, rigorous and impactful, supporting decisions on education policies, programmes, and practices at local, regional or statewide levels. RELs contribute research to understand variations in educational experiences based on context and student groups, influencing outcomes and identifying potential solutions.
This strategic distribution of tasks among the four centres ensures a comprehensive and focused approach to education research. While NCER and NCEE address broader educational challenges and programme evaluations, NCES concentrates on statistical data collection, and NCSER specifically targets research related to special education. In addition, RELs enhance the use of the evidence generated to propose specific and targeted solutions to schools in need.
This collaborative and divisional structure enhances the efficiency and effectiveness of the Institute’s efforts to improve education outcomes in the United States.
Source: IES (n.d.[8]), Institute of Education Sciences, https://ies.ed.gov/.
Narrowing the scope of FINEEC’s work could allow it to spend more time explaining the findings of its evaluations in detail. The evaluations conducted by FINEEC provide crucial insights for EDUFI, education providers, and the OKM, thereby informing their decision-making processes.
However, ensuring that FINEEC’s findings are communicated effectively is essential to enhance policy formulation. Establishing robust communication channels between FINEEC, the OKM, and EDUFI is critical to facilitate this. Implementing formal mechanisms for regular discussions among these entities about FINEEC's evaluation outcomes could lead to more targeted and effective policy responses to identified issues. These discussions need not always culminate in immediate policy actions but could instead highlight areas requiring further investigation or alternative solution testing.
In this regard, each organisation could play to its strengths: EDUFI could lead in designing innovative solutions, the OKM could allocate resources for piloting and assessing these solutions, and FINEEC could oversee the evaluation process. This approach is not unprecedented; for example, in 2020, FINEEC evaluated a pilot project on free early childhood education for five-year-olds, assessing its impact on participation rates, costs and organisational aspects (FINEEC, 2020[9]). Such evaluations provide the OKM with critical data, informing more grounded and effective policy decisions.
Although FINEEC’s autonomy is legally defined and widely recognised as crucial for safeguarding the integrity of its work, there is some divergence in how stakeholders interpret its scope and its practical implications. This discrepancy is primarily observed between the OKM and FINEEC itself. Many of the stakeholders interviewed, including FINEEC, interpret the autonomy of FINEEC not only to mean that it has independence in terms of how it conducts its evaluations and publishes its findings but also that it is subject to minimal steering by the OKM in relation to the evaluations selected to be part of the Evaluation Plan. While some stakeholders consider this independence vital for ensuring FINEEC's objectivity and the credibility of its findings, others express concern that it reduces FINEEC’s incentives to adapt to changing priorities, particularly those of the OKM.
This disagreement about the nature of FINEEC’s autonomy leads to substantial challenges, notably with regard to the nature of communication between the OKM and FINEEC. For example, the OKM has expressed concern that any requests or proposals it may make with respect to the work of FINEEC could be regarded as a directive and be perceived as infringing on FINEEC’s autonomy. This concern is further compounded by the fact that the OKM is a key funder of FINEEC. For these reasons, the OKM may be less than fully clear in articulating its needs to FINEEC.
Interviews with and surveys of FINEEC’s stakeholders reveal that FINEEC’s autonomy is widely recognised as being important for safeguarding the integrity of its work from undue external interference. FINEEC’s autonomy is established in its governing legal framework and is reinforced by its financial independence. The act establishing FINEEC states, “The Finnish Education Evaluation Centre operates in the capacity of an independent expert organisation for external evaluations of education” (Ministry of Education and Culture, Finland, 2013[10]). This autonomy is widely understood to grant it the freedom to independently decide how it conducts evaluations, including its choice of methods and evaluation team members, timetables, scope of reports and other decisions pivotal to educational evaluations. Autonomy also protects FINEEC from third-party influence, including the OKM, EDUFI and education providers.
However, there are some notable differences in how this autonomy is interpreted by FINEEC on the one hand and the OKM on the other. These differences in understanding have practical implications, particularly in undermining effective communication between the two entities and, thus, also undermining the timely measurement of the impact of reforms being implemented.
FINEEC tends to interpret its autonomy in a broader, less restrictive manner. Specifically, it interprets its autonomy as having the freedom to make independent decisions on various aspects of educational evaluation, including, for example, what topics to evaluate and how, as well as which assessments to conduct at different levels of education. In this interpretation of its autonomy, FINEEC has wide-ranging operational flexibility.
For its part, the OKM views FINEEC’s autonomy as more limited. The OKM interprets FINEEC’s autonomy as meaning it has the liberty to choose how it operates, such as the freedom to select its methodologies, tools and internal organisation, but not the right to make independent choices about what areas or levels of education to evaluate.
Differences in how FINEEC’s autonomy is interpreted particularly surface in relation to the elaboration of the Evaluation Plan. Although there are several discussions between FINEEC and the OKM prior to the approval of the plan, in practice, the OKM has little influence over what is included in FINEEC’s Evaluation Plan (see Key Finding 3). The OKM’s lack of influence over what is included in the Evaluation Plan often results in evaluations that lack direct policy relevance for the OKM. From the OKM’s perspective, this means it does not have access to the evidence base it needs to support its decision making.
To ensure it has the evidence base it needs, the OKM has supplemented FINEEC’s resources to undertake additional evaluations that the OKM considered strategic but were not included in the original Evaluation Plan. Ministry officials note, however, that the current period of fiscal restraint will make it more difficult to find supplementary funds to undertake additional evaluations in response to evolving priorities.
This funding arrangement has generated some disagreement between FINEEC and the OKM in relation to which evaluations should be included in the Evaluation Plan and which should be left out and financed through the OKM’s development funds. The question of which evaluations should be encompassed or excluded from the Evaluation Plan is further complicated by the absence of synchronisation between its development and electoral cycles. As FINEEC, OKM staff and various stakeholders highlighted, this lack of alignment can result in a misalignment between the topics in the Evaluation Plan and the government’s current priorities This misalignment could emerge when incoming governments propose new reforms for evaluation or express the need to assess existing institutions or programmes. Given that the timeline for evaluations may span across multiple electoral cycles, some degree of misalignment is inevitable. The difficulty in adapting the Evaluation Plan to new needs is developed in more detail in Key Finding 3.
The disparity in the interpretation of FINEEC’s autonomy, from the broader, strategic autonomy envisioned by FINEEC and the more operational autonomy envisioned by the OKM, leads to tensions between the organisations and hinders effective communication. There is a need to clarify the nature of FINEEC’s autonomy and the OKM’s role in guiding the work of FINEEC.
As a first step, FINEEC and the OKM should openly discuss their perspectives and map points of divergence. This step would involve discussing decision-making processes, methodological freedom and the extent of independence in setting objectives and tasks.
Additionally, the two entities could establish regular and structured communication channels, particularly at the strategic level. These channels could facilitate ongoing dialogue, ensuring alignment and efficiently addressing misunderstandings.
The dialogue between FINEEC and the OKM, as well as the resolutions taken as a result, could be made publicly transparent. This would protect FINEEC’s valued and well-regarded autonomy.
While the process for developing FINEEC’s four-year Evaluation Plan is often praised for being participatory and deliberative, many stakeholders, including the OKM, report that the resulting plan could be designed with greater flexibility to accommodate undertaking new evaluation topics in line with new or emerging priorities.
Although a review and adjustment of the plan is conducted two years after its implementation, the need for greater adaptability remains evident due to concerns from educational institutions over the delays in FINEEC’s results publication. There is a risk that what was initially deemed relevant at the start of the cycle may no longer hold the same relevance upon publication. This highlights the importance of improving the ability to identify change or the emergence of new evaluation needs, and to improve the flexibility of the Evaluation Plan to respond to these changes, in order to maintain the relevance and impact of FINEEC’s work.
Despite the scheduled review and adjustment of the Evaluation Plan in the second year of its four-year duration (see Box 3.3 for more information on the Evaluation Plan), integrating new evaluation proposals proposed by the OKM remains challenging in practice. This is partly due to the absence of formalised channels for communicating the government’s changing priorities and adapting FINEEC’s activities accordingly. Furthermore, undertaking new, additional evaluations would entail replacing previously prioritised and planned evaluations. While some argue that this situation helps protect the Evaluation Plan and FINEEC’s work from political interference and short-term thinking, there is a need to consider how new priorities can be accommodated within the lifecycle of the plan without undermining the organisation’s autonomy (see Key Finding 3 for further analysis of FINEEC’s autonomy).
The FINEEC Evaluation Plan is a crucial document developed over four years to guide educational evaluations. It is crafted through a consultative process led by the Evaluation Council, appointed by the OKM, and operating in conjunction with FINEEC. The process adheres to legislative requirements for broad stakeholder collaboration, ensuring that the plan is co-created with input from key users of the evaluations, including on proposals for future themes. This collaboration guarantees that the themes are timely, relevant and resonate with stakeholder needs and motivations. The OKM is responsible for approving the Evaluation Plan.
The plan’s content is informed by a desire to fortify the national knowledge base in education and to integrate findings from prior research and external sources. It is designed to be responsive and adaptable, aligning with the dynamic nature of educational policies and societal needs. A routine review of the plan is conducted at least once midway (after the second year) through the four-year cycle. This review serves as a reflective exercise to examine the outcomes of completed evaluations and to deliberate on any necessary updates to the plan. For instance, during the coronavirus (COVID‑19) pandemic, the plan was promptly adjusted to evaluate the impact of the crisis on learning, demonstrating its capacity for flexibility in the face of unforeseen circumstances. As in the original Evaluation Plan, the OKM is responsible for approving any subsequent modifications based on the Council’s proposals.
Despite this adaptability, the four-year span of the plan also offers stability, providing predictability for education providers and ensuring that evaluation results can be seamlessly integrated into ongoing development projects. This stability is also strategic in preventing overlap with other surveys and assessments.
Source: FINEEC (2024[11]), Plan for Education Evaluation 2020–2023, www.karvi.fi/en/about-us/about-our-evaluations/plan-education-evaluation-2020-2023.
Like the OKM, education providers have expressed similar concerns about the lack of responsiveness of the Evaluation Plan. While educational institutions widely recognise and value FINEEC’s work and relevance, viewing it as an impartial entity that produces work of high quality and relevance, concerns have been raised regarding the timing of the publication of FINEEC’s results. Given the four-year cycle of the plan, some evaluations are not published until the end of this period or later. For example, in the 2020‑2023 Plan defined in 2019, the evaluations “Overall evaluation of the qualification system of VET”, “Evaluation of the distance learning pilot for Sámi languages”, and “Extending compulsory education: Evaluation of the new forms of guidance counselling”, were planned to be conducted in 2022 or 2023, and therefore the results were published between three and five years after the need to conduct them was detected. As institutional priorities can shift over these four years and new priorities emerge, some results, by the time of publication, might become less relevant to the institutions. Education providers also pointed out that they are not consulted during the review and adjustment of the four-year Evaluation Plan on the changing needs for evaluation (carried out two years after its implementation).
It is important to note that current legislation requires FINEEC to consult key stakeholders in preparing the plan. Therefore, any changes to the plan should consider stakeholder consultation processes, be discussed in the Evaluation Council, and be validated by the OKM.
FINEEC could build greater flexibility into its approach to developing and managing its Evaluation Plan. A formal communication mechanism between FINEEC and the OKM could be established, particularly at the strategic level, to enhance the flexibility of FINEEC’s Evaluation Plan. For example, at the time of the mid‑term review of the Evaluation Plan, the OKM could make a formal request to FINEEC about the inclusion of new evaluations in the Plan. FINEEC, for its part, could give a formal response on whether or not to include the assessment into the Plan, and the reasons for its decision. This would enable FINEEC to better align its work with changing governmental priorities without compromising its autonomy.
Given that current legislation requires FINEEC to consult key stakeholders in preparing the plan, it is important that changes to the plan are discussed and approved at least by the Evaluation Council. Other consultation mechanisms could be used in this process of reviewing and adjusting the plan in order to analyse possible changes in assessment needs (e.g. due to possible changes in the context) and to inform the process of adjusting the plan.
Additionally, a certain predetermined portion of funding could be set aside at the beginning of an Evaluation Plan cycle to function as a “reserve fund”. This reserve fund could be drawn upon at the mid-cycle review point to fund work on new or emerging priorities. Another option would be to have a scheduled opportunity to replace previously planned topics that have not yet commenced and which are of lesser priority with topics of emerging or increasing importance. This could happen during the existing second-year review of implementation.
To respond to the changing context and changing needs of educational institutions, the Evaluation Plan could be amended to allow for more frequent updates and the incorporation of new evaluation topics, thereby ensuring it remains aligned with their evolving needs.
FINEEC’s extensive work across numerous areas and educational levels, covering a broad spectrum of the education system, is highly valued by stakeholders for its comprehensiveness. This widespread recognition reflects FINEEC’s commitment to be a relevant actor in the entire education system. However, pressures on FINEEC to increase the scope of its work have led it to continually seek new funding sources to deliver what is expected. FINEEC has, to some extent, contributed to creating this pressure on itself by planning more activities than its budget allows. As a result, some FINEEC staff members report experiencing excessive workload and insufficient time to devote to each project, potentially compromising the quality of the institution’s work.
According to the findings of the FINEEC self-assessment questionnaire and statements made by some FINEEC staff consulted for this evaluation, the institution’s funding model might be creating incentives for it to take on more work than it can execute, causing some teams to be overloaded and/or limiting the time they allocate to each project, both of which could have an impact on quality.
As mentioned in the context section of this report, FINEEC receives an annual appropriation for operational expenditure directly from the state. This direct allocation should at least cover the execution of the Evaluation Plan and the funding of its permanent staff.
While FINEEC receives additional funds from the OKM, this funding is intended to finance temporary staff and operational costs of projects commissioned by the OKM that fall outside the Evaluation Plan. In practice, these additional projects also receive both technical and administrative support from FINEEC's permanent staff, but these resources do not cover the costs of their contributions. Instead, these costs have to be financed indirectly through the annual appropriation for FINEEC’s operational expenditure, which comes directly from the state. Certain permanent FINEEC staff have reported that this arrangement leads to substantial additional work overload.
The implication is that taking on supplementary work (i.e. work outside of the Evaluation Plan) may negatively affect FINEEC’s ability to implement its Evaluation Plan and/or impact the quality of the work it completes. On this point, some OKM representatives report that the information collected by FINEEC is sometimes not sufficiently exploited (i.e. several key analyses are not being carried out) or that certain recommendations are not sufficiently concrete or targeted (this issue is developed in more detail in Key Finding 1). When asked why they carry out the projects requested by the OKM despite the additional workload implications, FINEEC employees referenced the organisation’s funding pressures and the strategic need for, or personal interest in, the topics addressed in these evaluations.
The scope of FINEEC’s work has broadened to include evaluating educational levels not originally within its purview, such as early childhood education and care (ECEC), assessing new topics like recent educational reforms, and integrating new methodologies using digital platforms like Joda for learning outcomes and Valssi for quality management. Although the annual appropriations for operational expenditure coming directly from the state were not adjusted to cope with this increase in scope, it should be noted that the OKM, particularly its departments of ECEC and General Education, has provided additional resources to finance these new tasks.
Another challenge associated with FINEEC’s economic model is the workload pressures that fee-based services generate. In general, FINEEC employees perceive fee-based services positively, as they allow FINEEC to generate some additional income (unlike income from the OKM, income from this type of service must cover all costs associated with the project, including technical and administrative support from FINEEC). However, teams in charge of implementing fee-based services, such as the General Education Unit team, also reported being overloaded with work during some periods. The General Education Unit assesses learning outcomes in basic education using a sample-based approach, assessing 10% of students in each round. Education providers who are not part of the sample can request assessments of their schools from FINEEC for a fee. In 2023, fee-based services led to almost double the sample size originally planned for some of the tests.
Currently, to cope with the excessive workload experienced by some people or some teams, staff members are expending extra unfunded time training and spearheading internal development projects aimed at enhancing the competencies of their colleagues. The training aims to ensure that a greater share of FINEEC staff is familiar with the different methodologies needed to carry out the range of evaluations it is tasked with. This may improve efficiency in the long run, but in the short run, it further contributes to the funding shortfall. Furthermore, it is not evident that this approach will improve efficiency even in the longer term, given that a high proportion of FINEEC’s workforce is temporary staff (almost 40%), many of whom are employed on a project basis and therefore have relatively short tenures at the institution.
To better manage workload pressures so that they do not jeopardise the quality of FINEEC’s work, FINEEC should ensure that all permanent staff and activities of the Evaluation Plan are financed by the annual appropriation for operational expenditure originating directly from the state. Other projects (which are not part of the plan) requested either by the OKM or by education providers should be accepted only as long as they fund all costs associated with their conduct, including the supervision of staff, the provision of technical and administrative support and other overheads (e.g. information technology [IT], infrastructure, etc.). Accounting for the real costs of delivering projects will give FINEEC the flexibility to take on these additional projects without compromising its overall work.
While FINEEC undertakes evaluations and research on a broad spectrum of topics across all levels of education around key themes, these studies are often conducted in isolation, with the linkages or synergies between them left underexplored. For example, few attempts are made to use the outcomes of learning assessments to enhance thematic evaluations. This compartmentalisation not only limits the depth and breadth of insights that can be drawn from individual studies but also restricts the capacity to understand complex, inter-related issues across the education system.
Some stakeholders interviewed noted that this lack of complementarity may arise from a tendency for teams in FINEEC to operate in silos, with each focusing on their specific studies in isolation from others.
FINEEC organises its evaluations around key themes, seeking to generate synergies between the different evaluations carried out by the organisation. The 2020-2023 Evaluation Plan organised its evaluations around four thematic areas: 1) learning and competence development; 2) promoting equity; 3) improving the effectiveness of the education system; and 4) supporting continuous improvement.
In addition, FINEEC seeks to connect staff working on these different issues through working committees on each theme. The work of these committees translates into publications that summarise various evaluations carried out on the key issues. An example of this is the report, Equality and Participation in Education – An Overview of National Evaluations (FINEEC, 2021[4]), which, based on the results of several studies carried out by the organisation, sought to answer the questions: What is the state of equality and equal opportunities in education in Finland? Which factors enhance or hinder the implementation of equality in education?
Additionally, FINEEC recently launched a longitudinal assessment of learning outcomes in basic education, intended to track learning outcomes and changes in students’ competency levels, necessitating increased collaboration among teams (FINEEC, 2023[12]). Despite these efforts to foster complementarity between these strands of work, still more can be done.
The most evident lack of complementarity is between assessments of learning outcomes and thematic studies. Typically, work in these areas is planned and executed separately, with insufficient thought given to potential synergies between them. For example, insights from learning outcome assessments could inform and enhance the quality of thematic evaluation studies, yet little is done at present to link the findings of the former with the latter.
More concretely, while FINEEC conducted several thematic evaluations between 2020 and 2023 on topics such as bullying (FINEEC, 2023[13]), integration of migrant students (FINEEC, 2019[14]), and bi- and multilingualism in Swedish-language schools in Finland (FINEEC, 2019[15]), these studies have not been linked to evaluations of learning outcomes.
Specifically, aligning factors such as questionnaire design, expert involvement, and the use of overlapping samples would enable the use of data from one study to elucidate outcomes in the other. This approach would mutually enrich both types of studies, providing more thorough and insightful findings. For example, how the implementation of practices to support the inclusion of immigrant students impacts the academic performance of these same students could be analysed.
FINEEC’s work on higher education provides another good example of these untapped synergies. Specifically, its audits of higher education institutions provide a wealth of detailed insights into institutions’ operations, yet this rich data pool is not effectively incorporated to enhance the analysis in thematic reports. This rich institutional-level data could be used to develop and enrich thematic reports on current and emerging challenges in the higher education sector, such as evaluations of pedagogical reforms as well as field-specific studies. For example, through thematic evaluations, FINEEC could investigate the kind of pedagogical practices or internal university policies associated with students’ better academic or employment performance.
Fostering synergies between the different strands of work undertaken by FINEEC could result in findings that have important policy implications. This could be achieved by encouraging greater information sharing and collaboration between teams working on diverse levels of education, subjects and type of evaluations. This type of collaboration could exist at several stages: During the development of the Evaluation Plan to enhance synergy and complementarity between evaluations, during the design of evaluations to increase the coherence of the approaches used, and after evaluations, to find points of convergence or divergence between educational levels, subjects and type of evaluations. Working in this fashion would not only generate new insights but could also improve internal efficiency.
A notable effort by FINEEC to move in this direction is the implementation of the longitudinal assessment of learning outcomes in basic education mentioned above (FINEEC, 2023[12]). This required broad institutional efforts and cross-team collaboration. The initiative aims to track learning outcomes and changes in students' competence curves, focusing on key subjects and transversal competencies in accordance with the new Finnish national core curriculum. By integrating these longitudinal insights, FINEEC enhances its capacity to understand and address complex, inter-related issues across the education system, illustrating its innovative and forward-thinking approach to educational evaluation.
The Evaluation Council could play an important role in fostering greater complementarity by providing more strategic guidance to FINEEC. The Council could ask FINEEC to conduct evaluations on certain topics at several educational levels simultaneously. For example, it could request that FINEEC carry out a thematic evaluation on the teaching and learning of digital competencies in primary, secondary and higher education. This approach would allow for the identification of strengths and areas for improvement in integrating digital competencies across different educational levels, thus promoting a coherent and complementary approach to the development of educational policies and programmes related to digitalisation. Such strategic oversight could better orient FINEEC’s work, ensuring that activities are aligned with overarching goals and conducted in a more integrated and efficient manner.
Finally, more frequent and focused Evaluation Council statements on pivotal issues could be highly beneficial. These statements address cross-cutting issues at various levels of education, taking a position on reforms, issues related to evaluating the educational system, use of data, etc. In 2021, FINEEC published 9 statements, representing 11% of FINEEC’s publications in FINEEC’s publication series. During the years 2022 and 2023, the number of statements increased to 25 and 26 respectively, representing 23% and 53% of the total number of publications. Historically, these statements have been instrumental in reaching consensus on important matters related to the work of FINEEC and the assessment of the education system in general. For example, the FINEEC statement “Consolidated opinion on the reform of funding models for universities and universities of applied sciences” contributes to reaching consensus on crucial issues by providing a detailed and evidence-based analysis of the challenges and opportunities in higher education. The document identifies critical issues, such as competition between institutions, and aligns with national policies to improve the education system. It provides clear recommendations to guide institutions towards best practice, focusing on strategic funding to foster collaboration, improve educational trajectories and support staff and student well-being. In this way, FINEEC creates a starting point and common ground for education policy debate and formulation in Finland.
FINEEC conducts its assessment of learning outcomes on a sample basis, meaning that not all schools in the country are included in each evaluation. However, FINEEC offers the opportunity for municipalities in the country to include some or all schools in their districts in the evaluation sample, subject to a fee. The decision to participate is driven not just by a municipality’s financial resources but also by its technical and administrative capabilities. This creates inequalities, as relatively large municipalities with stronger resource bases and administrative capacity are better positioned to benefit from the valuable insights offered by these assessments.
Since many municipalities in Finland have opted to include all schools in their districts in the sample, this has also raised the possibility of creating school rankings at the municipal level, a broadly unwelcome outcome in Finland.
As previously mentioned in Key Finding 4, FINEEC’s evaluation of learning outcomes in basic education employs a sample-based approach, where 10% of students are evaluated in each round. This approach was decided in the 1990s and implemented first by the Finnish National Board of Education and, since 2014, by FINEEC. The foremost purpose of assessments of learning outcomes is to produce national-level data on how well students are acquiring the targeted competencies in a variety of subjects. At the school level, the sampled schools receive locally targeted feedback on their learning outcomes. School feedback provides schools with information on their performance in relation to national performance levels, which helps them identify where they need to improve. A sample-based approach aims to safeguard against teaching to the test and the creation of “ranking lists” that permit comparisons between schools.
In addition to the 10% sample, education providers (municipalities and private providers) can request FINEEC to evaluate some or all of the schools in their jurisdiction on a fee-paying basis. This model poses a challenge for some municipalities, particularly smaller ones or those in rural areas. While larger municipalities may have the financial resources and human capacity to co-ordinate the evaluations in their schools, smaller municipalities, on the other hand, tend to lack the necessary resources and staff. Small municipalities’ administrative and managerial staff usually have multiple roles and, therefore, do not have the time to dedicate to evaluation activities. Moreover, the competencies of these small municipalities to manage the assessments in their schools and then analyse the assessment results are often very limited. As a result, small municipalities may not have access to the same level of information as larger municipalities and, therefore, find themselves in a more disadvantaged position with respect to their capacity to understand the performance of their schools in detail and, by extension, undertake measures to improve quality where appropriate.
A related concern is using this data to construct school rankings, which, by enabling parents to select better schools for their children, undermines the principle of equitable access to education and potentially reinforces the divide between high- and low-performing schools. Indeed, some municipalities, especially large ones, have commissioned FINEEC to evaluate all schools in their jurisdictions. While these municipalities benefit from having objective and regular information on their students’ performance, the fact that all schools in large cities can be evaluated allows student performance to be compared and the schools themselves to be ranked.
Finland values and promotes equality and has tried to avoid school rankings specifically to safeguard this equality. However, since the results of the evaluations are in the public domain, some media outlets have started to generate rankings [see Helsingin Uutiset (2023[16]); Yle (2023[17]); and Helsingin Sanomat (2022[18])]. Authorities fear these rankings may encourage “school shopping” and undermine equality. Moreover, since the data upon which such rankings are constructed are typically not well described and contextualised, the rankings may even serve to misinform the public about the merits of changing schools (see related literature in Box 3.4). The extent to which these rankings influence or might influence decisions made by parents and students in Finland has not yet been formally studied. However, the existing literature suggests that school rankings influence parental behaviour (see Box 3.5).
Rankings may offer imprecise assessments of school quality for several reasons. First, they tend to be volatile and lack precision (scores may be influenced by one-time events, such as illnesses or external distractions, and can exhibit variation due to sampling differences in each cohort of students), as highlighted in several studies; (Chay, 2005[19]; Kane and Staiger, 2002[20]). Second, these rankings often reflect factors beyond a school’s control, such as the socio-economic background of students’ families, rather than indicating the actual quality of well-performing schools (Mizala, Romaguera and Urquiola, 2007[21]). Third, the public disclosure of school results may prompt schools to overly emphasise easily measurable test-specific skills, neglecting other less quantifiable aspects of education (Reback, 2008[22]; Neal and Schanzenbach, 2010[23]). Four, accountability systems can be manipulated, leading to questionable reported benefits as schools may engage in gaming or cheating without genuinely enhancing the evaluation of effective quality (Cullen, 2006[24]; Figlio and Getzler, 2006[25]) Finally, the revelation of unfavourable outcomes in the public domain could lead to unintended consequences for fairness. This may manifest through selective enrolment practices by both students and teachers, commonly known as cream-skimming, or result in motivational effects, as indicated in several studies (Clotfelter et al., 2004[26]; Figlio, 2004[27]; Schütz, 2007[28]).
Source: Nunes, Luis C., Ana Balcão Reis and Carmo Seabra (2015[29]), “The publication of school rankings: A step toward increased accountability?” https://doi.org/10.1016/j.econedurev.2015.07.008.
The influence of school-quality information on parental decision making in education has been a subject of research, with notable contributions from studies by Hastings, Van Weelden and Weinstein (2007[30]). Investigating the direct impact of providing explicit information about alternative schools to parents, these studies found a substantial increase in the selection of higher-scoring schools.
Building on this evidence base, a study by Pierre Koning and Karen van der Wiel (2013[31]) examines the effects of media rankings, particularly those published by the Dutch newspaper Trouw, on school choices in the Netherlands. This study’s findings reveal that parents respond to the published school‑quality information, particularly in the case of academic school tracks. The inflow of first-year students significantly increases by more than 20% when a track receives the most positive score from Trouw. At the same time, the authors see a decrease in the inflow of students to the worst-performing schools. However, the study finds that the overall impact of the publication of school-quality information is relatively small compared to the impact of a school’s overall reputation or proximity to home. The study suggests that the noisiness and volatility of quality rankings may explain the modest overall impact. Additionally, the information’s value may be diminished if perceived as transitory. Despite these nuances, the evidence suggests that school-quality information is reshaping the educational market, particularly for academic tracks in the Netherlands. According to the authors, over time, there is the possibility of quality either converging among schools or witnessing a strengthening differentiation among them.
In line with Koning and Wiel (2013[31]), the results of Nunes, Balcão Reis and Seabra (2015[29]) also suggest that the publication of the rankings in the media has clear effects on families and schools in Portugal. Following the publication of the rankings by the media, fewer students enrol in the lowest-rated schools and the likelihood of school closure increases.
Furthermore, the public revelation of unfavourable outcomes may lead to unintended consequences for equity, including the selective enrolment of both students and teachers or the emergence of motivational effects (Clotfelter et al., 2004[26]; Figlio, 2004[27]; Schütz, 2007[28]).
Source: Koning, P. and K. van der Wiel (2013[31]), “Ranking the schools: How school-quality information affects school choice in the Netherlands”, https://doi.org/10.1111/jeea.12005; Nunes, Luis C., Ana Balcão Reis and Carmo Seabra (2015[29]), “The publication of school rankings: A step toward increased accountability?”, https://doi.org/10.1016/j.econedurev.2015.07.008.
The current sample-based outcome assessment model, with the option to be included in the measurement for a fee, is having some negative, unintended consequences that need to be managed. Consideration could be given to limiting the access of municipalities to learning assessments on a fee-paying basis outside of the regular sample. However, such a measure may not be viable or desirable as it would undermine the statutory duty of education providers to evaluate themselves.
Another option to address the inequity of access to the services offered by FINEEC would be to reduce the barriers to access to learning outcomes assessments for small municipalities to level the playing field. FINEEC has recently developed a new IT tool that will allow learning outcomes assessments to be carried out on line, opening up an opportunity to lower barriers to accessing assessments. For schools that are not part of the sample but want to be assessed nonetheless, self-assessment could be facilitated, and some processes and analyses could be automated to lower the costs of conducting the assessments. For the correct application of the assessments, schools could receive online training that provides information on the application of the assessments and the correct interpretation of the data. The drawback of such an approach is that more municipalities will undoubtedly choose to be assessed (or to assess themselves), thereby raising the risk of rankings; however, such an outcome may be inevitable and can be perhaps mitigated by implementing strategies to inform the population on how to best interpret the results (see below for more on this).
Since some media outlets have already started to generate and publicise school rankings, FINEEC and the OKM may wish to consider taking a more active role in contextualising these results to limit their misuse. Currently, FINEEC conducts learning outcomes assessments regularly for mathematics and Finnish, making it possible to observe trends. Therefore, FINEEC and the OKM could join forces to raise awareness about the importance of observing trends and, thereby, facilitate the more informed interpretation of assessment results. In addition, other key indicators on schools could be assessed and included in the reporting, such as academic self-esteem and school motivation, school attendance, school retention, school coexistence climate, participation and citizenship training, healthy living habits and gender equity, among others, which are essential for assessing the overall value of education. Some indicators, such as school attendance and retention, would be easily accessible, as they could be obtained from administrative data. However, collecting data on some other indicators would be costly and, therefore, unlikely to be undertaken in the context of fiscal restraint.
FINEEC employs mixed-method approaches in its studies. However, some of FINEEC’s studies aim to respond to policy questions that can only be adequately answered by causal evidence. Notwithstanding this need, in most of these cases, the methods and analysis undertaken by FINEEC do not produce causal evidence. This shortcoming is compounded by the fact that despite lacking causal evidence, many FINEEC evaluations provide policy recommendations. This inadvertently pressures the OKM to adopt reforms that may not be adequately supported by causal evidence. These deficiencies have been noted by stakeholders and confirmed by the OECD through reviews of primary sources and specific examples.
OECD analysis of a sample of FINEEC’s reports reveals that FINEEC predominantly employs a mixed-method approach, integrating both qualitative and quantitative techniques, such as survey analysis, interviews and descriptive statistics analysis in its studies. This approach typically offers a holistic view of educational practices and their impacts on student outcomes, supporting a comprehensive understanding of how and why certain educational interventions succeed (Wenger-Trayner et al., 2019[32]). Combining qualitative and quantitative analyses can capture the complexity of how learners interact with and benefit from different teaching strategies, ultimately leading to more effective educational policies and initiatives. Through this integrated methodology, mixed-method evaluations deepen the understanding of educational systems and inform the development of more nuanced and effective educational practices and policies (Muhonen et al., 2018[33]).
The studies conducted with these methods are considered to be of significant value by stakeholders, especially for educational institutions such as VET schools, higher education institutions and municipalities. For instance, institutional audits offer comprehensive evaluations of internal processes, aiding institutions in enhancing mechanisms such as internal quality assurance. Furthermore, FINEEC’s thematic evaluations, which typically employ a range of methods, including expert consultation, provide institutions with insightful analyses of educational trends and enable them to adopt and learn from best practices.
Despite these notable contributions, the methods employed by FINEEC are often not well-suited to answering certain questions important for policy making, as outlined in the Evaluation Plan. Specifically, there is a notable gap in FINEEC’s ability to establish statistical causality to better inform policy making. For example, FINEEC does not routinely conduct causal evaluations or apply experimental or quasi-experimental designs, which are important methods for uncovering and understanding causal relationships. Instead, FINEEC’s methods tend to yield descriptive rather than inferential insights.
To illustrate, the 2023 FINEEC report, Evaluation of Anti-Bullying Methods: Usability, Sustainability and Outcomes of Seven Methods Selected for the Evaluation (FINEEC, 2023[13]), aimed to create criteria for evaluating the effectiveness of anti-bullying tools and promote the use of the most functional methods in schools and ECEC units. However, the methodologies applied were not well-suited to these impact-focused goals. The evaluation adopted a mixed-methods approach, combining quantitative data from surveys of school staff, pupils and tutors with qualitative data from group interviews. These methodologies, however, were not capable of establishing causality between the different anti-bullying models and bullying rates. These shortcomings were also evident in the evaluation of absenteeism prevention models (FINEEC, 2024[34]), which employed a methodology that did not permit an assessment of the impacts of the models but did facilitate an understanding of different actors' perceptions of the models and did identify possible areas for improvement.
Causal evidence is crucial for policy making as it provides a solid foundation for understanding the direct effects of policies or interventions on outcomes, thereby enabling more informed, effective and targeted decision making. For example, in the context of the observed decline in educational performance in Finland, there is a need to generate understanding and more evidence on the factors affecting student performance. Such evidence would permit policy makers to design interventions and programmes targeted at the needs of specific groups of underperforming students. Similarly, in order to develop policies that improve the employability of VET graduates, causal evidence is needed on the relationship between teaching practices and/or institutional processes on the one hand and employability outcomes on the other (see also Key Finding 8).
Despite the lack of use of causal methods, FINEEC frequently issues recommendations (see Key Finding 1), which readers may assume are based on such causal evidence. The OKM has voiced concern that this situation has inadvertently placed pressure on them, and potentially others as well, to adopt policies that are not adequately supported by evidence.
FINEEC could strengthen its capacity to conduct causal inference research to better support policy making. FINEEC could also adopt a policy of only making recommendations when evidence clearly supports it. However, this would require agreement on what constitutes adequate evidence for policy making or limit the circumstances in which it makes recommendations (see Key Finding 1).
In instances where the recommendations presented lack robustness for formulating policy recommendations, FINEEC could include more detailed notes on methodological limitations and clearly articulate what inferences can and cannot be drawn from their analyses. Additionally, including a comprehensive review of research literature on effective policy or educational interventions from other countries could enrich its analysis. This expanded review could be accompanied by a detailed analysis of external validity, ensuring that the findings from one context can be accurately applied to similar programmes, times or places beyond the scope of the original evaluation. In some countries, such as Switzerland and the United States, there are bodies dedicated to evaluating existing research and reports, categorising them according to their ability to answer causal questions, and then publishing reports summarising the existing evidence (see Box 3.6).
It should be noted that not all aspects of FINEEC’s activities require applying causal methods. Indeed, FINEEC answers many of the questions in its Evaluation Plan effectively and offers substantial inputs without relying on causal methods.
The WWC, established in 2002 as part of the Institute of Education Sciences (IES) within the US Department of Education, is dedicated to providing credible and succinct information in the field of education. Over the past decade, the WWC has played a central role as a trusted source of scientific evidence, focusing on high-quality research to address the crucial question of “What works in education?” Recognising the variability in education research, the WWC is crucial in ensuring that decisions are informed by reliable information to improve student outcomes.
Managed by IES staff and supported by leading firms, the WWC identifies existing research, assesses its quality, and summarises and disseminates evidence from studies that meet WWC standards. The WWC prioritises topics for intervention reports and practice guides based on their applicability to a broad range of students, policy relevance, potential to improve educational outcomes and the availability of research.
A cornerstone of its approach is that individual study reviews form the basis for all WWC products. This rigorous review process ensures the use of consistent, objective and transparent procedures and standards. The review process conducted by WWC has three phases. In Phase 1, studies meeting eligibility criteria proceed to Phase 2, where eligible findings are rigorously assessed against WWC standards, influencing overall ratings. In the final phase, the WWC synthesises research and effectiveness ratings based on design strength and intervention effects, ensuring credible and reliable findings.
Evaluated studies can receive one of three ratings:
“Meets WWC Standards Without Reservations”, indicating strong and well-executed research design, providing the highest confidence in the intervention’s impact.
“Meets WWC Standards With Reservations”, acknowledging limitations in design or execution that may not entirely rule out alternative causes for the observed effect.
“Does Not Meet WWC Standard”, assigned to findings lacking sufficient evidence that the intervention caused the observed effect.
As a vital resource, the WWC website features effectiveness ratings and specifies the grade levels in which studies were conducted. This ensures that the education community can easily identify research that is effective and relevant to their specific needs and contexts.
Source: WWC (2022[35]), WWC Version 5.0. Procedures and Standards Handbook, https://ies.ed.gov/ncee/wwc/Handbooks.
SCCRE, operating under the Swiss federal government and the Swiss Conference of Cantonal Ministers of Education, focuses on strengthening educational research in Switzerland. Its performance mandate includes documenting Swiss and international educational research, co-ordinating research efforts, and analysing the progress in current educational research topics.
SCCRE's notable contribution is the Swiss Education Report, a comprehensive overview of the education system, first published in 2010. This report is instrumental in shaping joint education policy objectives, providing reliable information for quality assessment, planning measures and setting strategic goals for Swiss education. The continuous efforts of researchers are crucial for enhancing understanding of the complex causal relationships within the education system.
Source: SCCRE (2023[36]), Education Report Switzerland 2023, www.skbf-csre.ch/fileadmin/files/pdf/bildungsberichte/2023/BiBer_2023_E.pdf.
FINEEC collects important information about the operational functioning of higher education and VET institutions, while the OKM, FINEEC, and other institutions assess students’ learning and labour market outcomes. However, only limited efforts are taken to understand the relationship between these findings. There is an opportunity to leverage both of these valuable data sources to better understand whether and how certain operational policies and procedures at the institutional level are associated with better outcomes. For example, FINEEC could leverage insights from its quality assessments of VET institutions and audits of higher education institutions to better understand why, despite recent reforms aimed at improving quality control measures and internal procedures in VET and higher education institutions, the employability rates of their graduates have remained relatively constant over the past decade.
Finland has implemented quality assurance approaches in VET since the 1990s, following national policies and EU guidelines. Since its foundation, FINEEC has played an important role in evaluating VET providers’ quality management systems and supporting them in improving the quality of the education provided (FINEEC, 2024[37]). FINEEC also implements systemic assessments of learning outcomes in VET and conducts thematic evaluations in VET. To evaluate the quality of education provided in VET, FINEEC has relied on education providers’ self-assessments, indirect assessments of learning outcomes, and surveys that collect students’ opinions on teachers’ pedagogical practices. These approaches are usually complemented by the use of qualitative instruments (mainly interviews) conducted with a limited number of education providers [see, for example, FINEEC (2022[6]); FINEEC (2015[38]); and Aila, Tarja and Risto (2022[39])].
In higher education, FINEEC primarily supports institutions in strengthening quality improvement by working with them to develop and implement effective quality management systems that support continuous improvement and the achievement of high standards. For example, FINEEC conducts institutional audits of higher education institutions that adhere to the European standards for quality assurance. Even if higher education institutions are not required to be audited by FINEEC and can opt for other national or international auditing bodies to evaluate their programmes and quality, in practice, most typically choose to be audited by FINEEC. In addition to institutional audits, FINEEC produces thematic reports on relevant higher education topics that generate new insights and recommendations for improvement.
Education providers appreciate these undertakings as they have allowed for the development of a “culture of continuous improvement”. However, these providers have also noted that it is not always apparent how these audits and quality assessments have improved understanding of how their operational decisions are linked to student learning outcomes and graduate employability.
Considerable information about learning outcomes in VET and higher education is already available to explore such correlations. For example, the OKM already collects information on outcomes from these institutions in return for the funding they provide for their operations. Furthermore, as part of a research consortium called the Kappas! Project, the Universities of Helsinki and Jyväskylä conduct assessments of bachelor-level students’ foundational skills, including analytical reasoning, problem solving, writing effectiveness and writing mechanics. In the last assessment carried out between 2018 and 2020, 2 402 students from 7 universities of applied sciences and 11 universities participated (Ursin et al., 2021[40]). Despite having access to these rich data sources, FINEEC has not been exploiting them in their audits and quality assessments.
As noted in Key Finding 7, with the data and methods currently being used by FINEEC, it is currently not possible to know whether a given improvement in the self-assessed quality of pedagogical practices is associated with an improvement in learning outcomes or graduate employability. Therefore, it cannot be known whether improving the self-assessed quality of pedagogical activities or learning outcomes assessment results will necessarily lead to improved learning outcomes assessment results or enhanced graduate employability. While this is particularly a concern with respect to VET, considering that the employability of initial VET graduates has remained relatively low in comparison to the general track and to higher levels of education, it is no less important in higher education, as the employability of higher education graduates has remained relatively constant over the past decade (see Figure 3.1).
Source: Statistics Finland (2020[41]), “Employment of recent graduates with initial vocational qualifications weakened by 8 percentage points”, Transition from school to further education and work 2020, www.stat.fi/til/sijk/2020/sijk_2020_2022-01-20_tie_001_en.html.
According to some stakeholders interviewed by the OECD team, there is no clarity about which factors might impede improvement in graduate employability. While factors could include deficiencies in the quality of education at these levels of education, they could also include deficiencies in the education delivered at previous levels or a mismatch between supply and demand, both in terms of the volume of graduates/employees required and in terms of the competencies and skills delivered/needed.
While correlations between institutional-level policies and procedures and student outcomes may not always be sufficient as evidence to guide policy making (see Key Finding 7 for further information about the need for causal evidence), this information can nonetheless help to identify potential causal relationships that can be tested in future research.
It is important to note that neither the OKM nor EDUFI has access to these evaluation results at the level of individual institutions. This hinders their ability to understand how internal processes influence specific outcomes, such as student learning and graduate employability. This limitation is not due to legal restrictions but rather to the nature of the permissions obtained by FINEEC for collecting data, which guarantees that institutions cannot be identified in the reports.
Consideration could be given to more actively exploiting and linking the outcomes information collected by the OKM, the Kappas! Project and FINEEC, with the information gathered through FINEEC’s process evaluations and assessments, such as audits and quality assessments. This would facilitate a better understanding of the factors affecting outcomes and means for improving outcomes.
Furthermore, to better leverage the available information gathered through the evaluation of institutions, FINEEC could consider revising the permissions framework to allow for greater flexibility in how the results are used while still respecting potential privacy concerns. This could involve negotiating new terms with educational institutions. By adjusting these permissions, entities like FINEEC, the OKM and EDUFI could link evaluation results with publicly available institutional outcomes. Among other things, this approach could allow for a more nuanced understanding of how quality assessments and audits influence specific outcomes, such as student learning and graduate employability.
FINEEC plays an important and highly valued role in the VET sector, providing comprehensive support and insights that support institutional quality development through its evaluations of their learning outcomes [see, for example, FINEEC (2022[6]) and (2023[42])] and its thematic evaluations [see, for example, Aila, Tarja and Risto (2022[39]) and FINEEC (2023[43])] of the VET sector.
Despite these significant contributions, access to more reliable evidence is needed on VET institution graduates’ learning outcomes to support policy making. Specifically, in the absence of more direct measures of the skills performance of VET graduates, FINEEC often assesses VET learning outcomes indirectly using the grades given to students by teachers as a proxy. However, since the grading of VET students is not standardised in Finland, these grades may not be sufficiently robust and reliable (see Box 3.7). FINEEC is well aware of the limitations of using grades as a proxy for skills and has, as a result, complemented these indirect measures of learning outcomes with other qualitative data approaches and, in a few circumstances, direct assessments of learning achievement in general disciplines (e.g. mathematics). Still, more could be done to improve the reliability of data on the learning outcomes of VET graduates to support informed policy making.
Although FINEEC assesses learning outcomes in VET, these assessments are generally based on indirect measures. For several years, the assessment of learning outcomes in vocational upper secondary education, both vocational modules and general modules, has been based on the grades given to students by their teachers and/or employers’ representatives. While VET teachers and employers have direct knowledge of the students and their abilities and can, therefore, provide relevant input in the assessment, the grades they award may not be comparable, as institutions, teachers and employers may vary in how they assess student outcomes and may have different grading standards (OECD, 2022[44]). This not only affects the reliability of the assessments but also their validity, as an inconsistent measurement cannot accurately predict occupational competence (see Box 3.7).
This heterogeneity in assessments and marking has, in fact, been acknowledged in certain FINEEC studies (FINEEC, 2022[6]). Currently, in order to compensate for this lack of reliability, FINEEC complements the information gathered from the learning outcomes assessment with qualitative information, including VET providers’ self-assessments; interviews and focus groups with students and employers; and quantitative information, such as student surveys (see examples of studies using these methodologies in Key Finding 7). These other approaches are used, among other things, to understand whether students are learning and satisfied with the education they are receiving.
FINEEC has also conducted a few direct learning assessments of mathematical competence to measure its evolution over time. However, some concerns have been raised about the representativeness of this data and, by extension, the possibility that results may be biased. For example, while an assessment of mathematical competence and the factors connected to it at the end of upper secondary education in secondary schools and vocational education (FINEEC, 2017[45]) sheds light on the evolution of mathematical competencies developed during VET, the results need to be interpreted with caution as participation in the study was voluntary, and 48% of the sample did not respond. It is highly likely that the results are not representative of the whole population, which was acknowledged in the report, which noted as well that the students who participated were, on average, more motivated and advanced in their mathematical competence than those who did not participate.
Any vocational assessment may involve, first, a set of tasks that candidates are expected to perform and, second, procedures for evaluating these candidates based on those tasks. Assessments are often appraised in terms of reliability and validity:
Validity refers to the capacity of an assessment to accurately measure what it intends to measure. For a vocational assessment, this will mean the capacity to accurately measure the candidate's ability to perform well in the target occupation.
Reliability refers to consistency so that the assessment applies the same standards to different individuals, in different contexts, with different assessors and assessment bodies, and does not change over time.
These two characteristics are different but not independent. A highly valid assessment is necessarily also reliable, since a very inconsistent measure cannot yield accurate predictions of occupational competence. However, a reliable vocational assessment may have low validity.
Source: New Zealand Ministry of Education (2021[46]), Reliability and Validity, https://assessment.tki.org.nz/Using-evidence-for-learning/Working-with-data/Concepts/Reliability-and-validity; Darr, C. (2005[47]), A Hitchhiker’s Guide to Reliability, https://assessment.tki.org.nz/content/download/6109/62609/version/1/file/A+hitchhiker%27s+guide+to+reliability.pdf; OECD (2022[48]), “Assessment and certification in vocational education and training” in Engaging Employers in Vocational Education and Training in Brazil: Learning from International Practices, https://doi.org/10.1787/e26636eb-en.
A number of actions could be taken to improve assessments of learning outcomes in VET. For example, a provisional solution could be to improve the reliability of current assessments by creating scoring guidelines or standards, which can serve as an initial step towards refining evaluation processes. Additionally, FINEEC could include independent actors in the assessment process (Box 3.8 shows how Estonia employs this approach). These external actors are less likely to be biased as they do not (a priori) have a direct interest in obtaining good results and are, therefore, more likely to ensure that assessment standards are applied consistently.
In Estonia, the Education Ministry delegates responsibility for the professional qualifications system to a qualifications authority (Kutsekoda), steered by representatives of employers, unions and government working together. This body organises and co-ordinates the activities of professional councils and keeps the register of professional qualifications.
Professional councils, representing 14 sectors, approve and update professional standards and include representatives of trade unions, employer organisations, professional associations and public authorities. The professional standards set out the content of different occupations and the competencies expected of individuals in those occupations.
Professional councils select awarding bodies (public and private) to organise the assessment of competencies and issue qualifications. The awarding bodies are selected for five years through a public competition organised by the qualifications authority. VET providers may also be given the right to award qualifications if the institution’s curriculum complies with the professional standard and is nationally recognised.
To manage assessments, the awarding body sets up a committee involving sectoral stakeholders: employers, employees, training providers, and representatives of professional associations. Often, it also includes customer representatives and other interested parties. This ensures impartiality in awarding qualifications. The committee approves assessment procedures, including examination materials, decides on awarding qualifications, and resolves complaints. It may set up an assessment committee that evaluates the organisation and assessment results and reports to the qualifications committee. The assessment committee verifies the extent to which the applicant’s competencies meet the requirements of the professional qualification standards. A person’s competencies can be assessed and recognised regardless of whether they have been acquired through formal, non-formal or informal learning.
Source: Adapted from CEDEFOP (2014[49]), VET in Europe: Country Report 2014, www.cedefop.europa.eu/en/publications-andresources/country-reports/estonia-vet-europe-country-report-2014. See also Kutsekoda Estonian Qualifications Authority (2021[50]), Estonian Qualifications Authority, www.kutsekoda.ee/en/.
A better, albeit more costly, option would be to cease using indirect methods of evaluating learning outcomes in favour of employing direct assessments, either through sampling or on a population-wide basis. Standardised assessments provide the confidence that uniform criteria are applied to all candidates. This is particularly effective when measuring the cognitive and knowledge facets of professional competence via written exams. Using uniform criteria, particularly standardised assessments, guarantees consistency and enables the benchmarking of training providers. One possible approach for implementing standardised assessments of learning outcomes in VET would be to build on the foundation of the Kappas! Project, which assesses the foundational skills of bachelor-level students (mentioned in Key Finding 8) by integrating measures of other skills relevant for VET students (theoretical/knowledge dimension of occupational competence) and by extending this assessment to VET education providers as well.
In the productive sector, assessing job proficiency involves directly observing how candidates perform authentic work tasks. This goes beyond standard activities, such as fixing a leaking pipe for a plumber, to include negotiating with clients, diagnosing faults, collaborating with other artisans, managing costs and schedules and handling unforeseen challenges. These “soft” competencies, like teamwork, resilience and conscientiousness, are deemed crucial for workplace success (OECD, 2022[48]). Therefore, to comprehensively assess VET students’ skills, assessments of theoretical occupational competency and generic skills would need to be complemented by assessments of how skills are used in a work setting. Even when assessed tasks are diverse and embedded in work scenarios, it is possible to make assessments that adhere to standardised criteria, for example, by outlining a set of prescribed tasks for all candidates and mandating that candidates perform them under controlled circumstances. This would ensure they consistently cover essential elements in evaluating occupational competency (OECD, 2022[48]).
References
[39] Aila, K., F. Tarja and H. Risto (2022), The Status of Vocational Education and Training Providers’ Quality Management 2022, FINEEC, https://www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_T0722.pdf.
[49] CEDEFOP (2014), VET in Europe: Country Report 2014, https://www.cedefop.europa.eu/en/publications-andresources/country-reports/estonia-vet-europe-country-report-2014.
[19] Chay, K. (2005), “The Central Role of Noise in Evaluating Interventions That Use Test Scores to Rank Schools”, American Economic Review, Vol. 95/4, pp. 1237-1258, https://doi.org/10.1257/0002828054825529.
[26] Clotfelter, C. et al. (2004), “Do school accountability systems make it more difficult for low-performing schools to attract and retain high-quality teachers?”, Journal of Policy Analysis and Management, Vol. 23/2, pp. 251-271, https://doi.org/10.1002/PAM.20003.
[24] Cullen, J. (2006), Tinkering Toward Accolades: School Gaming Under a Performance Accountability System, https://econpapers.repec.org/paper/nbrnberwo/12286.htm.
[47] Darr, C. (2005), A Hitchhiker’s Guide to Reliability, https://assessment.tki.org.nz/content/download/6109/62609/version/1/file/A+hitchhiker%27s+guide+to+reliability.pdf.
[27] Figlio, D. (2004), “Do high grading standards affect student performance?”, Journal of Public Economies, Vol. 88/9-10, pp. 1815-1834, https://econpapers.repec.org/paper/nbrnberwo/12286.htm.
[25] Figlio, D. and L. Getzler (2006), “Accountability, Ability and Disability: Gaming the System?”, https://doi.org/10.1016/S0278-0984(06)14002-X.
[37] FINEEC (2024), Evaluation and support of quality management - Vocational education, https://www.karvi.fi/en/evaluations/vocational-education/evaluation-and-support-quality-management (accessed on 23 January 2024).
[34] FINEEC (2024), Evaluation of the engaging school community work, https://www.karvi.fi/en/evaluations/pre-primary-and-basic-education/thematic-and-system-evaluations/evaluation-engaging-school-community-work (accessed on 22 February 2024).
[11] FINEEC (2024), Plan for Education Evaluation 2020–2023, https://www.karvi.fi/en/about-us/about-our-evaluations/plan-education-evaluation-2020-2023 (accessed on 21 February 2024).
[12] FINEEC (2023), Competence in Mathematics and Mother Tongue and Literature at the Start of Third Grade – A Longitudinal Assessment of Learning Outcomes in Basic Education 2018–2020, https://www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_0123.pdf.
[13] FINEEC (2023), Evaluation of Anti-Bullying Methods: Usability, Sustainability and Outcomes of Seven Methods Selected for the Evaluation, https://www.karvi.fi/en/publications/evaluation-anti-bullying-methods-usability-sustainability-and-outcomes-seven-methods-selected-evaluation.
[42] FINEEC (2023), Professional Competence and Pedagogical Activities in the Basic Qualification in Business, the Vocational Qualification in Managerial Work and the Specialist Vocational Qualification in Management and Business Management, https://www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_1623.pdf.
[43] FINEEC (2023), Quality of Demonstration Activities and the Competence Shown in Vocational Competence Demonstrations, https://www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_0423.pdf.
[2] FINEEC (2023), Statement on amendments to the Act on early Childhood Education and Care, https://www.karvi.fi/fi/ajankohtaista/kannanotot-ja-lausunnot/lausunto-varhaiskasvatuslain-muuttamisesta-vn227742023.
[1] FINEEC (2023), Statement on amendments to the Act on the Promotion of Immigrant Integration and related acts, https://www.karvi.fi/fi/ajankohtaista/kannanotot-ja-lausunnot/lausunto-kotoutumisen-edistamisesta-annetun-lain-muuttamisesta-ja-siihen-liittyvista-laeista.
[3] FINEEC (2023), Statement on amendments to the Government Decree on the National Application Procedure for Post-comprehensive School Education, https://www.karvi.fi/fi/node/85.
[5] FINEEC (2022), Partnering with Working Life – Evaluation of Workplace Education and Training and Working Life Co-operation in Vocational Education and Training, https://www.karvi.fi/sites/default/files/sites/default/files/documents/Kumppanina-tyoelama-Arviointi-tyoelamassa-oppimisesta-ja-tyoelamayhteistyosta-ammatillisessa-koulutuksessa.pdf.
[6] FINEEC (2022), Vocational Competence and Pedagogical Activities in Vocational, Further Vocational and Specialist Vocational Qualifications in Electrical Engineering and Automation Technology, https://www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_1522.pdf.
[4] FINEEC (2021), Equality and Participation in Education – An Overview of National Evaluations, https://www.karvi.fi/sites/default/files/sites/default/files/documents/FINEEC_T1821.pdf.
[9] FINEEC (2020), An Experiment on Free of Charge Early Childhood Education and Care for Five-Year-Olds: The Second Phase of Evaluation. Participation in Early Childhood Education, Its Costs and Organisation, https://www.karvi.fi/en/publications/experiment-free-charge-early-childhood-education-and-care-five-year-olds-second-phase-evaluation-participation-early-childhood-education-its-costs-and.
[15] FINEEC (2019), Dealing with Bilingualism in Swedish-Language Schools. Results from an Evaluation Conducted for Grades 1-6 During the 2017-2018 Academic Year, https://www.karvi.fi/en/publications/dealing-bilingualism-swedish-language-schools-results-evaluation-conducted-grades-1-6-during-2017-2018-academic-year.
[14] FINEEC (2019), On the Way to Better Learning: Good Practices for Integrating Immigrants to Education, https://www.karvi.fi/en/publications/way-better-learning-good-practices-integrating-immigrants-education.
[45] FINEEC (2017), Never Too Late to Learn - Mathematical Proficiency at the End of the Secondary Education 2015 (in Finland), https://www.researchgate.net/publication/315457805_Oppia_Ika_Kaikki_-_Matemaattinen_osaaminen_toisen_asteen_koulutuksen_lopussa_Never_too_late_to_learn_-_Mathematical_proficiency_at_the_end_of_the_secondary_education_2015_in_Finland (accessed on 19 September 2023).
[38] FINEEC (2015), Evaluation of the Quality Management Systems of Vocational Education and Training Providers, https://www.karvi.fi/en/evaluations/vocational-education/evaluation-and-support-quality-management/evaluation-quality-management-systems-vocational-education-and-training-providers-2015.
[30] Hastings, J. et al. (2007), “Preferences, Information, and Parental Choice Behavior in Public School Choice”, https://doi.org/10.3386/W12995.
[18] Helsingin Sanomat (2022), Suburban upper secondary school in Espoo became one of the most successful upper secondary schools in the Helsinki Metropolitan Area – Search engine shows the results of upper secondary schools, https://www.hs.fi/kaupunki/art-2000008837178.html (accessed on 1 February 2024).
[16] Helsingin Uutiset (2023), Will there be a kindergarten plane?“ – Nasima Razmyar reproaches Yle and Hesar, social media asks if the deputy mayor would like to keep public information secret, https://www.helsinginuutiset.fi/paikalliset/5759729 (accessed on 1 February 2024).
[8] IES (n.d.), Institute of Education Sciences, https://ies.ed.gov/.
[20] Kane, T. and D. Staiger (2002), “The Promise and Pitfalls of Using Imprecise School Accountability Measures”, Journal of Economic Perspectives, Vol. 16/4, pp. 91-114, https://doi.org/10.1257/089533002320950993.
[31] Koning, P. and K. van der Wiel (2013), “Ranking the schools: How school-quality information affects school choice in the Netherlands”, Journal of the European Economic Association, Vol. 11, pp. 466-493, https://doi.org/10.1111/jeea.12005.
[50] Kutsekoda Estonian Qualifications Authority (2021), Estonian Qualifications Authority, https://www.kutsekoda.ee/en/.
[10] Ministry of Education and Culture, Finland (2013), “Act on the Finnish Education Evaluation Centre”, https://www.finlex.fi/fi/laki/kaannokset/2013/en20131295.pdf (accessed on 1 February 2024).
[21] Mizala, A., P. Romaguera and M. Urquiola (2007), “Socioeconomic status or noise? Tradeoffs in the generation of school quality information”, Journal of Development Economics, Vol. 84/1, pp. 61-75, https://doi.org/10.1016/J.JDEVECO.2006.09.003.
[33] Muhonen, H. et al. (2018), “Quality of educational dialogue and association with students’ academic performance”, Learning and Instruction, Vol. 55, pp. 67-79, https://doi.org/10.1016/J.LEARNINSTRUC.2017.09.007.
[23] Neal, D. and D. Schanzenbach (2010), “Left Behind by Design: Proficiency Counts and Test-Based Accountability”, Review of Economics and Statistics, Vol. 92/2, pp. 263-283, https://doi.org/10.1162/rest.2010.12318.
[46] New Zealand Ministry of Education (2021), Reliability and Validity, https://assessment.tki.org.nz/Using-evidence-for-learning/Working-with-data/Concepts/Reliability-and-validity.
[29] Nunes, L., A. Balcão Reis and C. Seabra (2015), “The publication of school rankings: A step toward increased accountability?”, Economics of Education Review, Vol. 49, pp. 15-23, https://doi.org/10.1016/j.econedurev.2015.07.008.
[48] OECD (2022), “Assessment and certification in vocational education and training”, in Engaging Employers in Vocational Education and Training in Brazil: Learning from International Practices, OECD Publishing, Paris, https://doi.org/10.1787/e26636eb-en.
[44] OECD (2022), Engaging Employers in Vocational Education and Training in Brazil: Learning from International Practices, OECD Publishing, Paris, https://doi.org/10.1787/e26636eb-en.
[7] OECD (2013), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, https://doi.org/10.1787/9789264190658-en.
[22] Reback, R. (2008), “Demand (and supply) in an inter-district public school choice program”, Economics of Education Review, Vol. 27/4, pp. 402-416, https://doi.org/10.1016/j.econedurev.2008.01.001.
[36] SCCRE (2023), Education Report Switzerland 2023, https://www.skbf-csre.ch/fileadmin/files/pdf/bildungsberichte/2023/BiBer_2023_E.pdf.
[28] Schütz, G. (2007), “School Accountability, Autonomy, Choice, and the Equity of Student Achievement: International Evidence from PISA 2003”, OECD Education Working Papers, No. 14, https://doi.org/10.1787/246374511832.
[41] Statistics Finland (2020), “Employment of recent graduates with initial vocational qualifications weakened by 8 percentage points”, Transition from school to further education and work 2020’, https://stat.fi/til/sijk/2020/sijk_2020_2022-01-20_tie_001_en.html.
[40] Ursin, J. et al. (2021), Assessment of Undergraduate Students’ Generic Skills in Finland: Findings of the Kappas! Project, https://www.researchgate.net/publication/354477740_Assessment_of_undergraduate_students%27_generic_skills_in_Finland_Findings_of_the_Kappas_project.
[32] Wenger-Trayner, B. et al. (2019), “Boundaries and boundary objects: An evaluation framework for mixed methods research”, Journal of Mixed Methods Research, Vol. 13/3, pp. 321-338, https://doi.org/10.1177/1558689817732225.
[35] WWC (2022), WWC Version 5.0. Procedures and Standards Handbook, What Works Clearinghouse, https://ies.ed.gov/ncee/wwc/Handbooks.
[17] Yle (2023), Language & School, https://yle.fi/a/74-20016772 (accessed on 1 February 2024).