This chapter outlines the objectives and methodology employed to assess the work of the Finnish Education Evaluation Centre (FINEEC). The chapter begins by presenting the objectives of the evaluation, which included assessing the scope, quality and usefulness of FINEEC’s activities. It then presents the evaluation framework and the dimensions evaluated, including the relevance, coherence and effectiveness of FINEEC’s work. The chapter also provides a detailed description of the evaluation methodology, which involves desk research, self-assessments, expert panels, interviews, focus groups and theory of change.
Finnish Education Evaluation Centre (FINEEC)
OECD Centre for Skills Evaluations
OECD Skills Studies
Abstract
The objective of this study is to assess the scope of the Finnish Education Evaluation Centre’s (FINEEC’s) work and its quality, considering whether FINEEC is carrying out the tasks that are most needed and/or desirable; the quality of the analysis is sound; and the results are useful. More specifically, this study:
Assesses the scope of FINEEC’s evaluation activities: The analysis looks at the subjects covered in FINEEC’s evaluation activities and the distribution of its activities over the different parts of the education system to identify gaps and imbalances.
Assesses the quality of FINEEC’s evaluations: This analysis looks at whether the tools and methods used in FINEEC’s evaluations are suitable to fulfil its mandate and in line with international practices.
Assesses the usefulness of FINEEC’s activities from a policy perspective at the system level and for education institutions: This analysis evaluates whether the distribution of FINEEC’s activities is appropriate from a policy perspective. It also assesses whether the outcomes from FINEEC’s evaluation activities form a reliable base for making policy decisions, provide well‑grounded and implementable policy recommendations, and contribute effectively to Finland’s education policy priorities.
This study employs the OECD’s Development Assistance Committee (DAC) Evaluation Criteria (OECD, 2021[1]). These evaluation criteria provide an internationally recognised framework and a structured approach for evaluating FINEEC’s activities. The criteria are widely recognised as playing a central role in improving the quality of global evaluation practice and supporting collaboration. They have enabled organisations to design and deliver evaluations that are relevant to the needs of decision makers and capture a wide range of intended and unintended results, producing valuable evidence and insights. This study concentrates on three evaluation criteria: relevance, coherence and effectiveness These criteria were selected to align with the objectives of the external evaluation of FINEEC mentioned above and after discussion with the Finnish Ministry of Education and Culture (OKM) on the study. Two guiding principles accompany the criteria. First, “the criteria should be applied thoughtfully to support high-quality, useful evaluation”. Second, “use of the criteria depends on the purpose of the evaluation”. In other words, the criteria should be applied thoughtfully and adapted to the context of the intervention and the evaluation.
The selected criteria allow for the examination of whether FINEEC is carrying out the most needed and/or desirable tasks and whether the quality of FINEEC’s analysis is sound and producing results useful for education providers and policy makers. Each criterion, referred to as “dimensions” for the purposes of this evaluation, is composed of two or three sub-dimensions. The dimensions with their corresponding sub‑dimensions are presented in Figure 2.1.
The dimension “Relevance” is evaluated by assessing how well FINEEC’s objectives, strategies and planning align with the needs of beneficiaries, national policies and the priorities of partner institutions. It also allows for an assessment of FINEEC’s ability to adapt to changing policy environments. This dimension has three sub-dimensions. The first sub-dimension, “Responsiveness to needs, policies and priorities”, assesses the extent to which FINEEC addresses stakeholder concerns. The second sub‑dimension, “Sensitivity and responsiveness to context”, analyses how FINEEC’s goals and tasks evolve in response to changing economic, environmental, social, cultural and political circumstances. The third sub-dimension, “Quality of design”, assesses whether FINEEC’s structure and organisation effectively facilitate the pursuit of relevant goals and whether these goals are clearly defined.
Source: Authors’ elaboration based on OECD (2021[1]), Applying Evaluation Criteria Thoughtfully, https://doi.org/10.1787/543e84ed-en.
The dimension “Coherence” is evaluated by assessing the alignment between FINEEC’s objectives and tasks both within the organisation and in relation to other actors in the education system. This dimension includes two sub-dimensions. The first sub-dimension, “External coherence”, assesses the compatibility of FINEEC’s objectives and work with other actors’ interventions in the education sector, nationally and abroad. The second sub-dimension, “Internal coherence”, assesses the interconnectedness and synergies between all interventions undertaken by FINEEC.
The final dimension, “Effectiveness”, is evaluated by assessing the degree to which FINEEC has achieved its intended objectives and outcomes, considering variations in results across services and beneficiary groups. This dimension has three sub-dimensions. The first sub-dimension, “Achievement of objectives”, examines whether FINEEC’s interventions have successfully produced the desired outcomes. The second sub-dimension, “Symmetry of results across levels of education and local authorities”, evaluates the inclusiveness and fairness of outcomes among diverse beneficiary groups. The third sub-dimension, “Factors influencing results”, analyses positive and negative effects originating from FINEEC, the OKM, local authorities, or the broader national context that contribute to the attainment or non-attainment of FINEEC objectives.
The evaluation of FINEEC’s performance across these dimensions makes it possible to reflect more deeply on FINEEC’s strengths and challenges with respect to contributing to the continuous improvement of the education system in Finland.
This evaluation combines different methodologies: desktop research, self-assessments, interviews, focus groups and Theory of Change. It also involved consulting with a wide range of FINEEC’s stakeholders and organising workshops with international experts to learn from their experiences, thereby facilitating reflection on FINEEC’s role and performance.
Once the dimensions to be examined during the evaluation were defined and validated with the technical counterparts (i.e. the OKM), an evaluation matrix was constructed. The evaluation matrix is a tool that allows for the development of a set of guiding questions based on the dimensions and sub-dimensions. The matrix also sets out the methodologies to be employed and information sources to be explored in order to answer these questions. Data collection instruments, including a self-assessment questionnaire and interview and focus group guides, were also drafted.
The information sources to be consulted were identified jointly with the study's technical counterparts and with FINEEC to ensure that the main stakeholders were consulted during the evaluation. An abbreviated version of the matrix can be found in Annex B.
The list of stakeholders consulted during the study is as follows:
Ministry of Education and Culture (OKM)
Minister and Permanent Secretary
Department for General Upper Secondary Education and Vocational Education and Training (VET)
Department for Higher Education and Science Policy
Department for Early Childhood Education, Comprehensive School Education and Liberal Adult Education.
Other educational administration institutions
Finnish National Agency for Education
Evaluation Council
Higher Education Evaluation Committee.
Local authorities and education providers
Association of Finnish Municipalities
Education officials from six municipalities
Association of Finnish Education Employers
The Finnish Association for the Development of Vocational Education and Training
Directors of six VET providers.
FINEEC
Directors and Strategic Advisors
Shared Development Service Unit
General Education Unit
Vocational Education and Early Childhood Education Unit
Higher Education and Liberal Adult Education Unit
Managers for focus areas
Process Managers
Methodology Team.
A self-assessment questionnaire was designed and shared with FINEEC staff. This self-assessment included more than 30 questions related to the dimensions and sub-dimensions selected for analysis in this evaluation. The responses provided by FINEEC staff were supported by documentation to support or further elaborate on aspects of their responses. Responses to the self-assessment questionnaire are available from FINEEC upon request.
Two missions to Finland were undertaken to conduct interviews, run focus groups and collect first-hand insights from FINEEC staff and key stakeholders. The responses received from the initial self-assessment questionnaire by FINEEC staff were instrumental in refining the guidelines for the interviews and focus groups planned for these missions. Each mission, spanning four days each, involved comprehensive discussions with FINEEC’s management and staff and a selection of stakeholders previously identified as critical to the evaluation process. These interactions allowed for a deeper exploration of the self‑assessment questionnaire responses and provided valuable perspectives on FINEEC’s operations and impact.
Two workshops were held with the participation of international experts, OECD experts, FINEEC staff, and stakeholders to facilitate reflection on FINEEC’s activities and performance and the range of possible courses of action that Finland could take. They convened international experts, including specialists from the National Center for Education Statistics of the United States, the Korean Educational Development Institute, and the Estonian Quality Agency for Education. In terms of OECD experts, specialists in Education Policy and Higher Education Policy and Benchmarking participated. All these experts were selected due to their extensive experience in the evaluation of education systems as well as to illuminate the diversity of possible evaluation approaches.
In these workshops, some of the findings of the initial consultation process carried out by the OECD with FINEEC and its stakeholders were presented to highlight some of the challenges FINEEC faces in actively contributing to improving the Finnish education system. For their part, international experts presented information about the organisation and the role of the institutions they represented, as well as the assessment models used in their respective countries. This permitted reflection on points of similarity or difference with regard to the work carried out by FINEEC. These presentations inspired further discussions on the challenges faced by FINEEC and how these could be addressed in the future.
Several internal documents and research literature were consulted throughout the evaluation with three main objectives in mind. The first objective was to identify and choose possible evaluation frameworks and methodologies relevant to a rigorous and high-quality evaluation of FINEEC. The second was to examine documentation and reports produced by FINEEC in order to better understand its mission, organisational structure, activities and outcomes. The final objective was to analyse literature on the current performance of the Finnish education system that would provide context for this evaluation.
A guide was prepared for the review of the reports published by FINEEC. Then, with the help of artificial intelligence, a random selection of FINEEC reports was analysed (see Annex F for details). This analysis covered four key areas: objectives, methodology, conclusions and quality of recommendations. In relation to the objectives, the purpose, type and scope of the evaluation were analysed. In terms of methodology, the types of data and statistical approaches used in the evaluations were analysed, distinguishing between quantitative and qualitative analyses.
Regarding the conclusions, the analysis looked at whether the evaluation findings support these and whether the limitations of the conclusions (and the data supporting them) are addressed in a transparent manner within the reports. Finally, the analysis of the quality of recommendations examined the specificity, feasibility and empirical basis of the recommendations given by FINEEC in its reports and whether recommendations are made at a general level or the level of each stakeholder. Annex F presents the guidance and the results of the analysis of the reports.
It is important to acknowledge the limitations of the methodology used in this study. Although comprehensive research was conducted, including interviews with FINEEC staff, the OKM and stakeholders; an international literature review; a review of reports produced by FINEEC; and discussions with global experts in workshops, these approaches have limitations. First, interviews and stakeholder responses may be subject to personal or institutional biases. Second, the literature review might not encompass all current or emerging perspectives. Third, the workshop discussions rely on the participants’ knowledge and experiences, which might not fully capture the diversity of perspectives across all relevant sectors. Fourth, the evaluation method involved reviewing only a random sample of FINEEC reports.
Reference
[1] OECD (2021), Applying Evaluation Criteria Thoughtfully, OECD Publishing, Paris, https://doi.org/10.1787/543e84ed-en.