Finnish Education Evaluation Centre (FINEEC)

Annex F. Review of a sample of FINEEC’s reports

Guide to review FINEEC’s reports and recommendations

To refine the review of the Finnish Education Evaluation Centre’s (FINEEC’s) reports, a detailed guide was crafted by the OECD team and subsequently executed with the support of artificial intelligence technology. This process involved the use of ChatGTP to systematically scan and analyse the selected FINEEC reports across four dimensions. The analysis was structured around a series of key questions designed to assess the quality of the reports, as shown below.

Evaluation objective

• What research question and/or objective does the evaluation seek to answer (Example: “How does the implementation of X impact student performance?”)?

• What type of evaluation is being conducted (Example: thematic evaluation; system evaluation; learning outcomes assessment; quality management assessment)?

• What is the scope of the evaluation (Example: exploratory [e.g. what students understand by “bullying”]; descriptive [e.g. students’ performance in mathematics by district and region]; observing correlations [e.g. What is the relation between students’ attendance and students’ performance?]; infer causality [e.g. How does VET graduates’ performance in the national learning outcomes assessment impact their employability?])?

Methodology used

• What type of data are collected/used (e.g. quantitative data [e.g. surveys/administrative data/standardised test]; qualitative data [e.g. interviews/focus groups])?

  • If quantitative data analysis is being used, what type of analysis is being conducted (Example: descriptive statistics [mean, median, mode, standard deviation; skewness]; inferential statistics [differences between groups (e.g. T-Tests, ANOVA)] or relationships between variables [e.g. correlation analysis, regression analysis])?

  • If inferential statistics are being used, what type of experiment is being conducted (Example: non-experimental/quasi-experimental/experimental)?

Main evaluation conclusions

• Are the conclusions supported by the results of the analysis of the evaluation?

• Are the limitations of the evaluation transparently highlighted?

Quality of the recommendations

• Are the recommendations specific and practical?

• Are the implications and specific considerations of the recommendations identified for each stakeholder group (e.g. schools, education providers, municipalities, education agency, Ministry of Education and Culture)?

• Are the recommendations based on evidence, either from the evaluation’s results or existing literature?

• Are the recommendations based on evidence from causal analysis?

Table A F.1. FINEEC’s reports reviewed

Level of education	Report title	Year of publication	Type of evaluation	Link
Early childhood education and care	Summary of The Evaluation of the Free Early Childhood Education and Care Trial for Five-Year-Olds in 2018–2021	2021	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_T1221a.pdf
Pre-basic and compulsory	Taking Action to Make Gender Equality Reality – Causes and Backgrounds of the Differences in Learning Outcomes Between and Within Genders in Basic Education	2021	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_1921.pdf
	Evaluation of Anti-Bullying Methods – Usability, Rootability and Effectiveness of the Seven Selected Methods	2023	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_1123.pdf
	English During the Coronavirus Pandemic – Proficiency in A-English at the end of 9th Grade in Spring 2021	2021	Assessment of learning outcomes	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_2222.pdf
Upper general	Options, Choices and New Beginnings – Evaluation of the Study Paths and Counselling of Young People in the Transition Phase Between Basic and Upper Secondary Education	2020	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_0620.pdf
Vocational education and training	Equality and Participation in Education – An Overview of National Evaluations	2021	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/FINEEC_T1821.pdf
	Partnering with Working Life – Evaluation of Workplace Education and Training and Working Life Co-operation in Vocational Education and Training	2022	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/Kumppanina-tyoelama-Arviointi-tyoelamassa-oppimisesta-ja-tyoelamayhteistyosta-ammatillisessa-koulutuksessa.pdf
	Key Findings of FINEEC’s Evaluations of Vocational Education and Training	2023	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/Karvin-esitys-EN-002.pdf
	Special Needs Support as a Resource – Evaluation of the Provision of Special Needs Support in Vocational Education and Training	2021	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_1721.pdf
	Quality of Demonstration Activities and the Competence Shown in Vocational Competence Demonstrations	2023	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_0423.pdf
	Vocational Competence and Pedagogical Activities in Vocational, Further Vocational and Specialist Vocational Qualifications in Electrical Engineering and Automation Technology	2022	Assessment of learning outcomes	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_1522.pdf
	The Status of Vocational Education and Training Providers’ Quality Management 2022	2022	Quality management	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_1222.pdf
Higher education	The Evaluation of Higher Education in Law	2021	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_2221.pdf
	Background Matters: Students with an Immigrant Background in Higher Education	2019	Thematic and system	www.karvi.fi/sites/default/files/sites/default/files/documents/KARVI_2219.pdf

Results of the review of FINEEC’s reports and recommendations

FINEEC’s evaluations are notable for their mixed-method approaches, offering a comprehensive framework that enriches the understanding of complex educational initiatives. These methods are applied in a number of evaluations, such as the Summary of the Evaluation of the Free Early Childhood Education and Care Trial for Five-Year-Olds in 2018–2021, which utilised surveys, administrative data and interviews to assess the programme’s impact. This methodological approach is also evident in the Evaluation of Higher Education in Law (2021) and the Study on Students with an Immigrant Background in Higher Education (2019), where diverse data collection methods provided a broad view of the subjects at hand. Similarly, the Evaluation of the Study Paths and Counselling of Young People in the Transition Phase Between Basic and Upper Secondary Education (2020) and the Evaluation of Anti-Bullying Methods (2023) exemplify how integrating quantitative and qualitative data can yield detailed insights into guidance services and anti-bullying strategies, respectively.

Additionally, the integration of feedback mechanisms across FINEEC reports stands out as a recurrent methodological strength. The incorporation of feedback from stakeholders, as seen in Evaluation of Workplace Education and Training and Working Life Co-operation in Vocational Education and Training (2022), not only captures the viewpoints and perspectives of stakeholders but also enhances the relevance of the findings to both learners and the labour market. Through the use of feedback mechanisms, FINEEC can draw from a rich pool of data, providing a multidimensional perspective on the effects and implementations of educational policies and practices, thus significantly contributing to the credibility and applicability of their evaluations for stakeholders and policy makers.

However, FINEEC’s current approach also presents several challenges and areas for improvement:

• Analytical approach and causality: The evaluations sometimes do not include detailed inferential statistical analysis and are not designed to provide robust establishment of causality, which is crucial for attributing specific outcomes to interventions. Without clear inferential statistical analysis, attributing changes in educational outcomes to specific initiatives is challenging. For example, in the report, Taking Action to Make Gender Equality Reality – Causes and Backgrounds of the Differences in Learning Outcomes Between and Within Genders in Basic Education (2021), the lack of causal statistical methods hampers the ability to confidently attribute changes in gender equality outcomes to the interventions evaluated. Similarly, in the report, Summary of the Evaluation of the Free Early Childhood Education and Care Trial for Five-Year-Olds in 2018–2021 (2021), while increases in participation were noted, the absence of an experimental or quasi‑experimental designs means that these increases cannot conclusively be linked to the policy in question. Similar attribution issues are observed in the Evaluation of the Study Paths and Counselling of Young People in the Transition Phase Between Basic and Upper Secondary Education (2020) and the Evaluation of the Provision of Special Needs Support in Vocational Education and Training (2021).

• Evidence-based recommendations: To be useful for policy making, recommendations need to be grounded in reliable evidence. If the recommendations from evaluations – such as those found in The Status of Vocational Education and Training Providers’ Quality Management (2022) and the Evaluation of Higher Education in Law (2021) – are not directly linked to clear, empirical findings from the evaluation or supported by a broader body of literature, their use for making policy decisions will be limited. It is crucial that recommendations are not only based on the evaluation's findings but also supported by existing research and literature, particularly when the evaluation itself does not lead to concrete recommendations. This triangulation with other literature can validate proposed actions and identify areas for further research, enhancing the recommendations’ relevance and applicability.

• Generalisability and representativeness: A representative sample is critical for establishing the generalisability of findings to a broader population. When an evaluation’s sample is not representative of the wider population, or there is insufficient information about the sample's representativeness, there are significant limitations on what can be inferred from its findings. The Evaluation of Anti-Bullying Methods – Usability, Rootability and Effectiveness of the Seven Selected Methods (2023), the report Equality and Participation in Education – An Overview of National Evaluations (2021), and the Evaluation of the Provision of Special Needs Support in Vocational Education and Training (2021) are examples of evaluation where the lack of information on the representativeness of its findings undermine confidence in the relevance of the findings beyond the sample.

• Interpretation of qualitative data: Qualitative data, which are important sources of evidence for understanding the context and findings in depth, are nonetheless vulnerable to interpretive biases. The potential biases in interpreting qualitative data, such as those outlined in the Evaluation of Higher Education in Law (2021) and the Evaluation of Workplace Education and Training and Working Life Co-operation in Vocational Education and Training (2022), underscore the need for careful consideration and validation of qualitative insights.

• Accuracy of self-reported data and feedback collection: The potential for biases in self-reported data and feedback collection is a recognised challenge of these data sources. The Status of Vocational Education and Training Providers’ Quality Management report (2022) acknowledges such biases, and the Evaluation of Higher Education in Law (2021) notes its possible limitations in capturing competencies and learning outcomes, which may lead to skewed interpretations of the data. Similar concerns are acknowledged in the Evaluation of the Study Paths and Counselling of Young People in the Transition Phase Between Basic and Upper Secondary Education (2020).

• Long-term impact assessment: The evaluations often lack a longitudinal perspective, making it difficult to assess the long-term impacts of educational initiatives. Without this perspective, as shown in the report Equality and Participation in Education – An Overview of National Evaluations (2021) and the report Background Matters - Students with an Immigrant Background in Higher Education (2019), it is challenging to gauge the sustainability and enduring effects of the policies and practices under review.

• Actionability for stakeholders: Recommendations are often broad and lack specificity for individual stakeholder groups. The Key Findings of FINEEC’s Evaluations of Vocational Education and Training report (2023) provides generalised suggestions without detailed guidance for implementation, and the Summary of the Evaluation of the Free Early Childhood Education and Care Trial for Five-Year-Olds in 2018–2021 (2021) report proposes systemic changes without clear actions for municipalities, complicating the translation of these recommendations into practice. This is also evident in the Evaluation of Workplace Education and Training and Working Life Co‑operation in Vocational Education and Training (2022) and the Vocational Competence and Pedagogical Activities in Vocational, Further Vocational and Specialist Vocational Qualifications in Electrical Engineering and Automation Technology (2022) report.