Finnish Education Evaluation Centre (FINEEC)

Annex B. Evaluation matrix

Table A B.1. Evaluation matrix

Dimension	Sub-dimension	Guiding questions	Sources
Relevance - Is FINEEC doing the right things?	Responding to needs, policies and priorities	To what extent are FINEEC’s goals and activities aligned with the needs and priorities of the Ministry of Education and Culture (OKM) and local authorities (e.g. municipalities)?	1. Ministry of Education and Culture (OKM) (Interviews and focus groups) 2. Finnish National Agency for Education (EDUFI) (Interviews) 3. Evaluation Council (Focus groups) 4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups) 6. Desktop research
		To what extent is the National Education Evaluation Plan (NEEP) aligned with Finland’s national education objectives?	6. Desktop research
		To what extent are FINEEC activities aligned with the objectives of the NEEP?	6. Desktop research
		Are there activities carried out by FINEEC that have higher priority than others? If yes, which ones and why?	1. OKM (Interviews and focus groups) 2. EDUFI (Interviews) 4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
		What focus areas and activities not currently in the NEEP could FINEEC develop to better support the OKM and local authorities (e.g. municipalities)?	1. OKM (Interviews and focus groups) 2. EDUFI (Interviews) 3. Evaluation Council (Focus groups) 4. Municipalities (Interviews and focus groups)
	Being sensitive and responsive to context	What changes in Finnish society and the educational system have had the greatest impact on the education evaluation system?	5. FINEEC (Self-assessment, interviews and focus groups) 6. Desktop research
		What changes or adjustments has FINEEC made in recent years to adapt to the changing needs and challenges of the Finnish education system?	5. FINEEC (Self-assessment, interviews and focus groups)
		How do the OKM and municipalities assess the capacity of FINEEC to respond to the changing context, needs and challenges of the Finnish education system?	1. OKM (Interviews and focus groups) 4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
		Finland has a highly decentralised education system. From an educational evaluation perspective, has this decentralisation affected the operation of education evaluation systems or the scope of their work in terms of reflecting the opportunities and challenges associated with a decentralised education system?	1. OKM (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
	Quality of design	Does FINEEC follow or base its work on international standards? If yes, what are these standards, and how are they integrated into FINEEC's work?	5. FINEEC (Self-assessment, interviews and focus groups) 6. Desktop research
		Does FINEEC have sufficient physical, human and financial resources to carry out all the activities in the NEEP 2020‑23? If not, what additional resources are needed?	5. FINEEC (Self-assessment, interviews and focus groups)
		Do you think that the process by which the NEEP is developed involves the right set of stakeholders at the right time, and does it give due consideration to context and expected evolutions?	1. OKM (Interviews and focus groups) 2. EDUFI (Interviews) 3. Evaluation Council (Focus groups) 4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
		To what extent were the views and needs of the OKM and the municipalities incorporated in the development of the NEEP 2020-23?	1. OKM (Interviews and focus groups) 2. EDUFI (Interviews) 4. Municipalities (Interviews and focus groups)
Coherence - How well does FINEEC fit?	External coherence	How does the work of FINEEC align with evaluations and assessments developed by other institutions in Finland? 1) What other educational evaluations and assessments are there in Finland besides FINEEC's? 2) To what extent do these other evaluations and assessments complement FINEEC’s? 3) To what extent do these other evaluations and assessments overlap with FINEEC’s?	1. OKM (Interviews and focus groups) 4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups) 6. Desktop research
		How is FINEEC’s independence safeguarded from third-party interests?	1. OKM (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
		Are there any institutional or governance elements that could be improved to achieve a higher level of independence for FINEEC?	1. OKM (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
		Does the governance of the Finnish evaluation system adhere to international best practices?	6. Desktop research
	Internal coherence	What synergies exist between the different activities carried out by FINEEC? How could these synergies be enhanced?	5. FINEEC (Self-assessment, interviews and focus groups)
		What conflicts (e.g. overlaps, inefficiencies, conflicts of interest) arise between the different activities carried out by FINEEC? How could these conflicts be minimised?	5. FINEEC (Self-assessment, interviews and focus groups)
Effectiveness - Is FINEEC achieving its objectives?	Achievement of the objectives	How is the achievement of FINEEC's objectives evaluated?	1. OKM (Interviews and focus groups) 3. Evaluation Council (Focus groups)
		To what extent has FINEEC succeeded in producing high-quality knowledge and a better understanding of the state of the education system and the main challenges it faces?	1. OKM (Interviews and focus groups) 2. EDUFI (Interviews) 4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
		To what extent has FINEEC succeeded in developing concrete and actionable policy recommendations for the OKM and the municipalities?	1. OKM (Interviews and focus groups) 2. EDUFI (Interviews) 4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
		What are the main areas of improvement to enhance the effectiveness and relevance of the services provided by FINEEC?	1. OKM (Interviews and focus groups) 2. EDUFI (Interviews) 4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews, and focus groups)
		If resources were available to improve just one area or service provided by FINEEC, what should this area or service be and why?	1. Ministry of Education and Culture (Interviews and focus groups) 2. Finnish National Agency for Education (Interviews) 4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
		Have there been any unintended effects (positive or negative) in the education system resulting from FINEEC’s work and outputs?	1. OKM (Interviews and focus groups) 2. EDUFI (Interviews) 4. Municipalities (Interviews and focus groups)
		Does FINEEC provide differentiated support to local authorities (e.g. municipalities) that have more limited capacity to plan and carry out local assessments and evaluations?	4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
	Differential results	Does FINEEC produce outputs and services of similar quality and relevance across levels of education? If not: 1) Please explain these differences and why they arise. 2) Is it your perspective that these differences pose a problem? Why or why not? 3) If these differences pose a problem, what measures has FINEEC taken to address these differences?	1. OKM (Interviews and focus groups) 2. EDUFI (Interviews) 4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
		Does FINEEC produce outputs and services of similar quality and relevance across municipalities? If not: 1) Please explain these differences and why they arise. 2) Do these differences pose a problem? Why or why not? 3) If these differences pose a problem, what measures has FINEEC taken to address these differences?	4. Municipalities (Interviews and focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)
	Influencing factors	What internal factors of FINEEC (e.g. management, human resources, financial, regulation) have had the greatest impact on the achievement or non-achievements of FINEEC's objectives?	5. FINEEC (Self-assessment, interviews and focus groups)
		What external (e.g. historical, social, legal, political) factors have had the greatest impact on the achievement or non-achievement of FINEEC’s objectives?	1. OKM (Interviews and focus groups) 3. Evaluation Council (Focus groups) 5. FINEEC (Self-assessment, interviews and focus groups)