OECD Reviews of Evaluation and Assessment in Education: North Macedonia

Assessment and Recommendations

Introduction

The OECD last reviewed the education policies of the Republic of North Macedonia in 2003 (OECD, 2003[1]). Today, sixteen years on, more children and young people participate in education and remain in school for longer. There have also been major institutional improvements since 2003, with the creation of separate government agencies that now concentrate professional capacity for key evaluation and assessment functions. This includes the National Examinations Centre (NEC), which manages the country’s state matura examination that is recognised across the region for its innovative design and integrity. It also includes the State Education Inspectorate (SEI), which leads and supports regular external and internal school evaluations in North Macedonia. School evaluations focus on many of the aspects of the school environment that research recognises to be essential for learning.

Yet despite these improvements, progress on the most important measure of education system quality – student learning outcomes – remains limited. Data from the OECD Programme for International Student Assessment (PISA) show that learning outcomes in North Macedonia are lower than international and regional averages, and show no sign of improvement. In 2015, half of the country’s 15-year-olds did not demonstrate basic proficiency in all three subjects in which they were assessed – mathematics, reading and science (OECD, 2016[2]). The apparent lack of improvement in student learning outcomes occurs at a time when increasing numbers of students in North Macedonia are progressing to tertiary education, but not subsequently into jobs. This situation points to an urgent need to reinforce the national framework for evaluation and assessment so that student learning is directed and assessed to more rigorous, relevant national standards. It also highlights the need to ensure that struggling students are detected early on and supported to master the essential knowledge and skills that they will need to participate fully in their country’s development as a competitive economy and prospective member of the European Union.

Aware of the need to strengthen evaluation and assessment policies, North Macedonia asked the OECD and UNICEF to undertake a review that would provide recommendations in three key areas. First, on how to develop a national assessment system that would enable the government to monitor national learning outcomes and support instructional improvement, while avoiding the distortions that resulted from past assessment models. Second, was to provide guidance on how to develop further the state  matura examination, especially with a view to better recognising and rewarding the competencies of upper secondary students from vocational programmes. A third priority concern was how to create an effective system for teachers’ professional and career development. These objectives accord with the national aims that are set out in the country’s new education strategy (see Box 1) which was published when this OECD‑UNICEF review began.

Box 1. The Republic of North Macedonia’s goals for education

In 2018, North Macedonia launched the Comprehensive Strategy for Education for 2018‑25. The strategy includes a number of actions relevant to evaluation and assessment and improving education quality, such as:

• Significantly increasing the share of children in pre-school and introducing a compulsory year of pre-primary education (ages 5-6).

• Reforming the curricula and programmes for compulsory education to increase their relevance and attractiveness, better aligning them to children’s stages of development and focus more on learning outcomes.

• Supporting the development and consistent use of quality textbooks while reducing reliance on textbooks for teaching.

• Better orienting vocational education and training (VET) programmes towards the needs of the labour market.

• Strengthening the competence of teaching staff at all educational levels.

• Strengthening management and leadership capacity at central and local government levels, and within schools, and ensuring harmonised and transparent policies.

• Developing a national assessment by 2020 and a new concept for the state matura, in particular for VET students.

Source: (MoES, 2018[3]), Comprehensive Education Strategy for 2018-25, Ministry of Education and Science, Skopje.

Main trends: despite strong participation in education, learning outcomes are not improving

Younger generations show similar levels of educational attainment as their peers in OECD and EU countries

Following a dip in the years after independence, participation in schooling has expanded steadily (Figure 1). This has translated into higher levels of educational attainment among younger generations in North Macedonia, similar to those found in OECD and EU countries. In 2017, while 38.3% of older adults (45-64 years) had left school without upper secondary education, this was the case for just 18% of young adults (25-34 years) (similar to the EU average of 16% of 25-34 year-olds) (Eurostat, n.d.[4]).

Figure 1. Gross enrolment rate by level of education in North Macedonia (2000-15)

Source: (UNESCO-UIS, n.d.[5]) The Republic of North Macedonia, UNESCO-UIS, http://uis.unesco.org/country/MK (accessed on 25 November 2018).

However, while participation in upper secondary has improved, gross enrolment at this level remains more than 10 percentage points lower than other countries in the region, and significantly below the EU average of 119%. Enrolment is lowest among students from a lower socio-economic background and in rural areas. Reasons for not attending school at this level relate to poor learning conditions and families’ and students’ low expectations (World Bank, forthcoming[6]).

Participation in higher education has expanded rapidly in the last two decades

The country’s higher education system has expanded rapidly in the last two decades. In 2017, there were 22 higher education institutions compared to only five in 2003/2004 (UNESCO-IBE, 2011[7]). The expansion of supply is reflected in increased gross enrolment, from 15% in 1991 to over 40% in 2015 (UNESCO-UIS, n.d.[5]). However, rapid growth has not been accompanied by sufficient quality controls. There is little selection into higher education programmes. All students who pass the state matura (which had a pass rate of 94.3% in 2017 among gymnasium students) can attend university and the quotas for government-funded places are very large. The rapid expansion of higher education during a period when North Macedonia’s performance in international assessments has remained very low suggests that many students are entering university with major gaps in their basic knowledge and skills and without the types of higher-order competencies required to advance successfully at this level.

There are also concerns that tertiary programmes do not reflect labour market needs. Recent graduates from higher education in North Macedonia are far less likely to be employed (55.4%) (Eurostat, 2017[8]) than tertiary graduates in EU countries (83%) (OECD, 2017[9]). An explanation for high unemployment among tertiary graduates are low skills, or skills mismatch. One factor contributing to the latter is limited diversity in the provision of higher education programmes, in particular few high quality, technical options in higher education. This leads many vocational upper secondary graduates to pursue academic subjects in higher education.

Learning outcomes are among the lowest in the region and are not improving

In PISA 2015, North Macedonia’s 15-year-olds performed almost four years behind their OECD peers, with an average score of 384 in science compared to 493 in OECD countries (OECD, 2016[10]). The country scored lower than its neighbours, including Albania (427 score points) and Montenegro (411 score points), and only slightly above Kosovo (378 score points) (OECD, 2016[10]).

The country’s performance in international assessments also shows little evidence of improvement. Reading performance in PISA declined by 21 points between 2000 and 2015 (Figure 2). In comparison, neighbouring Albania succeeded in increasing performance by 56 score points, the equivalent of nearly two years of schooling, over the same period.

Figure 2. Mean reading performance in PISA (2000 through 2015)

Note: Albania participated in PISA 2000, 2009, 2012 and 2015. North Macedonia in PISA 2000 and 2015. Montenegro in PISA 2006, 2009, 2012 and 2015. Serbia participated in PISA 2006, 2009 and 2012.

Source: (OECD, 2016[10]), PISA 2015 Results (Volume I): Excellence and Equity in Education, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264266490-en.

More than three out of five 15-year-olds lack basic reading skills

A particular concern in North Macedonia is the high and increasing share of students who are not acquiring basic mathematics or reading skills. The country has one of the highest proportions of students (52.2%) failing to demonstrate basic proficiency (Level 2) in all three domains of science, mathematics and reading among PISA-participating countries. Notably in reading, more than three out of five 15-year-olds lack basic reading skills (70.7%) as measured by PISA. This compares to 20% across OECD countries, 50% in Albania and 42% in Montenegro. While the share of low-performers has fallen over time in most of the region, low performers in North Macedonia increased by nearly 7  percentage points between 2000 and 2015 (Figure 3) (OECD, 2016[10]).

Figure 3. Change in the share of low performers in reading over PISA cycles

Source: (OECD, 2016[10]), PISA 2015 Results (Volume I): Excellence and Equity in Education, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264266490-en.

Neighbouring countries achieve better outcomes with the same level of education spending

At 3.7% of GDP in 2016, public expenditure on education in North Macedonia is lower than the OECD average (4.2% of GDP) and has declined in recent years (World Bank, forthcoming[6]) (OECD, 2018[11]). Between 2011 and 2016, North Macedonia’s public spending on education as a percent of GDP fell from 4.6% to 3.7%. The share of total government expenditure allocated to education also declined, from 13.3% to 11.6% (World Bank, forthcoming[6]).

Comparative analysis suggests that while increased funding will be important to improve education outcomes, there is also scope for North Macedonia to achieve better results with the resources it invests (OECD, 2016[2]). Neighbouring countries have been able to achieve higher participation rates and better learning outcomes with similar or lower levels of expenditure on education (Figure 4). This points to the need for more attention to the effectiveness and efficiency of resource allocation in North Macedonia, in particular greater efforts to optimise the school network, which is characterised by a large number of small schools and high student-teacher ratios.

Figure 4. PISA 2015 results and government expenditure in lower secondary education

Sources: (UNESCO-UIS, 2018[12]), Education: Initial government funding of education per student as a percentage of GDP per capita, UNESCO-UIS, http://data.uis.unesco.org/ (accessed on 25 January 2018); (OECD, 2016[10]), PISA 2015 Results (Volume I): Excellence and Equity in Education, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264266490-en.

School location and language of instruction are strongly related to outcomes

Students’ socio-economic background is less strongly associated with learning outcomes in North Macedonia than in many OECD countries, as measured by PISA, partly because outcomes are pervasively low and because certain ethnic groups – in particular Roma populations - are less likely to be enrolled in school at age 15. However, differences in learning outcomes by school location and ethnic group are significant. North Macedonia has among the largest rural-urban performance gaps of all PISA-participating economies. Fifteen-year-old students in rural areas perform 47 score points behind their peers in urban settings in science (compared to the average difference across OECD countries of 17 score points). This gap is equivalent to nearly 1.5 years of schooling (OECD, 2016[10]).

Participation in education and learning outcomes among ethnic minority groups, especially the Albanian community, are also markedly lower than the ethnic Macedonian community. Over half of Macedonian children attended pre-primary education, compared to less than one in five Albanians. The gap across ethnic groups becomes more pronounced as students advance in the system (USAID, n.d.[13]). While Albanians represent nearly 25% of the total population, they account for only 15.6% of secondary students and only 5.5% of tertiary enrolment (World Bank, forthcoming[6]). In PISA 2015, students who took the examination in Albanian were more than one year behind their peers who took the science examination in Macedonian, even after accounting for their socio-economic background (Figure 5).

Figure 5. Differences in science performance, by language of test, before and after accounting for socio-economic background (PISA 2015)

Source: (OECD, 2016[10]), PISA 2015 Results (Volume I): Excellence and Equity in Education, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264266490-en.

Evaluation and assessment in North Macedonia

This review analyses how policies for assessing student learning, appraising and supporting teachers, evaluating schools and evaluating the performance of the education system overall can be used to improve learning outcomes. The review draws on the OECD’s analysis of policies and practices for evaluation and assessment in over 30 education systems, and the rich evidence this has generated on the factors associated with progress in education quality in different country contexts (OECD, 2013[14]). Such a comparative perspective reveals three interrelated, systemic issues that will be important to address in order to support better learning in North Macedonia.

Setting goals for improving learning outcomes and measuring progress

In North Macedonia, the absence of a national discussion, goals or monitoring related to improvement in learning is striking. At the highest level, while the new Comprehensive Education Strategy is a step forward in many respects – by setting out important actions that have gained general national support - it does not articulate any national targets for improving learning outcomes. At the school level, evaluation is a valued process but encourages schools to focus on procedural compliance, rather than critically reviewing how their learning environment is supporting students to develop their knowledge and skills. None of the schools that the review team visited indicated that they had any goals or targets for improving student-learning outcomes.

At the classroom level, teachers lack the means – either through learning standards focused on outcomes or through assessment resources linked to curriculum expectations – to detect and diagnose students’ learning needs. This leaves the vast majority of the country’s students moving through school without acquiring essential competencies. Interviews conducted by the review team suggest that teachers set objectives for their students in terms of content knowledge to be acquired, rather than of individual learner improvement over time in relation to broader competencies. At present, evaluation and assessment tends to focus on identifying the very high achievers – those who will do well in academic competition, “Olympiads” – and those with learning difficulties. Teachers and schools lack the tools to assess learning more broadly and inclusively.

In a country where three in five 15-year-olds lack basic reading or numeracy skills, setting simple, measurable targets that the public, schools and teachers can understand would provide an essential focus for effort to improve learning outcomes. This review provides recommendations on how learning standards, classroom assessment resources and a national standardised assessment can be developed to help teachers make informed professional judgements about student learning and effective teaching strategies. These tools can also help schools, municipalities and the Ministry of Education and Science (MoES) to set and monitor appropriate, but stretching goals to drive forward improvement in learning. Communicating these goals and progress to the public will help to create national expectations and accountability for meeting national targets.

Strengthening professional competence

North Macedonia has strong technical expertise in its evaluation and assessment agencies – the State Education Inspectorate (SEI), the National Education Centre (NEC) and the Bureau for Development of Education (BDE). However, these institutions are not able to contribute effectively to policy making and implementation because they lack an independent voice and vital resources. Many of the institutions have a number of key functions that are empty – for example, at the time of the review team’s visit, the NEC did not have a director and lacked psychometricians and information technology staff. These institutions also lack sufficient funds – for example, the BDE’s budget is not adequate to provide the sixty hours of professional development for teachers that it is expected to offer. Within central government, the absence of dedicated research staff and limited support for data management and analysis – the country’s Education Management Information System (EMIS), for example, functions with just two members of staff – limits the ministry’s capacity for evidence-based policy making.

This review recommends how the capacity of agencies responsible for evaluation and assessment should be strengthened, as the means to build professional competence and independence. The key institutions in central government need leaders who can represent them at a political level with a strong technical voice. They also need adequate resources, so that they can fulfil their core functions effectively, and have some financial space to identify and implement improvements to their work. For example, the NEC’s matura datasets could be mined to better understand the kinds of questions or topics that students find most difficult, so that teachers can be helped to teach these topics more effectively in the future.

Supporting and resourcing schools to lead improvement

On paper, schools in North Macedonia have a wide range of support that they can draw upon. They receive regular external evaluations and undertake their own self-evaluations. Municipalities are also located close to schools and directly finance them. Each school has its own multi-professional support team that includes a pedagogical expert (“pedagogue”), a psychologist and a special educational needs (SEN) advisor. This degree of support for individual schools is uncommon in most OECD countries.

However, most schools operate in a very difficult context. Historic underfunding and a lack of transparency in funding allocations means that many schools do not have adequate resources to cover their basic running costs, and certainly not to invest in improvements in the instructional environment. The role of school leaders is restricted by external local political pressures, which limits their ability to effectively lead school improvement.

Raising educational outcomes in North Macedonia will depend significantly on strengthening schools’ capacity to design and lead instructional improvement. This review recommends how schools can be better supported centrally, through the school evaluation framework and data, to critically reflect on their performance and set their own objectives for teaching and learning. It also suggest how schools can become supportive communities for teachers’ professional development, by reinforcing existing promising practices like the teacher groups (the “Teacher Actives”), regular appraisal and the multi‑professional support teams. Teacher development will also be encouraged by the recognition and incentives provided by a new performance-based teacher career path.

Box 2. OECD Reviews on Evaluation and Assessment in Education (framework)

The OECD Reviews on Evaluation and Assessment in Education show how the components of evaluation and assessment – student assessment, teacher appraisal, school evaluation, school leader appraisal and system evaluation – can be developed in synergy to enhance student achievement in primary and secondary (Figure 6).

Figure 6. Interactions within the evaluation and assessment framework

This work has highlighted three hallmarks of a strong evaluation and assessment framework:

• Setting clear standards for what is expected nationally of students, teachers, schools and the system overall. Countries that achieve high levels of quality and equity set ambitious goals for all, but are also responsive to different needs and contexts.

• Collecting data and information on current learning and education performance. This is important for accountability – so that objectives are followed through – but also for improvement, so that students, teachers, schools and policy makers receive the feedback they need to reflect critically on their own progress, and remain engaged and motivated to succeed.

• Achieving coherence across the evaluation and assessment system. This means, for example, that school evaluation values the types of teaching and assessment practices that effectively support student learning, and that teachers are appraised on the basis of the knowledge and skills that promote national education goals. This is critical to ensure that the whole education system is working in the same direction, and that resources are used effectively.

Source: (OECD, 2013[14]), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264190658-en.

Raising learning outcomes through student assessment

In North Macedonia, using assessment to support learning is difficult because teachers’ assessment judgements are not based on established, national learning standards, and therefore do not convey reliable information on student achievement. While PISA results reveal that the majority of the country’s 15-year-olds are unable to perform basic cognitive tasks, the same students receive top marks in the classroom.

Classroom assessment practices are also predominantly summative and limited to a narrow range of lower-order tasks. Despite recent policy efforts to strengthen formative practices, students receive little quality feedback to help them understand how to advance in their learning. They also have few opportunities to demonstrate the more applied skills and complex transversal competencies that are part of the country’s curriculum.

Finally, while the state matura is one of the strengths of North Macedonia’s assessment system – its administrative procedures are sound and the results are trusted – the examination needs to evolve to keep pace with changes in the education system. In particular, this review looks at how the state matura might be adapted to support the government’s goal of improving the quality and attractiveness of VET pathways. It also suggests revisions to the matura’s design in order to promote learning across a wider range of subjects in general education, better discriminate across different levels of student achievement, and enhance the reliability of the results.

Policy issue 2.1 Making the results of student assessment more meaningful

In North Macedonia, inconsistent learning expectations across grades and subjects result in unreliable student marks. These inconsistencies are a consequence of limited support for teachers’ assessment literacy, and lack of coherence in the national curriculum, which combines resources, developed at different times according to different educational principles. For example mathematics curriculum for the grade 9, based on the Cambridge curriculum, focuses on competencies like posing research questions using statistical methods, while the mathematics curriculum for the 1st year of secondary (grade 10), not based on the Cambridge curriculum, gives far greater weight to performing discrete tasks such as calculating a mean.

At the same time, teachers’ limited training in assessment design means that they tend to revert to simple knowledge-recall tests with which they are most familiar, rather than assessing more complex interactions of knowledge and skills or higher-order abilities. The challenges around teachers’ assessment literacy are exacerbated by the country’s narrow marking scale of 1-5, which combined with strong societal expectations for high marks, results in classroom assessment marks being clustered around four and five. Student marks therefore contain little meaning with respect to what students can do, and do not effectively help students to understand their strengths and weaknesses.

Box 3. Recommended actions for reporting student results

2.1.1 Develop coherent national learning standards that set out what students should know and how they are expected to apply knowledge to promote more valid, reliable assessments. To achieve the latter, the country will needs to review and align national learning standards across different grades so that student learning is scaffolded towards increasingly complex, higher-order competencies. Particular priority should be given to standards in core learning areas, like mathematics and reading and writing, especially because the latter currently do not have standards in grades 1-3.

The development of leaning standards should be accompanied by the introduction of performance levels that set out student achievement against national standards, e.g. above, meeting or below national learning expectations. This is especially important in grades 1‑3, where there is no standardised description of student achievement at present.

2.1.2 Align student assessment with national learning standards by providing teachers with supports such as clear explanations of the criteria underlying different learning standards and their performance levels, rubrics for assessing students, marked examples of student work and examples of assessments to evaluate students. These materials can be provided via an online platform so that they reach more teachers, can be easily updated and facilitate teachers’ own contributions to online content. Once a new national assessment is developed (see Recommendations 5.2.1 and 5.2.2), teachers should be encouraged to use its items as inspiration for their own assessments and compare their students’ work with results on the national assessment to promote more accurate and reliable classroom assessment.

2.1.3 Enhance the accuracy and educational value of marking and reporting by extending the marking scale of classroom assessment. The scale might be extended to 1‑10, reflecting similar practices in the region. The new marking scale should be linked to the new national learning and performance standards (see Recommendation 2.1.1). The BDE can help teachers to use the new marking scale by creating moderation opportunities, like helping teachers to mark each other’s assessments and discussing in groups how to give marks.

The country might also consider introducing a project assignment at the end of lower secondary to inform students’ choice of upper secondary programme, motivate all students to apply themselves and reinforce more rigorous standards, especially in core subjects.

Policy issue 2.2 Focusing assessment practices on helping students learn

In North Macedonia, the intensive focus on summative marks and the predominant perception of assessment as a judgement of achievement obscures the other important function of assessment ‒ providing information to improve learning. This creates a situation where teachers are not making sufficient use of assessment results to help students understand their current proficiency and determine the next steps in their learning. This leads to many students moving from grade to grade without meeting expectations for their level.

Providing and recording easy-to-understand feedback is also a critical aspect of assessment for learning. In North Macedonia however, almost one-third of secondary school teachers surveyed for this review reported that they either “never or almost never” or just occasionally provided written feedback to students (in addition to a grade) (see Box 2). At the same time, while teachers have frequent contact with parents, the main tool to report student progress – the student report card – does not provide much descriptive information about student learning, especially after grade 6 when providing this information is no longer mandatory.

While the MoES has made some efforts to encourage teachers to use assessment more formatively, embedding the practice in classrooms is challenging. The experience of OECD countries shows that it requires significant and consistent support for teachers, such as resources related to formative assessment, professional development and incentives that encourage its use (OECD, 2013[14]). In North Macedonia, developing these kinds of support will need to be complemented by addressing systemic barriers that make it difficult for teachers to use assessment results to adapt instruction to students’ needs and interests. One barrier is the country’s dense and rigid curriculum. Curricula rigidity is reinforced by external school inspections (“integral evaluation”), which evaluate how closely schools adhere to the curriculum and, in turn, discourage schools to adapt the curriculum to their specific context.

Box 4. Recommendations for focusing assessment on student learning

2.2.1 Promote the use of diagnostic assessments, especially in early grades, to help teachers better understand how far their students are meeting national expectations and what skills and knowledge they still need to develop. Teachers could be required to undertake diagnostic assessments at the beginning of grades 1-3 and on an ad hoc basis as relevant using instruments based on the Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA) that have recently been adapted to the North Macedonian context. As teachers become more comfortable with diagnostic assessments, they should be encouraged to develop their own assessments, based on national learning standards. Teachers will also need guidance on how to use the results from the diagnostic assessments to identify student progress and tailor subsequent instruction.

2.2.2 Provide and record high quality feedback to help students and parents understand a student’s learning needs. The student report card should be updated to provide more space for descriptive feedback that explains why a student received a specific mark. This will help students and parents understand the next steps to improve learning. The country should also ensure that this more descriptive feedback is systematically recorded and shared, for example in the country’s Education Management Information System (EMIS), so that parents, students and other teachers can access feedback from previous teachers. This continuous documentation would help teachers to better understand student needs.

2.2.3 Remove barriers to providing formative assessment by systematically ensuring that all schools can allocate a certain amount of learning time as they wish. This would provide teachers with greater flexibility to use teaching time to respond to the learning needs that assessment results highlight. Greater curricula flexibility should be matched by changes to the school evaluation framework to focus on broader measures like school‑wide achievement of national learning standards, rather than detailed implementation of the curriculum.

To take advantage of greater autonomy, teachers will need more support to implement formative assessment. The BDE might support the country’s school-based teacher groups ‒ the “Teacher Actives” ‒ to focus on practical assessment issues, like questioning and feedback techniques and how to use the new diagnostic assessments.

Policy issue 2.3 Updating the state matura to encourage and assess better student learning in key areas

There are a number of ways in which the state matura could be revised to better recognise and encourage student learning. One issue is the range of assessed subjects. In contrast to national examinations in many OECD countries, mathematics is not compulsory in North Macedonia (OECD, 2015[15]). This results in very few upper secondary school students taking mathematics (roughly 13% in 2017). Internationally, mathematics, alongside reading and writing, is considered to be a core competence that students should acquire at school and an area where information on student achievement is important to inform university selection. A related issue is that while students in North Macedonia study many subjects at school (15) which is higher than in many OECD countries (Ofqual, 2012[16]), the matura only assesses four. The mismatch between timetabled subjects and those that are externally examined leaves students with little recognition and no certification for two‑thirds of the subjects that they study.

Another issue is the reliability and comparability of student results. An elective subject is marked at the school-level by markers from the same school, who develop the test themselves. Although schools receive guidance about how to develop and mark student tests, ultimately this method of marking risks that results are not reliable and comparable across different schools. There is a similar concern in relation to the project assignment. While the project was introduced in order to include an authentic assessment component within the matura, its educational value is reduced because there is little consistency in how projects are conceptualised across schools.

Third, the matura is currently not very effective at discriminating different levels of student achievement. This is partly because the 1-5 marking scale makes available only a limited range of marks. For example, in the 2017 English examination, a student in the 89th percentile received the same mark (4) as a student in the 65th percentile. It also reflects concerns with the tests themselves, which in some subjects do not appear to include items that are sufficiently discriminating. A number of subjects have unusually high student results, which prevents an accurate identification of the top-performing students. In the matura, a small number of students take certain subjects like mathematics and biology, and these tend to be the students who anticipate that they are likely to do very well. For example, around 60% of students got at least 75% of questions correct on their biology test, which was the case of only 7.6% of student who took the English test. Currently the NEC does not produce systematically the type of item-level analysis that would help item developers understand if item difficulty is appropriate for the examination’s target audience.

A final issue is the alignment of the matura with North Macedonia’s aim to improve VET. At present, students enrolled in four-year VET programmes (around 50% of the upper secondary cohort) are eligible to take the matura, giving them the choice of pursuing tertiary education or entering the labour market directly. However, the current matura’s design, where VET subjects are assessed at the school-level, means that students in the four-year VET do not graduate with any externally validated certification in their vocational field. This means that vocational students lack meaningful recognition of their vocational skills and that vocational results do not carry the same external authority as those in general academic subjects. As part of a wider reform to improve the quality and prestige of VET, the country is now considering the development of a distinct “VET matura”. However, this review suggests that national goals for VET could best be achieved by making the existing matura’s model more flexible and allowing it to certify achievements in VET fields.

Box 5. Recommendations for revising the state matura

2.3.1 Revise the matura’s design to provide more meaningful results in key subjects, by:

• Making mathematics a compulsory subject to motivate all students to master at least basic mathematics and help universities make a more informed decision regarding student selection into mathematics and other related courses.

• Creating two versions of the mathematics exam, at basic and higher levels, to provide mathematics certification that is useful and accessible for all students, while providing those students who wish to pursue mathematics at a higher level with the option to study more advanced concepts.

• Considering extending further the core subjects that are assessed to ensure a better match between the breadth of subjects studied and those that are assessed.

• Marking all subjects externally to increase the value of the subjects previously taken internally, and improve the overall reliability of the matura results.

• Standardising the project assignment, e.g. by limiting the topics and the format and providing online examples of acceptable project assignments and guidelines for school staff on how to oversee and assess projects. Regular external moderation, from the NEC or the BDE, could also be conducted for quality assurance.

2.3.2 Adapt marking and improve item quality to provide greater discrimination of student ability and motivate students to improve their learning. The NEC should analyse items following each administration of the matura to learn how students engaged with the test items. The analysis can inform future item development so that there are not too many items that are too easy, too difficult or have poor discriminating ability. Undertaking these procedures will help to improve item quality and normalise the distribution of the student marks. The country should also consider extending the marking scheme, in line with changes to the marking scheme for classroom assessment, to 1-10 (Recommendation 2.1.3), to provide greater scope to discriminate between different levels of achievement.

2.3.3 Strengthen the VET component by externally validating student achievement in the VET subject and linking the results to employer-recognised certification. The externally validated VET subject should provide students with a formal VET certification, integrated in North Macedonia’s national qualification framework, to signal readiness to employers and technical tertiary faculties. VET students should also be required to complete their project assignment for the matura in their chosen vocational subjects to provide greater recognition and time for the development of vocational skills.

To make VET certification more feasible, the current 150+ different specialisations should be reduced to a small number of subjects related to sectors that have been identified as important by economic and labour assessments. Over time, the current VET specialisations can be consolidated into VET families so that students do not pursue options that are too narrow, limiting their future employment options. The VET Centre should continue to oversee examination procedures to provide quality assurance. Since the Centre does not have the capacity to develop and mark all tests, the design and marking of the assessments might involve a body of employers or professional associations.

Using teacher appraisal to support and incentivise good teaching

High quality teaching is shown to be the most important school-level factor related to student learning outcomes. Effective education systems place a strong emphasis on selecting, training and retaining teachers with the competencies needed to help students succeed (Schleicher, 2016[17]). Appraisal supports such a culture of professionalism by first ensuring that all teachers have the aptitudes to teach, while also helping and incentivising teachers to develop higher levels of expertise and responsibility throughout their careers.

In recent years, North Macedonia has made several attempts to create more robust teacher selection and promotion methods, with the aim of establishing a more learner-centred system. Initiatives include the proposal for a merit-based career structure for teachers, as well as efforts to support teachers’ professional development in core areas, such as classroom assessment techniques. However efforts have not been sustained – the merit‑based career system is still not implemented – and do not amount to a comprehensive policy to support the teaching profession.

Policy issue 3.1 Ensuring that entry into, and progression along the teaching career path is based on professional competence

In 2016, the BDE in collaboration with the United States Agency for International Development (USAID) developed a plan to introduce a performance-based career path for teachers. The new career structure set out different steps linked to increasing levels of teaching competence and a new appraisal system determining how teachers would be promoted. Taking forward the plans to introduce a performance-based career path in North Macedonia would encourage teachers to develop new skills and take on new roles and responsibilities.

At the same time as enhancing the management of talent within the existing teacher workforce, North Macedonia needs to ensure that there is more rigorous selection and preparation for new entrants to the profession so that they can also become strong teachers in the future. High-performing education systems invest significantly in attracting and selecting talented and motivated candidates into teaching, and provide them with adequate training to develop the competencies required to become effective teachers (Schleicher, 2015[18]). However in North Macedonia, entry into teacher initial education is not selective, with almost all applicants to initial teacher education programmes being accepted (Mickovska et al., 2013[19]). This not only means that new entrants may lack key pre-requisites, like core academic knowledge and motivation, but impacts the status of the teaching profession and its attractiveness to high achieving students. It also contributes to a significant oversupply of new teachers, which is an inefficient use of government resources (since all initial teacher education places within the quota are subsidised by the government).

The above situation is exacerbated by the absence of other strong mechanisms to ensure the preparedness of teacher education graduates. North Macedonia lacks specific criteria for the accreditation of teacher education programmes as well as robust requirements for initial teacher licensing and adequate guidance for probation appraisal, and mentoring. Overall, the weak selection and quality assurance mechanisms for entry into teaching exacerbates the risk that new teachers enter the profession without a minimum level of knowledge and skills to be effective in the classroom. Aware of these challenges, the ministry has recently proposed to create an academy that would introduce an additional year of initial teacher education at the end of a candidate’s bachelors, with the purpose of selecting and training teacher candidates. While the intent of the academy is positive, it does not address the deeper problem of lack of selection and heterogeneity in the quality of initial teacher education programmes. Instead of introducing an additional layer of initial preparation it would be more efficient and effective to make initial teacher education more selective and rigorous.

Box 6. Recommendations for entry into, and progression along, the teaching career path

3.1.1 Introduce the planned performance-based teacher career structure. Issues like how teachers seeking a promotion will be appraised and the impact on teacher salaries were not addressed in the 2016 plans, so the first step for the ministry and the BDE will be to clearly define the process for the new external appraisal. This should include developing guidance for evaluators on the kinds of evidence they should collect to determine teachers’ readiness for promotion (e.g. classroom observations, reviewing teachers’ portfolios, and interviews with the candidate). Indicators and descriptors of quality teaching should also be developed to orient evaluators towards what they should focus on when observing teaching practices. The above will need to be accompanied by training for evaluators on how to appraise teachers’ competencies in line with the teaching standards.

The ministry will also need to determine how the new appraisal will impact other aspects of teacher policy, including linking career promotion to a salary increase to reward performance. Teachers will need to be supported to identify and undertake professional development that will help them advance up the new career path. One way to support this is by clearly identifying the teaching competencies targeted by accredited training programmes in the new professional development catalogue (Recommendation 3.3.1). Another is by providing school principals and a professional support team with training on how to orient teachers towards professional development that best meets their needs.

3.1.2 Select the most qualified candidates for teaching and ensure that they receive adequate support during probation. Greater selection of aspirant teachers into teacher education programmes could be achieved by reducing the quotas for government-funded tertiary places and requiring that candidates attain minimum matura marks in core subjects such as mother tongue and mathematics. In the future, universities may also be encouraged to also evaluate a candidate’s motivation and their socio-emotional skills, for example, through interviews. The ministry needs to introduce programme-specific accreditation criteria aligned with the 2016 teaching standards to help ensure that all accredited initial teacher education programmes provide quality theoretical and practical training.

The ministry should also consider more robust mechanisms for initial and full licensing of new teachers. One option is to introduce a national qualification examination at the end of initial teacher education so that all selected teachers meet minimum requirements. This new examination might replace the confirmation examination at the end of the probation period to avoid redundancy. It would also need to be accompanied by a stronger probation appraisal to evaluate classroom practice and other attributes that are hard to assess in an examination. BDE evaluators might become the final decision-maker for probation appraisal, given the high stakes that this decision carries for a teacher’s career. All novice teachers should also receive a mentor who can report on their performance across the year, both as input to their probation appraisal and to provide more formative feedback. Ensuring that all new teachers receive quality mentorship during their probation is important to support novice teachers in developing their pedagogical knowledge and skills, recognising that many have not benefited from a quality practicum.

Policy issue 3.2 Developing a culture of learning and feedback in schools

School-level support for teachers that is sustained and connected to their daily practices is among the most effective type of professional development and learning. In North Macedonia, there are already several in-school practices to help teachers. This includes an informal culture of collaborating, with teachers exchanging materials, discussing students’ learning and working with each other as part of in-school teacher subject groups called “Teachers Actives”. Each school also has a multi-professional support team comprising a pedagogue, a SEN advisor, a psychologist and a principal, who are supposed to help teachers respond to different student learning needs. Principals and pedagogues also lead regular in-school teacher appraisal.

While these practices are positive, they could provide stronger support for teachers. For example, it is unclear if the “Teacher Actives” are effectively in place in all schools, and they lack any external support or funding. Regular appraisal also needs to be more directly focused on the most important aspects of teaching for learning and linked to common teaching standards. At present, school-level evaluators, principals and pedagogues, do not receive the necessary training and technical support to make appraisal a meaningful exercise (Mickovska et al., 2013[19]). A further issue is the current overlap of appraisal processes – by inspectors from the SEI as part of external school evaluations, by BDE advisors and school-level appraisal by school leadership. These processes are not co-ordinated, are largely administrative and show little evidence of improving teaching practices.

Finally, the multi-professional support teams seem to be operating according to a very narrow definition of learning support. In a number of schools that the review team visited, the pedagogues perceived their role as being limited to helping teachers manage “problem” students at high risk of failure, instead of supporting teachers to understand the learning needs of each student and help them design lesson plans that create a more inclusive and effective learning environment.

Box 7. Recommendations for developing learning and feedback in schools

3.2.1 Guide principals and pedagogues to make regular teacher appraisal a more meaningful process. Given the redundancies and overlap created by three different regular appraisal processes, this review recommends phasing out the role of the BDE and SEI in regular appraisal. Instead, regular appraisal would be led solely at the school level by principals and pedagogues. This is in line with international practices and research which highlight the value of in-school appraisers leading regular appraisal as they have a more accurate understanding of a teacher’s performance and can create more open conversations that are conductive to the developmental objectives of regular teacher appraisal (OECD, 2013[14]).

Principals and pedagogues will need support to focus regular appraisal on evaluating teachers against the country’s new teaching standards (when they are introduced), and encouraging development towards higher levels of teaching competence through:

• Guidance on how to observe evidence of the new teacher competencies.

• Suggestions on how to use teacher portfolios more meaningfully, for example, by focusing on teachers’ learning goals in their development plan.

• Introducing teacher self-evaluation to encourage teachers to reflect on their teaching practices and development objectives.

• Establishing a development plan that identifies specific, discrete areas for learning and improvement for the coming year.

The school principals and pedagogues that undertake regular appraisals also need to receive training in appraisal techniques. Practical guidance on how to observe teaching and provide formative feedback should be included in principals’ initial preparation. Training sessions should be developed for this purpose for in-service principals and pedagogies.

3.2.2 Develop the “Teacher Actives” groups by giving them an official role in teacher professional development. For example, the BDE might provide examples of collaborative professional development activities that “Teacher Actives” can undertake like peer classroom observations and organising in-school training. The BDE should also designate and train co-ordinators of the “Teacher Actives” to ensure that “Teacher Actives” are developed across all schools. The “Actives” might be provided with some small discretionary funds to undertake their development activities.

3.2.3 Review the role of the in-school support team to focus on helping teachers to create an effective, inclusive learning environment. One option is to introduce a multi-tier support model to provide different “tiers” of support to meet learners’ different needs, similar to the approach used in Finland (Mitchell, 2014[20]). The support team might meet regularly as a group to review teachers’ learning plans to ensure that teachers have identified the different learning needs of the students in their class (e.g. who is on track to achieve national standards, who needs further support and who needs to be challenged) and put in place differentiated strategies to meet these needs. As well as helping teachers to address specific cases of struggling learners, they would also provide teachers with advice on classroom-wide approaches to improve learning outcomes, such as at the start of the school year to help teachers develop effective plans for the coming year and at the end to discuss strategies that have been more or less effective. These new roles should be reflected in the competency standards for school support staff, as part of implementation of the performance-based career structure (Recommendation 3.1.1). The ministry will also need to make sure that the initial training of school support staff is aligned with these standards and with modern concepts of SEN and inclusive education, and that mandatory training requirements are set for existing professional support teams to help them understand and apply new methods.

Policy issue 3.3 Strengthening external support for teachers’ professional development

While in-school professional development is important to support teachers in adopting more effective practices, there remains an important role for external training, especially in a context such as North Macedonia, where the gaps in teacher knowledge and skills are significant and genuine pedagogical leadership capacity within most schools remains weak. At present, however, there are concerns with both the availability and the quality of external training courses in the country. The take-up of professional development is relatively low compared to OECD and Western Balkan countries, and schools receive very little financial support to organise in-service training for their staff. As access to official training programmes is limited, teachers in North Macedonia often find and pay for training themselves, or turn to informal support, such as the Internet, to access teaching materials. The Internet makes it easier for teachers to collaborate beyond their schools and increases the range of teaching tools that teachers can draw upon (Schleicher, 2016[21]). However, the ministry needs to take a more active role in reviewing Internet content and platforms if this resource is to be leveraged effectively to improve teaching practice. The country’s market-based teacher professional development model also needs to be complemented by stronger mechanisms for quality assurance.

Box 8. Recommendations for supporting teachers’ professional development

3.3.1 Ensure that professional development meets teachers’ needs. In the past, the BDE was responsible for maintaining a catalogue of accredited teacher professional development programmes. This role should be re-established so that teachers receive professional development that meets minimum quality criteria. The accreditation process should check that programmes are targeting teachers’ core competencies as defined in the new teaching standards and aligned with the national priorities for teaching and learning set out in the Education Strategy 2018-25.

The ministry also needs to review both the scale and the way professional development is funded. The BDE requires significantly more resources if it is to provide the established ten hours of free training in national priority areas that all teachers are required to take every three years. In addition, earmarked subsidies should be given directly to schools as discretionary funds for them to use to choose training in line with their own needs and interests.

3.3.2 Develop more digital resources to support continuous professional development. Since more than two thirds of the country’s teachers already rely heavily on student assessment tools and lesson plans from the Internet (OECD and UNICEF, 2018[22]), the BDE could create a national online repository to build on this practice. The BDE could complement teacher-provided materials where there are gaps and ensure that materials meet minimum quality criteria. Material can also be peer reviewed. To encourage teacher collaboration, the repository might include an online forum where teachers can collaborate and solve problems that they face in their teaching practice.

3.3.3 Strengthen the role of the BDE. A broader concern for teaching in North Macedonia is the lack of a comprehensive approach to develop the profession. Recent policies and programmes have not been consistently supported - like the development of teaching standards and a performance-based career path, which remains unimplemented. Strengthening the BDE so that it is formally recognised as the key government body for supporting the teaching profession would help to ensure that teaching is recognised as a political priority. The reformed BDE would be responsible for key areas of teacher policy, formulating policy recommendations and advising the minister.

Aligning school evaluation with its core purposes of accountability and improvement

The purpose of school evaluation is to help schools improve their practices and keep them accountable for the quality of the education that they provide to their students. North Macedonia has a school evaluation framework that covers the key areas that are important for an effective school evaluation system. However, this framework has not been fully implemented or appropriated by stakeholders. Both external and self‑evaluation focus largely on complying with the framework, rather than encouraging a culture of reflection and improvement in schools. Fundamentally, this reflects a disconnect between the aims of the framework – to enhance school quality and school-led improvement - and the perception of evaluation among inspectors and schools as an administrative requirement. This is exacerbated by a useful, yet complicated evaluation framework, which inspectors and schools find difficult to apply, and by the lack of support given to schools on how to use evaluation results to inform improvement efforts.

Policy issue 4.1 Professionalising the State Education Inspectorate

External school evaluation in North Macedonia does not yet fulfil its stated core functions of ensuring school accountability and helping schools improve. The overwhelming perception of evaluation – external and internal – as reported to the review team by the inspectorate, principals and teachers, was as a process to ensure compliance with regulations and the national evaluation framework.

Leadership of the SEI is key for shaping how staff within the SEI and schools understand the role of school evaluation. In most OECD countries, the head of a school inspectorate is expected to combine deep understanding of school improvement, strong leadership skills and integrity. Inspectorate directors hold a senior leadership position within a country’s education system, regularly advising the ministry and the minister on issues of school quality. In contrast, the SEI Director in North Macedonia is not considered as a senior managerial position and does not play an advisory role to the minister. For example, the minimum eligibility requirements for the position of director are similar to those of other inspectors.

The SEI also lacks the technical capacity, integrity and independence to lead school evaluation at the national level. It was reported to the review team that the SEI evaluations are sometimes used for political purposes, for example to justify principal dismissals. While this points to a lack of integrity and independence, there are also few mechanisms to ensure that the SEI is made accountable for the quality of its work. In most OECD countries, school inspectorates are subject to a number of accountability requirements, such as the need to produce an annual report on the quality of their work, which is publicly debated in parliament. Inspectors must follow codes of practice and there are clear and explicit mechanisms for stakeholders, such as principals, teachers, students and parents, to make complaints. While some of these elements are present in North Macedonia, the reported practice of using evaluations to justify principal dismissals suggests that they are very weak at present.

Box 9. Recommendations for professionalising the State Education Inspectorate (SEI)

4.1.1 Guarantee the independence and integrity of the Inspectorate. The appointment requirements for the SEI director should focus on demonstrated competence in school improvement and a strong understanding of how evaluation impacts school quality. Adherence to national codes of conduct should also be enforced, with any violations resulting in dismissal.

Increased professional independence of the SEI will need to be balanced by greater oversight of, and accountability for, its work. All stakeholders should have clear and fair opportunities to redress any grievances. A board of respected national education experts could also be formed to help maintain the SEI’s independence and oversee the quality of its work. The role and prominence of the SEI’s annual report could also be reinforced by focusing explicitly on the quality of the organisation’s work (and not merely reporting activities undertaken), and debating its contents in parliament.

Accompanying the above measures with a national consultation to determine a shared vision of “a good school” in North Macedonia would help in developing greater national understanding and ownership of the role of evaluation in supporting school improvement and better student outcomes.

4.1.2 Build the professional capacity of the SEI. New inspectors currently receive only three days of initial training, which is inadequate to develop the skills needed to evaluate schools in a way that is consistent and valid. As an immediate priority, the SEI should design and deliver a set of training courses for existing inspectors with a focus on explaining the purpose of school evaluation and developing key evaluation competencies, such as how to conduct a classroom observation and report back to schools. The training should provide inspectors with practical opportunities to try out new techniques and receive feedback, and to participate in an evaluation visit. In the medium term, the ministry will need to revise inspector’s initial training to bring it more into line with the duration, structure and depth of well-established programmes in OECD and EU countries.

To expand the breadth and depth of the SEI’s experience and expertise, it should consider training and licensing experts as external consultants that can join the evaluation teams on an ad hoc basis. Possible external consultants might include experienced teachers from other schools and advisors from the BDE and the Vocational Education and Training Centre (VETC).

Policy issue 4.2 Ensuring that integral school evaluations focus centrally on improving school quality

While North Macedonia’s school evaluation framework includes many of the aspects that are important for creating an effective and supportive school environment, with 7 areas, 28 indicators and 99 parameters, it can be difficult for inspectors to implement and puts schools under a lot of pressure to compile and report data. As many countries have studied the effectiveness of their school evaluation practices, they have found it important to simplify their frameworks to focus on key aspects of school quality. This is important to move evaluation from a checkbox exercise, to a more focused, in‑depth review of the quality of school practices and how they can be improved.

A distinct aspect of school evaluation in North Macedonia is that inspectors are expected to appraise all teachers in the school individually. Since inspectors have limited time, individual classroom observations are often very short, just ten minutes, during which inspectors will simply check documents such as students’ portfolios and lesson plans. Teachers do not receive written feedback or their results from the classroom observation.

If school evaluations are to lead to improvement, they need to provide schools with information that helps them to understand what they do well, and where improvements can be made. At present, however, the review team’s interviews revealed that schools largely perceive evaluation to be an externally imposed process that is disassociated from their own planning and development efforts. An important way to ensure that evaluations trigger school action is providing useful, actionable feedback at the end of an evaluation, complemented by greater follow-up support where necessary.

Box 10. Recommendations for focusing integral school evaluation on school quality

4.2.1 Revise school integral evaluation to focus more centrally on the quality of teaching and learning. The indicators in the School Performance Quality Indicators (SPQI) framework that guide evaluation should be reduced to around 10 to 15 and revised to distinguish between a set of core indicators to be evaluated in every cycle and secondary indicators to be evaluated on a rotating/discretionary basis. This will make the framework more manageable for inspection teams and give them more time to focus on key indicators of teaching and learning. There are also gaps in the existing framework that need to be addressed. For example, indicators on school pedagogical leadership, the quality of self-evaluation and schools’ capacity to reflect on its practices should be included as part of the core indicators evaluated by the SEI.

In order to create more time to meaningfully evaluate teaching and learning, the individual teacher appraisals that are currently part of the integral evaluation should be replaced with more extended classroom observations of a sample of classrooms to gain a deeper understanding of instruction in the school. Individual teacher appraisals should instead be led by the school principal and the BDE (Recommendation 3.2.1). To reduce the administrative burden that integral evaluations place on schools, the SEI should simplify and digitalise the collection of administrative data.

4.2.2 Make sure that integral evaluations deliver constructive feedback to schools. The SEI needs to make sure that recommendations in the school evaluation reports are clear, specific and actionable. The SEI can start by reviewing a sample of national reports and interviewing schools to understand how feedback is used. It should also look at international practices on reporting. Low-performing schools will likely require additional assistance to act on feedback, and the ministry should explore ways to concentrate external support on where it can make the most difference. Towards this end, the country should consider gradually introducing a risk-based approach to school evaluation and follow-up that prioritises schools at risk on core indicators. As part of this new model, the follow-up visits by inspectors to all schools might be replaced by more sustained support for select schools that is led by the regional or municipal level. North Macedonia will need to consider the best structure to provide such support – for example, whether to create a body of school improvement officers that work across multiple municipalities or develop a separate improvement unit within the SEI.

Policy issue 4.3 Developing schools’ capacity to carry out meaningful self‑evaluation

While most schools in North Macedonia undertake regular self-evaluations and develop school action plans, few have appropriated these processes as internal tools to improve the quality of their practices. Schools need more practical support and training to undertake meaningful self-evaluation. At present, school actors with a leading role in self-evaluation do not receive any training or guidance to implement an effective self‑evaluation process that is embedded in school planning activities.

At the same time, school principals in North Macedonia are not sufficiently trained to carry out key pedagogical leadership tasks such as leading self-evaluation and planning activities or setting a clear vision for the school. Principals’ initial training does not provide time for preparation in these areas and, in addition, is purely theoretical in nature. Once in-service, there are also limited programmes for principals’ professional development. Moreover, political interference and the high level of turnover among principals make it difficult to build a professional school principal corps that is recognised for their expertise.

Box 11. Recommendations for developing school capacity

4.3.1 Provide support and training for school actors on self-evaluation, by considering the following actions:

• Revising self-evaluation guidance. As a first step, the country might review schools’ experience of self-evaluation to understand what a new comprehensive self-evaluation manual should include.

• Providing more training for school actors on self-evaluation. A mandatory, practical module on self-evaluation could be added to school principals’ initial preparation. Regular training should be offered to school staff involved in self‑evaluation and school boards. Schools that struggle the most with undertaking meaningful self-evaluation might be offered technical assistance from BDE advisors.

• Reviewing and providing feedback on self-evaluation during integral evaluations, by adding an indicator on the quality of self-evaluation practices to the integral evaluation framework.

• Creating an online portal for schools to share their experience with self‑evaluation. The ministry might also recognise schools with exceptional self‑evaluation practices through a title like “learning organisation” or “excellence in using evidence” and share these examples of good practice on the online portal.

4.3.2 Develop school principals’ instructional leadership skills. North Macedonia should consider creating a leadership academy to help professionalise the principal role. This academy would be in charge of providing initial preparation and in-service professional development for principals. As a first step, the academy should co-ordinate the finalisation and introduction of competency standards for principals, which would be used to inform their selection and training, and the evaluation of school leadership during school evaluations.

Protecting the principal appointment process from political interference will be essential to ensure that principals are selected based on their professional competence. Ways to achieve this include providing school boards with clear selection guidelines and ending municipalities’ role in validating principal appointments and renewals. The municipalities’ validation process should be replaced by validation by an external and independent body, such as the SEI. Decisions about principal renewal and dismissal should also be depoliticised by introducing a principal performance appraisal to provide a fair and independent measure of performance.

Policy issue 4.4 Providing schools with greater resources to enhance the quality and impact of school evaluation

Schools in North Macedonia will need additional resources to appropriate evaluation as a tool to drive their own improvement. Better access to their own data will allow schools to analyse and monitor their performance and compare themselves to others. Many countries use standardised assessment and examination results to inform school evaluation. North Macedonia’s SPQI framework includes few indicators on student-learning outcomes, limiting schools’ and the Inspectorate’s capacity to evaluate how far learning outcomes in a given school compare to national or local benchmarks.

Second, schools need predicable, adequate financial resources to introduce quality improvements. While schools in North Macedonia have significant discretionary power over the use of their resources, decades of underfunding and lack of transparency in the distribution of financial resources mean that they have few financial resources to implement improvement plans. Greater transparency and fairness of funding will need to be complemented by increased school funding overall.

Box 12. Recommendations for developing school resources linked to evaluation

4.4.1 Provide schools with indicators and tools to measure their performance, for example, by using results from the state matura as a measure of students’ learning outcomes in the school evaluation framework. The ministry should also make data more accessible to schools so that they can monitor key outcome indicators such as students’ learning outcomes, completion and drop-out rates. One option is to develop a school portal or “view” on EMIS that gives individual schools access to their own data and provides national and regional benchmarks.

North Macedonia’s plans to introduce a national assessment are very positive and will provide essential data for monitoring learning outcomes at the national level (see Policy issue 5.2). However, the intention to use the results for school ranking should be reconsidered. Using assessment results alone to rank schools and reward certain teachers is unfair, as it does not control for the school’s socio-economic profile. Instead, the assessment results can be included in the school evaluation framework to encourage schools to reflect on how they support students’ learning outcomes and school accountability. In the medium term, the ministry might consider developing a school index that contextualises school performance including factors such as financial inputs and socio-economic context.

4.4.2 Provide schools with adequate financial resources to implement their improvement plans. The ministry should consider making it compulsory for municipalities to use funding formulas and provide clear guidelines on the variables to be included (e.g. urban/rural, students’ socio-economic background). The ministry should also consider providing schools with small discretionary grants from central funding for professional development or implementing projects under their improvement plan.

Creating a stronger framework to monitor and evaluate national progress in education

System evaluation is central to education reform. It is important for holding the government and other stakeholders accountable for meeting national education goals. It also provides the information needed to define better policies and make sure that they have their intended impact. In North Macedonia, system evaluation is at a nascent stage of development. Recent years have seen some important steps towards establishing the institutions and instruments that can support system evaluation. However, many basic components are still lacking, and data systems and the processes for feeding information into decision-making are weak. Among the significant gaps are the absence of clear objectives for improving learning outcomes and a national assessment that would support efforts to raise achievement.

Policy issue 5.1 Centralising the use of EMIS and improving its capacity

North Macedonia’s EMIS is not used by the country’s policy makers to its full extent. For example, the sections of the MoES responsible for primary and secondary education directly collect data from schools, rather than retrieving data from EMIS. Providing data to numerous requestors is burdensome for schools and multiple collections risk compromising data quality.

This situation reflects a number of challenges in the current operation of EMIS. First, with only two staff members, the EMIS team lacks capacity and a strong voice within the ministry. EMIS also lacks strong quality control mechanisms such as data validation and auditing procedures to ensure that data are of the highest quality. Another issue is that data are stored across different databases, which are not linked, limiting data analysis. For example, student demographic data are stored in EMIS while matura results are stored in the NEC database. The functionality of EMIS is also currently limited, which means that users are not able to take full advantage of its data.

In addition, EMIS currently plays a limited role in monitoring national goals, which is an important aspect of system evaluation. This reflects the absence of measurable national goals in North Macedonia. While the country’s Comprehensive Education Strategy 2018‑25 lists some national objectives, they are not focused on outcomes, most notably not on improvement in student learning, and are not measurable.

Box 13. Recommendations for improving the use of EMIS

5.1.1 Formalise EMIS as the central source of data. The ministry should consider moving the EMIS unit closer to the ministry’s central leadership to give it greater authority over the collection of school-level data. Staff capacity in the EMIS unit should also be bolstered, by adding more staff and addressing current skill and position gaps including strong leadership, software development and quantitative analysis.

Establishing data definition and collection protocols would also help to clarify to whom schools are required to provide data and ensure that standard data definitions are applied across different schools and the education system overall. The country should also consider using students’ national identifications in all its databases and ensure that all data are digitised to allow for greater interoperability between databases, notably EMIS and the NEC database containing matura results. Finally, introducing regular quality assurance procedures for EMIS data (e.g. visiting a sample of schools to check data collection) would help to verify data accuracy and encourage more individuals to use the system.

5.1.2 Enhance the functionality of EMIS by introducing regular reporting procedures to help EMIS users make greater use of the system’s data. For example, regular reports of the most commonly used data (e.g. on participation and completion) could be made publicly available so that users can automatically retrieve data. Developing a user‑friendly public data portal would enable users such as schools, researchers and national policy makers to browse national education data and select schools and municipalities for comparison (e.g. by location or language of instruction).

5.1.3 Improve the articulation of national education goals and align future EMIS development with them. To help direct the country towards national priorities, the government will need to establish specific goals for improving student achievement, associated with measurable, achievable targets. Targets could be based on data from international assessments (e.g. reducing the share of low performers in PISA in line with European Union targets), and the national assessment when it is developed. Given the evidence of disparities in learning outcomes, other goals to improve equity might also be included, such as to close the performance difference between urban and rural areas and/or different ethnic groups. New goals and targets will need to be accompanied by the development of a national indicator framework to collect data and monitor progress publicly. The development of an indicator framework would also help to orient the future development of EMIS by easily identifying data gaps.

Policy issue 5.2 Designing a national assessment that supports national learning goals

From 2013 to 2017, North Macedonia administered a national standardised assessment, which aimed to compare teachers’ classroom marks with student results on the assessment. Teachers were supposed to be ranked based upon how closely their marks corresponded to students’ assessment results, with those who were ranked highly receiving a financial bonus. However, this reward system was never implemented and the assessment was abolished, largely on the grounds that it placed too much pressure on teachers and had a negative impact on teachers’ classroom activities.

North Macedonia is now planning to introduce a new national assessment. A well‑designed assessment would provide valuable information to monitor student performance at key stages of their education against national goals. The results can also be used to inform policies and future system planning, and help to improve the quality of teachers’ professional judgement at the classroom level as well (see Recommendations 3.3.1 and 3.3.2). In addition, the ministry had been considering the potential use of the results to rank and reward schools. Such a measure, however, could undermine the formative functions of the assessment and have a negative impact on teacher and school behaviour. Moreover, student results are influenced by a wide range of factors beyond a school’s control like student background, ability and motivation (OECD, 2013[14]). The means that ranking schools on assessment results alone will not provide an accurate measure of school performance. This review therefore recommends that North Macedonia should not use the assessment results for ranking and rewarding schools (see Recommendation 4.4.1).

Box 14. Recommendations for a new national assessment

5.2.1 Determine the purpose of the national assessment and align its design to the purpose. The ministry should first create a steering committee to make key decisions on the assessment’s development and build national support. The steering committee can help to determine which organisation will be responsible for the new assessment. Given the NEC’s experience in administering the matura and international assessments, it is best positioned to take on this responsibility. Next, the committee will need to determine the assessment’s primary function. This review recommends that the latter be focused on providing formative feedback to teachers and schools to help address key challenges in the country such as low learning outcomes and little support for teachers’ assessment capacity (see Policy issue 2.2), in addition to monitoring learning outcomes at the system level.

Once the assessment’s primary purpose has been determined, this should closely influence its design. The following points suggest how the assessment could be designed to best support a primarily formative purpose:

• Combine census (i.e. all students from a population of interest) and sample-based (i.e. a representative sample of students from the population) testing. A census assessment could provide formative information to help teachers adapt instruction to their students’ needs. However, census assessments can easily acquire high stakes, and are expensive and time-consuming to implement. To manage these costs, North Macedonia might implement a hybrid model of census assessments in grades 3 and 6 so that support can be directed towards struggling students and schools; and a sample-based assessment in grade 9 to avoid confusing the national assessment with a high school entry examination.

• Test mother tongue language and mathematics since they represent core skills. Additional subjects e.g. science and/or national history could be added in grade 9.

• Ensure that items assess learning rather than memorisation by following proper item-writing convention (e.g. reviewing items for potential bias and varying the placement of distractor choices (Anderson and Morgan, 2008[23]). Multiple-choice and closed-format responses can be used in grades 3 and 6, and more open-format questions added in grade 9.

• Consider computer-based delivery as it tends to be cheaper to administer (aside from the initial capital investment), less prone to human error and the results are delivered more quickly.

5.2.2 Pay careful attention to the dissemination and use of national assessment results to enhance their formative value. Different reports can be developed for individual schools and teachers, as well as a national public report. Each report should contain information to help the specific audience use the information to understand current performance and make improvements in the future. For example, reports for teachers can include item-level analysis to help them improve the teaching and assessment of similar content in the future. The national report should disaggregate results by demographic factors (e.g. gender, language of instruction, school type, municipality, student socio-economic status) to inform policy making.

Policy issue 5.3 Institutionalising system evaluation

Education policy making should draw on national information about how the system is currently operating and international research about what factors contribute to effective teaching and learning. In North Macedonia, however, decision-making is not always based on the available evidence, risking that political considerations are prioritised above what is most important for teaching and learning. This can mean that the system’s limited resources are not used as efficiently as they could be – for example, teacher numbers have consistently increased in past decades despite a falling student population (see Policy issue 3.1). It can also result in policies with negative consequences for teaching and learning, such as the intention to use the previous national assessment results to reward or penalise teachers (see Policy issue 5.2).

This situation reflects limited analytical capacity within central government. At the same time, while the agencies around the ministry, such as the BDE, the NEC and the SEI have some research responsibilities, they lack clear mandates, resources, capacity and stable leadership. Another issue is that education information is not well reported or analysed publicly. Unlike most OECD countries, North Macedonia does not have a national education report, which is guided by national policy goals and priorities.

System monitoring and evaluation are also weak at the local level. Following decentralisation, municipalities have taken on responsibilities for school resource allocation and staff recruitment but this has not been accompanied by increased oversight. The experience of decentralisation in other countries shows that in order to produce better outcomes, local governments need a framework to follow and to be held accountable (World Bank, 2006[24]). Municipalities are also not well resourced for managing education delivery. Each municipal government has just one or two members of education staff. It was reported to the review team that the municipalities do not come together to share good practices or experiences.

Box 15. Recommendations for institutionalising system evaluation

5.3.1 Build support for system evaluation through the creation of a policy analysis and research unit within the MoES. This review strongly supports the current ministry initiative to develop its own research unit, which will help to ensure information is used more systematically for policy making. In order for the new unit to guide different parts of the MoES and its associated institutions, it must be prominently situated within the ministry (e.g. by directly reporting to the minister and regularly attending meetings with the ministry’s leadership).

5.3.2 Develop a wider network of research entities that contribute to system evaluation. For the specialised agencies like the BDE, the NEC and the SEI to fulfil their duties, they need clearly defined roles that do not change in response to political considerations. This can be supported by developing mandates for each agency that are enshrined in legislation, agreeing a multi-year activity programme and related budget for each agency, and explicitly setting out the appointment process for directors for each agency to ensure candidates’ technical and professional competencies.

The research function of the individual agencies should also be carefully reviewed. The BDE should be formalised as the research arm of government and provided with the necessary resources and responsibilities, as part of a broader reinforcement of its role in supporting instructional improvement (see Recommendation 3.3.3). Given the extensive information that the NEC and the SEI collect, both should continue to have complementary research functions, with the necessary resources and staff skills that these functions require.

5.3.3 Promote the sharing and use of evaluation results, by annually publishing an analytical, public, education report to help hold the government accountable for educational improvement. The report might include prominent reporting against national goals and targets, accompanied by analysis of progress. These reports can also be used by the wider research community to direct secondary analysis into key issues that affect the education system.

Another measure to ensure that non-partisan evidence review becomes an integral part of the policy making process could be to introduce a government guideline that all major policies and programmes should first be piloted, and the pilot studied, before full-scale implementation. Major programmes should also be systematically evaluated to determine their effectiveness and inform future reforms.

5.3.4 Strengthen local accountability. The government in North Macedonia should consider setting out clearer expectations for how municipalities are expected to perform their role for education delivery (e.g. by setting out the principles that govern school funding or staffing). These principles would also provide the basis for local audit. Given the current opacity of local school funding arrangements and the evidence that funding is currently not efficient, the country should evaluate education resource allocation and use the results to inform the development of a more efficient resource allocation policy (see also Recommendation 4.4.2).

References

[23] Anderson, P. and G. Morgan (2008), “Developing tests and questionnaires for a national assessment of educational achievement”, https://elibrary.worldbank.org/doi/abs/10.1596/978-0-8213-7497-9.

[8] Eurostat (2017), Employment rates of recent graduates (aged 20–34) not in education and training, 2017, https://ec.europa.eu/eurostat/statistics-explained/index.php?title=File:Employment_rates_of_recent_graduates_(aged_20%E2%80%9334)_not_in_education_and_training,_2017_(%C2%B9)_(%25).png (accessed on 31 August 2018).

[4] Eurostat (n.d.), Database - Eurostat, http://ec.europa.eu/eurostat/web/education-and-training/data/database (accessed on 13 July 2018).

[19] Mickovska, G. et al. (2013), Policy and Practice Analysis of the Teacher Professional and Career Development in the Republic of Macedonia, USAID, Skopje, http://www.mcgo.org.mk/wp-content/uploads/2013/12/Policy-and-Practice-Analysis-of-the-Teacher-Professional-and-Career-Development.pdf (accessed on 13 July 2018).

[20] Mitchell, D. (2014), What Really Works in Special and Inclusive Education: Using Evidence-based Teaching Strategies, Routledge, London, https://www.routledgetextbooks.com/textbooks/_author/mitchell-9780415623230/.

[3] MoES (2018), Comprehensive Education Strategy for 2018-2025, Ministry of Education and Science, Skopje.

[11] OECD (2018), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris, https://dx.doi.org/10.1787/eag-2018-en.

[9] OECD (2017), Education at a Glance 2017: OECD Indicators, OECD Publishing, Paris, http://dx.doi.org/10.1787/eag-2017-en.

[10] OECD (2016), PISA 2015 Results (Volume I): Excellence and Equity in Education, PISA, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264266490-en.

[2] OECD (2016), PISA 2015 Results (Volume II): Policies and Practices for Successful Schools, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264267510-en.

[15] OECD (2015), Education at a Glance 2015: OECD Indicators, OECD Publishing, Paris, https://dx.doi.org/10.1787/eag-2015-en.

[14] OECD (2013), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264190658-en.

[1] OECD (2003), Reviews of National Policies for Education: South Eastern Europe 2003 : Volume 1: Albania, Bosnia-Herzegovina, Bulgaria, Croatia, Kosovo, OECD Publishing, Paris.

[22] OECD and UNICEF (2018), OECD-UNICEF survey of teachers and principals in North Macedonia.

[16] Ofqual (2012), International Comparisons in Senior Secondary Assessment: Full Report, Ofqual.

[17] Schleicher, A. (2016), Teaching Excellence through Professional Learning and Policy Reform: Lessons from Around the World, OECD, http://dx.doi.org/10.1787/9789264252059-en (accessed on 30 January 2018).

[21] Schleicher, A. (2016), Teaching Excellence through Professional Learning and Policy Reform: Lessons from Around the World, OECD, http://dx.doi.org/10.1787/9789264252059-en (accessed on 30 January 2018).

[18] Schleicher, A. (2015), Schools for 21st-Century Learners: Strong Leaders, Confident Teachers, Innovative Approaches, International Summit on the Teaching Profession, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264231191-en.

[7] UNESCO-IBE (2011), World Data on Education - The former Yugoslav Republic of Macedonia, http://www.ibe.unesco.org/ (accessed on 9 July 2018).

[12] UNESCO-UIS (2018), Education : Initial government funding of education per student as a percentage of GDP per capita, http://data.uis.unesco.org/ (accessed on 4 July 2018).

[5] UNESCO-UIS (n.d.), The Republic of North Macedonia | UNESCO UIS, http://uis.unesco.org/country/MK (accessed on 28 June 2018).

[13] USAID (n.d.), Main findings and recommendations from the nationwide assessment of reading and numeracy skills in early grade students.

[24] World Bank (2006), Keeping an eye on subnational governments: internal control and audit at local levels, World Bank, Washington D.C., http://web.worldbank.org/archive/website01006/WEB/IMAGES/INTERN-2.PDF.

[6] World Bank (forthcoming), “North Macedonia: Public Finance Review”, World Bank, Washington D.C.