OECD Reviews of Evaluation and Assessment in Education: Bosnia and Herzegovina

9. Sarajevo Canton

Context and features

Sarajevo Canton contains the capital of Bosnia and Herzegovina (BiH), Sarajevo City, and has the highest population density and strongest economy of all jurisdictions in the country (Table 9.1). Together, the city and canton of Sarajevo generated nearly 33% of the total GDP of the Federation of Bosnia and Herzegovina (FBiH) in 2017 (Sarajevo Canton, 2019[1]). Sarajevo Canton has important tourism, food processing, manufacturing and information technology sectors, and hosts a number of large international companies. Until recently, Sarajevo Canton had a single ministry with responsibilities for education policy. In spring 2021, however, these responsibilities were split between the new Ministry of Education and Upbringing, which governs pre‑school, school and adult education and a separate Ministry for Higher Education, Science and Youth, which is responsible for higher education, as well as science and youth policy areas (MoES, n.d.[2]; MoE, n.d.[3]). This profile examines the context and features of Sarajevo Canton’s evaluation and assessment system for education and highlights policy recommendations that can help strengthen this system to improve teaching and learning.

Sarajevo Canton previously had a pedagogical institute that was closed during the re-organisation of the ministry. The 16 professional staff who worked in the pedagogical institute (employees who mainly had degrees and experience in teaching), were transferred to work in a unit within the new Ministry of Education and Upbringing and are now considered ministry staff. This unit has responsibilities equivalent to that of a pedagogical institute, namely to monitor and provide support to the canton’s teachers and schools. However, canton officials report that school monitoring activities have not been conducted and implementation of the ministry’s other work has been slowed since the organisational changes were made. Positively, the Sarajevo Cantonal Assembly adopted plans to establish a new Institute for the Development of Pre-tertiary Education (Sarajevo Canton, 2019[4]), which will be responsible – among other things - for monitoring, assessing and improving pre-tertiary education, helping implement curricula reforms and providing expert support to teachers and schools.

Similar to other parts of FBiH, the cantonal budget and associated laws determine the financing of school education in Sarajevo Canton. Cantonal law states that a school principal is responsible for managing primary and secondary schools and a school board is responsible for governing them. In general, schools in Sarajevo Canton have some autonomy over management and governance decisions. For example, a school principal can request permission to hire a new teacher but this request must be approved by the ministry beforehand and any unassigned teachers in the canton have priority to fill the position before the principal can open the vacancy to the public (World Bank, 2021[5]). School principals also have responsibility for preparing school budgets, which must be approved by the school board and ministry; however, there are no guidelines on how to prepare these (ibid).

Sarajevo Canton is one of only two competent education authorities covered by this review that has a standalone strategy for the education sector (alongside Republika Srpska). This strategy covers the period of 2018‑22 and aligns with the general Development Strategy of Sarajevo Canton (2021-27). The canton’s main education goals include raising the quality of education, with a special emphasis on promoting science, technology, education and mathematics (STEM), lifelong learning and digital skills, as well as aligning the education system more closely with labour market demands (Canton Sarajevo, 2020[6]). Positively, the canton’s education strategy was developed in consultation with stakeholders and includes a set of clear targets, which can help organise resources and make policies more coherent. However, the limited use of evidence to underpin the education strategy (e.g. no data on learning outcomes) and the high turnover rates of ministers leaves the Sarajevo education system prone to political interference, making it difficult to establish and implement a sustainable reform agenda.

To help monitor and support the shift to online learning, during the COVID‑19 pandemic, Sarajevo Canton established a working group, as well as teams to support the development of digital content for online learning at the primary and secondary level. These bodies included representatives of the ministry, representatives of parents’ councils, teaching staff and school management. Schools in Sarajevo Canton were required to submit weekly reports on the implementation of online learning during the initial school closures caused by the COVID‑19 pandemic.

Table 9.1. Key facts: Sarajevo Canton and its education system

Economic Indicators Based on most recent available data
	Sarajevo Canton	BiH share	BiH average
Surface area	1 227 km2	2.4%	..
Population	413 593	11.7%	..
GDP (PPP) per capita	..	..	USD 14 509
Schooling indicators
	Pre-primary	Primary (ISCED 1‑2 or age 6‑14)	Secondary (ISCED 3 or age 15‑18)
			General	Specialised
No. of children or students	5 454	38 043	4 673	10 077
No. of schools	68	96	19	47
No. of teachers	396	2 989	1 605

Notes: Population figures are a 2020 estimate by BHAS based on the 2013 BiH population census. GDP (PPP) per capita figures are for 2020. BiH average GDP (PPP) per capita is in constant 2017 international $. For pre-primary level, the number of students and teachers refers to the numbers of children and staff in pre-school institutions. Specialised secondary programmes are vocational, technical or arts and religious programmes: data refers to the number of programmes within independent and mixed schools.

Source: (BiH, 2021[7]), Country Background Report for the OECD Review of Evaluation and Assessment in Education: Bosnia and Herzegovina, Unpublished. (BHAS, 2021[8]), Bosnia and Herzegovina in Figures 2020, https://bhas.gov.ba/data/Publikacije/Bilteni/2021/NUM_00_2020_TB_1_EN.pdf (accessed on 17 January 2022); (World Bank, 2022[9]), World Bank Open Data, https://data.worldbank.org/ (accessed on 17 January 2022). https://www.popis.gov.ba/popis2013/mapa/?lang=eng (for population)

Strengths and challenges

Sarajevo Canton has some important elements of an evaluation and assessment framework, such as a functioning education management information system (EMIS) and an external examination of student learning (in Grade 9). While the canton has started developing new learning outcomes for students, in line with its competence-based curricula reforms, teacher policies are not designed to promote the student-centred approaches that underpin competence-based approaches to education. Similar to other parts of BiH, the implementation challenges facing Sarajevo Canton are exacerbated by capacity and resource challenges that prevent the government from systematically monitoring and supporting the quality of teaching and learning in schools.

Sarajevo Canton is one of the most experienced education authorities in BiH when it comes to using external, standardised assessments of student learning

Among other jurisdictions in BiH, Sarajevo Canton is one of the most experienced in using external standardised assessments of student learning. Canton authorities piloted a low-stakes standardised assessment in 2018 to support system monitoring but this never materialised into a regular assessment, partly because of student absenteeism and lack of reliable marking procedures (BiH, 2021[7]). Despite this, the canton has a well-established external examination at the end of basic schooling (known as the Grade 9 matura) that assesses students in Bosnian, Croatian or Serbian language and literature, mathematics and the students’ first foreign language. While the canton did not administer the matura in 2020 because of the COVID‑19 pandemic, results are typically used as one of the criteria (alongside grade point average) for selecting students into either vocational or general secondary schools.

As the Grade 9 matura has high stakes for students, it influences what students learn and what teachers teach. This can create a “backwash effect” that can be either positive (e.g. by reinforcing curricula and learning outcomes) or negative (e.g. teaching to the test) (OECD, 2013[10]). While there are positive features of Sarajevo Canton’s Grade 9 matura, such as involving subject teachers in developing items, the test instrument consists mainly of multiple-choice questions with some short-answer items and does not include the types of open-ended tasks that can better capture higher-order competences. Moreover, the use of an external exam to sort students into secondary programmes has become less common internationally as policymakers seek to remove barriers to progression and reduce early tracking (Maghnouj, S. et al., 2020[11]). The canton’s lack of external examinations at the end of upper secondary (ISCED 3 or secondary education in BiH) is another notable difference between the way BiH education systems use examinations compared to international peers.

Several good practices that support system-level planning exist but schools lack capacity and support to improve instruction

Sarajevo Canton benefits from its proximity in location to the headquarters of international development partners and universities based in the BiH capital who regularly convene and can conduct research and support other activities to help strengthen the education sector. This context helps explain how the canton has established several good practices that support system-level planning. For example, the canton has a distinct education strategy and the Assembly of Sarajevo Canton aims to discuss reports from the education ministry on a regular basis. Despite some reported limitations (e.g. the inability to collect data on Roma students), the canton has one of the most sophisticated EMIS in the country, which collects and stores school-level data, as well as results from the external examination. Importantly, the ministry reportedly makes adjustments to its EMIS indicator framework when additional information is required. Despite these positive features, Sarajevo Canton struggles to implement education reforms, in part because of resource limitations, high turnover in ministry leadership and more recently, organisational changes to the bodies responsible for education within the canton. For example, since closing its pedagogical institute in 2021, Sarajevo Canton has not been able to carry out school monitoring activities. As school self-evaluation is not mandatory in the canton, the lack of any external or self-monitoring activities presents a major risk to the quality of school education.

Teacher policies are not designed to raise the quality of teaching or motivate teachers to improve their practice

Similar to other BiH jurisdictions, Sarajevo Canton has established a career structure for teachers, as well as systems for recruitment and promotion. However, recruitment procedures prioritise teacher candidates who have been unemployed over those with more teaching experience and promotion decisions are not systematically based on performance (World Bank, 2019[12]). As a result, the canton’s career structure does not motivate teachers to develop their competences, take on new tasks, and demonstrate high performance which could help improve the quality of schooling. This was not always the case: Sarajevo canton previously used a set of country-level teacher standards that were developed through an EU‑funded project and covered domains that research recognises as important for quality teaching, such as learning and teaching; monitoring and assessment; co-operating with family and community; and professional development (among others) (British Council, WYG and GIZ, 2017[13]).

These standards underpinned the canton’s external teacher appraisal processes and helped teachers to identify their own strengths and areas for improvement. However, the canton discontinued this practice, partly because teacher unions determined the standards created too much of an administrative burden. Without clear professional standards and appraisal processes, Sarajevo Canton will likely struggle to align its policies related to recruitment, promotion and teacher education, with the incentives and support needed to encourage the student-centred approaches that teachers should use and competences they should help their students develop.

The establishment and use of learning standards is a challenge

Sarajevo Canton started reforming its curriculum in 2016, with a goal to move beyond a narrow focus of recalling knowledge to an approach that is focused on developing competences. In line with the curriculum reform, the canton is developing subject curricula based on learning outcomes, which will help define the knowledge and competences that students should have achieved by the time they complete key stages in their schooling (BiH, 2021[7]). At present, however, teachers, students and schools do not have clear learning outcomes and assessment criteria to guide the educational process and review student performance. This context contributes to important challenges, such as grade inflation and an implementation gap between the goals of the canton’s competence-based curricula and what is taught and learnt in classrooms.

Policy recommendations

This review provides recommendations that are relevant for Sarajevo Canton as for other competent education authorities in BiH. However, the following points may be particularly salient for this authority:

Share the canton’s experience with standardised assessments while ensuring the reliability and validity of the Grade 9 matura

This review recommends that Sarajevo Canton concentrate on sharing its experience with standardised testing by helping to introduce a high-quality external BiH Matura examination that would help improve the reliability and rigour of upper secondary certifications (ISCED 3 or secondary education in BiH) across the country (see Chapter 2). This initiative would also help signal the mastery of students’ core competences to potential employers and help establish more merit-based and equitable selection into tertiary education. In parallel, however, authorities in Sarajevo Canton should take steps to improve the quality of its existing Grade 9 matura so that it can help make fair, trusted and transparent decisions about students’ pathways after basic schooling. The extent to which current external examinations in BiH jurisdictions fulfil their purposes depends upon the strength of test characteristics, which are themselves affected by several decisions related to the test’s design and procedures (see Table 9.2).

Table 9.2. Purposes, important characteristics and components of external examinations

	Purpose	Important test characteristics	Design and procedural components
Primary purpose(s)	Certify and select students	Integrity Reliability Validity (construct and content) Test subjects Item types Testing mode Testing conditions	Test subjects Item types Testing mode Testing conditions Marking Management and leadership
Secondary purpose(s)	Exert positive backwash effects

Source: Adapted from (OECD, 2020[14]), Strengthening national examinations in Kazakhstan to achieve national goals, https://doi.org/10.1787/0bf8662b-en.

As Sarajevo Canton is undertaking major reforms to align its curricula with more competence-based approaches to education, the matura test instruments need to align with the new curricula and a clear set of learning outcomes. This will require clearly defining the competences that students should achieve by the time they complete key stages in their schooling and developing assessment criteria to guide the educational process. The canton’s external Grade 9 matura also has a role to play in either helping to reinforce the competence-based curricula and desired assessment practices or hold back these efforts by encouraging teachers and students to narrowly focus on recalling the knowledge needed for success on more traditional types of examinations. Specifically, using a combination of multiple-choice items and open-ended items in the Grade 9 matura would help improve the validity of this exam. To ensure the Grade 9 matura and other external assessments are valid instruments that have positive backwash effects, canton authorities can take the following steps:

• Develop high-quality item banks. To develop valid exam instruments, canton authorities should develop item banks that reflect the goals and standards of the CCC that the Agency for Pre-school, Primary and Secondary Education (APOSO) has developed as well as criterion-based assessments. Item developers could draw on examples from PISA and other competence-based assessments to introduce more applied item types into local standardised tests. For instead of asking mathematics questions to assess whether students can recall complex procedures to solve logarithms, questions might ask students to formulate, use and interpret mathematical concepts to solve problems in real life contexts (OECD, 2019[15]). Teachers should be involved in writing these items, which can increase their familiarity with assessing higher-order competences.

• Recruit and train technical staff to manage standardised tests. To support local examinations and assessments, the units responsible for developing and managing the Grade 9 matura should recruit and train the technical staff needed to undertake this task. In particular, they will need individuals with expertise and experience in psychometrics and statistics, who will be able to develop and ensure the quality of competency-based items, as well as analyse results at a more granular level to inform instruction. This could, for instance, be done through co-operation with the University of Sarajevo or as one of the mandates for the canton’s new Institute for Development of Pre-tertiary Education.

Given that students in Grade 9 are still young (generally around 14‑15 years old), and have 3‑4 years of schooling left (depending on their secondary track), it is right to prioritise the validity of the external examinations at this level. However, there are trade-offs associated with the marking of more complex, open-ended test items, as these may require subjective judgements, which in turn makes reliability harder to ensure. Students in Sarajevo Canton take the Grade 9 matura in their schools and results are marked by an external commission of teachers. At present, there are no external moderation procedures to ensure that all students take the exam under the same conditions or that marking is consistent. One way to address this challenge is to conduct random external visits during the administration of the exam to ensure that all schools are following the Grade 9 matura rulebook. Another way is to have qualified staff from the Institute for Development of Pre-tertiary Education (once it is developed) perform spot checks on samples of student work to ensure that the marks granted by the commission of teachers do not deviate from marking guides. Authorities could also move towards computer-based test administration and marking, trends that have accelerated in response to the COVID‑19 pandemic (OECD, 2020[16]). Introducing moderation processes would be an effective way for canton authorities to prioritise the validity of the Grade 9 matura while maintaining its reliability and integrity as a trusted instrument for allocating students into different secondary pathways.

In the medium to longer term, canton authorities should also consider investing their resources in implementing a standardised assessment at earlier levels of schooling instead of as a selection examination. Creating a low-stakes standardised test that does not have consequences for student’s future pathways but primarily serves to support their learning could help to better consolidate the foundational numeracy and literacy skills of younger students in basic schooling while also providing a benchmark against local, regional, country-wide or international standards.

Re-introduce the use of professional standards to inform teacher appraisal processes

Sarajevo Canton should re-introduce the use of teacher standards and mandate the regular, standards-based appraisal of teachers’ work for development purposes. This practice should take place within schools and be conducted by the school leadership team or experienced teachers (see Chapter 3). Many OECD countries require that regular teacher appraisals take place annually to help ensure that teachers receive regular feedback, as well as opportunities and support to improve their practice (OECD, 2013[10]). If these standards are to be considered more than a narrow administrative tool, but rather means to reinforce expectations for teachers’ roles in line with the competence-based curriculum, it will be important that the standards align with the canton’s other teacher policies, such as those related to recruitment, promotion and professional development. Participating in the development of a state-level online platform that collects and disseminates resources for using the standards in teachers’ daily work and creating opportunities for job-embedded learning related to the standards are among some of the ways that Sarajevo Canton can ensure the standards serve as a meaningful and relevant reference for strengthening the quality of teaching and learning. Such a platform could be led by APOSO, in partnership with pedagogical institutes.

Strengthen the capacity of schools to drive their own improvement

Sarajevo Canton does not currently conduct external school evaluations, and self-evaluations are not mandatory. Moreover, principal appointments remain vulnerable to politicisation despite changes to selection procedures over the past decade. The canton’s lack of professional standards for principals, as well as appraisal processes and training opportunities also means that principals are often unprepared to make well-informed judgements about how their school is performing in relation to broader system priorities and how to develop school improvement plans. Other jurisdictions in BiH, including Republika Srpska and West Herzegovina Canton, have developed – or are in the process of developing – professional standards and appraisal processes for school principals. This policy action can help strengthen the capacity of principals in Sarajevo Canton to drive improvements in their schools. Investing in school principals and providing schools with support is especially important since the canton does not currently have a separate pedagogical institute to provide expert advice to schools (although they are in the process of forming the Institute for Development of Pre-tertiary Education).

The Ministry of Education can help promote improvements to school performance by working with APOSO to define a core set of five to ten school quality indicators that principal and other actors can use to better understand a school’s level of performance. These indicators can be tailored to Sarajevo Canton’s specific context but indicators should address aspects of schooling that are most important to students’ learning and development, as well as student outcomes (e.g. outcomes related to student learning and progress, equity, etc.). Sarajevo Canton could also leverage its EMIS to provide summary reports on the state of education that describe what schools in the canton are doing well and what they can do to improve, which could inform school-led improvement efforts. In the longer term, the canton should re-introduce external school evaluations and associated supports to raise school performance.

References

[8] BHAS (2021), Bosnia and Herzegovina in Figures, Agency for Statistics of Bosnia and Herzegovina, Sarajevo, https://bhas.gov.ba/data/Publikacije/Bilteni/2021/NUM_00_2020_TB_1_EN.pdf (accessed on 17 January 2022).

[7] BiH (2021), Country Background Report for the OECD Review of Evaluation and Assessment in Education: Bosnia and Herzegovina, Unpublished.

[13] British Council, WYG and GIZ (2017), Occupational standard teacher in primary and secondary education, European Union, http://www.poljskolabl.rs.ba/wp-content/uploads/2017/04/docs_2%20Standard%20zanimanja_nastavnik.pdf (accessed on 29 June 2021).

[6] Canton Sarajevo (2020), Draft Development Strategy of Sarajevo Canton 2021-27 [Utvrđuje se Nacrt Strategije razvoja Kantona Sarajevo 2021-27], https://skupstina.ks.gov.ba/sites/skupstina.ks.gov.ba/files/nacrt_strategije_razvoja_ks.pdf.

[11] Maghnouj, S. et al. (2020), OECD Reviews of Evaluation and Assessment in Education: Serbia,, OECD Publishing, Paris,, https://doi.org/10.1787/225350d9-en.

[3] MoE (n.d.), Sarajevo Canton Ministry of Education [Ministartvo Za Odgoj I Obrazovanje Kantona Sarajevo], https://mo.ks.gov.ba/pocetna.

[2] MoES (n.d.), Sarajevo Canton Ministry of Science, Higher Education and Youth [Ministarstvo za nauku Visoko Obrazovanje I Mlade], https://mon.ks.gov.ba/.

[16] OECD (2020), “Coronavirus special edition: Back to school”, Trends Shaping Education Spotlights 21, https://doi.org/10.1787/339780fd-en (accessed on 25 January 2021).

[14] OECD (2020), “Strengthening national examinations in Kazakhstan to achieve national goals”, OECD Education Policy Perspectives, No. 24, OECD Publishing, Paris, https://doi.org/10.1787/0bf8662b-en.

[15] OECD (2019), PISA 2018 Assessment and Analytical Framework, OECD Publishing, Paris, https://doi.org/10.1787/b25efab8-en.

[10] OECD (2013), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Publishing, Paris, https://doi.org/10.1787/9789264190658-en.

[1] Sarajevo Canton (2019), Invest in Sarajevo Canton, Sarajevo Canton, https://mp.ks.gov.ba/sites/mp.ks.gov.ba/files/kanton_sarajevo_inv_sazetak_2019_eng.pdf.

[4] Sarajevo Canton (2019), Proposal of the Decision on the Establishment of the Institute for Development of Pre-tertiary Education, Official Gazette of Sarajevo Canton, https://skupstina.ks.gov.ba/sites/skupstina.ks.gov.ba/files/odluka_institut_pred_univ_obrazovanje_0.pdf.

[9] World Bank (2022), World Bank Open Data, https://data.worldbank.org/ (accessed on 17 January 2022).

[5] World Bank (2021), Bosnia and Herzegovina: Functional Review of Education Service Delivery: Phase II - Strengthening Institutions to Create a More Effective Education Workforce, https://documents1.worldbank.org/curated/en/365261617260855095/pdf/BiH-Phase-2-Education-Functional-Review-Report.pdf.

[12] World Bank (2019), Bosnia and Herzegovina Review of Efficiency of Services in Pre-University Education. Phase I: Stocktaking, World Bank, https://documents1.worldbank.org/curated/en/719981571233699712/pdf/Bosnia-and-Herzegovina-Review-of-Efficiency-of-Services-in-Pre-University-Education-Phase-I-Stocktaking.pdf (accessed on 21 October 2021).