Jeroen Backs
Flemish Ministry of Education and Training
Bieke de Fraine
Flemish Ministry of Education and Training
Miekatrien Sterck
Flemish Ministry of Education and Training
Jonathan D’haese
Flemish Ministry of Education and Training
Katrijn Ballet
Flemish Ministry of Education and Training
This chapter analyses the use of evidence in the policy-making process of implementing standardised testing in the Flemish Community of Belgium. It uses the OECD Framework of Strategic Education Governance to analyse the evidence use journey. The chapter starts with an introduction to the broad context of evidence use in the Flemish education system. It then focuses on the actors and their roles in using evidence in policy making. The chapter gives a short introduction to the policy reform and analyses the use of evidence in this process from a governance perspective considering the knowledge governance, the capacity within the system, stakeholder involvement and the whole-of-system perspective. The conclusion reflects upon how these different building blocks work together.
This chapter focuses on strengthening the use of evidence in policy making in the Flemish Community of Belgium. Key characteristics of the Flemish system are freedom of education1 and high levels of school autonomy.
The Flemish education system currently faces several challenges. These include declining student outcomes (such as falling literacy standards); significant teacher shortages; and educational inequality related to socio-economic status, migrant background and segregation within the system. A challenge not unique to Flanders is that policy making is a highly complex process involving diverse actors with diverse interests. The new attainment targets, for example, were annulled by the constitutional court after complaints by some education providers that they constituted an infringement of the freedom of education.
The nature and extent of evidence use in decision making is currently rather diverse and ad hoc in Flanders. The drive to use evidence in policy and practice has systematically been gaining momentum in the Flemish Ministry for Education and Training. Important steps have been taken to produce evidence for policy and practice. However, some major challenges remain. The use of research results is not yet a systematic practice. There remains a risk of fragmented use of research results and “cherry picking” of research findings and evidence. Research and evaluation are sometimes used as a tool for negotiation driven by political agendas rather than as an instrument for development. Policy makers sometimes lack the necessary capacities and time to gather and use evidence. At the same time, the research capacity to conduct policy and practice-oriented research is also rather limited.
The use of evidence in policy making is embedded into the professional standards that civil servants working in the Flemish Ministry of Education should adhere to. Every policy advisor is expected to use valid and reliable evidence related to their topic and keep abreast of recent research related to their field. Furthermore, they are expected to suggest themes for further research and maintain their network of contacts within the field of education and research.
The Ministry for Education and Training’s Department of Education and Training is responsible for policy development. Within the Department of Education and Training, a small team works on policy research and evaluation to support policy makers and policy advisors in collecting and making use of valid and reliable knowledge. The Research and Policy Evaluation Team is part of the larger Strategy and Knowledge Division, where knowledge and long-term policy planning come together (Figure 5.1). Other teams in this unit are the Legal Advice Team, the International Education Policy Team, the Strategic Planning Team and the Data Team. The Data Team supports policy makers with data and data analysis.
The Research and Policy Evaluation Team identifies and formulates knowledge needs. It supports the development of further knowledge primarily by tendering external research (policy and practice-oriented research and international research), occasionally conducting research (e.g. providing support to evaluation commissions) and fostering networks such as contact with researchers (to learn about research that has been conducted with other funds). The team supports the mobilisation, dissemination and use of knowledge through tools and mechanisms such as a website, one-off events and seminars with brokers. The team brokers research to colleagues in the Ministry for Education and Training. The team members are experts in educational research and evaluation and mainly have a research background but do not necessarily have expertise in a specific domain of educational policy. Research initiated by this team has a strong link to the policy agenda and is organised in line with the policy cycle. The Research and Policy Evaluation Team builds bridges within the ministry and between the ministry and other actors.
Flanders has no real tradition of standardised testing. In primary education, there has been a tradition of administering tests developed by two of the umbrella organisations for schools. Since 2018/19, primary schools are required to administer tests to students in Grade 6 and must choose from a toolkit of validated tests, which includes the central parallel tests and tests of the umbrella organisations. In 2019, the new coalition government agreed to introduce standardised assessments. From the 2023-24 school year onwards, primary and secondary schools will therefore be implementing the first Flemish tests. Flemish tests are standardised and validated cross-network and cross-sector tests, whose primary aim is to support the internal quality assurance of schools and thus help to improve educational quality. Standardised testing monitors learning outcomes and measures pupils’ progress. This knowledge can contribute to schools’ internal quality assurance system and the external quality control by the inspectorate. It also helps the pedagogical guidance services to provide support to schools. The tests are set up centrally, taken digitally, and will be analysed and reported on in a dashboard. They are initially limited to Dutch (reading comprehension and writing) and mathematics.
The key driver for introducing standardised tests in Flanders was a decline in student attainment. International assessments such as the Programme for International Student Assessment (PISA), Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS) have indicated a gradual decline in literacy and numeracy, and this trend has been reflected in national sample-based assessments (“peilingen”).2
While the research clearly indicated a decline, it took time for this trend to be acknowledged by the general public and in the educational field. There was initially no consensus that student results were declining, although this shifted in 2017 when the results of PIRLS 2016 were released and a “PIRLS shock” occurred in Flanders. Nonetheless, several voices in Flemish education remained convinced that the quality of education is still high (citing the very high results in PISA in 2003). There has been more awareness in recent years of the decline in results, thanks in large part to the media fostering awareness among the general public. Moreover, recognising the decline and the ambition to turn the tide has become a very important element in the political discourse.
To analyse the evidence use journey, we use the OECD Framework of Strategic Education Governance (Shewbridge and Köster, 2019[1]). This framework explores how to best achieve policy objectives for an education system characterised by multi-level governance structures and a multitude of stakeholders. The framework identifies six interrelated domains that collectively promote more effective governance processes: accountability, capacity, knowledge governance, stakeholder involvement, strategic thinking and a whole-of-system perspective (Figure 5.2). These domains of strategic education governance can help government authorities steer complex education systems towards established goals. We examine four of these domains separately in our analysis of evidence use in a policy process. The other two domains, accountability and strategic thinking, are mentioned in a cross-cutting way across the different sections.
Source: Shewbridge and Köster (2019[1]), Strategic Education Governance, https://www.oecd.org/education/ceri/SEG-Project-Plan-org-framework.pdf.
Several years before the introduction of standardised testing, a strong knowledge base was built around evaluation and assessment in Flanders. In 2011, Flanders participated in an OECD Review on Evaluation and Assessment. The OECD recommended that the Flemish Community seek ways to collect more performance information to support school evaluation (Shewbridge et al., 2011, p. 112[2]). In the following years, several Flemish studies explored the possibilities for implementing standardised tests specifically in the Flemish context. These include:
A study commissioned by the Flemish government and conducted by researchers at the University of Antwerp showed that the general aversion in Flanders to standardised tests had diminished, and there was more positive commentary about standardised tests (Penninckx et al., 2017[3]). The study laid out the possibilities for large-scale testing and highlighted possible scenarios for Flanders based on a detailed analysis of case studies of standardised tests in other countries.
A feasibility study was carried out in 2020 by independent researchers and focused on the impact of testing, the organisational aspects, legal issues and technological challenges.
The design for the new standardised testing in Flanders drew on these pieces of research evidence and other studies on testing. Policy makers grappled with conflicting research evidence on high-stakes testing. They considered the ample evidence from international research that shows that high-stakes testing can lead to unintended use (Box 5.1), such as fraud and student selection. The research on the beneficial effect of high-stakes testing on student allocation to educational tracks was considered, but conflicts with the tradition of free track choice in Flanders.
Examples from abroad demonstrated that publishing school rankings and naming and shaming schools based on results is counterproductive. There is also evidence that high-stakes testing leads to more “shadow education” – the phenomenon of private tutoring covering subjects already taught in school (Carnoy and Loeb, 2002[4]). On the other hand, high-stakes testing has been shown to be beneficial for equity in student allocation, as students are more objectively allocated to study programmes, and there is a lower impact of socio-economic status in this allocation (Bol et al., 2014[5]).
Another important challenge that policy makers faced was that the topic of standardised tests showed some polarisation among academics. It seems that researchers are often either explicitly in favour of, or against, standardised testing, and express a strong ideological stance on the issue.
Consequently, it was decided that the implementation of the standardised tests would be scientifically monitored to evaluate to what extent intended and unintended effects occur.
The approach to standardised tests in Flanders is also rooted in policy makers’ strong understanding of the Flemish cultural context. The result of the decision to combine insights from research with an understanding of what can be achieved given the cultural context is that the proposed tests will be low stakes (Box 5.2).
Cultural perceptions of accountability: The decision to propose low-stakes tests reflects in part the still predominant cultural aversion to accountability measures in education among Flemish stakeholders. The dislike of accountability in an educational context is linked to the concept of freedom of education as a constitutional right in Flanders. Moreover, accountability is often perceived as creating more work for teachers. Accountability is often construed as “vertical accountability” (top-down), while there is no strong tradition of horizontal accountability in Flanders.
Tradition of free track choice: While Flemish policy makers were aware of the research on high-stakes tests and equity in study programme allocation, the decision was taken to uphold the Flemish tradition of allowing free choice of study options.
Involving the inspectorate: Recommendations from different studies suggested that giving the inspectorate access to school results would highlight the accountability component of the tests. The OECD, on the other hand, saw opportunities to place standardised tests within the strengths of the current accountability system that focuses on dialogue for improving practice (Shewbridge and Köster, 2021, p. 69[6]). It was decided that the inspectorate would receive the results of all schools every year thus allowing it to use the results as input for school visits and perform their work more efficiently.
Combining insights from research with an understanding of the cultural context, it was concluded that low-stakes tests would be the most appropriate approach. Striking a balance between research arguments on the one hand and cultural/practical arguments on the other requires dialogue and openness towards other perspectives and concerns. This is generally a time-consuming exercise to reach a compromise. In the Flemish case, it is sometimes unclear who can take the final decision.
For example, researchers from the Research Centre have argued in favour of digital writing assignments in Grade 4. They carried out a pilot in a small sample of schools that was successful. However, many stakeholders were reluctant and argued that this would mainly measure typing skills in 10-year-olds. The researchers then decided not to include writing assignment testing in primary schools in 2024. The topic will be rediscussed at a later stage.
Generally, the Research Centre writes a report (e.g. on testing students with special educational needs, testing assistants or data literacy) that is discussed in the steering committee. During the discussion, arguments are given on feasibility, workload and impact on schools and teachers. Sometimes the researchers are asked to update their report. When the topic is considered strategic, policy advisors at the ministry summarise all the information for the minister with suggestions for policy decisions. The minister takes the final decision.
In Flanders, many staff members of umbrella organisations and teacher unions have a research background. And some researchers and policy makers have a teaching background. Most test developers have been teaching Dutch or mathematics. This helps understand each other’s perspective and can also impact the process of decision making.
Having readily available evidence is, however, not sufficient. It is important to compile and synthesise evidence to make a strong case. Policy makers and stakeholders involved reflect upon the different pieces of evidence in diverse ways, all with their own interests and opinions. In addition, research evidence rarely translates into simple conclusions or statements.
Policy team: A small policy team made up of civil servants from the Department of Education and a contracted IT business analyst supervises the introduction of the standardised tests. This team collaborates intensively with other colleagues within the ministry with specific expertise, such as communication, curriculum, legal issues, technology, the inspectorate’s role, etc. The project leader has a strong academic background in educational effectiveness research. This is an important asset in building bridges between researchers and policy makers.
While there is a strong mix of skills within the team, the number of full-time staff is limited for the complexity of the reform project. This makes implementation challenging. Civil servants give evidence-informed advice to political decision makers. However, their role requires them to be committed to loyally implementing political decisions that have been taken, making them distinct from researchers who have academic freedom. This requires civil servants to have skills in working with evidence, communicating, collaborating and advising the political level (European Commission, 2023[7]). These skills are necessary to be able to act as boundary spanners, bridging science and policy by gathering relevant external scientific knowledge and communicating it to the relevant users within the public policy organisation.
Stakeholders: Capacity to understand and interpret scientific evidence varies. Discussions on standardised testing can be highly technical and require a scientific background. At the same time, stakeholders have strong opinions on this topic, according to their specific interests.
Researchers: The Flemish Research Centre for Central Assessment in Education (Box 5.3) was established to develop the tests. Academics (e.g. psychometricians, psychologists, mathematicians, educational scientists) in Flanders are strongly involved in the policy development on this topic. However, academic reports must be translated into policy decisions. This is a complex and multi-layered process. One example is the policy decision process on measuring student learning gains over time (see Box 5.4).
In addition, researchers’ views and opinions are sometimes ideologically driven and can be opposed to the political vision. More generally, vulnerability to ideological and political bias among researchers has been flagged as an issue of research production in education and other social sciences (van Damme, 2022[8]).
The Flemish Research Centre for Central Assessment in Education is a consortium of 40 academics from all Flemish universities and two universities of applied sciences. It was a deliberate decision to opt for capacity building in the academic world rather than outsourcing the development to a commercial test development partner to develop valid, reliable tests and keep up with new developments in educational measurement, adaptive testing and artificial intelligence applications.
The centre develops the test items. The researchers also produce academic reports on crucial topics, such as measuring learning gains, standard setting and test validity for students with special needs. The reports generally do not support one clear “best evidence buy” but describe different scenarios and their implications. The research reports are discussed in a steering group made up of policy makers, researchers and stakeholders. The steering group’s role is to give advice, drawing on the expertise of its members.
The work environment can be challenging for researchers engaged in the centre. Standardised testing is a highly politicised and strongly debated subject in Flanders, and researchers often have to defend their positions.
The following capacity challenges were highlighted in relation to researchers and academics in the Flemish context:
The number of researchers working on these topics is limited compared to other (larger) systems.
Few researchers carry out educational policy research (research to support educational policy making). There is a perception that this type of research is not highly valued in Flemish academia, despite ample funding made available by the ministry.
In 2022, the Flemish Research Centre for Central Assessment in Education wrote a technical report on measuring learning gains. Researchers and stakeholders discussed the report at the steering committee meetings. Stakeholders appreciated the researchers’ technical and academic input, but their own perspective is more value-based and predominantly concerned with the effects of this measurement on teachers and school leaders. In particular, several stakeholder groups (teacher unions, umbrella organisations, the Flemish Education Council) expressed great concerns about the measurement of learning gains. The researchers highlighted that learning gains are the best measure of true school effects, but stakeholders feared that school quality would be narrowly judged based on student learning gains.
The policy team at the ministry summarised both the research evidence and the concerns to support the minister in his decision making. This summary report, which provided policy advice, was written for the minister but also shared with the researchers. The researchers’ reaction was rather negative, saying there was no correct balance between their research input and stakeholders’ concerns. In their view, the policy advice was too concerned with pragmatic and feasible solutions. In the end, the minister decided that in the first years, learning gains would only be measured between Grades 4 and 6 for school feedback and in parallel the researchers will study if and how feedback on learning gains in secondary schools might be implemented in later years. This illustrates the types of compromises decision makers make to reconcile evidence and the nature of policy making.
Capacity building is thus an integral part of the knowledge governance and stakeholder engagement processes to develop and use research evidence for policy-making purposes.
Implementing standardised testing is a very complex, technical and time-consuming process which is also highly value-laden. To successfully implement the standardised testing within the agreed-upon time frame, policy makers had to start almost immediately by establishing the Research Centre that would be in charge of developing the tests. There was no time for in-depth consultation with stakeholders on the introduction of the standardised tests, which gave stakeholders the feeling of being faced with a fait accompli.
Different platforms for stakeholder involvement have been established along the way. One such platform is a high-level forum established by the ministry to act as a feedback mechanism between stakeholders regarding the tests. The forum meets to discuss specific topics at key stages of the policy-making process. Another example of stakeholder involvement is the steering group that discusses the work of the Research Centre for Central Assessments. In addition, there are “satellite steering group meetings” to discuss very specific topics, such as the automatic scoring of writing assignments. All of these platforms involve the researchers to provide evidence and stakeholders are invited to collectively appraise these pieces of evidence. Moreover, staff in several stakeholder organisations include former academic researchers involved in these platforms.
Schools, as the end users of the tests and the feedback reports, are also consulted in the decision-making process by the Research Centre (through "denk tanks"; see Box 5.5) and by the ministry (for example, through webinars with Q&A sessions).
In the present stage of the proposed reform, stakeholder engagement takes place through denk tanks – user groups organised by the Research Centre. The topics for discussion in the denk tanks are the test items (for mathematics and Dutch) and the visualisation of the school feedback dashboards. The denk tanks are stakeholder dialogue mechanisms that emphasise co-creation – they present a way of checking proposals from researchers with professional knowledge. These user groups are comprised of teachers, school leaders, counsellors, teacher trainers and inspectors. The researchers present their ideas and proposals and the user groups give their advice based on their practical knowledge of daily life in schools.
Engaging stakeholders poses multiple challenges. First, discussions at the steering group sometimes become highly technical and difficult to follow for stakeholders that lack a scientific background. Strong brokerage skills are thus very important for the policy team to keep everyone aboard.
Second, while each platform has clearly defined goals, in practice they become blurred. For example, a process of providing feedback on a report can devolve into a negotiation process. It is important to try and find common ground in dialogue and listen to concerns. The highly technical nature of the project also proved challenging in this regard. A lot of misconceptions exist about the tests, making it challenging to develop a common language.
The third difficulty is that stakeholders consulted in dialogues do not necessarily represent the diversity of opinion within the groups they represent. For example, the Flemish Education Council has written four pieces of critical advice on introducing the tests. The discussions on the last text did not result in a consensus with respect to how the text could be revised to reflect the diversity of all stakeholders. Ultimately, the advice text highlighted the risks and disadvantages of the Flemish tests. This text was approved by the majority of stakeholders yet a few stakeholders added a letter to the text in which they expressed an opposing view.
In conclusion, stakeholder engagement is closely linked to knowledge governance in the sense that thoughtful engagement with research evidence requires stakeholders to have opportunities to collectively appraise evidence while bringing their own contextual and professional knowledge to the discussion. Different platforms have been installed to give stakeholders these opportunities. However, stakeholders have voiced criticism of their involvement in the standardised tests reform. While the current denk tanks (Box 5.5) represent a promising avenue for strengthening stakeholder involvement, there is a perception that more could have been done to engage stakeholders early in the process.
The implementation of the tests is closely linked to a number of other large and complex policy dossiers. In particular, three other major developments affect the implementation of the tests: 1) the revision of the curriculum (and the introduction of new attainment targets); 2) the reform of the pedagogical guidance services; and 3) a significant shortage of teachers. Concerning the shortage of teachers, stakeholders have raised the question of whether the tests can have an impact on the attractiveness of the teaching profession.
It is challenging to align these different developments in the various policies due to the strict time frame of the implementation of the standardised tests and the highly value-laden and sensitive character of the curriculum reform. However, within the Ministry for Education and Training, there is very close co-operation between the teams working on the Flemish tests and the team working on curriculum development, exchanging information and sources of evidence. Given the small scale of the Flemish education system, there are also researchers involved in both the process of test development on the one hand and the new curriculum development on the other.
To maintain a whole-of-system perspective, the standardised tests need to be linked to other policy goals. It will be very important to clarify for schools how the standardised tests will be linked to attainment targets and the broader educational quality reference framework (see: https://www.onderwijsinspectie.be/en). There are roles here for the Flemish Education Inspectorate and the pedagogical guidance services to document expectations of how schools can interpret and position the data from the standardised tests in a broader array of evidence (Shewbridge and Köster, 2021, p. 8[6]).
The process of introducing standardised tests in Flanders has drawn on strong evidence, including from other countries. Background research was undertaken that subsequently informed the design of the proposed reform. The decision to make the tests low stakes reflected policy makers’ engagement with research evidence but also their own knowledge of the Flemish cultural context. The decision-making process on the design of the tests was a case for thoughtful engagement with research in the policy community in the sense that the decision makers combined research evidence, contextual knowledge (e.g. stakeholders’ beliefs and practice) and political values. The Council of State recognised this in its advice on the decree on standardised testing, stating that the policy decision to provide an exception on public transparency in order to avoid public rankings was built on evidence from research.3
We can conclude that having readily available evidence is not enough – it is important to compile and synthesise evidence to make a strong case. Policy makers and stakeholders reflect upon the different pieces of evidence in diverse ways, all with their own interests and opinions in mind. In addition, research evidence rarely translates into simple conclusions or statements. Despite this, the ministry was successful in “speaking as one voice” throughout the reform process. However, stakeholders perceived this as a top‑down discourse. Flemish policy makers spoke about “informing” teachers and stakeholders and “explaining” aspects of the tests to them. Discussing with or entering into dialogue with teachers was not part of their discourse.
Flemish policy makers had to cope with the challenges of time pressures and expectations of delivering the reform within a given time period while also effectively engaging stakeholders, which also requires time. The implementation of the Flemish tests has taken place within a tight and ambitious time frame, and the first large-scale test for 140 000 students in two subjects is planned for May 2024. The tests are expected to be modern, digital and adaptive, and all IT systems and supporting databases must be operational within this timeline. There is an important balance to strike between investing the necessary time in stakeholder engagement and building trust while delivering reform within an acceptable time frame. These conclusions demonstrate the ways in which the different components of the Strategic Education Governance Framework play out in close combination. Stakeholder engagement is closely linked to knowledge governance in the sense that thoughtful engagement with research evidence requires stakeholders to have opportunities to collectively appraise evidence while bringing in their own contextual and professional knowledge. Such opportunities for collective appraisal of evidence involving various groups of stakeholders can help build a better understanding of evidence as well as trust in the process. However, these were notably absent in the Flemish case throughout the process of introducing standardised tests. Knowledge governance is, in turn, linked to capacity, both because creating such opportunities requires central (policy) capacity and because stakeholders also need special competences to be able to engage with research. Capacity building is thus an integral part of the knowledge governance and stakeholder engagement processes.
[5] Bol, T. et al. (2014), “Curricular tracking and central examinations: Counterbalancing the impact of social background on student achievement in 36 countries”, Social Forces, Vol. 92/4, pp. 1545-1572, https://doi.org/10.1093/sf/sou003.
[4] Carnoy, M. and S. Loeb (2002), “Does external accountability affect student outcomes? A cross-state analysis”, Educational Evaluation and Policy Analysis, Vol. 24, pp. 305-331, https://doi.org/10.3102/01623737024004305.
[7] European Commission (2023), “Competence frameworks for policymakers and researchers”, web page, https://knowledge4policy.ec.europa.eu/projects-activities/competence-frameworks-policymakers-researchers_en#theframeworkmodels.
[3] Penninckx, M. et al. (2017), “Delphi study on standardised systems to monitor student learning outcomes in Flanders: Mechanisms for building trust and/or control?”, Studia Paedagogica, Vol. 22/2, pp. 9-31, https://doi.org/10.5817/SP2017-2-2.
[2] Shewbridge, C. et al. (2011), OECD Reviews of Evaluation and Assessment in Education: School Evaluation in the Flemish Community of Belgium 2011, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, https://doi.org/10.1787/9789264116726-en.
[6] Shewbridge, C. and F. Köster (2021), Promoting Education Decision Makers’ Use of Evidence in Flanders, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/de604fde-en.
[1] Shewbridge, C. and F. Köster (2019), Strategic Education Governance – Project Plan and Organisational Framework, OECD, Paris, https://www.oecd.org/education/ceri/SEG-Project-Plan-org-framework.pdf.
[8] van Damme, D. (2022), The Power of Proofs: (Much) Beyond RCTs, Center for Curriculum Redesign, https://curriculumredesign.org/wp-content/uploads/The-Power-of-Proofs-CCR.pdf.
← 1. Freedom of education means that anyone can start a school and receive government financing as long as a core set of requirements is fulfilled and there is freedom of choice for parents and students.
← 2. Peilingen are annual national sample-based tests. They are developed each year for the sixth grade of primary education and the second, fourth or sixth grade of secondary education. Every school year, one subject is tested in primary schools and one in secondary education (Shewbridge and Köster, 2021[6]).
← 3. Council of State section Legislation opinion 72.476/1 of 31 January 2023 on a preliminary draft decree on Flemish tests in education.