Stakeholder involvement can support better policy outcomes and implementation and generate credibility and trust. There is a strong tradition of stakeholder involvement in Flemish education. As an example, the OECD case study was designed to gather feedback from stakeholders on various aspects of the introduction of standardised tests. This chapter includes their perceptions of the mechanisms and consultations for stakeholder involvement during the initial stages and their motivations for involvement.
Promoting Education Decision Makers' Use of Evidence in Flanders
Educational Research and Innovation
3. Stakeholder involvement
Abstract
Stakeholder involvement as a cornerstone of strategic education governance
Central to this OECD case study is the involvement of stakeholders. The Flemish Department for Education and Training (the Department) engaged the OECD team to consult with different stakeholder groups and give each an opportunity to express their perspectives on the introduction of standardised tests. The aim of this consultation being to document different voices and views.
Stakeholder involvement is a cornerstone of the strategic education governance framework (Chapter 2). This recognises that the main benefits of involving stakeholders more directly in the policy-making process are as follows (Burns, Köster and Fuster, 2016[1]):
Better policy outcomes: ensuring that policies are in line with the needs and interests of stakeholders, while including their knowledge and expertise, can make a policy more fit‑for‑purpose.
Better implementation: giving the opportunity to influence the stakes of a policy and simultaneously enhancing the understanding of the policy can raise legitimacy and create ownership by stakeholders.
Greater trust: providing direct contact and dialogues between policy makers and stakeholders can generate credibility and trust.
An earlier OECD study on the role of the Flemish attainment targets in systemic quality assurance identified the need for continuous dialogue to share different interpretations of the policy, to point to the original aims and background, and to jointly develop new understandings and solutions (Rouw R., 2016[2]).
OECD research on governance in complex education systems points to four elements that support effective stakeholder involvement:
Clear and active communication and transparency: Stakeholder engagement is based on clear and active communication, ideally tailor-made to a diversity of audiences, and particularly reaching out to the most relevant stakeholders (Burns, Köster and Fuster, 2016[1]). For stakeholders who are not so knowledgeable in policy-making processes, it needs to be clear where decision making happens and how and where they can participate and hold other actors accountable. Transparency entails gathering data and providing stakeholders with information about inputs, processes, outputs and outcomes to prepare their effective participation.
Careful selection: Identifying and selecting stakeholders can be done for participation in different stages of the policy process. In complex systems, this has become particularly challenging since the number of groups with stakes in education has multiplied. Seeking for a broad and inclusive engagement arena is preferable, but may result in the voice of key stakeholders being diluted. Balancing openness with the recognition of the value of key stakeholders requires a sensible and transparent approach (Rouw R., 2016[2]).
Capacity building: Different stakeholders require capacity to assume roles and deliver on responsibilities. In many instances, capacity cannot be taken for granted, but needs to be invested on and built deliberately (Burns, Köster and Fuster, 2016[1]). Capacity building also includes developing the competences for participating in stakeholder engagement processes.
Facilitative leadership: Leadership to engage stakeholders requires facilitative skills and attitudes. Facilitative leadership contributes to empower and mobilise stakeholders, to create trust, to promote consensus and to move collaboration forward, a facilitative leadership. The engaging leader or facilitator is sometimes depicted as a steward, focused on the process, with a high “technical credibility” (Ansell and Gash, 2007[3]).
Chapter 2 provides details of the stakeholders invited to participate in the OECD case study. In Figure 3.1, the OECD team provides an overview diagram presenting key elements related to standardised tests and how these and the various stakeholder groups surround Flemish schools. Central bodies interviewed include the Flemish education inspectorate (the Inspectorate), the Agency for educational services (AGODI) and the Agency for higher education, adult education, qualifications and study grants (AHOVOKS). Stakeholder representative bodies are indicated in blue, as although at the central level, they represent the perspectives of students, teachers and parents.
Figure 3.1. OECD mapping of stakeholders and key elements related to standardised tests
A snapshot of stakeholders invited to participate in the OECD case study
Note: The key elements identified are (from the outside in): the ‘OK’ Quality framework is a common reference for the inspectorate, umbrella organisations and schools; the central annual sample (peilingen) tests students in primary and secondary education; the Dataloep platform provides data feedback to schools; the central attainment targets are set for students in primary and secondary education; and the umbrella organisations Catholic Education Flanders (KOV) and the Educational secretariat of Flemish cities and municipalities (OVSG) offer tests to primary schools. IDP stands for the inter-Diocesan tests offered by KOV.
Box 3.1. Discussions with stakeholders on their involvement
Stakeholder involvement
Stakeholder involvement integrates the perspectives and knowledge of schools, school boards, networks and other stakeholders in policy making. In Flanders, this safeguards the constitutional principle of freedom of education while creating suitable feedback channels and strengthening accountability to parents and the public. It also helps to identify capacity needs and address concerns, which may be overlooked otherwise.
What concerns do you have regarding the introduction of standardised tests?
Are you satisfied with your involvement with the introduction of standardised tests?
How can your professional knowledge and expertise contribute to the introduction of standardised tests?
The OECD team explored how each stakeholder group could contribute to the introduction of standardised tests and invited them to raise any concerns they had (Box 3.1).
Stakeholder involvement at the initial stages to develop standardised tests
This chapter documents feedback from stakeholders gathered during the OECD case study. As such, it represents the various perceptions, including motivations and concerns, at the initial stages of the development of standardised tests (Figure 3.2).
Figure 3.2. Timeline of initial stakeholder involvement
Situating the feedback gathered during the OECD case study
Note: VLOR is the Dutch acronym for the Flemish strategic advisory council for education and training. “The Department” is the Flemish Department of Education and Training. ‘SG’ stands for steering group.
In the last quarter of 2020, the Department of Education and Training (the Department) had established two main mechanisms to support the development of the standardised tests:
The university centre is a consortium of higher education institutions cooperating with the Department on introducing standardised tests. Their main tasks consist of developing the test items, statistical analyses, designing the feedback and developing scripts for the test administration. The University centre comprises all five universities and two higher education colleges (with teacher training facilities) in Flanders.
A steering group will steer the work of the university centre. The steering group comprises representatives of the Department, the Education Inspectorate, the education providers/umbrella organisations, the education trade unions and the Flemish students association.
On their own initiative, the Flemish strategic advisory council for education and training (VLOR) created a working group to give advice on the implementation of standardised tests in Flanders. The VLOR works independently of the Minister and the Department and can provide advice or organise consultations on all educational matters for which the Flemish Community is competent. In January 2021, the VLOR published a text highlighting its concerns and advice on the conditions necessary for the implementation of the standardised tests.
The Minister decided to establish a specific stakeholder consultation platform to facilitate communication and feedback at key stages of the development of standardised tests. The first meeting of this “High-level forum” was convened in May 2021.
The high-level forum is mandated to supervise the key decisions in policy development for introducing the standardised tests. It is a forum for feedback and input from stakeholders regarding all the policy aspects. These include timing, communication, which students will participate in the tests, etc. At the meetings of the forum, stakeholders are given information on recent policy developments.
Stakeholder perceptions of their involvement at the initial stages
Broad support for the early focus on scientific expertise for test development
During discussions with the OECD team, nobody contested the need for scientific experts to play a prominent role in the early stages of development. On the contrary, stakeholders expect scientific rigour and perceive the high reliability and quality of standardised tests as their added value. In support of this, the existing central sample tests (peilingen) were often cited and in some discussions also international assessments. The OECD noted a perception that many existing tests used in schools were not of the desired quality. Notably, parental and student representatives raised several doubts and concerns about inconsistencies and varying quality of existing tests used in schools. Many stakeholders referred to evidence from the Inspectorate that supports this (see also Chapter 5).
The university centre unites academic partners from different backgrounds (psychometrics, statistics, Dutch language, mathematics, teacher education) and different institutions to support the development of standardised tests. This is symbolically important in achieving broad academic representation and is a clear strength for the development at early stages, even though individual academic staff may take a more critical stance towards the introduction of standardised tests. In addition, representatives from the Department advise that schools traditionally find it easier to engage with academic actors in joining new initiatives compared to engaging with government actors directly.
Stakeholders voice strong criticism of the initial process
During discussions with the OECD team, with the exception of one interview, stakeholders were strongly critical of the initial process around the introduction of standardised tests. Some took time to describe to the OECD team the regular way of policy making in Flanders (a ‘democratic way’) and illustrated how the process around the introduction of standardised tests had thus far strongly deviated from that. Some stakeholders also pointed out that while there was no specific legal requirement for stakeholder consultation regarding the setting up of the university centre to develop standardised tests, the established culture in the educational field had been to consult with stakeholders at early stages of policy development.
The VLOR depicts the typical education policy cycle in Flanders, as:
The Minister prepares a decree in collaboration with the cabinet and the Department. At that stage, the Minister seeks advice, for example, from the Ministry of Finance and the VLOR. The parliament (decree) and the government (implementing decrees) then decide exactly what the policy measure will look like. This is followed by implementation in the educational field, with the cooperation of the Department and other educational organisations. The inspectorate, feedback from schools, networks and other organisations form an evaluation of the policy, which very often results in the preparation of new policy measures.
Given this context and widely communicated view on Flemish education policy-making tradition, the OECD team noted that the initial approach to develop standardised tests had left many stakeholders with the perception that they were not involved in the project. Several stakeholders expressed frustration at a lack of consultation opportunities and specifically referenced the OECD case study as a significant step to increase stakeholder involvement.
Motivation for greater and more structured involvement
During all discussions with stakeholders, the OECD team enjoyed open and constructive exchanges. Each stakeholder communicated clear motivations and visions for how the standardised tests would best serve educational improvement. Nobody contested that there was democratic legitimacy for the introduction of standardised tests. This was included in the political manifesto of the current government. Several stakeholders specifically mentioned the legitimacy for the government to require more objective information in an area of significant public investment. Many stakeholders were enthused by the prospect of the availability of reliable and regular data on student outcomes.
The OECD team had discussions with stakeholders before the high-level forum was established. At that time, they were missing a consultation mechanism that would provide more structured feedback. During the majority of discussions with the OECD team, stakeholders referred to the fact that the VLOR had taken the initiative to issue a position statement on the introduction of standardised tests. The implication was that this had happened in the vacuum created by a lack of consultation with stakeholders and their frustration at a lack of official involvement at initial stages. However, the OECD team also noted in some discussions that the process to reach agreement on the VLOR statement had not been easy. Students voiced an inherent tension that the VLOR had issued a position statement while at the same time many of the same stakeholders were involved in the steering group to support the work of the university centre.
During discussions with the OECD team, stakeholders communicated their ideas of how they could have greater involvement in the introduction of standardised tests:
Network representatives expressed motivation to capitalise on their established relationships with schools and support the integration of standardised tests in schools’ self-evaluation processes. One network voiced surprise that the government had not already approached them to build capacity and support in the field.
Parental representatives appreciate when the government makes direct contact with them and underlined the possibility to design and circulate surveys to their members. This can get timely feedback to policy makers. They cited the example of quick-turnaround surveys during the ongoing health crisis.
Student representatives noted the possibility to send a questionnaire to their members to get feedback on student attitudes and expectations on the use of standardised tests.
The OECD team did not have the opportunity to speak with school leader representatives. Going forward their involvement will be critical in defining expectations and support for the use of standardised tests in schools. The pedagogical advisory services (PBD) of umbrella organisations have close contact and good relationships with many school leaders. For example, the OECD team noted that the umbrella organisation Flemish Provincial Education (POV) is prioritising the engagement of school leaders in preparing for standardised tests.
The stakeholder reflection seminar pursued these motivations further, asking participants to think about how they would use the standardised tests and the necessary preparations to ensure their effective introduction and use in Flemish schools. Stakeholder feedback during the seminar echoed and expanded on many of the above points. Chapter 5 presents a summary overview.
Unclear communication from the government on the purpose(s) of standardised tests
Without doubt, the biggest concern raised by stakeholders during discussion with the OECD team was a lack of clarity in communication about the purpose of the standardised tests. Stakeholders reported that there had been confusing and contradictory messages from the government on expectations for how results would be used. It was thought that a narrative on the purpose of the standardised tests was missing and in its absence, speculation and confusion were growing. Below are some examples of the way this was expressed during discussions with the OECD team:
“Use of results is entirely missing in government communication at the moment. It is not adequate to simply obtain and give results, there is a need to do something with them.”
“There are mixed communications that school rankings are not wanted, but at the same time there is political expectation for results at the school level. Obviously this is incompatible. There is no clarity on how tests will be used.”
“There is no framework on the impact of tests for schools, teachers and networks. Much fear is due to a lack of vision on what to do with the tests.”
“From the perspective of the educational field, communicating standardised tests as ‘a revolution’, but without an idea of how, makes this all more delicate and complex.”
These points were echoed in the stakeholder reflection seminar (see Chapter 5). There was an expectation that the different scenarios that had been developed would bring forward a decision on the major purpose(s) of the new standardised tests. Many stakeholders were eager for this clarity so as to better engage with the necessary preparations for developing and working with the standardised tests.
Developments in stakeholder involvement
Policy development and implementation of the standardised tests are ongoing. Since, its direct consultation with stakeholders, the OECD team notes several developments in how the Department is working to involve stakeholders.
Communication initiatives: In June, the Department included an interview of two experts in the teacher magazine (Klasse) and organised an online seminar where school leaders could submit their questions and concerns.
Working groups with the university centre: In autumn 2021, the university centre invited teachers, school leaders and representatives of the pedagogical advisory services to participate in working groups. The groups are organised around three topics: (1) the selection of the attainment targets that the standardised tests will cover, (2) test adaptations for students with special educational needs and (3) content and format of the feedback of results to schools. Participation rates in these groups are high, and stakeholders indicate their motivation to participate and to discuss the policy development.
Second high-level forum: By the end of September 2021, the high-level forum received a preparatory text with proposals, based on the feasibility study.
The OECD team notes that the VLOR issued a second position statement in September 2021. In this statement, the VLOR lists several concerns on the introduction of the standardised tests. Their main plea is for a public debate on the standardised tests.
References
[3] Ansell, C. and A. Gash (2007), “Collaborative Governance in Theory and Practice”, Journal of Public Administration Research and Theory, Vol. 18/4, pp. 543-571, http://dx.doi.org/10.1093/jopart/mum032.
[1] Burns, T., F. Köster and M. Fuster (2016), Education Governance in Action, OECD, http://dx.doi.org/10.1787/9789264262829-en.
[2] Rouw R., M. (2016), “United in Diversity: A Complexity Perspective on the Role of Attainment Targets in Quality Assurance in Flanders”, OECD Education Working Papers, Organisation for Economic Co-Operation and Development (OECD), http://dx.doi.org/10.1787/5jlrb8ftvqs1-en.