Promoting Education Decision Makers' Use of Evidence in Flanders

7. Lessons from the OECD case study

Abstract

This Chapter presents a brief overview of lessons learnt in the OECD case study. For each domain of the strategic education governance framework, it presents some key points and reflections on what these imply for the further development of standardised tests.

Stakeholder involvement (Chapter 3)

Prioritising clear and active communication

The OECD case study has identified unclear communication as a point of weakness in the initial stages of developing standardised tests. The establishment of the high-level forum is a constructive step. This can serve as an authoritative communication channel at key stages in the development of the standardised tests and also collect feedback in a timely and transparent way from key stakeholders. It can also be a platform for expert contributions and testimonies from the educational field as the project unfolds.

The overriding feedback from stakeholders in the OECD case study, including from test developers, is the need for a clear steer from the central authorities on the purpose(s) of the standardised tests. There is opportunity to more actively involve stakeholders in the next stage of development, such as to provide input into clarifying the purpose(s) and uses of the standardised tests. This will pave the way to enable stakeholders to take up their roles and responsibilities in preparing for the introduction of standardised tests.

Committing to stakeholder involvement and ensuring key voices are heard

Educational policy development in Flanders has a tradition of involving different stakeholders (especially umbrella organisations in school networks and trade unions). Stakeholders were strongly critical about a lack of consultation processes at the early stages of the introduction of the standardised tests. An important lesson for the government is to take stakeholder involvement seriously at every stage of the policy development.

One important group of stakeholders tends to be overlooked: the school leaders. The absence of a representative body for school leaders weakens this important voice in official channels to support policy development. There is a need to reflect on ways to systematically involve school leaders, for example, with a rotating representation of school leaders from each network in the high-level forum. There are ways to mobilise existing professional connections with the pedagogical advisory services and going forward, as suggested below, via direct interactions with the university centre. Mobilising awareness, support and feedback channels for school leaders will be critical to the successful introduction of the standardised tests.

Ensuring facilitative leadership

The OECD case study has identified much motivation among stakeholders for greater and more structured involvement in the introduction of standardised tests. Strong technical credibility in facilitative leadership can help heighten the engagement of different stakeholder groups. Here, the strong credibility for the university centre as a centre of scientific expertise will provide fertile ground for gaining regular feedback from the educational field during test development.

The department can take the opportunity to empower student voice by supporting the Flemish Student Association’s suggestion to conduct a survey among its membership. This can be a way to seek feedback on key aspects of the standardised tests development.

Organising contributions from the educational field to support the university centre

There is motivation for involvement in test development and opportunity in establishing a coalition of test development partners across educational networks to support the university centre’s work. The university centre can facilitate this by providing clear guidance on scheduling and expected time commitments. The pedagogical advisory services can facilitate and organise the participation of expert teachers and schools. It will be important to engage expertise from teachers and the test developers in the pedagogical advisory services to support the development of specific test items in Dutch and mathematics. There is also a need to engage school leaders and other staff in designing and developing the feedback reports for schools. The development of standardised tests presents an opportunity to bring schools together across different networks and strengthen horizontal collaboration. This interaction between test developers and test users (schools) is an important research-based principle of supporting more systematic use of evidence.

Strategic thinking and whole-of-system perspective (Chapter 4)

Developing, sharing and consolidating common goals and how standardised tests will support these

The OECD case study has found that there is a shared concern on the overall quality of education in Flanders and a body of evidence to support this. Such widespread recognition is pivotal and presents an opportunity to mobilise stakeholders in creating a common vision for the role of standardised tests in supporting the quality of educational provision going forward. The OECD case study has found enthusiasm for the opportunities that standardised tests could offer. There is strong support for a vision of standardised tests as tools to support school quality development. In contrast, stakeholders were unanimous in voicing concern on the eventual publication of results of these tests as indicators of school performance. It is important to consider safeguard measures to support the use of data for school development, including to ensure schools are encouraged to continue to develop and innovate their practice.

Taking a long-term perspective and adapting to changing contexts and new knowledge

The development and introduction of standardised tests in Flanders is a groundbreaking project. The OECD case study has revealed a gradual evolution in attitudes towards the potential that standardised tests could bring for Flemish schools and enthusiasm among many stakeholders. There is value in taking a long-term perspective on the introduction of standardised tests and refining and evolving their development through concrete experiences in the educational field. First, the development of the standardised tests presents an opportunity to bring together the research community (test developers) and schools. Such collaboration will provide many other advantages for professional learning and development on both sides. Once developed, the initial introduction of standardised tests will bring new experiences and opportunities for rich, concrete feedback from the broader educational field. The first few administrations of the standardised tests will generate much knowledge for how to optimise the use of results at the school level.

There would be value in considering (and consulting about) how the introduction of standardised tests will be placed within the context of school (self) improvement processes and practices – as well as broader education strategic planning initiatives to support schools in their ongoing development. For example, standardised tests can be used to develop curriculum as well as monitor equity gaps over time for specific groups, which can then lead to higher-level strategic supports to be implemented. A clear opportunity to solidify initial expectations is to ensure a coherent approach and communication from the Flemish education inspectorate and the pedagogical advisory services on how to use these results for school development as part of the broader view of educational quality (anchored in the ‘OK’ quality framework).

Coordinating action and learning from experiences in the educational field

The OECD case study has noted that there is a complex and healthy debate on the reasons for the observed decline in the overall quality of education in Flanders. Feedback from the educational field highlights growing concerns on the prestige of the teaching profession and quite some variation in openness to change among Flemish schools, particularly between the primary and secondary levels. These points indicate a need to carefully establish positive experiences with the standardised tests in the educational field and to nurture support for these as effective and relevant educational tools for professionals.

The working realities for teachers necessitate a coordinated guidance from the central authorities on the expected use of the standardised tests and the associated time and resource requirements for teachers and schools. This needs to be based on systematic input from the educational field on their requirements and experiences as the standardised tests are being developed and introduced.

Capacity and knowledge governance (Chapter 5)

Ensuring technical capacity for standardised test development and administration

The OECD case study has identified much motivation among stakeholders for greater and more structured involvement in the introduction of standardised tests. Strong technical credibility in facilitative leadership can help heighten the engagement of different stakeholder groups. Here, the strong credibility for the university centre as a centre of scientific expertise will provide fertile ground for gaining regular feedback from the educational field during test development.

The OECD case study has noted some concerns and supporting evidence on the capacity to administer digital tests across Flemish schools. There will need to be a careful evaluation of schools’ capacity to administer digital tests and due attention to field trials when first administering the tests.

Laying foundations for the systematic use of standardised test results by professionals

The OECD case study has documented the major motivations for the introduction of standardised tests. Feedback from stakeholders suggests that standardised tests would be most valuable as tools to: provide reliable information on outcomes for students, teachers and schools; provide comparative feedback to schools for reflection on their development; strengthen and promote a culture of feedback for student learning; deepen professionals’ skills for using data and evidence more systematically; provide data for educational research and policy; and augment the evidence base for school inspections. There is opportunity to use the motivations identified by the stakeholder groups to develop clear, comprehensible feedback reports for schools. In preparing for the use of test results, feedback from stakeholders notes the necessity for: a clear and uniform communication strategy; planning time and resources in schools; preparing guidance for schools; and supporting professionals. The OECD case study allows a mapping of these points to the research-based framework supporting a more systematic use of evidence by decision makers in their work.

Skills: There is a need to give adequate attention to the capabilities of teachers and other school staff to work with the results of standardised tests and other assessments. For teachers, to use these as one form of evidence to give feedback to students and parents on learning progress. At the school level, to interpret the results for the school in light of central benchmarks and to feed this into plans for school development. Teachers voiced the opportunities for deepening professional skills in using data. There is opportunity in committing to investment in professional development and in ways that can support collaborative practices in schools.

Availability: The standardised tests present an opportunity to make available regular and reliable student performance data to schools in the common areas of mathematics and Dutch. The OECD case study documents that the rapidity of results feedback will play into their perceived value and relevance for educators. Notably, students expressed a desire for the standardised tests to bolster the culture of feedback to students on their progress more generally and more rapid feedback would best support this.

Organisational processes: School leaders will drive the preparation of the necessary processes and structures to create the space for effective use of the standardised tests. This can be supported at the system level by preparation of common guidance material for schools – a process that will need to engage school leaders and teachers in a structured way. As in all educational systems, capacities for quality assurance and development at the school level vary across Flemish schools. The OECD case study notes that evidence on school capacity from the Flemish education inspectorate is widely known and referred to by the majority of stakeholders. This indicates a maturity in the educational field as to recognising differing realities and starting points across schools. Notably, the pedagogical advisory services affiliated to the umbrella organisations have nurtured ties with many schools and in recent years have increased their focus on providing support to schools to improve their quality assurance processes. This knowledge and expertise can be tapped into when introducing the standardised tests.

Interaction: The design and development of feedback from the standardised tests will be strengthened by the direct interaction between researchers (analysts and feedback designers) and schools. Importantly, this presents an opportunity to promote horizontal collaboration and learning across the different educational networks.

Standards: The development of guidance material for schools will provide a common anchor for expectations on the use of standardised tests. This should clarify the ways that standardised tests are connected with the existing central anchors of the attainment targets and the broader ‘OK’ quality framework. There are roles here for the Flemish education inspectorate and the pedagogical advisory services to document expectations of how to best interpret the evidence provided by the standardised tests and how to position these in a broader array of evidence at the school level.

Accountability (Chapter 6)

Ensuring the ‘fit’ of accountability instruments

The OECD case study has noted the perception of ‘accountability’ in Flemish education as a matter of internal responsibility and great resistance to the public availability of school performance information. The development of the ‘OK’ quality framework represents a considerable achievement in the Flemish education system. By design, this framework embodies the organised agreement on expectations of school quality and also leaves the necessary room for disagreement and local flexibility in continuing to develop targeted quality goals for the school’s specific community. The ‘quality triangle’ concept is embedded in the educational field, with the schools having the main responsibility for their educational quality, their pedagogical advisory services providing support and the Flemish education inspectorate as the major accountability mechanism. Professional dialogue and rich feedback are valued in inspection processes.

There is opportunity to place standardised tests within the strengths of the current accountability system that focuses on dialogue and deepening an understanding between available data and links to ideas for improving practice. The standardised tests offer opportunities to help ‘firm up’ two important sides of the quality triangle: inspection and pedagogical advisory services. On one side, the Flemish education inspectorate will benefit from the availability of regular, comparable information on student performance in two key areas. This will further enrich the evidence base for school inspections and may support greater reactivity by supporting the implementation of a differentiated approach to inspections. On the other side, the pedagogical advisory services will have a greater evidence base to work with schools seeking their support. In recent years, these services have increased their focus on supporting schools with their quality assurance processes. This would not change the existing accountability mechanism that allows the inspectorate to obligate a school with noted quality issues to engage with pedagogical support services and initiate an improvement trajectory. Both sides of the triangle will continue to anchor their work in the common ‘OK’ quality framework. This will be key to helping interpret and use the results of standardised tests in a constructive and proportionate way.

Enhancing critical reflection on substantive expectations

The development of standardised tests that provide common feedback to all Flemish schools will provide an objective and external perspective for school development. Schools carry the main responsibility for the quality of their educational provision. The availability of regular, reliable data on student performance with comparative benchmarks will be a basis to further strengthen their critical reflection. To fully support the learning function, there would be value in exploring mechanisms for designing data use and interpretation by teachers and school leaders to support informed practice and strategic planning.

In turn, the Flemish education inspectorate can confer a valuable perspective to schools on how to interpret and use the results of the standardised tests as part of their quality assurance processes. Regular exchanges with schools via school inspection processes will strengthen the knowledge and capacity of school inspectors regarding effective and innovative ways of working with the standardised tests at tools for school development.