Strategic thinking requires an approach to policy development that emphasises adaptive capacity and co-operation among stakeholders. It looks to establish shared goals and co-ordinate action. Taking a step back, the OECD team asked stakeholders to think about the Flemish system as a whole. Is there a shared concern on the quality of education? If so, how could standardised tests play a role in addressing this?
Promoting Education Decision Makers' Use of Evidence in Flanders
Educational Research and Innovation
4. Strategic thinking and whole-of-system perspective
Abstract
Strategic thinking and whole-of-system perspective
Education systems are complex structures with dynamic relationships between stakeholders and decision makers at various levels. Education governance needs to juggle this dynamism and complexity at the same time as steering a clear course towards common goals. To this end, it increasingly relies on strategic thinking (Burns, Köster and Fuster, 2016[1]; Burns and Köster, 2016[2]). This is in line with a growing literature arguing that strategic planning, in the sense of a deliberate, precise and integrated long-term plan, is unsuited in complex environments (Van der Steen and Van Twist, 2018[3]). In education governance, strategic thinking differs from strategic planning in two key ways.
First, it emphasises flexibility and adaptive capacity, rather than being a matter of developing a long-term plan that assumes linear and predictable developments of the system. This is important because addressing complex issues requires being able to respond to varying local conditions and needs, as well as being aware of and prepared for potentially diverging and even unexpected effects of policy interventions (Frankowski et al., 2018[4]).
Second, it is a collaborative process aiming to strengthen capacity for strategic thinking at all decision‑making levels of the system, rather than being a matter of the central level or individual decision makers rolling out a plan on all other levels of the system. Policy and reform require simultaneous and sustained interventions at as many parts of the system as possible (Mason, 2008[5]). Effective governance therefore needs to emphasise collaborative dynamics between different parts of the system. It has to build on strategic thinking, collaboration and trust – in contrast to supervision and control, which have been traditional forms of governance in many systems (Osborne, 2006[6]).
New policies have greater potential to succeed if stakeholders share the goals and components of reforms, and take action in alignment with them (Burns, Köster and Fuster, 2016[1]). Introducing new ways of thinking and working together can be difficult in education governance, because existing education systems cannot be turned off, redesigned and restarted. Change therefore needs to be introduced in an iterative manner, even if the change itself is contradictory to current practice (OECD, 2017[7]). New approaches and ways of thinking need to be learnt by doing and their implementation has to be inclusive (Hynes, Lees and Müller, 2020[8]).
The OECD strategic education governance framework conceptualises strategic thinking in education governance as broadly involving three main processes that influence each other:
Develop common goals: in order to balance short-term priorities with common goals, decision makers need to first develop long-term goals that incorporate various perspectives of stakeholders across the system.
Adapt to changing contexts: decision makers need to adapt strategies as contexts change and new knowledge emerges from a broad range of sources.
Co-ordinate action: decision makers need to co-ordinate action and balance tensions by fostering co-operation among stakeholders and education actors, who may have different short-term priorities and work realities.
The OECD team asked each stakeholder group to take a step back and think about the introduction of standardised tests at the macro level. First, is there a shared understanding in Flanders that there is a need to focus more on educational quality and its improvement? If so, what role could standardised tests play towards achieving this? Second, how could standardised tests best align with and complement existing efforts, such as the attainment targets and the ‘quality triangle’ approach outlining responsibilities for schools, pedagogical advisory services and the inspectorate? This macro perspective fits within the strategic education governance framework in the domains of ‘strategic thinking’ and ‘whole-of-system perspective’ (Box 4.1).
Box 4.1. Discussions with stakeholders on strategic thinking and whole-of-system perspective
Strategic thinking
Strategic thinking links knowing where to go with strategies of how to get there – especially when contexts change. With evidence from the national sample survey (peilingen) and international assessments, there is a shared concern among stakeholders on the overall quality of education provided to Flemish pupils. Attainment targets provide ‘anchors’ to help advance educational efforts towards agreed goals for students.
Is our understanding correct that there is ‘shared concern’ on the overall quality of Flemish education?
What would be the role of standardised tests in advancing Flanders towards a common goal of educational excellence?
How could standardised tests best complement the attainment targets as a central ‘anchor’?
Whole-of-system perspective
Taking a whole-of-system perspective enables a look at the “big picture” of education in Flanders. It supports better learning and efficiency for stakeholders within the system through greater alignment and co-ordination of efforts. In this way, it seeks to maximise synergies and minimise the duplication/waste of time and effort of those involved.
How would standardised tests best align with existing quality assurance efforts?
How could the introduction of standardised tests be most efficient for your work?
A shared concern on the overall quality of education in Flanders
During discussions with the OECD team, all stakeholders agreed that there is a shared concern on the overall quality of education. There is high awareness and debate about this in the educational field. This greater awareness has been supported, among other ways, by the collective conferences on the results from the national sample surveys (peilingen). This multi-stakeholder discussion engages participants in debate and raises the profile of national results. Given their alignment to the attainment targets, the results of the peilingen have driven the debate forward with authority in the educational field. All stakeholders spoke with respect for the ‘scientific rigour’ of the national sample survey. Alongside cyclical results from participation in international assessments, specifically the OECD’s Programme for International Student Assessment (PISA) and the IEA’s Progress in International Reading Literacy Study (PIRLS) and Trends in International Mathematics and Science Study (TIMSS), this has augmented the evidence base at the system level.
The OECD team did not note any contradiction or contesting of an observed phenomenon of overall decline in student performance in Flanders. As communicated by several stakeholders ‘the diagnosis is there’. However, some stakeholders raised the point that the results from national and international assessments do not reflect the full range of educational quality.
Evidence of performance in primary education
Certainly, results from the national sample assessments (peilingen) at Grade 6 indicate significant proportions of Flemish students who do not demonstrate that they have obtained the expected attainment targets (Figure 4.1). Looking at different content areas in mathematics, in 2016 sometimes only 50% or fewer students were found to meet the attainment targets (see left chart in Figure 4.1). In all but one content area, the percentage of students demonstrating that they met the attainment targets had declined since the national assessment in 2009. The national sample assessments in reading comprehension indicate more stability in the proportion of students demonstrating they meet the attainment targets (see right chart in Figure 4.1). However, the most recent assessment in 2018 indicates a decline. Evidence from the international assessment PIRLS indicates a significant decline in the average reading performance of Flemish Grade 6 students between the 2006 (547 points) and 2016 (525 points) assessments (Mullis et al., 2017[9]). Flemish students also sat the PIRLS test in June 2021 and results will be published in December 2022. This will provide another indicator to assess the performance trend.
Figure 4.1. Primary education: evidence from national sample assessments
Percentage of students assessed in Grade 6 who obtained the attainment target in the specified learning area
Source: Compiled from data available at Steunpunt Toetsontwikkeling en Peilingen (Steering Group for Development of the Sample Tests) (www.peilingsonderzoek.be).
There is an observed decline in the average performance of Flemish students in the TIMSS international assessments of mathematics and science in Grade 4 (Table 4.1). The observed decline is starkest between the more recent assessments in 2019 and 2015.
It is notable that the observed decline in reading, mathematics and science is across the entire performance distribution, that is, from students who are able to complete the most difficult tasks to students who are able to complete the easiest tasks (Table 4.2). However, it is most significant in the middle of the performance distribution (students who are able to perform on the international benchmarks ‘high’ or intermediate’).
Table 4.1. Primary education: changes in average performance in international assessments
Flemish student performance in the Trends in International Mathematics and Science Study (TIMSS Grade 4)
Note: Data in bold denotes a statistically significant difference.
Source: (Mullis et al., 2020[10]).
Table 4.2. Primary education: changes in performance distribution in international assessments
Percentage of students performing at each of the international benchmarks (TIMSS Grade 4, PIRLS Grade 4)
Note: Data in bold denotes a statistically significant difference. TIMSS is the Trends in International Mathematics and Science Study; PIRLS is the Progress in International Reading Literacy Study.
Table 4.3. Primary education: performance in different content areas of international assessments
Relative strengths and weaknesses of Flemish students (TIMSS Grade 4)
Note: Data in bold denotes a statistically significant difference. TIMSS is the Trends in International Mathematics and Science Study.
Source: (Mullis et al., 2020[10]).
A more fine-grained look into the areas assessed in the international assessments reveals that the observed decline is in the content areas numbers and data for mathematics and in life science and earth science for science (Table 4.3). Flemish students had a high average performance in test items assessing measurement and geometry. In both the mathematics and science assessments, Flemish students’ performance declined in test items that assessed the cognitive processes ‘knowing’ (covering the facts, concepts and procedures students need to know) and ‘applying’ (focusing on students’ ability to apply knowledge and conceptual understanding to solve problems or answer questions). In the science assessment, Flemish students performed relatively better on test items assessing ‘reasoning’, where students need to go beyond the solution of familiar problems that may have been routinely practiced in lessons to encompass unfamiliar situations, complex contexts, and multistep problems (Mullis et al., 2020[10]).
Evidence of performance in secondary education
Evidence from the national sample assessment, similar to what has been observed in primary education, reveals high proportions of Flemish students in Grade 8 are not able to demonstrate they have obtained the expected attainment targets in various content areas (Figure 4.2).
Figure 4.2. Secondary education: evidence from national sample assessments at Grade 8
Percentage of students assessed who obtained the attainment target in the specified learning area
Note: Results are for students in the general stream of education (‘A-stroom’).
Source: Compiled from data available at Steunpunt Toetsontwikkeling en Peilingen (Steering Group for Development of the Sample Tests) (www.peilingsonderzoek.be).
In 2018, this was the case for over 50 per cent of the Flemish students assessed in the mathematical content areas of operations, calculating with polynomials and proportions and there had been no improvement since the equivalent assessment in 2009 (see left chart in Figure 4.2). Also, the national assessments of information processing in 2011 reveal over 50 per cent of Flemish students in Grade 8 did not demonstrate the expected minimum levels in consulting tables, graphs, plans, drawings and maps and that this had declined since 2004 (see right chart in Figure 4.2). For the majority of content areas in Figure 4.2, there is no notable improvement, with the exception of data handling and geometry spaces.
Figure 4.3. Secondary education: evidence from the OECD’s PISA (students aged 15)
Mean performance for participating Flemish students (PISA scale score)
Note: Data displayed in bold indicate statistical significance. For example, for reading literacy the differences in mean performance between 2000 (532), 2009 (519) and 2018 (502) are statistically significant.
Evidence of Flemish grade 8 students’ performance in the International Civics and Citizenship Education Study (ICCS) is more encouraging. There is a significant improvement between 2009 (514 points) and 2016 (537 points) (Schulz et al., 2018[11]). The average age of participating students was nearly 14 years old (13.9). The OECD’s PISA assesses students at age 15, meaning that students can be in different grades. In PISA 2018, 72% of participating students were in Grade 10, 23% in Grade 9, 4% in Grade 8 and 1% in Grade 11 (OECD, 2020[12]). Evidence from PISA indicates decline in the Flemish students’ average performance across the various cycles (Figure 4.3). In each assessment, students complete test items to assess reading, mathematics and science. However, the main focus of each assessment rotates, meaning it is possible to compare performance on a larger set of test items in each of the domains every nine years. In reading, there has been a steady decline in the average performance of Flemish students (between 2000 and 2009 and again between 2009 and 2018). The same is observed for mathematics between 2003 and 2012 and for science between 2006 and 2015. In all areas, the decline over the latter period (between 2015 and 2018) is not statistically significant (OECD, 2019[13]).
A complex debate on educational quality and concerns for equity
During discussions with the OECD team, all stakeholders highlighted that there is a lively debate and many differing opinions/approaches to understanding the reasons behind the observed overall performance decline in Flanders. Most frequently cited points about the broader policy environment relating to quality include: attracting and retaining excellent teachers, teacher education and continued professional development; and different contexts and provision for schools in terms of student composition. When stakeholders referred to differing student composition across schools, they made arguments related to fairness and the need for adequate contextual understanding to interpret results on educational outcomes in a meaningful way.
Compared internationally, both novice and experienced teachers in Flanders report high levels of satisfaction with their salaries, which is an important factor in the attractiveness of the profession (OECD, 2020[14]). In both primary and secondary education, statutory and actual salaries for teachers, particularly upper secondary teachers, in Flanders are above both the OECD and European Union averages (OECD, 2021[15]). This commitment to teacher salaries is reflected in comparatively greater expenditure on education in Flanders: expenditure on educational institutions per full-time equivalent student in 2018 was USD 13 507 in Flanders, compared to an OECD average of USD 10 454 (OECD, 2021[15]). As in other OECD countries, teacher salaries in Flanders remain less competitive than those for other similarly educated workers, but are most competitive at the upper secondary level (OECD, 2021[15]).
However, there is also international evidence to back up the concerns raised on attracting and retaining excellent teachers. First, the perceived attractiveness of the teaching profession has declined in Flanders over recent years (Table 4.4). Notably, between the TALIS 2013 and 2018 surveys more Flemish lower secondary teachers agree that they wonder whether it would have been better to choose another profession and less agree that the advantages of being a teacher clearly outweigh the disadvantages. Along with this, there was a significant decrease in the percentages of lower secondary teachers and school leaders agreeing that the teaching profession is valued in society. Second, reports on stress levels and negative impacts on mental health are also above the OECD average (Table 4.4).
Table 4.4. Feedback from educators on the prestige of the teaching profession and stress levels
Teacher and school leader reports in TALIS 2018 and 2013
Source: Compiled from data in (OECD, 2020[14]).
Third, there is evidence of staff shortages gathered from Flemish school leaders as part of the PISA 2018 survey. Compared to the OECD average, there are reportedly higher proportions of Flemish teachers working part time and while the percentage of fully certified teachers compares favourably, there are higher proportions of Flemish students in schools with a reported lack of teaching staff or inadequate or poorly qualified teaching staff (Table 4.5). There is also a concern for equity with the percentage of fully certified teachers reportedly lower in socio-economically disadvantaged schools.
Table 4.5. Secondary education: feedback from school leaders on staff qualification and shortages
Based on reports from school leaders in participating schools (PISA 2018)
Source: Compiled from data in (OECD, 2020[12]).
Figure 4.4. Secondary education: feedback from school leaders on school admission policies
Percentage of PISA students in schools where the school leader reported these admission policies
Source: Compiled from data in (OECD, 2020[12]).
During discussions with the OECD team, stakeholders also noted the importance of the strategic goal to reduce inequities in Flemish education. In consideration of this, many stakeholders voiced great sensitivity about the possibility of the results of standardised tests being available to the broader public. Various stakeholders pointed to the highly competitive context within the Flemish education system and raised strong concerns about the potential use of results by the media to publish simplistic and misleading rankings of school performance. Several stakeholders specifically raised the need to find a legal basis to prevent such ‘misuse’.
To illustrate the competitive context and the minimal tradition of making information on schools public, stakeholders contributed several anecdotal points. These included: initial anxiety in the education field regarding the publication of inspection reports for individual schools and attempts by the media to sensationalise these; the failure of multi-stakeholder equal opportunity platforms (Lokaal OverlegPlatform) to make concrete agreements on student intake; some schools relying heavily on early streaming/tracking of students; and middle class parents paying for people to camp outside certain schools to enrol their children.
The Belgian constitution guarantees freedom of education, including the freedom to choose a school and even to establish a school. Data from PISA 2018 reflect this, revealing that Flemish students have a greater choice of secondary schools in their area, compared to their international counterparts (83% have at least two or more schools in the area, compared to 63% on average in the OECD) (OECD, 2020[12]). In contrast to the OECD average, Flemish school leaders report that residence in a particular area is not a very widespread criterion in admitting students to school (Figure 4.4). Much more commonplace are students’ academic performance and/or interest in a special programme, which is largely explained by the tracking in Flemish secondary education and the fact that many schools offer only one or two tracks. In theory, schools (both primary and secondary) cannot refuse students based on their academic performance. Students in Flanders get priority admission when an older brother or sister attends the school or when one of the parents is a member of staff. This latter policy reportedly plays a more important role in Flemish secondary education compared to on average in the OECD. These results are striking in how admission policies may differ among Flemish schools and lend support to concerns raised by parents and students of a highly competitive context.
Support for standardised tests as tools for school development
The OECD noted from the different discussions with stakeholders a coherent expectation on the role that standardised tests could play in advancing Flemish education towards educational excellence. Standardised tests would be a tool for schools, providing regular, reliable student test results in two key areas. The standardised nature would give all Flemish schools access to objective and comparable feedback. The expectation is that the availability of such data would stimulate schools to focus on outcomes and further strengthen the culture of quality assurance at the school level. The OECD team attributes the coherence of this in part to the fact that several stakeholders made explicit reference to the Flemish Education Council (VLOR) position statement on conditions for high quality tests:
Their potential added value lies in a development-oriented use of the tests. Under the right conditions, the information from tests can support schools to take responsibility for developing their own educational quality.
The OECD team notes that this may also be influenced by an awareness of previous research testing out different scenarios for the development and use of standardised tests in Flanders (Penninckx et al., 2017[16]).
The major argument made for introducing standardised tests primarily as a tool for school quality development was the need to gain trust in the educational field. During several discussions, the OECD team noted enthusiasm to capitalise on and nurture the openness to embrace the opportunities that standardised tests would bring. The OECD team heard that, in general, attitudes regarding the potential usefulness of standardised tests had evolved over recent years and noted a sense that this was a pivotal moment. However, stakeholders also underlined that this openness was by no means universal and that there is a need to build trust through concrete experiences in schools. Notably, the teacher union representatives during discussions with the OECD team commented that they are yet to be convinced of the real value that standardised tests would bring to teachers and schools and raised the challenge of whether it would be better to invest resources in professional development and capacity building.
According to reports in the TALIS 2018 survey, the majority of Flemish teachers and school leaders view their colleagues as open to change and their schools as places that have the capacity to adopt innovative practices. However, this is not the case in all Flemish schools and, in particular, openness to change and accepting new ideas in lower secondary education is low by international comparison, as reported by both teachers and school leaders (Figure 4.5). Among the OECD countries with available data, primary teachers are more open to change, but the differences are ‘particularly pronounced’ in Flanders between primary and lower secondary teachers (OECD, 2019[17]).
Figure 4.5. Openness to change: feedback from teachers and school leaders
Percentage of teachers or school leaders agreeing or strongly agreeing with the specified statements (TALIS 2018)
Note: The first four points are reported by teachers (“Most teachers in the school…”); the last four points are reported by school leaders (“The school quickly…”). There is no OECD average for primary teachers as not all OECD countries chose to administer the survey at this level.
Source: Compiled from data in (OECD, 2019[17]).
In Chapter 5, the OECD team presents an overview of the various motivations cited by different stakeholders during the stakeholder reflection seminar. During discussions with the OECD team, several stakeholders made specific reference to consistent findings from the Inspectorate that point to insufficient focus on goals/outcomes at the school level (for evidence from inspections see Chapter 6). In all discussions with pedagogical advisory services, representatives noted the variation in capacity (and sometimes motivation) among schools and emphasised recent efforts to heighten their focus on supporting schools’ quality assurance efforts. They argue that the availability of regular, objective data from the standardised tests would bolster their support efforts (see also Chapter 5).
Student representatives expressed strong support for the role that standardised tests could play in promoting a more rigorous approach to grading in schools and within networks. They raised prominent concerns regarding current assessment practices and the reliability of many tests used in schools. According to reports from Flemish lower secondary teachers, there is not an established culture to work with other teachers in school to ensure common standards in student evaluations (22% of Flemish lower secondary teachers reported this in TALIS 2018, compared to 40% on average in the OECD) (OECD, 2020[14]). Although limited to assessing competences in Dutch and mathematics, student representatives expressed their hopes that the introduction of standardised tests would promote a more rigorous approach to grading equivalency at the school level (see also Chapter 5).
References
[2] Burns, T. and F. Köster (eds.) (2016), Governing Education in a Complex World, Educational Research and Innovation, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264255364-en.
[1] Burns, T., F. Köster and M. Fuster (2016), Education Governance in Action: Lessons from Case Studies, Educational Research and Innovation, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264262829-en.
[4] Frankowski, A. et al. (2018), “Dilemmas of central governance and distributed autonomy in education”, OECD Education Working Papers, No. 189, OECD Publishing, Paris, https://dx.doi.org/10.1787/060260bf-en.
[8] Hynes, W., M. Lees and J. Müller (eds.) (2020), Systemic Thinking for Policy Making: The Potential of Systems Analysis for Addressing Global Policy Challenges in the 21st Century, New Approaches to Economic Challenges, OECD Publishing, Paris, https://dx.doi.org/10.1787/879c4f7a-en.
[5] Mason, M. (2008), “What is complexity theory and what are its implications for educational change?”, Educational Philosophy and Theory, Vol. 40/1, pp. 35-49, http://dx.doi.org/10.1111/j.1469-5812.2007.00413.x.
[9] Mullis, I. et al. (2017), PIRLS 2016 International Results in Reading, International Association for the Evaluation of Educational Achievement (IEA), https://files.eric.ed.gov/fulltext/ED580353.pdf.
[10] Mullis, I. et al. (2020), TIMSS 2019 International Results in Mathematics and Science, International Association for the Evaluation of Educational Achievement (IEA).
[15] OECD (2021), Education at a Glance 2021: OECD Indicators, OECD Publishing, Paris, https://dx.doi.org/10.1787/b35a14e5-en.
[12] OECD (2020), PISA 2018 Results (Volume V): Effective Policies, Successful Schools, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/ca768d40-en.
[14] OECD (2020), TALIS 2018 Results (Volume II): Teachers and School Leaders as Valued Professionals, TALIS, OECD Publishing, Paris, https://dx.doi.org/10.1787/19cf08df-en.
[13] OECD (2019), PISA 2018 Results (Volume I), OECD, http://dx.doi.org/10.1787/5f07c754-en.
[17] OECD (2019), TALIS 2018 Results (Volume I): Teachers and School Leaders as Lifelong Learners, TALIS, OECD Publishing, Paris, https://dx.doi.org/10.1787/1d0bc92a-en.
[7] OECD (2017), Systems Approaches to Public Sector Challenges: Working with Change, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264279865-en.
[6] Osborne, S. (2006), “The New Public Governance?”, Public Management Review, Vol. 8/3, pp. 377-387, http://dx.doi.org/10.1080/14719030600853022.
[16] Penninckx, M. et al. (2017), Zicht op leerwinst. Scenario’s voor gestandaardiseerd toetsen (Scenarios for standardised tests), Acco, Leuven.
[11] Schulz, W. et al. (2018), Becoming Citizens in a Changing World, Springer International Publishing, Cham, http://dx.doi.org/10.1007/978-3-319-73963-2.
[3] Van der Steen, M. and M. Van Twist (2018), “Strategies for robustness: Five perspectives on how policy design is done”, Policy and Society, Vol. 37(4), pp. 491-513, https://doi.org/10.1080/14494035.2018.1520782.