Measuring Innovation in Education 2023

1. Measuring innovation in education: three approaches based on the innovation survey methodology

Koen Van Lieshout
OECD

Stéphan Vincent-Lancrin
OECD

Abstract

This introduction presents the purpose of this part of the publication, which includes four chapters. Its objective is to showcase how innovation in education could be measured by using a survey methodology, that is, asking some actors about the intensity and nature of innovation in their organisation. This follows a methodology that OECD countries have implemented for decades in the business sector, and more recently in the public sector and education. The different approaches proposed exemplify different methods for achieving this objective, which can be undertaken either for statistical purposes at a system level or for self-reflection at the institutional level. Those public goods are meant to be implemented or adapted by local administrators or actors.

How could administrators or institution leaders use a survey methodology to better understand how much innovation is happening in their system or institution, whether actors have different perspectives on it, and how can they approach innovation itself from different perspectives? This is what this part of this publication is about.

First, it explores the research literature and practice on how to measure important dimensions of the innovation process and outcomes and its applications to education, particularly at the level of educational institutions such as schools (chapter 2). This purpose is met through an evaluation of existing research on innovation, including relevant questionnaire surveys in education.

Second, it presents three sets of model questionnaires for measuring innovation in education (chapters 3, 4 and 5) that are based on this research as well as on the previous OECD experience in this area (OECD, 2014[1]; Vincent-Lancrin et al., 2019[2]). The questionnaires are designed to help local, regional or national public authorities, school principals, or tertiary education managers, to:

• Better understand the process of innovation in education and associated activities and the main factors that influence innovation.

• Monitor changes in pedagogical and administrative practices, including the adoption of innovative practices.

• Identify differences in innovation at the primary, secondary and tertiary education levels (and possibly other sectors such as early childhood and training).

• Identify the drivers or sources of innovation in different domains and at different levels.

• Collect data that can be used to link innovation to existing evidence on educational outcomes.

The three sets of questionnaires include: a main questionnaire that covers all innovation activities (chapter 3), a module of questions for inclusion in other surveys that collects data on the innovation culture of educational institutions (chapter 4), and a questionnaire on the use of innovation to improve equity in education (chapter 5). The first questionnaire is primarily designed to collect statistically representative data (chapter 3), while the primary function of the other two questionnaires is self-reflection (chapters 4 and 5), although they can also be used to collect representative data.

As summarised in Table 1.1 different questionnaire versions were developed for school leaders and teachers and versions of the innovation culture module and equity questionnaire were also produced for students.

Table 1.1. Summary of model questionnaires by page length

	School leaders	Teachers	Students	Chapter
Main innovation questionnaire	14	14	-	See chapter 3
Innovation culture module	5	5	3	See chapter 4
Innovation and equity questionnaire	14	14	12: tertiary students 9: secondary students	See chapter 5

None of the questionnaires have undergone cognitive testing, consisting of face-to-face interviews with a small number of diverse individuals drawn from the population of interest (for instance teachers and school leaders for the main innovation questionnaire). Cognitive testing is strongly recommended before implementing any of these questionnaires to ensure that all questions are understood by potential respondents, as intended, and that respondents can provide reasonably accurate responses. The footnote provides a link to a useful ‘how to’ guide to cognitive testing, developed by Gordon Willis (2004[3]).

Innovation is not necessarily an improvement over existing processes, goods, or services. It is possible for an innovation to make matters worse, for instance a new teaching method could reduce student performance or make learning less pleasurable. Some of the problems caused by innovation are due to conflicting goals, for instance an innovation that successfully reduces costs could have detrimental effects on learning, or back-office innovations could increase instead of decrease the workload of educators.

Measuring innovation is primarily about measuring processes (OECD/Eurostat, 2018[4]) instead of measuring innovation outcomes, although innovation surveys can provide limited data for some types of outcomes. Nevertheless, some processes and practices are more likely to lead to better outcomes, for instance collaborating with external sources of expertise or conducting pilot tests. Collecting this data can help governments and educators to determine if best practices are in use and if improvements are needed. In addition, innovation data can be linked to external outcome data of interest, such as student academic performance or satisfaction with their learning environment, to determine if there is a significant positive or negative relationship between innovation and outcomes. However, while this can be useful as an indicator of the possible success or failure of an innovation, a comparison of different types of interventions on outcomes requires a different methodology based on policy evaluation methods that are generally unsuited to an innovation survey (OECD/Eurostat, 2018, pp. 229-237[4]). This is particularly important for pedagogical innovations or evaluating social inclusion or equity effects, where self-selection bias is likely.

References

[1] OECD (2014), Measuring Innovation in Education: A New Perspective, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/9789264215696-en.

[4] OECD/Eurostat (2018), Oslo Manual 2018: Guidelines for Collecting, Reporting and Using Data on Innovation, 4th Edition, The Measurement of Scientific, Technological and Innovation Activities, OECD Publishing, Paris/Eurostat, Luxembourg, https://doi.org/10.1787/9789264304604-en.

[2] Vincent-Lancrin, S., J. Urgel, S. Kar and G. Jacotin (2019), Measuring Innovation in Education 2019: What Has Changed in the Classroom?, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/9789264311671-en.

[3] Willis, G. (2004), Cognitive interviewing: A tool for improving questionnaire design, Sage publications.