Measuring Innovation in Education 2019

Annex B.  Composite indices of innovation

The analyses reported throughout this book have shown considerable variation in the amount of change in educational practices and thus the potential extent of innovation. In order to provide an overview of change across school and classroom practices and to draw some conclusions about the level of innovation in each country, it may be considered helpful to combine some of this information and look at the extent and focus of innovation within education in different countries.

There may be important differences between practices at different education levels (primary or secondary) or across disciplines. For this reason, broader composite indices have been created to group together practices and represent innovation at the discipline level- maths, science and reading and at the education level- primary and secondary, besides and index for overall educational innovation. Additionally, composite indices for ICT practices and more specific educational practices have been computed. This allows readers and policy makers to identify which aspects of countries’ education system(s) appear to have experienced relatively more innovation, and identifies countries that are innovating throughout the education system.

Creating the indices

The indices draw from the analysis reported in this book. The approach used is broadly based on the guidance provided in the 2018 OECD handbook on constructing composite indicators. In particular, the indices are derived (as far as possible) from the definition of innovation discussed in the introduction and the process of creating them takes into account the need for appropriate data and imputing missing values.

The indices are based on the effect sizes of changes in responses to specific questions between baseline and endline years. Effect sizes reflect the size and direction of changes seen across two points in time, with a large positive effect size indicating a large increase over time and a large negative effect size indicating a large decrease. Effect sizes give a standardised measure of the change and can thus be easily added together.

Table B.1. Data sources for indices

Study name	Questionnaire used	Grade/age covered
TIMSS	Principals Teachers Students	4th grade 8th grade
PIRLS	Principals Teachers Students	4th grade 8th grade
PISA	Principals Students	15-year-olds

Education level, discipline level, and overall indices of innovation

These indices are constructed in order to represent change in practices across different grades, disciplines or throughout the whole education system. Given that both increases and decreases indicate change which can be part of innovation, the absolute value of the effect size has been used to create these indicators. An index that kept the sign of the effect size would make countries that have large changes in both directions appear to have no change at all.

In order to have a fair representation of innovation, different disciplines have been given different weights at different levels. Primary and secondary levels were given equal weights, whereas maths, science, and reading were given different weights defined on the basis of the relative instruction time spent on each one of the disciplines in every respective grade (source: Education at a Glance 2011) For instance, as reading instruction time is roughly twice as large as science instruction time in primary education, change in reading practices was given twice as much weight as change in science practices for this particular level.

ICT and thematic indices

These indices illustrate change in more specific educational practices. However, it is relevant in this case to not only analyse whether the use of certain practices has met significant change, but also whether the use has more often increased or decreased. Thus, besides the value of composite indices with absolute effect sizes, the graphs for ICT and thematic practices also demonstrate the decomposition of the change into increases and decreases.

The conceptual grouping of these indicators was done to maintain a more or less balanced representation of practices across both grades and across all the disciplines. This allowed us to go ahead with an unweighted average rather than weighting by grades or disciplines.

Missing values

Variation in the coverage of PISA and TIMSS/PIRLS means that school and classroom change effect sizes are therefore not available for all education systems across all of the questions asked. Furthermore, data are missing when certain questions (or questionnaires) were omitted at the national level at certain points in time. This is not an issue when reporting responses to a single question, but it does pose a potential problem when seeking to combine information across questions. In order to analyse as many countries as possible whilst keeping a wide range of questions in the analysis, it has been necessary to manage the missing data through a combination of deletion and estimations processes.

An iterative process has been used to manage observations (education systems) and variables (questions) with missing data, and some systems/countries and questions have had to be omitted in the construction of an index:

1. Education systems that had effect size data for fewer than 20% of the potential question set were excluded.

2. Following this, questions with high proportions of missing data were dropped. Specifically, those questions with effect size missing for more than 50% of the remaining database were excluded.

3. Education systems with less than 60% valid data on the remaining questions were then excluded from the analysis.

Following the deletion process, some of the remaining education systems still had portions of missing data. Data was typically missing when education system had not participated in one of the surveys. As information for a whole dataset was missing, it was not possible to undertake an imputation at the indicator level. However, it was possible to estimate the effect of a missing dataset on the final index.

The estimation process uses information from countries having all the data points in order to estimate the impact of including a dataset on the index computation. We use this information to adjust the indices of countries missing one dataset. The process goes as follows:

• For education systems with all the information available, a subset of indices was computed, each one of them excluding one of the datasets from the index computation ($I_{-A}$). The index including all the data was also calculated (I). For instance if other countries missed PISA, countries with all the information available will have an index excluding PISA ( $I_{-A}$ ) and one with PISA (I).

• The ratios of complete index to sub-indices were calculated for each country ($I/I_{-A})$.

• The cross-country mean ratio of full index to every sub-index was computed, giving us a dataset factor effect for each potential missing data source. (${DF}_A=Mean(I/I_{-A}))$

• Finally, countries missing data from one source (A) had their index computed with all their information available ($I_{m\left(A\right)}$). This index is then corrected by multiplying it by the dataset factor of the corresponding missing database, giving us the final composite index ($I=I_{m\left(A\right)}*{DF}_A$).

Criteria for including questions in the indices

Highly correlated questions may unduly influence an index that seeks to explore the extent to which change occurs over different aspects of education, particularly given the existence of missing data. For this reason, where question effect sizes are highly correlated [0.6 or more using Person’s r] and the wording of the questions is the same across different grades or subjects, only the question with the highest absolute effect size at the OECD level has been included in the classroom, school and overall indices. Where the effect sizes of different questions within a module are correlated, but the wording differs, both questions have been included as separate items within the indicator. Questions have also been retained for indices at subject and grade level where the possibility of correlation is not a problem.

Table B.2. Number of available questions – Main indices

Countries and regions	Overall Index	Primary Index	Secondary Index	Maths Index	Science Index	Reading Index
Australia	125	62	66	28	49	-
Austria	-	59	-	-	-	33
Belgium (Fl.)	-	-	-	-	-	33
Canada (Alberta)	-	59	-	-	-	-
Canada (Ontario)	110*	62	51*	28	43*	33
Canada (Quebec)	110*	62	51*	28	43*	33
Czech Republic	-	62	-	-	-	-
Denmark	-	62	-	-	-	33
France	-	-	-	-	-	33
Germany	-	62	-	-	-	33
Hungary	125	62	66	28	49	33
Israel	100***	-	66	-	-	33
Italy	125	62	63	28	49	33
Japan	91**	-	66	28	49	-
Korea	91**	-	66	-	48	-
Latvia	-	-	-	-	-	33
Lithuania	125	62	66	28	49	33
Netherlands	-	62	-	-	-	33
New Zealand	125	61	-	28	49	33
Norway	125	62	61	28	49	33
Poland	-	59	-	-	-	33
Portugal	-	59	-	-	-	33
Slovak Republic	-	62	-	-	-	33
Slovenia	125	62	66	28	49	33
Spain	-	59	-	-	-	33
Sweden	125	62	66	28	49	33
Turkey	-	-	61	28	48	-
U.K. (England)	110*	62	51*	28	43*	33
United States	125	62	61	28	49	33
U.S. (Massachusetts)	-	-	51	-	-	-
U.S. (Minnesota)	-	-	51	-	-	-
Hong Kong, China	125	62	66	28	49	33
Indonesia	100***	-	61	-	-	33
Russian Federation	125	62	66	28	49	33
Singapore	125	62	56	28	43*	33
South Africa	-	-	51*	-	-	28

Note: * Missing PISA data- database effect estimation applied; ** Missing PIRLS data - database effect estimation applied; ***Missing TIMSS 4th grade data- database effect estimation applied.

Source: PISA, TIMSS and PIRLS databases.

Table B.3. Number of available questions – ICT indices

Countries and regions	ICT availability	ICT use
Australia	7	18
Austria	5	11
Canada (Ontario)	5	15
Canada (Quebec)	5	15
Chile	6	15
Czech Republic	5	11
Denmark	5	11
Finland	5	11
Hungary	7	18
Ireland	5	11
Italy	7	18
Japan	6	15
Korea	6	15
Lithuania	7	18
Netherlands	5	11
New Zealand	7	18
Norway	5	15
Poland	5	11
Portugal	5	11
Slovak Republic	5	11
Slovenia	7	18
Spain	5	11
Sweden	7	18
Turkey	4	-
U.K. (England)	5	15
United States	5	15
Hong Kong, China	7	18
Russian Federation	7	18
Singapore	7	18

Source: PISA, TIMSS and PIRLS databases.

Table B.4. Number of available questions – Thematic indices

Countries and regions	Assessment Index	Homework Index	Active Learning in Science Index	High order skills index	Knowledge transmission and acquisition index	Learning resource availability index	Rote learning Index	Professional Development- Teacher training index	Professional Development- Peer learning index	External relations and HRM index	Other practices index
Australia	10	8	8	16	8	-	9	16	9	8	-
Austria	-	-	-	-	5	11	-	-	-	-	7
Canada (Alberta)	-	-	-	-	5	-	-	-	-	-	-
Canada (Ontario)	10	-	6	11	8	12	8	16	9	6	-
Canada (Quebec)	10	8	6	11	8	12	8	16	9	6	-
Chile	-	-	-	-	-	-	8	16	-	-	-
Czech Republic	-	-	5	13	5	11	-	-	-	-	7
Denmark	-	-	5	13	5	11	-	-	-	-	7
Finland	-	-	-	-	-	11	-	-	-	-	7
France	-	-	-	11	-	-	-	-	-	-	-
Germany	-	-	5	13	5	-	-	-	-	-	7
Hungary	10	7	8	16	8	14	9	16	9	8	7
Ireland	-	-	5	-	-	-	-	-	-	-	-
Israel	10	8	-	14	5	11	6	-	6	7	7
Italy	10	8	8	16	8	14	9	16	9	-	7
Japan	-	8	8	10	6	-	8	16	9	7	-
Korea	-	8	8	10	6	-	8	16	9	7	-
Latvia	-	-	-	11	-	-	-	-	-	-	7
Lithuania	10	7	8	16	8	14	9	16	9	8	7
Netherlands	-	-	5	13	5	11	-	-	-	-	7
New Zealand	9	-	8	-	8	14	9	16	9	8	7
Norway	10	8	8	16	8	12	9	16	9	8	6
Poland	-	-	5	13	5	11	-	-	-	-	7
Portugal	-	-	-	-	5	11	-	-	-	-	7
Slovak Republic	-	-	5	13	5	11	-	-	-	-	7
Slovenia	10	7	8	16	8	14	9	16	9	8	7
Spain	-	-	5	13	5	11	-	-	-	-	7
Sweden	10	7	8	16	8	14	9	16	9	8	7
Turkey	-	8	-	10	6	-	8	16	9	7	-
U.K. (England)	10	8	6	11	8	12	8	16	9	6	-
United States	10	8	8	16	8	12	9	16	9	8	6
U.S. (Massachusetts)	-	8	-	-	-	-	-	-	-	-	-
U.S. (Minnesota)	-	8	-	-	-	-	-	-	-	-	-
Hong Kong, China	10	8	8	16	8	14	9	16	9	8	7
Indonesia	10	7	-	14	5	-	6	-	6	7	6
Russian Federation	10	7	8	16	8	14	9	16	9	8	7
Singapore	10	8	6	11	8	14	8	16	9	6	-
South Africa	10	-	-	-	5	-	-	-	6	-	-

Developing and reporting the indices

The indices developed are intended to show the extent of change or innovation in one country when compared with other countries. They can be used to rank countries according to their relative levels of innovation across levels, disciplines and in more specific educational practices.

Discipline, education level and overall innovation indices for each country =100 x (weighted average of absolute effect sizes)

ICT and thematic innovation indices do not accord any weight to values, therefore the composite indices for each country= 100 x (unweighted average of absolute effect sizes)

The number of questions included depends on whether data exist in PISA and/or TIMSS/PIRLS and therefore differs across education systems. It also clearly depends on the indicator itself: up to 33 questions are used in the reading innovation index compared to 49 in science for example. The number of questions included across ICT and thematic indices also varies considerably.

It is possible for the absolute effect sizes to take a value that is greater than one; however in practice they mostly range between 0 and 1; the indices can therefore take values from 0 to positive infinity but in practice they never cross 100 for the broad composite indices. For the ICT and specific composite indices the index itself has the same range as the broader ones but their decomposition shows the negative and positive contributions as well.

Cautions

Question inclusion

The indices combine information from a large and diverse pool of questions asked on different surveys. On the assumption that each question can provide additional information about the extent of change and innovation in an education system, the process employed to develop the indices has drawn on as many of the questions as possible and their inclusion has been determined by the availability of valid data. However, a more theoretical approach focusing on the most relevant questions, or a statistical approach to data reduction may provide different results.

Education system coverage

The indices provide some information about a subset of the education systems discussed in the previous chapters. This subset has been determined by the availability of data. It may be the case that other systems sit at the extremes of the ranking. It should be noted that the inclusion or removal of education systems would also impact on the estimation of missing values. Although it gives a robust synthesis of change covered by our change indicators, the country ranking should not be over-interpreted.

OECD average

The OECD average is computed for all the education systems for which data are available for all years concerned. In calculating the weights of regions that do not correspond to an entire OECD member the following procedure has been followed. Education systems that are part of a country for which the overall data is available are not considered – this being the case for the different states in the United States. Conversely, education systems that do not have a figure for the whole country they belong to have been given weight equal to 1- this being the case, for example of Ontario and Flanders (Belgium) among others.

Time periods

The effect size of the change in responses to a particular question is typically calculated across the same two points of time for each country but the two points in time may differ by question. The indices therefore show a tendency to change or innovate across slightly different time periods, rather than the extent of change over a specific time period.

Interpreting the findings

The indices reported help the reader to consider the benefits of such a composite innovation indicator based on change measures, but may not provide a fully accurate representation of the level of change and innovation within a country. Whilst the indicator is based on many questions and observations, the missing data imputation and correction which were needed to construct the innovation indices invites the reader to be cautious. The innovation indices are mere indicators of innovation, and small differences in levels are almost certainly not meaningful.

A higher score on the indicator suggests that an education system is characterised by more change than other systems. However, there is currently no theory that could be applied to describe the different levels in terms of adequacy of innovation. Similarly, the scale does not provide information about what is necessary to move from one point to another. Additional work could be undertaken to develop qualitative descriptions of different points on the scale, but this should be preceded by improved data collection.

Component indicators of the ICT based and thematic composite indices

Table B.5. Indicators included in the composite index of innovation in computer availability in schools

Practice	Grade
Availability of computers (including tablets) to use during maths lessons	Primary and secondary
Availability of computers (including tablets) to use during science lessons	Primary and secondary
Availability of computers (including tablets) to use during reading lessons	Primary
Availability of desktop computers for use at school	Secondary
Availability of portable laptops or notebooks for use at school	Secondary

Table B.6. Indicators included in the composite index of innovation in ICT use in schools

Practice	Grade
Practising skills and procedures on computers in maths	Primary and secondary
Practising skills and procedures on computers in science	Primary and secondary
Study natural phenomena through simulations on computers in science	Primary and secondary
Processing and analysing data on computers in maths	Secondary
Processing and analysing data on computers in science	Secondary
Students using computers to write stories and texts in reading	Primary
Using computers to look for information in reading	Primary
Frequency of use of computer or a tablet at school	Primary
Use of digital devices for foreign language learning or mathematics	Secondary
Using digital devices for playing simulations at school	Secondary
Use of school computers for group work and communication with other students	Secondary
Teacher participation in a programme integrate information technology into mathematics	Primary and secondary
Teacher participation in a programme to integrate information technology into science	Primary and secondary

Table B.7. Indicators included in the composite index of innovation in active learning practices in science education

Practice	Grade
Students conducting scientific experiments and investigations in science	Primary and secondary
Study natural phenomena through simulations on computers in science	Primary and secondary
Students doing practical experiments in laboratories	Secondary
Students designing and planning science experiments	Primary and secondary
Scope for students to design their own experiments	Secondary

Table B.8. Indicators included in the composite index of innovation in homework practices

Practice	Grade
Frequency of homework in maths	Secondary
Frequency of homework in science	Secondary
Monitoring homework completion in maths	Secondary
Monitoring homework completion in science	Secondary
Students correcting their own homework in maths	Secondary
Students correcting their own homework in science	Secondary
Discussion of homework in class in maths	Secondary
Discussion of homework in class in science	Secondary

Table B.9. Indicators included in the composite index of innovation in assessment practices

Practice	Grade
Frequency of correction of assignment and feedback in maths	Secondary
Frequency of correction of assignment and feedback in science	Secondary
Importance of classroom tests in maths	Secondary
Importance of classroom tests in science	Secondary
Importance of national or regional achievement tests in maths	Secondary
Importance of national or regional achievement tests in science	Secondary
Written tests on reading	Primary
Emphasis on classroom test in reading	Primary
Emphasis on national or regional tests in reading	Primary
Oral examination and summarising of read text in reading	Primary

Table B.10. Indicators included in the composite index of innovation in fostering higher order skills

Practice	Grade
Students explaining their understanding of text in reading	Primary
Students explaining style and structure of text in reading	Primary
Students drawing inferences and generalisations from text in reading	Primary
Students identifying main ideas of text in reading	Primary
Students comparing read text with their own experiences in reading	Primary
Opportunities for students to explain their ideas in reading	Secondary
Making predictions about what will happen next in read text in reading	Primary
Observing and describing natural phenomena in Primary	Primary and secondary
Students designing and planning science experiments	Primary and secondary
Students drawing conclusions from an experiment in science	Secondary
Teacher explaining relevance of broad science topics	Secondary
Teacher explaining practical application of school science topics	Secondary
Scope for students to design their own experiments	Secondary
Solving problems with no obvious method of solution in maths	Secondary

Table B.11. Indicators included in the composite index of innovation in independent knowledge acquisition

Practice	Grade
Reading non-fiction books in reading	Primary
Reading textbooks and resource materials in science	Primary and secondary
Using computers to look for information in reading	Primary
Using computers to look up for ideas and information in maths	Primary and secondary
Using computers to look up for ideas and information in science	Primary and secondary

Table B.12. Indicators included in the composite index of innovation in rote learning practices

Practice	Grade
Memorising rules, procedures and facts as a pedagogical technique in maths	Primary and secondary
Memorising rules, procedures and facts as a pedagogical technique in science	Primary and secondary
Watching teachers demonstrate an experiment in science	Primary and secondary
Use scientific formulas and laws to solve routine problems	Secondary
Students doing practical experiments in laboratories	Secondary
Teaching new vocabulary systematically in reading	Primary

Table B.13. Indicators included in the composite index of innovation in formal teacher training

Practice	Grade
Teacher participation in mathematics content	Primary and secondary
Teacher participation in science content	Primary and secondary
Teacher participation in a program on maths pedagogy/instruction	Primary and secondary
Teacher participation in a program on science pedagogy/instruction	Primary and secondary
Teacher participation in a program on maths curriculum	Primary and secondary
Teacher participation in a program on science curriculum	Primary and secondary
Teacher participation in a program on mathematics assessments	Primary and secondary
Teacher participation in a program on science assessments	Primary and secondary

Table B.14. Indicators included in the composite index of innovation in teachers’ peer learning

Practice	Grade
Collaborating in planning and preparing instructional material	Primary and secondary
Visiting another classroom to learn more about teaching	Primary and secondary
Discussing how to teach a particular topic	Primary and secondary

Note: In secondary education, these questions were asked to both maths and science teachers; in primary education, no distinction was made on the basis of disciplines.

Table B.15. Indicators included in the composite index of innovation in availability of school learning resources

Practice	Grade
Availability of a school library for students	Primary
Availability of a library or a reading corner in the classroom	Primary
Allowing students to borrow books from the classroom library	Primary
Students visiting a library other than their classroom library	Primary
Availability of desktop computers for use at school	Secondary
Availability of portable laptops or notebooks for use at school	Secondary
Frequency of use of computer or a tablet at school	Primary
Availability of computers (including tablets) to use during reading lessons	Primary
Availability of computers (including tablets) to use during maths lessons	Primary and secondary
Availability of computers (including tablets) to use during science lessons	Primary and secondary
Availability of a science laboratory for students	Primary and secondary

Table B.16. Indicators included in the composite index of innovation in school external relations and human resource management (HRM) practices

Practice	Grade
Parental help in reading	Primary
Incentives to recruit or retain maths teachers	Secondary
Incentives to recruit or retain science teachers	Secondary
Incentives to recruit or retain teachers other than maths and science	Secondary
Degree of parental involvement in school activities	Primary and secondary
Public posting of school achievement data (e.g. in the media)	Secondary
Tracking achievement data over time by an administrative authority	Secondary

Table B.17. Indicators included in the composite index of innovation in other miscellaneous educational practices

Practice	Grade
Teaching strategies for decoding sounds and words in reading	Primary
Same class-ability groups in reading classes	Primary
Mixed-ability groups in reading classes	Primary
Reading individually with students in reading	Primary
Use of school computers for group work and communication with other students	Secondary
Student grouping by ability into different classes	Secondary
Student grouping by ability within classes	Secondary