The 2022 OECD Green Budgeting Index methodology section covers the following areas:
Structure of the 2022 OECD Green Budgeting Index
Coding the data
Weighting and aggregation
Statistical analysis
Limitations of composite indexes
Green Budgeting in OECD Countries 2024
2. 2022 OECD Green Budgeting Index methodology
The methodology used to build the 2022 OECD Green Budgeting Index (Figure 1.4) draws on the recommendations and steps identified in the Handbook on Constructing Composite Indicators: Methodology and User Guide (OECD/EU/JRC, 2008[12]). The 2022 OECD Green Budgeting Index is based on the four building blocks in the Green Budgeting Framework: 1) institutional arrangements, 2) methods and tools, 3) accountability and transparency, and 4) the enabling environment in budgeting (OECD, 2020[1]), each with an equal weight (25%).
The index ranges from 0 (not implementing) to 1 (high level of green budgeting practices). Country green budgeting practice scores were determined by adding together the weighted scores of each dimension, varying from 0 to 1. The structure, indicators and weightings of the index were selected by the OECD based on the relevance to the building blocks and have been reviewed by country delegates to the OECD Paris Collaboration on Green Budgeting. The various statistical analyses that were applied to the methodology were to ensure validity and reliability of the composite index.
The 2022 OECD Green Budgeting Index was built from data collected in the 2022 OECD Survey on Green Budgeting, encompassing responses from 36 OECD countries1 and referring only to central/federal governments. Respondents were predominantly senior budget officials within central budget authorities. Responses draw upon self-reporting representing the country’s own assessment of current practices and procedures as of 30 June 2022. A data cleaning process took place from September to December 2022. The process checked for internal and external consistency in the survey responses. During the review process, countries were invited to clarify and verify any substantial changes from the earlier data, potential inconsistencies and outliers.
Although the 2022 OECD Green Budgeting Index was developed in co-operation with OECD countries and is based on best practices and/or theory, the Index structure, indicators and weights may be further refined over time to ensure they continue to capture the changing practices and elements that underpin a comprehensive green budgeting approach.
While the 2022 OECD Green Budgeting Index allows for cross-country comparison, it is neither context specific nor does it purport to measure the overall quality of green budgeting in each country or how well a country’s selected approach to green budgeting operates. Rather, the index presented is descriptive in nature and allows countries to examine the results at the building block level to identify areas for further improvement.
Structure of the 2022 OECD Green Budgeting Index
The Index is assessing the implementation of green budgeting across OECD countries by quantifying the measures that are in place across each of the four building blocks in the Green Budgeting Framework: 1) institutional arrangements, 2) methods and tools, 3) accountability and transparency, and 4) the enabling environment in budgeting (OECD, 2020[1]). Each of the Index’s four building blocks carries an equal weight (25%) (Figure 2.1).
Figure 2.1. 2022 OECD Green Budgeting Index: Building blocks, variables and weights used
Source: OECD authors’ elaboration.
Coding the data
Each of the answer options in the 2022 OECD Survey on Green Budgeting were translated into quantitative indicators, varying from 0 to 1, with the value of one representing the highest value of the indicator and the value of zero representing the lowest. Although all indicators respect the [0,1] range, the method for assigning values to responses for questions vary by the type and structure of the question. For binary-type questions (yes/no), the value of 1 represents the existence (yes) of the indicator and the value of 0 represents the nonexistence (no). For ordinal-type questions, multiple answer options are possible and values are assigned according to their significance or importance, e.g. [0, 0.5 or 1]. For some questions with multiple response options, several options were attributed the maximum value [1] as they are considered equally important. Once the maximum value is obtained, additional responses are not taken into account.
The 2022 OECD Green Budgeting Index refers to measures under implementation as of 30 June 2022. If a measure was under consideration or being developed at the time, but not yet implemented, it was assigned a value of zero.
Weighting and aggregation
All building blocks were assigned an equal weight (25%). Variables within a building block were weighted depending on the i) the number of indicators making up each variable and ii) the different relevance of each variable. The sum of the weights assigned to variables in each building block is 1. Given the number of indicators per building block, each indicator has a final combined weight of 0.014 under the building block “Institutional arrangements”, 0.023 under the building block “Methods and tools”, 0.023 under the building block “Accountability and transparency” and 0.063 under the building block “Enabling environment”. A linear aggregation method was applied to first aggregate the indicators into variables, the variables into building blocks and then the building blocks into a composite indicator. To obtain a country score, the weighted scores for each indicator were totalled to obtain an overall index score for each country from 0 to 1. The weightings for the building blocks, variables and indicators were selected by OECD based on the relevance to the OECD Green Budgeting Framework and have been reviewed by county delegates of the OECD Paris Collaborative on Green Budgeting. The weights assigned to each building block and building block variables are displayed in Figure 2.1.
The tables below (Tables 2.1 to 2.4) present the structure and weight of each of the four building blocks composing the 2022 OECD Green Budgeting Index, including the variables and indicators and show how the 2022 OECD Survey on Green Budgeting questions were used across each building block.
Institutional arrangements
Institutional arrangements look at the way governments have set the basis for green budgeting. The 2022 OECD Green Budgeting Index contained ten questions relative to the legal basis for green budgeting, national environmental strategies, net-zero strategies, the objectives adopted by countries, and the policymaking and budgeting process (Table 2.1).
Table 2.1. Methodology for institutional arrangements
1. Including weights for variable and building block.
2. For non pre-defined answer categories, additional data validation was performed: to be validated for the index and to assure an equal treatment across answer categories and countries, the explanations provided had to match the expectations of the underlying survey question.
Methods and tools
The methods and tools refer to concrete instruments to implement green budgeting. The 2022 OECD Green Budgeting Index contained nine questions relative to the medium-term perspective, carbon related tools, impact appraisal, budget classification and review, accounting, and green finance (Table 2.2).
Table 2.2. Methodology for methods and tools
1. Including weights for variable and building block.
Accountability and transparency
The building block on the implementation of green budgeting refers to accountability and transparency. The 2022 OECD Green Budgeting Index contains eleven questions relative to Monitoring and reporting, oversight, and budget transparency (Table 2.3).
Table 2.3. Methodology for accountability and transparency
1. Including weights for variable and building block.
2. For non pre-defined answer categories, additional data validation was performed: to be validated for the index and to assure an equal treatment across answer categories and countries, the explanations provided had to match the expectations of the underlying survey question.
Enabling environment in budgeting
The fourth building block refers to the enabling environment in budgeting. The 2022 OECD Green Budgeting Index contained four questions on performance and programme budgeting, co-ordination mechanisms and capacity building (Table 2.4).
Table 2.4. Methodology for enabling environment in green budgeting
1. Including weights for variable and building block.
Statistical analysis
Sensitivity testing
The sensitivity analysis in this report is based on Monte Carlo simulations. The purpose of the simulations was to test the sensitivity of the weights selected for each of the four building blocks. The results of the simulation are to show which country would have been presented differently if the building blocks were based on a different combination of weights.
Monte Carlo simulations generate an interval of values which reflect the range of possible scores for indicators in each country, depending on five thousand different weighting scenarios. The results for the 2022 OECD Green Budgeting Index are presented in Figure 2.2. Vertical lines represent the 90% confidence intervals derived from the random weights analysis.
The size of the interval varies depending on a country’s green budget data profile (mixed or rather even). Countries with an even answer profile, that is, a profile gave similar answers for the majority of questions, will fall into a narrow interval, while countries with an uneven profile will fall into a broader interval (Freudenberg, 2003[13]).
The results from the sensitivity analyses at pillar levels, the four building blocks, show that, for the largest part of the countries analysed, total scores are not very sensitive to the choice of values given to the categories. The total scores of Canada, Greece, and Spain appear to be more sensitive to the weightings applied. It is important to note that not all the combinations of weights used to create these intervals can be viewed as realistic outcomes (Arndt et al., 2015[14]).
One possible practical application of the analysis of the upper and lower bounds of these intervals is to consider how countries can be grouped together with a strong degree of confidence (Arndt et al., 2015[14]).
The group of countries with black diamonds on the left-hand side of Figure 2.2 have a total score for their composite indicator that is above the average (expressed as a horizontal dotted average line), and have scores above the average for 90% of random combinations (this is the case for: Denmark, France, Ireland, New Zealand, Norway, Sweden, and the United Kingdom). Conversely, on the right-hand side of the graph countries that are marked with black diamonds score below the average for 90% of random combinations. These two groups of countries can therefore be said to have indicator values which are significantly different from each other independent of the weighting scheme.
Cronbach alpha testing (scale reliability coefficient)
Cronbach’s Alpha is a coefficient of reliability based on the correlations between indicators. This statistic is generally used to investigate the degree of correlation among a set of variables and to check the internal reliability of items in a model or survey. A Cronbach’s alpha coefficient equal to zero means that the variables are independent (e.g. the selection is not correlated and therefore is statistically not relevant), while a coefficient equal to one means that the variables are perfectly correlated. In general, a coefficient of above 0.7 is considered to be an acceptable indication that the variables are measuring the same underlying construct. The value recorded of the coefficient is of 0.75 which indicates that the variables are measuring the same underlying construct.
Figure 2.2. 2022 OECD Green Budgeting Index: Sensitivity analysis
Note: Figure presents the sensitivity of the index to various weighting assumptions (results from the Monte Carlo simulation where 5 000 different weights were assigned). Diamonds represent the indicator scores, and vertical lines represent the 90% confidence intervals derived from the random weights analysis. Index comprised between 0 (no use) and 1 (high use). Cronbach’s alpha: 0.75 (computed with SPSS). A Cronbach’s alpha close to or above 0.7 indicates a high degree of correlation among a set of variables.
Source: OECD author’s analysis.
Limitations of composite indexes
Composite indexes are useful to summarise large amounts of discrete, qualitative information and “are much easier to interpret than trying to find a common trend in many separate indicators” (Nardo M, 2005[15]). However, these indexes are easily and often misinterpreted by users due to a lack of transparency as to how they are generated and the resulting difficulty to truly unpack what they are actually measuring. The 2022 OECD Green Budgeting Index, while providing cross-country comparison, is not context-specific, nor can fully capture the complex realities of the quality, use and impact of green budgeting approaches. This comparison should hence not be seen as a measurement of quality or a ranking. It shows that countries have adopted multi-initiative approaches to green budgeting by using each of the four building blocks. It should also be noted that countries with the same or similar index scores may have very different profiles (e.g. more balanced scores or an uneven score across building blocks). Examining the results at the building block level is therefore crucial to identify areas for further improvement.
Composite indexes should be seen as a means of initiating discussion and stimulating public interest (OECD/EU/JRC, 2008[12]). To help countries interpretate the findings and to provide a more detailed analysis of the strengths and weaknesses of their green budgeting approach, the results of the 2022 OECD Green Budgeting Index should be complemented with in-depth country assessments. These country assessments allow to provide a more detailed analysis of the quality of country practices in the context of diverse and country-specific governance structures, administrative cultures, institutional set-ups and priorities.
Note
← 1. Data for Costa Rica and Slovenia are not available.