Government at a Glance 2023

Annex A. Methodology for composite indexes on green budgeting, gender budgeting, and communications by independent fiscal institutions

General background

The narrowly defined composite indexes described here represent the best way of summarising discrete, qualitative information. “Composite indexes are much easier to interpret than trying to find a common trend in many separate indicators” (Nardo et al., 2005[1]). However, their development and use can be controversial. These indexes are easily and often misinterpreted by users due to a lack of transparency about how they are generated, which makes it difficult to truly unpack what they are actually measuring.

The OECD has taken several steps to avoid or address common problems associated with composite indexes. The composites presented in this publication were developed using the steps identified in the Handbook on Constructing Composite Indicators (OECD/European Union/EC-JRC, 2008[2]) that are necessary for the meaningful construction of composite or synthetic indexes.

Each composite index is based on a theoretical framework representing an agreed concept in the area it covers. The variables comprising the indexes are chosen based on their relevance to the concept. Each index is constructed in close collaboration with the relevant OECD expert groups, which advised on the variables and the weighting schemes to use for the composite.

A number of statistical analyses were also conducted to ensure the validity and reliability of the composite indexes. The survey questions used to create the indexes are the same across countries, to ensure indexes are comparable. In order to eliminate scale effects, all indicators and variables were normalised between “0” and “1” for comparability. To build the composites, all indicators were aggregated using a linear method. The index scores were determined by adding together the weighted scores of each indicator. Statistical tools (i.e. Cronbach’s alpha) were also employed to establish the degree of correlation among a set of variables comprised in each index and to check their internal reliability. This implies that all of the variables comprised in each index have intrinsic value but are also interlinked and capture the same underlying concept. Finally, sensitivity analysis using Monte Carlo simulations was carried out to establish the robustness of the index scores to different weighting options.

2022 Green Budgeting Index

Green budgeting refers to integrating climate and environmental considerations into the budgetary process. It involves the use of special initiatives, processes and analytical tools with a view to promoting policies and investments that help achieve climate and environmental goals and commitments. The 2022 OECD Green Budgeting Index is designed around the four building blocks in the OECD Green Budgeting Framework: 1) institutional arrangements; 2) methods and tools; 3) accountability and transparency; and 4) the enabling environment in budgeting (OECD, 2020[3]). Each building block is weighted equally (25%).

Data used for the construction of the 2022 OECD Green Budgeting Index are derived from the 2022 OECD Survey on Green Budgeting. Survey respondents were predominantly budget officials within central budget authorities in OECD countries. The variables and weights comprised in the index were selected based on their relevance to the concept by a group of experts within the OECD and in consultation with country delegates to the OECD Paris Collaborative on Green Budgeting. While the 2022 OECD Green Budgeting Index allows for cross-country comparison, it is not context specific, nor can it fully capture the complex realities of the quality, use and impact of green budgeting approaches. This comparison should hence not be seen as a measurement of quality or a ranking. It shows that countries have adopted multi-initiative approaches to green budgeting by using each of the four building blocks (OECD, forthcoming[4]).

Variables and weights

The components used in the construction of this index, and the weights given to each, are indicated in the figure below.

Figure ‎A.1. 2022 OECD Green Budgeting Index: Variables and weights used

Evolution over time

The 2022 Green Budgeting Index is an update of the 2021 OECD Green Budgeting Index reflecting changing practices and new items in response to country requests. The objective of the update was to strengthen the composition of the four building blocks. New additions include developments that are affecting the way climate and environmental considerations are being integrated into government budgets. Examples include net-zero emissions strategies, medium-term budgeting tools, oversight mechanisms and co-ordination across levels of government. The two versions of the index are therefore not comparable.

Although the composite index was developed in co-operation with OECD countries and is based on best practice and/or theory, both the variables and the weights comprised in the composite may be further refined over time to ensure they continue to capture the changing practices and elements that underpin a comprehensive green budgeting framework.

Statistical analyses

Sensitivity analysis was carried out to establish the robustness of the indicators to different weighting options through Monte Carlo simulations. The results of the sensitivity analysis at building block level for the 2022 OECD Green Budgeting Index show that, for most of the countries analysed, total scores are not very sensitive to the choice of values given to the categories. Cronbach’s alpha coefficient is equal to 0.75, indicating that the building blocks are measuring the same underlying construct (OECD, forthcoming[4]).

2022 OECD Gender Budgeting Index

Gender budgeting refers to the integration of a clear gender perspective within the overall context of the budgetary process. It involves the use of special processes and analytical tools with a view to promoting policies and investments that help achieve gender equality goals. To strengthen the implementation of gender budgeting, the OECD has recently updated its Framework on Gender Budgeting, now capturing five building blocks: 1) institutional and strategic arrangements; 2) enabling environment; 3) methods and tools; 4) accountability and transparency; and 5) impact (Gatt Rapa and Nicol, 2023, forthcoming[5]). The 2022 OECD Gender Budgeting Index is designed around these five building blocks. Each building block is weighted equally (20%).

Data used for the construction of the 2022 OECD Gender Budgeting Index are derived from the 2022 OECD Survey on Gender Budgeting. Survey respondents were predominantly senior budget officials within central budget authorities in OECD countries. The variables and weights comprised in the index were selected based on their relevance to the building block by a group of experts within the OECD and in consultation with country delegates to the Senior Budget Officials (SBO) Network on Gender Budgeting. While the 2022 OECD Gender Budgeting Index allows for cross-country comparison, it is not context specific, nor can it fully capture the complex realities of the quality, use and impact of gender budgeting approaches. This comparison should hence not be seen as a measurement of quality or a ranking. It shows that countries have adopted multi-initiative approaches to gender budgeting by using each of the five building blocks (Gatt Rapa and Nicol, 2023, forthcoming[6]).

Variables and weights

The components used in the construction of this index, and the weights given to each, are indicated in the figure below.

Figure ‎A.2. 2022 OECD Gender Budgeting Index: Variables and weights used

Evolution over time

The 2022 Gender Budgeting Index is an update of the 2018 OECD First Pass at an Index on Gender Budgeting (OECD, 2019[7]) reflecting changing practices and new items in response to country requests. For example, two new pillars have been added to the index: 1) accountability and transparency; and 2) impact. The two versions of the index are therefore not comparable.

Although the composite index was developed in co-operation with OECD countries and is based on best practices and/or theory, both the variables and the weights comprised in the composite may be further refined over time to ensure they continue to capture the changing practices and elements that underpin a comprehensive gender budgeting framework.

Statistical analyses

Sensitivity analysis was carried out to establish the robustness of the indicators to different weighting options through Monte Carlo simulations. The results from the sensitivity analysis at the building block level for the 2022 OECD Gender Budgeting Index show that, for the majority of countries analysed, total scores are not very sensitive to the choice of values given to the categories. Cronbach’s alpha coefficient is equal to 0.84, indicating that the building blocks are measuring the same underlying construct (Gatt Rapa and Nicol, 2023, forthcoming[6]).

2021 OECD Independent Fiscal Institutions’ (IFI) Communications Index

The OECD Principles for Independent Fiscal Institutions call on IFIs to develop effective communication channels from the outset, as these are key to achieving impact (OECD, 2014[8]). The 2021 OECD IFI Communications Index provides a measure of communication practices at the national level across OECD IFIs. The index considers institutional arrangements in relation to three different aspects of communications, each with different weights: 1) disseminating research (52%); 2) promoting research (32%); and 3) tracking influence (16%).

Data used for the construction of the 2021 OECD IFI Communications Index are derived from the OECD Independent Fiscal Institutions Database (OECD, 2021[9]). The data were collected via desk research and then verified and validated by relevant senior officials in the OECD’s Working Party of Parliamentary Budget Officials and Independent Fiscal Institutions. The variables and weights comprised in the index were selected based on their relevance to the concept by a group of experts within the OECD and in consultation with delegates to the Working Party of Parliamentary Budget Officials and Independent Fiscal Institutions. The weights reflect the relative importance given to the variables contributing to the three dimensions.

Variables and weights

The components used in the construction of this index, and the weights given to each, are indicated in the figure below.

Figure ‎A.3. 2021 OECD IFI Communications Index: Variables and weights used

A detailed explanation on the components of the 2021 OECD IFI Communications Index is available online at https://www.oecd.org/gov/govataglance.htm, including the variables, answer options, scores and weights used to construct the composite index, as well as the statistical analysis carried out.

Statistical analyses

Sensitivity analysis was carried out to establish the robustness of the index scores to different weighting options through Monte Carlo simulations. The results from the sensitivity analysis at variables level for the 2021 OECD IFI Communications Index show that, for the majority of the countries analysed, total scores are not very sensitive to the choice of values given to the categories. Cronbach’s alpha coefficient is equal to 0.79, indicating that the variables are measuring the same underlying construct.

References

[6] Gatt Rapa, K. and S. Nicol (2023, forthcoming), Gender Budgeting in OECD Countries 2023, OECD Publishing, Paris.

[5] Gatt Rapa, K. and S. Nicol (2023, forthcoming), OECD Framework for Gender Budgeting, OECD Publishing, Paris.

[1] Nardo, M. et al. (2005), Tools for Composite Indicators Building, European Commission, https://publications.jrc.ec.europa.eu/repository/handle/JRC31473.

[9] OECD (2021), Independent Fiscal Institutions Database (Version 2.0), OECD, Paris, http://www.oecd.org/gov/budgeting/OECD-Independent-Fiscal-Institutions-Database.xlsx.

[3] OECD (2020), OECD Green Budgeting Framework: Highlights, OECD, Paris, https://www.oecd.org/environment/green-budgeting/OECD-Green-Budgeting-Framework-Highlights.pdf.

[7] OECD (2019), Government at a Glance 2019, OECD Publishing, Paris, https://doi.org/10.1787/8ccf5c38-en.

[8] OECD (2014), Recommendation of the Council on Principles for Independent Fiscal Institutions, OECD, Paris, https://www.oecd.org/gov/budgeting/OECD-Recommendation-on-Principles-for-Independent-Fiscal-Institutions.pdf.

[4] OECD (forthcoming), Green budgeting in OECD Countries - 2022 OECD Green Budgeting Survey Results, OECD Publishing, Paris.

[2] OECD/European Union/EC-JRC (2008), Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Publishing, Paris, https://doi.org/10.1787/9789264043466-en.