Government at a Glance 2023

Annex B. Methodology for the infrastructure governance indicators

The OECD Infrastructure Governance Indicators (IGIs) are intended to support and monitor the implementation of the OECD Recommendation on the Governance of Infrastructure (hereafter “the Recommendation”), adopted by the OECD Council on 17 July 2020 (OECD, 2020[1]). The Recommendation is based on 10 pillars that relate to how governments plan, prioritise, fund, budget, deliver, operate and monitor infrastructure assets. It presents a whole-of-government approach, covering the entire life cycle of infrastructure projects and placing special emphasis on regional, gender, resilience, environmental perspectives and the gender perspective. The overarching nature of the Recommendation’s pillars allows for exhaustive analysis of the multiple governance dimensions that are at play in infrastructure planning, decision making and delivery. They therefore provide a robust conceptual framework for the development of the IGIs. The pillars represent both conceptual categories and functional areas of work. As such, the pillars are not standalone entities and interact with one another to support a comprehensive overview of infrastructure governance.

The IGIs serve as a diagnostic tool to help countries assess their current stage of development and identify the dimensions that may require more attention. In particular, the IGIs aim to achieve the following goals:

• map OECD countries’ state of play regarding infrastructure governance, identifying strengths and weaknesses

• provide tools for countries to self-assess their performance in each of the infrastructure governance pillars highlighted in the Recommendation

• provide a comprehensive view and deeper understanding of the different pillars that compose the infrastructure governance framework

• allow countries to identify changes in their performance on infrastructure governance through time

• draw attention to how much data are available and needed to measure infrastructure governance, as well as the benefits of building a comprehensive database in the field

• contribute to the discussion on the relationship between infrastructure governance and infrastructure outcomes.

In addition to a general assessment, the IGIs also serve to pinpoint specific areas within each pillar that may require further development from each country. Results at a more granular level (i.e. performance on the sub-components of each dimension) allow for a more in-depth assessment.

As with other composite indicators, the methodology used for building the IGIs is based on the Handbook on Constructing Composite Indicators (OECD/European Union/EC-JRC, 2008[2]). It has also been shared and discussed with experts and public officials from the Network of Senior Infrastructure and PPP Officials (SIP) and the Working Party of the Leading Practitioners on Public Procurement (LPP).

Structure of the IGIs

The IGIs are measured and presented in composite indicators, one for each of the pillars arising from the Recommendation, plus the cross-cutting pillar on environmentally sustainable and climate-resilient infrastructure. Each pillar can be disaggregated into groups of variables, called sub-pillars. These sub-pillars reflect countries’ performance at a more granular level. The nested structure helps countries understand the driving forces behind each of the composite indicators.

Implementation of the IGIs by phase

The implementation of the IGIs is being carried out in three phases. Three composite indicators were built in the first phase. In the second phase, five composite indicators were built, measuring the following pillars: 1) transparent, systematic and effective stakeholder participation; 2) coherent, predictable, and efficient regulatory framework; 3) a whole-of-government approach to managing threats to integrity; 4) evidence-informed decision making; and 5) environmentally sustainable and climate-resilient infrastructure. In the third and final phase, the composite indicators for the remaining pillars will be developed (see Figure B.1). The results for the full set of indicators will provide an overarching analysis of countries’ performance across all dimensions of the Recommendation and on the cross-cutting pillar on environmentally sustainable and climate-resilient infrastructure. This edition of Government at a Glance presents and discusses the results for four pillars of the second phase (see Figure B.2). Results from the first phase are available in the OECD Infrastructure Toolkit (OECD, n.d.[3]).

Figure B.1. Implementation of data collection by phase

Data collection and validation

The IGIs were built using data collected via OECD survey instruments, namely the Survey on the Governance of Infrastructure and other relevant data collected from OECD policy communities. The survey was designed based on inputs from relevant divisions/directorates of the OECD and in consultation with the SIP and the LPP. Invitations to participate in the survey were sent to all OECD countries, including delegates from the SIP and main contact points in country delegations. SIP officials co-ordinated responses across government, which in some cases came from specific sectors (transport being the most common) or other competent ministries. Respondents were predominantly senior officials in the central/federal ministries of infrastructure, public works and finance, as well as in infrastructure agencies and other line ministries.

The process included various steps to ensure the highest standards in data quality and accuracy. Before the survey was launched, the questionnaire and the glossary of key terms were discussed with relevant divisions/directorates of the OECD and circulated among the delegates of the SIP for comments. A data validation process was used to check for internal and external consistency in the survey responses, comparing the answers to previous answers provided in related questionnaires, and verifying that supporting evidence was systematically provided before validating the responses.

Selection of variables and re-coding

The sub-pillars were constructed from a set of variables that aim to measure the adoption and adequacy of governance practices in line with the Recommendation. The variables were selected in order to measure countries’ performance in infrastructure governance in terms of inputs and processes (e.g. policy tools, norms of interaction, decision-making methodologies and monitoring strategies). The proposed composite indicators did not include variables related to outputs or outcomes (e.g. levels of investment, quality of infrastructure services, or amounts of capital stock and achievement of policy objectives). It is important to note that the selection of variables and re-coding, and thus the structure of the composite indicators, could be subject to change in future editions of the IGIs to account for changes in institutional, political and economic settings across OECD countries. An overview of the sub-pillars under each of the four pillars presented in this edition of the Government at a Glance is shown below in Figure B.2.

The OECD Survey on the Governance of Infrastructure was designed to collect qualitative data. Therefore, the responses to the survey questions were re-coded using numerical values between 0 and 1, where 1 is the maximum value and indicates complete alignment with the best practices highlighted in the Recommendation, and 0 is the minimum value indicating the absence of such practices in the country.

For sector-specific questions, the survey covered five sectors – transport, energy, social, water and government office buildings. However, complete information was only available for the transport sector. In order to ensure comparability between countries responses, the composite indicators were built taking into account only the transport sector.

Under the pillar on regulatory framework, the sub-pillar on the governance of economic regulators aggregates two variables, the independence and accountability of economic regulators. These variables were built using the 2018 Indicators on the Governance of Sector Regulators. These indicators capture the governance arrangements of economic regulators in the energy, e-communications, rail transport, air transport and water sectors, and are structured around three pillars: independence, accountability and scope of action. To build both the variables, a simple average of the equivalent sector indicators for which data were available was calculated. As the Indicators on the Governance of Sector Regulators take the values 0-6, with 0 indicating the most effective governance arrangement, they were reverse coded to build the independence and accountability of economic regulators variables of the governance of economic regulators sub-pillar.

One of the survey questions under the pillar on environmentally sustainable and climate-resilient infrastructure used data from the OECD 2021 Indicators of Regulatory Policy and Governance (iREG). The iREG present up-to-date evidence on regulatory policy and governance practices, measuring three key principles – stakeholder engagement, regulatory impact analysis (RIA) and ex post evaluation. They are based on responses to the 2021 Regulatory Indicators Survey provided by government bodies responsible for regulatory reform.

To simplify the processing of the data, under the whole-of-government approach to manage threats to integrity pillar, the survey to collect data includes separate sections on risk-based assessment (under the risk-based approach sub-pillar) and on internal control (under the internal control and audit sub-pillar), even though internal control (i.e. the measures aimed at mitigating the identified risks) is part of the risk management process.1

Figure B.2. Infrastructure Governance Indicators: Pillars, sub-pillars and their corresponding weights used in this publication

Missing data

Due to the cross-cutting nature of the concept of infrastructure governance, the OECD surveys on the governance of infrastructure require respondents from different institutions to provide information on the infrastructure governance frameworks and practices in a country. The composite indicator for each pillar was not calculated for countries that reported not having the information to answer two or more survey questions for any one of its sub-pillars. Consequently, those countries were not included in the OECD average indicator value for that pillar. As the data used to build the composite indicators are qualitative, data imputation was not used to deal with missing data. However, it should be noted that where country responses were only based on practices applicable in a certain sector or sectors, these were retained and important caveats provided in relation to those.

Weighting and aggregation

To build the composite indicators, all the sub-pillars within each pillar were given equal weight. However, the variables within a sub-pillar were weighted differently depending on: 1) the number of variables that make up each sub-pillar, as the larger the number of variables within a sub-pillar the lower the weight each variable will have; and 2) the relevance of each variable, where greater weight was given to variables that are more relevant in measuring a specific sub-pillar. The weights assigned to the variables in each sub-pillar add up to 1. The weighted scores of all the variables are totalled to arrive at a sub-pillar score that ranges from 0 to 1.

The linear aggregation method was used to first aggregate the variables into a sub-pillar (i.e. weighted arithmetic mean), and then the sub-pillars into a composite indicator (i.e. arithmetic mean). Experts and public officials from the SIP and the LPP were consulted over the assignment of weights and the aggregation type before the final set of weights was confirmed.

Multivariate analysis

Multivariate analysis was employed to study the overall structure of the data collected. The analysis was used to further help guide methodological choices with respect to variable grouping and aggregation. The techniques used in the multivariate analysis are detailed below.

Factor analysis

Factor analysis was used to check the structure of the data along the variable dimension, to help identify groups of variables that are statistically similar and that could be regrouped under a sub-pillar where such grouping is conceptually relevant. The analysis was run separately for each pillar. Principal component factor analysis was used to extract the principal components and consider them as factors (groups of variables). The groups of variables offered by the factor analysis were interpreted together with the conceptual framework underpinning the composite indicators exercise.

The results were carefully reviewed to look for any set of variables that measure the same underlying dimension and that could be regrouped to avoid double-counting. The results offered several cases where the factors matched well the conceptual groupings (sub-pillars). In the case of variables with high levels of covariance but belonging to different initial conceptual groupings, the results were discussed with experts to determine if the variables needed to be regrouped. Following this consultation with experts, some sub-pillars under the pillar on environmentally sustainable and climate-resilient infrastructure were re-adjusted. In these cases, variables initially placed in different sub-pillars, but which were found to measure similar or highly related concepts, were combined under the most relevant sub-pillar.

Cronbach coefficient alpha

The Cronbach coefficient alpha (c-alpha) was used as a measure of internal consistency and scale reliability. The coefficient shows how related the variables are as a group and to what extent they measure the same underlying concept. A c-alpha of 0.7 is usually recommended as an acceptable reliability threshold (Lafortune and Ubaldi, 2018[4]). The c-alpha test was used to measure internal consistency for each pillar. The coefficients for all the pillars except for the pillar on regulatory framework were over the threshold of 0.7. The coefficient for the pillar on regulatory framework was just below the threshold, at 0.69. This might be due to a combination of reasons. For example, two of the variables under this pillar were built using the 2018 OECD Indicators on the Governance of Sector Regulators, which themselves comprise composite indicators aggregating different underlying dimensions. On the other hand, the other variables under this pillar measure specific dimensions.

Sensitivity analysis

To assess the robustness of the composite indicators, Monte Carlo simulations were used to study how uncertainty in the weighting schemes affects the composite indicator values. This technique uses 1 000 sets of randomly generated simulated weights to calculate possible composite indicator scores for each country under different weighting schemes.

Measuring balance in sub-pillar scores

Good infrastructure governance requires improvements across multiple dimensions. Ideally, countries should make progress in all sub-pillars, and low scores in some should not be compensated with high scores in others (i.e. sub-pillars for a country should not show a wide range of values). For each pillar, a rating scale based on the coefficient of variation will be used to rate country profiles from balanced (low variability in country sub-pillar scores under a pillar) to unbalanced (high variability in country sub-pillar scores under a pillar). For each pillar, this analysis will show how balanced country profiles are with respect to sub-pillar scores and help identify countries with relatively high indicator values but with great variability in their sub-pillar scores. The analysis for each country will be presented in the OECD Infrastructure Toolkit (OECD, n.d.[3]).

References

[4] Lafortune, G. and B. Ubaldi (2018), “OECD 2017 OURdata Index: Methodology and results”, OECD Working Papers on Public Governance, No. 30, OECD Publishing, Paris, https://doi.org/10.1787/2807d3c8-en.

[1] OECD (2020), “Recommendation of the Council on the Governance of Infrastructure”, OECD Legal Instruments, OECD, Paris, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0460.

[3] OECD (n.d.), Infrastructure Toolkit, OECD, Paris, https://infrastructure-toolkit.oecd.org/ (accessed on 15 January 2023).

[5] OECD (forthcoming), Risk Management in Public Procurement: Good Practices and Guidelines.

[2] OECD/European Union/EC-JRC (2008), Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Publishing, Paris, https://doi.org/10.1787/9789264043466-en.

Note

← 1.  For more information on risk management and its application, for example, to the procurement stage, please refer to OECD (forthcoming[5]).