Gender Budgeting in OECD Countries 2023

Annex A. 2022 OECD Gender Budgeting Index methodology

Data for the 2022 OECD Gender Budgeting Index (Figure ‎3.1) are derived from the 2022 OECD Survey on Gender Budgeting, encompassing responses from all 38 OECD member countries and referring only to central/federal governments. Respondents were predominantly senior budget officials within central budget authorities. Responses represent the country’s own assessment of current practices and procedures as of 1 March 2022. Data collection ceased in June 2022 and a data cleaning process took place from June to December 2022. The process checked for internal and external consistency in the survey responses. During the review process, countries were invited to clarify and verify any substantial changes from the earlier data, potential inconsistencies and outliers.

Surveys were previously undertaken in 2016 and 2018, and in 2018, a first pass at an Index on Gender Budgeting was developed by the OECD (OECD, 2019[21]). In 2022, the OECD strengthened its survey methodology to reflect evolving gender budgeting practices and wider considerations highlighted by the OECD Senior Budgeting Officials Network on Gender Budgeting. These developments expanded the number of building blocks identified as essential for an effective and sustainable gender budgeting practice, from three in 2018: 1) strategic framework, 2) tools of implementation, 3) enabling environment, to five in 2022: 1) institutional and strategic arrangements; 2) methods and tools; 3) enabling environment; 4) accountability and transparency; and 5) impact. These revisions further informed the updated 2023 OECD Framework for Gender Budgeting (Gatt Rapa and Nicol, 2023, forthcoming[1]) and the 2022 OECD Gender Budgeting Index; rendering the 2018 and 2022 versions of the Index incomparable.

The Methodology used to build the 2022 OECD Gender Budgeting Index draws on the recommendations and steps identified in the Handbook on Constructing Composite Indicators: Methodology and User Guide (OECD/EU/JRC, 2008[22]). The 2022 OECD Gender Budgeting Index is based on the five building blocks of the 2023 OECD Gender Budgeting Framework (Gatt Rapa and Nicol, 2023, forthcoming[1]).

The structure, indicators and weightings comprising the index were selected by OECD experts based on their relevance to the building blocks and have been reviewed by country delegates to the OECD Network on Gender Budgeting.

To compare the degree of implementation of gender budgeting measures across countries, the Index ranges from 0 to 1, with countries having an advanced gender budgeting practice with a score of 0.6 and above, an intermediate practice with a score between 0.3 and 0.6, and an introductory practice with a score of 0.3 and below.

While the 2022 OECD Gender Budgeting Index was developed in co-operation with OECD countries and is based on best practices and/or theory, the Index structure, indicators and weights may be further refined over time to ensure changing practices and elements that underpin a comprehensive gender budgeting approach are captured.

The Index does not purport to measure the overall quality of gender budgeting in each country or how well a country’s selected approach to gender budgeting operates. Rather, the index presented is descriptive in nature. The various statistical analyses that were applied to the methodology were to ensure validity and reliability of the composite index.

Structure and weights of the 2022 OECD Gender Budgeting Index

The Index is assessing the implementation of gender budgeting across OECD countries through quantifying the measures in place across each of the five building blocks of the 2023 OECD Gender Budgeting Framework: 1) institutional and strategic arrangements; 2) methods and tools; 3) enabling environment; 4) accountability and transparency; and 5) impact (Gatt Rapa and Nicol, 2023, forthcoming[1]). Each of the Index’s five building blocks carries an equal weight (20%).

Figure ‎A.1. 2022 OECD Gender Budgeting Index: building blocks, variables and weights used

Source: OECD authors’ elaboration.

The tables below present the structure and weight for each of the five building blocks composing the 2022 OECD Gender Budgeting Index, including the variables and indicators and show how the 2022 Survey questions were used across each building block.

Institutional and strategic arrangements

Table ‎A.1. Methodology for institutional and strategic arrangements

Building block (weight)	Variables (weight)	2022 OECD Gender Budgeting Survey Questions	Answer options / Indicators (score)	Maximum combined index score1
Institutional and strategic arrangements (20%)	Legal basis or authority for conducting gender budgeting (33.33%)	6. What is the legal basis or authority for conducting gender budgeting? Please select all that apply.	Maximum score: 1 a. Constitution (1) b. Organic budget law (1) c. Other legislation (1) d. Ordinances / Regulations (0.5)	0.067
	Clear gender equality goals and policies guiding gender budgeting efforts (33.33%)	7. What policies guide gender budgeting efforts? Please select all that apply.	Maximum score: 1 a. National gender equality strategy (0.5) b. National gender equality goals (0.5) c. Sector-specific gender equality goals (0.25) d. Priorities identified in gender needs assessment (0.5)	0.067
	Central Budgeting Authority role in leading gender budgeting (33.33%)	5. Who has the main responsibility for gender budgeting? Please select one.	Maximum score (Q5 and 5.1 combined): 1 a. The central budget authority (CBA) (e.g. Ministry of Finance) (1) b. The gender equality institution (e.g. Ministry for Gender Equality, Ministry of Social Affairs) (0) c. Other (0)	0.067
		5.1. Who has a supporting role in implementing gender budgeting? Please select all that apply.	a. The central budget authority (CBA) (e.g. Ministry of Finance) (0.5) b. The gender equality institution (e.g. Ministry for Gender Equality, Ministry of Social Affairs) (0) c. President/Cabinet/Prime Minister’s Office (0) d. None (0) e. Other (0)

1. including weights for variable and building block.

Enabling environment

Table ‎A.2. Methodology for enabling environment

Building block (weight)	Variables (weight)	2022 OECD Gender Budgeting Survey Questions	Answer options / Indicators (score)	Maximum combined index score1
Enabling environment (20%)	Supporting elements (total = 100%): a. Guidelines on how to apply gender budgeting (15.38%) b. Expert consultative group advises gender budgeting (15.38%) c. Inter-agency group(s) on gender budgeting (15.38%) d. Training and capacity-building (15.38%) e. Programme budgeting (15.38%) f. General availability of gender/sex disaggregated data (15.38%) g. Sector-specific gender disaggregated data (7.69%)	18. Please indicate what elements the government has put in place to support the implementation of gender budgeting. Please select all that apply.	Maximum score: 1 per supporting element a. Standard guidelines or annual budget circular from central budget authority detail how to apply gender budgeting (1) b. Expert consultative group advises on the application of gender budgeting (1) c. Inter-agency group(s) to ensure co‑ordination and/or exchange of good practices on gender budgeting (1) d. Training and capacity-building in the application of gender budgeting (1) e. Programme budgeting (1) f. General availability of gender/sex disaggregated data (i.e. such data is routinely available to support policy analysis across all or most key areas of public service) (1) g. Sector-specific availability of gender/sex disaggregated data (i.e. such data is routinely available to support policy analysis for some selected areas of public service) (1)	0.20 for all supporting elements 0.031 for each answer a. to f. 0.015 for answer g.

1. including weights for variable and building block.

Tools for evidence gathering

Table ‎A.3. Methodology for methods and tools

Building block (weight)	Variables (weight)	2022 OECD Gender Budgeting Survey Questions	Answer options / Indicators (score)	Maximum combined index score1
Methods and tools (20%)	Methods and tools used for budget formulation (33.33%)	10. Please indicate specific tools used as part of your approach to gender budgeting, and their scope. Please select all that apply. (Applies for all or selected budget measures)	Maximum score: 1 a. Gender dimension in performance setting/performance budgeting (1 if applies for all budget measures, 0.5 if applies for selected budget measures) b. Ex ante gender impact assessment of budget measures (1 if applies for all budget measures, 0.5 if applies for selected budget measures) d. Gender budget tagging (tagging budget items according to their gender impact) (1 if applies for all budget measures, 0.5 if applies for selected budget measures) e. Requirement for gender information to accompany budget proposals (1 if applies for all budget measures, 0.5 if applies for selected budget measures)	0.067
	Methods and tools used in budget approval (33.33%)	10. Please indicate specific tools used as part of your approach to gender budgeting, and their scope. Please select all that apply. (Applies for all or selected budget measures)	Maximum score: 1 h. Gender dimension in spending reviews (1 if applies for all budget measures, 0.5 if applies for selected budget measures) i. Gender dimension to evaluation and/or performance audit (1 if applies for all budget measures, 0.5 if applies for selected budget measures) c. Ex post gender impact assessment of budget measures (1 if applies for all budget measures, 0.5 if applies for selected budget measures)	0.067
	Methods and tools used in budget implementation and control (33.33%)	10. Please indicate specific tools used as part of your approach to gender budgeting, and their scope. Please select all that apply. (Applies for all or selected budget measures)	Maximum score: 1 f. Distributional assessment of tax and welfare measures by gender (1 if applies for all budget measures, 0.5 if applies for selected budget measures) g. Gender impact assessment of the budget (1 if applies for all budget measures, 0.5 if applies for selected budget measures)	0.067

1. including weights for variable and building block.

Accountability and transparency

Table ‎A.4. Methodology for accountability and transparency

Building block (weight)	Variables (weight)	2022 OECD Gender Budgeting Survey Questions	Answer options / Indicators (score)	Maximum combined index score1
Accountability and transparency (20%)	Publication of gender information in budget documentation (33.33%)	13. What gender information is included in, or accompanies, the annual draft budget documentation? Please select all that apply.	Maximum score: 1 a. Standalone gender budget statement (1) b. Separate chapter relating to gender budgeting (1) c. Gender budgeting discussion integrated into the general narrative of ministries and programmes (1) d. Annex or appendix on gender budgeting (1) e. Other (0.5)	0.067
	Publication of other gender budgeting information (total = 33.33%): a. Ex post gender budget statement alongside annual reporting documents (5.56%) b. Performance report that shows progress towards gender goals (5.56%) c. Ex ante gender impact assessments (5.56%) d. Ex post gender impact assessments (5.56%) e. Spending reviews with gender dimension (5.56%) f. Evaluations and/or performance audits with gender dimension (5.56%)	14. Is any additional gender budgeting information published by the government? Please select all that apply.	Maximum score: 1 per additional publication a. Ex post gender budget statement alongside annual reporting documents (1) b. Performance report that shows progress towards gender goals (1) c. Ex ante gender impact assessments (1) d. Ex post gender impact assessments (1) e. Spending reviews with gender dimension (1) f. Evaluations and/or performance audits with gender dimension (1)	0.067 for all additional publications 0.011 for each answer a. to f.
	Parliamentary/Supreme Audit/ Independent Fiscal Institution oversight (16.67%)	15. How do you ensure external accountability in relation to gender budgeting? Please select all that apply.	Maximum score: 1 a. Regular reporting to parliament on gender budgeting (1) b. Parliamentary committee hearings on gender budgeting (1) c. Parliamentary debate on gender budgeting (1) d. Oversight by the Supreme Audit Institution (1) e. Oversight by an Independent Fiscal Institution (parliamentary budget office or fiscal council) (1)	0.033
	Engagement with Civil Society (16.67%)	15. How do you ensure external accountability in relation to gender budgeting? Please select all that apply.	Maximum score: 1 f. Structured engagement/events with civil society (1)	0.033

1. including weights for variable and building block.

Impact

Table ‎A.5. Methodology for impact

Building block (weight)	Variables (weight)	2022 OECD Gender Budgeting Survey Questions	Answer options / Indicators (score)	Maximum combined index score1
Impact (20%)	Scope of gender budgeting information promoting gender perspective in policy development and resource allocation (50%)	21. From your perspective, how often would you say information from gender budgeting is used in budget decision making? Please select one.	Maximum score: 1 a. Never (0) b. Rarely (0.25) c. Sometimes (0.5) d. Often (0.75) e. Always (1)	0.10
	Frequency of gender budgeting information used in budget decisions (50%)	22. From your perspective, has gender budgeting been effective in promoting a clear gender perspective in policy development and resource allocation? Please select one.	Maximum score: 1 a. Yes, across government as a whole (i.e. gender budgeting has been effective in promoting a clear gender perspective as a general principle of policy development and resource allocation) (1) b. Yes, but only in some areas (i.e. gender budgeting has been effective in promoting a clear gender perspective in certain ministries, departments or institutions) (0.5) c. No (i.e. no significant results to date in promoting a clear gender perspective) (0) d. Not yet able to judge, e.g. insufficient information available (0)	0.10

1. including weights for variable and building block.

Coding and aggregating the data

Each of the answer options in the 2022 Gender Budgeting Survey were translated into quantitative indicators, varying from 0 to 1, with the value of one representing the highest value of the indicator and the value of zero representing the lowest. Although all indicators respect the [0,1] range, the method for assigning values to responses for questions vary by the type and structure of the question. For binary-type questions (yes/no), the value of 1 represents the existence (yes) of the indicator and the value of 0 represents the nonexistence (no). For ordinal-type questions, multiple answer options are possible, and values are assigned according to their significance or importance, e.g. [0, 0.5 or 1]. For some questions with multiple response options, several options were attributed the maximum value [1] as they are considered equally important. Once the maximum value is obtained, additional responses are not taken into account. Thus, the maximum score per variable can not be bigger than 1.

The 2022 OECD Gender Budgeting Index is referring to measures under implementation as of 1 March 2022. If a measure was under consideration or being developed at the time, but not yet implemented, it was assigned a value of zero.

A linear aggregation method was applied at each stage of aggregation: to obtain a country score, the total indicator score (capped by the indicated maximum score) for each variable was multiplied by its variable weight and its building block weight. The index scores for each variable were then summed to obtain an overall score for each country from 0 to 1. The weights assigned to each building block and building block variables are displayed in Figure ‎A.1 and Tables A.1 to A.5.

Statistical analysis

Sensitivity testing

Monte Carlo simulations generate interval of values which reflect the range of possible scores for indicators in the case of each country, depending on five thousand different weighting scenarios. The Monte Carlo simulation for the 2022 OECD Gender Budgeting Index is testing the sensitivity of the weights at the building block level (equal weight of 20% for each of the five building blocks). The results are presented in Figure ‎A.2. Vertical lines represent the 90% confidence intervals derived from the random weights analysis. The size of the interval varies depending on each country’s gender budgeting data profile (mixed or rather even). Countries with an even answer profile (i.e. which gave similar answers for the majority of questions) will fall into a narrow interval, while countries with an uneven profile will fall into a broader interval (Freudenberg, 2003[23]).

The results from the sensitivity analyses at building block levels show that in general the total scores of Korea and Portugal appear to be more sensitive to the weightings applied. It is important to note that not all the combinations of weights used to create these intervals can be viewed as realistic outcomes (Arndt, C., et al., 2015[24]). One possible practical application for the analysis of the upper and lower bounds of these intervals is to consider how countries can be grouped together with a strong degree of confidence (Arndt, C., et al., 2015[24])

The group of countries with black diamonds on the left-hand side of the graph not only have a total score for their composite indicator that is above the average (expressed as a horizontal dotted average line), but also have scores above the average for 90% of random combinations (this is the case for: Austria, Canada, Korea, Iceland, Mexico, Spain and Sweden). Conversely, on the right-hand side of the graph countries that are marked with black diamonds score below the average for 90% of random combinations (with Australia and Japan clearly well below the average). These two groups of countries can therefore be said to have indicator values which are significantly different from each other independent of the weighting scheme.

Cronbach alpha testing (scale reliability coefficient)

Cronbach’s Alpha is a coefficient of reliability based on the correlations between indicators. This statistic is generally used to investigate the degree of correlation among a set of variables and to check the internal reliability of items in a model or survey. A Cronbach’s alpha coefficient equal to zero means that the variables are independent (e.g. the selection is not correlated and therefore is statistically not relevant), while a coefficient equal to one means that the variables are perfectly correlated. In general, a coefficient of above 0.7 is considered to be an acceptable indication that the variables are measuring the same underlying construct. The value recorded of the coefficient is of 0.84 at the building block level which indicates that the building blocks are measuring the same underlying construct.

Figure ‎A.2. 2022 OECD Gender Budgeting Index: Sensitivity analysis

Note: Figure presents the sensitivity of the index to various weighting assumptions (results from the Monte Carlo simulation where 5 000 different weights were assigned). Diamonds represent the indicator scores and vertical lines represent the 90% confidence intervals derived from the random weights analysis. Index comprised between 0 (no use) and 1 (high use).

Cronbach’s alpha: 0.84 (computed with SPSS). A Cronbach’s alpha close to or above 0.7 indicates a high degree of correlation among a set of variables.

Source: OECD authors’ analysis.

Statlink

Limitations of composite indexes

Composite indexes are useful to summarise large amounts of discrete, qualitative information. “Composite indexes are much easier to interpret than trying to find a common trend in many separate indicators” (Nardo et al., 2005[25]) However, these indexes are easily and often misinterpreted by users due to a lack of transparency as to how they are generated and the resulting difficulty to truly unpack what they are actually measuring. The 2022 OECD Gender Budgeting Index, while providing cross-country comparison, is not context specific, nor can fully capture the complex realities of the quality, use and impact of gender budgeting approaches. This comparison should hence not be seen as a measurement of quality or a ranking. It shows that countries have adopted multi-initiative approaches to gender budgeting by using each of the five building blocks.

Composite indexes should be seen as a means of initiating discussion and stimulating public interest (OECD/EU/JRC, 2008[22]). To help countries interpretate the findings and to provide a more detailed analysis of the strengths and weaknesses of their gender budgeting approach, the results of the 2022 OECD Gender Budgeting Index should be complemented with in-depth country assessments. These country assessments allow to provide a more detailed analysis of the quality of country practices in the context of diverse and country-specific governance structures, administrative cultures, institutional set-ups and priorities.