Measuring regulatory performance is challenging, starting with defining what to measure, dealing with confounding factors, attributing outcomes to interventions and coping with the lack of data and information. This chapter describes the methodology developed by the OECD to help regulators address these challenges through a Performance Assessment Framework for Economic Regulators (PAFER), which informs this review. The chapter first presents some of the work conducted by the OECD on measuring regulatory performance. It then describes the key features of the PAFER and presents a typology of performance indicators to measure input, process, output and outcome. It finally provides an overview of the approach and practical steps undertaken for developing this review.
Driving Performance at Peru's Energy and Mining Regulator
Annex A. Methodology
Abstract
Analytical framework
The analytical framework that informs this review draws on the work conducted by the OECD on measuring regulatory performance and the governance of economic regulators. OECD countries and regulators have recognised the need for measuring regulatory performance. Information on regulatory performance is necessary to better target scarce resources and to improve the overall performance of regulatory policies and regulators. However, measuring regulatory performance can prove challenging. Some of these challenges include:
What to measure: evaluation systems require an assessment of how inputs have influenced outputs and outcomes. In the case of regulatory policy, the inputs can focus on: i) overall programmes intended to promote a systemic improvement of regulatory quality; ii) the application of specific practices intended to improve regulation, or, iii) changes in the design of specific regulations.
Confounding factors: there is a myriad of contingent issues that have an impact on the outcomes in society which regulation is intended to affect. These issues can be as simple as a change in the weather, or as complicated as the last financial crisis. Accordingly, it is difficult to establish a direct causal relationship between the adoption of better regulation practices and specific improvements to the welfare outcomes that are sought in the economy.
Lack of data and information: countries tend to lack data and methodologies to identify whether regulatory practices are being undertaken correctly and what impact these practices may be having on the real economy.
The OECD (2014[1]) Framework for Regulatory Policy Evaluation starts addressing these challenges through an input-process-output-outcome logic, which breaks down the regulatory process into a sequence of discrete steps. The input-process-output-outcome logic is flexible and can be applied both to evaluate practices to improve regulatory policy in general, and also to evaluate regulatory policy in specific sectors, based on the identification of relevant strategic objectives. It can be tailored to economic regulators by taking into consideration the conditions that support the performance of economic regulators (Box A A.1.).
The OECD Best Practice Principles for Regulatory Policy: The Governance of Regulators (OECD, 2014[2]) identifies some of the conditions that support the performance of economic regulators. They recognise the importance of assessing how a regulator is directed, controlled, resourced and held to account, in order to improve the overall effectiveness of regulators and promote growth and investment, including by supporting competition. Moreover, they acknowledge the positive impact of the regulator’s own internal process on outcomes (i.e. how the regulator manages resources and what processes the regulator puts in place to regulate a given sector or market) (Figure A A.1).
Box A A.1. The input-process-output-outcome logic sequence
Step I. Input: indicators include for example the budget and staff of the regulatory oversight body.
Step II. Process: indicators assess whether formal requirements for good regulatory practices are in place. This includes requirements for objective setting, consultation, evidence-based analysis, administrative simplification, risk assessments and aligning regulatory changes internationally.
Step III. Output: indicators provide information on whether the good regulatory practices have actually been implemented.
Step IV. Impact of design on outcome (also referred to as intermediate outcome): indicators assess whether good regulatory practices contributed to an improvement in the quality of regulations. It therefore attempts to make a causal link between the design of regulatory policy and outcomes.
Step V. Strategic outcomes: indicators assess whether the desired outcomes of regulatory policy have been achieved, both in terms of regulatory quality and in terms of regulatory outcomes.
Source: (OECD, 2014[12])
Figure A A.1. The OECD Best Practice Principles on the Governance of Regulators
Source: Adapted from (OECD, 2014[2]).
The two frameworks are brought together into a Performance Assessment Framework for Economic Regulators that structures the drivers of performance along the input-process-output-outcome framework (Table A A.1).
Table A A.1. Criteria for assessing regulators’ own performance framework
Source: OECD Analysis.
Performance indicators
For regulators, performance indicators need to fit the purpose of performance assessment, which is a systematic, analytical evaluation of the regulator’s activities, with the purpose of seeking reliability and usability of the regulator’s activities. Performance assessment is neither an audit, which judges how employees and managers complete their mission, nor a control, which puts emphasis on compliance with standards (OECD, 2004[3]).
Accordingly, performance indicators need to assess the efficient and effective use of a regulator’s inputs, the quality of regulatory processes, and identify outputs and some direct outcomes that can be attributed to the regulator’s interventions. Wider outcomes should serve as a “watchtower”, which provides the information the regulator can use to identify problem areas, orient decisions and identify priorities (Figure A A.2).
Figure A A.2. Input-process-output-outcome framework for performance indicators
Note: This framework was proposed in the initial methodology for the performance assessment framework for economic regulators (PAFER) discussed with the OECD Network of Economic Regulators (NER). It has been refined to reflect feedback from NER members and the experience of other regulators in assessing their own performance.
Source: (OECD, 2015[4]), Figure 3.3 (updated in 2017).
Approach
The analytical framework presented above informed the data collection and the analysis presented in the report. The present report looks at the internal and external governance arrangements of Peru’s Energy and Mining Regulator (Organismo Supervisor de la Inversión en Energía y Minería, Osinergmin) in the following areas:
Strategic objectives: to identify the existence of a set of clearly identified objectives, targets, or goals that are aligned with the regulator’s functions and powers, which can inform the development of actionable performance indicators;
Input: to determine the extent to which the regulator’s funding and staffing are aligned with the regulator’s objectives, targets or goals, and the regulator’s ability to manage financial and human resources autonomously and effectively;
Process: to assess the extent to which processes and the organisational management support the regulator’s performance;
Output and outcome: to identify the existence of a systematic assessment of the performance of the regulated entities, the impact of the regulator’s decisions and activities, and the extent to which these measurements are used appropriately.
Data informing the analysis presented in the report was collected via a desk review, a fact-finding mission and a peer mission to Peru:
Questionnaire and desk review: Osinergmin completed a detailed questionnaire which informed a desk review by the OECD Secretariat. The Secretariat reviewed existing legislation and OSIPTEL documents to collect information on the de jure functioning of the regulator, and to inform the basis of the fact-finding mission. This questionnaire was tailored to OSIPTEL, based on the methodology already applied by the OECD to Colombia’s Communications Regulation commission (OECD, 2015[4])), Latvia’s Public Utilities Commission ( (OECD, 2016[5])), Mexico’s three energy regulators (OECD, 2017[6]); (OECD, 2017[7]); (OECD, 2017[8]); (OECD, 2017[9]) and Ireland’s Commission for Regulation of Utilities (OECD, 2018[10]).
Fact-finding mission: the mission was conducted by the OECD Secretariat on 25‑28 June 2018 in Lima and was the key tool to collect and complete the de jure information obtained through the questionnaire with the de facto state of play. The work of the fact-finding mission tailored the PAFER methodology to Osinergmin features. Information collected was completed and checked with Osinergmin for accuracy, and issues for further discussion were also flagged.
Peer mission: the mission took place on 11-14 September 2018 in Lima and included peer reviewers in addition to OECD Secretariat. This mission took place concurrently with the peer mission for the PAFER review of Peru’s Supervisory Agency for Investment in Telecommunications (Organismo Supervisor de Inversión Privada en Telecomunicaciones, OSIPTEL), which helped the peer mission team understand the systemic issues facing regulatory authorities in Peru. This mission met with key stakeholders in Osinergmin as well as externally. At the end of the mission, the team discussed preliminary findings and recommendations jointly with senior management from Osinergmin and OSIPTEL to test their feasibility and goodness of fit.
During the fact-finding and peer missions, the team met with Osinergmin’s leadership team as well as a number of staff from across the institution, other government institutions and external stakeholders, including:
National Institute for the Defense of Competition and Intellectual Property (Instituto Nacional de Defensa de la Competencia y Protección de la Propiedad Intelectual, Indecopi)
Ministry of Economy and Finance (Ministerio de Economía y Finanzas, MEF)
Ministry of Energy and Mines (Ministerio de Energía y Minas)
Presidency of the Council of Ministers (Presidencia del Consejo de Ministros, PCM)
Environmental Protection Agency (Organismo de Evaluación y Fiscalización Ambiental, OEFA)
Peruvian Association of Consumers and Users (Asociación Peruana de Consumidores y Usuarios, ASPEC)
Peruvian Organisation of Consumers and Users (Organismo Peruano de Consumidores y Usuarios, OPECU)
Association for the Promotion of National Infrastructure (Asociación para el Fomento de la Infraestructura Nacional, AFIN)
Journalists
National Society of Mining Petroleum and Energy (Sociedad Nacional de Minería, Petróleo y Energía, SNMPE)
Economic Operating Committee of the National Interconnected System (Comité de Operación Económica del Sistema Interconectado Nacional, COES)
Sociedad Minera Cerro Verde S.A.
Peru’s Gas Transport Company (Transportadora de Gas del Perú S.A., TGP)
Naturgy Energy Group S.A.
Peruvian Society of Environmental Law (Sociedad Peruana de Derecho Ambiental)
References
[10] OECD (2018), Driving Performance at Ireland's Commission for Regulation of Utilities, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264190061-en.
[9] OECD (2017), Driving Performance at Mexico's Agency for Safety, Energy and Environment, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264280458-en.
[7] OECD (2017), Driving Performance at Mexico's Energy Regulatory Commission, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264280830-en.
[8] OECD (2017), Driving Performance at Mexico's National Hydrocarbons Commission, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264280748-en.
[6] OECD (2017), Driving Performance of Mexico's Energy Regulators, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264267848-en.
[5] OECD (2016), Driving Performance at Latvia's Public Utilities Commission, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264257962-en.
[4] OECD (2015), Driving Performance at Colombia's Communications Regulator, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264232945-en.
[1] OECD (2014), OECD Framework for Regulatory Policy Evaluation, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264214453-en.
[2] OECD (2014), The Governance of Regulators, OECD Best Practice Principles for Regulatory Policy, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264209015-en.
[3] OECD (2004), The choice of tools for enhancing policy impact: Evaluation and review, OECD, Paris, http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=gov/pgc(2004)4&doclanguage=en (accessed on 16 November 2018).