Regulatory enforcement and inspections should be evidence-based and measurement-based: deciding what to inspect and how should be grounded on data and evidence, and results should be evaluated regularly.
OECD Regulatory Enforcement and Inspections Toolkit
Report
Table of contents
Share
Facebook
Twitter
LinkedIn
Criterion 1. Evidence-based enforcement1
Key questions:
Are enforcement and inspection aspects reviewed during Impact Assessment for regulations, and considered in ex post review of regulations?
Do the mandates of regulatory enforcement and inspections institutions reflect goals set by governments in terms of risk reduction and pursuing public interest?
Do the indicators and data used to assess the performance of regulatory enforcement and inspections institutions similarly focus on outcomes such as risk reduction, economic growth, social well-being etc.?
Are effectiveness evaluations implemented in practice, and do they have real consequences in terms of inspection and enforcement structures, methods, resources and tools?
Sub-criterion 1.1. Enforcement and inspection aspects are reviewed during the Impact Assessment process for new regulations – and evidence-based enforcement is “anchored” as a key aspect to be checked during both design and ex post review of regulations
Considering the implementation and enforcement stage of proposed regulations is an integral part of “best practice” Regulatory Impact Assessment (and Impact Assessment more broadly). In principle, ex post reviews should also be looking at enforcement and broader implementation questions. However, available evidence suggests that specific attention is not always paid to these issues. Many RIAs treat inspections and enforcement as a given, something that more-or-less automatically should or will happen (and be effective), rather than specifically considering the different possible options, their costs and benefits, as well as their relative effectiveness (or lack thereof). Likewise, ex post reviews often look at the cost and effectiveness of a given regulation as a whole, without considering the inspection and enforcement part in its specificity, looking at whether it was correctly designed and implemented, what would be the alternatives etc. Feedback is also not always sought from the inspectors themselves, or from stakeholders specifically on the topic of inspections and enforcement.
In order for the inspection and enforcement system to be adequately designed and achieve the best balance of efficiency and effectiveness, it is necessary to properly consider whether inspections and enforcement would be needed for a proposed regulation, how they should be organised and resourced, what methods they should follow.2 These same issues should be considered when reviewing regulations ex post.
Evidence: RIA guidelines/procedures, contents of published RIAs
Sub-criterion 1.2. The mandates of institutions in charge of regulatory enforcement and inspections reflect goals in terms of risk reduction and pursuing public interest
There should be no automatic assumption that enforcing a given set of legislation requires an ad hoc institution – and that “enforcing legislation” is an adequate definition of purpose. Rather, institutions empowered to inspect and enforce should have a purpose that is stated in terms of what harm to the public they will mitigate, of what positive outcomes they will contribute to achieve. Institutions, that are not able to define their mission in this way, typically create significant costs without clear benefits, and have little incentives to improve approaches and methods to achieve better results. Nor is it possible to properly assess their performance, since it is defined in a purely “circular” way: their task is to inspect and enforce, the more they do of it, the better, regardless of what benefits or costs this entails for the public.
It is thus essential that each institution or structure with inspection and enforcement power have a mandate that is clearly defined in terms of risk management3 and/or outcomes for social well-being to which it should contribute. The mandates can be set in a variety of ways (generally through primary or secondary legislation, but they can also be supplemented through strategy documents approved by the supervisory board or ministry etc.).
Evidence: official mandates as adopted in relevant legislation/official documents
Sub-criterion 1.3. Do the indicators and data used to assess the performance of regulatory enforcement and inspections institutions similarly focus on outcomes such as risk reduction, economic growth, social well-being etc.?
Using volume of activities (inspections) or violations detected (and sanctions) as indicators gives perverse incentives to inspecting agencies, since they then have an incentive to achieve low compliance levels, which will give them more volume – and runs contrary to their mission. Performance should be assessed against the achievement of social well-being (safety, health, environmental protection etc.) and, as an intermediate step towards this and a proxy for these goals, against improvements in compliance. It is indispensable that this be defined as a central task and indicator for inspection and enforcement structures.
Just as mandates set for enforcement and inspections institutions need to correspond to improvements in social well-being and reductions in risk levels, specific objectives and indicators defined to measure performance within these mandates should have a similar focus. For each specific inspectorate and enforcement area, key indicators can be identified that relate meaningfully to their mandate. This can be done by first defining a number of high-level “goals”, “aims” or “priority areas” (since most inspectorates are responsible for a broad regulatory domain that covers several distinct goals) and, within each of these, specific objectives and indicators measuring to what extent these objectives are being achieved. Because there are a number of issues involved in properly measuring outcomes from inspections and enforcement activities (attribution problems, problematic quality of data in some cases, difficulty to measure certain outcomes e.g. environmental on frequent basis, time-lag between activities and outcomes etc.), in practice these can be complemented by some indicators pertaining to levels of compliance (specifically decreasing non-compliance, particularly among high-risk groups).
Evidence: performance indicators as officially adopted and publicised
Sub-criterion 1.4. Effectiveness evaluations are implemented in practice, and really inform choices in terms of inspection and enforcement structures, methods, resources and tools
In order for inspections and enforcement to be genuinely evidence-based, evaluations of effectiveness of existing practices and institutions must be regularly conducted – and actions taken on the basis of these evaluations. Such evaluations would consider the evolution of performance indicators over several years, trends in improvement or worsening of key indicators of over time (whether there are any inflexion points), comparison with other jurisdictions (looking both at the level of indicators and at the relative trends), when possible and relevant specific surveys (which can include representative surveys of businesses or other stakeholders, but also e.g. epidemiological surveys), as well as more “qualitative” instruments (consultations with various stakeholder groups), etc.
If assessments and evaluations show the system or institution to be lacking, objectives not being met and/or costs being too high, stakeholders satisfaction being low etc., changes should be proposed and, eventually, implemented, affecting the elements that have been found wanting (institutional structures, legislation, processes, approaches, tools etc.).
Evidence: Official policy on evaluations, examples of public evaluations and follow up reports
Notes
← 1. i.e. grounded in the best available research and existing data and informed by Impact Assessment, cost-benefit analysis, experiential evidence from the field and relevant contextual evidence.
← 2. See e.g. the Netherlands’ “Table of Eleven” (as in www.oecd.org/gov/regulatory-policy/44912386.pdf), in (OECD, 2010[3]).
← 3. In order to determine the relative importance of different risks, the adequacy of different methods, the potential and real effectiveness of actions, it is essential to take as basis, as much as possible, reliable science. When science is not conclusive enough and uncertainty is significant, decisions on risk assessment, choice of approach, resource allocation etc. should be transparent about this uncertainty, and the choices that have been made. Likewise, if a decision is reached to treat a given issue with more (or less) intensity than its risk level estimated based on scientific evidence would warrant, there should be full transparency and clarity on what values this decision was based on, and what trade-offs were accepted as part of this decision. Science cannot be the only guide to policy decisions, and often enough uncertainty makes it indispensable to make judgment calls anyway – but clarity about values, criteria and trade-offs is essential to ensure that the decisions are evidence-based, and can be properly assessed by stakeholders. The definition of “science” should here be understood to not be limited only to natural sciences, but to include also social sciences.