Quantification should be attempted where feasible and cost effective, as it can bring additional rigour to assessments of impacts and potential outcomes.
Evaluations typically need to draw on both qualitative and quantitative methods of analysis. In many cases, the qualitative considerations will be among the more important (e.g. environmental amenity, perceptions of safety, etc.). However, the greater the quantification of impacts, the easier it will generally be to make an overall assessment where subjective elements are present.
An estimate of costs expressed in money terms will often help in making judgments as to whether benefits that cannot be so expressed are “worth it”. For example, would the amenity value of retaining heritage features of the built environment in a potential industry development area outweigh the estimated income gains from change of use? Would preservation of native fauna be worth the estimated costs of restricting agricultural development? An ability to pose such questions can help inform necessary value judgments at the political level.
More refined quantitative methods such as multivariate or regression analysis can also provide a rigorous means of determining causality; that is, for distinguishing impacts due to a regulatory intervention from those potentially attributable to other changes or influences; see (Malyshev, 2006[4]); (OECD, 2011[5]).
Data requirements are best considered at the time a regulation is being made, as part of wider consideration of the type of ex post review that would be most appropriate.
Reviews can fail to produce credible findings and recommendations for lack of adequate ‘evidence’. Standard data collections within government may not have the granularity or specificity needed to evaluate all relevant impacts of a regulation. In such circumstances it may be that the data needed to assess performance has to be collected as part of the regulatory regime itself. This can be done under compliance reporting obligations and/or through survey instruments. If the latter, the usual precautions against response bias apply.
Regulated entities will generally be a useful source of qualitative information, but should be encouraged to provide quantitative evidence as well. Administrative data sources are increasingly being used in the quantification of impacts; see (Crato and Paruolo, 2019[6]).
The increasing availability of open data, “big data” and new statistical techniques have considerable potential both to enhance evaluations and enable innovations in how these are conducted. Patterns and responses may be discernible that would not have been possible using traditional statistical methods. This is a relatively new area and one that holds out considerable scope for learning across jurisdictions.
The observed impacts of a regulation should ideally be compared with “counterfactuals” – how things would have turned out otherwise.
At issue in a regulatory review is not just whether a given regulatory regime has on balance achieved its goal or yielded certain benefits, but whether better results may be achievable in future by adopting modifications or using alternative policy instruments, or indeed without further government intervention at all. In this sense an ex post review must also involve some ex ante analysis. The difference in this case is that actual data on impacts to date should be available. This can provide a more tractable foundation for analysing how variations could have made a difference in the past.
As noted previously, one useful technique for understanding “counterfactuals” is to benchmark domestic regulations against those found in other jurisdictions that address the same policy issue using alternative approaches. As also noted, the most useful jurisdictions for benchmarking purposes will be those where the policy objectives and broad institutional structures are similar to those domestically. It is a technique well-suited to federal systems of government, therefore, as well as at the local government level (Box 5.2).
