This chapter reviews how Slovenia’s regulatory policy encourages ministries to set goals, priorities and indicators when developing or amending regulation. Finally, it makes recommendations for how Slovenia could support the use of priorities and indicators in its regulatory to improve the effectiveness of regulation.
Regulatory Policy in Slovenia
Oversight Matters
OECD Reviews of Regulatory Reform
Chapter 7. Goal setting, priorities, monitoring and indicators
Abstract
Data and analysis drives evidence-based policy, but the only way for governments to measure the impact of policy and spending is through the targeted development of indicators to track the effectiveness of policies and regulations. Both the 2012 Recommendation on Regulatory Policy and Governance and the 2015 Recommendation of the Council on Budgetary Governance repeatedly assert that governments should clearly define and track policy priorities to measure the success or failure of regulations, policies and spending. By monitoring the effects of policy after implementation, governments learn which policies and regulations are effective and which can be improved to support citizens’ well-being.
Box 7.1. Goal setting, priorities, monitoring and evaluation of spending in the Recommendation of the Council on Budgetary Governance
Recommendation 2: Closely align budgets with the medium-term strategic priorities of government, through:
a) developing a stronger medium-term dimension in the budgeting process, beyond the traditional annual cycle;
b) organising and structuring the budget allocations in a way that corresponds readily with national objectives;
c) recognising the potential usefulness of a medium-term expenditure framework (MTEF) in setting a basis for the annual budget, in an effective manner which i) has real force in setting boundaries for the main categories of expenditure for each year of the medium-term horizon; ii) is fully aligned with the top-down budgetary constraints agreed by government; iii) is grounded upon realistic forecasts for baseline expenditure (i.e. using existing policies), including a clear outline of key assumptions used; iv) shows the correspondence with expenditure objectives and deliverables from national strategic plans; and v) includes sufficient institutional incentives and flexibility to ensure that expenditure boundaries are respected
d) nurturing a close working relationship between the Central Budget Authority (CBA) and the other institutions at the centre of government (e.g. prime minister’s office, cabinet office or planning ministry), given the inter‑dependencies between the budget process and the achievement of government-wide policies; e) considering how to devise and implement regular processes for reviewing existing expenditure policies, including tax expenditures (see recommendation 8 below), in a manner that helps budgetary expectations to be set in line with government-wide developments.
Recommendation 8: Ensure that performance, evaluation and value for money are integral to the budget process, in particular through:
a) helping parliament and citizens to understand not just what is being spent, but what is being bought on behalf of citizens – i.e. what public services are actually being delivered, to what standards of quality and with what levels of efficiency;
b) routinely presenting performance information in a way which informs, and provides useful context for, the financial allocations in the budget report; noting that such information should clarify, and not obscure or impede, accountability and oversight;
c) using performance information, therefore, which is i) limited to a small number of relevant indicators for each policy programme or area; ii) clear and easily understood; iii) allows for tracking of results against targets and for comparison with international and other benchmarks; iv) makes clear the link with government-wide strategic objectives;
d) evaluating and reviewing expenditure programmes (including associated staffing resources as well as tax expenditures) in a manner that is objective, routine and regular, to inform resource allocation and re-prioritisation both within line ministries and across government as a whole;
e) ensuring the availability of high-quality (i.e. relevant, consistent, comprehensive and comparable) performance and evaluation information to facilitate an evidence-based review
f) conducting routine and open ex ante evaluations of all substantive new policy proposals to assess coherence with national priorities, clarity of objectives, and anticipated costs and benefits;
g) taking stock, periodically, of overall expenditure (including tax expenditure) and reassessing its alignment with fiscal objectives and national priorities, taking account of the results of evaluations; noting that for such a comprehensive review to be effective, it must be responsive to the practical needs of government as a whole.
Source: OECD (2015), Recommendation of the Council on Budgetary Governance, OECD, Paris.
Indicators tied to specific and measurable policy objectives make it easier and more efficient for governments evaluate the impact of policy and program spending and then, if necessary, make changes to legislation or spending programs to improve the lives of citizens. Connecting government goals to priorities to measurable indicators is critical for countries to be able to ascertain how well policy works in practice. Good monitoring can lead to better more targeted ex post evaluations. They also promote learning by doing, because ministries can track how well certain kinds of regulations have worked in the past.
Box 7.2. Recommendation 4 of the 2012 Recommendation of the Council on Regulatory Policy and Governance
Integrate Regulatory Impact Assessment (RIA) into the early stages of the policy process for the formulation of new regulatory proposals. Clearly identify policy goals, and evaluate if regulation is necessary and how it can be most effective and efficient in achieving those goals. Consider means other than regulation and identify the trade-offs of the different approaches analysed to identify the best approach.
Source: OECD, 2012 Recommendation of the Council on Regulatory Policy and Governance, OECD Publishing, Paris. http://dx.doi.org/10.1787/9789264209022-en.
Some OECD countries have developed frameworks to track the effectiveness of regulations, policies and spending. The systems to track effectiveness also help to spur change within their own administrations to make them more objective and data driven. Of course, developing indicators for a new regulation or amendment can be time consuming and costly. Many countries only require the development of indicators for significant reforms. For example, Canada only requires a performance measurement and evaluation plan for high impact regulations, i.e. those with more than CAD 10 million in impacts on businesses and/or citizens (see Box 5.1 in Chapter 5 for a description of Canada’s threshold test).
Box 7.3. Canada’s Performance Measurement and Evaluation Plans
From 2007 to 2012, regulatory proposals with a high-level of impact (over CAD 10 million to the economy) were required to be accompanied by a Performance Measurement and Evaluation Plan (PMEP) and they were optional for medium impact proposals. These PMEPs helped regulators in several key ways, according to the Treasury Board secretariat. These plans:
ensured a clear and logical design that ties resources and activities to expected results;
described the roles and responsibilities of the main players involved in the regulatory proposal;
helped ministries make sound judgments on how to improve performance on an ongoing basis;
demonstrated accountability and benefits to Canadians; four ensure reliable and timely information is available to decision makers in the regulatory organisations and central agencies as well as to Canadians;
and ensured that the information gathered will effectively support an evaluation.
Along with a review from the Treasury Board Secretariat and the approval from the appropriate authority, each PMEP had 6 key components to help integrate performance management into the regulatory policy cycle and broader departmental goals:
Description and Overview of the Regulatory Proposal
Logic Model – How the regulatory proposal is linked through activities and outputs to impact on stakeholders (e.g. businesses and beneficiaries).
Indicators – A quantitative or qualitative means for gauging the initiatives performance.
Measurement and Reporting – How to approach and present ongoing performance measurements and reporting of the indicators.
Evaluation Strategy – A high-level framework to evaluate performance of the regulatory proposal over time.
Linkage to the Program Activity Architecture – How the regulatory proposal links to the broader program goals.
The requirement to develop PMEPs was removed when the Cabinet Directive on Regulatory Management (CDRM) replaced the CDSR in 2012 as a result of broader changes in how the Canadian government tracked ministry results. At the time, PMEPs duplicated existing requirements in place across government under the natural evaluation and reporting cycle. For example, regulatory programs were subject to the evaluation requirements under the Treasury Board Policy on Evaluation.
In summer 2016 the Treasury Board Policy on Results replaced the Policy on Evaluation and the Policy on Management, Resources and Results Structures. Departments now must evaluate their performance, including that of their regulatory programmes, according to the time frames and cycle established in the new Policy on Results.
Source: Treasury Board Secretariat (2009), Handbook for Regulatory Proposals: Performance Measurement and Evaluation Plan, December, www.tbs-sct.gc.ca/hgw-cgf/priorities-priorites/rtrap-parfa/guides/pmep-pmre/pmep-pmre-eng.pdf.
Good objectives should be SMART: Specific, measurable, attainable, real and time-based and should be measured using indicators that measure the real impact on citizens.
Despite the recommendation that countries develop SMART targets to see if regulations meet policy objectives, it is still quite rare for OECD members to require regulators to identify a process for assess progress in achieving the regulation’s goals. Only 8 countries require regulators to develop indicators to measure progress in achieving goals of primary laws or subordinate regulation. Tying regulatory goals to long-term goals or agenda is even less common. Only 4 OECD members reported that the require regulators to have a process to measure if the proposed regulation is contributing towards a country’s long-term goals.
Figure 7.1. OECD Country responses to “When developing regulation, are regulators required to identify a process for assessing progress in achieving a regulation’s goals?”
Source: 2014 Indicators of Regulatory Policy and Governance Survey.
In Slovenia, public sector managers are not held to account for the achievement of government objectives which are operationalised through their individual organisations and collectively across the public service (OECD, 2012b). As a result, very few new regulations include plans to track the progress and impact of new regulations.
Goal setting and priorities in regulation in Slovenia
In Slovenia, the Rules of Procedure require that regulators develop indicators to monitor the impact of new or amended regulations. According the Rules of Procedure, all regulatory drafters should conduct a comprehensive analysis of the problem and set achievable targets.
The targets should be:
specified in a way, that leaves no room for different interpretations;
measurable – measurements shall be carried out according to pre-established criteria or standards;
acceptable to the political actors and stakeholders;
and achievable within a reasonable time.
The Slovenian criteria include most of the SMART criteria recommended by the OECD and presented at the workshops in Ljubljana in December 2016. However, the Slovenian criteria are currently missing:
Specific: Although the guidance does say that the target should be specified in a way that leaves no room for different interpretation, an indicator should be specific enough to the expected impacts of the regulation that there are relatively few external factors that could affect the indicators movement. This ensures that the indicator is actually measuring the impact of the regulation after promulgation.
Attainability: The targets can be reasonably achieved by the proposed regulation, but the target is also not so easy that it does not promote action.
Relevant: The indicator should be relevant to the proposed regulation. It should be closely linked to the desired benefits of the proposed regulation. E.g. a good indicator for food safety may be the number of e. coli infection cases thought to be caused by food poisoning, if the regulated products are susceptible to that bacteria.
A lack of targeted monitoring makes it harder for regulators in Slovenia to review the effectiveness of regulations and as a result ex post evaluations are often ad hoc or focus on reducing administrative burdens. It continues to be rare for ex post evaluations or RIAs to mention the effectiveness of current regulations (see more in Chapter 6).
In most cases, the objectives are not quantitative and performance is not systematically evaluated. The evaluation is theoretically conducted when amending regulation. According to the Court of Audit in 2012, however, 71.7% per cent of draft regulations provided no information on the monitoring and implementation of the regulation. The report also found that 78.3% of proposals had not planned a framework or indicators to track progress of the law after it was implemented.
A lack of effective indicators or monitoring also means that alternatives to regulation that require them, like performance-based regulation, are much less likely to be considered (see more on this in Chapter 5). Performance-based regulation tends to be more successful in countries that have developed performance indicators to use in the practice of a performance standard.
During workshops with Slovenia, the Statistical Office stated that it rarely received requests to develop indicators or to help monitor the implementation of specific regulations. On the other hand, the EU had a detailed program to develop indicators and data to monitor interventions after they have been put into place (see Box 7.4).
Box 7.4. Arrangements for future monitoring and evaluation of regulation in the European Commission
The Impact Assessment Guidelines of the European Commission require that new regulatory proposals “Identify core progress indicators for the key objectives of the possible intervention, provide a broad outline of possible monitoring and evaluation arrangements, and ensure that evaluations are designed and timed in a way that the results can be used as input for future impact assessments”
They also state that where a preferred option has been identified, the agency should:
describe briefly how the data needed for monitoring are to be collected;
outline the nature, frequency and purpose of subsequent evaluation exercises.
And questions to be addressed in the Impact Assessment include:
what will the monitoring data and evaluation findings be used for?
to what extent do monitoring/evaluation structures already exist? Does new capacity need to be put in place?
is the baseline situation sufficiently well-known or will further data collection be necessary once the proposal has been adopted?
who are the key actors in providing and using such information? (e.g. the Commission, Member States, intermediaries such as Agencies, operators/beneficiaries, etc.)?
in general terms, what will be the roles of these actors? How will information be shared and eventually aggregated?
what will be the additional use for gathering this information? If they imply administrative burden which is significant, it should be measured through the Standard Cost Model as part of the IA.
Source: 2014 Regulatory Indicators Survey results and adapted from website http://ec.europa.eu/smart-regulation/impact/commission_guidelines/docs/iag_2009_en.pdf (accessed 9 August 2017).
Assessment and recommendations
Slovenia already requires regulators to consider how to track the impact of regulations after they have been put into place. Guidance on RIA in Slovenia also includes some definitions on what makes a good indicator. However, currently few proposals actually include any strategy for tracking results and impacts. The relative lack of oversight of this requirement may be why few proposals are compliant. However, efforts should be targeted towards those regulations with the highest impacts on society to ensure that resources are used effectively in enhancing evidenced-based policy in Slovenia.
Like detailed RIAs, the Slovenian government could prioritise the development of indicators for high-impact regulations. Few proposed regulations in Slovenia currently have indicators to track the impact and progress of new regulations. Given the resource constraints on proposers, the government should prioritise which regulations will need a more extensive performance management and tracking plan based on how big the societal impacts are expected to be. This requirement could be tied to a new procedure for prioritising RIA as discussed in Chapter 5. Ensuring that resources are spent on impactful regulations will give the government a better idea of which regulations are most in need of amendments.
As Slovenia also enhances its regulatory oversight process in general, a part of oversight towards the end of the development of the regulation should be the oversight of the performance management program to track the impact of regulations after promulgation. Of course, the oversight body would only need to look at those performance management programs and indicators for the draft proposals with the highest expected impacts.
Line ministries could be encouraged to co-ordinate with the Slovenian Statistical Office. During the workshops, the Statistical Office noted that it could be used as a source to help identify and develop indicators for tracking the impact of regulatory proposals. However, they were rarely contacted. As part of a broader reform on the requirements and guidance for indicators for high-impact regulations, regulators could be encouraged or required to co-ordinate with the Statistical Office when considering indicators to track the prospective goals of regulations. Drawing from the expertise and resources of the statistical, regulators will be able to ensure that any indicators are methodologically sound and that the statistical office and regulators are able to take advantage of economies of scale. For example, a proposer could just request an additional question or two on a survey the statistical office already conducts and is also related to the proposed regulation.
The definition of good indicators should be expanded to include all of the criteria of a SMART indicator. The guidance should also include a broader idea of a good performance management plan for a draft regulation to ensure that indicators are developed, tracked, and integrated back into policy making.
The government should set goals and objectives for Better Regulation reforms to track progress of major regulatory reforms. Slovenia is undertaking a number of recommended reforms to tackle continued barriers for businesses such as the spatial planning reform. These Better Regulation reforms should be matched with objectives and indicators to help track the actual effectiveness of reforms and to inform evaluation of the reforms.
As part of the annual regulatory planning, high-priority or importance regulations should also include a requirement to have matched objectives and indicators to track the performance of the regulations.
Over the long-term, indicators for high-impact regulations should be integrated into ex post evaluation and amendments to those regulations. The government should have a long-term strategy for integrating indicators and performance management data into ex post evaluation to inform new regulations and when modifying old ones. Current RIA guidance does include a requirement to “analyse the current situation”, however, this guidance could be strengthen to make explicit mention of indicators developed during the drafting of regulations.
Bibliography
OECD (2012a), Recommendation of the Council on Regulatory Policy and Governance, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264209022-en.
OECD (2012b), Slovenia: Towards a Strategic and Efficient State, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264173262-en.
OECD (2015), “Recommendation of the Council on Budgetary Governance”, OECD Publishing, Paris, https://www.oecd.org/gov/budgeting/Recommendation-of-the-Council-on-Budgetary-Governance.pdf.
Court of Audit (2012), “Do we evaluate the effects of the proposed regulation on society in Republic of Slovenia (2)?”, Audit Report, www.rs-rs.si/rsrs/rsrs.nsf/i/k7aaecfafa8dfd535c1257a62001c1180/$file/ria_2.pdf
Court of Audit (2007), “Do we evaluate the effects of the proposed regulation on society in Republic of Slovenia?”, Audit Report, www.rs-rs.si/rsrs/rsrs.nsf/i/k672e7926a481c380c1257298001f8274/$file/ria_ucinki_rsp.pdf.