This chapter discusses the importance of policy evaluation, and provides tools and strategies to promote and produce high quality evaluations. As distinguished from policy monitoring, policy evaluation seeks to analyse linkages between policy interventions and effects. Policy evaluations can therefore enhance the quality of decision-making and provide tailored advice to improve policy formulation and implementation. Despite these findings, evidence collected by the OECD suggests that, regardless of strong commitments, policy evaluation often constitutes the weakest link in the policy cycle and countries are still facing substantial challenges to promote policy evaluation. The first section of this chapter provides guidance on how to build an institutional framework and promote the quality and use of evaluations in policy-making. The following section highlights the need to review the effect of regulations.
Policy Framework on Sound Public Governance
5. Toward robust policy evaluation
Abstract
Evaluating performance and results helps to understand better why some policies work and others do not. By producing, using and promoting evidence on policy performance, policy evaluation supports the quality of decision-making (see chapter 2), providing tailored advice to improve policy formulation (chapter 3) and implementation (chapter 4). Policy evaluation, along with other practices such as user and staff feedback built into policy implementation processes, enables the strategic use of feedback loops in the policy-making process to improve policy performance, as it connects policies outcomes, impacts, and policymakers’ decisions (learning dimension), as well as government and beneficiaries (stronger focus on the accountability dimension).
Policy evaluation is the structured and objective assessment of an ongoing or completed policy or reform initiative. The aim is to determine, inter alia, the relevance and fulfilment of objectives along with the initiative’s efficiency, effectiveness, impact and sustainability1. As distinguished from policy monitoring – which, as explained in chapter 4, is essentially a descriptive exercise – policy evaluation seeks to analyse linkages between policy interventions and effects. It strives to create deeper understanding of observed policy success or failure as an end in itself and as a means to correct course and improve performance to enhance results and outcomes.
Robust policy-evaluation systems imply that evaluations are part and parcel of the policy cycle; that evaluations are carried out rigorously and systematically; that the results are used by decisions-makers; and that information is readily available to the public (Lázaro, 2015[66]). Moreover, evaluation methods need to be taking into account during the policy-formulation and design phase and integrated into an overall approach, in order to ensure that the necessary information and data required for effective policy evaluation can be collected during the implementation phase. Policy evaluation needs to be incorporated into the design of strategies for pursuing the SDGs (OECD, 2019[47]). Nevertheless few countries have established mechanisms to evaluate SDG achievements. Mainstreaming evaluation of the SDGs and Agenda 2030 therefore constitutes and important element on the policy-making agenda for all countries that are engaged in designing and pursuing the SDGs.
Given the overall importance and benefit of policy evaluation, several OECD Recommendations - such as the ones on Open Government and Public Integrity - underline the importance of undertaking evaluations. More specifically:
The Recommendation on Regulatory Policy and Governance (OECD, 2012[27]) [OECD/LEGAL/0390] recommends that Adherents “[c]onduct systematic programme reviews of the stock of significant regulation against clearly defined policy goals, including consideration of costs and benefits, to ensure that regulations remain up to date, cost justified, cost effective and consistent, and deliver the intended policy objectives.”
The Recommendation on Budgetary Governance (OECD, 2015[48]) [OECD/LEGAL/0410] recommends that governments ensure that “performance, evaluation & value for money are integral to the budget process”. To this end, the Recommendation suggests countries evaluate and review “expenditure programmes (including associated staffing resources as well as tax expenditures) in a manner that is objective, routine and regular, to inform resource allocation and re-prioritisation both within line ministries and across government as a whole”.
Nevertheless, policy evaluation very often constitutes the weakest link in the policy cycle and countries are still facing substantial challenges to promote policy evaluation (see Figure 5.1).
Figure 5.1. Government’s current challenges for promoting policy evaluations
Note: For the main institution n=42 (35 OECD Member countries). For the Health ministries n=31 (28 OECD Member countries). 9 countries (7 OECD Member countries) did not participate on this survey. Moreover, 2 countries (1 OECD Member country) are not included as they answered that none of the policies that fall in their institution's responsibility are evaluated. For the PSR ministries n=25 (20 OECD Member countries). 11 countries (10 OECD Member countries) did not participate on this survey. Moreover, 6 countries (5 OECD Member countries) are not included as they answered that none of the policies that fall in their institution's responsibility are evaluated. Answers reflect responses to the questions, “What are the government's current challenges for promoting policy evaluations?” for the main institution and “What are current challenges for promoting policy evaluation in your institution?" for Health and PSR, where 0 indicates that is a "rare challenge", 5 is "Neutral", and 10 is a "principal challenge".
Source: OECD Survey on Policy Evaluation (2018[67])
As Figure 5.1 shows, developing and/or implementing a strategy for promoting a whole-of-government approach on policy evaluation is a key challenge for many countries. Such a strategy should ideally address two main issues, thus providing guidance on how to:
Build an institutional framework for policy evaluation, which provides amongst others (a) the legal basis to undertake policy evaluations; (b) macro-level guidance on when and how to carry out evaluations; and (c) the identification of mandated institutional actors with allocated resources to oversee or carry out evaluations.
Promote the quality and use of policy evaluations across government, including efforts related to building human resources capacity, ensuring appropriate stakeholder engagement, etc.
Building an institutional framework for policy evaluation
As for any area of public governance, a fit-for-purpose adequate institutional framework constitutes a solid basis to embed the practice of evaluations across government in a systematic and systemic way. That said, in this case as in most others, no single one-size-fits-all approach need govern the design and adoption of such frameworks
Indeed the legal and policy anchoring of evaluation can vary substantially across countries. Some countries have specific stipulations in their constitutions; others focus on primary or secondary law to anchor policy evaluation; while still others opt for flexible arrangements linked for instance to particular public-sector reform strategies.
Robust policy evaluation systems can benefit from clearly designated institutional actors with a well-defined mandate and specific resources to oversee and/or carry out policy evaluation. Here, the landscape is also quite diverse. While in some countries there is one, or a small number of organisation(s) promoting and/or coordinating policy evaluation across government, such a centralised element can also be absent, without necessarily precluding the existence of a strong evaluation culture.
Even if a clear central coordinating entity exists, it can be of a very different nature depending on the country:
Some countries created departments or offices located within their CoG institutions (the Presidency, Cabinet Office, Government Office or Prime Minister Office).
Other countries have established independent agencies that set and coordinate evaluations across government. Moreover, certain line ministries can also play a central role in promoting and/or co-ordinating policy evaluation across government.
What is evaluated might also affect the institutional set-up that is required. For instance, while independent bodies might be the best option for conducting or overseeing ex post evaluation of sensitive regulations with significant impact, line ministries and agencies in charge of the implementation of the evaluated policy might be more adequate to conduct less sensitive evaluations (OECD, 2018[52])
Box 5.1. Canada’s Policy on Results
In July 2016, the Government of Canada launched a Policy on Results, which seeks to improve the achievement of results across government and better understand the desired and obtained results and the resources used to achieve them.
Responsibilities on policy evaluation are shared between the Privy Council Office and the Treasury Board. These bodies are respectively responsible for promoting the use of evaluation findings into policy-making and defining and updating the evaluation policy.
The Policy establishes that all government departments should have an evaluation unit. On the other hand, Line Ministries are responsible for establishing a Departmental Results Framework. For the implementation of the policy, the Treasury Board of Canada can require departments to undertake specific evaluations and participate in centrally-led evaluations; initiate or undertake Resource Alignment Reviews; and approve Line Ministries Departmental Results Frameworks and changes to their organisations’ core responsibilities.
Source: Treasury board of Canada Secretariat (2016[68]), Policy on results https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=31300 (Accessed August 2nd 2019).
Supreme Audit Institutions can play a critical role in the evaluation process through their audits, evaluations and advice, thus holding the government to account for the use of public resources (see Box 5.2 for an example of Chile’s Supreme Audit Institution’s role in strengthening good governance). In addition to evaluating policies and programmes on a performance or value-for-money basis, SAIs can act as an “evaluator of evaluators” in government by auditing the effectiveness of an evaluation system and those responsible for it.
Box 5.2. Chile’s Supreme Audit Institution’s role in strengthening good governance
In 2014, the OECD conducted a Public Governance Review of the SAI of Chile. The Report finds that “Chile's supreme audit institution (Contraloría General de la República de Chile or CGR) is at the forefront of an evolution of Supreme Audit Institutions and has undertaken ambitious initiatives for institutional strengthening, capacity development, transparency and citizen participation. The CGR has introduced strategic planning, restructured its workforce and become an exemplary institution with respect to transparency within the Chilean public sector.
The CGR recognises its crucial role in contributing to good public governance, and has undertaken this OECD review to support ongoing initiatives, maximise the positive impact of its work on enhancing good public governance, and improve accountability and the quality of government decision-making. The CGR has the opportunity to further strengthen its solid reputation to position itself as a leader, providing objective and credible information that is widely recognised as useful for addressing challenges to good public governance. The review explores how the CGR’s audit assignments could be adjusted to enhance the institution’s impact on good public governance, and how it could further leverage knowledge gathered through existing and new audit assignments to deliver additional value to its diverse range of stakeholders.
Source: OECD (2014[69]), Chile’s Supreme Audit Institutions: Engancing Strategic Agility and Public Trust, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264207561-en.
Promoting the quality and use of evaluations
Promoting policy evaluation across the policy cycle requires more than ticking the box that evaluations are produced. Ensuring the systematic production of policy evaluations is a necessary but insufficient condition to enhance the quality of public governance and service delivery. Poor quality evaluations will hardly contribute to better learning, higher accountability, or better decision-making and policy design, or better results for people. Likewise, high-quality evaluations may be completely ignored for actual policy decisions, due to a lack of incentives.
While the idea of fostering an evaluation culture can sound somewhat aspirational, concrete actions can be taken to promote the relevance and uptake of policy evaluations. These can for instance include the promotion of political commitment and stakeholder engagement (see chapter 1 and 2), or the support for skills development in the area of policy evaluation. Moreover, according to the OECD The Path to Becoming a Data-Driven Public Sector Report, fostering a Data Driven Public Sector culture can be a very effective way to enhance the quality of ongoing evaluations through the application of relevant data (OECD, 2019 [60]). Indeed, data in the public sector leads to an understanding of performance oriented towards an iterative approach to subsequent planning. Not only does an increase in the amount of data associated with policy outcomes allow for agile policy adjustments in the short-term, it can also generate better insights into the policy process in the mid- to long term. As a result, policymakers can assess whether policies have had the desired effect, and if that data is publish in Open Data format, so can other stakeholders. Policy evaluation can therefore become a more open, inclusive and ongoing process.
The quality of policy evaluations is an essential factor in guaranteeing the robustness and validity of any policy evaluation effort. Both quality control (deliverable oriented) and quality assurance (process oriented, i.e. doing the right things in the right way) are essential in this respect. For this reason, the OECD Recommendation on Budgetary Governance (2015[48]) [OECD/LEGAL/0410] recommends for instance that Adherents “ensure the availability of high-quality (i.e. relevant, consistent, comprehensive and comparable) performance and evaluation information to facilitate an evidence-based review”.
An analysis of the data collected through the OECD Survey on Policy Evaluation shows that governments are focusing in enhancing skills and capacities within the public service to conduct or commission policy evaluations; and in promoting stakeholder engagement, to ensure that evaluations are targeted properly and that recommendations for improvement are practical and user-centred (OECD, 2018[67]) .
As shown in Figure 5.1, the use of policy evaluation results in policy-making represents an important challenge for governments in the area of policy evaluation2. Factors such as overall quality, timing of evaluations and political commitment to the evaluation process can increase the use (and therefore impact) of policy evaluation recommendations. Therefore, countries have developed specific initiatives to promote the strategic use of policy evaluation results. According to the results of the OECD Survey on Policy Evaluation, almost 50 % of surveyed countries (60%) for instance promote the use of policy evaluation through the incorporation of their findings in budgeting (OECD, 2018[67]). This aligns with the OECD Recommendation on Budgetary Governance [OECD/LEGAL/0410], which suggests that governments should take into account the results of evaluations to reassess the alignment of overall expenditure (including tax expenditure) with fiscal objectives and national priorities (OECD, 2015[48]) . More than one third of surveyed countries also foster the use of evaluations by discussing their results at the highest political level (Council of Ministries or equivalent). A similar number of countries has established co-ordination platforms to promote the use of evidence produced by policy evaluations.
Box 5.3. Country practices in fostering the use of evaluation results in policy-making
According to the responses provided by countries to the OECD Survey on Policy Evaluation, countries are adopting different practices to fostering the use of evaluation results in policy-making:
Norway has launched a web service (https://evalueringsportalen.no/), which gathers the findings of the evaluations carried out by the central government in one platform. By increasing accessibility to evaluation results, the Government aims to increase the use and reuse of knowledge and results from evaluations in all state policy areas, in future evaluations, and in society as a whole. It is moreover important for legitimacy and transparency in relation to government activities.
The United States of America has created an Interagency Council on Evaluation Policy, co-chaired by the Office on Management and Budget (OMB) and the Department of Labor, composed of about ten high-capacity evaluation officers from government agencies, who meet on a monthly basis, to discuss evaluation results.
Moreover, in several countries evaluations results are discussed at the parliamentary level:
In Germany, the Bundestag (the lower House of Parliament) requires annually approximately 80 reports from the Federal Government regarding the evaluation of single policies or specific regulations and measures of administrative action of the government. Over the past five years, the country has issued the following evaluation reports: Scientific Advisory Council of the Federal Government on Global Change (Wissenschaftlicher Beirat der Bunderegierung Globale Veränderungen); Main Report Monopolies Commission 2016 (Hauptgutachten Monopolkommission 2016); Evaluation of the Age Allowance Act (Evaluation des Altersgeldgesetzes).
In Japan, the government submits a report each year to the Japan’s bicameral legislature Diet on the status of Policy Evaluation and on how the results of such evaluation have been reflected in policy planning and development.
Source: (OECD, 2020[70])).
Reviewing the impact of regulations
Evaluating regulations through ex-post regulatory reviews is primordial to ensure regulations in place are both relevant and adapted to their aims (OECD, 2017[1]). Following implementation, unintended consequences might emerge and need to be addressed. Alternatively, societal or technological changes may make a regulations obsolete. When ex-post regulatory reviews are not carried out, regulatory costs and red tape tend to incrementally increase, at the expense of businesses and citizens (OECD, 2017[1]). Ex-post reviews can therefore shed light on potential areas of improvement and therefore become a tool for regulatory planning.
The OECD Regulatory Policy Committee is discussing the development of Best Practice Principles for Ex Post Regulatory Reviews; based in the fact that despite their importance to improve the current regulation stocks and the design and administration of new regulations, they remain the less developed area of regulatory policy.
The document aims to provide general guidance in system governance, under the following overarching principles:
Regulatory policy frameworks should explicitly and permanently incorporate ex-post reviews as an integral part of the regulatory cycle.
A sound system for the ex-post review of regulation would ensure comprehensive coverage of the regulatory stock over time, while ‘quality controlling’ key reviews and monitoring the operations of the system as a whole.
Reviews should include an evidence-based assessment of the actual outcomes from regulations against their rationales and objectives, note any lessons and make recommendations to address any deficiencies. The Principles will address several governance dimensions, such as methodologies, public consultation and sequencing, capacity building and committed leadership for evaluations.
Core questions for consideration
Does your country’s current legal and policy framework foster systematic policy evaluation across government? Are the necessary mechanisms in place to ensure ex-post evaluations of regulations?
How does your government ensure the quality of evaluations across government? Are there specific mechanisms in place?
To what extent does your government engage with stakeholders during the policy evaluation process?
To what extent does your government provide a degree of transparency during the policy evaluation process?
How does your government promote the use of the findings of policy evaluations? Does your government consider evaluation results for budgetary discussions? How is performance information use dot improve policy and service design and delivery? In other words, does your government institutionalise feedback loops to optimise the impact of the evaluation of policy performance and sustain its influence on policy-making?
In the context of pursuing the Agenda 2030 SDGs, is policy evaluation and its feedback loops taking on added importance? How is your government preparing for assessing its progress in pursuing this Agenda and in reporting to citizens on this progress?
Has your government established measures to promote the realisation of ex-post regulatory reviews?
Additional resources
OECD legal instruments:
Recommendation of the Council on Regulatory Policy and Governance (2012) [OECD/LEGAL/0390]
Recommendation of the Council on Budgetary Governance (2015) [OECD/LEGAL/0410]
Other relevant OECD tools:
References
[66] Lázaro, B. (2015), Comparative study on the institutionalisation of evaluation in Europe and Latin America, Eurosocial.
[70] OECD (2020), Improving Governance with Policy Evaluation: Lessons From Country Experiences, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/89b1577d-en.
[60] OECD (2019), The Path to Becoming a Data-Driven Public Sector, OECD Digital Government Studies, OECD Publishing, Paris, https://doi.org/10.1787/059814a7-en
[47] OECD (2019), Governance as an SDG Accelerator : Country Experiences and Tools, OECD Publishing, Paris, https://dx.doi.org/10.1787/0666b085-en.
[52] OECD (2018), OECD Regulatory Policy Outlook 2018, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264303072-en.
[67] OECD (2018), Survey on Policy Evaluation.
[1] OECD (2017), Government at a Glance 2017, OECD Publishing, Paris, https://dx.doi.org/10.1787/gov_glance-2017-en.
[23] OECD (2016), Open Government:The global context and the way forward, OECD Publishing, Paris.
[69] OECD (2014), Chile’s Supreme Audit Institution: Enhancing Strategic Agility and Public Trust, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264207561-en.
[27] OECD (2012), Recommendation of the Council on Regulatory Policy and Governance, http://www.oecd.org/regreform (accessed on 4 October 2019).
[68] Treasury Board of Canada Secretariat (2016), Policy on results.
Notes
← 1. This definition is adapted from the Open Government: The global Context and the Way forward (OECD, 2016[23]), which is based on the “OECD DAC Glossary” in Guidelines for Project and Programme evaluation
← 2. Chapter 2 of this Framework addresses the broader issues related to evidence-informed policy-making.