This chapter reviews the objectives, achievements, and limitations of this stocktaking of principles and standards for the use of evidence in OECD countries. The chapter recognises the results of this report as a first global effort on the road to create a supportive institutional framework for evidence-informed policy-making. This work complements OECD’s previous work on building capacity for evidence-informed policy-making, as well as on a data driven public sector. In addition, the chapter discusses opportunities for future work. It points out the relevance of understanding current practice and exploring the potential for a shared approach to standards. This is an important step to ensure the reliability of public-sector decision-making processes and to ultimately nurture trust in decision-making in government.
Mobilising Evidence for Good Governance
4. Conclusion
Abstract
What has this report achieved?
This report looks to provide a stocktaking of principles and standards for the use of evidence. Although other mappings exist at a national level, this report is perhaps unique in attempting to map these approaches across OECD countries. As a result, the number of approaches enables lessons to be reached about the strengths and gaps of existing practice. In discussing a range of detailed case study examples, there is a wealth of material that can be used to enable countries to reflect on their own practice and facilitate knowledge sharing.
In the future, this mapping could facilitate a shared approach to evidence across the OECD and could represent an initial exploratory step in identifying and developing guidance at the international level in the area of evidence and evaluation. Some stakeholders have already called for a standardised approach to standards to avoid the confusion and duplication which can arise from the proliferation of evidence standards (Vine, 2018[1]).
At this stage, the current mapping remains descriptive. It opens the floor for future discussions on the issues that should be consider for the review and production of evidence. This report represents an important component of OECD’s work on evidence-informed policy and it also provides an important broadening of the discussion about evidence-based policy making, complementing the OECD work on a data driven public sector, which emphasises this area as an important policy cluster for the public sector (OECD, 2019[2]). The report should also be considered alongside work on Building capacity for evidence-informed policy-making (OECD, 2020[3]). In addition, a broader report focused on the role of policy evaluation in terms of institutionalisation, quality and use is also bringing comparative insights on a range of guidelines and approaches to structure the analysis and evaluation of public policies and programmes (OECD, 2020[4]).
Opportunities for future work
This report provides an initial resource for agencies and national experts thinking about the role of standards and principles in their evidence architecture. Reflecting both the nature of existing practice and the thematic interests of the project, the current review has primarily focused on public interventions and especially in the social policy area even if the ambition is to generate findings of general significance. This work could be complemented by future work exploring the synergies with emerging policy approaches in a data-driven public sector, where technological opportunities are also shaping a new environment for policy making. While the report makes reference to work on artificial intelligence for example, this is an area of rapid technological and policy development which could deserve future investigations.
The focus of the analysis of the standards of evidence reviewed in this report concerns primarily public interventions, as discrete decisions related to the concrete supply of public services, which is a bit more specific than the evaluation of policies as a whole, but is complementary. As a result, the report makes a specific but focused contribution to the full range of evidentiary considerations involved in policy-making. For example, it is recognised that evidence can play a role in the policy issues as developing a financial aid system in the area of development of in developing clinical guidelines for how physicians respond to an issue in the health area. However, such issues are not directly addressed in the current report and the communities of experts involved in development policy or in health care may either have developed similar stocktaking approaches. Engaging with such communities on the findings of the report may help to consolidate and improve common ground to strengthen evidence informed policy making. This is particularly true from a whole of government perspective, as there is a need to promote the evidence underlying public policy from across the range of policy fields.
Mapping existing practice is an important first step in understanding current practice and exploring the potential for a shared approach to standards. This report adopts a stocktaking neutral approach to offer a general framework to describe and understand existing approaches to principles and standards. While the case can be made that some of the methods for assessing evidence described in this report are more rigorous than others, consensus has yet to emerge among the experts. Therefore, this report does not rate the quality of the approaches nor does it aim at any form of comprehensive ranking. What the report offers is to encapsulate in some portable format a presentation of the key steps that are generally involved in the design, implementation and evaluation of public interventions. While the approach is not normative as such and that all of the steps do not need to be considered jointly, it is clear that ignoring any of the steps presented here would not be conducive in any way to sound public governance. Considering some of them, at least in some significant way and ensuring space in the public sector for such considerations, should still be seen as an essential step in consolidating trust by citizens in the soundness of public-sector decision-making processes. Whilst the report is comprehensive, future work could also make this technical material accessible to a wider audience. One option would be to turn the material into a toolkit for public servants and experts to be used and promoted throughout the public sector, in some portable or web-based format.
Countries might also consider updating their standards to reflect recent and ongoing developments in policy evaluation and experimentation. As discussed in the Effectiveness chapter, existing approaches to standards of evidence rarely address systems based approaches to evaluation, which deal with open-ended problems and issues (Askim, Hjelmar and Pedersen, 2018[5]; Tõnurist, 2019[6]). These tensions between traditional linear approaches to impact evaluation and systems based approaches are part of the OECD approaches under its Observatory for Public Sector Innovation: integrating these insights could represent a next step for updating standards of evidence in the future.
Finally the field of data itself is rapidly changing as a result of the COVID 19 pandemic, with researchers, economists turning to high frequency and big data to track the economy in real time,1 as traditional economic indicators might always be produced with some lag. This involves data from non-traditional data sources, including scraping the web, satellites, sensors, data from platforms. The OECD has started to analyse some of the current practices and applications (Johnstone N., 2019[7]). With the widespread deployment of the “Internet of Things”, such data sources are likely to growth exponentially in the future. The challenge will be to apply some quality filters and to ensure that some of these new data sources can be mobilised to formulate reliable analysis that can lead to meaningful conclusions for policy making.
References
[5] Askim, J., U. Hjelmar and L. Pedersen (2018), “Turning Innovation into Evidence-based Policies: Lessons Learned from Free Commune Experiments”, Scandinavian Political Studies, Vol. 41/4, pp. 288-308, http://dx.doi.org/10.1111/1467-9477.12130.
[7] Johnstone N., A. (2019), Using Digital Technologies to Improve the Design and Enforcement of Public Policies, https://doi.org/10.1787/99b9ba70-en.
[3] OECD (2020), Building Capacity for Evidence Informed Policy Making.
[4] OECD (2020), Improving Governance through Policy Evaluation: : Lessons From Country Experiences, OECD Publishing, https://doi.org/10.1787/89b1577d-en.
[2] OECD (2019), The Path to Becoming a Data-Driven Public Sector, OECD Digital Government Studies, OECD Publishing, Paris, https://dx.doi.org/10.1787/059814a7-en.
[6] Tõnurist, P. (2019), Evaluating Public Sector Innovation Support or hindrance to innovation?, OECD, Paris.
[1] Vine, J. (2018), Standardising standards The case for shared standards in the evidence sector Alliance for Useful Evidence.
Note
← 1. As highlighted in the OECD Work on New Approaches to Economic Challenges and the Innovation Lab. See. http://www.oecd.org/naec/projects/naecinnovationlabevents/.