First lessons from government evaluations of COVID-19 responses: A synthesis

Abstract

While the COVID-19 pandemic has hit different countries with varying intensity, responding to the crisis has presented an unprecedented challenge to most governments. In this context, evaluations provide critical tools to support real time sharing of lessons on what is working, what is not, what could work and for whom. This paper draws lessons from evaluations that governments have carried out themselves of their COVID-19 responses. It provides a synthesis of the evidence from 67 such evaluations produced in OECD countries during the first 15 months of the pandemic. These first evaluations show that many governments came to similar conclusions, and allow us to identify important insights that can feed into ongoing policy responses to the crisis – as well as increase future resilience.

Key messages

Evaluations provide critical tools to support real-time sharing of lessons on what is working, what is not, what could work and for whom. Despite the significant time and resource constraints that they were facing at the height of the COVID-19 crisis, governments understood this need to draw lessons from their policy responses to the pandemic. By April 2021, most countries in the sample analysed in this paper had already conducted evaluations on each of the key stages of the risk management cycle: preparedness, crisis management, and response and recovery.

These first evaluations show that many governments came to similar conclusions, thus allowing us to identify important insights that can both feed into ongoing policy responses to the COVID-19 crisis and iimprove future resilience. In particular, the evaluations analysed in this paper underline that:

• Pandemic preparedness was generally insufficient, particularly in light of the major human and financial costs associated with global health crises similar to the COVID-19 pandemic.

• Governments took swift and massive action to mitigate the economic and financial effects of the pandemic, but should carefully monitor the longer-term budgetary costs of these measures.

• Trust requires transparency, not only through frequent and targeted crisis communication, but, more importantly, by engaging stakeholders and the public in risk-related decision-making.

Important gaps remain in the evidence base available to date, however, and would warrant further investigation:

• There is insufficient evidence on critical sectors’ preparedness for pandemics, whereas early evaluations suggest that they were crucial to an effective crisis response.

• The effectiveness of lockdown and restriction measures should probably be further assessed, given their impacts on individual liberties.

• It is worth exploring the impact of lockdowns on domestic violence, alcohol consumption, youth, and mental health.

Finally, issues relating to policies’ proportionality and coherence are still largely under-explored by these first evaluations, despite their usefulness for policy debate, particularly when resources are scarce and cross-government co-ordination is crucial.

Introduction

While the COVID-19 pandemic has hit different countries with varying intensity, responding to the crisis has presented an unprecedented challenge to most governments in the OECD – both in scale and in the depth of impact on health, on the economy and on citizens’ well-being. At the same time, the pandemic has brought structural and social issues to light, including the erosion of public trust in government and in expert advice, which was compounded by a wave of mis- and disinformation. To respond to these challenges, OECD governments have deployed, relatively quickly, signficant human, financial and technical resources to manage and mitigate the impacts of the crisis.

Yet, there are many challenges still to be met if OECD countries are to successfully recover and build back better. Real-time sharing of lessons is critical for ongoing crisis response, for recovery efforts and for increasing governments’ resilience in the long run. In this context, evaluations help to understand what is working (or could work), what is not, and for whom. They also provide citizens and stakeholders information on whether public funds have achieved their intended objectives and are producing the expected results.

This paper responds to a request from OECD members to draw lessons from the evaluations that governments have carried out over the first year and a half of their COVID 19 responses. It is not the objective of this paper to evaluate countries’ COVID-19 responses. Rather, it is to provide useful insights in the hopes that these can feed into ongoing policy efforts and future resilience.

Scope, methodology and limitations

This paper synthesises available evidence produced by government institutions during the first fifteen months of the pandemic response.1 The OECD Secretariat called upon member countries to share evaluations undertaken by government institutions on the COVID-19 response and received over 256 contributions from 31 OECD countries by May 2021. Each dataset and report was then assessed based on three main criteria: whether it was an evaluation, whether it was related to the COVID-19 crisis and whether it was published or available to read. Therefore, non-evaluative exercises such as compliance audits or monitoring reports were not included in this analysis, insofar as they do not provide information on what has worked and why. In the end, 67 evaluations contributed by 18 OECD countries.2 were selected for this paper’s analysis.

To identify the lessons presented in this paper, the OECD Secretariat conducted a software-assisted qualitative content analysis, where recurrent themes were analysed based on a coding exercise. An initial review of the available evidence shows that most evaluations in the sample examine several topics, but that these can all be classified into three main policy areas that constitute the bulk of countries’ COVID-19 responses: pandemic preparedness, crisis management, and response and recovery measures.

The results from this analysis were then assessed in terms of whether there was sufficient and robust evaluative evidence to be able to provide interesting and useful lessons. Where possible, therefore, key insights are provided on the relevance, coherence, efficiency and effectiveness of government responses to the COVID-19 crisis.3

For the purpose of this analysis, first lessons describe both “good practices” identified in a single evaluation or country, and “key insights”, which draw consistent assessments and recommendations from several evaluations. Key insights look at both positive and negative results, i.e., lessons on what worked (for whom, why, under what circumstances) as well as what didn’t work, in order to draw a balanced picture of the crisis response. The insights also outline what could have been done better according to the evaluations.

However, all of the insights require further investigation. The conclusions presented should be considered preliminary, as the pandemic is ongoing, the available evidence does not fully capture countries’ recovery efforts, and countries continue to evaluate their policy responses.

This paper identifies 14 key insights from the 67 evaluations.

Figure 1. Key insights from evaluations of COVID-19 responses

Framing evaluations of COVID-19 responses

The OECD Recommendation of the Council on the Governance of Critical Risks identifies several major phases of any risk management cycle (OECD, 2014[1]) and recommends that governments adopt strategies for, and develop policies in, each phase of this cycle to boost their resilience to risks. To understand what was done well – or not – in preparing and responding to the COVID-19 pandemic, evaluations must therefore draw lessons on the relevance, coherence, effectiveness, efficiency and impact of policies put in place in each phase of this risk management cycle.

The Framework for Evaluating COVID-19 Responses developed for the purpose of this analysis identifies three main types of policy responses that governments need to evaluate to better understand what has worked well and what has not in managing the pandemic. These three main policy responses correspond to the major phases of the risk management cycle: pandemic preparedness, crisis management, and response and recovery (see Figure 2). The empirical relevance of this framework has been tested in the sample of contributing OECD countries, and evidence shows that a majority of them (11/18) have conducted at least one evaluation on each of these main types of policy response.

Figure 2. Framework for evaluating COVID-19 responses

Source: Authors

Pandemic preparedness refers to governments’ ability to anticipate a pandemic before it materialises and prepare for a global public health emergency, by developing the right knowledge and capacities (OECD, 2015[2]). Indeed, understanding hazards and threats, and building capacities for risk foresight and assessment, allow countries to better target prevention policies and mitigation programmes to reduce their vulnerability to risk (OECD, 2015[2]). Additionally, risk management protocols should be set up to establish standard operating procedures and pre-defined plans for dealing with pandemics (OECD, 2015[2]). Critical sectors, such as pharmaceutical industries and healthcare providers, also have a key role to play in improving countries’ resilience to pandemics. To understand the extent to which countries were prepared for the pandemic, this paper looks at how evaluations assess governments’ pandemic anticipation capacities and health emergency protocols, as well as the preparedness of critical sectors.

The next major phase of the risk cycle where evaluations may draw lessons is crisis management, that is, the policies and actions that governments deployed to deal with the crisis once it materialised. Crisis management entails responding appropriately, at the right time, and in a co-ordinated manner across government (OECD, 2015[2]). Indeed, evidence from the OECD shows that managing modern, complex crises likes a pandemic involves many actors beyond emergency services. This requires co-ordination and creates governance challenges, since crisis management operations are often led and managed by centres of government but carried out at sub-national levels (OECD, 2015[2]). Crisis management also requires clear communication to the public and transparency in decision making, particularly as large-scale crises can have a critical impact on people’s trust in government (OECD, 2015[2]). This paper looks at what the evaluations conclude about governments’ crisis communication strategies, the governance arrangements they have put in place to manage the crisis, and the measures they have deployed to co‑ordinate a whole-of-society response.

Finally, response and recovery policies are aimed at, first, mitigating the impacts of the pandemic and economic crisis on citizens and businesses and, subsequently, supporting the economic recovery and reducing welfare losses. These policies include lockdown and restriction measures taken in order to limit the spread of the virus; economic and financial support to households, businesses and markets to lessen the impact of the economic downturn; health measures aimed at protecting and treating the population; and social policies aimed at protecting the most vulnerable populations.

The following section maps the existing evidence on these three types of policy response. The rest of the paper then offers an overview of the main insights from the evaluations.

Where do we need more evidence?

The majority of respondent countries (11/18, see Annex Table 1.A.1 in Annex 1.A) have conducted at least one evaluation on each of these main types of policy response. However, countries have not (yet) analysed them to the same extent and in the same way. First, fewer evaluations in the sample pertain to the critical phases of the crisis, namely pandemic preparedness (28 evaluations) and the management of the crisis response (34), than evaluations relating to response and recovery measures (62). This could be because a wider range of measures were deployed in the response and recovery phase.

A more detailed analysis of the evaluations in the sample shows that several types of policy response could be further examined by countries (see Table 1):

• There is less evidence on the preparedness of critical sectors, namely the health sector (11 evaluations). Further evidence would be useful, as the crisis has shown how crucial the health sector is to countries’ resilience to pandemics. Given that health systems are still in crisis mode, evaluations may come when times are calmer.

• Measures deployed to co‑ordinate a whole-of-society response, such as mechanisms for co‑operation across different levels of government or for engaging stakeholders in key decision-making processes, have not been sufficiently analysed (16 evaluations analyse this topic). In particular, countries may wish to draw further lessons on how to work with sub-national governments and local actors, and on how well their decentralisation models are adapted to managing important shocks such as the COVID-19 pandemic.

• 27 evaluations analyse lockdown and restriction measures, 23 of which look at actual (rather than intended) effects. The evaluations provide some evidence about the relevance and coherence of these measures, and underline the need for more evidence on their impact – especially in areas such as the economy and the environment.

• Economic and financial measures are the most evaluated in the sample (45 evaluations in total). These evaluations focus mostly on process, adequacy and intermediate effects and do not measure impacts.

• Countries’ health policies have been assessed in 27 evaluations, mostly in terms of procedural issues related to measures for controlling infection and monitoring intensive care capacities during the acute phase of the crisis. Several of these evaluations suggest more evidence is needed on the impacts of the pandemic on health. Moreover, other important health issues are not mentioned at all in the sample of 67 evaluations: for example the impact of the pandemic on mental health, especially that of young people, perhaps because these impacts may not be apparent for a number of years.

• Whilst 35 evaluations look at social policies, they contain little evidence on the social impacts of the crisis, whether it be on poverty levels or inequality. In particular, topics such as the educational impact of school shut downs which are mentioned only in passing need to be further explored.

Table 1. Mapping the existing evaluations of COVID-19 responses

Category of policy response	Types of policies	Total evaluations	Ex post and ex durante evaluations
EMERGENCY PREPAREDNESS	Disaster Risk Anticipation and Foresight	31%	31%
	Emergency protocols	33%	33%
	Preparedness of Critical Sectors	16%	16%
MANAGEMENT OF CRISIS RESPONSE	Crisis Communication	30%	30%
	Governance of Crisis Response	33%	31%
	Whole-of-Society Response	24%	24%
RESPONSE AND RECOVERY	Lockdown and Restrictions	40%	34%
	Economic and Financial Support	67%	46%
	Health policies	40%	33%
	Social policies	52%	40%

Note: n= 67 evaluations, of which 36 ex post evaluations and 17 ex durante. A red cell in the evaluation column signifies that less than 25% of the 67 evaluations address this topic, e.g. only 16% of evaluations addressed the topic of "Preparedness of Critical Sectors". Orange cells refer to the range 25-34%, yellow to 35-44%, light green to 45-54% and dark green cells to over 54% of evaluations addressing the respective topic.

Source: OECD 

As countries are still in the early stages of evaluating their COVID-19 policy responses, more evidence on these topics may become available in the coming months and years. The following sections present the broad lessons that can be drawn from these evaluations for each of the three policy areas.

What lessons can we draw from evaluations of COVID-19 responses?

Pandemic preparedness

Most countries in the sample (13 out of 18) have examined the extent to which the government was adequately prepared to manage such a global shock. Specifically, the evaluations on this topic focused on three main types of policy or measure that contribute to pandemic preparedness:

• Risk management protocols to follow during pandemics (i.e. ‘pandemic management protocols’);

• government capacity for risk anticipation, through foresight and risk assessment (i.e. ‘risk anticipation capacities’);

• the overall preparedness of critical sectors for pandemics (i.e. ‘critical sector preparedness’, see Figure 3).

Figure 3. Main focus of evaluations of pandemic preparedness

Note: n =67. 67 evaluations were conducted across 18 countries. 11 evaluations examined at least one issue related to the preparedness of critical sectors to pandemics, in particular relating to the preparedness of the health sector. 22 evaluations examined at least one aspect of issues related to pandemic management protocols. 21 evaluations examined at least one issue related to risk anticipation capacities, either forecast and assessment, such as early warning systems, or national and sectoral risk assessments, and emergency planning.

Source: OECD

Pandemic management protocols

Around one-third of all evaluations (22) analysed pandemic management protocols. These evaluations place a strong emphasis on the existence of procedures and guidelines to follow in the event of a pandemic (Box 2). Indeed, as pointed out in several evaluations, both clearly defined responsibilities – especially in terms of leadership – and risk management frameworks, are essential for preparing for major unknown and unexpected risks such as pandemics. This is why the 2005 World Health Organisation’s International Health Regulations mandate countries to have preparedness plans for public health crises, and to regularly evaluate and update them. OECD data shows that, in general, pandemic protocols were at the heart of pandemic response capacities in governments (OECD, 2021[3]). For example, the Office of the Auditor General in Canada examines the role of such protocols in the Public Health Agency Canada’s overall preparedness to pandemics (see Box 1).

Risk anticipation capacities

21 evaluations were conducted on government capacities to anticipate risks, whether through early warning systems, risk assessment exercises, foresight, or building critical material stocks (e.g. personal protective equipment). While pandemics featured prominently in national risk assessments prior to 2020 (OECD, 2018[4]), preliminary lessons show that countries nonetheless had insufficient capacity to anticipate shocks of this magnitude. This is, in part, due to data collection and expertise deficiencies (see Box 2).

Data from the 2018 OECD report on Assessing Global Progress in the Governance of Critical Risks suggests that early warning systems and other anticipation tools, while costly, are crucial for risk management (OECD, 2018[5]). OECD countries may wish to further invest in evaluating this area to improve risk anticipation capacities.

Critical sectors preparedness

Finally, data from respondent countries suggest that robust evidence is still lacking on the effectiveness of critical sectors’ preparedness for risk (see Figure 3). As early as 2011, OECD identified the need for countries to improve their national critical infrastructures in anticipation of global pandemics (OECD, 2011[6]). Nevertheless, only a few OECD countries (7 out of 34) considered “health care and public health sector” as critical infrastructure sectors when surveyed in 2018 (OECD, 2018[5]). There are no doubt many lessons still need to be learned from the COVID-19 pandemic on this issue.

Box 1. Good practice: guidelines and protocols for pandemic response in Canada

An evaluation by the Office of the Auditor General of Canada

One report by Canada’s Auditor General assessed the overall preparedness of the Public Health Agency of Canada, which is the lead federal organisation for planning and co‑ordinating the national response to infectious diseases that pose a risk to public health. The evaluation recognises how the agency has been able to learn from the 2009 H1N1 pandemic by taking steps to develop protocols and national guidance for potential pandemics. The evaluation encourages the Public Health Agency to continue the practice of updating its protocols to prepare for future pandemics.

Source: (Canadian Office of the Auditor General, 2021[7])

Yet, as several respondent countries underline, standard operating procedures in crisis management plans are not enough to manage events such as global pandemics and must be complemented by other tools, such as inter-agency co‑operation mechanisms, staff lending, or emergency training or drills (OECD, 2018[5]) (OECD, 2015[2]). In this regard, the OECD’s upcoming work on “Delivering results across agencies” may offer insights on how countries have co‑operated across agencies to reinforce emergency response capacities and have collaborated across government.

Box 2. Key insight 1: Investment in risk anticipation capacities and in critical sectors was insufficient in most countries for adequate pandemic preparedness

Policy context:

The COVID-19 pandemic was not an unexpected ‘black wwan’ event, as most national risk assessment frameworks had anticipated some form of pandemic. Yet, many OECD countries overlooked lessons that could have been drawn from previous global virus outbreaks, such as SARS or H1N1, and as a result, were not adequately prepared. Pandemic preparedness requires risk anticipation and assessment, as well as strategic foresight capacities. Well-developed emergency plans and procedures also allow governments to prepare and to equip critical sectors when a risk materialises.

Main assessments and recommendations:

• Several evaluations point to the need for better risk anticipation capacities and more systematic horizon scanning, to better anticipate shocks of this magnitude. This requires, in particular, having the necessary data infrastructure to monitor infection rates and other key health indicators, as well as having the necessary expertise inside government to review this data and provide advice to decision-makers (Danish Parliamentary Committee on COVID-19, 2021[8]) (US Government Accountability Office, 2021[9]) (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Canadian Office of the Auditor General, 2021[7]).

• Another important aspect of andemic preparedness was the critical stock of equipment, such as personal protective equipment (PPE) or testing capacities. Evaluations examined the relevance of the existing stock of masks and the efficiency of distribution, but results among countries are very heterogeneous (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]) (Swiss Federal Chancellery, 2020[11]) (Belgian Health Care Knowledge Centre, 2020[12]) (UK National Audit Office, 2020[13]) (New Zealand Controller and Auditor General, 2021[14]) (Australian National Audit Office, 2020[15]) (US Government Accountability Office, 2020[16]) (US Government Accountability Office, 2021[17]) (US Government Accountability Office, 2021[9]) (US Government Accountability Office, 2020[18]). Recommendations regarding PPE include diversifying supply sources, constituting stocks based on thorough risk assessments and monitoring national production capacities and stocks regularly (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]) (Swiss Federal Chancellery, 2020[11]) (UK National Audit Office, 2020[13]) (New Zealand Controller and Auditor General, 2021[14]) (Australian National Audit Office, 2020[15]).

• Respondents also underline countries’ overall lack of emergency response planning and training, in particular with regard to pandemic response (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Canadian Office of the Auditor General, 2021[7]); (Danish Parliamentary Committee on COVID-19, 2021[8]) (New Zealand Controller and Auditor General, 2021[14]) (Australian NAO, 2020[19]) (US Government Accountability Office, 2020[16]). In France, for instance, risk assessments had focussed particularly on terrorist attacks and few pandemic response drills had been conducted in recent years (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]). Respondents also generally stress the need to have detailed and up-to-date operational plans and processes that describe the roles of different actors in responding to national health emergencies. The evaluations recommend that these plans be developed in co‑operation with the main crisis response stakeholders (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Canadian Office of the Auditor General, 2021[7]); (Danish Parliamentary Committee on COVID-19, 2021[8]) (New Zealand Controller and Auditor General, 2021[14]).

Note: Key insights synthesise evidence and early lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Crisis management

At least one aspect of government management of the COVID-19 crisis response has been analysed in 12 countries in the sample. These evaluations have focused on three main types of policies or measures:

• the governance arrangements put in place to manage the crisis (i.e. ‘governance arrangements’);

• crisis communication strategies and tools (i.e. ‘crisis communication’);

• and mechanisms to promote a whole-of-society response in managing the pandemic, encompassing stakeholders, citizens, sub-national governments, the private sector, etc. (i.e. ‘whole-of-society response’, see Figure 4).

Figure 4. Main focus of evaluations of crisis management measures

Note: n =67. 67 evaluations were conducted across 18 countries. 11 evaluations focused on at least one aspect of the governance arrangements related to the crisis response. 20 evaluations examined at least one aspect of crisis communication. 16 on issues related to whole-of-society response.

Source: OECD

Governance arrangements

22 evaluations in the sample looked at how governments have organised their management of the crisis, whether by setting up structures to co‑operate among agencies and to gather scientific advice on the pandemic, or by deploying tools to facilitate internal communication. Overall, these evaluations underscore the rapidity and agility with which governments have put these mechanisms in place. Nonetheless, there is scope to streamline some of these structures and tools and make their decision-making processes more transparent and accountable (see Box 3, Box 5, and Box 6 for more detail on key insights on these topics).

Data collected by the OECD shows that there is ample evidence regarding the effectiveness of inter-agency co‑operation in managing the crisis (see Box 3) (OECD, 2020[20]). Many evaluations outline the ways in which governments set up specific bodies or committees to lead inter-agency co‑operation in managing the crisis on specific policy topics.

Box 3. Key insight 2: Interagency co-operation requires commitment from leadership and fit-for-purpose governance structures with clear mandates

Policy context:

Given that government responses to the crisis have required the mobilisation of resources in virtually all policy domains and involved a wide range of actors in different policy areas, co‑operation among government institutions and agencies has been vital to ensure a coherent response. Governments put in place, if needed, structures and mechanisms to ensure a co‑ordinated response (almost half of OECD countries deployed new institutional arrangements to manage the pandemic (OECD, 2021[3])), and developed protocols to clarify the responsibilities of each actor in managing the crisis response. Overall, the structures used by governments to manage the pandemic illustrate the complexity of this shock, in which traditional crisis management authorities, such as the centre of government, must contend with multiple stakeholders with potentially overlapping responsibilities (OECD, 2020[20]).

Main assessments and recommendations:

• Respondent institutions note the flexibility and rapidity shown by many governments in activating and setting up governance arrangements to facilitate inter-agency co‑operation to manage the crisis. These committees, teams and tasks forces were often multi-disciplinary and represented several ministries, but may have focused on one aspect of the crisis response, such as health or economic measures (Australian National Audit Office, 2020[21]) (French mission to control the quality of the management of the health crisis, 2020[22]) (Danish Parliamentary Committee on COVID-19, 2021[8]).

• On top of existing crisis management arrangements, many countries created ad hoc structures that played a co‑ordination role on specific topics. The multiplicity of structures meant that there could be an overlap in staff attending each meeting, which improved communication across the different groups/ committees (Danish Parliamentary Committee on COVID-19, 2021[8]) (Australian National Audit Office, 2020[23]) (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Canadian Office of the Auditor General, 2021[7]). On the other hand, this overlap meant that division of labour and chains of command were not always clear to the major stakeholders involved (Australian National Audit Office, 2020[23]).

• As a result, evaluations underline the importance of clarifying responsibilities and mandates of each of these crisis management groups at the outset. This includes making the decision-making processes of these groups clear, as well as their relative hierarchies (who has the final say and who is responsible for the decisions) and their organisational charts (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]) (Swiss Federal Chancellery, 2020[11]) (US Government Accountability Office, 2021[24]).

• Several evaluations highlight the need to send high-level civil servants and senior staff to these crisis management groups, to ensure their ministries are engaged in decision-making (Australian National Audit Office, 2020[21]) (French mission to control the quality of the management of the health crisis, 2020[22]). Crisis management structures benefit from strong leadership. Having a high level of seniority helps give more weight to the discussions, ensures stronger accountability and facilitates speedier decision-making (Belgian Health Care Knowledge Centre, 2020[12]).

Note: Key insights synthesise evidence and lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

These evaluations show varying results regarding the effectiveness of these measures (see Box 4). Mostly, the evaluations underline the need to streamline these structures to avoid overlapping roles, and to ensure a high level of representation from relevant agencies to give their decisions greater authority. These recommendations are very much in line with the OECD Recommendation of the Council on the Governance of Critical Risks, which recommends that Members should:

“assign leadership at the national level to drive policy implementation, connect policy agendas, and align competing priorities across ministries and between central and local government through the establishment of multidisciplinary, interagency approaches (e.g. national co‑ordination platforms) that foster the integration of public safety across ministries and levels of government.” (OECD, 2018, p. 125[5]).

Box 4. Good practices: a focus on inter-agency co‑operation in Australia, Belgium and France during the COVID-19 crisis

Australia: Relying on flexible task forces to increase the national medical stockpile

In Australia, the Department of Industry, Science, Energy and Resources and the Department of Health co‑operated to manage an increase in the national medical stockpile. The departments used a flexible taskforce approach to manage the procurement and engage executive management in decision-making. They established a process for managing conflicts of interest in both departments, which was deemed fit-for-purpose by the Australian National Audit Office.

France: Having representation at the right level of seniority

The French government created an Inter-ministerial Crisis Unit in March 2020 to co‑ordinate the actions of various ministries in their collective response to the pandemic. In spite of its inter-ministerial nature, the Unit was perceived as being under the tutelage of the Ministry of the Interior. As a result, many ministries either did not participate in the Unit’s meetings or chose to send ‘junior’ colleagues who were unable to make commitments on behalf of their ministries.

The evaluation by the “Mission on the quality control of the management of the health crisis” recommended that all ministries participate at a proper level of seniority to allow for better inter-agency co‑operation within the Inter-ministerial Crisis Unit.

Belgium: Assessing the management of hospital surge capacity in the first wave of the COVID-19 pandemic

In Belgium, the Hospital & Transport Surge Capacity (HTSC) committee was set up in March 2020. As an ad hoc working group of the standing Risk Management Group (RMG), its mission was to “monitor the number and type of COVID-19 patients admitted to acute hospitals […] and to propose decisions to the RMG on the organisation of adequate hospital care” (Belgian Health Care Knowledge Centre, 2020[12]). As part of its mission, the HTSC intensively interacted with other bodies and working groups that were involved in the management of the COVID-19 crisis.

According to the evaluation conducted by the Belgian Health Care Knowledge Centre, various factors contributed to the successful functioning of the committee: respect of confidentiality rules by participants, strong leadership, agreement about what was communicated and respecting these agreements.

Source: (Australian National Audit Office, 2020[15]) (Belgian Health Care Knowledge Centre, 2020[12]); (French mission to control the quality of the management of the health crisis, 2020[22])

Another important policy issue assessed by contributing countries and institutions is that of internal communication among agencies and different levels of government (see Box 5).

Box 5. Key insight 3: Adaptive crisis management relies on predefined and effective internal communication and reporting processes, which can prove challenging among levels of government

Policy context:

Effective internal communication is a key component for successful inter-agency co‑operation. This includes communication among actors who may not be used to working together during normal times. During the COVID-19 crisis, government agencies had to work with many different public or quasi-public entities to build a coherent response to the pandemic.

Main assessments and recommendations:

• Communication among national agencies and actors was generally positively assessed for those who had previous experience in working together. Several evaluations reported that this communication sometimes relied more on personal relationships than established communication channels, nevertheless (Swiss Federal Chancellery, 2020[11]) (US Government Accountability Office, 2021[9]).

• However, evaluators reported that conveying relevant information in a timely manner to the local or regional level and to frontline staff was a major challenge encountered by governments in their response to the crisis (Swiss Federal Chancellery, 2020[11]) (French mission to control the quality of the management of the health crisis, 2020[22]) (UK National Audit Office, 2020[13]). Indeed, managing the crisis required national governments to co‑ordinate with a variety of national, regional and local actors that have different mandates and do not usually collaborate. Some countries, such as France (French mission to control the quality of the management of the health crisis, 2020[22]), took a decentralised approach to passing on key health information and instructions to local governments and health actors. This resulted in loss of information and signficant delays. This particular evaluation points to the need to simplify information channels and chains of command in times of crisis by having the national government talk directly to the local level. OECD data suggest that, overall, dynamic and interactive forms of internal information are most effective. Indeed, while it is important to “push information out” through, for example, emails or internal websites, using both traditional and new digital platforms for internal communication can lead to greater buy-in from stakeholders (OECD, 2021[25]).

• Several evaluations underlined the important challenges encountered by national governments in collecting data from multiple health actors involved in the crisis response (national agencies, sub-national governments, hospitals). These challenges included the lack of clear standards and homogeneous processes for reporting data to the national government, the variety of information technology systems used by these different actors, and the fact that surveys did not reflect the realities of the field (Swiss Federal Chancellery, 2020[11]; Belgian Health Care Knowledge Centre, 2020[12]; Canadian Office of the Auditor General, 2021[7]) (US Government Accountability Office, 2021[9]).

• Reports showed the value of building data standards and processes together with local health actors, to make sure that they are understood by all relevant stakeholders and adapted to operational realities. For example, the GAO recommended that the Department of Health and Human Services establish an expert committee made up of health care professionals and public agencies to review and inform data collection and reporting standards for key health indicators (US Government Accountability Office, 2021[9]). Having a clear point of contact in the national administration for questions about data standards (for example to understand the definition of ICU bed availability) was also recommended in several evaluations. These conclusions are in line with the 2021 OECD Recommendation on Enhancing Access to and Sharing of Data, which recommends promoting inclusive representation and engaging relevant stakeholders in the data ecosystem.

• Contributors also emphasised the interoperability of IT systems among health actors, or between emergency crisis response actors at the national and local levels, as an important issue for further investigation (Swiss Federal Chancellery, 2020[11]; French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]). The report by the Swiss Federal Chancellery on “Crisis management during the COVID-19 pandemic” mentioned that such IT systems should be built so that they can be used for internal management and not just for reporting purposes (this is especially important given time constraints during crises).

Note: Key insights synthesise evidence and lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

The final topic related to the governance of crisis management responses examined in the evaluations is the role of experts and scientific bodies in providing advice to governments during the ‘sense-making’ period when scientific information and advice feeds into the decision-making process (OECD, 2020[20]; OECD, 2018[5]). In the sample of evaluations analysed, 13 evaluations by 8 countries studied scientific advice, notably from public health experts, during the COVID-19 crisis response phase. According to the French national mission to control the quality of the management of the COVID-19 crisis, for example, the dispersion of scientific expertise across public health bodies may have made it more difficult to pool knowledge and skills. The evaluation also suggests that the creation of an agile and responsive scientific council would offset existing institutional rigidities (French mission to control the quality of the management of the health crisis, 2020[22]). Overall, the evaluations noted that scientific and expert committees provided important advice to governments on complex matters, but that a wider range of expertise – in particular that of practitioners and civil society – could be reflected in these bodies (see Box 6).

Box 6. Key insight 4: Scientific advisory bodies provided valuable evidence to inform decision-making during this complex crises, but could rely on more varied sources of expertise

Policy context:

The crisis has put governments in a challenging situation where they had to ensure clear, trusted and legitimate decision-making processes informed by the best available evidence, while there were many “unknown unknowns” and very little time for dialogue and gathering information. Governments were also faced with the need to review and synthesise information from multiple sources, and to use it to feed into governments’ responses to the COVID-19 crisis. In many cases, putting in place ad hoc structures to gather scientific advice was seen as the best approach given such a high level of uncertainty (OECD, 2020[20]).

Main assessments and representation:

• Many respondent countries created ad hoc scientific committees to provide advice to the government on managing the COVID-19 pandemic, sometimes as early as January 2020 (New Zealand Parliament, 2021[26]) (US Government Accountability Office, 2021[9]) (Belgian Health Care Knowledge Centre, 2020[12]; Swiss Federal Chancellery, 2020[11]). The evaluations generally assess this approach as having improved government decision-making on complex issues. In some cases, these committees have also filled an expertise and analysis gap in government institutions (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]). However, several evaluations note that these scientific committees or groups had difficulties accesssing data or information (French mission to control the quality of the management of the health crisis, 2020[22]) (New Zealand Parliament, 2021[26]) (US Government Accountability Office, 2021[9]). Furthermore, prior to the COVID-19 pandemic, countries had already recognised the need to strengthen their domestic capacities for scientific advice in crises, by developing national mechanisms for providing scientific advice to governments in emergencies – in particular for sense-making in complex and novel crises. (OECD, 2018[27])

• Several evaluations also point to the fact that representation from the field or civil society organisations was lacking in these committees (Belgian Health Care Knowledge Centre, 2020[12]; Swiss Federal Chancellery, 2020[11]; French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]). Openness about evidence bases and decision-making processes is crucial when authorities are dealing with health risks characterised by great uncertainty, which includes formalising processes for consulting scientific committees or advice (Danish Parliamentary Committee on COVID-19, 2021[8]).

Note: Key insights synthesise evidence and lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Crisis communication

The evaluations in the sample also provide good-quality evidence on how governments communicated to a variety of stakeholders during the crisis. Indeed, issues related to government communication of COVID-19 responses have been examined in 20 out of 67 evaluations from 11 countries.

Many governments in OECD countries made unprecedented use of communication channels and information technologies to reach out to citizens about the spread of the virus and the policies undertaken to control infection, and to share instructions on how to comply with lockdown and social distancing requirements (OECD, 2020[28]; OECD, 2020[29]; Wendling, 2013[30]). Some governments also worked with local actors and civil society organisations to communicate on the pandemic. For example, the Government of Canada worked with the leaders of Indigenous peoples and other ethnic minorities to deliver the government’s messages to their respective communities.

Governments may have wanted to pay special attention to the impact of these new communication channels and methods on the effectiveness of their crisis management and on the acceptability of measures rolled out to mitigate the impact of the crisis. This is the case, for example, in Switzerland, where the Swiss Federal Chancellery examined the impact of its communication strategy during the height of the crisis (see Box 7). This is an important exercise, as 58% of centres of government that responded to the OECD 2020 Understanding Public Communication Survey identified crisis communication as the most challenging communication competency, namely due to human resource and co-ordination-related issues. A majority of countries stated in the same survey that they deploy surge capacity or staff to support their communication during crises.

Box 7. Good practice: diversifying communication channels during a crisis in Switzerland

The Swiss government used various external communication channels to provide information to its citizens about the current situation and infection control measures. In addition to frequent press conferences by the Federal Council and experts, the federal administration used poster campaigns, webpages, social media and the “ALERTSWISS” app to disseminate information. For example, during the highest alert level, the Federal Council gave three press conferences per week, complemented by press briefings with specialists every second week.

The evaluation by the Swiss Federal Chancellery concludes that using a variety of communication channels allowed the govenrment to reach a large portion of the population and was particularly effective.

Source: (Swiss Federal Chancellery, 2020[11])

Evaluations relating to crisis communication looked at the consistency of the information communicated by governments (13 evaluations), whether the information was well received by the public (10 evaluations), and what efforts governments made to target specific or hard-to-reach populations (12 evaluations).

Overall, the evaluations concluded that governments have made considerable efforts to provide easily available information to the public, but that information was not always timely, consistent and comprehensive (see Box 8).

Box 8. Key insight 5: More targeted, informed and coherent messaging is needed to foster trust

Policy context:

A clear and consistent communication strategy was quickly recognised by many governments as crucial for responding to the crisis. The general population’s acceptance of decisions has direct consequences for compliance with mitigation measures, particularly in contexts where mis- and dis-information spreads rapidly (OECD, 2020[29]). Governments greatly increased the frequency of their external communication to the public and relied on a variety of channels to do so, from press conferences to social media and online databases. Well-established risk communication practice (OECD, 2016[31]) highlights the importance of communicating about the actions each person can take to reduce his or her own risk exposure and that of others.

Main assessments and recommendations:

• Several evaluations deemed external communication to have been swift and regular enough, relying on a variety of both traditional and digital communication channels, and that the mix of high-level and technical messaging particularly contributed to increasing trust and acceptance within the population (Swiss Federal Chancellery, 2020[11]) (US Government Accountability Office, 2021[17]) (Danish Parliamentary Committee on COVID-19, 2021[8]). In fact, data from the OECD Survey on Understanding Public Communication indicate that most countries defined crisis communication protocols, which may explain why crisis communication was effective in many countries during the COVID-19 pandemic.

• Some governments paid special attention to producing communication material in multiple languages (whether in the form of social media posts, podcasts, or apps) and to targeting specific audiences, such as hard-to-reach populations (US Government Accountability Office, 2020[16]) (US Government Accountability Office, 2021[9]) (Australian National Audit Office, 2020[21]) (Danish Parliamentary Committee on COVID-19, 2021[8]). Evaluations stress that this is a good practice insofar as messaging should be inclusive and tailored for each audience to ensure that crisis communication is effective.

• Yet, governments encountered important issues linked to the consistency and clarity of messages relating to critical health issues. The increased frequency of communication and rapidly changing narratives may have led to mixed messages and hindered mitigation and recovery efforts (US Government Accountability Office, 2021[17]) (US Government Accountability Office, 2021[32]) (Belgian Health Care Knowledge Centre, 2020[12]) (Danish Parliamentary Committee on COVID-19, 2021[8]). For instance, rapid changes in guidance on wearing masks or use of PPE, meant that frontline staff did not always comply with them (UK National Audit Office, 2020[13]) (US Government Accountability Office, 2021[17]) (Belgian Health Care Knowledge Centre, 2020[12]). Evaluations underline that clear and consistent communication is essential during emergencies. Some evaluations note that limited and targeted messages may be more effective than large outreach strategies (US Government Accountability Office, 2021[32]) (Danish Parliamentary Committee on COVID-19, 2021[8]). This was also considered important for promoting public uptake of COVID-19 vaccines (US Government Accountability Office, 2021[32]). Furthermore, governments could consider implementing more collaborative forms of crisis communication, moving away from a one-way, top-down approach, and engaging in conversations with a variety of stakeholders to co-construct messages and learn from their concerns and experiences.

• Some evaluations also point out that the media may have received scientifically incorrect information from the government, which created uncertainties and may have affected the uptake of mitigation measures (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]) (Swiss Federal Chancellery, 2020[11]) (US Government Accountability Office, 2021[17]). This was the case regarding the effectiveness of masks as personal protective equipment, or the efficacy of hydroxychloroquine as a treatment for COVID-19 patients. In order to reduce confusion and increase trust in government responses, evaluations highlight the importance of disclosing the scientific rationale for health guidelines. The OECD Principles of Good Practice for Public Communication Responses to Help Counter Mis- and Disinformation stress the importance of transparency and honesty in communication (OECD, forthcoming[33]).

• Several evaluations point to the usefulness of publicising health and infection data on a single national dashboard to communicate the current status of the pandemic across different areas and show the scientific rationale for policy responses to the crisis (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]) (US Government Accountability Office, 2021[9]).

Note: Key insights synthesise evidence and lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Whole-of-society response

16 evaluations looked at how governments have involved the ‘whole-of-society’ in their COVID-19 responses. These evaluations examine how different levels of government have worked together to manage the pandemic response, how citizens and stakeholders have been engaged in decision-making processes related to crisis response, and what measures could have helped safeguard democratic rules.

Federal respondent countries (Australia, Belgium, Canada, Switzerland, the United States) put a greater emphasis than non-federal countries on evaluating the co‑operation among levels of government, 4 for example, regarding the distribution of responsibilities between national and sub-national governments during different stages of the crisis. Yet, OECD data suggest that sub-national governments everywhere were often at the front line of the COVID-19 crisis, as they are typically responsible for healthcare, social and transportation services. Co‑operation with local administrations was thus both essential and complex, given that national governments had to contend with the varying local realities of sub-national governments (OECD, 2020[34]). In many countries, the quality of co-ordination among levels of government has been a key determinant in the effectiveness of the response to the health and economic crisis (OECD, 2020[34]). This is the case in Austria, for example, where the National Crisis and Disaster Management Committee (SKKM) co‑ordinated the COVID-19 response among federal ministries, provinces, rescue organisations and the media. Further efforts to gather evidence on the effectiveness of multi-level governance response to crises may be required, especially since preliminary lessons suggest that countries encountered important challenges in this regard (see Box 9).

Evaluations also suggest that governments need to make more of a concerted effort to involve stakeholders and citizens in decision making for crisis management, particularly considering that democratic accountability mechanisms may have been impaired in most countries during the acute phase of the pandemic (see Box 9). Making information widely available to citizens can enable and empower other public actors to reuse it for the benefit of citizens. For example, in Canada, the civil society organisation ‘COVID-19 Canada Open Data Working Group’ created a single interactive dashboard to communicate data released by public authorities to citizens (COVID-19 Canada Open Data Working Group, 2021[35]). Governments can also look at interesting examples from sub-national governments that have conducted online deliberative processes during the pandemic, such as in the US state of Oregon, where a citizen assembly was convened to discuss the COVID-19 recovery.

Box 9. Key insight 6: Governments could involve civil society, the private sector and local actors more to increase transparency in decision making and facilitate the implementation of crisis management responses

Policy context:

The pandemic has affected every sector of society and every level of government. First, sub-national authorities are responsible for crucial aspects of pandemic management, such as containment measures, healthcare, social services, and economic development (OECD, 2020[34]). Businesses and civil society organisations also supported national efforts in many countries – whether by repatriating citizens from abroad or distributing personal protective equipment to citizens. At the same time, in many countries, emergency measures have been accompanied by a restriction of civic freedoms and rights (OECD, 2020[28]). Indeed, at the start of the pandemic, 20 out of 38 OECD countries declared a state of emergency to give the executive special powers to prevent the spread of COVID-19 and to mitigate its effects on society. To address this range of issues, governments have sought to engage all relevant stakeholders in the effective design and delivery of programmes – with varying levels of success.

Main assessments and recommendations:

• Several contributing countries encountered major difficulties in ensuring a co‑ordinated response among all actors of the health sector and supply chain (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]) (US Government Accountability Office, 2021[17]). Indeed, health strategies such as vaccine delivery, for example, involve co‑operation between national and sub-national governments, as well as with practitioners, the private sector, civil society organisations, and/or the media. A lack of clarity in roles and responsibilities may have led to duplication, confusion, or gaps in delivery (Swiss Federal Chancellery, 2020[11]; French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Canadian Office of the Auditor General, 2021[7]).

• Several evaluations recommend establishing clear response strategies and processes for engaging with these stakeholders on pandemic responses in the future (US Government Accountability Office, 2021[9]) (US Government Accountability Office, 2021[17]) (Canadian Office of the Auditor General, 2021[7]).

• Several evaluations highlight the need to put in place mechanisms to mitigate the loss of democratic accountability relating to legislation passed by Parliaments that were given extraordinary powers (New Zealand Parliament, 2021[26]) (Danish Parliamentary Committee on COVID-19, 2021[8]). These evaluations do not question the necessity of these extraordinary power measures but suggest increasing the publicity of the underlying evidence for measures or allowing more time for evaluation of these measures ex post.

• Evaluations also underline the importance of involving citizens in decision-making, including in Parliamentary scrutiny committees (New Zealand Parliament, 2021[26]) and in scientific expert committees (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Belgian Health Care Knowledge Centre, 2020[12]).

Note: Key insights synthesise evidence and lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Response and recovery

Evaluations on response and recovery measures constitute the bulk the evidence on COVID-19 responses received by the OECD. Response and recovery measures encompass policies aimed at mitigating the economic, financial, social and health effects of the COVID-19 pandemic, as well as those related to lockdowns and restrictions. Policies relating to these measures were assessed in one way or another in 62 evaluations (93% of the sample), and all but one country have evaluated measures aimed at mitigating the effects of the crisis.

Most of the evaluations on response and recovery have focused on four main types of policy or measure:

• economic and financial support,

• social policies,

• health policies,

• and lockdown and restriction measures (see Figure 5).

Figure 5. Main focus of evaluations of response and recovery measures

Note: n =67. 67 evaluations were conducted across 18 countries. 45 evaluations examined at least one aspect of economic and financial support.

Source: OECD

Economic and financial support

Most of the evaluations related to response and recovery measures focus on the measures that government rolled out in support of the economy (45 evaluations from 15 countries). Indeed, most OECD governments took swift and massive actions to financially support households and businesses following the shutdown of economic activities. At this stage, these evaluations tend to focus on the relevance and efficiency of these measures, with some assessing intermediate effects. Most evaluations agree that it is too early to measure the impacts of these policies. More evidence on this topic may also become available in the future, as most OECD member countries state that they intend to evaluate the economic and financial emergency measures that were taken at the onset of the pandemic (Working Party of Senior Budget Officials, 2020[36]), but might not have had, as yet, the resources or time to do so.

In their evaluations of economic and financial policies to mitigate the effects of the COVID-19 crisis, countries have examined three main types of measure:

• tax-based measures,

• balance-sheet measures,

• spending measures.

Sixteen evaluations looked at tax-based measures, which included lowering tax rates, deferring tax payments and providing tax credits. At this stage, evaluations of tax-based measures have mainly focused on process and efficiency, as it is challenging to assess their additional effects at this stage of the crisis (see Box 10). The Government Accountability Office in the United States has produced one such evaluation, examining how increased volumes of federal tax returns could be managed by the Internal Revenue Service to improve the overall effectiveness of the government’s economic support to households and businesses, as part of the tax provisions within the CARES Act (United States Government Accountability Office, 2020[37]).

Overall, the evaluations suggest that countries encountered administrative difficulties in rolling out tax schemes. Data from government officials in OECD and partner countries indicate that rapid delivery and minimising the risk of missing deserving recipients took priority over maximising the efficiency of targeting relief measures (OECD, 2021). This necessary prioritisation provides some context for the results obtained from early evaluations of the impact of tax measures.

Box 10. Key insight 7: Tax measures have played an important role in supporting the liquidity of households and businesses, but did not always target the right beneficiaries

Policy context

Tax-based measures were an important part of governments’ efforts to directly support households and businesses during the crisis (OECD, 2020[38]). These measures have taken various forms across countries, including tax payment deferrals and more flexible tax repayment schedules, the removal of import duties and value-added taxes on medical products, and enhanced tax refunds and loss-offset provisions. These measures mostly sought to support corporate and household liquidity, employment and investment.

Main assessments and recommendations:

• Several evaluations underline administrative difficulties with rolling out tax schemes. In particular, evaluations mention the problems faced by countries in identifying eligible recipients for the support schemes, as the tax data and records used to identify eligible individuals and businesses were based on previous fiscal years (2019 and prior). According to some implementing authorities, limiting the schemes only to taxpayers with current and previous records was important to reduce the risks of fraud. However, this led to both the exclusion of potential beneficiaries and the inclusion of ineligible individuals in a number of countries (US Government Accountability Office, 2021[17]; UK National Audit Office, 2020[13]). Moreover, using tax records to identify eligibility did not appear to have reduced fraud in the countries that carried out these evaluations. Matching tax records with other government data became particularly important to avoid improper or duplicate payments (National Audit Office, 2020[39]; New Zealand Controller and Auditor General, 2021[14]; OAG of Canada, 2021[40]).

• Evaluations also mention that these measures were associated with significant budget costs. The United States Joint Committee on Taxation estimated that tax credits put in place to mitigate the cost of emergency paid sick leave, along with expanded family medical leave that certain employers had to provide under the Families First Coronavirus Response Act and the employee retention tax credits for employers under the Coronavirus Aid, Relief, and Economic Security Act of March 2020, would result in around USD 172 billion in lost revenue for fiscal years 2020-2030 (US Government Accountability Office, 2020[16]). The Australian Taxation Office also estimated the revenue foregone for the various tax-based measures under its responsibility (Australian National Audit Office, 2020[23]).

• Yet, evaluations agree that it is too early to know the true budgetary effects and impacts of these measures, or the potential cost of inaction. For instance, in the United States, the GAO indicates that, for several tax measures, the uptake by beneficiaries cannot be quantified yet (US Government Accountability Office, 2021[17]; US Government Accountability Office, 2020[16]). Evaluators in Norway indicated some concern that the use of short-term tax measures could weaken the tax base and affect the stability of the tax system if these changes become permanent, while acknowledging that it is too early to provide a robust assessment (Commission "Norway Towards 2025", 2021[41]).

Note: Key insights synthesise evidence and lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

In addition, 16 evaluations examined the impact of government balance sheet measures. These types of measures have a delayed impact on the cash deficit as they generate assets in the form of loans or equity shares and liabilities in the form of guarantees. For instance, several countries, including Italy and Norway, have put state guarantees in place so that businesses can access credit markets. Evaluations on this topic point out that it is too early to assess the impact of these measures, but that they may be associated with increased budgetary risks for governments (see Box 11).

Box 11. Key insight 8: Balance-sheet measures likely kept businesses afloat on the short term, but they come with risks linked to their longer-term budgetary costs

Policy context

Besides traditional spending and tax-based support, emergency fiscal packages have relied heavily on unconventional fiscal policy instruments such as government loans, guarantees and equity injections. In some countries, these measures made up the largest share, in nominal terms, of the government fiscal response (Moretti, Braëndle and Leroy, 2021[42]). The need to keep businesses afloat, especially SMEs, was the overarching concern of most countries (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; US Office of Management and Budget, 2020[43]; Commission "Norway Towards 2025", 2021[41]; Bank of Italy, 2020[44]). Governments provided access to liquidity and/or guaranteed loans to reduce the risk of insolvencies. Some countries also emphasised the contribution of these measures to employment support (Bank of Italy, 2020[44]; US Office of Management and Budget, 2020[43]). These measures tended to be aimed at specific sectors, such as aviation, agriculture, municipalities, or education.

Main assessments and recommendations

• Evaluations underline the rapid implementation of balance sheet-based policies. In the United States, the Small Business Administration launched the Paycheck Protection Programme of forgivable loans within a week after the CARES Act passed (US Office of Management and Budget, 2020[43]). In France, the National Independent Mission on the assessment of the COVID-19 crisis management and on the anticipation of pandemic risks stressed how swiftly the loan guarantee scheme was implemented (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]).

• This speed may have come at the price of effective targeting. According to the United States Government Accountability Office (GAO), for instance, the lack of guidance and the complexity of the regulations on loan forgiveness under the Paycheck Protection Programme seem to have prevented the disbursement of loans to some recipients in need (US Government Accountability Office, 2020[16]). Loans worth USD 4 million were also approved even though borrowers had not yet received essential information regarding the eligibility for loan forgiveness.

• Contributors emphasised the risk and uncertainty around the longer-term budgetary costs of these measures. Unlike additional expenditures or tax-based support, balance sheet-based policies have a delayed fiscal impact. There are no immediate cash outflows for guarantees and immediate future cash inflows for loans. In France, the evaluation highlights the significant future risk to public finances of the loan guarantee scheme, citing estimates of a default rate on state-guaranteed loans of 5% (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]). In the United States, the cost of the Paycheck Protection Programme was deemed difficult to calculate due to the conditionality of loan forgiveness and changes in the level of loan forgiveness (US Government Accountability Office, 2020[16]).

• Evaluations identified positive short-term impacts, such as allowing employees to return to work and businesses to reopen (US Office of Management and Budget, 2020[43]) or limiting the number of bankruptcies (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]). Some institutions warn that estimating the impact of a given measure may be difficult given multiple forms of government support (UK National Audit Office, 2020[13]).

Note: Key insights synthesise evidence and lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Spending measures, which are those that have an immediate effect on the cash deficit and debt of a country, were analysed in 36 evaluations. In fact, evaluations relating to government expenditure make up most of the evaluations related to economic and financial measures. Key insights from these evaluations can be found in Box 12, Box 13 and Box 14.

Evaluations in the sample show that these three main types of measure – tax-based, balance-sheet based, and spending - sought to target different groups of beneficiaries (see Figure 6), such as households, businesses, and employees.

Twenty-nine evaluations looked at economic measures aimed at households. Such measures often support specific target groups by providing additional financial resources. In Colombia, for example, an unconditional cash transfer aimed at vulnerable households was rolled out to mitigate the economic effects of the crisis. The country’s Department of National Planning conducted an impact evaluation of this “Ingreso Solidario” measure, which was published in May of 2021 (Department of National Planning, Colombian government, 2021[45]). The evaluation concludes that Ingreso Solidario increased the probability that households maintained some source of income during the pandemic and that it did not create disincentives for participation in the labour market. Evaluations of economic measures aimed at households are generally positive about their intermediate effects (see Box 12). These evaluations mention that a closer look at long-term effects and impacts might be needed (see also Box 15).

Box 12. Key insight 9: Measures aimed at households were generally successful at supplementing existing social safety nets

Policy context

Economic measures targeting households seek to mitigate the impact of the crisis on income and purchasing power. They are often targeted at vulnerable groups, who face increased risks of income loss and insecurity as a result of the crisis. These measures are usually temporary and often rely on existing welfare systems (for funding or disbursement), and thus are dependent on the systems’ overall efficiency.

Main assessments and recommendations

• Evaluations show that automatic stabilisers in the form of countries’ standard social protection systems had to be complemented with specific temporary measures to target those whose income and purchasing power suffered most from the shock (Commission "Norway Towards 2025", 2021[41]; French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]).

• State-funded job retention schemes, cash transfers, tax credits, income support for the self-employed, and extention of sick leave protections were seen as effective in limiting income losses incurred by households during the crisis (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Italian Ministry of Economy and Finance, 2020[46]; Australian National Audit Office, 2020[21]; Belgian National Social Security Institut for Self-employed Workers, 2020[47]; UK Department of Health and Social Care, 2020[48]; National Insurance Institute of Israel, 2020[49]; National Planning Department of Colombia, 2020[50]; US Office of Management and Budget, 2020[43]). Specifically, these measures were seen as generally successful in compensating the income loss of more vulnerable workers, and the households they live in, including self-employed workers, young people, large families and those below the poverty line, as well as in reducing their risks of poverty and liquidity problems (UK Department of Health and Social Care, 2020[48]; Italian Ministry of Economy and Finance, 2020[46]; US Government Accountability Office, 2021[17]; France Stratégie, 2020[51]).

• Evaluations recommended, however, that some households receive further state support in order to support their consumption levels (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Italian Ministry of Economy and Finance, 2020[46]; US Government Accountability Office, 2020[18]). This is particularly the case for households that were socio-economically fragile before the crisis and less well-protected by the existing social security system (Italian Ministry of Economy and Finance, 2020[46]; France Stratégie, 2020[51]). Indeed, according to several evaluations, these measures sometimes still excluded some of the most vulnerable groups of the population (such as unemployed youths, isolated elderly people or single-parent families), either due to their limited scope or to a lack of capacity to identify them (France Stratégie, 2020[51]; National Audit Office, 2020[39]). Evaluations recommend improving the scope and reliability of the data collected to identify households eligible for income-protection schemes in order to avoid excluding them in the future (National Audit Office, 2020[39]; US Government Accountability Office, 2021[17]).

• Non-recourse to emergency support was another issue reported by some countries. To ensure that eligible beneficiaries effectively access the offered support, application procedures should be simplified and access to rights should be more automatised (France Stratégie, 2020[51]; US Government Accountability Office, 2021[9]; Contraloría General de la República, 2020[52]). Awareness could also be raised through better outreach and communication efforts (US Government Accountability Office, 2021[17]; National Audit Office, 2020[39]; Belgian National Social Security Institut for Self-employed Workers, 2020[47]; National Planning Department of Colombia, 2020[50]), which could be targeted by tracking and reporting the number of eligible beneficiaries who were notified of the existing support and who have yet to file for it (US Government Accountability Office, 2021[17]; US Government Accountability Office, 2020[18]).

Note: Key insights synthesise evidence and lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Twenty-seven evaluations examined economic and financial measures aimed at businesses. Such measures mostly aim to support employment (e.g. job retention schemes) or ensure liquidity through loan guarantee schemes or tax-based support (e.g. tax reduction or deferrals). Measures may apply to businesses in general or be targeted to specific sectors that were heavily affected by the pandemic (e.g. tourism, aviation). Evaluations underline the fact that countries massively and rapidly supported businesses that were not always well targeted. However, it is too early to assess the impact of these policies – whether positive of negative (see Box 13). For example, in March 2020, Belgium amended a 2016 law to temporarily introduce income support measures for the self-employed. A public policy institute (Institut national d’assurances sociales pour travailleurs indépendants) found that the measures would be more effective if they were better targeted (Geeraert and De Maesschalk, 2020[53]).

Box 13. Key insight 10: Countries rapidly deployed measures aimed at supporting businesses but had to adapt them frequently, affecting their relevance and effectiveness

Policy context

While a number of countries had automatic stabilisers in place to contain an large part of the economic shock resulting from the pandemic, governments had to adopt other measures as well, given the sudden, global nature and severity of the crisis. A number of businesses faced labour force and supply chain interruptions, mandatory closures, and other serious challenges. Across OECD countries, businesses thus benefited from various and sometimes overlapping measures: furlough and wage subsidy schemes, tax deferrals, asset write-offs, and mechanisms to offset additional paid sick leave expenditures or guaranteed loans. These measures were aimed at preserving employment and investments, providing job security and ensuring corporate liquidity and solvency (Demmou et al., 2021[54]; Demmou et al., 2021[55]).

Main assessments and recommendations

• While schemes were, overall, rapidly implemented and timely (French Court of Accounts, 2021[56]; Belgian National Social Security Institut for Self-employed Workers, 2020[47]; US Government Accountability Office, 2021[9]), their efficiency and relevance were found to be more limited. Evaluations stress that governments had to adapt the objectives and modalities of these schemes often to account for changes in restriction measures or to respond to initial design failures and better target those in need (National Audit Office, 2020[39]; French Court of Accounts, 2021[56]; OAG of Canada, 2021[40]; Australian National Audit Office, 2020[23]; Belgian National Social Security Institut for Self-employed Workers, 2020[47]). As a result, some evaluations mention that these changes blurred messages about businesses’ eligibility and therefore diminished the measures’ effectiveness (Belgian National Social Security Institut for Self-employed Workers, 2020[47]; US Government Accountability Office, 2020[16]; New Zealand Controller and Auditor General, 2021[57]) (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Commission "Norway Towards 2025", 2021[41]; Belgian National Social Security Institut for Self-employed Workers, 2020[47]).

• Evaluators found that it was too early to assess the total impact of these schemes (National Audit Office, 2020[39]; US Government Accountability Office, 2020[16]; French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]). While they noted strong evidence that unemployment, output losses and business bankruptcies were contained, they also suggested that support schemes might have sustained non-viable firms (Commission "Norway Towards 2025", 2021[41]; National Audit Office, 2020[39]; OAG of Canada, 2021[40]) or firms that were not in need of support and exploited the rules (National Audit Office, 2020[39]; New Zealand Controller and Auditor General, 2021[57]), perhaps due to the speed of implementation and low conditionality (Commission "Norway Towards 2025", 2021[41]; OAG of Canada, 2021[40]).

• Anticipating a wave of insolvencies, some evaluations also suggested adapting and adjusting bankruptcy and restructuring laws and procedures (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Commission "Norway Towards 2025", 2021[41])to facilitate the restructuring of firms in temporary distress. For example, the French National Independent Mission on the assessment of the COVID-19 crisis management and the anticipation of pandemic risks recommended that the Finance and Justice Ministries study the trajectory of distressed companies to assess the impact of collective procedures on business bankruptcies or survivals and adjust these procedures accordingly.

Note: Key insights synthesise evidence and early lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Finally, 29 evaluations looked at economic and financial measures aimed at supporting jobs and workers. Measures related to employment have a special role to play, as they support several recipient groups at the same time. Support measures for the self-employed affected by the pandemic, for instance, are often designed to support both their businesses and their household income. Since self-employed workers are also one of the more vulnerable groups in times of crisis, many of the evaluations in the sample focused on this population.

For example, learning from the 2008 global financial crisis, when the country did not have a short-time work or partial furlough scheme to support employment, the UK created the Coronavirus Job Retention and Self-Employment Income Support schemes in March 2020. In October 2020, the UK National Audit Office (NAO) published a report examining the relevance, efficiency and effectiveness of these schemes and concluded that they were deployed quickly and were largely successful in protecting employment during lockdown. Nevertheless, the NAO observed that deployment was made more challenging by the lack of existing similar schemes (UK National Audit Office, 2020[13]).

In general, evaluations assessing policies to support self-employed workers during the crisis looked at their relevance and coherence, the efficiency with which the institutions in charge disbursed the support, and their short-term effects (see Box 14).

Box 14. Key insight 11: Learning from the global financial crisis, countries massively deployed measures to support self-employed workers

Policy context

The term ‘self-employed workers’ describes people who work for themselves and are registered as a business. This diverse group makes up a considerable part of the labour force in OECD countries (16.6% in the OECD countries on average in 2019) (OECD, 2020[58]). As this group is usually not eligible for unemployment insurance, they are one of the most financially vulnerable groups in times of crisis.

Main assessments and recommendations

• Support to the self-employed has taken various forms across countries. Some countries extended their existing social security benefits to this group, e.g. by giving them access to unemployment insurance (Office of the Auditor General, Canada, 2021[59]) (US Government Accountability Office, 2020[16]). Other countries extended existing support systems for self-employed workers (Belgian National Social Security Institut for Self-employed Workers, 2020[47]), while others set up new programmes or tax deferments to compensate for loss of revenue due to the pandemic or restrictive measures (French mission to control the quality of the management of the health crisis, 2020[22]) (UK National Audit Office, 2020[13]) (Italian Ministry of Economy and Finance, 2020[46]) (Office of the Auditor General, Canada, 2021[59]); (Belgian National Social Security Institut for Self-employed Workers, 2020[47]).

• Overall, the employment support schemes for the self-employed were assessed positively, as they provided quick income support to self-employed people with encouraging short-term effects (UK National Audit Office, 2020[13]) (US Government Accountability Office, 2020[16]) (Office of the Auditor General, Canada, 2021[59]). Several contributions stress, however, that it is too early to evaluate long-term effects and impact. The UK National Audit Office claims that the UK’s scheme provided an effective bridge during the early phases of the pandemic to allow some people to return to work once the national lockdown eased. It supported 77% of potentially eligible recipients and the average grant came close to fully compensating beneficiaries for the effects of the pandemic. The HM Revenue and Customs also made their self-employment income support scheme available two weeks before schedule (UK National Audit Office, 2020[13]).

• However, many evaluations underline that the trade-off for rapid rollout was simplified eligibility procedures, which may have resulted in some self-employed workers being accidentally excluded from these measures (UK National Audit Office, 2020[13]) (US Government Accountability Office, 2021[17]) (Office of the Auditor General, Canada, 2021[59]) (New Zealand Controller and Auditor General, 2021[14]). Indeed, multiple contributing countries used tax data from previous fiscal years to verify claims by different recipient groups and to calculate the amount to pay (UK National Audit Office, 2020[13]) (US Government Accountability Office, 2021[17]) (New Zealand Controller and Auditor General, 2021[14]). This has excluded some self-employed workers who recently opened their businesses and who had not previously submitted tax returns. This also meant that post-payment controls were preferred to pre-payment controls and an increased risk of incorrect payments and fraud had to be consciously accepted (Office of the Auditor General, Canada, 2021[59]) (UK National Audit Office, 2020[13]).

• In order to minimise fraud and reach the right beneficiaries, some countries have adapted their measures several times – thus introducing greater complexity and minimising clarity for the public (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Belgian Health Care Knowledge Centre, 2020[12]).

• Some evaluations conclude with the need to improve the protection system for the self-employed for future economic shocks, both in terms of national solidarity and economic efficiency (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Belgian Health Care Knowledge Centre, 2020[12]).

Note: Key insights synthesise evidence and lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Source: In the text.

A more detailed analysis of evaluations on economic and financial measures show that there is less evidence on the effectiveness of tax-based support and balance-sheet measures for these target groups, than that on expenditure-based measures for these groups (see Figure 6). Specifically, the evaluations that mention tax-based support and balance-sheet measures mostly looked at their effects on businesses. This focus is consistent with the main goal of balance-sheet measures, which was to ensure corporate liquidity and support credit markets, suggesting that there is a good evidence base on these measures to date. Still, the impacts of economic and financial measures on households, employees and businesses need to be further evaluated in the coming months and years.

Figure 6. Evaluations on the effects of economic and financial support measures on different recipient groups

Note: n=67. There are 67 evaluations among 18 OECD countries. 29 evaluations assessed spending/expenditure measures directed at households.

Source: OECD

Finally, evaluations also looked at infrastructure measures, measures to support financial markets and measures to support longer-term recovery – but such assessments are, so far, less common.

Social policies

Social measures were also mentioned in quite a few evaluations of COVID-19 responses (35 evaluations over 13 countries). However, 8 of these evaluations were ex ante assessments that looked at the predicted and potential impacts of the crisis on poverty and inequality. For example, the National Council for the Evaluation of Social Development Policy (CONEVAL) in Mexico conducted an ex ante evaluation to assess the impacts of the pandemic on poverty and how the existing social protection mechanisms in place could help mitigate these impacts. It underlines the need for increased social protection for the most vulnerable population groups.

Many ex post or ex durante evaluations on this topic note that the pandemic may have had impacts of inequality and poverty without providing an assessment of how and why this might be the case. There are, however, several evaluations that analyse these effects. These evaluations looked at topics such as the distribution of school meals, emergency housing during lockdown and confinement periods, or how governments have supported household income to fight against poverty (see Box 15). Interestingly, all three upper-middle-income countries in the sample (Colombia, Costa Rica, and Mexico) have focused their early evaluations solely on social, household and employment issues.

Over the longer term, more countries may wish to look at the social impacts of the pandemic. Some countries have already begun doing so, for instance when it comes to the impacts of the pandemic on young people (OECD, 2020[60]). In Canada, the government is examining how government spending and policies, including emergency aid and recovery measures for the COVID-19 crisis, will affect women and men differently through the lenses of its Gender-Based Analysis Plus (GBA+) framework. The framework is used to assess the impact of policies and programmes across social groups, acknowledging intersecting identity factors such as gender and age. In general, the impact of general lockdown measures and increased unemployment on domestic violence and the general effects of the crisis on youth have not been mentioned in any of the evaluations in the sample and may be worth exploring in the future.

Additionally, among the 67 evaluations covered in this report, few cover education (16), and, specifically, the effect of school closures. Surprisingly, few national statistical offices or education ministries undertook special data collections related to the pandemic and its effects using probability samples. This reflects the sudden and unexpected nature of the COVID-19 pandemic and the fact that the restrictions associated with lockdowns created less than ideal conditions for conducting survey-based research (Thorn, W. and S. Vincent-Lancrin, 2021[61]).

Nevertheless, in Germany, Ireland, France, the United Kingdom and the United States, data based on probability samples, and thus representative of some well-identified national populations, were collected on a variety of inputs and outcomes. The picture offered by those statistical studies shows that distance and remote education were put in place at short notice and allowed education to continue at home for the majority of children, coupled with a form of in‑person instruction for children with special needs (Thorn, W. and S. Vincent-Lancrin, 2021[61]). While those arrangements represented an imperfect substitute for normal classes, they nonetheless ensured that most, though not all, children continued to have a connection with their teachers and schools.

At this point, however, the evidence regarding the impact of the first wave of school closures on academic progress is generally inconclusive. Looking ahead, a more thorough and long-term monitoring of the consequences of the pandemic on children’s schooling and well‑being is needed. High-quality data collection may have become easier for the subsequent waves of school closures and should provide researchers and analysts with a better understanding of what occurred during this period and of the behaviour and views of those involved and affected by the disruption to school education.

Box 15. Key insight 12: To minimise the social impacts of the crisis, countries had to implement policies targeted at the most disadvantaged groups in society

Policy context

The crisis has affected all parts of the population but its adverse effects are felt more strongly in vulnerable social groups, including people living in poverty, the elderly, homeless persons, children and young people (OECD, 2020[60]). To prevent social disruption and avoid an increase in poverty, inequality and exclusion, governments have expanded their social protection systems and implemented policies targeted at the most disadvantaged groups in society.

Main assessments and recommendations

• Particular attention has been paid to poverty and inequality reduction. In Colombia, Israel and Italy, government interventions were seen as having had a positive impact on the poorest people and mitigating the impact of the crisis on overall inequality indexes (NPD and IADB, 2021[62]) (National Insurance of Israel, 2020[63]) (MEF, 2020[64]). However, several evaluations point to the challenges of identifying and protecting vulnerable beneficiaries under short time constraints (Contraloría General de la República, 2020[52]; Cour des Comptes, 2021[65]). In Costa Rica, for example, some people who qualified for the benefit did not receive the Bono Proteger programme, thus preventing it from having the desired impact on poverty levels (Contraloría General de la República, 2020[52]).

• Several evaluations point out that assessing the impact of the crisis on poverty and inequality could also allow governments to identify gaps in their social protection systems (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]) (US Government Accountability Office, 2020[18]). These findings are also in line with the recommendations from the OECD policy paper on “Youth and COVID-19” which suggests that tools to measure intergenerational impacts could also allow governments to anticipate the impact of response and recovery measures across different age groups and avoid exacerbating intergenerational inequalities, which have been accentuated by the crisis (OECD, 2020[60]).

• Policies aimed at mitigating the social impacts of the crisis also paid particular attention to the effects of remote learning and school closures, which were undertaken by a large number of countries to reduce the spread of the virus (UK NAO, 2021[66]; Knesset Research and Information Center, 2020[67]; Commission "Norway Towards 2025", 2021[41]; Cour des Comptes, 2021[68]). Ensuring continuous learning for students was a particular challenge for countries. The most vulnerable groups were particularly affected by such measures (HM Government, 2020[69]; Commission "Norway Towards 2025", 2021[41]; US Office of Management and Budget, 2020[43]; UK Department of Health and Social Care, 2020[48]; Knesset Research and Information Center, 2020[67]; Cour des Comptes, 2021[68]). At home for instance, vulnerable groups suffered from higher rates of infection and COVID-19 related deaths, more temporary layoffs in lower status occupations, a relatively greater loss of income compared to the general population, and a higher incidence of mental health issues. For disadvantaged children, schooling in this environment translated into more difficulty with devices and connectivity and more loss of contact with school and teachers (Thorn, W. and S. Vincent-Lancrin, 2021[61]). Generally, school closures also have a high cost in terms of student performance, well-being, physical and mental health, motivation and insertion in the labour market (HM Government, 2020[69]; Commission "Norway Towards 2025", 2021[41]; Cour des Comptes, 2021[68]; UK NAO, 2021[66]).The few evaluations available on this topic concluded that it was essential to keep schools open as much as possible (Danish Parliamentary Committee on COVID-19, 2021[8]; HM Government, 2020[69]; Cour des Comptes, 2021[68]) (GAO, 2020[70]).

Note: Key insights synthesise evidence and early lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Source: In the text.

Health measures

Health measures put in place by OECD countries in response to COVID-19 were examined in 27 evaluations across 12 countries. These evaluations look at issues related to measures for infection control, as well as intensive care and regular health care capacities during the acute phase of the crisis. The evaluations that focus on health measures look mostly at issues related to process, such as that of data collection or supply of vaccines. Several of these evaluations suggest that more in-depth analysis of the impacts of the pandemic on health is needed (see Box 16). For instance, there is little evidence available on the effectiveness of countries’ vaccination policies at this stage, or on the impact of telemedicine on key health indicators. Other issues are not mentioned at all in the sample of evaluations, such as the impact of the pandemic on mental health, especially that of youth, and on alcohol consumption. Yet, evidence from the OECD suggests that the COVID‑19 crisis has heightened the risk factors generally associated with poor mental health and alcohol consumption (OECD, 2021[71]; OECD, 2021[72]).

Box 16. Key insight 13: Countries encountered important implementation challenges in mitigating the health impacts of the crisis

Policy context

In order to control the spread of the virus, OECD countries introduced more or less stringent infection control measures (mandatory use of protective masks, social distancing, quarantine measures, etc.) and deployed massive testing and tracing strategies. In addition, vaccination campaigns were launched starting from the end of 2020 to immunise the population. Multiple actors, both public and private, have been involved in these efforts. The pandemic not only had an impact on intensive care, but also on regular health care insofar as hospitals were overwhelmed with treating COVID-19 patients during the peak of the pandemic.

Main assessments and recommendations

• A major challenge pinpointed by evaluations was the lack or poor quality of real-time data on intensive care unit beds available in hospitals in some countries (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]) (Belgian Health Care Knowledge Centre, 2020[12]). Yet, these data are crucial to assess hospital capacity and take steps to surge or de-surge this capacity, to decide on new admissions of COVID-19 patients, to create additional capacity, or assess the need for inter-hospital transport. Evaluations pointed to the importance of developing an inventory of critical care capacities both at the national and regional level, to assess capacities in normal times, and to better understand how to extend them in times of crisis. Evaluations also suggest working on shared IT systems among the health ministry, local health authorities and hospitals to allow for real-time reporting (US Government Accountability Office, 2021[9]) (see also Box 5).

• Evaluations underline that national governments encountered important challenges in deploying testing and tracing efforts, in particular as they required a high level of co‑ordination with local actors. As a result, the detection of large-scale outbreaks and the subsequent isolation of infected people was extremely difficult (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]). Evaluations recommend that governments establish clear protocols for authorities to work together to test and trace, in order to improve pandemic response capacities. Evaluations also highlighted the insufficient checks on quarantine and self-isolation orders imposed on infected people and travellers, given the lack of compliance with quarantine and testing requirements and public health agencies’ difficulty in performing controls. Some evaluations mention that a nationwide quarantine could be a solution to improve compliance with these rules (Canadian Office of the Auditor General, 2021[7]; Holden Expert Group, n.d.[73]).

• Evaluations emphasised that deploying vaccination strategies has posed important logistical challenges as they required the concerted effort of many stakeholders. Evaluations recommend a clear communication of vaccination plans to all involved entities to ensure that they have sufficient time to prepare and organise distribution (GAO, 2020[70]) (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]).

• Evaluations observe a reduction in medical consultation and in-person visits to clinics and hospitals for non-COVID-19 patients, and indicate that communications around regular health care were not clear in several countries (Danish Parliamentary Committee on COVID-19, 2021[8]; Cour des Comptes, 2021[74]). Online medical consultations were not always seen as an effective solution to maintain good levels of care for patients (State Comptroller of Israel, 2020[75]; Corona Investigative Committee, 2020[76]; Dutch Ministry of Sports and Health, 2020[77]). Evaluations pointed to the potential long-term negative consequences for population health outcomes due to challenges in maintaining care continuity, and suggest a further evaluation of the impacts of the pandemic in this regard (State Comptroller of Israel, 2020[75]; Corona Investigative Committee, 2020[76]; Dutch Ministry of Sports and Health, 2020[77]).

Note: Key insights synthesise evidence and early lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Lockdown and restriction

Restrictions on the movement of people were assessed in 27 evaluations conducted by 10 countries. OECD data indicates that school restrictions (16 evaluations), curfews and bans on public gatherings (14), and travel restrictions (13) were important concerns for countries. There is no correlation between the number of evaluations published on lockdown and restrictions measures and the average score of a stringency index, as calculated by the COVID-19 Government Response Tracker of the University of Oxford (University of Oxford, 2020[78]). For example, Norway published 7 evaluations that examined lockdown and restriction policies on peoples’ behaviour despite a relatively low score of 47 (on a scale of 100) on this index. Evaluations in the sample suggest that lockdown and restriction measures came at a high cost for society, but recommend pursuing efforts to evaluate the costs and benefits associated with these measures, in order to optimise them in the future. Several evaluations do recognise that this balance will be difficult to assess, given the far-reaching implications of these types of measures. In terms of process, lockdown measures were seen as not always coherent or well implemented (Box 17).

Box 17. Key insight 14: Countries were quick to close their borders and implement travel restrictions, but encountered issues with coherence of, and compliance with, lockdown measures

Policy context

The lockdown and restriction measures implemented by countries generally include border closures and controls, travel restrictions, school closures, curfews and bans on public gatherings. Countries limited mobility within and across borders to curb the spread of the virus and indirectly diminish the detrimental economic, social and health impacts of the crisis.

Main assessments and recommendations

• Evaluations mention that governments generally acted relatively quickly to close borders when it became apparent that the pandemic was no longer under control, although these findings may be very country-dependent. In general, border restrictions on international travel are beneficial in the short term to contain the spread of the virus, but their effectiveness lessens once the virus starts spreading within countries (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Canadian Office of the Auditor General, 2021[7]; Commission "Norway Towards 2025", 2021[41]; Danish Parliamentary Committee on COVID-19, 2021[8]) (Italian Ministry of Economy and Finance, 2020[46]). Countries tried to reallocate resources where to repatriate citizens residing abroad or at least ensure that their needs were met at a distance by swiftly and regularly informing them about the health situation and existing support (US Government Accountability Office, 2021[17]; Cour des Comptes, 2021[79]). Evaluations also show that countries within the European Union were able to co‑operate fruitfully to implement these measures (Commission "Norway Towards 2025", 2021[41]; French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]).

• Evaluations mention, however, that there were challenges with compliance with lockdown measures, party because they were sometimes difficult to understand for citizens or inconsistent. Evaluations recommend clearer provisions, more accurate control of their implementation, with a consideration for vulnerable groups that might be disproportionately affected by such measures (Canadian Office of the Auditor General, 2021[7]; UK Department of Health and Social Care, 2020[48]).

• The restrictions of visits in hospitals and retirement homes had high costs in terms of mental health and well-being of both patients and their relatives (US Government Accountability Office, 2020[18]; Belgian Health Care Knowledge Centre, 2020[12]). Evaluations therefore recommend that solutions such as building glass panes and organising more distance meetings be put in place (US Government Accountability Office, 2020[18]; Belgian Health Care Knowledge Centre, 2020[12]).

• While restrictive measures such as bans on public gatherings in restaurants and cultural places and mandatory teleworking had high economic, social and well-being costs, they were assessed as somewhat beneficial for the environment, with reduced carbon emissions associated with reduced national and international travel (Commission "Norway Towards 2025", 2021[41]; French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; US Government Accountability Office, 2021[32]; GOV.UK, 2020[80]; Italian Ministry of Economy and Finance, 2020[46]). Continuing to evaluate the costs and benefits associated with restrictive measures, such as strict lockdowns and curfews, would help countries optimise them in the future (French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks, 2021[10]; Commission "Norway Towards 2025", 2021[41]; Danish Parliamentary Committee on COVID-19, 2021[8]).

Note: Key insights synthesise evidence and lessons from the 36 ex post and 17 ex durante evaluations. The insights summarised in these boxes aim to facilitate collective learning to build resilience to future shocks. All lessons in this box come from the material listed above and are supported by multiple evaluations.

Conclusions and the way forward

Countries have deployed significant efforts to respond to this crisis, which affected all sectors of society. Despite time and resource constraints, OECD governments undertook evaluations of their COVID-19 responses. Yet, as the crisis is still ongoing and it is still early to assess the true impacts of many of the measures deployed, the evidence from these evaluations is focused on the relevance and efficiency of these policies, and much less on their effectiveness. More importantly, issues relating to policies’ proportionality and coherence are still largely underexplored – at the same time, they may be particularly useful for policy debate when resources are scarce and cross-government co‑ordination is crucial.

A mapping of the overall evidence base available to the OECD Secretariat shows that some types of policy responses, such as health and social policies, as well as lockdown and restriction measures or critical sector preparedness, still need further analysis. More importantly, the evaluations do not cover all aspects of countries’ policy responses to the crisis, suggesting that evidence is still lacking in some areas – such as the impacts of the crisis on mental health, on youth or on education levels.

Nevertheless, the existing evidence base provides decision makers with key insights to draw upon in developing both their ongoing recovery efforts and future plans to increase governments’ resilience to crises. Going forward, the OECD Secretariat will update this paper to expand on the 14 insights included in this edition, particularly with regard to response and recovery measures, as evidence of their impact may be particularly useful to inform policies as countries progressively move out of the crisis.

References

[19] Australian NAO (2020), Management of the Australian Public Service’s Workforce.

[15] Australian National Audit Office (2020), Planning and Governance of COVID-19 Procurements to Increase the National Medical Stockpile.

[21] Australian National Audit Office (2020), Services Australia COVID-19 Measures and Enterprise.

[23] Australian National Audit Office (2020), The Australian Taxation Office’s Management of Risks Related to the Rapid Implementation of COVID-19 Economic Response Measures, https://www.anao.gov.au/sites/default/files/Auditor-General_Report_2020-21_24a.pdf.

[44] Bank of Italy (2020), Some preliminary estimates of the effects of support measures on the labour market, https://www.bancaditalia.it/pubblicazioni/note-covid-19/2020/Scenari_impatto_COVID_19.pdf.

[12] Belgian Health Care Knowledge Centre (2020), Assessing the management of hospital surge capacity in the first wave of the COVID-19 pandemic in Belgium, https://doi.org/D/2020/10.273/35.

[47] Belgian National Social Security Institut for Self-employed Workers (2020), Temporary extension of the financial safety net for self-employed workers facing loss of income following the Coronavirus crisis.

[7] Canadian Office of the Auditor General (2021), Pandemic Preparedness, Surveillance, and Border Control Measures, https://doi.org/2561-343X.

[41] Commission "Norway Towards 2025" (2021), White Paper, https://doi.org/978-82-583-1475-9.

[52] Contraloría General de la República (2020), Informe de auditoría operativa sobre la eficacia y eficiencia del Bono Proteger implementado por el Ministerio de Trabajo y Seguridad Social y el Instituto Mixto de Ayuda Social ante la emergencia sanitaria provocada por la enfermedad Covid-19, https://cgrfiles.cgr.go.cr/publico/docs_cgr/2020/SIGYD_D/SIGYD_D_2020023096.pdf.

[76] Corona Investigative Committee (2020), The eldery care in the pandemic, http://coronakommissionen.com/wp-content/uploads/2020/12/summary.pdf.

[68] Cour des Comptes (2021), La contribution du service public numérique éducatif à la continuité scolaire pendant la crise sanitaire, https://www.ccomptes.fr/fr/publications/le-rapport-public-annuel-2021.

[79] Cour des Comptes (2021), L’aide au retour des Français retenus à l’étranger par la pandémie de covid 19, https://www.ccomptes.fr/fr/publications/le-rapport-public-annuel-2021.

[65] Cour des Comptes (2021), L’hébergement et le logement des personnes sans domicile pendant la crise, https://www.ccomptes.fr/fr/publications/le-rapport-public-annuel-2021.

[74] Cour des Comptes (2021), Réanimation et soins critiques en général, https://www.ccomptes.fr/fr/publications/le-rapport-public-annuel-2021.

[35] COVID-19 Canada Open Data Working Group (2021), COVID-19 in Canada, https://art-bd.shinyapps.io/covid19canada/.

[8] Danish Parliamentary Committee on COVID-19 (2021), The handling of COVID-19 in the spring of 2020.

Demmou, [55] Demmou, L. et al. (2021), Insolvency and debt overhang following the COVID-19 outbreak: Assessment of risks and policy responses, https://doi.org/10.1787/747a8226-en.

Demmou, [54] Demmou, L. et al. (2021), Liquidity shortfalls during the COVID-19 outbreak: Assessment and policy responses, https://doi.org/10.1787/581dba7f-en.

[45] Department of National Planning, Colombian government (2021), Impactos del programa Ingreso Solidario frente a la crisis del COVID-19 en Colombia, http://dx.doi.org/10.18235/0003261.

[77] Dutch Ministry of Sports and Health (2020), An impact analysis of the COVID-19 on the regular health care.

[51] France Stratégie (2020), La lutte contre la pauvreté au temps du coronavirus, https://www.strategie.gouv.fr/publications/lutte-contre-pauvrete-temps-coronavirus.

[56] French Court of Accounts (2021), The solidarity fund for companies, https://www.ccomptes.fr/sites/default/files/2021-03/20210318-06-TomeI-fonds-solidarite-destination-des-entreprises.pdf.

[10] French independent national mission to evaluate the COVID-19 crisis management and on the anticipation of pandemic risks (2021), Final report.

[22] French mission to control the quality of the management of the health crisis (2020), Report on the quality control of the management of the health crisis, https://www.gouvernement.fr/sites/default/files/document/document/2020/11/mission_controle_qualite_de_la_gestion_de_crise_sanitaire_-_v14.pdf.

[70] GAO (2020), Covid-19: Federal Efforts Could Be Strengthened by Timely and Concerted Actions, https://www.gao.gov/products/gao-20-701.

Geeraert, [53] Geeraert, M. and V. De Maesschalk (2020), “Extension temporaire du droit passerelle en tant que filet de sécurité financière pour les indépendants confrontés à une perte de revenus à la suite de la crise du coronavirus”, Revue Belge de sécurité sociale, Vol. 1, https://socialsecurity.belgium.be/sites/default/files/content/docs/fr/publications/rbss/2020/rbss-2020-1-extension-temporaire-du-droit-passerelle-en-tant-que-filet-de-securite-financiere-pour-les-independants.pdf.

[80] GOV.UK (2020), Analysis of the health, economic and social effects of COVID-19 and the approach to tiering, https://www.gov.uk/government/publications/the-health-economic-and-social-effects-of-covid-19-and-the-tiered-approach.

[69] HM Government (2020), Analysis of the health, economic and social effects of COVID-19 and the approach to tiering, https://www.gov.uk/government/publications/the-health-economic-and-social-effects-of-covid-19-and-the-tiered-approach.

[73] Holden Expert Group (n.d.), 2021, https://www.helsedirektoratet.no/rapporter/samfunnsokonomisk-vurdering-av-smitteverntiltak-covid-19/Samfunns%C3%B8konomisk%20vurdering%20av%20smitteverntiltak%20-%20Tredje%20rapport,%20del%201.pdf/_/attachment/inline/d4970fb8-37e8-4e67-8900-835937bc80ca:c.

[46] Italian Ministry of Economy and Finance (2020), The impact of Covid-19 and government interventions on the socio-economic situation of Italian families in the first three months of the pandemic.

[67] Knesset Research and Information Center (2020), The Availability of Digital Devices and Internet Connections for Distance Learning among School-age Children, https://main.knesset.gov.il/EN/activity/mmm/TheAvailabilityofDigitalDevices.pdf.

[64] MEF (2020), L’impatto del Covid-19 e degli interventi del Governo sulla situazione socio-economica delle, https://www.finanze.gov.it/export/sites/finanze/.galleries/Documenti/Varie/Nota-tematica-n.-3-Impatto-del-Covid-19-sulla-situazione-socio-economica-delle-famiglie-italiane.pdf.

[42] Moretti, D., T. Braëndle and A. Leroy (2021), “Balance sheet-based policies in COVID‑19 fiscal packages: How to improve transparency and risk analysis?”, Journal on Budgeting, Vol. Volume 2021/Issue 2, https://doi.org/10.1787/16812336.

[39] National Audit Office (2020), Implementing employment support schemes in response to the COVID-19 pandemic, https://doi.org/9781786043399.

[49] National Insurance Institute of Israel (2020), An estimate of the impact of the corona crisis on the socio-economic situation, https://www.btl.gov.il/Publications/corona/Pages/mashber-calcali1206-5241.aspx.

[63] National Insurance of Israel (2020), The impact of the recession following the Corona crisis on living standards, poverty and inequality (Study No. 133), https://www.btl.gov.il/Publications/corona/Pages/mechkar1330520-600.aspx.

[50] National Planning Department of Colombia (2020), The Impact of Emergency Cash Assistance During the COVID-19 Pandemic in Colombia, https://www.poverty-action.org/study/impact-emergency-cash-assistance-during-covid-19-pandemic-colombia.

[57] New Zealand Controller and Auditor General (2021), Management of the Wage Subsidy Scheme, https://oag.parliament.nz/2021/wage-subsidy/docs/wage-subsidy.pdf.

[14] New Zealand Controller and Auditor General (2021), Ministry of Health: Management of personal protective equipment in response to Covid-19, https://oag.parliament.nz/2020/ppe.

[26] New Zealand Parliament (2021), Assessing Parliament’s response to the COVID-19 pandemic.

[62] NPD and IADB (2021), Impactos del programa Ingreso Solidario frente a la crisis del COVID-19 en Colombia, https://publications.iadb.org/es/impactos-del-programa-ingreso-solidario-frente-la-crisis-del-covid-19-en-colombia.

[40] OAG of Canada (2021), Canada Emergency Wage Subsidy, https://www.oag-bvg.gc.ca/internet/docs/parl_oag_202103_02_e.pdf.

[3] OECD (2021), Government at a Glance 2021, OECD Publishing, Paris, https://doi.org/10.1787/1c258f55-en.

[25] OECD (2021), OECD Report on Public Communication: The Global Context and the Way Forward, OECD Publishing, Paris, https://dx.doi.org/10.1787/22f8031c-en.

[25]

[25]

[71] OECD (2021), “Tackling the mental health impact of the COVID-19 crisis: An integrated, whole-of-society response”, OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing, Paris, https://dx.doi.org/10.1787/0ccafa0b-en.

[72] OECD (2021), “The effect of COVID-19 on alcohol consumption, and policy responses to prevent harmful alcohol consumption”, OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing, Paris, https://dx.doi.org/10.1787/53890024-en.

[20] OECD (2020), “Building resilience to the Covid-19 pandemic: the role of centres of government”, OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing, Paris, https://dx.doi.org/10.1787/883d2961-en.

[58] OECD (2020), OECD Labour Force Statistics.

[38] OECD (2020), “Tax and fiscal policy in response to the Coronavirus crisis: Strengthening confidence and resilience”, OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing, Paris, https://dx.doi.org/10.1787/60f640a8-en.

[28] OECD (2020), “The Covid-19 crisis: A catalyst for government transformation?”, OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing, Paris, https://dx.doi.org/10.1787/1d0c0788-en.

[34] OECD (2020), “The territorial impact of COVID-19: Managing the crisis across levels of government”, OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing, Paris, https://dx.doi.org/10.1787/d3e314e1-en.

[29] OECD (2020), “Transparency, communication and trust: The role of public communication in responding to the wave of disinformation about the new Coronavirus”, OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing, Paris, https://dx.doi.org/10.1787/bef7ad6e-en.

[60] OECD (2020), “Youth and COVID-19: Response, recovery and resilience”, OECD Policy Responses to Coronavirus (COVID-19), OECD Publishing, Paris, https://dx.doi.org/10.1787/c40e61c6-en.

[5] OECD (2018), Assessing Global Progress in the Governance of Critical Risks, OECD Reviews of Risk Management Policies, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264309272-en.

[4] OECD (2018), National Risk Assessments: A Cross Country Perspective, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264287532-en.

[27] OECD (2018), Scientific Advice During Crises: Facilitating Transnational Co-operation and Exchange of Information, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264304413-en.

[31] OECD (2016), Trends in Risk Communication Policies and Practices, OECD Reviews of Risk Management Policies, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264260467-en.

[2] OECD (2015), The Changing Face of Strategic Crisis Management, OECD Reviews of Risk Management Policies, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264249127-en.

[2]

[2]

[1] OECD (2014), “Recommendation of the Council on the Governance of Critical Risks”, OECD Legal Instruments, OECD/LEGAL/0405, OECD, Paris, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0405.

[6] OECD (2011), Future Global Shocks: Pandemics, OECD, Paris, https://www.oecd.org/fr/gov/risques/Pandemics.pdf.

[33] OECD (forthcoming), Principles of Good Practice for Public Communication Responses to Help Counter Mis- and Disinformation.

[59] Office of the Auditor General, Canada (2021), COVID-19 pandemic: Canada Emergency Response Benefit.

[75] State Comptroller of Israel (2020), The State of Israel Response to the COVID-19 Crisis - Special Interim Report, https://www.mevaker.gov.il/sites/DigitalLibrary/Documents/2020/COVID-19/2020-COVID-19-003-preface-EN.pdf?AspxAutoDetectCookieSupport=1.

[11] Swiss Federal Chancellery (2020), Rapport concernant l’évaluation de la gestion de crise pendant la pandémie de COVID-19, https://www.newsd.admin.ch/newsd/message/attachments/64454.pdf (accessed on 4 July 2021).

Thorn, [61] Thorn, W. and S. Vincent-Lancrin (2021), Schooling During a Pandemic: The Experience and Outcomes of Schoolchildren During the First Round of COVID-19 Lockdowns, OECD Publishing, https://doi.org/10.1787/1c78681e-en.

[48] UK Department of Health and Social Care (2020), Coronavirus Act 2020: the public sector equalities duty impact assessment, https://www.gov.uk/government/publications/coronavirus-act-2020-equality-impact-assessment/coronavirus-act-2020-the-public-sector-equalities-duty-impact-assessment.

[66] UK NAO (2021), Support for children’s education during the early stages of the Covid-19 pandemic, https://www.nao.org.uk/work-in-progress/support-for-childrens-education-during-the-covid-19-pandemic/.

[13] UK National Audit Office (2020), Implementing employment support schemes in response to the COVID-19 pandemic, https://www.nao.org.uk/report/implementing-employment-support-schemes-in-response-to-the-covid-19-pandemic/.

[37] United States Government Accountability Office (2020), COVID-19: Urgent Actions Needed to Better Ensure an Effective Federal Response, https://www.gao.gov/assets/gao-21-191.pdf.

[78] University of Oxford (2020), Covid-19 Government Response Tracker, https://www.bsg.ox.ac.uk/research/research-projects/covid-19-government-response-tracker (accessed on 29 June 2021).

[9] US Government Accountability Office (2021), COVID-19: Critical Vaccine Distribution, Supply Chain, Program Integrity, and Other Challenges Require Focused Federal Attention.

[17] US Government Accountability Office (2021), COVID-19: Federal Efforts Could be Strengthened by Timely and Concerted Actions.

[24] US Government Accountability Office (2021), Critical Vaccine Distribution, Supply Chain, Program Integrity, and Other Challenges Require Focused Federal Attention.

[32] US Government Accountability Office (2021), Federal Efforts Accelerate Vaccine and Therapeutic Development, but More Transparency Needed on Emergency Use Authorizations.

[16] US Government Accountability Office (2020), Covid-19: Opportunities to Improve Federal Response and Recovery Efforts.

[18] US Government Accountability Office (2020), Urgent Actions Needed to Better Ensure an Effective Federal Response.

[43] US Office of Management and Budget (2020), The Economic Impact of Coronavirus Response Funds: First Quarterly Report.

Wendling, [30] Wendling, R. (2013), “The use of social media in risk and crisis communication”, OECD Working Papers on Public Governance, Vol. No. 24.

[36] Working Party of Senior Budget Officials (2020), The use of performance information during the COVID-19 crisis.

List of evaluations of COVID-19 policy response analysed in this paper

Annex Table 1.A.1. Evaluations analysed by each country

Country	Title	Author	Publication date (quarter)
AUS	Services Australia COVID-19 Measures and Enterprise Risk Management	Australian National Audit Office	2020 Q4
AUS	The Australian Taxation Office’s Management of Risks Related to the Rapid Implementation of COVID-19 Economic Response Measures	Australian National Audit Office	2020 Q4
AUS	Management of the Australian Public Service’s Workforce Response to COVID-19	Australian National Audit Office	2020 Q4
AUS	Planning and Governance of COVID-19 Procurements to Increase the National Medical Stockpile	Australian National Audit Office	2020 Q4
AUS	COVID-19 Procurements and Deployments of the National Medical Stockpile	Australian National Audit Office	2021 Q2
BEL	Extension temporaire du droit passerelle en tant que filet de sécurité financière pour les indépendants confrontés à une perte de revenus à la suite de la crise du coronavirus	Institut national d’assurances sociales pour travailleurs indépendants (INASTI)	2020 Q4
BEL	Assessing the management of hospital surge capacity in the first wave of the COVID-19 pandemic in Belgium	Centre Fédéral D'Expertise des Soins de Santé	2020 Q4
CAN	Pandemic Preparedness, Surveillance, and Border Control Measures	Office of the Auditor General of Canada	2021 Q1
CAN	Canada Emergency Wage Subsidy	Office of the Auditor General of Canada	2021 Q1
CHE	Rapport concernant l’évaluation de la gestion de crise pendant la pandémie de COVID-19 (1re phase / février - août 2020)	Swiss Federal Chancellery	2020 Q4
COL	The Impact of Emergency Cash Assistance During the COVID-19 Pandemic in Colombia	National Planning Department and Innovations for Poverty Action	2020 Q4
COL	Impact evaluation of Ingreso Solidario	National Planning Department and Inter-American Development Bank	2021 Q1
CRI	Operational Audit Report on the effectiveness and efficiency of the Bono Proteger implemented by the Ministry of Labour and Social Security and the Instituto Mixto de Ayuda Social in the face of the Health Emergency caused by the COVID-19 disease	Contraloría General de la República	2020 Q4
DNK	Report on the authorities' handling of the first phase of the Covid 19 pandemic	Independent study group	2021 Q1
FRA	L'aide au retour des Français retenus à l'étranger par la pandémie de Covid-19	Cour des Comptes	2021 Q1
FRA	La contribution du service public numérique éducatif à la continuité scolaire pendant la crise sanitaire	Cour des Comptes	2021 Q1
FRA	L'hébergement et le logement des personnes sans domicile pendant la crise	Cour des Comptes	2021 Q1
FRA	Réanimation et soins critiques en général	Cour des Comptes	2021 Q1
FRA	Les établissements de santé face à la première vague de Covid-19 : exemples néo-aquitains et franc-comtois	Cour des Comptes	2021 Q1
FRA	Le fonds de solidarité à destination des entreprises	Cour des Comptes	2021 Q1
FRA	La SNCF face à la crise sanitaire	Cour des Comptes	2021 Q1
FRA	National independent Mission on the assessment of the Covid-19 crisis management and on the anticipation of pandemic risks - Final Report	Independent mission on the assessment of the COVID-19 crisis	2021 Q2
FRA	La lutte contre la pauvreté au temps du coronavirus	France Stratégie	2020 Q3
FRA	Rapport de la mission relative au contrôle qualité de la gestion de crise sanitaire	Mission relative au contrôle qualité de la gestion de crise sanitaire	2020 Q2
GBR	Implementing employment support schemes in response to the COVID-19 pandemic	National Audit Office	2020 Q4
GBR	Investigation into the free school meals voucher scheme	National Audit Office	2020 Q4
GBR	The government’s approach to test and trace in England – interim report	National Audit Office	2020 Q4
GBR	Support for children’s education during the COVID-19 pandemic	National Audit Office	2021 Q2
GBR	Investigation into government funding to charities during the COVID-19 pandemic	National Audit Office	2021 Q2
GBR	Analysis of the health, economic and social effects of COVID-19 and the approach to tiering	Department of Health and Social Care	2020 Q4
GBR	Coronavirus Act 2020: the public sector equalities duty impact assessment	Department of Health and Social Care	2020 Q3
ISR	The Availability of Digital Devices and Internet Connections for Distance Learning among School-age Children	Knesset Research and Information Center	2020 Q3
ISR	An estimate of the impact of the corona crisis on the socio-economic situation	National Insurance Institute of Israel	2020 Q4
ISR	The impact of the recession due to the corona crisis on poverty, inequality and standard of living	National Insurance Institute of Israel	2020 Q2
ISR	The impact of the recession following the Corona crisis on living standards, poverty and inequality (Study No. 133)	National Insurance Institute of Israel	2020 Q2
ISR	The State of Israel Response to the Covid-19 Crisis - Special Interim Report	State Comptroller	2020 Q4
ISR	Social Security - Israel Journal of Social Policy	National Insurance Institute of Israel	2020 Q2
ISR	Special analysis of the Research Division of the Bank of Israel: The impact of the Bank of Israel's intervention in the government bond market at the height of the crisis	Bank of Israel	2020 Q2
ITA	The impact of Covid-19 on income inequality	Ministry of Finance	2020 Q2
ITA	The impact of Covid-19 and government interventions on the socio-economic situation of Italian families in the first three months of the pandemic	Ministry of Finance	2020 Q3
ITA	The impact of the Covid-19 pandemic on the Italian economy: illustrative scenarios	Bank of Italy	2020 Q2
ITA	Some preliminary estimates of the effects of support measures on the labour market	Bank of Italy	2020 Q4
ITA	The impact of the Covid-19 crisis on the access to the capital market of the Italian SMEs	Bank of Italy	2020 Q1
LTU	Ex-ante assessment of “Future DNA” spending	STRATA	2020 Q2
LTU	Expected impact assessments of Covid-19 recovery measures on inequality and poverty levels	Ministry of Social Affair and Labour	2020 Q2
MEX	La política social en el contexto de la pandemia por el virus SARS-CoV-2 (COVID-19) en México	CONEVAL	2020 Q2
NLD	An impact analysis of the COVID-19 on the regular health care	Sports & Health Ministry	2020 Q4
NOR	Socio-economic assessment of infection control measures - covid-19 - first report	Holden Expert Group	2020 Q2
NOR	Socio-economic assessment of infection control measures - covid-19 - second report	Holden Expert Group	2020 Q2
NOR	Socio-economic assessment of infection control measures - "Forced holiday" in December - Effects on the Norwegian economy	Holden Expert Group	2020 Q4
NOR	Socio-economic assessment of infection control measures - Third Report, Part I	Holden Expert Group	2021 Q1
NOR	White paper 2021:4	Commission ''Norway Towards 2025''	2021 Q1
NOR	First Interim Report - Input to the National Budget 2021 from the committee Norway towards 2025	Commission ''Norway Towards 2025''	2020 Q3
NOR	Second Interim Report - Input to the Perspective Report 2021 from the Norway to 2025 committee	Commission ''Norway Towards 2025''	2021 Q1
NZL	Ministry of Health: Management of personal protective equipment in response to Covid-19	Office of the Auditor-General	2020 Q3
NZL	Our intentions: Looking at how the Wage Subsidy Scheme is managed	Office of the Auditor-General	2021 Q2
NZL	Assessing Parliament’s Response to the Covid-19 Pandemic	Office of the Clerk of the House of Representatives	2021 Q1
SWE	The elderly care in the pandemic	Corona Investigative Committee	2020 Q4
SWE	Children and young people during the corona pandemic	Institute for Labour Market and Education Policy Evaluation (IFAU)	2021 Q1
USA	The Economic Impact of Coronavirus Response Funds - First Quarterly Report	OMB	2020 Q3
USA	The Economic Impact of Coronavirus Response Funds - Second Quarterly Report	OMB	2020 Q4
USA	COVID-19: Opportunities to Improve Federal Response and Recovery Efforts	Government Accountability Office	2020 Q2
USA	COVID-19: Federal Efforts Could Be Strengthened by Timely and Concerted Actions	Government Accountability Office	2020 Q4
USA	COVID-19: Federal Efforts Accelerate Vaccine and Therapeutic Development, but More Transparency Needed on Emergency Use Authorizations	Government Accountability Office	2020 Q4
USA	COVID-19: Urgent Actions Needed to Better Ensure an Effective Federal Response	Government Accountability Office	2020 Q4
USA	COVID-19: Critical Vaccine Distribution, Supply Chain, Program Integrity, and Other Challenges Require Focused Federal Attention	Government Accountability Office	2021 Q1
USA	The Effects of Pandemic-Related Legislation on Output	Congressional Budget Office	2020 Q3

Source: In the text

Notes

← 1.  Government is understood in a wide sense, as calls for contributions were sent to networks from the Centres of government, from the Supreme Audit Institutions, from Parliamentary Budget Offices and Independent Fiscal Institutions.

← 2.  Australia, Belgium, Canada, Colombia, Costa Rica, Denmark, France, Italy, Israel, Lithuania, Mexico, New Zealand, Netherlands, Norway, Switzerland, Sweden, United Kingdom, United States.

← 3.  Ex post impact evaluations were not available in the sample.

← 4.  50% of evaluations by federal countries assessed at least one aspect of whole-of-government response, compared to 15% of evaluations by the non-federal countries.