This study is the most recent edition in a regular series that aims to prompt critical thinking around development co-operation and humanitarian aid evaluations, with a view to strengthening evaluation systems.
Evaluation Systems in Development Co-operation 2023
1. Background and methodology
Abstract
1.1. Background
Evaluation is a process that critically examines an intervention1 (e.g., a policy, programme or project) to better understand if and how it has delivered its intended results. Accountability and learning are the dual objectives of evaluation. Across the public sector, international development co-operation and humanitarian assistance have regularly been subject to relatively high levels of evaluation. This reflects the unique accountability dynamics of international co-operation, and the high degree of domestic scrutiny associated with this external spending. Evaluation is more critical than ever in the current global co-operation context, characterised by more complex and interlinked humanitarian and development challenges and increasingly scarce development co-operation resources.
Evaluation Systems in Development Co-operation is the most recent in a series of similar reviews and provides an update to reports published in 2010 and 2016 (Box 1.1). The overall aim of this series is to prompt critical thinking to strengthen evaluation systems for members of the Development Assistance Committee (DAC) Network on Development Evaluation (EvalNet) and beyond.
The study provides details on the entities, structures, mandates, policies, procedures, and resources that support the evaluation function in participating development organisations. It includes four substantive chapters organised around the following issues: the role of evaluation in development co-operation, covering the number and types of evaluations conducted; the policy and institutional arrangements put in place to govern evaluation systems; the evaluation process, from deciding on which evaluations to undertake to how findings are disseminated; and the use of evaluation findings to maximise development effectiveness and impact.
The core of the study is a comparative report that presents overall trends in evaluation systems, covering sustainable development and humanitarian assistance, with a focus on common challenges, good practice, and innovative ways of working. This is complemented by individual profiles of all participating organisations, which provide an in-depth look at the policies, principles, and ways of working.
Box 1.1. DAC reviews of evaluation systems: a brief history
The DAC Network on Development Evaluation conducts periodic reviews of its members’ evaluation systems to support its efforts to increase the effectiveness of development co-operation policies and programmes by promoting high-quality, independent evaluation and the development of institutional evaluation systems. Back in in 1975, at a time when evaluation policy and practice were still nascent, the DAC published its first study: “Aid Evaluation – The Experience of Members of the Development Assistance Committee and of International Organisations.”
The DAC first convened an expert group on evaluation in 1982, which took stock of member evaluation practices. Once formally established in 1983, the OECD DAC Network on Development Evaluation (EvalNet) periodically reviewed how members were conducting evaluations, both in terms of methods and systems. EvalNet meetings often featured peer exchanges and member updates on these topics.
Evaluation Systems in DAC Members' Agencies was published in 2004. This paper discussed some of the challenges members’ evaluation systems were facing, drawing mainly on data contained in DAC Peer Reviews between 1996 and 2004, as well as information provided over the years to the Secretariat by Evalnet members.
The next study in this series, ‘’A comparative Study of Evaluation in Policies and Practices in Development Agencies’’ was conducted in 2007. The study was commissioned by France to map and compare evaluation practices across development agencies, primarily to stimulate an internal debate within Agence Française de Développement (AFD) on its evaluation systems during a time of reform. Nine EvalNet agencies were reviewed as part of this study, and results confirmed that there was indeed variation among the strategies and practices adopted by different evaluation units.
In 2010, Better Aid: Evaluation in Development Agencies used an approach that has been used ever since. The report provided a snapshot of management and resourcing in 30 development evaluation agencies. It covered the topical trends in evaluation – how evaluations are moving away from project outputs to assess broader impacts of development assistance, adapting to new aid modalities and cross-cutting issues, and increasingly involving country partners. The report identified areas that required collaborative efforts, such as improving communication and supporting the use of evaluation findings to enhance the contribution of evaluation to achieving development goals.
Next in the series, Evaluation Systems in Development Co-operation, was conducted in 2016. The study reviewed the evaluation systems of 37 bilateral and 9 multilateral development agencies, providing insights into the role and management of evaluations. It covers topics such as institutional structures and governance, resourcing and management, and evaluation policies and processes. Additionally, the study reflects on general trends in evaluation systems and includes a comparative analysis with the 2010 study.
1.2. Scope and methodology
This study covers 34 of the 41 EvalNet members and observers2, representing 51 distinct institutions providing development co-operation and humanitarian assistance (Table 1.1). Throughout the text, “participating organisation” refers to those evaluation units on which data were collected, which includes both EvalNet members and EvalNet observers. For a full list of members and observers see Annex A.
To note, OECD member countries disburse and implement their development co-operation and humanitarian assistance in a variety of ways, often via multiple ministries or agencies. Their evaluation systems, and the institutional set up of central evaluation units, reflect these institutional structures. Several EvalNet members have multiple organisations that conduct relevant evaluations. In these cases, all relevant organisations have been included where possible, and are presented together in the member profile (Annex C). While the study does ask participants to report on the overall number of centralised and decentralised evaluations undertaken in 2021, it does not cover decentralised evaluation systems, which in some cases account for the vast majority of evaluations conducted by a participant, as discussed further in Chapter 2.
The study included a literature review of policy documents provided by members and published information on member and observer evaluation systems in 2021. This was followed by a qualitative survey with a questionnaire sent to participating organisations in 2022. Survey results were reviewed and enhanced through individual semi-structured interviews with 28 heads of evaluation units. Data profiles were prepared by the authors for all participating organisations, which were then validated by relevant organisation staff. It is important to note that this study focuses on centralised evaluation units. It does not attempt to provide detailed information on the results, monitoring and evaluation systems, including decentralised evaluations, undertaken by other parts of the participating organisations.
Figure 1.1. Research methods
All data and evidence presented in this report draw on the information collected specifically as part of the report process unless otherwise stated or referenced. Where a comparison is drawn with 2010 or 2016, this refers to data and evidence presented in the 2010 and 2016 iterations of this study (Box 1.1). It is important to note that the datasets from 2010, 2016 and 2022 vary slightly depending on EvalNet participation at the time of the study. Where relevant, the 2022 dataset has been adapted to ensure that comparisons over time are meaningful. Additionally, the data reported in the main report are current as of 15 March 2023. However, as minor revisions to country profiles were received after this date, there might be slight discrepancies in the trends reported in the report and the individual data reported in the profiles.
Table 1.1. Participating organisations
Notes
← 1. For the purposes of this report, references to development evaluation will be understood to mean evaluations of both development co-operation and humanitarian assistance activities, including contributions to multilateral institutions and United Nations (UN) organisations. Several members also have a mandate to evaluate the development effects of non-development specific policies and actions, including trade and foreign policy.
← 2. All 41 EvalNet DAC members and observer organisations participating in EvalNet as of end 2021 were invited to take part in the study. Seven members and observers with more nascent evaluation systems and limited data are not included in the overall trends analysis: the Asian Infrastructure Investment Bank (AIIB), Greece, Hungary, Poland, Slovak Republic and Slovenia. Lithuania joined the DAC at the very end of the study period and is therefore also not included.