In 2017, an external evaluation assessed the quality of decentralised reviews and evaluations in Norwegian development co-operation and found them to be generally poor. The reviews were useful and timely, but the data and methodology were not sufficiently robust. To improve the quality, Norwegian Agency for Development Cooperation (Norad) now provides training, templates for terms of reference, and guidance tools. In addition to triggering institutional and process changes, a key message from the quality review is the need for realism in terms of the scope and resource needs of decentralised evaluations.
Improving the quality of decentralised reviews and evaluations in Norwegian development co-operation
Abstract
Challenge
Most of Norway’s evaluations of development co-operation are decentralised, i.e. commissioned by the unit at headquarters in Norway or at the embassy that manages the relevant programme. For Norway, these evaluations are a key component of project management and results-based management and should inform learning on what works, how and why. In 2017, an external evaluation of decentralised evaluations confirmed that staff considered them useful and timely for their work. However, the review identified several important challenges: evaluation methodologies were not sufficiently robust, terms of reference (ToRs) were not sufficiently clear and too ambitious, final reports contained weaknesses, and support and training for staff commissioning evaluations was insufficient.
Approach
Norwegian Agency for Development Cooperation (Norad) took a number of steps to improve the quality of its decentralised reviews and evaluations:
Clear support function. The Results Management Section in Norad was given a clearer mandate as a support function for decentralised evaluations.
Guidance, training and advice. The Results Management Section developed a practical training course for caseworkers, a template for developing ToRs, as well as a guide and a checklist on how to quality assure draft reports. It also provided technical advice when needed.
Communicating key principles for evaluation to the organisation as a whole and staff commissioning the decentralised evaluations, including that they should be:
Tailored: evaluations should be tailored to obtain specific information rather than large “tick the box” evaluations that cover all OECD DAC evaluation criteria.
Feasible: the scope of the evaluation and evaluation questions must be designed in a way that they can be addressed with a robust methodology within the available time and budget.
Setting clear quality requirements: staff were advised to include clear requirements in the ToRs, with a focus on the clarity of the methodology in inception reports, and to perform more critical quality assessments of draft reports.
Results
External quality reviews are now carried out annually. For example, a study assessing the quality of decentralised evaluations published from 2018-2019 was conducted in 2020. The study revealed that the overall quality of both ToRs and decentralised evaluation reports was still low and many of the findings from the evaluation in 2017 remained valid.
However, as most measures were implemented in late 2018 and 2019, seeing significant results by 2020 was not to be expected. Norad hopes that the next follow-up study on evaluations conducted in 2020 will reveal more positive results for ToRs, quality of reports and importantly, quality of the methodology. The annual quality reviews provide an excellent tool to monitor the effects of Norad’s measures over time.
Lessons learnt
Do not assume all evaluations are of acceptable quality. Norad was surprised by the poor quality of decentralised evaluations. Findings from the reviews enabled it to target specific weaknesses and work to improve these by developing fit-for-purpose guidance, templates, and training.
Strengthening the evaluation capacity of project experts is a critical challenge. Decentralised evaluations need to be tailored to the project to provide useful information and keep the scope and resource needs realistic. However, project experts are often not familiar with developing evaluation ToRs, calls for tender, assessments of consultants’ qualifications and methodological approaches. Norad is therefore considering developing more appropriate systems for quality assurance.
Balance is needed to improve quality within the limits of current capacity. Relevance and usefulness are more important than academic top scores. However, the methodology underpinning the findings in the reviews conducted must be robust.
Further information
Norad, Evaluation on The Quality of Reviews and Decentralised Evaluations in Norwegian Development Cooperation, 2017, https://www.norad.no/om-bistand/publikasjon/2017/the-quality-of-reviews-and-decentralised-evaluations-in-norwegian-development-cooperation.
Norad, Quality Assessment of Decentralised Evaluations in Norwegian Development Cooperation, 2020, https://www.norad.no/om-bistand/publikasjon/2020/kvalitetsvurdering-av-desentraliserte-evalueringer.
Supplementary Material – Guidance and checklist to quality assure draft evaluation report from Norad, https://www.slideshare.net/secret/9AJQtKvkfa1Rk5.
Supplementary Material - Reviews and evaluations how to make a ToR from Norad, https://www.slideshare.net/secret/oEskCHtjSudcVJ.
OECD resources
OECD, Evaluation of development programmes, https://www.oecd.org/dac/evaluation.
OECD (2019), OECD Development Co-operation Peer Reviews: Norway 2019, OECD Development Co-operation Peer Reviews, OECD Publishing, Paris, https://doi.org/10.1787/75084277-en.
To learn more about Norway’s development co-operation see:
OECD (2021), "Norway", in Development Co-operation Profiles, OECD Publishing, Paris, https://doi.org/10.1787/aaf0304f-en.
