Improving the quality of decentralised reviews and evaluations in Norwegian development co-operation

Challenge

Most of Norway’s evaluations of development co-operation are decentralised, i.e. commissioned by the unit at headquarters in Norway or at the embassy that manages the relevant programme. For Norway, these evaluations are a key component of project management and results-based management and should inform learning on what works, how and why. In 2017, an external evaluation of decentralised evaluations confirmed that staff considered them useful and timely for their work. However, the review identified several important challenges: evaluation methodologies were not sufficiently robust, terms of reference (ToRs) were not sufficiently clear and too ambitious, final reports contained weaknesses, and support and training for staff commissioning evaluations was insufficient.

Approach

Norwegian Agency for Development Cooperation (Norad) took a number of steps to improve the quality of its decentralised reviews and evaluations:

• Clear support function. The Results Management Section in Norad was given a clearer mandate as a support function for decentralised evaluations.

• Guidance, training and advice. The Results Management Section developed a practical training course for caseworkers, a template for developing ToRs, as well as a guide and a checklist on how to quality assure draft reports. It also provided technical advice when needed.

• Communicating key principles for evaluation to the organisation as a whole and staff commissioning the decentralised evaluations, including that they should be:

  • Tailored: evaluations should be tailored to obtain specific information rather than large “tick the box” evaluations that cover all OECD DAC evaluation criteria.

  • Feasible: the scope of the evaluation and evaluation questions must be designed in a way that they can be addressed with a robust methodology within the available time and budget.

  • Setting clear quality requirements: staff were advised to include clear requirements in the ToRs, with a focus on the clarity of the methodology in inception reports, and to perform more critical quality assessments of draft reports.

Results

External quality reviews are now carried out annually. For example, a study assessing the quality of decentralised evaluations published from 2018-2019 was conducted in 2020. The study revealed that the overall quality of both ToRs and decentralised evaluation reports was still low and many of the findings from the evaluation in 2017 remained valid.

However, as most measures were implemented in late 2018 and 2019, seeing significant results by 2020 was not to be expected. Norad hopes that the next follow-up study on evaluations conducted in 2020 will reveal more positive results for ToRs, quality of reports and importantly, quality of the methodology. The annual quality reviews provide an excellent tool to monitor the effects of Norad’s measures over time.