OECD Development Co-operation Peer Reviews: Austria 2020

6. Austria’s results, evaluation and learning

Abstract

This chapter considers the extent to which Austria assesses the results of its development co-operation; uses the findings of evaluations to feed into decision making, accountability and learning; and assists its partner countries to do the same.

It begins by looking at Austria’s system for managing development results, i.e. whether the objectives of its development co-operation policies and programmes can be measured and assessed – from output to impact. It then reviews the evaluation system for its alignment with the Development Assistance Committee (DAC) evaluation principles: is there an evaluation policy, are roles and responsibilities clear, is the process impartial and independent? Finally, it explores whether there is systematic and transparent dissemination of results, evaluation findings and lessons and whether Austria learns from both failure and success, and communicates what it has achieved and learnt.

In Brief

Austria sets impact goals to be delivered by each federal institution. However, reporting on institutional performance uses input-related corporate indicators, rather than assessments of outcomes or impact. The Austrian Development Agency and the Development Bank of Austria have improved their approach to results-based management, linking results to the Sustainable Development Goals (SDGs). Austria also draws on multilateral institutions’ corporate scorecards and results reporting. It could make better use of results information, in particular to communicate with the public and partners about Austria’s contribution to sustainable development in partner countries and territories, and globally.

Austria is developing a comprehensive evaluation policy, which will apply to the five key institutions responsible for development co-operation. While the Austrian Development Agency has an independent evaluation function, the Development Bank of Austria and the Ministry of Foreign Affairs situate their evaluation functions alongside policy and strategy. Austrian Development Cooperation undertakes strategic evaluations, but its approach to project evaluation could be more proportionate. Evaluation resources might be reallocated to increasing the number of strategic evaluations, including looking at challenges across Austria’s development system as a whole.

Institutional learning is a challenge for Austria given the fragmented nature of its development co‑operation system. Lessons are shared informally amongst Austrian development actors, but systematic dissemination is limited. The Austrian Development Agency systematically publishes evaluation reports and evaluation briefs on its website. It disseminates the results, evaluation findings and lessons of Austrian Development Cooperation and these inform programming. The Development Bank of Austria publishes short reports of ex-post evaluations.

Management for development results

Austria’s results system could focus more on its overall contribution to development

Austria has not yet fully realised its intention to enhance management for development results.1 Development co-operation objectives are set for each federal ministry in federal budget documents, but there is no comprehensive, whole-of-government statement outlining the expected results of Austria’s official development assistance (ODA) (Chapter 2). In addition, success at the ministry level is measured using input-related indicators rather than by assessing outcomes or impact.

In funding Austrian Development Cooperation (ADC), the Federal Ministry of Europe, Integration and Foreign Affairs (Ministry of Foreign Affairs, MFA) is expected to achieve sustainable poverty reduction; consolidation of peace and human security; preservation of the environment; and gender equality, including addressing the needs of children and people with disability. However, rather than assessing the outcomes or impacts in these areas, the ministry is required to report on input-related corporate indicators, such as the share of projects and programmes that promote gender equality or that meet ADC quality standards (MOF, 2018[1]).2

The ODA matrix in the three-year programmes on Austrian development policy for 2013-2015 (MFA, 2012[2]) and 2016-2018 (MFA, 2016[3]) described the outcomes Austrian Development Cooperation sought to achieve. However, neither the expected outcomes of multilateral co-operation, nor of the broader bilateral efforts of other Austrian stakeholders were described. This results in a partial picture of Austria’s overall contribution to the SDGs and to sustainable development in its partner countries and territories. The 2016-2018 document and its 2017 update (MFA, 2017[4]) did, nevertheless, include in the ODA matrix a list of additional Austrian stakeholders supporting priority themes in countries, territories and regions supported by ADC.

This fairly comprehensive approach to outlining measurable results has not been carried forward to the 2019-2021 three-year programme document, which does not contain an ODA matrix. The document does, nevertheless, link Austria’s five thematic priorities to the 2030 Agenda for Sustainable Development, its focus on leaving no one behind, and the Sustainable Development Goals (SDGs). This provides Austria with the basis for using the SDGs as a shared framework for results (Zwart, 2018[5]). Such an approach would define outcome indicators that are aligned to the SDGs and ensure synchronisation with partner governments’ results frameworks.

Individual Austrian institutions have improved their results-based management

In 2017, the Austrian Development Agency (ADA) enhanced its approach to managing project and programme results, introducing a results-focused monitoring and tracking system into its electronic aid management system. This includes a selection of SDG indicators, many of which are disaggregated by gender, age and level of vulnerability. Information about disability is obtained via the use of a policy marker.

Country strategies now include results matrices showing the link between Austria’s support, partner countries’ sustainable development objectives and the SDGs. While Austria aims to use partner countries’ data and systems where possible, the approach taken to programming and choice of implementing partners constrains its ability to do so (Chapter 5). At the project level, logical framework matrices are systematically included in project design documents, and reported against, enabling ADC to monitor the chain of expected results from outputs to eventual impact. This is good practice.

The Development Bank of Austria (OeEB) has recently introduced a development effectiveness rating tool (DERa) developed by Deutsche Investitions-und Entwicklungsgesellschaft mbH (DEG),3 which allows the bank to measure the development effects of its interventions.

Austria joins other donors in encouraging multilateral and regional institutions to apply results-based management in their operations and draws on their reporting. The Federal Ministry of Finance, for example, draws on performance reporting against the multilateral development banks’ corporate scorecards and results frameworks.

Austria could be clearer about how its activities contribute to sustainable development

Austrian institutions are using more readily available information about results to improve their programme management. However, it could make greater use of this information to communicate with the Austrian Parliament, public and partner governments about Austria’s contribution to the SDGs and to sustainable development in partner countries. Annual updates of the ODA matrix, as envisaged in the 2016-18 three‑year programme, have not eventuated, constraining parliament’s ability to exercise its oversight role.

Evaluation system

A new evaluation policy will apply to the five key development actors

Austria is finalising an inter-ministerial evaluation policy which will apply to ADA, OeEB and the ministries of finance, foreign affairs, and sustainability and tourism, and other development actors that choose to adopt it (MFA, 2019[6]). This is a very positive step towards achieving evaluation coverage of the entire aid system, as recommended in the 2015 peer review (OECD, 2015[7]).

The draft policy is comprehensive. It draws on evaluation principles, criteria and standards developed by the Development Assistance Committee (DAC), the German-speaking Evaluation Society (DeGEval),4 the United Nations Evaluation Group5 and the multilateral development banks’ Evaluation Cooperation Group.6 It outlines ten evaluation principles,7 including impartiality and independence from policy-making and programming functions. In evaluating its projects and programmes, Austria will look at the criteria of relevance, effectiveness, efficiency, coherence, impact, and sustainability, in line with the DAC evaluation criteria.8 Humanitarian evaluations will also consider connectedness, coverage and co-ordination. It will not be mandatory to apply all criteria. Instead, criteria will be selected based on their appropriateness to each evaluation. This is good practice. Roles and responsibilities are defined for each of the five development actors.

In ADA the evaluation function is kept independent from policy, programming and delivery.9 However, while the evaluation functions in OeEB10 and the Ministry Foreign Affairs11 are separate from programming and delivery, they sit alongside policy and strategy, which does not allow for sufficient independence. The ministries of finance and foreign affairs rely on evaluations undertaken by their multilateral partners, which is good practice.

Evaluations are prioritised, but could be more selective

ADA is responsible for evaluating Austrian Development Cooperation activities, and requires all projects to be evaluated.12 In addition, one or two strategic evaluations are done each year. By way of example, the 2019-20 Evaluation Plan, which is developed by the Ministry of Foreign Affairs in close collaboration with ADA, envisages that two country strategies will be evaluated in 2019 and that strategic evaluations in 2020 will focus on the human rights approach, a systematic review of results of the whole-of-government approach in ADC, and the evaluability of gender activities (ADC, 2019[8]). Implementation of the recommendations of strategic evaluations are monitored on a regular basis and reported to ADA’s Supervisory Board on an annual basis. This establishes a direct feedback loop to an oversight body in an effort to increase commitment to follow-up on recommendations, and promote institutional learning and use of evaluation findings at all levels.

While this approach is commendable, ADA could make better use of limited resources. Rather than evaluating every project it could be more selective about what is evaluated and draw more on the results of monitoring by implementing partners, as seen in Kosovo (Annex C). This point was raised in the report of a recent meta-evaluation of project and programme evaluations undertaken by ADA between 2016 and 2018 (Silvestrini and Bäthge, 2019[9]). Additional issues for the Ministry of Foreign Affairs and ADA to consider from this report include:

• ensuring that budgets are sufficient to achieve the intended scope of evaluations

• ensuring that terms of reference are adequate

• ensuring a proper inception phase

• ensuring evaluation results are disseminated and discussed.

Evaluations could assess challenges facing Austria’s development system

While the draft evaluation policy tasks each institution with evaluating its own development co-operation efforts, it does not explore the extent to which Austria achieves synergies across its development cooperation system as a whole. This is something that the Ministry of Foreign Affairs might consider as part of its responsibility for co-ordinating Austrian development policy. Resources might usefully be reallocated to increasing the number of strategic evaluations, including looking at challenges across Austria’s development system as a whole (Chapter 4).

Institutional learning

Institutional learning remains a challenge within the Austrian system

The fragmented nature of Austria’s development co-operation system makes institutional learning a particular challenge. Informal sharing of information does occur amongst Austrian stakeholders – for example ADA invites other stakeholders to attend learning events where it presents projects, evaluations and results. However, this approach is limited in its ability to disseminate lessons to a wide range of staff across Austrian development institutions and co-ordination offices, let alone to partners.

ADA has improved its dissemination of evaluation results. Evaluation reports are now systematically published on ADA’s website,13 and since 2017 these have been accompanied by evaluation briefs. Recommendations are followed up systematically by management and findings inform policy, as seen in the work by the Ministry of Foreign Affairs and ADA to prepare the latest round of country strategies. ADA’s knowledge management unit maintains a repository of information which is disseminated through newsletters and “good to know” messages. It also organises an annual conference, which serves as a retreat for ADA staff and the MFA’s Directorate General for Development Co-operation, facilitating mutual reflection, knowledge exchange and learning on questions of importance for global development co‑operation and ADA as an organisation. While internal lessons are available, the system does not yet capture lessons from implementing partners. The Development Bank of Austria posts summaries of evaluations on its website,14 but it could follow ADA’s example and publish the full reports, including methods and findings.

References

[8] ADC (2019), Strategic Evaluation Plan 2019/2020, Austrian Development Cooperation, https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluation_plan_2019-2020.pdf (accessed on 27 August 2019).

[11] Austrian Development Agency (2009), Guidelines for Project and Programme Evaluations, Austrian Development Agency, Vienna, https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Projektabwicklung/Englisch/Guidelines_for_Project_and_Progamme_Evaluations.PDF (accessed on 2 August 2019).

[6] MFA (2019), DAC Peer Review Memorandum, Federal Ministry of Europe, Integration and Foreign Affairs, Vienna.

[10] MFA (2019), Working together. For our world. Three-Year Programme on Austrian Development Policy 2019-2021, https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Publikationen/3_JP/Englisch/3JP_2019-2021_EN.pdf (accessed on 25 June 2019).

[4] MFA (2017), The future needs development. Development needs a future. Three-Year Programme on Austrian Development Policy 2016-2018, Update 2017, Federal Ministry for Europe, Integration and Foreign Affairs, https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Publikationen/3_JP/Englisch/2016-2018_3-YP_UPDATE_2017.pdf (accessed on 17 July 2019).

[3] MFA (2016), The future needs development. Development needs a future. Three-Year Programme on Austrian Development Policy 2016-2018, Federal Ministry for Europe, Integration and Foreign Affairs, https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Publikationen/3_JP/Englisch/2016-2018_3-YP.pdf (accessed on 17 July 2019).

[2] MFA (2012), Three-Year Programme on Austrian Development Policy 2013-2015, Federal Ministry for European and International Affairs , Vienna, http://www.entwicklung.at/fileadmin/user_upload/Dokumente/Publikationen/3_JP/Englisch/2013-2015_3-YP.pdf (accessed on 13 August 2019).

[1] MOF (2018), Bundesvoranschlag 2019 Untergliederung 12 Äußeres, Bundesministerium für Finanzen (Federal Ministry of Finance) Vienna, https://service.bmf.gv.at/BUDGET/Budgets/2018_2019/bfg2019/teilhefte/UG12/UG12_Teilheft_2019.pdf (accessed on 21 August 2019).

[7] OECD (2015), OECD Development Co-operation Peer Reviews: Austria 2015, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264227958-en.

[9] Silvestrini, S. and S. Bäthge (2019), Meta-Evaluation of ADA Project and Programme Evaluations, CEval GmbH, Saarbruecken, http://www.ceval.de.

[5] Zwart, R. (2018), “Using the SDGs as a shared framework for results: demonstrating good practice”, Discussion Paper for the OECD-DAC Results Community, OECD Publishing, Paris, http://www.oecd.org/dac/results-development/docs/OECD-SDG-Case-studies-discussion-paper-FINAL.pdf (accessed on 13 August 2019).

Notes

← 1.  All stakeholders of Austrian development policy agreed to enhance results management in a mission statement on 18 December 2012 and published in the three-year programme for 2013-15 (MFA, 2012[2]). This agreement is retained in the mission statements in the three-year programmes for 2016-2018 (MFA, 2016[3]) and 2019-2021 (MFA, 2019[10]).

← 2.  The MFA is required to report on the following four indicators: Indicator 12.4.1: Percentage of projects that provide access to water, energy, land and basic services, generate income and reduce poverty; Indicator 12.4.2: Share of ADC projects/programmes that promote equality between women and men; Indicator 12.4.3: Budget for financing new and current projects implemented in accordance with ADC quality criteria; Indicator 12.4.4: Share of ADC projects/programmes dedicated to environment and natural resources (MOF, 2018[1]).

← 3.  Deutsche Investitions-und Entwicklungsgesellschaft mbH (DEG) is a subsidiary of the German Kreditanstalt für Wiederaufbau Group (KFW Group). The Development Effectiveness Rating (DERa) tool uses five outcome categories: decent jobs; local income; market and sector development; environmental stewardship; and community benefits. The first three categories assess what was achieved and the other two, how these effects were achieved: For more information, see https://www.deginvest.de/International-financing/DEG/%C3%9Cber-uns/Was-wir-bewirken/Wir-messen-Wirksamkeit/.

← 4.  For more information, see https://www.degeval.org/home/. ADA and the Federal Ministry for Education, Science and Research are institutional members of DeGEval.

← 5.  For information about the UN Evaluation Group see www.uneval.org.

← 6.  For information about the Evaluation Cooperation Group, see https://www.ecgnet.org/about-ecg.

← 7.  These principles are independence, impartiality, credibility, transparency, usefulness, feasibility, fairness, accuracy, participation and partnership.

← 8.  Although these DAC evaluation criteria are currently being reviewed. For details see https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm.

← 9.  The Evaluation and Statistics executive unit, based in ADA’s Managing Directorate, has 2.5 full-time equivalent staff working on evaluation.

← 10.  Monitoring and evaluation are part of the development policy function in OeEB. An evaluation desk was established in 2018 and currently evaluates some, but not all, of the bank’s projects.

← 11.  The Department for Development Cooperation: Strategy, Public Relations and Evaluation of the Federal Ministry of Europe, Integration and Foreign Affairs is responsible for development co-operation strategy and evaluation. An evaluation unit was established in 2017 with 1.5 full-time equivalent staff.

← 12.  ADA’s Guidelines for Project and Programme Evaluations distinguish between internal evaluations, which are undertaken by the partner which implements the project, and external evaluations. The latter are of two types – those which are commissioned and managed by the ADA evaluation unit, and those which are commissioned and managed by another ADA unit at headquarters or by an ADA Co-ordination Office (Austrian Development Agency, 2009[11]).

← 13.  ADA’s evaluation reports are published here: https://www.entwicklung.at/en/ada/evaluation/evaluation-reports/.

← 14.  English language ex-post evaluations of three loan projects can be found at https://www.oe-eb.at/en/development-effects/measuring-results.html.