OECD Development Co-operation Peer Reviews: Japan 2020

6. Japan’s results, evaluation and learning

Abstract

This chapter considers the extent to which Japan assesses the results of its development co-operation, uses the findings of evaluations to feed into decision making, accountability and learning, and assists its partner countries to do the same.

The chapter begins with a look at Japan’s system for managing development results, i.e. whether the objectives of its development co-operation policies and programmes can be measured and assessed from output to impact. It then reviews the alignment of Japan’s evaluation system to the DAC evaluation principles, looking specifically at whether an evaluation policy is in place, whether roles and responsibilities are clear, and whether the process is impartial and independent. Finally, it explores whether there is systematic and transparent dissemination of results, evaluation findings and lessons and whether Japan learns from both failure and success and communicates what it has achieved and learned.

In brief

Japan’s projects are thoroughly planned and evaluated following to the “Plan, Do, Check, Action” cycle drawn from Japanese manufacturing. Ex-ante evaluations are required for all official development assistance (ODA) loan projects – these outline expected outputs, external risk factors, and lessons learned from similar past projects. At the project level, Japan has logframes and outcome targets, but it does not set out in measurable ways the broader development results it seeks to achieve or contribute to. Country Development Cooperation Policies do not include indicators and are not used to define the results that Japan wishes to achieve even though these policies systematically build on the partner country’s development plan and the Sustainable Development Goals (SDGs). Like other Development Assistance Committee (DAC) members, the key constraints to adopting a results-based management system relate to the organisational culture and the lack of an incentive structure.

Japan uses the DAC evaluation criteria and also assesses diplomatic goals in evaluations, in line with the objective of the Development Cooperation Charter. The 2014 Peer Review suggested Japan might adopt a more strategic approach to the projects that are evaluated, which is work in progress. While ex-post evaluations are still standard practice for every project costing over USD 1.8 million, midterm evaluations are no longer obligatory for the Japan International Cooperation Agency (JICA). JICA has also adopted a pragmatic approach to conducting ex-ante and ex-post evaluations of multiple projects with the same purpose. JICA is currently working to build its evaluation methodology for private sector investment finance based on good practice standards.

The new DAC evaluation criteria will provide Japan with opportunities to match evaluation efforts with learning and accountability needs. Japan’s continued emphasis on comprehensive and joint evaluations with partner countries is particularly noteworthy. It has invested in building the evaluation capacity of national authorities in Viet Nam and Myanmar and has conducted a number of joint evaluations with partner countries, including the Philippines, Thailand and Viet Nam. Japan also supports the statistical capacity of partner countries through in-kind technical co-operation and earmarked funding.

JICA has a public website on lessons learned which must be consulted before a new project design is approved so as to incorporate lessons from similar projects. The findings of ex-post evaluations are also available publicly on JICA’s official website. Evaluation results are fed back into strategies and thematic guidelines, reflected in new and ongoing projects, and fed into the programmes and policies of partner countries.

JICA has a strong internal knowledge management system in place to share lessons and challenges across 19 communities of practice in specific sectors and areas. Japan would gain from extending this system to all its development co-operation actors. JICA considers better knowledge management as the key to promoting innovation in development co-operation.

Managing for development results

Building on its strong monitoring and evaluation mechanisms at project-level, Japan can improve its comprehensive results-based management approach

Japan’s projects are thoroughly planned and evaluated following the “Plan, Do, Check, Action” cycle drawn from Japanese manufacturing. Ex-ante evaluations are required for all ODA loan projects and outline the rationale for the project and a description of target outcomes, such as improved living standards for a regional infrastructure project or reduced pollutants in channels for a sewerage system project.1 The evaluation looks at expected outputs, external risk factors and lessons learned from similar past projects.

The broader development objectives are not established. Japan has logframes and outcome targets for projects, but it does not set out in measurable ways the broader development results it seeks to achieve or contribute to through a clearly articulated chain of expected results from inputs and outputs to outcomes and impact.2 JICA is renowned for its bottom-up approach that leads directly to tangible outcomes at a local level and in which all stakeholders participate. The link between policy, its implementation and the results on the ground could be established through strengthened dialogue between the bottom-up approach and institutional or system-wide reform in partner countries (Yoshida, 2018[1]). This is all the more important given the size of Japanese investments as the fourth largest DAC donor overall.

A more strategic approach to results is needed. While Country Development Cooperation Policies, or country strategies, build on partner countries’ development plan and the SDGs, they do not include indicators and are not used to define the results that Japan wishes to achieve or how it contributes to meeting the SDGs. As a result, third-party country assistance evaluations find it difficult to assess the actual impact, broad relevance or specific contribution of Japan’s development co-operation since these results or collective efforts with the partner country and stakeholders are not set out at the beginning. An overall results framework for each country could help bring together all of Japan’s investments and identify synergies among the various investments and instruments. Project and programme indicators are drawn from partner country national data sources where possible and align to national strategies (Chapter 5). However, they could more explicitly demonstrate how investments link to the SDGs or contribute to partner countries’ own development outcomes.

Statistical support to partners and joint quality assurance monitoring are good practice. Japan supports the statistical capacity of partner countries through in-kind technical co-operation and earmarked funding provided through the Association of Southeast Asian Nations (ASEAN), the United Nations Economic and Social Commission for Asia and the Pacific, and the World Bank to national statistical agencies and regional statistics institutes.3 Every six months, and sometimes more frequently depending on the stage of project implementation, JICA, together with local and national authorities, monitors progress for quality assurance and issues a report of social and environmental conditions, which helps to avoid parallel monitoring efforts.4 This participatory approach, including by piloting the problem-driven iterative adaptation in conflict-affected areas, is good practice.

Monitoring and results are used for accountability to track project and output delivery and to verify internal controls, as well as to inform new phases or future projects. Where possible, on-the-spot monitoring allows for the adjustment of some parameters, which is more easily done for loan projects. There is an opportunity for JICA to look beyond the immediate delivery of outputs of its own projects to consider how these are part of a broader outcome.

Building a culture of results and learning, backed by appropriate incentives, is a challenge for Japan as for other DAC members. Japan could draw on work by the OECD’s Development Co-operation Directorate to develop tools and guidance for implementing the Guiding Principles on Managing for Sustainable Development Results. This work could help Japan to learn from other DAC members’ successes and failures in establishing a coherent system-wide approach to results that is linked to the objectives it wants to achieve. A shift towards managing for results could alleviate the need for systematic evaluations in order to adopt a more strategic approach to evaluation, as recommended in the 2014 Peer Review (OECD, 2014[2]).

The evaluation system

Japan’s evaluations follow DAC criteria and also consider diplomatic objectives

Evaluations are planned annually and the project cost is the determining factor for whether or not to conduct an external evaluation for JICA. The Ministry of Foreign Affairs (MOFA) has an annual evaluation plan drafted by the Evaluation Division and approved by the Management (OECD, 2016[3]). JICA does not have an evaluation plan, but it publishes requests for proposals for external evaluations via its procurement portal on an annual basis. MOFA conducts its own policy evaluations but hires third parties for ODA evaluations of country or regional assistance, themes, or aid modalities. External evaluators are commissioned for all JICA projects costing over USD 9 million (JPY 1 billion). JICA carries out internal ex-post evaluations for all projects costing over USD 1.8 million (JPY 200 million), and less than USD 9 million, Ex-post evaluations are budgeted separately from the initial project cost. The regional or sector department is responsible for appropriating the budget for ex-ante evaluations as part of the initial project cost and the evaluation department appropriates the budget for ex-post evaluations. In fiscal year 2017, there were 86 external ex-post evaluations of JICA projects (JICA, 2019[4]).

Japan evaluates investments from a diplomatic perspective. Japan assesses its investments relative to national interest, in line with the 2015 Development Cooperation Charter. These evaluations are guided by standardised questions.5 For example, the 2017 Country Assistance Evaluation of Cambodia highlights the strong economic co-operation between the two countries, and suggests that Japan should consider that there may be differences between the needs of the general Cambodian population and those identified or prioritised by the Cambodian Government (Waseda University, 2018[5]). This is a good example of how Japan incorporates the ‘coherence’ criteria recently added to the DAC evaluation criteria.

Japan has made some progress since the 2014 Peer Review. The last peer review suggested Japan might adopt a more strategic approach to the projects that are evaluated by using criteria such as risk or innovation (Annex A), and progress is being made in this regard. While ex-post evaluations are still standard practice for every project costing over USD 1.8 million, midterm evaluations are now no longer obligatory for JICA in order to allow flexibility in project monitoring. JICA has also adopted a pragmatic approach to conducting ex-ante and ex-post evaluations of multiple projects with the same purpose. For example, ex-post evaluations can integrate ODA loans and technical co-operation projects or grant aid and technical co-operation projects that share the same purpose (JICA, 2019, p. 7[4]). All evaluations are published online, including summaries in English.

Japan has an opportunity to make its evaluations more strategic. Both MOFA and JICA have a sound evaluation policy and guidelines (MFA, 2019[6]). These are aligned with the previous DAC evaluation criteria and will be adapted to the new DAC evaluation criteria.6 Japan should make most use of this adaptation process to match evaluation efforts with learning and accountability needs in order to develop more strategic evaluations that look beyond project delivery to more systemic and programme-based outcomes. JICA is currently working to build its evaluation methodology for private sector investment finance based on good practice standards that consider the particularities of this modality.

Annual evaluation reports compile the results of all evaluations. MOFA’s ODA evaluations, including the ratings and response measures to the recommendations, are compiled in annual reports and available online. JICA’s external evaluations ratings (A-D) based on subcomponents related to effectiveness and impact; relevance; efficiency; and sustainability equally feature in annual online reports. The reports analyse evaluation results and feature overviews of the year’s evaluations, including those of less successful interventions.7 This is an important learning practice, highlighting how things can be done differently in future interventions (JICA, 2019[4]). The JICA report draws practical lessons to inform future projects and policies, such as challenges and recommendations in external ex-post evaluations in Afghanistan (Chapter 7).

Box 6.1. The results of randomised control trial studies by JICA

JICA conducts a plethora of research on the impacts of Japan’s development co-operation. A few randomised control trials are summarised below:

• Value of microcredit in increasing rice productivity in Tanzania. The Bangladesh Rural Advancement Committee (BRAC) and JICA provided microcredit in the form of a fertiliser voucher and cash with the aim to increase rice cultivation productivity in Tanzania. The randomised control trial found weak evidence that the programme increased the use of chemical fertiliser. Credit use did not result in an increase in paddy yield, profit from rice cultivation, or household income for borrowers. In this case, the evidence supported previous researchers’ findings on the importance of extension services prior to distribution of the input (Nakano and Magezi, 2019[7]).

• Increasing public pension participation rates in Mongolia. A study in Mongolia found that providing information about subsidiary monetary benefits (survivors’ and disability pensions) did not increase participation substantially. What did increase participation was providing information about the mobile phone payment of pension funds and dispatching foreign aid officials (JICA) to a pension administrative agency. In this case, perceived transaction costs and trust directly affected demand for pension services (Tanaka et al., 2019[8]).

• Value of non-formal education for disadvantaged pupils in Bangladesh. The impact of the Kumon learning method, which enables each student to learn in accordance with his or her academic ability, on improving the cognitive and non-cognitive abilities of disadvantaged pupils in Bangladesh studying at informal primary schools was assessed. The results show that students of both grades in the treatment schools where the Kumon method is used record substantial and significant improvement in their cognitive abilities after a period of eight months, compared to students in the control schools. The results suggest the wider applicability of a properly designed non-formal education program in solving the learning crisis in developing countries (Sawada et al., 2017[9]).

Japan values joint evaluations and strengthening the capacity of partner countries

Japan’s continued emphasis on comprehensive and joint evaluations with partner countries is particularly noteworthy. It has invested in building the evaluation capacity of national authorities in Viet Nam and Myanmar and has conducted a number of joint evaluations with partner countries including the Philippines, Thailand and Viet Nam. Japan places high value and importance on conducting joint site visits and joint evaluations with official counterparts in ministries and implementing agencies and strengthening their capacity through its technical assistance; this was confirmed by government officials in Ghana and Cambodia. Evaluations play a key role in informing subsequent phases and future project design.

JICA has also been evaluated along with other development partners. In one recent example, the Government of the Philippines sought permission to conduct an evaluation of the various projects with different modalities, terms, and conditions provided by JICA, the World Bank, and the French Development Agency (AFD) in the aftermath of Typhoon Yolanda (JICA, 2019, p. 35[4]).

Institutional learning

Japan shares evaluation findings widely and they inform new projects

Evaluation findings are widely incorporated into future work. Although JICA does not provide official management responses to evaluations8, once an ex-post evaluation has been completed, monitoring determines the extent to which measures and actions have incorporated the evaluation recommendations. The findings of ex-post evaluations are available publicly on JICA’s official website. Evaluation results are fed back into strategies and thematic guidelines, reflected in new and ongoing projects, and fed back to the programmes and policies of partner countries. For example, in Ghana, a final evaluation of the community health project recommended involving the central government for reasons of sustainability, but also to ensure that central government “owned” the tools and systems to make the national scale-up more likely. This recommendation was taken on board by JICA in a follow-up project.

JICA’s public website on lessons learned from its evaluations9 must be consulted before a new project design is approved to ensure experience is fed into similar projects. In this way, it is able to scale-up good practices and results across its broad range of programming. In a few cases though, referring back to previous projects’ successes and learning from failures can stifle experimentation and contextualisation.10 Japan could also do more to systematically draw lessons from partner governments and other stakeholders.

JICA does well at sharing knowledge across communities of practice

JICA has a strong internal knowledge management system in place to share lessons and challenges across 19 communities of practice in specific sectors and areas. The system works well to share information and to connect Tokyo with country offices. In addition to JICA’s publications and seminars, JICA’s knowledge is widely shared through the country-based ODA Task Force meetings, consisting of Japanese embassies and JICA’s overseas offices, as well as overseas offices of the Japan External Trade Organisation (JETRO), the Japan Bank for International Cooperation (JBIC), and private companies in more than 130 partner countries and regions. MOFA’s International Co-operation Bureau is responsible for disseminating evaluations and lessons learned via the intranet and conducts regular trainings on ODA evaluations. For JICA, better knowledge management is seen as key to promoting innovation in development co-operation.

References

[4] JICA (2019), JICA Annual Evaluation Report 2018, Japan International Cooperation Agency, https://www.jica.go.jp/english/our_work/evaluation/reports/2018/c8h0vm0000f4nmle-att/full_2018_a4.pdf.

[6] MFA (2019), ODA Evaluation Guidelines, https://www.mofa.go.jp/policy/oda/evaluation/basic_documents/pdfs/guidelines12th.pdf (accessed on 25 February 2020).

[7] Nakano, Y. and E. Magezi (2019), The impact of microcredit on agricultural technology adoption and productivity: evidence from randomized control trial in Tanzania, https://www.jica.go.jp/jica-ri/publication/workingpaper/l75nbg000018sx0m-att/JICA_RI_WP_No.193.pdf (accessed on 25 February 2020).

[3] OECD (2016), Evaluation Systems in Development Co-operation: 2016 Review, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264262065-en.

[2] OECD (2014), OECD Development Co-operation Peer Reviews: Japan 2014, OECD Development Co-operation Peer Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264218161-en.

[9] Sawada, Y. et al. (2017), Individualized Self-learning Program to Improve Primary Education: Evidence from a Randomized Field Experiment in Bangladesh, https://www.jica.go.jp/jica-ri/publication/workingpaper/l75nbg00000g3kap-att/JICA-RI_WP_No.156.pdf (accessed on 25 February 2020).

[8] Tanaka, T. et al. (2019), Barriers to Public Pension Program Participation in a Developing Country, https://www.jica.go.jp/jica-ri/publication/workingpaper/l75nbg0000196kst-att/JICA_RI_WP_No199.pdf (accessed on 25 February 2020).

[5] Waseda University (2018), Country Assistance Evaluation of Cambodia: Third Party Evaluation, https://www.mofa.go.jp/policy/oda/evaluation/FY2017/pdfs/cambodia.pdf (accessed on 25 February 2020).

[1] Yoshida, K. (2018), “The policy-implementation-results linkage for education development and aid effectiveness in the “Education 2030” era”, Compare: A Journal of Comparative and International Education, Vol. 48/1, pp. 39-55, http://dx.doi.org/10.1080/03057925.2017.1283583.

Notes

← 1.  For a list of ex-ante evaluations, see www.jica.go.jp/english/our_work/evaluation/oda_loan/economic_cooperation/index.html.

← 2.  While the Administrative Review and the Government Policy Evaluation Act reinforces its role, Japan does not have a structured results-based management system in place.

← 3.  Total support to statistical capacity building was USD 7.7 million in 2017 and 2018.

← 4.  An example of environment and social monitoring for the Olkaria V Geothermal Project in Kenya can be found here: www.jica.go.jp/english/our_work/social_environmental/id/africa/kenya/c8h0vm000090rho9-att/c8h0vm0000ez6qn7.pdf.

← 5.  Questions for the ODA evaluation of diplomatic viewpoints include: (1) How is Japan’s ODA geopolitically important?; (2) How important is the bilateral relationship between Japan and the recipient country?; (3) How has Japan’s ODA contributed to strengthen bilateral relationship politically and economically?; (4) How has Japan’s ODA promoted the recipient country people’s understanding of Japan? Source: www.mofa.go.jp/mofaj/gaiko/oda/files/000337119.pdf.

← 6.  For an explanation of the criteria, see www.oecd.org/dac/evaluation/revised-evaluation-criteria-dec-2019.pdf.

← 7.  For example, a water supply and sewerage project in Peru improved services in provincial cities, but the volume of sewerage received exceeded the planned volume, pointing to the importance of accurate demand forecast.

← 8.  MOFA provides official management responses to evaluations through feedback meetings.

← 9.  Available at www.jica.go.jp/english/our_work/evaluation/lessons/index.html.

← 10.  This was mentioned by some staff members in the field who sometimes felt limited by what they could do based on past projects.