Effective Results Frameworks for Sustainable Development

5. Strengthening monitoring systems

Guiding principles

Development organisations strengthen and maximise use of partner countries’ monitoring and statistical systems, enhancing national capacity to plan and develop results frameworks as well as to produce and analyse data, including disaggregated data to capture results related to populations left behind.

Monitoring and evaluation systems are set up to produce credible quantitative and qualitative evidence that meets the needs and capacities of the development organisation and its partners.

OECD (2019[1]) Managing for Sustainable Development Results, Guiding Principles 3 and 6

Monitoring and evaluation are vital sources of evidence and data. Timely, complete and reliable data are essential, not only for guiding implementation at project level, but also for designing policies, informing strategic programming, fostering institutional learning, and reporting on progress towards achieving the SDGs. Credible quantitative and qualitative evidence also supports country ownership, transparency, mutual accountability and inclusive partnerships. Crises show the importance of getting timely, secure and disaggregated data to guide the response effectively. Yet, at the same time, they risk provoking disruptions to monitoring and reporting capacity, calling for innovative approaches such as remote monitoring.

Major opportunities are arising with the data revolution. Still, the continued scarcity of solid, objective and disaggregated data and statistics in countries continues to pose a significant risk to delivering Agenda 2030. Many countries continue to lack data systems that are fit for policymaking and accountability to their own people. Furthermore, statistical and monitoring systems of partner countries are underutilised by development organisations.

This requires development organisations to strengthen their efforts to establish agile monitoring and evaluation (MEL) systems that are not only fit for their needs and those of their partners but are also able to connect to and draw on partner country actors’ systems and data, and to harness digitalisation. In addition to this, the GPEDC Geneva Summit Declaration calls on development partners to support countries’ national statistical and information management systems, including in their digital transformation, and their ability to improve data quality and disaggregation, and to strengthen their capacity in the collection and processing of information, data management and use for statistics in development. Supporting locally led development also requires adapting MEL systems to align to, use and strengthen local actors’ systems.

Issues covered in this chapter

• Monitoring and evaluation systems are fit for purpose

• Countries’ monitoring and statistical systems are strengthened and used effectively

• Both quantitative and qualitative information is collected

5.1. Monitoring and evaluation systems are fit for purpose

5.1.1. Why is this important?

Fit-for-purpose monitoring and evaluation (M&E) systems provide management and their partners with credible quantitative and qualitative evidence that is vital to steer, learn, report and communicate on development efforts.

• Monitoring helps to guide project implementation towards the expected results, providing indications of implementation progress and other important context-related information. It is a continuing process organised by the project manager that involves the systematic collection of data.

• Evaluation provides an objective assessment of the relevance, coherence, effectiveness, efficiency, impact and sustainability of an intervention. It happens at a given time during the course or at the end of an intervention.

Project-monitoring requires that data be collected at regular intervals. Data is usually collected by staff working on the ground and using paper or spreadsheets. Two interconnected evolutions are taking place: i) remote monitoring approaches, building on experience working in crisis situations (such as Covid-19 or conflict-sensitive contexts), and ii) digitalisation, eased by increasing internet connectivity and new IT tools. Each comes with specific opportunities and challenges.

Monitoring remotely

Remote monitoring is in line with recent localisation trends emphasising the need to empower local partners. Distant monitoring methods used in conflict-sensitive contexts were extended during the COVID-19 pandemic, when monitoring and reporting capacity was disrupted, with staff having difficulties accessing project-implementation sites and interviewing stakeholders.

Ensuring the quality of data shared by partners requires specific measures. This includes developing clear guidelines setting reasonable monitoring standards. It also calls for balancing the use of digital technologies with other data-collection methods. This might be needed to reach the marginalised and most vulnerable people who do not use online services due to a lack of network infrastructure, lack of affordable internet service and devices, and/or gaps in skills and ability.

Harnessing digitalisation

Digitalisation offers many benefits to development organisations’ leadership, staff and partners. For senior management, benefits include the ability to quickly view and communicate progress, with more efficient M&E and reporting processes. For staff and partners, the benefits may include the ability to access and analyse data reported in the system by themselves and others.

Still, many development organisations are struggling to establish adequate information systems. Even sophisticated digitalisation tools can be difficult to implement and require significant time and resources to set up properly. Digitalisation efforts often require a change from “business as usual” information-sharing processes. It also involves significant up-front and ongoing financial and human-resource investments. For this reason, dedicated attention is needed to gain staff and management buy-in for new processes, and to communicate new processes to partners.

Specific consideration should be given as to how to handle qualitative information. While there are interesting opportunities to use machine-learning and artificial intelligence to process qualitative data, these innovations should be balanced with a human-centred approach that is able not only to look back at what has already been done, but also to look towards the future.

5.1.2. Self-assessment questions

Are your monitoring and evaluation systems set up to produce credible quantitative and qualitative evidence? Are the systems adequately resourced? In the list below, consider which priority actions are needed to ensure your system meets your needs and capacities and those of your partners.

To what extent does your information system support the collection, aggregation, analysis and sharing of results information? Is your system digitalised in a way that meets your needs while ensuring connectivity with the systems of your partners?

Do you have experience with remote monitoring? If so, what worked well and what needs to be further developed to ensure that quality and comprehensive information on programme implementation is delivered on time (e.g. guidelines, training, IT equipment)?

5.1.3. Actions to consider and pitfalls to avoid

Monitoring systems

• Design monitoring plans associated with programmes and projects in ways that specify the frequency, scope and intended use of the monitoring data, i.e. the elements that should be monitored regularly to help make informed decisions. These elements relate to:

  • Time: are the activities delivered on time?

  • Cost: is the budget spent according to plan?

  • Scope: do the actions follow the plan?

  • Quality: are the outputs being delivered with the expected quality?

  • Human resources: are staff performing their tasks as expected? Is the team working efficiently?

  • Context: are risks and opportunities arising?

  • Use: are the intended uses of results data clear and spelled out?

• Align data-collection tools to the results framework: monitoring reports are made by comparing programme and project achievements against expected results, timelines and costs.

• Tailor data-collection methods to each context. In some contexts, using non-digitalised collection methods is needed to reach those furthest behind. In others, qualitative or remote methods might be more fitting.

• Where it exists, use the organisation’s digitalised information systems for reporting data at the country or programme level. This will help aggregate data and therefore facilitate its use in centralised systems.

  • Avoid: Ignoring standard practices for collecting similar data in the country or territory of interest. Are other government entities or partners already collecting equivalent data? Can these processes be synchronised or harmonised to increase the quality, depth and sustainability of data collection in that particular context?

Digitalisation/Information systems

• Secure internal buy-in, communicating on the benefits of digitalisation, broadly consulting internally and/or making it a high-level priority.

• Take an iterative approach that seeks to achieve progress through gradual change and feedback. This can involve piloting new approaches before expanding their use or progressively increasing the number of projects captured in the information system.

• Use the digitalisation process to reflect on and clarify what information is actually essential, seizing the opportunity to rationalise what elements to capture and how.

• Provide appropriate training to help ensure the success of new digital tools.

• Select systems that are manageable, adaptable and reviewed on a regular basis to ensure they still fit the needs of their users.

5.1.4. Tips to go a step further

Monitoring in crisis situations: Remote methods for data-monitoring using telephone, emails (for documented evidence) and (online) survey tools may be required in conflict situations and other crises which limit access to key stakeholders. Specific accompanying measures are needed.

• Simplify results monitoring and add flexibility in deadlines as a short-term response (e.g. keeping financial reporting while adjourning ”narrative” results reporting requirements).

• Draft a plan on how to collect data when travel is restricted. This can include:

  • collecting contact details;

  • setting up digital training methods to instruct local data collectors on efficient data-collection practices;

  • planning regular data discussions to verify the accuracy of the data.

• Use larger samples when relying on digital data-collection methods, given that higher non-response rates can be expected. Concerns related to data protection favour using appropriate digital technologies, such as secure mobile applications.

• Triangulate information. Triangulation of quantitative and qualitative data from various sources using diverse methods is useful to get a comprehensive and objective picture.

  • Complement quantitative data, especially that collected via digital methods, with additional data-collection methods and qualitative information. This is needed to overcome reporting gaps and get the views of marginalised groups such as the elderly, poor or illiterate, as well as, in some cultures, women.

  • Check data provided by partners and use third-party monitoring whenever possible to reduce reporting bias. This can be done through citizen journalists and peer data collection, using existing relations with communities, or relying on secondary sources provided by GPS, satellite imagery, photos and videos. Reducing reporting bias also requires using female surveyors to speak to women.

• Develop risk assessments and mitigation plans This is needed to ensure the safe involvement of local partners. The mitigation plan should identify alternative data sources to replace or complement the primary data source. It should also include a remote verification plan listing specific tasks, documentation and possibly remote training options for staff so that the data collected meets the quality agreed with partners.

Evaluation: for specific information and guidance on evaluation, consult the six DAC evaluation criteria – relevance, coherence, effectiveness, efficiency, impact and sustainability – and the two principles for applying those criteria here. These criteria provide a normative framework used to determine the merit or worth of an intervention (policy, strategy, programme, project or activity). They serve as the basis upon which evaluative judgements are made. Broader information on evaluation systems is available on the EvalNet website. See also Table 2.2 for insights on how to assess SDG indicators with an evaluative eye.

5.1.5. Good practices

Digitalisation, information systems and artificial intelligence

Investing in digital systems that support the identification, management and reporting of results data has been a recognised gap and a priority for governments and international development partners.

Several development organisations have used a phased approach to secure staff and implementing-partner buy-in when upgrading their information systems or transitioning to a new information platform. In the case of the European Union, increasing the number of projects captured in the system with each reporting cycle allowed for the observation of increased efficiencies over time. Other organisations: i) relied on external experts for reporting data on an interim platform, reducing the immediate workload of staff and partners; ii) ensured that the final system would feed information back (e.g. in the form of dashboards), providing a benefit to staff in exchange for the additional workload; or iii) planned final trainings that went beyond ‘point and click’ functionality, to focus on topics related to results-based management and reporting good practice.

Some organisations, such as the Asian Development Bank, are conducting early-stage pilots of using basic artificial intelligence to support the problem and results analyses that inform project design. An important consideration in exploring such initiatives has been managing expectations, namely about the continued importance of human inputs to ensure contextual relevance and the examination of innovative approaches.

Box 5.1. Strengthening digital interlinkages between results frameworks

Information systems to support the collection and aggregating of results monitoring data

In areas with robust internet connectivity and established data protocols, cloud-based monitoring platforms enhance the speed and usability of results data reporting, leading to better management and decision-making. Interactive online templates used at the design stage can also facilitate vertical alignment to strategic priorities in a dynamic manner. While the internet and advanced digital solutions can greatly streamline data collection, their utility is contingent on a thorough understanding of the decision-making processes integral to implementing the programme or intervention. In situations where real-time, cloud-based data entry isn't feasible, alternative strategies should be in place, such as the use of standardised spreadsheet templates which can be shared via email or even physical copies, if necessary.

In practice, deploying systems for better management of results data is more than adding a new app

To ensure results data availability for programme and project monitoring, and to ensure more efficient and timely corporate results reporting, the European Union designed and rolled out OPSYS in 2022-23, a homegrown solution that allows the encoding of all logframes in a structured system, incentivises the use of standardised indicators to report on various corporate results frameworks, and facilitates quick access to indicator values in a dashboard. In addition to customising the IT solution (rather than relying on an off-the-shelf product), thorough change management strategies, training and close involvement of operational and managerial units across the results chains were shown to be important for the successful deployment of the system.

Sources: World Bank (2012[2]), Designing a Results Framework for Achieving Results, https://documents1.worldbank.org/curated/en/331541563854787772/pdf/Designing-a-Results-Framework-for-Achieving-Results-A-How-to-Guide.pdf, EU (2023[3]), OPSYS, https://wikis.ec.europa.eu/display/ExactExternalWiki/FAQ+-+OPSYS

5.1.6. Resources

• USAID (2023), Monitoring Toolkit, A set of guidance, tools, and templates for monitoring USAID strategies, projects, and activities

• Finland, Guidelines for programme design, monitoring and evaluation

• BMZ, Digital Global - Tools of Development Co-operation

• GIZ, Online Monitoring Tool – More efficient reporting for the German Development Co-operation by means of web-based data collection and analysis

• OECD (2020), Managing for Sustainable Development Results in Uncertain Times

• OECD (2021), Development Co-operation Report 2021: Shaping a Just Digital Transformation

5.2. Countries’ monitoring and statistical systems are strengthened and used effectively

5.2.1. Why is this important?

National monitoring and statistical systems play a key role in designing public policies, monitoring their effects and reporting on progress towards sustainable development. Robust national systems that increasingly incorporate administrative data and enable countries to regularly collect their own reliable and timely data is a long-term goal for governments and all development actors. Still, many countries lack national statistical and administrative data systems that can support effective and inclusive policymaking. Census data and key surveys managed by national statistical offices are infrequent or outdated, while administrative data systems such as routine registration of births and deaths and the production of vital statistics, are often incomplete. Two challenges must be addressed:

• Lack of funding from both domestic and external sources for data and statistics. Challenges to funding for development data include the public-good nature of data, uncertain fiscal returns, political economy constraints and complementarities between different types of data that imply low returns on investment.

• Ineffective support to national statistical and monitoring systems, due to:

  • Fragmented approaches, with more diverse stakeholders that support and need data and statistics and competing priorities leading to the development of parallel monitoring systems, instead of financing national systems and retaining the institutional capacity for data collection and use over time within national statistical systems.

  • Insufficient strategic focus and direction to support statistical systems, including specifying comparative advantage and sustainable outcomes, and adopting a mainstreaming approach that goes beyond specific support for sectoral data. Having shared guidance for international support to national statistical and monitoring capacity would help to strengthen co-ordination between development partners and alignment to country priorities.

Long-term commitments to invest in countries’ national statistical and monitoring systems are key to producing more relevant, granular, timely and harmonised data in the future. These investments must be done in an effective and co-ordinated way.

In using national systems, development partners strengthen country ownership, help to build co-ordinated approaches and mutually reinforcing partnerships, and foster transparency and mutual accountability.

5.2.2. Self-assessment questions

What efforts are you making to strengthen countries’ statistical and monitoring capacities? To what extent are your investments supporting countries’ long-term strategic priorities in the development of national statistical and monitoring capacity? Are your efforts co-ordinated with other development partners?

How do your monitoring approaches ensure that they maximise the use of countries’ monitoring and statistical systems? Whenever you collect your own data through parallel processes, is there a clear rationale for doing so, and is there precise guidance regarding the sustainability of data-collection efforts, interconnectedness with the national statistical system, and availability of data?

5.2.3. Actions to consider in order to strengthen country data systems

Governments

• Strengthen statistical and administrative data systems:

  • Invest in a sustainable, equitable and integrated national statistical system that reflects national policy priorities, enables the monitoring of the national development plan and the SDGs, and helps improve service delivery.

  • Follow a demand-driven, bottom-up approach in installing statistical capacity in local governments.

• Adopt a national strategy and budget for the development of statistics to secure domestic resources and mobilise and strengthen the alignment of development partners’ support.

• Improve data governance with a legal basis and governance framework in line with the UN Fundamental Principles of Official Statistics, providing a clear institutional mandate to the national statistical office to ensure quality assurance and co-ordination between data-producing agencies and new data actors in the extended data system.

• Organise regular co-ordination meetings with development stakeholders to co-ordinate support, harmonise statistical production and foster transparency and mutual accountability.

Development partners

• Ensure that support to national statistical and monitoring systems is co-ordinated among the different national stakeholders.

• Support more harmonised approaches to funding at the international level.

• Invest in the long term to support the capacity of national statistical systems, aligning data support with countries’ needs and priorities, including their digital transformation and ability to improve data quality and disaggregation.

• Ring-fence a share of project and programme resources to help close sector-related data gaps based on a common agreement on data priorities and needs, in line with the SDG indicators.

• Co-ordinate funding of data collection efforts to avoid duplication and increase value for money, collaborating to map existing data sources such as household surveys and administrative surveys, making duplications visible and taking action to prevent them in the future.

5.2.4. Actions to consider in order to improve the use of country data systems

Governments

• Deepen efforts to use data and results information to inform policy decisions and as a basis for programming, monitoring and accountability.

• Provide timely, open and easy access to data to users and decision-makers at all levels, within and beyond government.

• Invest in improved capacity of decision-makers to read, interpret and analyse data and results information.

• Strengthen trust in data by taking measures to protect data of individuals and social groups in line with existing international standards.

Development partners

• Use the country’s national statistical system to report on development interventions.

• When you cannot avoid collecting your own data through parallel processes, take steps to ensure that your data-collection efforts are sustainable and connected with the national statistical system; and make your data available while respecting safeguards in line with international standards and principles for open data.

• Support data use and capacity for data use throughout the national statistical system. Support data literacy among civil society organisations and media to promote evidence-based dialogue on development co-operation.

5.2.5. Good practices

• Many development partners (e.g. Finland, Sweden and the United Kingdom) support the strengthening of national monitoring systems to collect foundational data such as civil registration and vital statistics, censuses and surveys, which underpin the reporting of many SDGs. They also help to develop innovative methods and new data, such as satellite data, to ensure that statistical systems are modernised and sustainable. They increasingly emphasise the sharing and co-ordination of data and information amongst relevant entities.

• Paris21 offers platforms for dialogue between statistical actors in countries with development actors. Partnerships with the Global Partnership for Sustainable Development Data promote open data, global standards and the disaggregation of data to ensure that no one is left behind. The Bern Network promotes the Clearinghouse for Financing Development Data, which enables anyone to track and analyse the state of financing for data, connect to partners and access a broad range of resources to advocate for more and better data.

5.2.6. Resources

• OECD (2020), Key trends in development co-operation for national data and statistical systems

• GPEDC (2022), Bern Call to Action

• GPEDC (2022), Effective Development Co-operation Summit Declaration

• GPEDC (2020), Enhancing Effectiveness to Accelerate Sustainable Development: A Global Compendium of Good Practices.

• GPEDC (2021), Data for Development Dialogue: Good Practice at Country Level

• OECD (2021), Data for Development Profiles: Official Development Assistance for Data and Statistical Systems

5.3. Monitoring, evaluation and learning systems are locally led

5.3.1. Why is this important?

Engaging local actors has gained traction in development co-operation, with the development effectiveness agenda calling for inclusive ownership from the early 2000s, and later the people-driven 2030 Agenda. As conveyed in a donor statement endorsed at the 2022 Effective Development Co-operation Summit, locally led development is promoted as a way to enhance the effectiveness of development co-operation and its ability to support sustainable development impact, through enhancing inclusive ownership, increasing accountability and fostering legitimacy, with decisions and actions driven by local actors. Locally led development also aims to shift away from traditional donor-recipient relationships to promote partnerships built on mutual trust.

A programme is locally led when it recognises and enables local actors’ agency in “framing; design; delivery, including control over resources; and accountability, in given local and operating contexts” (OECD, 2023[4]). Local actors’ agency can be seen along a spectrum that ranges from being omitted, to consulted, co-responsible and primarily responsible. The degree of agency may vary between the four dimensions above, determining the extent to which a given programme is considered to be locally led.

Supporting locally led monitoring, evaluation and learning requires giving or ceding more agency to local actors in defining the lines of accountability and learning, e.g. by co-creating the development of results frameworks and indicators, co-designing the monitoring process, selecting the interventions to be evaluated, and leading on the production, sharing and use of evidence.

Longstanding calls for localised approaches to development are yet to have a significant impact on MEL systems and norms. As applications of locally led development for MEL systems are still at an early stage, guidance is all the more needed to help headquarters and programme managers support locally led MEL systems – thereby fighting the “functioning inertia” that occurs when development actors are unable to operationalise a vision.

5.3.2. Self-assessment questions

• Local actors

  • To what extent are you in charge of MEL activities in the projects and programmes that you implement in partnership with development co-operation providers? Does your degree of agency vary depending on the type of project or partners? What would be needed to increase your agency in MEL activities?

• Development partners

  • To what extent are you aligning your MEL activities with your local partners’ MEL systems and priorities, and what efforts are you making to strengthen local actors’ systems? What steps could be taken to increase local actors’ agency in MEL activities? What challenges or barriers have to be overcome in order to take those steps?

5.3.3. Actions to consider and pitfalls to avoid

To develop and strengthen locally led MEL approaches

Local actors

• Engage in a dialogue with your funding partners to further the alignment of their MEL systems and processes to local systems and priorities. Where appropriate, co-ordinate with other local partners to advocate for this alignment.

• Co-define with your development partners what should be measured as part of MEL activities. Set goals, targets and indicators relevant to your context and needs (e.g. what aspects of development projects or programmes are you interested in learning more about and for what reason?).

• Promote co-ordinated approaches and systems among your funding partners (e.g. unique reporting standards, local representation in MEL steering committees, etc.).

• Suggest using methodologies and tools for data collection and analysis that are suitable for your local organisation, tailoring them to the population’s capacity and cultural preferences. Make sure these methodologies increase local ownership, participation and representation.

• Determine what capacity, external expertise and information your organisation needs to lead MEL activities in the long term and request relevant training from your partners to achieve this.

• Define your learning goals: how and for what purposes will you use the results information collected through monitoring and evaluating the programmes and projects implemented in your area? Discuss these goals with your funding partners and engage in dialogue to agree which learning approaches will best support local needs.

Development partners

• At corporate level:

  • Include success targets for localising MEL activities in the localisation strategy.

  • Develop guidance to promote the alignment to local results frameworks and use of monitoring systems at programme and project levels to increase local agency in MEL.

  • Wherever needed, empower managers to adapt the M&E approach.

  • Facilitate training sessions to sensitise ministry or agency staff about the value of local capacity, knowledge and expertise, and accept its validity and relevance for HQ’s work on MEL.

• At programme and project level:

  • Involve local actors in the planning and design phases of MEL (e.g. hire local experts in senior MEL roles or involve local actors in M&E steering committees) and make sure that M&E processes respond to their learning goals.

    • Avoid: designing MEL frameworks that only respond to upward accountability needs and do not reflect local expertise, perspectives and definitions of success and impact.

  • Prioritise investing in local organisations’ data sets, local M&E capacity and local M&E practitioners to build the local evidence base rather than establishing parallel data-collection systems.

  • Build long-term partnerships based on mutual trust with different local MEL practitioners and organisations. Engage with these partners throughout the MEL planning and implementation stages and aim for a mutually beneficial relationship.

    • Avoid: resorting to exclusively using local partners in the data-collection stages of MEL (e.g. as investigators or respondents), without involving them in the design and analysis stages.

  • Set a context-based strategy to progressively increase the level of agency of local actors in MEL. This involves planning demand-driven capacity-strengthening activities for local actors.

For data collection, analysis and learning, local actors and development partners should:

• Integrate participatory methodologies during data collection and data analysis. M&E methodologies such as most significant change (MSC) and outcome harvesting involve local communities in a meaningful and non-extractive way and allow more space for subjective qualitative insights from local stakeholders.

• Co-create data collection tools to ensure that they are context-appropriate and accessible to all population groups (e.g. translated into local languages and centred around local voices). While technology and digital solutions may help reach some isolated population groups, keep in mind that not all actors have equal access to them.

• Analyse M&E data in accordance with the local context. Involving local MEL experts (e.g. researchers, data analysts, etc.) can ensure that the evidence generated is supported by the cultural and societal context and that findings are relevant to the community.

• Design learning agendas in conjunction with local actors and actively involve them in producing knowledge products that are tailored to and generate value for local audiences.

  • Avoid: using information collected during M&E activities for donor-reporting purposes only, as this hinders local learning efforts.

• Ensure that information collected and knowledge generated is shared with concerned communities. Using approaches such as peer-to-peer learning or knowledge-exchange platforms can strengthen local actors’ awareness of programme/project successes and failures and support their own development endeavours. (See Section 5.1 on using results information for learning).

5.3.4. Tips to go a step further

Embed local voices in MEL activities. Meaningful engagements with local communities require valuing local actors’ perspectives on development co-operation and acknowledging that locally generated knowledge might diverge from what has traditionally been considered as MEL expertise.

• Seek partnerships with diverse groups and engagement with marginalised communities, using existing guidance and good practice examples such as those related to religious actors and faith-based organisations with longstanding commitments to local populations, or indigenous peoples.

• Consider alternative measurement methodologies for capturing local “intangible” realities (e.g. actors’ sense of agency, motivation, feelings of trust vis-à-vis development partners, etc.) beyond numeric, quantifiable or short-term metrics seen as bringing objective data for reporting purposes.

Use self-assessment tools to evaluate your progress towards locally led MEL. There is a growing number of tools and metrics available for development partners and organisations to measure whether their MEL systems and co-operation interventions are aligned with locally led development principles. Some examples include:

• The decision-mapping tool (DMAT) developed by King’s College London, which makes it possible to explore the allocation of decision-making power across the programme cycle. It helps practitioners understand how decision-making power is distributed and managed in MEL systems and explore what steps can be taken to strengthen local communities’ agency in them.

• The community led development assessment tool created by the Movement for Community-led Development, which can be used by governments and NGOs alike to determine how their programming (including M&E activities) aligns with locally led development characteristics and undertake course corrections, as needed.

Jointly monitor and manage risks throughout programme implementation. At the project level, due diligence processes may affect the trust relationship between partners and lead to decisions (such as higher controls and cumbersome procedures) that hamper the agency of local stakeholders in MEL. Shifting from an ex-ante, heavy due diligence process to a more proportionate approach based on actual risks and emphasising dialogue among stakeholders can better support locally led approaches. There are four ways to do this:

• Tailor and simplify due diligence requirements, making them proportionate to the partner’s capacity, to the nature of the partnership and to the types of risks that are involved. Stop being exhaustive and focus instead on the risks that may materialise and could be damaging.

• Collaborate with other international partners to harmonise or reduce requirements put on local actors, e.g. using the due diligence process of another organisation for your own purposes or conducting joint assessments.

• Strengthen the capacity of local actors as a key risk-mitigation measure, rather than imposing additional controls. Such capacity-strengthening should be based on real dialogue. It can involve financial support or higher overheads for reform measures or additional staff.

• Engage with partners to jointly manage risks throughout the collaboration. A good practice is to consider the initial due diligence process as a starting point for a dialogue to mitigate risks as the project is implemented.

5.3.5. Good practices

• USAID has developed resources on locally led MEL, such as guidance tools and examples of locally led MEL in action, to support partners and communities in implementing these goals. They include insights and lessons learned from an internal evaluation of the Local Works programme in Serbia, where local partners provided recommendations to enhance locally led MEL based on their firsthand experience.

• Following a partner-led approach to development co-operation, Iceland includes District Councils in programme steering committees that play a direct role in the framing, design, approval and review of MEL activities (in addition to other decisions across the programme cycle). Iceland has also supported District Councils in recruiting local M&E officers, has provided them with financial assistance and training, and has hired local teams for evaluations in Togo and Uganda.

• Investing in digital solutions for locally led MEL systems, USAID has introduced an approach to collaborative monitoring through which grantees from their agricultural programmes in Ukraine input their own context-specific data into a centralised database, where information is aggregated and analysed. This monitoring approach allows for more accurate decision-making and programme adaptation based on real-time local information.

• Liberia, with the support of UNDP, has developed a MEL system framework for its National Adaptation Plan (2020-2030), which allows the tracking of progress on its prioritised climate adaptation actions and defines success metrics in line with its national objectives.

• Groups that are left behind often suffer from data marginalisation and risk being invisible in national statistics and administrative data. Citizen-generated data can help empower marginalised groups and provide insights on the situations of groups left behind. The Leave No One Behind partnership promotes the generation and use of community-driven data, aiming to augment the voice and agency of marginalised groups who are at risk of being overlooked in the implementation of the SDGs.

5.3.6. Resources

• Coger, T., Corry, S. and Gregorowski, R. (2021), Reshaping Monitoring, Evaluation and Learning for Locally Led Adaptation

• ITAD (2022), Facilitating change: Localising monitoring, evaluation and learning

• King, M. (2023), Why does local agency matter? Enabling the local decision space for aid recipients

• Movement for Community Led Development (2023), (Un)Learning Labs: Community-led Monitoring, Evaluation, and Learning

• ODI (2021), Are we there yet? Localisation as the journey towards locally led practice: Models, approaches and challenges

• Outreach International (n.d.), What is Community-Led Monitoring and Why is it Important?

• UNAIDS (2023), Community-led monitoring in action

• USAID (2023), Learning Lab with resources on Locally Led Monitoring, Evaluation and Learning

5.4. Both quantitative and qualitative information is collected

5.4.1. Why is this important?

Quantitative information is important to measure what is being achieved and assess progress towards objectives. Quantitative information is often favoured as it is collected routinely (through the monitoring system) and is easy to aggregate and communicate. Moreover, number-based results can be compared and are perceived as more objective than qualitative information.

Qualitative information is vital to complement quantitative information and document how change is happening. It is best suited for measuring progress that is complex, multi-faceted or multidimensional. Qualitative information increases understanding of the diversity of causal pathways for different constituents. It also helps to account for the institutional and human dynamics that affect outcomes, enabling the information on why interventions work in some places and not in others. Qualitative information in the form of stories and narratives is also crucial to communicate with people and engage with the public.

Qualitative information is often perceived to be impressionistic. It is more easily questioned for bias, often being criticised for ‘cherry picking’ cases. Gathering robust qualitative information around attitudes, behaviour, relationships, influence, systems, power dynamics, social norms and public goods is indeed challenging. Institutions need to go beyond stories of change and case studies towards more rigorous approaches, including qualitative research methods, to measure and show qualitative results.

Bringing numbers to qualitative results can facilitate their communication and use. Hybrid (or mixed) indicators quantify qualitative data, involving qualitative assessment factors (e.g. using checklists) for measuring them. They best capture data on multifaceted concepts like efficiency or participation. Hybrid indicators are measured by indexes, milestone scales or scorecards.

Qualitative indicators are also possible and can be preferable for measuring performance in certain areas, for instance in the democracy and governance sector. They take the form of a narrative assessment that measures change over time against specific, predetermined criteria.

5.4.2. Self-assessment question

• To what extent and how are you collecting qualitative information to complement quantitative data?

5.4.3. Actions to consider when collecting qualitative information

• Select and shape indicators using an iterative process, thinking about which indicator types and formats are most appropriate and considering resources and potential data sources.

  • Use an explicit rationale to select the type of indicators (quantitative, hybrid, qualitative) that will be used (domain of activity, level of complexity, multifaceted approaches).

• Apply a robust methodology: adopting a qualitative research approach in the generation of qualitative evidence will make qualitative information more credible. Data sources need to be selected with transparency and rigour, and case selection must be built in a systematic manner – this is imperative to make robust generalisations as well as to strengthen acceptance of qualitative and mixed-method evidence to evaluate attribution.

  • Note that rigorous case selection for qualitative information is not the same as for quantitative information. Since qualitative information helps us understand the diversity of causal pathways, random selection is unlikely to be very useful. Moreover, there is no scientifically optimal sample size. However, transparent and reasoned case-selection with specific, predetermined criteria is important.

• Bring clarity as regards the use of qualitative information and how such information can be linked to quantitative indicators.

• When considering digitalisation, pay attention to how to handle qualitative information, in order to keep a human-centred approach that is able to look towards the future.

5.4.4. Good practices and resources

• Many institutions combine quantitative monitoring with qualitative methods, e.g. using partner testimonials to complement quantitative data and ensure reporting at output and outcome level is evidence-based. They increasingly combine data with reliable stories about what was (or wasn’t) achieved, and lessons learned from the experience. The use of hybrid indicators to quantify qualitative information is also increasingly used with different categories or levels of scorings.

• Evaluation teams are also increasingly adopting iterative mixed-method approaches, for instance employing a range of qualitative and quantitative methods across a sample of countries to enable ongoing analysis, triangulation and validation of findings.

• The Qualitative Impact Protocol (QuIP) developed by Bath Social and Development Research (BSDR) has been designed to offer a credible and cost-effective assessment of the extent to which changes in wellbeing that have been self-reported by intended beneficiaries can be attributed to specified social and development interventions in complex contexts.

• Skovdal, Morten and Cornish, Flora (2015). Qualitative research for development: a guide for practitioners

5.5. Summary checklist for strengthening monitoring systems

• Does your monitoring plan specify the frequency, scope and intended use of the monitoring data?

• Is proper guidance on collection tools and methods widely shared within the organisation?

• Are staff trained to use the organisation’s digitalised information systems (where these exist) for reporting data at country or programme level?

• Do you have specific methods and procedures in place for remote monitoring in case a crisis limits access to key stakeholders?

• Is your investment in strengthening statistical and monitoring systems informed by a long-term vision, aligned with partner country needs and priorities, and co-ordinated with other stakeholders?

• To what extent are you investing in the MEL systems of local organisations and involving local actors in designing and implementing your MEL activities?

• How can you improve the agency of local actors in monitoring, evaluation and learning activities and systems?

• In which ways can you improve data governance, ensuring quality assurance while protecting the data of individuals and social groups?

• Whenever you collect your own data through parallel processes, is there a clear rationale for doing so, and is there precise guidance regarding the sustainability of data-collection efforts, interconnectedness with the national statistical system and availability of data?

• How can you improve the capacity of decision-makers to read, interpret and analyse data and results information?

• Are you equipped to collect robust qualitative information and able to specify how it can complement quantitative data?

References

[3] EU (2023), OPSYS, https://wikis.ec.europa.eu/display/ExactExternalWiki/FAQ+-+OPSYS (accessed on 29 April 2024).

[4] OECD (2023), Framing DAC member approaches to enabling locally led development, https://one.oecd.org/document/DCD(2023)47/en/pdf.

[1] OECD (2019), Managing for Sustainable Development Results: Guiding Principles, Best Practices in Development Co-operation, OECD Publishing, Paris, https://doi.org/10.1787/44a288bc-en.

[2] World Bank (2012), Designing a results frameworks for achieving results, https://web-archive.oecd.org/2013-10-21/253482-WB%202012%20designing%20results%20framework.pdf (accessed on 29 April 2024).