Monitoring and Evaluation of Child and Youth Policies and Outcomes in Ireland

4. Monitoring for results

Once the results framework has been agreed upon, the next step in building a results-based M&E system consists in obtaining the necessary data that will provide evidence on the performance of the programme or policy under scrutiny (Kusek and Rist, 2004[9]). Data collection, collation, and analysis is the main role of the monitoring function. Monitoring is the continuing process that involves the systematic data collection or collation on specified indicators or other types of information. Monitoring provides managers and other stakeholders of an intervention with indications as to the extent of implementation progress, achievement of intended results, occurrence of unintended results, use of allocated funds and other important intervention- and context-related information (OECD, 2023[12]).

There are two main types of monitoring: implementation monitoring and results monitoring, and both are essential in tracking progress across the results framework (Kusek and Rist, 2004[9]). Implementation monitoring tracks the means in place (i.e. inputs, activities, and outputs) to achieve a given outcome, while results monitoring focuses on the outcomes themselves (see Figure 4). The relevance of monitoring outcomes is self-explanatory, but implementation monitoring is also of utter importance, since it provides information on how well the programme or policy of interest is producing the expected outputs using available inputs and activities.

Figure 4. Types of monitoring across the results framework

Source: Authors.

4.1. Data collection for monitoring

Thus, as it occurred with the definition of indicators, data must be collected for each level of the results chain, and not only for the outcome/impact level. An appropriate planning of the monitoring function therefore should describe the methodology required to collect and process the relevant data on the complete set of indicators identified in the results framework (FAO, 2023[22]). The plan should also indicate the tools required to collect such information and the frequency of deployment of those tools. The ultimate objective of this planning exercise is to systematically organise the data collection and management processes around the indicators defined in the results framework.

When planning for data collection, it should be considered what data have already been produced either internally or from other sources and are readily available (secondary data), and what further data collection processes need to be developed (primary data). One of the main advantages of primary data is that it can be designed to exactly fulfil the information needs of the particular intervention that needs to be monitored and evaluated. However, it requires more resources for the design of data collection templates, data collection and storage, and subsequent data analysis (Markiewicz and Patrick, 2016[23]). Secondary data are less resource intensive, but constraints can be experienced in relation to data access and overall data quality, reliability, and relevance.

4.1.1. Building on BOBF

First, collating and analysing data for the more than 100 indicators in the BOBF indicator set would be a challenging task. Hence, a careful selection out of those indicators would have to be made for the M&E framework, to keep the economic cost of monitoring realistic.

Second, all BOBF indicators were constructed based on secondary data sources. For each indicator in the set, the indicator set report (DCEDIY, 2022[6]) provided detailed information including the indicator definition, the data source, the frequency of data collection, and the indicator’s relevance to BOBF. As indicated in the report, the data underlying the indicator set come from a range of data providers and a variety of data sources. Criteria for selecting the data sources to construct the indicators included the requirement for frequently collected, transparent, and internationally comparable data, if possible. Interviews held with DCEDIY confirmed that data availability was a key decision variable when assessing potential data sources.

The indicator set report acknowledged some of the challenges associated with this use of secondary sources for data collection (DCEDIY, 2022[6]). First, and given the wide array of data sources, data for each indicator were collected at different intervals. For example, some indicators could be constructed annually, while data for others was exclusively collected every five years. This variety means that it is of vital importance to be aware that when monitoring and analysing changes on the indicators of interest over time, data reflect changes across different periods, depending on the frequency of collection of the underlying data. Secondly, the report noted that the BOBF indicator set relied on the data quality verification processes of each data provider (DCEDIY, 2022[6]). No additional quality assurance system was set in place for the construction of indicators beyond the one undertaken by the data provider.

4.2. Ensuring human and financial resources to undertake the monitoring function

In addition to the methodology, data sources, and frequency of data collection, an appropriate planning of the monitoring function should also indicate the roles and responsibilities with respect to the processes of data gathering and management. Building and maintaining a monitoring system requires providing sufficient financial, human, and technical resources for the task (Kusek and Rist, 2004[9]).

In terms of human resources, a monitoring system that can produce trustworthy, timely, and relevant information on the performance of government policies requires skills, experience, and real institutional capacity. This capacity has to include, at a minimum: i) the ability to successfully construct indicators; ii) the means to collect and process data in relation to the indicators defined in the results framework; and iii) the understanding of what to do with the information once it has been processed. Statistical capacity and knowledgeable staff on information technology and Management Information Systems (MIS) are also a must. Building such technical capacity within government structures is a long-term effort. On some occasions, external technical assistance and training may be necessary to initially support these tasks, but such support should always be oriented on the national government developing its own capacity to design, implement and make use of their own monitoring systems. In order to build in-house capability and access external expertise when necessary, a business case could be developed for the funding of M&E activities.

4.2.1. Building on BOBF

A three‑member Implementation Team was established as part of the Policy Innovation Unit to lead on the implementation of BOBF. With an annual implementation budget of EUR 70 000, the team was responsible for reporting on the implementation of BOBF and supporting the governance of BOBF, among other tasks. Despite the human and financial resources, monitoring the progress made by BOBF was considered challenging, mainly due to the ambiguous nature of many of the commitments (see Section 3.2) and the additional administrative tasks assigned to the team.

In addition, as indicated in the indicator set report (DCEDIY, 2022[6]), the development and implementation of the BOBF indicator set was undertaken by the Research and Evaluation Unit at DCEDIY, in collaboration with an external provider. Interviews held with DCEDIY for the elaboration of this report confirmed that this collaboration was seen as an added value given its track record in working on the design, development, implementation, and evaluation of public policies and services.

Since the adoption of BOBF, additional human resources have been set in place. DCEDIY now has a dedicated Data and Analytics Unit (DAU), headed by a Senior Statistician from the Irish Government Statistical Service (IGSS). DAU currently has seven full-time staff members, including the Head of Unit, two additional statisticians/analysts, a policy analyst from the Irish Government Economic and Evaluation Service (IGEES), a generalist administrative officer, and two executive officers. This Unit is additional to the Research and Evaluation Unit, which predates BOBF and has a range of research, data and analytical expertise.

Moreover, relevant advances have been made in terms of capacity building on the area of M&E since the adoption of BOBF. Since 2019, a series of Guidance Notes were developed by the Research and Evaluation Unit as a resource to support the Department’s staff and other policy units on the uptake and use of evidence to drive policy change (Government of Ireland, 2023[24]). Although the development of these Guidance Notes was not specifically linked to BOBF but rather aimed to provide general support on key stages of the research to policy process, some of them provide useful information on key M&E concepts. For example, Guidance Note 3 covers the concept of evaluation, its role within the programme life cycle, and an introduction to evaluation design and types of evaluations (DCEDIY, 2019[25]). Similarly, Guidance Note 7 builds on the above and explores the role and importance of programme logic models on the planning and evaluation of projects and policies (DCEDIY, 2021[26]). In addition to these Guidance Notes, the Research and Evaluation Unit also developed a two‑day training course for civil service staff on “Understanding Evaluation in Human-Related Government Services”. The training content was developed in collaboration with the Centre for Effective Services under the Goal Programme for Public Service Reform and Innovation, an initiative aimed to support systemic change in public services in Ireland and Northern Ireland with the goal of improving outcomes for people using public services (Centre for Effective Services, 2019[27]).