Monitoring and Evaluation of Child and Youth Policies and Outcomes in Ireland

3. Development of a results framework

A results framework is an explicit description (in the form of a graphical display, matrix, and/or summary) of the different levels of results expected from a particular policy or programme (World Bank, 2012[11]). Its main objective is to describe how and why a particular intervention will achieve its objective(s), including causal relationships and underlying assumptions and risks (OECD, 2023[12]). Results frameworks help policy makers to monitor and assess how well a policy or programme is performing against planned targets, and therefore facilitate an evidence‑based approach to M&E. Results frameworks can be developed not only for discrete projects or programmes, but also for overall institutional frameworks and even country strategies.1

While there is no “one size fits all” approach, some key steps can contribute to the development of effective and useful results frameworks. Each of these steps is described in the paragraphs below.

3.1. Identification of policy outcomes

The identification and definition of outcomes is an essential first step in building a results framework. Building the system is a deductive process in which the rest of the elements (inputs, activities, and outputs) are all derived from the set of desired outcomes to be achieved.

Despite the fact that setting goals is a fundamental part of the policy making process, not all governments undergo the process of transforming those relatively broad and long-term goals into agreeable and well-defined outcomes to be monitored and evaluated. At this point, it is fundamental to distinguish between policy goals and outcomes. According to the OECD definition, a policy’s goal or impact is a “higher-order objective to which a development intervention is intended to contribute”, while an outcome is the “likely or achieved short-term and medium-term effect of an intervention’s outputs” (OECD, 2002[10]). Goals are therefore long-term, while outcomes tend to be medium- or short-term and the relatively direct consequence of the policy’s outputs.

As an example, the ultimate objective of a youth employment programme is to improve the socio‑economic development of young people in vulnerable circumstances by improving their job placement. Such improvement could be considered the programme’s goal or impact, while the necessary outcomes to achieve that goal could be: i) improved employability of young people in vulnerable circumstances, and ii) increased employment of young people in vulnerable circumstances (Figure 1).

When specifying the set of policy outcomes to monitor and evaluate, the following characteristics should be considered:

• Outcomes should be formulated through a participatory and consultative process. Setting policy goals and outcomes in isolation is likely to lead to a lack of ownership and buy-in from key stakeholders. This participatory approach, ideally, should start with the development of policy goals, continue through the identification of outcomes, and extend to the definition of indicators throughout the whole results chain (see Section 3.3). The involvement of both internal and external stakeholders should be actively encouraged, since it will help increase commitment to achieving the agreed outcomes.

• Outcomes statements should be formulated positively rather than negatively. National concerns and problems to be addressed should be reformulated into a set of desirable outcomes. Stakeholders are more likely to reach consensus and to respond better to positive statements (Kusek and Rist, 2004[9]; Levin, Schneider and Gaeth, 1998[13]; Tversky and Kahneman, 1981[14]). For example, “increased employment opportunities for young people in vulnerable circumstances” is a better choice than “reduced unemployment rates among young people in vulnerable circumstances”.

• Outcomes should be disaggregated sufficiently so as to cover a single outcome in each outcome statement. This step will become critical when outcome statements need to be converted into specific indicators (see Section 3.3). For example, outcome statements of the sort “improved educational attainment and employment opportunities for young people in vulnerable circumstances” should be avoided. These types of statements are actually the combination of two separate outcomes, and therefore should be enumerated independently.

Figure 1. Illustrative identification of a policy’s goal or impact and derived outcomes

Source: Authors.

3.1.1. Building on BOBF

According to the policy framework, BOBF adopted an outcomes approach based on five national “outcomes” for children and young people. Each of those outcomes was further disaggregated into what the document refers to as “aims” (Box 1).

BOBF did not only describe the overall policy framework goals but went further into reformulating them into specific long-term outcomes. The difference with the theoretical framework described above, however, comes from the fact that BOBF seemed to use a different terminology. What BOBF referred to as national “outcomes” seemed to be the long-term national goals or desired impact, while the distilled “aims” that were derived from those goals could be considered the outcomes of the policy framework.

BOBF followed a consultative approach in order to identify the goals and outcomes of interest of the policy framework. BOBF built on the learning gained through its predecessor the National Children’s Strategy, Our Children – Their Lives (2000‑10), and responded to the substantial public consultation that took place in 2012 to inform the government’s direction for the following seven years (Department of Children and Youth Affairs, 2014[1]). The public consultation included 66 705 responses from children and young people, and over 1 000 submissions from stakeholders and the general public. Inputs were received from a wide range of government departments, the National Children’s Advisory Council, the National Youth Work Advisory Committee, statutory bodies, and Non-Governmental Organisations (NGOs).

A participatory approach was also followed, as recommended, for the definition of indicators contained in BOBF’s indicator set (DCEDIY, 2022[6]). The development of the indicator set was led by the Research and Evaluation Unit of the Department of Children, Equality, Disability, Integration and Youth (DCEDIY) in collaboration with the Centre for Effective Services (CES), and the process followed a highly collaborative Delphi technique.2 Following an extensive desk-based review of national and international literature on indicators for children and young people’s outcomes and supported by the guidance of an Expert Advisory Panel, a selection of indicator areas was identified for inclusion in the Delphi surveys (DCEDIY, 2019[5]). Two rounds of Delphi surveys were conducted with specialists across a wide range of fields, occupations, and expertise. The Delphi approach also drew on a summary of the outcomes resulting from consultations previously held with children and young people to identify areas of importance to them. Finally, a range of bilateral meetings were held with government departments, agencies, and experts to consult on the indicator areas that had been initially prioritised and to help identify measures to represent them.

Box 1. BOBF’s “outcomes” and related “aims”

Outcomes “Children and young people are or have…”	Active and healthy, with positive physical and mental well-being	Achieving their full potential in all areas of learning and development	Safe and protected from harm	Economic security and opportunity	Connected, respected, and contributing to their world
Aims “Children and young people are or have…”	1.1. Physically healthy and make positive health choices	2.1. Learning and developing from birth	3.1. Secure, stable, caring home environment	4.1. Protected from poverty and social exclusion	5.1. Sense of own identity, free from discrimination
	1.2. Good mental health	2.2. Social and emotional well-being	3.2. Safe from abuse, neglect and exploitation	4.2. Living in child/youth-friendly sustainable communities	5.2. Part of positive networks of friends, family and community
	1.3. Positive and respectful approach to relationships and sexual health	2.3. Engaged in learning	3.3. Protected from bullying and discrimination	4.3. Opportunities for ongoing education and training	5.3. Civically engaged, socially and environmentally conscious
	1.4. Enjoying play, recreation, sport, arts, culture and nature	2.4. Achieving in education	3.4. Safe from crime and anti-social behaviour	4.4. Pathways to economic participation and independent living	5.4. Aware of rights, responsible and respectful of the law

Source: Department of Children and Youth Affairs (2014[1]), Better Outcomes Brighter Futures: The National Policy Framework for Children.

The importance and added value of these consultative processes in the identification and definition of policy outcomes and indicators should not be underestimated. In fact, this participatory approach is in line with the European Child Guarantee Council Recommendation (Art. 11.e), which encourages the participation of all relevant stakeholders, including children themselves, throughout the preparation, implementation, and monitoring and evaluation of child-related policies (Council of the European Union, 2021[15]).

3.2. Articulation of the policy logic and results chain

Once the desired policy outcomes have been agreed and identified, the next necessary step to build a results framework is to explicitly develop the policy logic, that is, the different levels of results that need to take place for the policy or intervention to eventually lead to the impact of interest.

More specifically, the results chain is the causal sequence of a policy or programme that stipulates the different stages that need to take place for the achievement of the specified objectives. The results chain describes how inputs and activities will produce outputs, which then will lead to specific outcomes and, eventually, policy impact (Figure 2). Each element of this chain contributes to the next, and the links between each element are as important as the results themselves.

Figure 2. The results chain

Source: OECD (2002[16]; 2023[12]).

The most common method to structure a results chain is the Theory of Change (ToC) approach. The ToC is an analytical framework aimed to model the change process, that is, a description and explanation of how and why a particular intervention is expected to contribute to the desired change in a particular context (Belcher, Claus and Davel, 2020[17]). A documented ToC makes it explicit what a policy does (inputs, activities and outputs) and what it aims to achieve (outcomes and impacts).

The ToC approach first emerged in the United States in the 1990s, in the context of improving evaluation theory and practice (Stein and Valters, 2012[18]). This analytical framework derives from the more traditional logic planning models, such as the logical framework (logframe), and has since then increasingly become mainstream. Logframes have been often criticised for being too rigid and linear, and unable to capture the complexity and context in which a policy is deployed, as well as the effect of external factors on the outcomes of interest. As a result, they are believed to be better suited to describe the logic behind small or medium sized projects or programmes.

As discussed in Stein and Valters (2012[18]), the transition from logical models to ToC is therefore the result of an increased desire to explore change in a way that takes into account the complexities and uncertainties of the context in which a policy or programme is implemented. A ToC is considered to provide a more comprehensive description and illustration of how and why a desired change is expected to happen in a particular context, since it includes a description of assumptions and risks that could have an effect in the results chain (Box 2). The ToC approach allows testing and reviewing the hypothetical causal linkages and underlying assumptions between activities, outputs, and outcomes during implementation. This approach is especially appropriate in the case of complex contexts where the expected change can be influenced by many factors, and not only the policy under study. In those cases, the ToC is able to provide the basis to highlight whether and how a particular policy is making a difference, while also capturing unintended and unexpected results of a policy.

Box 2. Identification of assumptions and risks in a Theory of Change

A rigorous Theory of Change (ToC) identifies and makes explicit the key assumptions underpinning the theory of how change is expected to occur in a given context, as well as the major risks that may affect it (UNHCR, 2018[19]; UNDP, 2017[20]; OECD, 2023[12]).

Assumptions are the causal links, conditions and events that need to exist for a particular intervention to work and generate the desired change (Mayne, 2023[21]). They are the enabling conditions presumed to be true or in place at the beginning of the intervention (OECD, 2023[12]). Assumptions can be established for every level of the results chain, and they derive from relevant academic research, prior implementation experience, and best practices.

Assumptions refer to the causal relationship that exists between one result and another. They help provide answers to the questions: i) what leads to what, and how?, and ii) which conditions and/or events are needed for a causal link to work? For example, in the case of the youth employment programme described in Section 3.1, it is assumed that if young people receive personalised career guidance, this will improve their employability, as long as staff at the Public Employment Services are appropriately trained on career guidance and resourced appropriately (assumption #1 in Figure 3).

Of particular importance are contextual or external assumptions, which refer to the influence that factors outside the sphere of control of the intervention can have in facilitating the expected change. Continuing with the youth employment programme example, assumption #4 highlights that for company recruitment subsidies and traineeship programmes to result in increased employment opportunities for young people, it is necessary that the economic context also favours the generation of employment.

Like with assumptions, the identification of risks is a crucial and challenging element in the design of a ToC. Exploring assumptions before risks can help with the identification of the latter, since assumptions and risks are often inversely related (UNDP, 2017[20]). In the case of the youth employment programme, for example, if the economic context became one of a recession, it would be likely that neither company recruitment subsidies nor traineeship programmes would result in increased employment opportunities for young people.

It is important to highlight that during the design of ToC, it is not always possible to anticipate the full range of risks that might derail the results chain. However, it is essential to devote resources to identify the most essential ones so that if they materialise, policy makers can adapt the ToC accordingly. Identifying relevant risks at the start of an intervention also contributes to the design of strategies that can help mitigate those risks.

As with the rest of the elements of the ToC, the identified assumptions and risks should be tested (verified) by evidence and theory arguments at the evaluation stage (Mayne, 2023[21]). The verification process will contribute to understanding whether and how the intervention is working, assess causality, learning, and revising the links between the different results levels, if necessary.

For the purpose of exemplification, Figure 3 showcases a simplified version of the youth employment programme’s ToC mentioned in Section 3.1. As shown in the Box, the ToC describes the complete results chain that the programme is expected to trigger to generate a change in the outcomes of interest, leading to the high-level impact of improved socio‑economic development of young people in vulnerable circumstances.

Figure 3. Example of a simplified version of a ToC

Source: Authors.

3.2.1. Building on BOBF

In the case of BOBF, an explicit and comprehensive articulation of the logic of change that underpinned the policy framework was not developed. Instead, the framework identified six “transformational goals”, which were defined as key areas with the potential to strengthen the support system around the child and young person and therefore have a positive effect on the set of five national “outcomes”. For the realisation of those goals, the policy framework also identified a broad set of 69 government “commitments” (Department of Children and Youth Affairs, 2014[1]). In addition to the commitments attached to the six “transformational goals”, BOBF also enlisted 89 commitments that directly referred to the five national outcomes.

As it occurred with the definition of the policy framework outcomes (see Section 3.1), the terminology used by BOBF here was slightly different. Although named “transformational goals”, the six products described under this rubric were, in essence, the sorts of outputs that should be generated by the government action with the ultimate objective of generating a change in the outcomes of interest. These outputs, were, namely: i) support to parenting; ii) earlier intervention and prevention services; iii) involvement of children and young people in the matters that affect them; iv) quality services; v) strengthened transitions; and vi) cross-government and interagency collaboration and co‑ordination. Similarly, the government “commitments” were in fact the set of necessary government actions or activities that should be put in place to generate the aforementioned outputs.

Building on BOBF, an M&E framework for Ireland’s child and youth strategy should make it explicit why and how the government commitments and transformational goals, if realised, are expected to contribute to the identified policy outcomes, and eventually generate the desired impact. The framework should document any theoretical or contextual assumption that could help explain why a particular change could be expected from a certain government action under specific circumstances. It should also make explicit references to potential external risks that could derail the results chain. The absence of a complete results chain and policy logic for BOBF made it difficult to evaluate the impact of any government action under the BOBF policy framework.

3.3. Design of an indicator set

After having elaborated the logic of change underpinning a programme or policy, indicator development is the next core activity in building a results framework. The importance of indicators derives from the fact that they are the drivers of all subsequent data collection, analysis, reporting, evaluation, and evidence dissemination (Kusek and Rist, 2004[9]).

Indicators are thus essential elements of a results framework since they enable organisations to monitor and report accurately on how a programme or policy is performing (OECD, 2023[12]). An indicator can be a quantitative or qualitative variable related to an intervention and its results, or to the context in which such intervention takes place. Indicators provide simple, verifiable, and reliable means for tracking changes and the performance of projects, programmes, and policies.

Indicators should be developed and agreed on by key stakeholders for each level of the results chain, since they are needed to monitor progress with respect to inputs, activities, outputs, outcomes, and goals (Kusek and Rist, 2004[9]). Tracking indicators on a regular basis throughout the whole results chain enables decision makers to understand the performance of policies and programmes, and it alerts them to any performance discrepancies, shortfalls in reaching targets, and any other deviations from the desired outcomes.

Selecting and defining indicators can be a difficult task. However, Kusek and Rist (2004[9]) provide the following methodological considerations that could help guide this process:

• It is recommended that a minimal number of indicators is selected. A recurrent problem with results frameworks is that they are often based on a too large set of indicators. In those cases, the marginal added value of each additional indicator in terms of information may not be significant, while it can create a considerable burden for the processes of data collection, analysis, and reporting. Every indicator has cost and work implications, and thus they should be chosen carefully and judiciously.

• For indicators to be useful and effective for M&E purposes, indicators should follow the SMART criteria described in Box 3.

• Qualitative indicators should be used with caution. Although indicators may be quantitative or qualitative, it is desirable to start with simple and quantitatively measurable variables rather than inserting qualitative indicators upfront. Qualitative information is important and necessary, but it is more time consuming to collect, measure, and verify, since it often involves subjective judgements and perceptions.

• Proxy indicators should be assigned carefully. Indirect (or proxy) indicators should be used only when data for direct indicators are not available or regular data collection is not feasible. Nonetheless, proxy indicators should be used with caution since they can pose a threat to the validity of results measurement. They can correctly track the desired result, but they can also be affected by other contributing factors, which would make attribution to the policy or programme challenging to assert.

Box 3. Characteristics of SMART indicators

• Specific. Indicators should reflect simple information that is communicable and easily understood. They should be as clear, direct, and unambiguous as possible.

• Measurable. Indicators should be objectively verifiable.

• Achievable. Indicators and their measurement units must be achievable and sensitive to change during the lifespan of the programme or policy.

• Relevant. Indicators should reflect information that is important for the policy or programme object of analysis, and likely to be used by managers and decisionmakers.

• Time bound. Indicator progress can be tracked at a desired frequency for a set period of time.

Source: World Bank (2012[11]), Designing a Results Framework for Achieving Results: a How-To Guide, https://documents1.worldbank.org/curated/en/331541563854787772/pdf/Designing-a-Results-Framework-for-Achieving-Results-A-How-to-Guide.pdf.

3.3.1. Building on BOBF

The BOBF indicator set includes over 100 indicators across 70 indicator areas, with one or more indicators per area. The indicators spanned the five BOBF “outcomes” and the 20 associated “aims” (four aims associated to each outcome), as well as different age groups between 0‑24 years. While interesting and useful to track progress for children and young people – which is indeed the main objective of the BOBF indicator set –, the economic cost of collating, analysing, and reporting data on more than 100 indicators is considerable and should be carefully considered when designing an M&E framework. Although there is not a specific or correct number of indicators for a results framework to be effective, the following questions are useful to ask when considering the addition of an indicator: i) is this indicator absolutely necessary to measure whether progress toward the desired goal is being achieved?; ii) will it create additional burdens in terms of data collection, analysis, and/or reporting?; and iii) how will this indicator help with the policy or programme M&E?.

BOBF’s indicators are almost exclusively outcomes focused. Additional indicators would thus have to be identified (and possibly developed) for the results chain of an M&E framework, which includes not only outcomes, but also inputs, activities, outputs, and assumptions. In line with its purpose which was to track how children were faring across the five national outcomes, relatively few of the BOBF indicator areas focused on the outputs that are generated by government action. 15 out of 70 indicator areas included indicators on access to, take‑up, or quality of services (OECD, 2024[4]). For example, out of the 16 indicator areas under Outcome 1 “Active and Healthy”, two covered output-level indicators: immunisation uptake (two indicators3) and access to mental health services (one indicator). The other 31 indicators were outcome level indicators. To properly monitor and evaluate whether government activities generate the desired results or outputs (i.e. parenting support, quality services, strengthened transitions, etc.), additional output-level indicators would need to be developed or articulated (and related data collected, if necessary).

Similarly, the indicator set does not define any indicators at the activity-level. Instead, the policy framework enumerated a list of considerably broad government commitments that were expected to trigger a change in the outcomes of interest. 69 commitments were attached to the framework’s “transformational goals”, and the other 89 referred to the five national “outcomes” in a direct manner. The framework also identified the government departments and/or agencies that were responsible for taking action with respect to such commitments. The set of 158 commitments had however a rather ambiguous nature, and no specific and verifiable indicators were created to enable the monitoring of these commitments. Instead, the annual report for the final year of implementation of BOBF shows that the monitoring of commitments was based on a self-evaluation undertaken by the government departments and agencies that were responsible for delivering on the commitments (DCEDIY, 2022[8]). For each of the commitments, those entities were asked to value their progress on a qualitative scale that goes from “done” to “ongoing”, “in progress”, and “not being progressed”. Ideally, this qualitative, self-reported system could be complemented with additional quantitative indicators that would contribute to the external verification of the delivery of governmental activities.

With respect to the more than 100 outcome‑level indicators, it is important to highlight that a very large number of BOBF’s indicator areas were based on self-reported personal experiences, behaviours, beliefs, and attainments (OECD, 2024[4]). The use of such proxy indicators are second-best solutions to fill data gaps and should only be used when no better data is available, pending the development of better data.

The BOBF’s indicator set report acknowledged the potential risks of self-reported indicators and included strategies to mitigate them. During the design of the indicator set, it was considered essential that the indicator areas prioritised by the Delphi process (see Section 3.1) would be retained in the final indicator set even if limited or no data were available for such area (DCEDIY, 2022[6]). For this purpose, where no appropriate data were identified for a particular indicator area, a range of markers known as “placeholders” were included in the indicator set. For instance, the indicator “number of children who receive direct support and/or accommodation from a domestic violence service each year” is a proxy for “number of households where domestic violence occurs”. Similarly, the indicator for “abuse” – total number of children at the end of the reporting period listed as “at ongoing risk of significant harm” on the Child Protection Notification System – is a proxy for the real extent of abuse. The inclusion of these placeholders ensured that areas that were identified as relevant by the Delphi process were not omitted from the set based uniquely on data availability reasons, and most importantly, it encouraged the future development of data on those topic areas considered essential.

Finally, and given the wide array of datasets used to construct the BOBF indicators set, the used indicators presented a high degree of variation in terms of their scope for disaggregation. Overall, the set of indicators does not have a consistent disaggregation in terms of key variables such as geographical location, age‑groupings, and vulnerability status of children and young people. The absence of certain disaggregation variables made it impossible to monitor outcome changes among particularly relevant sub-populations of children and young people. For example, age disaggregation would be of special relevance for poverty indicators, since no matter what statistical measure of poverty is adopted, adolescents aged 12 to 17 years old are much more frequently exposed to poverty than younger children (OECD, 2024[4]).

Notes

← 1.  For illustration purposes, see UNICEF’s results framework for their strategy 2022‑25 (UNICEF, 2021[33]), as well as related Theory of Change which the framework is built on (UNICEF, 2021[34]).

← 2.  The Delphi technique is a scientific method to organise and structure an expert discussion around a particular research topic (Beiderbeck et al., 2021[32]). The technique relies on inviting a series of experts to assess and comment on different questions related to the specific topic, with the objective of identifying an emerging consensus.

← 3.  One indicator for meningitis C vaccine and another one for the first dose of measles, mumps, and rubella vaccine.