Guidance for a Monitoring and Evaluation System for Italy’s Universal Civil Service

2. Understanding and outlining UCS results chain through a ToC

A first step to build an M&E system is to set out the logic of the intervention by describing its results chain: how inputs and activities will produce outputs, which will lead to specific outcomes and, eventually, impacts (Figure 2). Each element of the results chain contributes to the next, and the links between elements are as important as the results themselves (OECD, 2023[18]; World Bank, 2012[19]). Identifying these elements and their linkages is fundamental, as they will be the main focus of M&E activities.

A common method to structure a results chain is through the “Theory of Change” (ToC) approach.1 The ToC is an analytical framework aimed to model the process of change, describing and explaining how an intervention is expected to bring about the desired outcomes and impact in a particular context (Belcher, Claus and Davel, 2020[20]). A documented ToC makes explicit what a policy does (inputs, activities, and outputs) and what it aims to achieve (outcomes and impacts). It allows to test and review the hypothetical causal linkages and underlying assumptions between activities, outputs, and outcomes during implementation. This approach is particularly important in complex contexts where the expected change can be influenced by multiple factors. The ToC provides the basis to defend that a particular policy is making a difference, while capturing unintended and unexpected results.

Figure 2. The results chain

Source: OECD (2002[13]), Glossary of Key Terms in Evaluation and Results Based Management (in English, French and Spanish); OECD (2023[18]), Impact by Design: Effective Results Frameworks for Sustainable Development.

Defining the ToC for any intervention requires a process of consultation, exchange, and consensus-building among key stakeholders. Being typically iterative and involving various actors, the process can be time‑ and resource‑intensive.

The UCS key elements are set in regulatory documents, but an explicit and comprehensive ToC has not been elaborated

The Italian UCS has existed for a long time and has evolved over time. Key information on the current intervention logic of the UCS is provided in Legislative Decree (LD) 40/2017 (Gazzetta Ufficiale della Repubblica Italiana, 2017[6]) and other programmatic documents,2 as well as the annual reports to the Parliament. However, an official and comprehensive description of the UCS ToC has not been elaborated.3

Defining a clear ToC is fundamental to ensure consistency in the understanding and implementation of the UCS across all key stakeholders (including the DGSCU, the entities and young people themselves), especially in view of UCS evolution over time as well as the different perspectives and interests at stake (for instance, the impact of the UCS on territories and young people). A clear ToC would also orientate the design, evaluation and selection of programmes, projects, and activities, to which the UCS devotes considerable attention (OECD, forthcoming[21]). Finally, a ToC would provide a solid basis for UCS updates and adjustments. For instance, the existence of a ToC for the UCS could have eased the adaptation process during the COVID‑19 pandemic, by providing a solid basis for discussions on the adjustment of specific elements of the UCS.

Although it is best to outline the ToC in the early stages of an intervention, there is still merit in undertaking such a process now for the UCS – especially since it is undergoing a reform process (DGSCU, 2021[22]).

Suggestion for an outline of UCS ToC based on UCS key elements

Desk research and stakeholder consultations undertaken for this report allowed for the identification of elements of the UCS results chain and make a first attempt to outline its potential ToC (Figure 3). This initial outline is intended as a resource to stimulate discussion and further work on the definition of an official ToC for the UCS: it will need to be adapted based on internal feedback, further stakeholder consultations and UCS developments.

In brief, this initial outline of the UCS ToC captures the following results chain: the set-up of volunteering projects, training activities and mentoring (activities) will allow for the delivery of volunteering placements, programmes and projects, and training and development (outputs). Such outputs are expected to strengthen young people’s human capital and improve the provision of services of public utility (outcomes). Overall, this results chain would contribute to strengthen young people’s active citizenship, employability, and employment, as well as social cohesion and resilience, ultimately supporting the “unarmed and nonviolent defence of the homeland” (Gazzetta Ufficiale della Repubblica Italiana, 2017[6]) (impacts). Various assumptions would sustain the feasibility of this model: a continued government commitment to the UCS, the availability of quality placements, sustained volunteer motivation, supportive entities, and stable legal and regulatory environment are key enabling factors.

Defining the UCS ToC would support a better understanding of the building blocks and significant aspects of the intervention – including, among others, the contribution of UCS actions to the ultimate goal of unarmed and nonviolent defence of the homeland; UCS impact on young people and territories; and the focus on various groups of young people, such as those with disabilities, migrant background, or not in employment, education or training (NEETs). For example, interviews conducted as part of this project emphasised that the UCS can enhance various aspects of young people’s lives – active citizenship, employability, as well as psychological and physical well-being, among others. Being a multifaceted matter, it is essential for UCS authorities to reach consensus on the priority areas (which are likely interconnected and not necessarily mutually exclusive) that the UCS should focus on, monitor, and evaluate. Clarifying these priorities would allow the identification of the strategies that would need to be pursued to achieve the desired change. For instance, enrolling as many young people as possible in the UCS could be expected to maximise the intervention’s value; yet, targeting specific groups could be a relevant alternative approach to support those young people who would benefit most from the interventions.

Once the UCS ToC has been defined and agreed, M&E efforts can be structured around its key elements, allowing for systematic tracking and assessment of progress. In a cascade process, the ToC helps to: (i) identify the appropriate indicators, data collection methods, baselines for comparison, and evaluation approaches; (ii) set realistic expectations; and (iii) identify potential risks and challenges. It also supports adaptive management, as any deviations or unexpected outcomes can be analysed and addressed promptly. Ultimately, it should enable UCS managers and evaluators to understand the logic of the UCS, assess its effectiveness, and make evidence‑based decisions for UCS improvement.

Recommendation 1: Ensure a clear and shared understanding of UCS ToC, to be used as a reference for implementing UCS programmes/projects and as a basis for M&E activities

Outlining a ToC for the UCS can ensure a clear and shared understanding of the nature, objectives, and logic of the intervention. This ToC can serve as a key reference for programming, implementation, monitoring and evaluation activities. Therefore, it is recommended that UCS authorities ensure a clear and shared understanding of the UCS ToC, to be used as a reference for implementing UCS programmes/projects and as a basis for M&E activities. In this process, UCS authorities are advised to:

• Ensure that the development process of the ToC for the UCS is inclusive and participatory.

• Clearly identify specific elements under each component of the ToC and the subsequent links between these elements.

• Enumerate the assumptions behind the model.

• Accompany the visual representation of the ToC with a narrative description of each of its items.

• Regularly review and update the ToC as the UCS evolves and more evidence becomes available.

Internal discussions at the DGSCU level will need to be complemented with consultation of key stakeholders, such as representatives of the entities that are implementing UCS programmes, young people, and researchers. This participatory and iterative process (e.g. in the form of consultations, workshops and focus groups) will be fundamental for consensus building and buy-in. This process could benefit from moderation by an expert in M&E or consultation processes. The draft ToC outline in Figure 3 could be used as a basis for the refinement of the UCS ToC.

Figure 3. Visual representation of the UCS Theory of Change: An initial outline

Note: This diagram is an example of a ToC outline for the UCS as prepared by the authors, to be potentially used as a basis for discussion by UCS authorities for the definition of the UCS ToC.

Source: Authors, based on consultations with UCS managers and desk research on UCS documents, including LD 40/2017 (Gazzetta Ufficiale della Repubblica Italiana, 2017[6]), NRRP (Italian Government, 2021[11]) and DGSCU (2022[23]).

Notes

← 1.  See “A short history of results frameworks” in OECD (2023[18]) for an overview of different approaches.

← 2.  For instance, the 2023‑25 three‑year plan for the programming of Universal Civil Service (DGSCU, 2022[23]) and its preceding versions, the calls for programmes/projects, and UCS description in the National Recovery and Resilience Plan – NRRP (Italian Government, 2021[11]).

← 3.  Existing research attempted to reconstruct the logic of the UCS prior to LD 40/2017 and to identify the links between its building blocks. See the National Institute for the Analysis of Public Policies (INAPP)’s surveys and analyses of the National Civil Service conducted in relation to “Occasional Surveys – Civil Service” (INAPP[55]).