Comprehensive results-based Monitoring and Evaluation (M&E) systems are a key public management tool that can help policy makers track progress and demonstrate the impact generated by a specific intervention. They can assist in setting and co‑ordinating policy goals, identifying promising practices, detecting weaknesses, and designing corrective actions. M&E systems are a crucial tool to promote transparency and accountability of policy making.
Monitoring and evaluation are separate but complementary practices. Although there is not a unique way to design and implement a results-based M&E system, some basic and necessary steps have to be considered, including: outlining the intervention logic, setting indicators and data collection systems, planning for evaluation, and reporting and dissemination of findings. Such processes must be supported through well-functioning institutional mechanisms and sufficient capacities to conduct M&E activities.
Based on overall guidance and best practices in building results-based M&E systems, this report provides a general assessment of the M&E elements in Italy’s Universal Civil Service (UCS). It highlights the strengths of the system currently in place, identifies gaps, and suggests areas for improvement.
Overall, the report comes to the following main conclusions:
Regulatory and programmatic documents inform on key elements of UCS’ Theory of Change, but a complete development of the policy logic and results chain is lacking. The UCS has existed for a long time and its nature has evolved over time. Key information on the current intervention logic of the UCS is provided in Legislative Decree 40/2017 and other programmatic documents, as well as the annual reports to the parliament. However, an official and comprehensive description of the UCS’ Theory of Change has not been elaborated. Therefore, the UCS current framework does not make it explicit why and how the UCS actions and transformational goals are expected to contribute to the identified outcomes, and eventually generate the desired impact.
Important monitoring efforts are dedicated to tracking UCS implementation, but evidence on UCS outcomes and impacts is limited. The UCS monitoring covers a managerial function via descriptive indicators collected on a regular basis, allowing for comparisons over time. The key indicators used to monitor the UCS at the central level are included in Helios, the main database of the UCS, and presented in the annual report to the parliament. While the UCS regularly monitors implementation focusing on inputs, activities, and outputs, it does not systematically monitor UCS performance in terms of outcomes and impacts. Moreover, the UCS currently does not have a comprehensive description of its monitoring system and indicators.
The Department of Youth Policies and Universal Civil Service and entities conduct monitoring activities on their interventions, but monitoring is sometimes seen as a formality, and only some information is aggregated at a central level. At the UCS level, monitoring primarily comprises inspections, centralising entity data on programme/project implementation, and preparing annual reports to the parliament. Central monitoring also involves monitoring the training provided to volunteers, ensuring compliance with legal requirements and guidelines. Entities are responsible for the implementation and monitoring of their programme/projects and the UCS currently does not impose a single, comprehensive monitoring system with pre‑defined tools for data collection beyond basic data on implementation. As a result, tools such as questionnaire to volunteers – which play a crucial role to understand volunteers’ expectations and satisfaction, identify weaknesses to be improved and capture the UCS outcomes and impacts – vary across entities. This decentralisation has its merits in terms of giving autonomy to the entities given their specific characteristics and diverse missions, but risks dispersing valuable information and becoming a box-ticking exercise if entities are not sufficiently trained on the monitoring function and made aware of its importance. At the same time, the UCS is lacking specific tools to monitor changes that happen in the territories and communities thanks to its interventions. Such risks and limitations are amplified by the scarce human and financial resources dedicated to monitoring and evaluation at both the central and entity level.
There is growing interest in evaluation activities on the UCS, but evaluation evidence so far is scattered and not systematic. Evaluation evidence on the UCS is scattered. Formal evaluation activities on the UCS at the level of the Department of Youth Policies and Universal Civil Service are limited, with the positive exception of research by the National Institute for the Analysis of Public Policies on the previous edition of the intervention (the National Civil Service). The Department has recently started pilot actions through consultations and end-of-placement questionnaires to volunteers, taking significant first steps towards starting the collection of information on UCS outcomes and impacts. When it comes to individual programmes/projects, only a few entities (mainly from the third sector) have carried out ad hoc evaluations or assessments of their interventions. Since these evaluations are scattered and mainly limited to third sector entities, it is not possible to extrapolate their conclusions at the UCS level. Overall, a strong body of evaluation evidence on the UCS is currently lacking, also given the absence of a strategic plan for evaluation exercises.
Despite major communication efforts on the UCS overall, reporting and dissemination of M&E findings of the UCS is currently limited. Despite a legal framework that encourages data sharing and transparency, the dissemination of evidence gained through the monitoring and evaluation of UCS is mainly limited to the annual report to the parliament. While the UCS conducts rich communication and dissemination actions on the UCS overall, such actions do not embed information on M&E results in a systematic manner.