Guidance for a Monitoring and Evaluation System for Italy’s Universal Civil Service

5. Establishing an evaluation plan

Complementarily to monitoring, the evaluation function seeks a clear understanding of the changes induced by the intervention and aims to form judgements on the intervention’s performance. Evaluation evidence can: i) support decision making on competing alternatives, offering insights into which types of interventions prove more successful in attaining desired outcomes; ii) improve decisions on resource allocation by identifying the most effective and efficient types of intervention; iii) foster agreement on the causes of and solutions to a problem by illustrating causality; iv) detect emerging problems, including issues that may have not been covered in the intervention design and may require attention of policy makers; and v) support public sector reform, informing citizens on the effectiveness of reform efforts (Kusek and Rist, 2004[24]).

In the case of the UCS, the evaluation function will be crucial to complement the monitoring efforts on UCS outcomes and impacts. Evaluation could also be useful to understand if the processes and mechanisms of the UCS are functioning as expected. Depending on their information needs, UCS authorities would need to define evaluation questions, to be answered through appropriate evaluation methods and designs. Evaluations should be proportionate and appropriate for their expected use, and the aim, scope and analysis of the evaluation, as well as its format and resources, should be adapted to the needs of its intended uses (OECD, 2022[3]). Examples of evaluation questions for the UCS could encompass:

• What is the impact of the intervention on UCS volunteers and the territories?

• Does the intervention reach and affect different people in different ways? How effective is it for specific groups of young people?

• How are UCS initiatives contributing to specific transversal issues (e.g. gender equality)?

• Are there any unexpected effects generated by the intervention?

• What types of interventions are best contributing to the desired change? Could the outcomes be achieved through different approaches?

• How effective are the training modules?

• What are the key regulatory barriers to UCS implementation?

Evaluation efforts need to be planned early (to ensure that evaluation results are robust and available in a timely fashion) and strategically (to meet multiple evaluation needs) (OECD, 2022[3]). An evaluation plan is the strategic and management document used to summarise the approach to evaluation and to guide the evaluation efforts along the cycle of an intervention, taking into account the purpose, evaluation questions, evaluation type and timeline (for more information on evaluation types and timelines (see for instance UNDP (2009[40])). The plan can be updated by authorities as new evaluation needs emerge.

Evaluations can be designed and conducted internally or externally, but externally commissioned evaluations can ensure higher objectivity and independence. Various evaluation types suit different questions and purposes; ensuring clarity when commissioning such exercises to external evaluators will be a prerequisite for the usefulness and relevance of evaluation exercises.

Existing evaluation exercises on the UCS are scattered

Evaluation evidence on the UCS is scattered but increasing. Formal evaluation activities on the UCS at the DGSCU level are limited, but additional evaluation evidence has been produced by INAPP and at a programme/project level. The list below includes a non-exhaustive overview of analyses and exercises that create a first basis of evidence on the effectiveness of the UCS.

Formal evaluation activities on the UCS at the DGSCU level are limited but increasing

LD 40/2017, in its Article 21, mentions that the evaluation of results of UCS programmes in the territories and the local communities concerned is the subject of a specific annual report prepared by the Prime Minister’s Office, with possible support from third-party organisations with proven expertise in the field, and published on the official website (Gazzetta Ufficiale della Repubblica Italiana, 2017[6]). Nonetheless, there is no evidence of formal evaluation activities of the UCS having been conducted so far. Evaluation evidence on UCS contribution to the desired change is limited and an evaluation plan for the UCS is currently not in place. Therefore, the main report at a central level remains the relation to the Parliament (DGSCU, 2021[22]), whose function mainly relates to monitoring.

It is nonetheless important to note that the DGSCU has conducted some ad hoc exercises to collect information on the outcomes of the UCS on the young volunteers:

• In 2021, the Minister for Youth Policies requested the public consultation “Growing Youth, Growing Italy” (Far Crescere i Giovani, Far Crescere l’Italia). Although not an evaluation exercise, the survey allowed to obtain information from UCS volunteers who had been/were in placement, with the aim to improve the UCS. Conducted online in 2021, the survey involved 14 500 young people, asking about their knowledge of UCS objectives and the usefulness of UCS for youth empowerment (DGSCU, 2021[38]). The consultation revealed, among others, that:

  • More than 65% of respondents were overall satisfied with their UCS experience.

  • The main reasons to participate in the UCS was the desire to challenge themselves, enriching their CV and starting a new experience.

  • More than 60% of respondents were interested in pursuing a job in the same sector of activity as their UCS placement.

  • 40% believed that the period of service was useful for their personal growth.

  • 80% of young people saw the UCS as a way to approach the world of work and considered it essential to receive a certificate of competence at the end of the experience.

• In addition, the DGSCU has been administering new end-of-placement questionnaires (see “Setting SMART indicators based on UCS ToC” and “Planning for the collection and processing of data”) on volunteer’s satisfaction, collecting relevant data on UCS outcomes and impacts.

The INAPP project on M&E of the UCS provides a rich basis of analysis on the previous version of the Civil Service (National Civil Service)

The most systematic analysis and evaluation activities on the UCS are carried out by National Institute for the Analysis of Public Policies (INAPP), which has been aiming to strengthen the knowledge base related to the monitoring, evaluation and profiling of the volunteers involved in the National/Universal Civil Service.1 With funds from the European Social Fund in the context of the Youth Guarantee initiative of the European Commission, INAPP promoted a methodological experimentation with two innovative multidimensional indices on active citizenship (built around the dimensions of democratic values, community life, and protest and social change) and employability (built around the dimensions of education, work experience, activation, and mobility). The data for the indices were collected through a nationwide sample survey to 3 500 National Civil Service volunteers (run in 2016), which were linked to administrative data allowing to follow their employment status over time,2 and complemented with information collected through additional consultations (interviews/focus groups). This research contributed to providing, among others:

• Information on the profiles of participants and their background, divided between “standard profile” and “NEET profile” and the identification of their family background (De Luca, 2022[29]).

• Measurements and analyses of outcomes and impacts (De Luca, 2023[15]; De Luca and Ferri, 2021[16]; De Luca, 2023[17]):

  • Identification of volunteers’ motivations to join the UCS, the level of satisfaction related to the experience, its perceived usefulness, its re‑orientation effect (i.e. making new education/professional choices as a consequence of participating in the UCS), and skills and competences acquired through the experience (based on volunteers’ self-assessments).

  • Evidence of a positive correlation between active citizenship for employability, and of UCS effects on employability (mobility and activation). For instance, research finds that the level of employability of the UCS volunteers after the placement increases on average by 12%, while the share of “inactive” youth is drastically reduced. Moreover, interviews to a sample of volunteers show that, 12‑18 months after the end of the experience, 52% of those interviewed who had participated to the SCU under ordinary calls were in employment.

• Policy messages and recommendations, such as for instance (De Luca, 2023[15]; 2022[41]):

  • The need to consider the gender and geographic dimensions of the UCS: women seem to choose SCU as a post-graduate experience, while men mainly do so to reactivate into the labour market; and UCS volunteers from southern regions have more disadvantaged family backgrounds and lower educational levels compared to those from central-northern Italy.

  • The need to pay attention at the candidate selection phase, to avoid benefiting well-performing profiles to the detriment of more disadvantaged young people, while ensuring project quality for the territories.

  • The potential to capitalise the UCS experience while designing other youth policies, given the relevance of UCS dimensions of “Participation” and “Active Citizenship” for youth employability.

  • The importance to move beyond employability as an abstract concept and start measuring it as a policy outcome.

  • The need to understand the UCS contribution to employability and other outcomes in comparison with other public interventions supporting young people.

The research provided an important evidence base for policy making, including for the 2023‑25 three‑year plan for the programming of the UCS (DGSCU, 2022[23]). Evidence of INAPP research was also shared with the Consulta and in dissemination material on the UCS3 and received wide media coverage overall. Nonetheless, there is no information on how systematically INAPP results are considered in UCS programming, as also highlighted by the limited dissemination of INAPP reports on the UCS website.

It is important to note that the data collected by INAPP relate to previous versions of the Civil Service – i.e. prior to the latest provisions included in LD 40/2017. Looking ahead, this experience could be capitalised for the analysis of the current version of the UCS, including specific aspects that followed the 2017 reform, such as the multi-year planning or the impact of the COVID‑19 pandemic on the UCS.

Evaluations at a sub-UCS level (programme/project, entity or territorial level) are rare

Desk research shows that only a few entities have carried out ad hoc evaluations or assessments of their interventions. Key examples show a positive impact of UCS actions on volunteers’ employability and skills:

• An ad hoc analysis of UCS projects conducted under Centro Servizi Volontariato Torino (Vol.To) in 2019 (Brescia, Caria and Mapelli, 2020[42]) investigates the social impact achieved by the UCS, based on volunteer data collected through questionnaires, storytelling, reclassification of economic and financial data and the determination of Social Return on Investment. The analysis identifies a positive social impact on volunteers, including enhanced employability and skills development. A following study expands the evidence through a social impact assessment of Vol.To projects of the 2020 call (Calandra et al., 2023[43]), showing that despite no significant change in perceived self-efficacy, SCU enhanced human capital and employability; led to increased employment; fostered skill acquisition, career orientation, and networking, contributing to reduced precarious work; and aligned with the Agenda 2030 goals.

• An AnciLab survey of 3 276 volunteers and former UCS volunteers looks into youth policies that young people consider most important and explores various topics about UCS volunteers, including gender differences, the impact on education on employability and the representation of diverse groups of young people among volunteers. The research confirms the role of the Civil Service as a tool for acquiring skills and supporting youth employability (AnciLab, 2022[44]).

• A survey conducted by Associazione Mosaico (2022[45]) among UCS volunteers from the 2022 call in the Lombardy region showed that 73% of those who completed the UCS received a job offer within 5 months, and 88.5% of them accepted the offer. 52% of these young individuals considered the UCS experience as either significant or highly important in triggering the job offer.

• An impact assessment on former volunteers hosted by social co‑operatives belonging to Confcooperative – conducted by CEVAS and Confcooperative (Leone and De Bernardo, 2017[46]) – shows that more than half of former volunteers continued to work in the non-profit sector after the National Civil Service experience, and therefore supports the idea that the UCS can be a powerful point of attraction and entry into the labour market. The analysis also highlights the role of entities and their networks for the employment opportunities of volunteers.

Most of the evaluations available at this level belong to entities from the third sector – given that, as required by the Third Sector Code, entities must produce a report on their Social Balance Sheet. Since these evaluations are scattered and mainly limited to third sector entities, extrapolating their conclusions at the UCS level is not possible. Expanding the body of project evaluations to all project and entity types would benefit learning and improvement at the UCS level.

At the same time, reasons for the lack of evaluation exercises by other entities may include their costly nature and the lack of evaluation competences among key stakeholders. In this respect, the evaluation culture and practice at the entity level should be better supported and encouraged (see “Strengthening human, financial, and institutional capacities, mechanisms and policy learning”).

Recommendation 4: Define a forward-looking evaluation plan, allowing for evaluations throughout the programme cycle

In order to manage evaluations effectively and maximise their usefulness, it is recommended that UCS authorities develop a feasible evaluation plan considering theoretical, practical, organisational, and budgetary aspects (European Commission, 2021[25]). UCS authorities are advised to specify the following elements in the plan:

• Subject and rationale: Evaluation’s purpose, background, and type, including key evaluation questions. Specific questions can be detailed further at a later stage, for instance in the evaluation’s Terms of Reference. Among others, of relevance for the UCS would be the analysis of the UCS impact on UCS volunteers’ personal and professional growth, active citizenship, and employment after the placement, as well as the evaluation of different UCS actions on the territories and sectors.

• Methods and data requirements: Depending on evaluation type and questions, different methods will be employed to exploit and complement the monitoring evidence. Given the heterogeneity of the UCS programmes/projects, evaluation exercises will most probably require a mix of qualitative and quantitative information. Counting with a robust monitoring system will provide valuable data for evaluation purposes and optimise the efforts for the collection of additional data.

• Data availability: Arrangements to ensure necessary datasets are accessible for specific evaluations (e.g. when using counterfactual methods). Systematic and timely data collection is key, given the risk of higher costs and lower accuracy of data collected retrospectively. Ensuring that UCS data are collected regularly at the level of the candidates/volunteers is a prerequisite for successful evaluation.

• Roles and responsibilities: Clear definition of roles and responsibilities of the actors involved in the evaluation, including evaluation team members, stakeholders, and participants. This decision also involves the choice between internal and external evaluation. These aspects are relevant in consideration of LD 40/2017, Article 21, stating that the evaluation of UCS results in the territories and local communities could be supported by third-party organisations with proven expertise in the field (Gazzetta Ufficiale della Repubblica Italiana, 2017[6]).

• Timeline and milestones: Schedule that includes key milestones and deadlines for each phase of the evaluation, from data collection to reporting. The milestones and deadlines should be related to the evaluation subject and methods.

• Estimated budget: This element is tied to the methods chosen and the duration of the evaluation service.

In addition, the DGSCU could also consider supporting evaluation practice at a programme/project level, for instance by providing guidance and training on evaluation to entities; raising awareness on the importance of such exercises; and making considerations on the potential allocation of funds to support the evaluation practice (see also section “Strengthening human, financial, and institutional capacities, mechanisms and policy learning”).

Notes

← 1.  See ongoing INAPP project “Monitoraggio e valutazione del Servizio Civile Universale fra cittadinanza attiva e Occupabilità (PTA INAPP 2022‑24)” (INAPP, n.d.[52]) and the surveys and analyses of the National Civil Service conducted in relation to “Occasional Surveys – Civil Service” (INAPP, n.d.[55]).

← 2.  The survey data were linked to the database of the Italian Ministry of Labour and Social Affairs on Compulsory Communications (Comunicazioni Obbligatorie), which contains data on dependent employment. See 2023 report by the Italian Ministry of Labour and Social Affairs (2023[56]).

← 3.  See for instance DGSCU (2022[53]).