Guidance for a Monitoring and Evaluation System for Italy’s Universal Civil Service

3. Setting SMART indicators based on UCS ToC

Indicator development is the next core activity that follows from the definition of the ToC. Indicators guide data collection, analysis, reporting and dissemination of evidence (Kusek and Rist, 2004[24]), hereby enabling organisations to accurately monitor and report on how an intervention is performing (OECD, 2023[18]).

Indicators (quantitative and/or qualitative) should be developed for each level of the ToC, as they will inform on progress related to inputs, activities, outputs, outcomes, and impacts (Kusek and Rist, 2004[24]). The set of indicators should provide a comprehensive picture of the intervention’s progress and effectiveness. Indicators should be well-defined, directly linked to the intervention, and measured periodically, with specified measurement units. Regularly tracking indicators enables decision makers to understand the performance of the intervention, identify any deviations from the expected performance, and adjust if needed. If indicators are not on track, it prompts reflection on implementation strategies, the suitability and effectiveness of interventions, and even the appropriateness of selected indicators and targets themselves (European Commission, 2021[25]; 2021[26]).

When developing the set of indicators, it is important to consider the following methodological guidelines:

• Define “SMART” indicators (World Bank, 2012[19]), meaning indicators that are:

  • Specific: Clear, direct, as unambiguous as possible; reflecting simple information that is communicable and easily understood.

  • Measurable: Objectively verifiable.

  • Achievable: Achievable and sensitive to change during the lifespan of the intervention.

  • Relevant: Reflecting information that is relevant to the intervention, and likely to be used by managers and decision makers.

  • Time bound: Trackable at a desired frequency for a set period.

• Select an appropriate number of indicators, considering the costs of data collection and processing, and their added value in terms of informative capacity. There must be a balance between the need to monitor all aspects of the intervention and the potential drawbacks of having too many indicators.

• Consider the feasibility of data collection. If data collection is too burdensome, it might lead to inconsistent or incomplete information. Monitoring should start with straightforward, quantifiable variables. Complementary qualitative information is key, but its acquisition and measurement are more complex and time‑intensive due to the involvement of subjective judgments and perceptions.

• Ensure traceability and comparability over time and potentially with other interventions, facilitating consistency and learning. When available, baseline data serve as a reference for comparison, facilitating a more accurate assessment of the effectiveness of the intervention. Indicators that reflect both short- and long-term impacts contribute to ensure a long-term perspective in M&E.

• Employ proxy (indirect) indicators thoughtfully, only when direct indicators lack data or regular collection is impractical. While helpful in tracking the desired outcome, proxy indicators might be influenced by other factors, making it challenging to attribute changes solely to the intervention.

UCS monitoring mainly tracks inputs, activities and outputs

Monitoring UCS interventions is mandatory (DGSCU, 2021[27]). At the UCS level, once the programmes/projects have been selected, monitoring mainly consists of ordinary and extraordinary inspections and the centralisation of basic entity data on programme/project implementation in databases. The key indicators used to monitor the UCS at the central level are included in Helios, the main database of the UCS, and presented in the annual report to the Parliament (Table 1). The DGSCU also monitors the general progress of the training provided to volunteers, to ensure that it complies with the requirements of the legislation, the guidelines, and the content of programmes/projects.

The current UCS monitoring mostly covers a managerial function to track progress in implementation via descriptive indicators, which are collected on a regular basis, allowing for comparisons over time. The UCS regularly monitors inputs, activities, and outputs (see Table 1, which lists the main indicators from the 2021 report to the Parliament and matches them with the ToC elements presented in Figure 3). A similar level of detail can be found in comparable interventions in other OECD countries, such as the Canada Service Corps1 and the French Service Civique.2 At the same time, UCS outcomes and impacts are not systematically monitored.

Table 1. Matching key UCS indicators from the report to the Parliament with the UCS ToC outline

Main indicators in the 2021 report to the Parliament	Correspondence with UCS ToC as per Figure 3
	Description	Inputs	Activities	Outputs	Outcome	Impacts
Annual financial allocation to the UCS	Inputs: Financial resources
UCS interventions: Projects and calls for the selection of volunteers Number of available/required positions for volunteers, Italy/abroad, by sector Number of interventions (programmes and projects) and distribution by: Region and macro-region (North, Centre, and South and Islands) Distribution of projects by sector and scope of action Focus on vulnerable young people (NEETs and unemployed young people as part of the Youth Guarantee) Status of the intervention (programme: not admissible, excluded, not activatable, activatable; project: rejected, withdrawn, not evaluable, not activatable, activatable) Presence of qualifying elements (co-programming, networks, opportunities for young minors, mentoring, service period in an EU country) Objectives aligned with the Goals of Agenda 2030 (1, 2, 3, 4, 5, 10, 11, 12, 13, 15, 16) Projects with reserved positions for young individuals with fewer opportunities (low education and economic difficulties)	Focus sectors Activities: Set-up of volunteering projects Outputs: Volunteering placements Transversal principles
Entities Number of owner entities (Enti titolari, listed in the Register, Albo), by type (third sector/public) Number of host entities (Enti di accoglienza), by type (third sector/private – public) Number of implementing sites (sedi) Number of entities conducting additional measures (as per LD 40/2017): Mentoring of volunteers, to facilitate their labour market entry Employment of young people with fewer opportunities within UCS programmes Management of volunteers, including the provision of sustenance and accommodation for the entire duration of stay in EU countries	Focus sectors Activities: Training activities Mentoring Outputs: Volunteering placements Training and development
Volunteers Number of young people aged 15‑29 years receiving support, by gender (non-binary, female, male) – indicator No. 14 of the set of common indicators related to the objectives of the Recovery and Resilience Facility (RRF)* Number of hours of volunteering activities Number of received applications by geographical area, submitted by Italian and foreign citizens (EU/Non-EU) Number of volunteers initiated relative to number of received applications Number of volunteers initiated into the UCS, Italy/abroad, by call and demographic and socio‑economic characteristics: gender, age, origin (Italy by region/abroad), education level UCS volunteers in Italy: number of volunteers by region, sector, and by disadvantage status (disabilities, low education, economic disadvantages) Number of projects and volunteers abroad by sector, geographic area (Africa, Europe, South America, Asia), gender, age, education level Coverage level of advertised positions, by territory Number of dropouts (withdrawals, interruptions), by region, causes (optional: inability to balance study/work and UCS; family reasons; finding a job) Number of UCS closures (termination of relationship volunteers/entity), by reason Number of replacements Number of disciplinary proceedings against volunteers	Focus sectors Activities: Set-up of volunteering projects Outputs: Volunteering placements
Training Number of volunteers who received training Hours of general and specific training provided	Activities: Training activities Outputs: Training and development
Activities supporting the deployment of the UCS** Activity by Region and Autonomous Province, including: Evaluation of intervention programmes Training Promotion and information Human and financial resources committed for the UCS, by year Control and inspection activities Specific activities related to the COVID‑19 emergency and additional activities Activity within the DGSCU, i.e. Human resources by category (managers/staff) and contractual type Financial resources, by source Initial allocations, final forecasts, committed sums, and payments Inspection activity aimed at verifying compliance with regulatory provisions related to proper project management and correct use of volunteers: Number of conducted inspections (planned/disposed after irregularity reports) Communication Regulatory activities Litigation concerning UCS Reduction of disputes Register and programme/project appraisal: Number of UCS entities registered in the Register of UCS entities Average duration (days) required to complete registration in the Register Average duration (days) for appraisal of interventions proposed by entities Composition of the National Consultation for UCS (Consulta)	Inputs: Financial resources Human resources Activities: Training activities Outputs: Volunteering placements

Notes: The list is not exhaustive and includes the indicators that are deemed most relevant for the purpose of this exercise.

* See https://ec.europa.eu/economy_finance/recovery-and-resilience-scoreboard/common_indicators.html?lang=en and the methodological fiche at www.italiadomani.gov.it/content/dam/sogei-ng/indicatori-comuni/Scheda_RRFCI14.pdf.

** Some indicators reflect administrative procedures as per programmatic documents such as Presidenza del Consiglio dei Ministri (2017[28]).

Source: Authors based on DGSCU (2021[22]), NRRP (Italian Government, 2021[11]), Presidenza del Consiglio dei Ministri (2017[28]), interviews with UCS stakeholders and Figure 3.

UCS indicators could be improved in terms of coverage and disaggregation

As shown above, the UCS monitors its inputs, activities, and outputs regularly, providing a good overview of the progress in implementation. Indicators could nonetheless be improved as regards the i) coverage of young people, ii) disaggregation of the information, and iii) availability of qualitative information:

• Coverage: the information collected on young people mainly relates to volunteers admitted to the UCS. Collecting information not only on volunteers who completed the experience, but also on those who were selected but did not take up the volunteering opportunity and volunteers who withdrew from the UCS before finalising the experience could provide useful information on the UCS relevance and effectiveness. Similarly, collecting complementary information on applicants who were not selected into the UCS can enhance the understanding of UCS outreach and provide a foundation for further evaluation research.

• Disaggregation of the information: the information collected is disaggregated by demographic profile of young volunteers, year and call, region, and entity type. Whenever possible, it is advisable to ensure disaggregation according to the following factors:

  • Demographic and socio‑economic characteristics of young people: age, gender, level of education, activity/occupation status at the time of the application (in employment, in education, NEET, other), origin (Italy by region/abroad), disadvantage status (disabilities, low education, economic disadvantages). The employment and educational situation may also reflect the economic sector of employment, the type of contract and field of study. Disaggregation could also encompass the family background, e.g. the education level and occupational status of both parents and/or volunteers’ assessment of the economic situation of their family (De Luca, 2022[29]). Overall, this information allows to profile young people and to analyse the inclusivity and reach of the UCS by:

    • Clustering the volunteers according to key characteristics (for instance: students, experienced workers, NEETs, disadvantaged groups, etc.).

    • Identifying the diversity of young people in UCS interventions compared to the diversity of the overall young population in Italy.

    • Analysing the gender dimension of the UCS.

    • Identifying the “missing volunteers”, i.e. any groups that should be better targeted.

  • Year and call. This information allows to consider the specificities of each call for applications by entities and young volunteers and to provide a time perspective to the analysis.

  • Stage in the UCS process: application, interview, before the placement, during the placement, at the end of the placement and 6‑12 months after the placement. This information allows to identify issues that may arise at different times along the UCS cycle and the effects of the UCS after the placement.

  • Region and municipality. This information allows to reflect the geographic spread of UCS interventions and, indirectly, inform on its relevance to the territory. The level of disaggregation in UCS monitoring already allows to track the “Southern quota”, whereby 40% of NRRP investments must be allocated to the southern regions of Italy.3 As the UCS intends to have an impact on the territories, it compels entities to make structural investments, such as increasing the number of implementing sites throughout the territory and in specific regions. Collecting information by municipality can provide a nuanced overview of the distribution of the interventions, for instance allowing to identify urban/rural differences.

  • Sector and sub-sectors. This information provides an overview of the thematic focus of UCS interventions.

  • Entity type: private, public, third sector. This information provides an overview of the typologies of entities involved, supporting reflections on whether the involvement of more/diverse entities is needed.

• Insights on the qualitative dimensions of the UCS. It is possible to collect qualitative information both through open or closed questions in questionnaires to volunteers and Local Project Operators (Operatori Locali di Progetto, OLPs), but closed lists of options would facilitate response aggregation. For instance, when inquiring regarding young people’s motivation to apply to the UCS, pre‑defined options could encompass: personal growth; need to give; need to receive; need to change/do something new/find a new meaning in life; have a better idea of the future; acquire new skills; try an alternative holiday; live an different experience; ensure an income; find employment; be prepared for the labour market; put the educational competences into practice; engage in social solidarity/active citizenship; defend one’s homeland; experience nonviolence; religious or spiritual reasons; others. A similar approach could be used for other dimensions of the UCS, for instance the content of the specific training provided to volunteers. A list of key thematic areas could be identified in entities’ reports on specific training provision (DGSCU, 2023[30]).

Additional indicators would be needed to monitor UCS outcomes and impacts

Table 1 highlights that additional indicators are needed to monitor and report on how the UCS contributes to achieving the desired change (outcomes and impacts). Although such information is not systematically collected by the UCS at the time of writing, existing regulatory documents provide a foundation for future improvements of UCS M&E (Box 1).

Selecting and feeding SMART indicators on outcome and impact indicators is often challenging. First, selecting relevant indicators is challenging because of the intangible nature of some outcomes, the possibility of unintended consequences, the interconnectedness of various components of the ToC, and difficulties related to causal attribution. Second, collecting related data is challenging due to the reliance on self-reported information, the need to involve UCS applicants and volunteers in various stages of the M&E process, and the difficulty to collect specific data on sectors and territories.

Using the ToC and each of its elements as a reference to define the UCS M&E system can help overcome some of these challenges. An example of such application is illustrated in Table 2. Following the outcomes and impacts presented in Figure 3, Table 2 provides a menu of additional indicators to consider (based on examples from entities’ reports and other examples in the literature). This list can be used as a basis for discussion with the DGSCU to enhance current M&E efforts. The list is not prescriptive nor exhaustive and should be refined based on indicator relevance, feasibility of data collection and considerations on available technical and financial resources, IT systems and data protection.

Once indicators have been selected, the table should be completed with information on “data collection frequency”, a “baseline”, a “target” (a pre‑agreed value for an indicator to be achieved at the end of the period under consideration), a “milestone” (an intermediate value to be achieved at a given point in time during the period under consideration), a “date to achieve targets” and “reporting channels”. A reflection would be needed of which indicators could be monitored regularly, and which other indicators would need to be collected through additional evaluation activities.

Indicators such as those presented in Table 2 are intended to apply to all UCS programmes/projects. Nonetheless, such set of indicators would not be able to summarise additional information that entities may collect individually on their UCS actions. A strategy to best use such additional information would be to adopt a double set of indicators, with specific indicators complementing UCS common indicators:

• A core set of common indicators for all interventions, along the lines of Table 2, would allow for consistent data collection across entities and their programmes/projects, facilitating cross-comparisons, and providing an overview of the overall progress and performance of the UCS.

• Complementary sets of specific indicators by sector of intervention and territory would allow for a more detailed overview of the activities, beneficiaries of programmes/projects, and results in the different sectors, territories, and communities.

Box 1. Regulatory documents and pilot initiatives support improvements in future M&E activities of the UCS

The regulatory framework in Italy acknowledges the importance of M&E and the need to make progress in this domain. Regulatory documents focus on the needs to collect information on various UCS results, including participants’ satisfaction upon completion of their placement; the social, territorial and community impact of the UCS; and UCS ability to reach specific groups of young people. In parallel, DGCSU pilot actions are showing promise for more advanced monitoring in future editions of the UCS:

The impact analysis of LD 40/2017 suggests assessing the satisfaction of UCS participants through online questionnaires to be completed at the end of their placement. In also highlights the importance of tracking the percentage of UCS volunteers who secure employment within 12 months of completing the service (Presidenza del Consiglio dei Ministri, 2017[28]). As a pilot action, end-of-placement questionnaires have been administered to volunteers from the first call of applications of the NRRP (see section “DGSCU is implementing promising pilot initiatives focusing on results”).

• The three‑year UCS plan mentions that the DGSCU will adopt, as a pilot initiative, common indicators to assess the outcomes of UCS interventions. This pilot aims to assess the achievement of UCS objectives and response to social challenges, while considering their different territorial dimensions (DGSCU, 2022[23]).

• The three‑year plan also mentions that, in alignment with the DGSCU obligations related to the implementation of cross-cutting principles of the NRRP and the Disability Directive, the DGSCU may work to enhance the Civil Service Information System’s capacity to collect disaggregated data throughout the lifecycle of programmes/projects. This effort aims to capture details such as gender participation and measures supporting disadvantaged young people, particularly those with disabilities or other vulnerable conditions (DGSCU, 2022[23]).

• Improvements in evaluation planning and practice are expected as the available NRRP funds should allow for a “reallocation of (…) national resources towards strengthening activities such as programme monitoring and the evaluation of results in the territories and communities” which would “ensure the acquisition of best practices for the quality of future programmes” (DGSCU, 2021[22]).

• DGSCU is also involved in M&E activities for the experimentation of the Civil Peace Corps (CCP), in co‑ordination with the CCP Monitoring Committee. Such experimental CCP projects, under the UCS actions conducted abroad, assign volunteers to actions related to non-governmental peace in areas at risk of/in conflict, or environmental emergencies (Italian Ministry of Labour and Social Policy, 2015[31]; DGSCU[32]). The monitoring activities of CCP actions have predominantly entailed the identification and analysis of CCP volunteers’ profiles, expectations, and assessment of the CCP experience. Such information was collected through short questionnaires administered to volunteers at the beginning, midway point, and end of the CCP experience (DGSCU, 2019[9]).

Table 2. Menu of options to complete UCS M&E system with outcome and impact indicators

Indicators	Disaggregation*	Comments by authors
Outcomes
Outcome – Human capital: UCS volunteers accumulate increased knowledge, skills and abilities through non-formal learning and are better oriented regarding the development of their professional and personal life (soft skills; personal, social, active citizenship competences)
Volunteering placement completion rate	Sector, region, municipality, entity type, demographic and socio‑economic characteristics	The indicator reports on the percentage of participants who successfully complete the placements. Monitoring this indicator helps assess UCS ability to provide meaningful experiences and support for volunteers. It also gives an idea of the continuity of the actions on the territory, indirectly suggesting the existence of potential impacts of the UCS actions.
% of UCS volunteers who self-report/ are assessed to have made improvements in personal and professional outcomes thanks to the UCS	Sector, entity type, general/specific training, demographic and socio‑economic characteristics	The indicator seeks to demonstrate personal growth and capacity by tracking participants’ acquisition of new skills and competences during the service period. Personal and professional outcomes can encompass a longlist and should be predefined to allow for clustering – options could include leadership, communication, problem-solving, teamwork skills; self-confidence, self-esteem, resilience, empathy, other aspects related to the psychological and emotional well-being of participants; and active citizenship skills and civic engagement. A way to measure these indicators is to use pre‑, mid-term and post-placement questionnaires to measure volunteers’ perceived improvement in specific skills. Allowing respondents to complement closed questions with an open question on this dimension would allow to better nuance the understanding of a dimension which is complicated to measure.
% of UCS volunteers having converted the UCS experience into educational credits		The indicator is based on the assumption that universities may, within the limits established by current regulations, award educational credits to volunteers who have undertaken UCS activities relevant to professional development and academic curriculum.
Level of satisfaction of UCS volunteers with the UCS experience		The indicator evaluates the level of satisfaction among UCS volunteers regarding their overall experience, training, and volunteering placement. It captures participants’ perceptions of their experience, indirectly reflecting UCS effectiveness in providing opportunities for personal growth, skill development, and community engagement. Monitoring this indicator helps ensure that the UCS is meeting the needs and expectations of UCS volunteers. Further levels of disaggregation related with the characteristics of the programme/project could be considered, for instance asking volunteers to identify the main beneficiaries of the intervention (the volunteers themselves, the entity, the final beneficiaries, the community, etc.).
Level of satisfaction of UCS volunteers with training		The indicator provides an indication of the level of satisfaction of UCS volunteers with different aspects of the general and specific training received, for instance: preparation of trainers in their disciplinary fields; quality of training; consistency of contents with the activities carried out; timeliness and clarity of training planning; organisation and schedules; quality of teaching materials and support tools; adequacy of equipment; adequacy of spaces; relationship with other volunteers.
Outcome – Services of public utility: UCS volunteers engage in community service activities, increasing in the offer of services of public utility in the territories
Number of UCS programmes/ projects concluded	Sector, region, municipality, entity type, demographic and socio‑economic characteristics	The indicator counts the UCS programmes/projects that have been completed within a specified timeframe, offering an overview of UCS activity level. It provides insight into the volume of efforts successfully deployed to contribute to the desired change.
Number of volunteer hours contributed		The indicator quantifies the amount of time youth volunteers dedicate to the UCS. It helps demonstrate the level of commitment and contribution of UCS volunteers.
UCS volunteers’ assessment of the impact of actions on the territory		The indicator tries to capture the impact of volunteers’ efforts on the actual beneficiaries, providing insights on the effectiveness and relevance of the implemented actions. It could pre‑identify a list of outcomes and measure them through a rating system/Likert scale.
Level of satisfaction of UCS volunteers with the UCS setting, tools and delivery		The indicator informs on the level of satisfaction of UCS volunteers with different operational aspects of the UCS programme/project delivery, including but not limited to: the reception by the entity; information received on the organisation and service; consistency between activities carried out and project description; relationship with people in the entity; working hours; co‑ordination and organisation of activities; equipment and resources available; spaces; support received by the staff of the entity; support received by the OLP; climate and relations within the working group; existing rules in the place of service; compensation received; appropriateness of the role and activities performed; ability to express ideas suggestions, proposals (see UCS end-of-placement pilot questionnaire).
Level of satisfaction of UCS entities with the UCS volunteers’ actions		The indicator assesses the satisfaction levels of the host organisations, measuring their perception of the volunteers’ contributions and overall UCS effectiveness. It can indirectly inform on UCS volunteers’ contribution to positive changes in the territory. In any case, a degree of caution is required with such indicators considering potential biases in the assessment due to various interests at stake.
Impacts
Impact – Active citizenship: UCS volunteers develop a sense of civic responsibility, becoming active citizens in their communities
% of USC volunteers reporting intention to participate in community projects (or similar) as a result of the placement	Sector, region, municipality, entity type, demographic and socio‑economic characteristics	Gives an indication of UCS influence on long-term civic engagement by measuring UCS volunteers’ intentions to continue their involvement in civic and community activities after completing their placement. It examines whether the UCS instils a sense of social responsibility and encourages youth to become active citizens in their communities.
% of USC volunteers who are engaged in community projects (or similar)		Gives an indication of UCS influence on long-term civic engagement by measuring the number of UCS volunteers who are involved in civic and community activities right after completing their placement and in following periods (at the end of the placement, and 6/12/18 months after the placement). It examines whether the UCS instils a sense of social responsibility and encourages youth to become active citizens in their communities.
Composite indicator measuring Active Citizenship	Demographic and socio‑economic characteristics	See literature on the use of a composite indicator to measure Active Citizenship (Hoskins and Mascherini, 2008[33]), around the dimensions of Protest and social change, Community life, Representative democracy, and Democratic values. The indicator was adapted in research of the National Institute for the Analysis of Public Policies on the National Civil Service (De Luca, 2022[29]), encompassing all dimensions but the one on Representative democracy.
Impact – Employability and employment: UCS volunteers’ experiences in the UCS enhance their professional development, including for disadvantaged youth
UCS volunteers’ occupational or education status at the end of the placement, and after 6/12/18 months, compared to situation before placement	Sector, region, municipality, entity type, demographic and socio‑economic characteristics	The indicator assesses the longer-term effects of the UCS on individuals’ career decisions and educational pathways. It highlights the potential for UCS to positively impact participants’ professional trajectories and contribute to their overall development. It also allows to calculate additional indicators, such as the reduction in quotes of inactive youth after their participation to the UCS.
Multidimensional Employability index	Demographic and socio‑economic characteristics	The construction of this multidimensional index requires information on the following variables: Education and training: variables related to the “educational qualifications”; the condition of being “in training activities or not” at the time of the survey and to any competence certifications obtained. Labour market experiences: “number” and “type” of working and traineeship experiences acquired before the UCS. Labour market activation: active job search; CV review and update; career design planning, in terms of clearness of professional goals. Mobility: mobility resources (foreign languages skills, experiences abroad and planning) and mobility “experiences” (experiences of mobility – from one city to another- already been made and if an individual has already left their household and the availability to move and live abroad). See De Luca, Ferri and Di Padova (2019[34]) for methodological specifications.
Number of agreements with potential employers to promote the employment of volunteers who have completed the placement	Sector, region, municipality, entity type, participating in the UCS or not	The State, Regions and Autonomous Provinces may enter into agreements with associations of private companies, associations representing co‑operatives and other non-profit bodies, to facilitate the placement in the labour market of young people who have performed the UCS. E.g. provision in Law 74 of 21 June 2023, whereby UCS volunteers who have successfully completed the UCS without any negative record are entitled to a reserved quota of 15% in the recruitment of non-managerial personnel within public administrations, as well as within special companies and institutions instrumental to the activities of local authorities (Article 1, paragraph 9‑bis).
Number of job posts created for UCS volunteers	Sector, region, municipality, entity type, participating in the UCS or not	Jobs created in supported entities as well as in other organisations who have signed agreements with the UCS.
% of UCS volunteers stating that the UCS majorly contributed to future education/ career choices	Sector, region, municipality, entity type, demographic and socio‑economic characteristics	The indicator attempts to measure the contribution of UCS in terms of future educational and professional life choices in terms of career objectives or further education in specific areas.
Skills and abilities acquired by the UCS volunteers	Sector, region, municipality, entity type	See the discussion around the certification of skills/competences in OECD (forthcoming[21]). Identification of the main types of skills and abilities that UCS contributes to strengthen. This would require advancing specific methodologies considering the complexity of the recognition of competences acquired through the UCS. Skills and competencies may include social, professional, communication, relational, digital, organisational and managerial, language skills, others. A complementary option would be for UCS entities to provide an assessment on those dimensions, yet with major complications related to the nature and measurability of soft skills and to establish a causality between UCS interventions and such outcomes, as well as potential ethical issues related to volunteers’ assessment – see OECD (forthcoming[21]) for further information on the certification of competences.
Impact – Social cohesion and resilience: The interventions contribute to address societal and development issues, strengthening the territories and country’s resilience and capacity to mitigate the social and economic impact of the crisis, paying a special attention to the green and digital transitions
Territory/Community impact	Sector, region, municipality, entity type	Indicators related to community impact aim to evaluate UCS influence on the community or the beneficiaries it serves. They should measure the tangible and intangible benefits generated by volunteers’ service activities for the community. Measuring such impact is methodologically complicated, as stressed in evaluations of other volunteering programmes – e.g. European Commission (2017[35]) – due to the informal nature of interactions between community/volunteers, and the lack of tools to measure impact at local level. Examples of indicators include: improved infrastructure, enhanced social services, or increased community engagement; they nonetheless require a cleared definition, while considering challenges for the collection of related information.

Notes: This list is intended to be a menu of options for consideration by UCS managers.

* See the explanation in the section above on UCS indicators could be improved in terms of coverage and disaggregation.

Source: Authors based on DGSCU (2022[23]; 2021[22]), other programmatic documents, examples extracted from reports by UCS entities, and European Commission (2017[35]).

Recommendation 2: Follow the UCS ToC to identify SMART UCS indicators along the results chain

In order to build a well-functioning and comprehensive set of indicators, it is recommended that UCS authorities follow the key elements of UCS ToC and assign SMART indicators to each of them. Particular attention would be needed to i) improve the existing set of input, activity, and output indicators in terms of coverage and disaggregation of the information; and ii) complement such indicators with a selection of outcome and impact indicators. In this process, UCS authorities are advised to:

• Consider the principles of “SMART” indicators.

• Select an optimal number of indicators, with a right balance between information needs and costs for the collection and processing of information and data.

• Ensure that indicators are available for each level of the ToC, expanding the exercise presented in Table 2 to cover inputs, activities, outputs, outcomes, and impacts.

• Ensure that indicators are disaggregated according to key factors of relevance for the UCS.

• Ensure a longitudinal perspective in M&E, with indicators to be fed at different points in time.

• Ensure a good coverage of the target population, encompassing local project operators, young volunteers as well as young applicants who were not selected into the UCS, wherever relevant.

• Consider the possibility to apply an approach with core and specific indicators to better capture the effects of the UCS on the sectors, territories, and communities.

• Identify complementary outcome and impact indicators that could be of interest for collection through evaluation exercises.

Notes

← 1.  Canada Service Corps systematically collects data on participant age ranges, starting and end dates, whether the participant completed their placement, type of placement, planned and actual number of hours spent on service placement, hours spent on training, province or territory where the placement took place, social identity identifiers and education level of participant, among others (Canada Service Corps, 2023[51]). It also implements an anonymous post-participation survey collecting data on identity factors, motivations for joining the programme, appreciation of different aspects of the programme, skills development and impact of the volunteering opportunity on the person’s civic and community engagement as well as on the community (Gagliardi, Perez-Raynaud and Robinson, forthcoming 2024[54]).

← 2.  The French Service Civique regularly monitors indicators such as: number of volunteers by gender, skills/education level, labour market status, age, province, region and sector; quarterly flows; new volunteers by type of organisation; duration of the volunteering (completed/interrupted, including reasons for interruption); weekly hours of volunteering. Information extracted from the programme’s website at https://injep.fr/wp-content/uploads/2022/08/Donnees-regulieres-2021_service-civique.xlsx. The Activity report template for accredited organisations to the French programme also includes sections on “Impact of the Civic Service” and “Learnings and observations” (Gagliardi, Perez-Raynaud and Robinson, forthcoming 2024[54]).

← 3.  The “40% clause”, introduced at the time of the conversion of Decree‑Law No. 77/2021, as amended, into Article 2, paragraph 6‑bis of Law No. 108/2021 – Annex Part 1, provides that the central Administrations involved in the implementation of the NRRP must ensure that at least 40% of the resources that can be allocated territorially, regardless of their financial source, are allocated to Italy’s southern regions.