Guidance for a Monitoring and Evaluation System for Italy’s Universal Civil Service

4. Planning for the collection and processing of data

Once the set of indicators has been identified following the ToC, data collection and analysis need to be organised accordingly. Proper planning of the monitoring function should therefore describe the methodology for data collection and processing (FAO, 2023[36]), including data collection tools and their frequency, as well as roles and responsibilities for data collection and analysis.

When planning data collection, it is important to balance the use of existing secondary data against the need for additional primary data. Primary data can be tailored specific information needs but require resources for data collection, storage, and analysis (Markiewicz and Patrick, 2016[37]). Secondary data are resource‑efficient but may pose challenges in terms of accessibility and quality.

Key data collection methods and tools for the UCS may include administrative records (e.g. volunteer registration forms for demographics, timesheets for hours of service, training attendance, etc.); feedback forms, interviews, and focus groups on volunteers’ and entities’ satisfaction and UCS results; pre‑, during- and post-programme surveys to measure changes in employability and citizenship skills (or any other relevant impacts); and standardised programmes/projects reports, among others. Such tools are already in use in the UCS but would need to be enhanced in terms of thematic coverage; organisation and standardisation of data collection; as well as consistency and regularity in implementation.

UCS interventions are monitored, but there is room for improvement in data collection and centralisation

Entities’ monitoring systems are validated by the DGSCU but tools and approaches for data collection vary across entities

At the programme/project level, each entity that is listed in the Register (Albo) is required to provide information on its monitoring system, whose adequacy is assessed by the DGSCU. Such information should encompass the monitoring functions and roles, key tools and data collection processes, planned analysis and dissemination activities, and their timeframe. In particular, monitoring systems are expected to enable the tracking of: progress in project activities, aligning with the indicators and timeline outlined in the project plan; delivery of training activities, tutoring and the certification of competences; as well as the satisfaction of volunteers and the overall context in which the UCS experience happens. Entities designate a person responsible for monitoring actions (on volunteers’ demographic characteristics and on training provision), the preparation of the annual report on the UCS programmes/projects, and the administration of satisfaction questionnaires to volunteers.

Since the entities are responsible for the implementation and monitoring of their programmes/projects, the UCS currently does not impose a single, comprehensive monitoring system with pre‑defined tools for data collection. This set-up responds to DGSCU’s decision to grant autonomy to entities considering their specific characteristics and diverse missions and is in line with other international practices (e.g. the Canada Service Corps), where organisations implement and monitor their activities autonomously. Nonetheless, using heterogeneous data collection tools risks the dispersion of relevant information. A good balance needs to be struck between the central collection of information and the degree of autonomy left to the entities, taking into account the often‑scarce resources at their disposal to carry out in-depth M&E.

Every year by the end of March, entities are to submit to the DGSCU an annual report on the results achieved with projects in the previous year. The report should clearly inform (among other aspects) on the objectives, indicators and targets related to the outputs produced; the activities conducted; the resources committed; the training courses provided; the entity’s self-assessment of the achievement of the objectives; and the level of satisfaction of volunteers (DGSCU, 2021[38]). A summary report also needs to be published on the entity’s website. Monitoring by entities should consist of a flow of continuous observation of the system, highlighting its strengths and the gaps that need to be addressed. Consultations with entities, however, highlight that monitoring is often seen as a burdensome and formal exercise to fulfil the requirements of the law. Consequently, in various cases monitoring reports only consist of a description of the demographic characteristics of the volunteers (e.g. distribution by age, regions, and sectors) or a summary of the projects.

Together with data on programme/project implementation, questionnaires to volunteers represent a key source of information for such annual reports. Pre‑selection,1 follow-up2 and satisfaction3 questionnaires are key to understand: a) the characteristics, motivation, expectations and level of satisfaction of candidates/volunteers; b) the outcomes and impacts of the interventions on young volunteers and the territories; as well as c) success factors and areas for improvement (Table 2 shows that questionnaires to young people are relevant to feed various indicators on the UCS performance). Questionnaires (in paper or online) are sent out to volunteers every three or four months: at the beginning, in the middle and at the end of the service. Entities also send out questionnaires to their staff in charge of the UCS (including the local project operators) to collect data on the implementation of the project and verify its progress in relation to the initial plan. By being administered at different stages of the interventions, the questionnaires can collect information on the same question at different points in time and allow to detect emerging issues over the course of the projects. Administering such questionnaires also some time after the end of the placement would provide valuable information from an evaluative perspective, for instance allowing to understand young people’s education or labour market status of young people, or their motivation for active citizenship, at different points in time.

Yet, some issues can be identified regarding the current use of such questionnaires. First, there is a variable amount of information generated at a programme/project level, which is currently not summarised in a UCS-level dashboard or database. Second, the use of such tools varies across entities, from cases where questionnaires are just treated as a formal step, to virtuous cases where they are used for evaluation and leaning purposes (see examples in sub-section below). And third, follow-up of volunteers finishes at the end of the placement without capturing effects over a longer period.

There are good examples of M&E systems for UCS programmes/projects

LD 40/2017, Article 7 states that the regions and autonomous provinces of Trento and Bolzano, after signing one or more agreements with the Presidency of the Council of Ministers, may perform functions including monitoring the management of the activities carried out by the UCS in the territories of each region or autonomous province, as well as evaluating the results of the interventions carried out by UCS entities carried out in their territories. At the time of writing, a technical working group is working on such an agreement.

Meanwhile, there are already good examples of thorough monitoring exercises conducted by Regions and Autonomous Provinces, which are collecting a wealth of information on the different elements of the UCS ToC. Two of them (by the Friuli-Venezia Giulia Region and the Autonomous Province of Trento) are presented below.

The Friuli-Venezia Giulia Region collects data to assess the quality and value of the UCS experience in the Region, through questionnaires on expectations and different evaluative aspects of the experience to all volunteers involved in UCS projects in the Region (DGSCU, 2021[22]), see Box 2. This example also highlights the risk of dispersion of information mentioned above: since the data are not aggregated at a central level, valuable information risks not being exploited at its most by UCS authorities.

Box 2. Questionnaire for end-of volunteer service: A virtuous example from the Friuli Venezia Giulia Region and relevance for M&E indicators

The final questionnaire administered by the Friuli Venezia Giulia Region has a high level of detail and covers the following sections with a mix of closed and open questions. The themes and subthemes covered in the questions could be used to build and feed related indicators.

 

Area and information	Relevance for indicators following the ToC outline (Figure 3)
Personal details: educational attainment; participation in programmes abroad; employment status; recent participation in training or professional development; involvement in internships or work abroad; previous voluntary work experiences; paid roles in social contexts; additional employment during the year of UCS	Output – “Volunteering placements”
Project information: sector, entity; service location; weekly service days and hours	Output – “Programmes and projects”
Civil service: before the UCS, motivation for: nonviolent defence, collaboration and teamwork, solidarity, association involvement; familiarity with the UCS history and nature; how awareness of the project was gained; reasons for joining the UCS	Outcome – “Human capital”
Commencement of service: start date; role within the project; primary activities; consistency of activities with project description; activity alignment with project objectives; conforming to expectations of involvement in activities; interaction frequency with project beneficiaries; reception by hosting entity; interaction with others in the entity; supervision by the LPO; participation in project activities; satisfaction with service conditions; assessment of various project aspects; assessment of the project in various areas; assessment of project strategy	Outputs – “Volunteering placements” and “Programmes and projects” and outcomes – “Human capital” and “Services of public utility”
Training: hours of project-specific and of general training; assessment of training quality; training provider	Output – “Training and development” and outcome – “Human capital”
Project suspension and resumption	Output – “Volunteering placements”
Overall assessment: importance of the UCS; comprehensive project assessment; evaluation of project outcomes; acquired skills; future applicability of training; rating of overall UCS experience; recommendations of UCS experience to others; intentions for future civic engagement; identification of encountered difficulties; suggestions for project enhancements; concluding remarks.	Outcomes – “Human capital” and “Services of public utility” and impacts – “Active citizenship”, “Employability and employment” and “Social cohesion and resilience”

Source: Authors based on Regione Autonoma Friuli Venezia Giulia (2021[39]) and Figure 3.

While some organisations limit themselves to administering questionnaires, others organise additional monitoring activities, such as face‑to-face and online meetings with young volunteers and local project operators. The Three‑Point Monitoring system built in in 2020 by the Autonomous Province of Trento is another positive example, which focuses on the volunteers, the local project operators, and the civil service office (Box 3).

DGSCU is implementing promising pilot initiatives focusing on results

It is worth mentioning that the DGSCU has recently taken significant first steps towards the collection of information on UCS outcomes and impacts. In addition to the above‑mentioned monitoring activities, UCS authorities have piloted actions to collect information on volunteers’ satisfaction at the end of the service (as mentioned in section “Understanding and outlining UCS results chain through a ToC”). The DGSCU prepared and administered end-of-placement satisfaction questionnaires to volunteers who participated in the first UCS call for applications under the NRRP in 2021. These questionnaires were administered to volunteers on a trial and voluntary basis and covered aspects such as demographic and socio‑economic characteristics of the volunteers, involvement in volunteering activities prior to the UCS, motivations for joining the UCS, activities developed as part of the UCS, assessment of the training and of the activities, level of satisfaction with the project delivery and the UCS experience, UCS contribution to obtaining a job offer, and suggestions for improvement. The administration of such questionnaires may be extended to all the UCS interventions and, although at the time of writing there are no concrete plans by the UCS authorities to make such questionnaires mandatory, this option is under consideration. Such questionnaires have the potential to collect information on outcomes and impacts of the UCS and set a useful basis for further developments of tools for regular data collection.

Box 3. The “3P” Monitoring system in Trento for the Provincial Civil Service

The Three‑Point (abbreviated to 3P) Monitoring is the digitalised monitoring system in Trento’s civil service. Its objective is to accompany the UCS activity with the aim of continuous improvement.

Monitoring of local project operators (OLPs)

The OLPs must organise a monthly meeting with volunteers to check progress of activities, analyse the learning process, and discuss proposals for improvement. They also fill in the following digital tools:

• Standard monthly OLP report: eight questions in the form of a check list (on specific training, the monitoring meeting, volunteer’s electronic attendance register, evaluation of the project’s progress), completed in three minutes.

• Mid-Project OLP Report: five sections (on the coherence between project proposal and implemented activities; OLP efficiency; OLP satisfaction; impact on the entity; improvement) with a total of 29 questions. The report helps to make any corrections along the way.

• Final OLP report on project progress: five questions (on project implementation; results achieved; effectiveness – or not – of the projects and reasons behind it; relationship between the project and the entity’s activity; any proposals for redesigning). The report gives an assessment of how the project has progressed and whether it should be replicated.

• Final OLP Report on the participant: seven questions (on the activities carried out by the volunteer; his/her contribution to the realisation of the project; his/her maturation; the technical and transversal skills acquired; indications to the volunteer for the development of the work and citizenship dimension; the reporting of skills related to active citizenship acquired during the project). The report provides an evaluation of the volunteer’s participation in the project.

Monitoring of UCS volunteers

Volunteers in service must fill in several different forms depending on the month of the project:

• At the end of the first month, they complete the initial diary form to assess the placement phase.

• Halfway through the project, the mid-project questionnaire is sent by the civil service office.

• At the end of the project, the end-of-service questionnaire is sent to them by the same office.

• In the other months, they complete the “Standard diary form”.

After all the forms have been completed by the volunteers, the system sends an email with the content just entered, which must be forwarded to the responsible OLP.

Source: Authors based on project website: https://servscup.wordpress.com/2020/08/29/monitoraggio-3p-cose/.

Only some monitoring data are collected in the UCS central database

The monitoring efforts for the UCS are collated in different datasets belonging to the central database “Helios”; including:4

• A primary dataset, part of the Single IT system, used by the UCS to collect and manage data related to volunteers, projects, and participating organisations.

• Accredited entities dataset, storing information about the host entities participating in the UCS.

• Project dataset, including information on the call, evaluation, and approval.

As they belong to the same database, the above datasets are fully interoperable. The data primarily support administrative reporting. Yet, the possibilities to conduct analyses on the UCS outcomes and impacts – also from a longitudinal perspective – based on these data are scarce, because UCS monitoring efforts mainly focus on inputs, activities, and outputs (see “Setting SMART indicators based on UCS ToC”).

Recommendation 3: Systematically structure monitoring activities

Data collection, treatment, aggregation, and reporting efforts within the UCS can become more effective and efficient if they are systematically structured and managed along the ToC and across all entities. Therefore, it is recommended that UCS authorities systematically structure monitoring activities. In this context, fit-for-purpose IT tools and systems represent a precondition for data collection. In this process, UCS authorities are advised to:

• Ensure co‑ordination of UCS M&E activities through a clear governance system. Central co‑ordination ensures consistency and homogeneity in data collection methods, instruments, and reporting requirements across all dimensions and actors of the UCS. As M&E tasks are identified more in detail, the roles and responsibilities for data gathering and management – within the UCS and at the entity level – should be identified accordingly, to make sure that all actors involved contribute M&E efforts in the most constructive way.

• Prepare indicator fiches or a summary table of the set of indicators to facilitate effective data collection, detailing each indicator’s name, its description/rationale, data collection methods/tools/ sources, the level of disaggregation, and the frequency of data collection.

• Ensure that enhanced tools, such as unified data collection tools, are implemented across all entities/programmes/projects. A first example would be the standardisation of the mandatory questionnaires for volunteers and OLPs, which could also include questions on outcomes and impacts.

• Explore agile systems for the administration of such questionnaires (i.e. through a mobile app). Entities should be given the choice to use these tools in addition or in replacement of their usual monitoring tools and to explore technical compatibility between them. The use of IT tools that are compatible with data processing needs should be prioritised. While open-source tools have advantages, ensuring compatibility and functionality for data management, treatment, analysis, and reporting should take precedence.

• Store data in a central system, following such structured approach to data collection, in order to prevent the dispersion of information across different levels of the UCS. Opportunities for interoperability and links to complementary administrative databases could also be considered.

• Recognise the sensitivity of data treatment and consider data protection and privacy regulations while enhancing data collection. Accordingly, protocols should be established to protect sensitive data, ensuring compliance with relevant laws and regulations.

• Implement a quality assurance system to ensure data reliability and consistency.

Notes

← 1.  See an example from Comunità Papa Giovanni XXIII: https://serviziocivile.apg23.org/questionario-italia/.

← 2.  See an example from the Marche Region: Questionario SCR Marche di inizio servizio Survey https://www.surveymonkey.com/r/KCHR5YJ.

← 3.  See an example from the Campania Region: Servizio Civile Regione Campania, http://serviziocivilecampania.it/questionario-volontari/.

← 4.  It should be noted that the UCS also needs to provide data for ReGiS (the central management system of the NRRP) or for other information requests that the DGSCU may receive.