This chapter evaluates the impact of the ATIVAR.PT internship programme on employment probability, earnings, and unemployment benefit claims. To account for the selection of participants into the programme, the counterfactual impact evaluation deploys a propensity score matching methodology. The findings highlight significant improvements in employment probabilities and earnings for participants, alongside a neutral effect on the usage of unemployment benefits. These effects are similar to those found in related evaluations in Portugal. Moreover, the analysis indicates differential impacts across subgroups, with the internship being particularly beneficial for lower-educated jobseekers and those residing in non-urban areas.
Impact Evaluation of Active Labour Market Policies in Portugal
Connecting People with Jobs
Abstract
Launched in Portugal in 2020, the ATIVAR.PT internship programme is designed to enhance the labour market integration of young individuals and the professional retraining of the unemployed by facilitating the acquisition of practical work experience. This chapter presents the findings from the counterfactual impact evaluation of the programme, focusing on its effects on the labour market outcomes of participants in comparison to a control group of similar jobseekers who did not participate in the programme. The evaluation includes individuals registered as unemployed from May 2020 through August 2021. Labour market outcomes are assessed up to 24 months after their registration with the Public Employment Service (PES) of Portugal (Instituto do Emprego e Formação Profissional, IEFP).
The chapter begins by outlining the characteristics of the participants in the internship programme. It then assesses the impact of the programme on the employment probability and on earned income, applying the methodology outlined in Chapter 3. The results are compared with findings from related literature. The analysis also examines how participation in the programme affects unemployment benefit claims. Moreover, the chapter explores the programme’s impact across various subgroups of the population based on their gender, education level and place of residence. The chapter concludes by summarising the key outcomes of the evaluation, offering insights into the programme’s overall efficacy in enhancing labour market integration for young jobseekers in Portugal.
Participants in the ATIVAR.PT internships differ significantly from unemployed individuals who do not participate in the programme, as detailed in Table 4.1 (columns 1 and 2). Reflecting the programme’s targeting, internship participants are, on average, 14 years younger than other jobseekers. They are more likely to be single (by 44 percentage points) and less likely to have dependents. Education levels among participants are notably higher, with 75% holding a tertiary education, compared to less than 20% in the broader unemployed population.
In terms of desired occupations, those enrolled in the internship programme are more likely to seek employment as professionals or technicians (by 6 percentage points) in comparison with other registered jobseekers. Conversely, they are less inclined towards roles in service and sales (by 18 percentage points) or craft and related trades. Moreover, a higher proportion of participants, relative to other jobseekers, are searching for a job for the first time through the PES (by 58 percentage points) and are less likely to be eligible for unemployment benefits, which aligns with their limited previous work experience and lower past earnings and unemployment benefits received before their current registration as jobseekers. Nevertheless, about 40% of the participants had some previous work experience within the last two years before their registration in the PES. Additionally, participants and non-participants are not evenly distributed across Portugal, as evidenced by their registration at different PES offices. A smaller proportion of participants than non-participants live in predominantly urban areas (a 10-percentage points difference between the two groups).
This descriptive analysis reveals a strong degree of self-selection in the programme, highlighting that any direct comparison of future labour market outcomes of participants and non-participants would be, to some degree at least, driven by these differences in their personal characteristics and the jobs they are looking for. Hence, as outlined in Chapter 3, propensity score matching is used to overcome this challenge. This methodology generates a control group of jobseekers (Table 4.1, column 3) who are closely matched with internship participants in terms of observable characteristics, offering a more appropriate basis for comparison.
Comparison of observable characteristics by treatment status
Note: The group of unmatched non-participants refers to all unemployed individuals in the sample that do not participate in ATIVAT.PT internships while the matched group refers to the individuals identified as comparable to internship participants through nearest-neighbour propensity score matching. Importantly, due to the use of nearest neighbour matching with replacement, the observations in column 3 are weighted to mirror the participant number, ensuring an accurate comparison between participants and their matched counterparts. The table includes individuals registered as unemployed from 13 May 2020 to 31 August 2021, for whom the outcomes of interest (employment status, earnings, and unemployment benefits) and the relevant control variables displayed in the table can be observed.
Source: OECD calculations based on Social Security (ISS) and Public Employment Service (IEFP) data.
The impact evaluation results reveal that ATIVAR.PT internships have a significant positive impact on the probability of employment for participants shortly after the programme’s start (Figure 4.1). This employment effect of about 50 percentage points six months after unemployment registration reflects the period when internships are active for all participants in the sample (as illustrated in Figure 3.5 in Chapter 3). This increase in employment probability is an expected outcome due to the programme’s design, which guarantees employment during the internship period. However, it is important to bear in mind that non-participants may also have found employment through alternative (more or less effective) means during the same time period. Therefore, even though this effect is anticipated, quantifying its magnitude remains crucial for understanding how much the internships help participants compared to what would have happened to them had they not taken the internships, and thus for understanding the programme’s effectiveness.
As participants transition out of the programme (which typically lasts nine months), the employment effect drops to approximately 10 percentage points 15 months after registering as unemployed. This decline indicates that the programme’s immediate effect on employment lessens after its completion, as some participants face challenges in securing subsequent employment.
Nevertheless, the employment probability for participants remains consistently about 10 percentage points higher than for non-participants in the control group from 15 to 24 months after registering as unemployed (the end of the observation period), indicating a lasting effect for at least nine months after the internships have concluded. This sustained impact, despite the natural attenuation post-completion, underscores the programme’s effectiveness in improving employment prospects, likely through the acquisition of experience, skill development, and networking during the internship.
The introduction of the “ATIVAR.PT Employment Award” subsidy alongside the internships, aimed at reducing wage costs for employers offering permanent contracts to former interns (see Chapter 3), may further encourage the hiring of programme participants and could possibly explain part of the positive effect on employment. However, the available data do not specify whether employment contracts are subsidised, limiting the ability to precisely determine the subsidy’s role in the observed employment effects. Individuals in the control group might also enter subsidised employment through different avenues at higher or lower rates than the internship participants. Therefore, it is not possible to say a priori whether the effect of the internship on non-subsidised employment would be smaller, greater or similar. This lack of data on subsidised employment for individuals in the treatment and control groups prevents a clear separation of the impacts attributable to subsidised versus unsubsidised employment contracts. Therefore, it would be beneficial for these data to be made available in the future for similar impact evaluations to better understand the effect of the internship programme, as well as the take‑up of the subsidy.
Percentage point change in employment probability
Note: The analysis presents nearest-neighbour propensity score matching results which matches individuals based on a number characteristics: history in employment over the past two years (number of employment spells, employment status and earnings), unemployment benefits received over the past two years, being an unemployment benefit recipient at registration, looking for a job first time through PES, the occupation of the job searched (1‑digit), age, gender, marital status, unemployment status of the spouse, education level, having dependents, the local employment agency of registration (geographical location), the quarter of registration into unemployment (time trend). The confidence intervals are shown at the 5% level of significance and represented by the whiskers delimiting the dotted lines on the charts. Outcomes are observed monthly since registration in unemployment.
Source: OECD calculations based on Social Security (ISS) and Public Employment Service (IEFP) data.
The international literature on the impact of Active Labour Market Programmes (ALMPs) on employment probability, as summarised by Card, Kluve, and Weber (2018[1]), does not detail the effects of internships but rather includes various types of ALMPs. This meta‑analysis indicates a median effect size of 1 percentage point within a similar time horizon (up to one‑year post-programme completion), with a range from ‑7 (5th percentile) to 13 (95th percentile) percentage points. Compared to these figures, the 10-percentage point effect observed for the internship programme in Portugal stands out as notably high.
However, this median effect masks substantial variability among different types of ALMPs. The positive impacts of training programmes, for example, typically emerge after two to three years. Consequently, analyses limited to the first year post-programme do not yet detect these benefits, or they even suggest negative outcomes due to participants temporarily leaving the labour market to engage in training (known as “lock-in effects”). Conversely, interventions such as counselling or sanctions tend to show immediate, substantial effects, although these may not persist over the long term.
This variability underscores the importance of evaluating the long-term impact of internships. It remains to be determined whether internships, like counselling that provide immediate support and guidance, offer pronounced short-term benefits that diminish over time, or if they, like training programmes that focus on skill development, facilitate sustained positive outcomes through the acquisition of experience and skills.
Within the Portuguese context, a counterfactual impact evaluation of the Youth Employment Initiative (Duarte et al., 2020[2]) analysed the effects of internships of varying durations: 6, 12, and 18 months. The findings of that study for a similar time frame to the one in this report (i.e. the effects assessed for the period between six months and one year after the end of internships) indicate a positive 8 percentage point impact on employment for six‑month internships and a 16-percentage point impact for 12‑month internships. Compared to these, the ATIVAR.PT 9‑month internships for youth demonstrate an effect of a similar magnitude during nine months after the end of the programme.
In addition to determining whether internships have increased access to employment, it is crucial to examine employment quality and the programme’s effect on participants’ financial stability. The evaluation thus assesses the impact of internships on both monthly and cumulative earnings over time.
The results of the impact evaluation indicate that ATIVAR.PT internships significantly increase participants’ monthly earnings (Figure 4.2, Panel A). The most substantial increase occurs during the internship, with participants earning, on average, approximately EUR 344 (around half the minimum wage in 2020) more than non-participants six months after registering as unemployed, reflecting the immediate financial benefits derived from paid internships.
Over time, the increase in monthly earnings decreases to about EUR 120 (around 20% of the minimum wage) by the 15th month following unemployment registration, coinciding with the conclusion of the internships for participants in the sample. This reduction in the effect on earnings may be due to the completion of internships and the participants’ transition to unemployment/inactivity or other forms of employment, which may not offer the same level of compensation. However, by month 24 after unemployment registration – at least nine months following the completion of internships – the effect on earnings increases slightly to EUR 180 (around 30% of the minimum wage). This persistent positive effect suggests that the programme contributes to securing better quality jobs and maintaining a financial advantage even after internship completion.
The analysis of cumulative earnings complements the monthly earnings findings: although there is a continuous increase, the rate of growth in cumulative earnings due to the internship programme decelerates as the internships conclude, before slightly increasing again (Figure 4.2, Panel B). By the 24th month, participants have, on average, accumulated additional EUR 4 834 in earnings due to their programme participation. This substantial and significant increase in cumulative earnings over the two‑year period demonstrates a net financial gain for participants, underscoring the programme’s effectiveness not only in enhancing access to employment but also in improving financial stability.
Change in monthly earnings at month t (Panel A) and cumulative earnings up to month t (Panel B)
Note: The analysis presents nearest-neighbour propensity score matching results which matches individuals based on a number characteristics: history in employment over the past two years (number of employment spells, employment status and earnings), unemployment benefits received over the past two years, being an unemployment benefit recipient at registration, looking for a job first time through PES, the occupation of the job searched (1‑digit), age, gender, marital status, unemployment status of the spouse, education level, having dependents, the local employment agency of registration (geographical location), the quarter of registration into unemployment (time trend). The confidence intervals are shown at the 5% level of significance and represented by the whiskers delimiting the dotted lines on the charts. Outcomes are observed monthly since registration in unemployment.
Source: OECD calculations based on Social Security (ISS) and Public Employment Service (IEFP) data.
This study also examines the internship’s impact on participants’ dependency on unemployment benefits (UB), measuring the effects both monthly and cumulatively over the duration of the study.
Initially, the internship programme results in a decrease in the monthly amount of UB received by participants, with an average reduction of approximately EUR 10 six months after registration in the PES. This reduction reflects participants’ diminished need and eligibility for UB due to employment during their internships.
As participants conclude their internships, the monthly UB received by participants gets close to zero, becoming statistically non-significant from month 15 onwards. This null effect possibly indicates that, although participants experience higher employment rates and earnings after their participation in the internship, thus reducing their UB claims, those who become unemployed are potentially more eligible for (higher) UB, based on their previous employment and earnings, including those accrued during the internship1 and any subsequent employment. These two dynamics could neutralise each other, possibly resulting in a net null effect on the monthly UB amount. However, further analysis is required to capture more precisely the programme’s role in mitigating financial stress for participants, particularly those with limited prior work experience. If the internship leads to enhanced access to unemployment benefits, the programme could act as a crucial safety net, potentially facilitating broader externalities beyond financial stability. Collecting data on social outcomes, such as improvements in social inclusion, material deprivation and well-being, and health outcomes, like changes in mental and physical health, would be important to further explore these potential broader impacts.
Regarding cumulative UB, the evaluation reveals a significant negative effect of the internship programme over the 24‑month period following unemployment registration. The reduction in UB starts at EUR ‑26.5 by six months after unemployment registration, extending to EUR ‑63 by 24 months. This statistically significant reduction at 24 months demonstrates that programme participants have claimed less UB overall, implying government savings. However, the modest magnitude of this effect underscores the necessity of a comprehensive cost-benefit analysis. Such an analysis would weigh the government savings from reduced UB claims and potentially other reduced expenditures (like social assistance or healthcare costs) against the total costs of the programme. The aim of such analysis would be to determine not only the effectiveness of the programme, as evidenced in the evaluation in this report, but also its cost-effectiveness.
Change in unemployment benefits at month t (Panel A) and cumulative unemployment benefits up to month t (Panel B)
Note: The analysis presents nearest-neighbour propensity score matching results which matches individuals based on a number characteristics: history in employment over the past two years (number of employment spells, employment status and earnings), unemployment benefits received over the past two years, being an unemployment benefit recipient at registration, looking for a job first time through PES, the occupation of the job searched (1‑digit), age, gender, marital status, unemployment status of the spouse, education level, having dependents, the local employment agency of registration (geographical location), the quarter of registration into unemployment (time trend). The confidence intervals are shown at the 5% level of significance and represented by the whiskers delimiting the dotted lines on the charts. Outcomes are observed monthly since registration in unemployment.
Source: OECD calculations based on Social Security (ISS) and Public Employment Service (IEFP) data.
While previous sections focused on the aggregate effects of the internship programme, this section delves into the differential impacts across various subgroups of the unemployed population. Examining how the internship’s outcomes vary among these subgroups offers insights into its differential effectiveness and underlying mechanisms. This analysis is critical for understanding which segments benefit most and may guide strategic adjustments to the programme’s targeting and design. The analysis considers jobseeker characteristics such as gender, level of education, residence in urban versus non-urban municipalities, age and previous work experience.
In terms of gender differences, the analysis reveals no significant heterogeneity in outcomes. Two years post-unemployment registration (at least nine months after completing internships), the effects on employment and earnings for women (10.7 percentage points and EUR 168, respectively) and for men (12 percentage points and EUR 212) show no statistical difference (Figure 4.4, Panels A and B).
However, the impact on employment varies markedly by education level. Individuals with upper secondary education or less exhibit the most substantial increase in employment probability (16.6 percentage points), indicating the programme’s pronounced benefit for this group. Conversely, those with tertiary education experience a smaller, yet positive and significant, increase in employment probability of 9.9 percentage points. This suggests that while higher-educated jobseekers are more likely to participate in the programme, it is the lower-educated jobseekers who derive the greatest benefit.
Despite these variations in employment outcomes, the programme’s effects on job quality, as measured by earnings, do not show significant statistical differences across education levels. This indicates that while the programme significantly helps lower-educated individuals in securing employment, it does not result in a differential impact on the quality of employment compared to their higher-educated counterparts.
When examining the type of municipality of residence, individuals living in non-urban areas show a more pronounced positive outcome in employment (13 percentage points) compared to those in urban areas (9.7 percentage points), although these differences are significant at the 10% level but not at the 5% level. Similar to the education level analysis, there are no statistically significant differences in the effects on earnings by place of residence.
The analysis by age, dividing participants into groups below 30 and 30 years old or above, does not reveal heterogeneous effects.2 It is important to highlight that for the internship programmes included in this evaluation, eligibility was limited to individuals under 35, with the median age of participants being below 24. This specification means that the subgroup of individuals above 30 years old is exclusively those between 30 and 35 years old, representing a relatively small sample. Consequently, the precision of the estimates for this older subgroup is limited, reflected by large confidence intervals. Nonetheless, examining the subgroup below 30 years old is particularly relevant, as this age groups is the focus of various other policies in Portugal and the European Union more generally, such as the Youth Guarantee scheme. The persistence of the programme’s main effects within this younger subgroup reaffirms its effectiveness in enhancing employability and financial stability among youth.
Finally, the analysis shows no significant differences for participants with and without previous work experience in the last two years before their registration in the PES. Both groups exhibit similar improvements in employment probability and earnings, indicating that the programme is equally effective regardless of prior work experience.
In terms of the programme’s impact on unemployment benefits (UB) receipt across different subgroups, the analysis reveals no significant differential effects (Figure 4.4, Panel C).
Change in employment, monthly earnings and monthly unemployment benefits 24 months after unemployment registration
Note: The analysis presents nearest-neighbour propensity score matching results which matches individuals based on a number characteristics: history in employment over the past two years (number of employment spells, employment status and earnings), unemployment benefits received over the past two years, being an unemployment benefit recipient at registration, looking for a job first time through PES, the occupation of the job searched (1‑digit), age, gender, marital status, unemployment status of the spouse, education level, having dependents, the local employment agency of registration (geographical location), the quarter of registration into unemployment (time trend).
Source: OECD calculations based on Social Security (ISS) and Public Employment Services (IEFP) data.
The evaluation of the ATIVAR.PT internship programme furnishes key insights into its efficacy within the unique economic and social landscape shaped by the COVID‑19 pandemic in Portugal.
The findings highlight that the programme successfully increased employment probability and earnings for its participants, alongside a null effect on unemployment benefit claims.
The internship effects on employment vary across subgroups of the populations and benefits more lower-educated jobseekers and non-urban residents. Therefore, re‑orienting the internship towards these groups or implementing similar measures to specifically target these groups could help improve their employment outcomes and address existing labour market disparities in Portugal.
An important consideration is the evaluation’s relatively short-term horizon, which captures the programme’s effects up to nine months after internship conclusion. While the findings indicate positive immediate impacts on employment and earnings, the lasting nature of these effects remains to be assessed. Moreover, the particular economic context of the COVID‑19 pandemic, during which the programme was launched, complicates the extrapolation of these results to other circumstances. This period was marked by considerable labour market disruptions and policy responses, which may have influenced the outcomes of the programme in ways that are difficult to anticipate.
To enhance future evaluations and better understand the programme’s impacts, it is crucial to address data limitations. Having comprehensive data on the type of contracts (permanent, temporary) and working time would help better understand the effects on the quality of employment. Additionally, the lack of information on whether employment contracts are subsidised limits the ability to clearly determine the role of the “ATIVAR.PT Employment Award” subsidy in creating employment in the open labour market, without government supports. Furthermore, data on social and health outcomes would be important to further explore the potential broader impacts of internships beyond employment outcomes, offering a more comprehensive assessment of participants’ overall well-being. Finally, the results on unemployment benefits underscore the necessity of a comprehensive cost-benefit analysis. Such an analysis would weigh the government savings from reduced unemployment benefit claims and potentially other reduced expenditures (like social assistance or healthcare costs) against the total costs of the programme.
References
[1] Card, D., J. Kluve and A. Weber (2018), “What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations”, Journal of the European Economic Association, Vol. 16/3, pp. 894-931, https://doi.org/10.1093/JEEA/JVX028.
[2] Duarte, N. et al. (2020), The evaluation of the youth employment initiative in Portugal using counterfactual impact evaluation methods, EU Publications, https://doi.org/10.2760/368100.
← 1. The internship grant is treated as earnings and is subject to social security contributions payments. If an intern pays social security contributions for at least 180 calendar days, which is less than the typical length of most internships, they are entitled to at least five months of unemployment assistance benefit.
← 2. There are also no significant differences between participants aged under 24 and those aged 24 and over.