Impact Evaluation of Active Labour Market Policies in Portugal

1. Assessment and recommendations

Abstract

The Portuguese labour market situation has improved in recent years, but some challenges persist, including high youth unemployment. As part of the response to the COVID‑19 pandemic, Portugal implemented the ATIVAR.PT internship programme to facilitate the integration of young people into the labour market and to support the professional retraining of the unemployed through practical work experience. The evaluation of this programme shows significant improvements in the employment prospects and earnings of the participants, particularly for lower-educated jobseekers and non-urban residents who tend to participate less in the programme. Despite substantial public spending on active labour market policies (ALMPs), Portugal has not yet developed a systematic approach to their impact evaluation. Portugal should improve data coverage, quality and availability for assessment of ALMPs and invest more in evidence‑based policy making.

1.1. Portugal makes strong investments in ALMPs, but could do more to address labour market disparities and build evidence on what works

1.1.1. The Portuguese labour market situation has improved, but some disparities persist

The Portuguese labour market has seen significant improvements across a number of dimensions in recent years. After the prolonged impact of the global financial crisis, Portugal’s unemployment rate declined steadily, reaching 6.6% in Q2 2024, slightly above the OECD average of 5.1%. In addition, the employment rate (72.7% in Q2 2024) and labour force participation rate (77.7% in Q2 2024) have improved markedly over the past decade, exceeding the OECD average of 70.3% and 74.0% respectively. For Portuguese women, both the employment and participation rates exceed the OECD average (by 7 and 9 percentage points respectively in 2023).

During the COVID‑19 pandemic, the Portuguese labour market showed resilience, with the initial rise in the unemployment rate being lower than on average in the OECD. However, the crisis and its recovery were not experienced equally by all categories of workers. Young people, temporary workers and those with lower educational attainment faced a disproportionate share of the job losses and experienced a more prolonged impact. Government support programmes, including job retention schemes, played a crucial role in mitigating the effects of the crisis and safeguarding employment.

Despite recent improvements and the relative resilience shown during the pandemic, several labour market challenges still persist. These include large gender disparities across key labour market indicators, an ageing population, high youth unemployment, imbalances by type of employment contract and the relatively low educational attainment of the older population. Of particular concern is the heavy toll that the COVID‑19 pandemic has taken on young people (aged 15‑24), with Portugal’s youth unemployment rate (20.3% in 2023) still well above the EU and OECD averages (14.5% and 10.5% respectively). Portugal’s recent Decent Work Agenda, launched by the government in 2023, includes dedicated measures that aim to improve the working life of young people.

1.1.2. Spending on ALMPs is substantial, but a more systematic approach to their evaluation is needed

In mainland Portugal, the Portuguese Public Employment Service (PES) operates as an independent body under the supervision of the Ministry of Labour, Solidarity and Social Security. The PES is responsible for providing both employment services and vocational training, the latter often being organised in co‑operation with the social partners. The first point of contact for jobseekers is the employment centre, where their needs are assessed by counsellors. Jobseekers with training needs are then referred to vocational training centres, which often try to leverage links with employers to match training provision to skills demands and arrange on-the‑job training with employers through apprenticeships or internships. Some vocational training centres, run in co‑operation with the social partners, offer specialised training in dedicated sectors, focusing on digital skills or energy transition for example.

Spending on ALMPs in Portugal is somewhat higher than the OECD average, at 0.43% of GDP in 2022 compared to 0.40% across OECD countries. However, ALMP expenditure has fluctuated considerably over the last two decades and is currently below several previous peaks since the early 2000s. Within the ALMP basket, training (0.19% of GDP), employment incentives (0.11% of GDP) and PES and administration (0.09% of GDP) and are the three main categories of ALMP expenditure in Portugal. The ATIVAR.PT internship programme, which is evaluated in this report, falls under the category of training, an area in which Portugal’s expenditure significantly exceeds that of its OECD counterparts (0.10% of GDP in OECD countries in 2022). The category of employment incentives, on the other hand, includes some additional measures provided within the ATIVAR.PT initiative, such as the ATIVAR.PT Employment Award and the ATIVAR.PT Incentive, which are not evaluated in this report due to data availability.

Despite above‑average public expenditure on ALMPs and the availability of rich linkable administrative data, Portugal does not routinely or systematically evaluate their impact. Most studies of ALMPs in Portugal have focused on programmes financed through the European Social Fund Plus (ESF+), which are required to undergo monitoring and evaluation. These studies conducted under ESF+ are usually relatively broad and assess each programme’s efficiency, effectiveness, value added, sustainability and intended impact objectives. Some studies have used counterfactual methods to evaluate employment support measures, including internships that preceded the ATIVAR.PT programme, for example internships under the Youth Employment Initiative, employment and professional internships. The ATIVAR.PT programme as a whole was also co-financed by the ESF under the REACT-EU (Recovery assistance for cohesion and the territories of Europe), with the evaluation of the internship component conducted in this report representing the first evaluation of an ATIVAR.PT measure (see Chapter 3 for an overview of the different ATIVAR.PT measures). Building on these few examples, combined with the OECD analysis presented in this report, Portugal should take steps to establish a more systematic approach to counterfactual impact evaluation (CIE) of ALMPs in order to contribute to a better understanding of the effectiveness of programmes on different population groups.

1.2. The evaluation of the ATIVAR.PT internship is based on non-experimental methods and good quality administrative data

1.2.1. The internship is a sizeable programme in terms of the number of recipients and overall expenditure

The ATIVAR.PT internship is one of the measures included in the Reinforced Programme of Support for Employment and Professional Training implemented in Portugal in response to the COVID‑19 pandemic. This internship aims to facilitate the integration of young people into the job market and support the professional retraining of unemployed individuals through the development of practical experience in the work context. Interns receive a monthly grant according to their level of qualification and other related benefits co-financed by the PES. The grant is treated as earnings and is subject to taxes and social security contributions.

The ATIVAR.PT internship was launched in the second half of 2020. According to data provided by the PES, expenditure on ATIVAR.PT internships, was almost EUR 130 million in 2022, constituting 17% of total expenditure on ALMPs. The enrolment of around 2 000 new entrants per month quickly brought the number of individuals actively engaged in internships to nearly 20 000 in a month, covering up to 30‑40% of the young unemployed under the age of 25 by mid‑2022.

In addition to the internship, the broader ATIVAR.PT initiative also offers other measures, such as the ATIVAR.PT Employment Award and the ATIVAR.PT Incentive, which provide employment incentives to employers who hire former interns or individuals who face significant barriers in the labour market but did not benefit from PES-funded internships. These smaller-size measures had fewer recipients and lower expenditure and are not evaluated in this report.

1.2.2. Young people with higher education are over-represented among internship participants

While the internships are open to registered unemployed jobseekers of all ages, the programme primarily targets jobseekers under the age of 35 who can access it from day one of their registration with the PES. Jobseekers over 35 (over 30 from 2022) can only access the programme after being registered as unemployed for at least six months and meeting additional requirements related to their qualification levels. Internships usually last up to nine months but can be longer in special cases.

Reflecting the programme’s target demographic, participants are generally younger than other jobseekers. They are more likely to be single and less likely to have dependents, often searching for their first job and less likely to be eligible for unemployment benefits due to limited previous work experience and lower past earnings.

Furthermore, employers in new and innovative employment sectors tend to use internships to attract individuals with high qualification levels. As a result, a significant proportion of interns have higher education and engage in professional and technical occupations that require advanced skills. This means that interns are very different from other jobseekers, who tend to have lower levels of education and are much less likely to seek professional occupations.

1.2.3. Non-experimental methods seek to resolve the problem of comparison of internship participants and non-participants

The ATIVAR.PT internship was not designed to be evaluated using experimental methods such as randomised controlled trials (RCTs). As a result, participants and non-participants are not randomly selected. Instead, the internship is open to a wide range of potential participants. Its voluntary nature and the specific requirements of available internships imply that participants may differ from non-participants. This means that simply comparing the outcomes of participants with those of non-participants may reflect some of these inherent differences (e.g. education level), rather than the effect of the internship on labour market outcomes. As participation is not determined randomly, it is necessary to use non-experimental methods to analyse the impact of the programme. The implementation of the programme does not offer any exogenous variation across regions in Portugal or certain groups of jobseekers that could be exploited in the counterfactual impact evaluation.

The analysis in this report uses propensity score matching to select a control group of non-participants for comparison that is as similar as possible to the participants. This technique uses detailed information on individuals to determine which characteristics are important in explaining who participates in the programme and summarises them into a “propensity score”. Once participants and non-participants have been matched based on these scores, it is possible to compare their labour market outcomes and estimate the impact of the internship on labour market outcomes, removing as far as possible the bias introduced by non-random selection into the programme.

1.2.4. Good quality linked administrative data provide key information necessary for the evaluation but some important limitations exist

The evaluation in this report uses linked administrative data from the Public Employment Service (PES) and the Social Security Institute (ISS). The combined dataset includes detailed information on registered jobseekers, their unemployment spells, demographic characteristics, participation in the ATIVAR.PT internships, employment spells, monthly earnings, and receipt of unemployment benefits. These data provide key variables for assessing the impact of the internship participation on standard employment outcomes, such as employment, earnings, and unemployment benefits, while controlling for socio‑economic characteristics and other factors that are crucial to mitigate potential bias in the evaluation.

However, the data have some limitations. They lack information on subsidised employment, in particular on the receipt of the ATIVAR.PT Employment Award, which is granted to employers who hire former interns on permanent contracts, and other employment incentives, such as ATIVAR.PT Incentive, provided to employers who hire individuals not benefiting from PES-funded internships. This means that it is not possible to assess the extent to which participation in the internship leads to unsubsidised employment.

In addition, important information on the type of job contract held is missing for the majority of employees, and data on occupations in employment are not available. As a result, the scope of the analysis is limited to standard employment outcomes (employment, earnings, and unemployment benefits receipt), preventing a more in-depth assessment of job quality after participation in the programme. Furthermore, the dataset does not provide sufficient information on participation in other ALMPs, which could have served as valuable additional controls for unobservable differences in motivation between participants and non-participants prior to the internship, or as a complementary outcome variable.

1.3. The ATIVAR.PT internship enhances employment and earnings for young people

1.3.1. Participation in the internship leads to significant improvements in the likelihood of employment and earnings

The analysis in this report focuses on internships that last up to nine months. The results of the impact evaluation show that the ATIVAR.PT internships increase the probability of employment by 50 percentage points shortly after the start of the programme. This increase is expected, as the internships guarantee employment for the duration of the programme. However, quantifying this effect is crucial as it measures the difference between being employed through internship contracts and what would have happened to participants if they had not taken the internships. As participants transition out of the programme, they still see a 10-percentage point difference in employment rates relative to non-participants and this effect remains stable for at least nine months after the end of the internship. This effect is high compared to estimates from international studies of different types of ALMPs and is comparable in magnitude to other evaluations of internships in Portugal. However, due to data limitations, it is not possible to assess to what extent subsidies provided to employers (within the broader ATIVAR.PT programme or through other ALMPs) can explain this effect or if this effect reflects rather unsubsidised employment.

Similar patterns are observed when looking at the impact of the internships on earnings. The most substantial increase occurs during the programme, with participants earning on average around EUR 344 more per month, reflecting the immediate financial benefits of the internship grants. Over time, the increase in monthly earnings falls to about EUR 120‑180 as internships end and participants move into unemployment, inactivity or other forms of employment which may offer lower levels of compensation.

The immediate negative effect of the programme on the receipt of unemployment benefits becomes statistically insignificant as participants complete their internships. On the one hand, participants experience higher employment rates and earnings after the programme, thus reducing their benefit claims. On the other hand, those who return to unemployment after the end of the internship are potentially more likely to qualify for (higher) unemployment benefits, based on the accrued employment and earnings during the internship and any subsequent employment. These two effects cancel each other out. For those who have not been able to find a job, improved access to unemployment benefits acts as a crucial safety net, potentially leading to gains beyond financial stability, such as improved social inclusion or mental health.

1.3.2. Lower-educated jobseekers and non-urban residents see larger employment gains from participation in the internship

The impact of the internship on employment varies significantly by education level. Individuals with upper secondary education or less exhibit the most substantial increase in employment probability of 16.6 percentage points two years after unemployment registration (the end of the observation period), while those with tertiary education experience a smaller but still significant increase of 9.9 percentage points. The results suggest that while the more educated are more likely to participate in the programme, it is the less educated who benefit from participation the most. In addition, individuals living in non-urban areas enjoy a higher impact of participation in the internship on the probability of employment at the end of the observation period (13 percentage points) compared to those in urban areas (9.7 percentage points), although this difference is at the threshold of statistical significance. It would be beneficial to scale up and/or re‑direct the internship towards the groups that benefit from participation the most or develop measures similar to the ATIVAR.PT internships that specifically target lower-educated and non-urban residents to ensure that they can fully benefit from this type of employment support.

1.3.3. The findings of the assessment should be interpreted with caution

It should be noted that the evaluation carried out in this report is based on a relatively short period of time covering the data between mid‑2020 to mid‑2023. This makes it possible to capture only the short-term effects of the programme, nine months after the end of the internship. While the findings suggest positive immediate effects on employment and earnings, the sustainability of these effects in the medium and long term remains to be assessed. Data limitations also prevent assessing the extent to which participation in the internship leads to unsubsidised employment. Furthermore, the exceptional economic context of the COVID‑19 pandemic, during which the programme was implemented, makes it difficult to extrapolate the findings to other circumstances. This period was marked by considerable labour market disruptions and policy responses, which may also have influenced the outcomes of the programme.

Key policy recommendations

Target internships to groups more likely to benefit from them and assess further longer-term effects of internships beyond the context of the pandemic

• Assess the sustainability of the positive effects of the internship on employment and earnings in the medium and long term and beyond the specific economic context of the pandemic.

• Ensure internships are available for lower-educated jobseekers and those in non-urban areas, by encouraging their participation in existing programmes and, if needed, developing similar measures specifically targeting them to boost their employment prospects.

• Carry out a comprehensive cost-benefit analysis of internship programmes, which would weigh the savings to the government from reduced expenditure against the total costs of these programmes.

Improve data coverage, quality and availability for evaluation of ALMPs

• Address the limitations of the linked administrative data by enriching the information available on subsidised employment, type of contract, occupations and other employment characteristics that are required to capture the quality of jobs obtained after participation in ALMPs and assess the extent to which internship boost non-subsidised employment.

• Facilitate access to administrative data that cover a long period, both before and after participation in ALMPs, to capture longer-term effects of ALMPs and construct employment histories that can capture some unobserved differences between participants and non-participants before their participation in the programme.

• Conduct complementary surveys to capture additional important aspects, such as job flexibility and subjective job satisfaction, which are not covered by the administrative data.

• Collect linked data on counsellors and jobseekers, including detailed information on counsellors’ characteristics and the nature and frequency of their interactions with clients. The inclusion of such data could enable analysis of the important role of counsellors in shaping individuals’ outcomes in the labour market.

• Supplement existing linked administrative data with information on social benefits, firm-level data or health and justice data. The availability of such data would provide opportunities to explore the wider impact of ALMPs, for example on participants’ incomes, firms’ hiring patterns and broader social issues, such as social inclusion, material deprivation, well-being, health outcomes or involvement in criminal activities.

Improve infrastructure for accessing administrative data on ALMPs, their metadata and develop a system for ALMP impact evaluation

• Develop a structured and systematic way of accessing administrative data for evaluation of ALMPs through a dedicated framework to improve the consistency, reliability and transparency of the data, facilitate efficient data access, minimise errors and improve the overall quality of research outputs.

• Improve the metadata by providing detailed descriptions of variables, their values, interpretation, classifications used and reasons for missing values. Consideration should be given to producing standardised metadata for similar requests, which would reduce the time spent on data preparation in the future and reduce potential for errors and information gaps.

• Building on past experiences in Portugal and successful examples of other countries, build a systematic approach to conducting ALMPs impact evaluations using linked administrative data. Use the evidence generated to inform the design, targeting and implementation of ALMPs.

Strengthen the ALMP system by ensuring stable funding and closer links with labour market needs

• Aim to secure stable and continuous funding for ALMPs and prioritise it towards measures and services that have been evaluated and been found to be effective.

• Strengthen the links of ALMPs with labour market needs by promoting PES contacts with employers and better leverage social partners in decisions surrounding the ALMP basket composition.

• Implement guidelines for vocational training centres to better and more systematically co‑operate with local employers to design training courses that meet local skills demands.