Introduction Measures for Newly-Arrived Migrants

10. How can governments assess whether measures are tailored to meet the rationale for integration?

Why is this an issue?

Though increasing attention has been paid to the subject in recent years, methods to ensure evidence‑based decision making on integration remain underused. Almost all OECD and EU countries have a strategic framework that addresses integration of at least some key categories of migrants such as beneficiaries of international protection, but few perform systematic evaluation of their integration measures. The lack of evaluation of integration policies and strategies makes it difficult to assess whether new policies achieve their intended goals or merely sound good in theory.

Integration is often a major expenditure for host-countries, and in light of that expenditure, countries should have a vital interest to ensure that their methods, training, and assessment services are relevant and effective in delivering the intended outcomes and that they are continuously updated and improved. This is especially important where participation is obligatory, or where countries have decided to impose penalties for failure to reach a certain language threshold. If migrants are to spend this time away from the labour market, they should do so in a way that will be most beneficial to them in the long run. Evaluation enables countries to understand whether migrants are accessing the right integration measures for their needs, if they are in fact learning the host-country language, and whether the skills and knowledge gained are actually facilitating access to the labour market and helping them meet other integration goals. In the wider sense, systematic evaluations of integration measures can also be understood to constitute a duty towards taxpayers, wherever schemes are financed from the public purse. Evaluation is a necessary condition for effective results-based management and can help authorities avoid overlap and waste.

As integration is a process that occurs over time, improvements in outcomes may not necessarily be tied to policies. Changes will occur even in the absence of any policy. Participation in a programme designed to find jobs for unemployed immigrants may not, for example, but the sole reason that a migrant becomes employed. EU-wide, more than 1 in 3 of the low-educated non-EU migrants who were unemployed in 2018 had found a job one year later, whether or not they participated in an integration programme. For established programmes, where outcomes do fall short of objectives, an evaluation can also help policy makers understand why that is the case. If migrants are not completing the programmes as designed, it may be that the programme is not meeting migrants’ needs. On the contrary, it could be that their needs would be met with fewer services. It could also be that communication about the benefits of the programme is falling short, or that targets are set to unreasonable levels during the amount of time allocated. Without data, it is also difficult to argue for changes, especially as each change comes with an investment of both time and money.

The understanding of not only outcomes, but also of the “why” and “how” programmes are most effective, enables authorities to make tailored improvements that could deliver significant return on investment as the economic contributions of impacted migrants increases. A study of the decision to develop individual integration plans in Finland found that migrants were subsequently offered more language hours and obtained significant improvement in employment outcomes (Sarvimäki and Hämäläinen, 2016[64]). Evaluation can provide valuable lessons regarding what measures could increase attendance by certain hard-to-reach groups, such as women and the elderly, as well as how to decrease drop-outs. An additional benefit of evaluation is knowing which programmes do not work before significant further investment is made. In some cases, insights gained through evaluation have led to programming being discontinued. For example, in Denmark, a decision to reduce welfare benefits in tandem with offering expanded and improved early language classes to refugees was discarded when it showed the reduction in benefits had no positive labour market effects. (Arendt et al., 2020[12]).

How to approach it

Begin by asking what would have happened in the absence of a measure

Countries must consider the risk of self-selection, as outcomes attributed to a programme can reflect participants’ individual characteristics instead. This is because the migrants who decide – or who are picked – to participate in measures might be systematically different from those who do not. A migrant’s age, gender, or education can impact participation choices and outcomes. Where migrants can self-select, increased motivation to succeed may play a key role in improving outcomes. The desire to find stable employment as quickly as possible, for instance, can positively affect completion rates and learning satisfaction for migrants who chose a vocation course, as opposed to those who are assigned to one. For instance, migrants motivated to choose vocational courses to prepare for rapid employment can affect success indicators such as completion rates, satisfaction with learning and teaching methods.

A proper evaluation always needs a benchmark to assess what would have happened in the absence of the policy – the so-called “control group”. Ideally, this control group should not systematically differ from the “treatment group” – that is, migrants participating in the measure – by any other relevant characteristic than programme participation. This is achieved, for example, by random allocation of participants to measures. An alternative is by identifying otherwise similar groups that did not benefit (for example, in the case of certain cut-off points for programme participation or around the margins of a policy change, as Denmark has done recently (Arendt et al., 2020[12]).1 Canada and several Nordic countries have taken the approach of testing new integration policy instruments via pilots prior to implementation. The Swedish bonus system for successful language course participants, for example, was piloted in the framework of a randomised experiment and discontinued in 2014 after results indicated that the programme was only effective in metropolitan areas (OECD, 2014[65]).

By identifying a group of individuals who are similar ex ante to those participating in a programme, policy makers can compare the two groups for an unbiased assessment of the policy intervention. When finding an adequate control-group is not possible, longitudinal studies can rely on merging cross-sectional survey data with administrative records, or on following-up with particular groups. These solutions minimise respondent loss biases and alleviate the need for a counterfactual by allowing the monitoring of the same individuals over time, through data linkage, or of specific cohorts with common characteristics (e.g. refugees by age or gender). They can also reduce the financial costs of longitudinal studies. By leveraging aggregate‑level data, collected for different purposes, these methodologies reduce the reliance on ad-hoc data collection, which requires more targeted expenses.

Information about the knowledge profile and possible selection biases of learners can be collected before the beginning of a programme through a pre‑assessment. Such an assessment needs to be based on consistent standards that ensure migrants are placed in an appropriate level. Ideally, it will also evaluate learning capacity, using, for example, educational background or tests of structural perception and logical thinking (i.e. Finland’s Testipiste; (Tammelin-Laine et al., 2018[66]). Then, throughout the project evaluation, several check-in periods allow for measurement of medium-term effects. Analysing progress along the results chain allows the evaluator to understand what factors and institutional frameworks (e.g. course size, course duration, childcare availability, use of virtual classrooms) increase success and to what extent, for which sub-groups. It follows that evaluation cannot be an afterthought but has to be factored in from the outset of a project with its allocated budget.

Evaluate introduction measures and continuously monitor the results

Systematic evaluation can provide policy makers with insights and evidence needed to optimise the development of integration programmes. Ideally, integration programmes incorporate a systematic and in-build element of evaluation from the very start (Box 10.1). This allows the programme designers to test the validity of assumptions along the programme chain. Scotland’s ‘Migration Policy Toolkit’ stresses that effective policy implementation and evaluation should also be based on clear intended outcomes from the very beginning, and the involved actors must share the agreed-upon objectives. All relevant stakeholders, from financing (national or local authorities), to implementation (language schools), to consumer (migrants) should be included in the process. National action plans have increasingly acknowledged the need for an in-build monitoring of integration policies using qualitative and quantitative data collection. Lithuania’s ‘Action Plan 2018‑2021’ adopted this approach and acknowledged the need to strengthen inter-institutional co‑operation between NGOs, public and local authorities. It also identified specific ministries, the Association of Local Authorities, Caritas of the Archdiocese of Vilnius, and the Lithuanian Red Cross Society (Minister of Social Security and Labour, 2018[67]). Reform of Lithuania’s strategic management procedures is currently underway, and measures provided for in the Action Plan will be evaluated along with measures financed by the AMIF national programme in 2022.

Box 10.1. Evaluation of New Initiatives, the Norwegian Integration Reform

In Norway, all new public policy should be subject to evaluation, both as means of learning, management and control. Evaluations may not merely be a post-assessment of a new legislation or reforms a given number of years after its introduction, but are also intended as a tool for the responsible units (i.e. ministries/national agencies) to receive continuous feedback to make necessary adjustments in the initial implementation phase. When Norway implemented a new Integration reform from 2021, several actions were taken to evaluate the implementation and effects of the new reform, for example the development of a new set of quantitative indicators of the reforms input, output and outcomes, and evaluations of the new Integration Act and other new reform initiatives. Common traits for (most of) these evaluations are that they: 1) are initiated from day one of the implementation, making it possible to follow the implementation ‘in real time’, 2) include a baseline of the situation before the implementation of the new policy for comparison, and 3) are conducted by independent actors (research institutes, consultancy firms, etc.) through open tenders. One example is an evaluation of a new policy initiative: an online resource for relevant public actors that are responsible for planning and implementing introductory programmes for newly arrived immigrants. The evaluation is conducted by an independent research foundation, FAFO, and paid for by the national Directorate of Integration and Diversity. The evaluation’s aim is to provide answers to whether the introduction of the online resource works as expected, and to provide a basis for continuous improvements, at both the national and the municipal level. The evaluation will follow the usage and implementation of the new resource over a four‑year period, with a budget of NOK 1.5 million a year (total budget of NOK 6 million).

Rigorous evaluation may have profound implications for policy makers in determining how much training is necessary, how much flexibility to introduce, and how to identify and improve government-sanctioned course offerings. For example, a recent evaluation carried out for the Estonian Government identified significant unmet demand for language training and made specific recommendations regarding funding to improve the ability to hire enough quality teachers to meet that demand (Estonian Centre for Applied Research (CENTAR) and Tallinn University, 2018[68]). Evaluators made the case for increasing flexible options by evaluating unemployment insurance fund data and tax data to examine two optional tracks offered by the Estonian Government against a control group of those who chose not to take a course, using a matching method. They found that a shorter “training card” course purchased on the market had a shorter lock-in effect and smaller dropout rate (13% v. 25%) than longer courses offered by the Unemployment Insurance Fund. Migrants who took a course had better results finding employment than those who did not, regardless of which course, though the positions were not necessarily higher paying (Kivi, Sõmer and Kallaste, 2020[69]). Estonia systematically studies its integration programme through the Estonian Integration Monitoring, which has been carried out eight times over the last two decades.2

Canada, Australia, France, Germany, the United Kingdom and Ireland are among the OECD countries that have undertaken evaluations of their programmes or strategies. Canada regularly requires evaluations to examine programme relevance, management, and impact of its Settlement Program (Immigration, Refugees and Citizenship Canada, Research and Evaluation Branch, 2017[70]). Acting on gaps identified by this evaluation, Canada has also performed a full evaluation of its language training (Box 10.2). Australia (AMEP Longitudinal Survey, see (Yates and Wang, 2015[71]), France (ELIPA 2, ongoing since 2019), and Germany (Evaluation of the Integration Courses, see (Tissot et al., 2019[72])) have launched longitudinal panels, in some cases comparing participants and non-participants. The United Kingdom’s ‘Inspection of the Vulnerable Persons Resettlement Scheme’ has been used to evaluate the efficiency and effectiveness of the 2015 Syrian refugees resettlement programme in terms of risks and success factors of the policy as well as the cultural integration programmes delivered by local authorities (Bolt, 2018[73]). Still, particular attention should be paid to testing the design and organisation of the programmes in addition to its value added. To this end, the Nordic countries have taken the approach of testing the efficiency of new integration policy instruments via pilots prior to implementation. Ireland’s ‘Monitoring report on Integration 2020’ was set to investigate four areas of integration – employment, education, social inclusion and active citizenship – in a European comparative perspective through the use of commonly agreed indicators (ESRI et al., 2020[74]). The report found that non-Irish individuals report to be in better health but present higher risk of poverty, lower incomes and home ownership rates compared to Irish nationals.

Box 10.2. An Evaluation of Canada’s Language Training Services

The quality of evaluation exercises depends strongly on the variety of data collected. Canada’s 2020 assessment of its Language Training Services aimed at understanding the specific success factors of language programmes for newcomers’ language skills progression by leveraging a range of quantitative and qualitative data from all relevant stakeholders, i.e. ministries, service providers and beneficiaries (Immigration, Refugees, and Citizenship Canada, Research and Evaluation Branch, 2020[75]).

Notably in this case, data collection aimed at obtaining comprehensive information on service provision, sample characteristics, and programming outcomes. Data on service provision was based on stakeholder interviews (such as IRCC programme staff, senior management, and service provider organisation) and contribution from an independent expert group through on-field assessments, panel groups, and document review. It introduced surveys to language training instructors, which helped gather information regarding the quality of assessments, appropriateness of the expected outcomes, and the resources provided. Rigorous sample profiling took place through mixed-mode surveys (of clients and a non-client control group) combined with administrative data. The measurement of progression was done by standardised tests (administered ex-ante and ex-post), the results of which were combined with sample profiles to distinguish the effects of different types of services. These were measured further through specific case study analysis of targeted groups such as refugees, employment-related beneficiaries, or women.

This two‑year data collection strategy allowed Canada to shed light on the ‘what’ and ‘why’ aspects of designing and implementing policies that are most effective, but most importantly, it showed some weaknesses of the national processes in these areas, notably the clarity of employment data collection and cost-effectiveness of the selected standardised tests. Following the analysis, evaluators provided ten different recommendations, which the ministry is integrating in its future policy adjustments.

Correctly attributing migrants’ results to programme outcomes is essential for improving existing policies and design better ones. To assess causality, studies should aim to find a counterfactual – a statistically comparable group not subject to the policy under evaluation. The presence of this control group allows measurement of a programme’s effectiveness after its implementation. Yet, this sort of evaluation can be difficult to achieve when working with migrants. Measuring the impact of language training, for instance, requires an adequate comparison group that did not participate in the training. This, however, is complicated in cases where participation is expected to be near universal for certain groups. For other policies, the limited number of beneficiaries undermines statistical reliability, and there is a risk the same group of people will be unavailable to re‑interview over time. It is also important to note that migrants – especially new arrivals – will have opportunities to integrate in the absence of formal training, depending on their exposure to host-country natives and other factors.

Measure integration measures against all objectives, not just those easily measured

Integration is a complex process that encompasses different aspects of social life. Once in the receiving country, immigrants have to integrate in the labour market, adapt to new institutions and practices, understand social norms, and create links with other groups and individuals in the new society. However, integration is also a significant undertaking for host communities. They are called to create programmes addressing legal and economic aspects of integration, but also to encourage the social, political and cultural involvement of newcomers and their children. Evaluations do not often reflect this multidimensionality – focusing on individuals rather than receiving communities, and prioritising a restricted number of policy instruments, mostly related to the outcomes of economic integration programmes. Despite the importance of economic self-sufficiency, policy makers also need to evaluate the social aspects of integration to foster a full and equal membership in host societies. For instance, Canadian evaluations of the national ‘Settlement and Integration Program’ include indicators such as volunteering rates, trust in public institutions, sense of belonging to the country (and province), and how welcoming are local communities perceived to be. Measures also consider the ability of communities to adjust their practices to welcome and integrate newcomers, as well as their retention capacity.

It is important that the instruments chosen to carry out an evaluation are capable of tracking progress and highlight areas of intervention. For instance, satisfaction surveys can help understanding the aspects of a policy that work (and those that do not), showing the channels through which programmes affect beneficiaries. With respect to integration policies, for example, they can show whether participants of vocational training programmes were satisfied with the learning methods and what challenges they encountered. However, findings coming from these surveys cannot be generalised due to their programme‑specific nature, and whether an individual is happy may tell us little about their actual progress. Thus, while informative, satisfaction surveys should be considered within a broader evaluation framework that assesses both tangible skills acquisition and the relevance of those skills to labour market demand. A system of comprehensive indicators is a fundamental component of such framework. To avoid becoming unduly burdensome, such indicators should be targeted and precise – something that is only possible where the evaluation has explicit and specific objectives. Once the objectives are established, indicators can be set following the SMART criteria – specific, measurable, actionable, realistic and time‑bound. Asking whether the indicators put in place answer those different questions makes the monitoring and evaluation of integration policies sensitive to the needs of both institutions and beneficiaries, as well as optimising resources. In assessing its Integration Act introduced in 2021, the Norwegian Government clearly stated that the evaluation’s objective was to measure whether the law was working as intended, rather than producing a management tool for public actors. Given the law’s focus on economic independence and early integration in Norwegian society, the indicators suggested for the evaluation were focused on the share of part-time and full-time workers, formal qualifications obtained in the country and language skills (Hernes, Staver and Tønnessen, 2020[76]).

Besides being targeted and well defined, indicators should be set across different policy areas. New Zealand has developed a framework consisting of outcome indicators ranging across five different areas – employment, education and training, English language, inclusion, and health and well-being. Frameworks based on interconnected indicators give a more complete overview of the process of integration as a whole. This translates into richer data for policy makers to recognise potential areas of intervention.

Once appropriate indicators are identified, countries can use benchmarking exercises to assess whether the policies are advancing migrants appropriately. It is essential to develop the right benchmarks based on the right project goals. For example, completion rates of training courses are a convenient benchmark for evaluation of courses, but to understand whether a programme is successful, evaluators should question how training programmes could be measured beyond this metric, in particular taking into account labour market conditions and specific orientation of courses, as well as the profile of the students. Benchmarks also can help countries evaluate their progress relative to similar countries. The NIEM – National Integration Evaluation Mechanism – is organised around this rationale and aims at evaluating factors influencing the successfulness of refugees’ integration policies on three main levels: the general legal and policy framework, the efficiency of implementation, and degree of collaboration between relevant parties. The tool, used in several EU countries, includes housing indicators that measure access to property rights, representation, the provision of targeted housing advice, counselling, and partnership on housing with expert NGOs (Wolffhardt, Conte and Huddleston, 2019[77]).

A full evaluation should also collect and process the information needed to identify which aspects of the programme are working well, rather than simply tracking markers of success. Evaluators should identify what migrants are learning and putting to use, then use that information to determine what aspects of integration might need more attention. Progress of the learner, mentoring, and dissemination of appropriate advice may be more important than outright achievement. Determining whether course participants are learning may require multiple methods, including satisfaction surveys, self-assessments, completion certificates, and portfolios. Evaluating the quality of courses could also involve monitoring service‑provider performance and accreditation.

Examine short-term versus long-term effects of integration measures

A country seeking to evaluate its integration measures must start by acknowledging that integration is a long-term process and that therefore, a long temporal horizon has to be incorporated in the structure of any measurement exercise. Evaluations should take place over time and be repeated consistently. Ideally, migrants should be evaluated at (or prior to) arrival in the host community. An early evaluation enables governments to map current skills as well as potential capabilities and difficulties. This information is relevant for designing needs-based language and training classes. Frequent quantitative and qualitative longitudinal studies on programme beneficiaries have the additional advantage of examining specific outcome metrics more closely. For instance, the United States conducts the Annual Survey of Refugees (ASR), a study that follows refugees in the five years after their arrival and serves as a registry of data for monitoring how they are adapting to their new lives in the host country. Conducted as a longitudinal study until 2016, the ASR monitored the progress of refugee families in terms of English proficiency, workforce participation, formal education, and permanent residence in order to better allocate resources for their support (Office of Refugee Resettlement, 2018[78]).

If improved labour-market performance is the goal, important questions also include whether migrants are likely to become self-sustaining enough to leave social or unemployment benefits or likely to take up a more highly qualified job, including in the longer term. In this context, it is interesting to assess whether vocational courses provide a greater chance of success in comparison to general language courses with the same target level. When feasible, checking in with migrants after the completion of the course to obtain information about labour market outcomes would better enable longer-term assessments of the added-value of programmes, for example regarding whether participants are not only able to find, but to sustain, employment.

Beyond tracking programmes’ effectiveness, these data collection practices enable the study of intergenerational differences in terms of integration outcomes. Indeed, the integration challenges of native‑born children of immigrants – which are different from those faced by their parents – also differ across hosting societies and according to the origin country of their parents. In 2016, Norway launched its ‘Children of Immigrants Longitudinal Study’ (CILS) project, to run through 2025, with the goal of investigating the intergenerational integration of immigrant children in terms of social, economic, cultural and educational outcomes through survey and administrative data. The CILS design, which has been implemented in Belgium, Germany, the Netherlands, Spain, Sweden, the United Kingdom, and the United States, is a comprehensive, standardised method for studying the integration outcomes of children of migrants.

Notes

← 1.  In Denmark, a decision to reduce welfare benefits in tandem with offering expanded and improved early language classes to refugees was discarded when it showed the reduction in benefits had no positive labour market effects. The study did find that increased course hours and quality improvements, notably through a focus on teacher training, yielded long-term benefits in spite of a significant lock-in effect.

← 2.  https://www.kul.ee/en/estonian-integration-monitoring-2020. Full report is available only in Estonian.