To date, language programmes across the OECD are rarely scientifically evaluated. This finding is somewhat surprising given that state‑funded language training represents the bulk of public integration expenditures. In light of such major investments, countries should have a vital interest to ensure that their methods, training, and assessment services are relevant and effective in delivering the intended outcomes and that they are continuously updated and improved. The quality of the programme is especially important where participation is obligatory, or where countries have decided to impose penalties for failure to reach a certain language threshold. If migrants are to spend this time away from the labour market, they should do so in a way that will be most beneficial to them in the long run. Rigorously evaluating the impact of language training on labour market integration is also a necessary step to identify gaps and improve the effectiveness of available training options. The understanding of not only outcomes, but also of the “why” and “how” programmes are most effective, enables authorities to make tailored improvements that could deliver significant return on investment as the economic contributions of impacted migrants increases. It can provide valuable lessons regarding what measures could increase attendance by certain hard-to-reach groups, such as women and the elderly, as well as how to decrease drop-outs.
Language Training for Adult Migrants
12. Evaluate the impact of language training and act on the results
WHAT and WHY?
WHO?
Evaluating the impact of language training on language acquisition and on the labour market integration of participants should guide authorities in charge of organising and funding language programmes to make informed choices when choosing a course‑provider, the overall methodology and incentives for both beneficiaries and course providers. A demonstrated impact on labour market integration is also a major driver of immigrants’ motivation to participate in language training in the first place (see Lesson 2). In the wider sense, systematic evaluations of language training can be understood to constitute a duty towards taxpayers, wherever language schemes are financed from the public purse. Evaluation is a necessary condition for effective results-based management and can help authorities avoid overlap and waste.
HOW?
Evaluation enables countries to understand both whether migrants are learning the host-country language and whether higher language proficiency is actually facilitating access to the labour market. Ideally, language programmes incorporate a systematic and in-build element of evaluation from the very start. This allows the programme designers to test the validity of assumptions along the programme chain. To do this, the actor responsible for financing the programme must engage independent, external survey design experts who are asking the right questions. All relevant stakeholders, from financing (national or local authorities), to implementation (language schools), to consumer (migrants) should be included in the process.
To conduct an effective evaluation, a mechanism for baseline assessment is necessary. Information about the knowledge profile and any selection bias (i.e. motivation to choose vocational courses to rapidly enter employment) of learners can be collected before the beginning of the language course through a pre‑assessment. Such an assessment should be based on consistent standards that ensure migrants are placed in an appropriate level. Ideally, it will also evaluate learning capacity, using, for example, educational background or tests of structural perception and logical thinking (i.e. Finland’s Testipiste; Tammelin-Laine, et al. [2018]). Then, throughout the project evaluation, several check-in periods allow for measurement of medium-term effects. Analysing progress along the results chain allows the evaluator to understand what factors and institutional frameworks (e.g. course size, course duration, child care availability, use of virtual classrooms) increase success and to what extent, for which sub-groups.
To measure the effect of language training, it is important to recognise that migrants – especially new arrivals – will also enhance their proficiency in the host-country language in the absence of formal training. At the same time, especially where courses are lengthy, there is also a cost for migrants involved, as the time spent on language training is not available for job-search. Measuring the impact of language training thus requires an adequate comparison group that did not participate in the training. This, however, is often difficult, especially in cases where participation is expected to be near-universal for certain groups. Self-selection of language training participants is also an issue that a proper evaluation needs to address.
It is also essential to develop the right benchmarks based on the project goals. Language level targets (measured by successful exam results) are a convenient benchmark for evaluation of courses, but they may tell us little about effectiveness of the programme from a labour-market integration standpoint. Evaluators should question how the success for a language course can be measured beyond examination results, in particular taking into account labour market conditions and the vocational orientation of courses, as well as the profile of the students. Progress of the learner may be more important than outright achievement. Determining whether course participants are learning may require multiple methods, including satisfaction surveys, self-assessments, completion certificates, and portfolios. Evaluating the quality of courses could also involve monitoring service‑provider performance and accreditation.
If improved labour-market performance is the goal, important questions also include whether migrants are likely to become self-sustaining enough to leave social or unemployment benefits or likely to take up a more highly qualified job, including in the longer term. In this context, it is interesting to assess whether vocational courses provide a greater chance of success in comparison to general language courses with the same target level. If it is feasible, checking in with migrants after the completion of the course to obtain information about labour market outcomes would better enable longer-term assessments of the viability of the programme, for example regarding whether participants are not only able to find, but to sustain, employment.
Embedding of evaluation design into language programmes has been rare in OECD countries to date. A proper impact evaluation is expensive and requires that part of the budget is reserved for that purpose. Data constraints are often a further obstacle, particularly given privacy law issues in some countries, which may prohibit the collection of biographical information that may impact learning outcomes. Insufficient participant numbers in smaller-scale programmes also pose a challenge. Rigorous evaluations require the selection of an adequate control-group to compare outcomes (usually attendance rates and test scores) before, during, and after the programme. Limited evaluation periods and the short duration of individual course types make it difficult to evaluate long-term successes. Wherever it is not possible to identify an adequate counterfactual group prior to programme implementation, comparing the outcomes of different cohorts of migrants is an alternative, which is often used to assess the impact of introduction programmes (Liebig and Huddleston, 2014). Good evaluations are almost invariably mixed method evaluations that use qualitative information to provide context in both design and interpretation.
Canada, Australia, France, Germany, and the United Kingdom are among the OECD countries, which have undertaken robust scientific evaluations of their language programmes. Canada, for instance, regularly requires evaluations to examine programme relevance, management, and impact of its Settlement Program (e.g. Immigration, Refugees and Citizenship Canada, Evaluation Division, 2017). Australia (AMEP Longitudinal Survey, see Yates, et al., 2015), France (ELIPA, see Bouvier, 2013) and Germany (Integration Panel, see Schuller, et al., 2011; Evaluation of the Integration Courses, see Tissot, et al., 2019) have launched longitudinal panels, in some cases comparing participants and non-participants. The United Kingdom’s ‘Equality Impact Assessment’ has been used to evaluate the access to ESOL courses and their impact of changes for men and women and for different age, ethnic, and vulnerable groups (Department of Business, Innovation, and Skills, 2011). Still, particular attention should be paid to testing the design and organisation of the programmes in addition to its value added. To this end, the Nordic countries have taken the approach of testing the efficiency of new integration policy instruments via pilots prior to implementation. The Swedish bonus system for successful language course participants, for example, was piloted in the framework of a randomised experiment and discontinued after results indicated that the programme was only effective in metropolitan areas (Aslund and Engdahl, 2012).
Rigorous evaluation may have profound implications for policy makers in determining how much language training is necessary, how much flexibility to introduce, and how to identify and improve government-sanctioned course offerings. For example, a recent evaluation carried out for the Estonian Government identified significant unmet demand for language training and made specific recommendations regarding funding to improve the ability to hire enough quality teachers to meet that demand (Centar, 2018). Evaluators made the case for increasing flexible options by evaluating unemployment insurance fund data and tax data to examine two optional tracks offered by the Estonian Government against a control group of those who chose not to take a course, using a matching method. They found that a shorter “training card” course purchased on the market had a shorter lock-in effect and smaller dropout rate (13% v. 25%) than longer courses offered by the Unemployment Insurance Fund. Migrants who took a course had better results finding employment than those who did not, regardless of which course, though the positions were not necessarily higher paying (Kivi, et al., 2020).
An additional benefit of evaluation is knowing which programmes do not work before significant investment is made. In 2017, Canada’s IRCC launched a Service Delivery Improvement initiative, investing nearly CAD 150M over five years and over CAD 35M on an ongoing basis to test innovative projects for improved effectiveness, subject to evidence‑based monitoring over a lifecycle of 1‑3 years.1 The goal was not only to determine what works, but also what does not work and why. In some cases, insights gained through evaluation have led to programming being discontinued. For example, in Denmark, a decision to reduce welfare benefits in tandem with offering expanded and improved early language classes to refugees was discarded when it showed the reduction in benefits had no positive labour market effects. The study did find that increased course hours and quality improvements, notably through a focus on teacher training, yielded long-term benefits in spite of a significant lock-in effect (Arendt, et al., 2020).2
References
Abrams, S. and Gerber, H. (2013), “Achieving through the Feedback Loop: Videogames, Authentic Assessment, and Meaningful Learning,” The English Journal, Vol. 103(1), pp. 95‑103.
Ankestyrelsen (2020), Fravær fra danskuddannelse for flygtninge og familiesammenførte, National Board of Appeal of Denmark, Copenhagen, Denmark.
Arendt, J.N. et al. (2020), “Integrating Refugees: Language Training or Work-First Incentives?” NBER Working Paper No. 26834, National Bureau of Economic Research, Cambridge, MA, the United States.
Aslund, O. and Engdahl, M. (2012), “The Value of Earning for Learning: Performance bonuses in immigrant language training,” IZA Discussion Paper No. 7118, IZA, Bonn, Germany.
Beacco, J.C. et al. (2014), “Linguistic integration of adult migrants: Guide to policy development and implementation,” Council of Europe, Strasbourg, France.
Bouvier, G. (2013), “L’impact des cours de français pour les nouveaux migrants,” Infos migrations, n. 55, Ministry of the Interior, Paris, France.
Chenven, L. (2004), “Getting to Work: A Report on How Workers with Limited English Skills Can Prepare for Good Jobs,” AFL-CIO Working for America Institute, Washington, the United States.
Chiswick, B. and P. Miller (2014), “International Migration and the Economics of Language,” IZA Discussion Paper No. 7880, IZA, Bonn, Germany.
Clausen, J. et al. (2009), “The Effect of Integration Policies on the Time until Regular Employment of Newly Arrived Immigrants: Evidence from Denmark,” Labour Economics, Vol. 16, pp. 409‑417.
Codagnone, C. and Kluzer, S. (2011), “ICT for the social and economic integration of migrants into Europe,” JRC Scientific and Technical Report, EUR 24 719 EN, Seville, Spain.
Cooke, M. and Simpson, J., (2008) ESOL: A Critical Guide, Oxford University Press.
Damas de Matos, A. and T. Liebig (2014), “The qualifications of immigrants and their value in the labour market: A comparison of Europe and the United States”, in OECD and European Union, Matching Economic Migration with Labour Market Needs, OECD Publishing, Paris. DOI: http://dx.doi.org/10.1787/9789264216501‑9‑en
Delander, L. et al. (2005), “Integration of Immigrants: The Role of language proficiency and experience”, Evaluation Review, Vol. 29, No. 1, pp. 24‑41.
Department for Justice and Equality (2016), “The Migrant Integration Strategy: A Blueprint for the Future,” Government of Ireland, Dublin, Ireland.
Department for Business, Innovation, and Skills (2011), “English for Speakers of Other Languages (ESOL): Equality impact assessment,” UK Government, London, UK.
Estonian Centre for Applied Research (CENTAR) and Tallinn University (2018), Estonian Language Training for Adults with Other Native Languages as Part of Estonian Integration and Employment Policy: Quality, Impact and Organisation, Tallinn, Estonia.
European Commission (2016), Mapping of good practices relating to social inclusion of migrants through sport: Final report to the DG Education and Culture of the European Commission, Brussels.
Fafo (2020), “’We have kept the wheels in motion.’ The municipalities’ integration work with refugees during the outbreak of the coronavirus,” Fafo-rapport 2020:16.
Federal Ministry for Europe, Integration, and Foreign Affairs (2016), Integration Report, Expert Council on Integration, Vienna, Austria, https://www.bmeia.gv.at/fileadmin/user_upload/Zentrale/Integration/Integrationsbericht_2016/Integration_Report_2016_EN_WEB.pdf.
Freire, P. (1994), Pedagogy of hope: reliving pedagogy of the oppressed, Continuum, New York.
Friedenberg, J. et al. (2014), Effective Practices in Workplace Language Training, TESOL Publications, Alexandra, the United States.
Galvão, I. and S. Cabrita (2020), Notebook of Theatrical Practices for the Learning of the Language, Conselho Português para os Refugiados, Lisbon, Portugal.
Gortz, M., Heinesen, E., Husted, L. and Hald, S. (2006), “Benchmarking Analysis of Danish Municipalities’ Integration Policies in the period 1999‑2005,” Institute for Local Government Studies, Copenhagen, Denmark.
Hartikainen, A., Ahola, M., Apiola, M. and Sutinen, E. (2020), “The immigrant integration online training programme in Finland.” (forthcoming).
Hennessy, S., Harrison, D. and Wamakote, L, (2010), “Teacher factors influencing classroom use of ICT in Sub-Saharan Africa,” Itupale online journal of African studies, Vol. 2(1), pp. 39‑54.
Immigration, Refugees and Citizenship Canada, Evaluation Division (2017), Evaluation of Settlement Program, Government of Canada, Ottawa, Canada.
Isphording, I. (2013), “Disadvantages of Linguistic Origin: Evidence from Immigrant Literacy Scores”, Ruhr Economic Papers No. 397, Bochum/Dortmund/Duisburg/Essen, Germany.
Kivi, L., Somer, M. and Kallaste, E. (2020), “Language Training for Unemployed Non-natives: Who Benefits the Most?” Baltic Journal of Economics, Vol. 20, No. 1, pp. 34‑58.
Kluzer, S., Ferrari, A. and Centeno, C. (2011), “Language Learning by Adult Migrants: Policy challenges and ICT responses,” Joint Research Centre Institute for Prospective Technological Studies, Seville, Spain.
Korte, W.B, and Husing, T. (2007), Benchmarking access and use of ICT in European schools 2006: Results from Head Teacher and A Classroom Teacher Surveys in 27 European countries. e‑L earning Papers, 2(1), 1‑6.
Kozar, O. and Yates, L. (2019), “Factors in Language Learning after 40: Insights from a Longitudinal Study,” International Review of Applied Linguistics in Language Teaching, 57(2), 181‑204.
Krumm, H. J., and Plutzar, V. (2008), Tailoring language provision and requirements to the needs and capacities of adult migrants, Retrieved from LIAM. https://rm.coe.int/CoERMPublicCommonSearchServices/DisplayDCTMContent.
Liebig, T. and Huddleston, T. (2014), “Labour market integration of immigrants and their children: Developing, activating and using skills”, in OECD, International Migration Outlook 2014, OECD Publishing, Paris. DOI: http://dx.doi.org/10.1787/migr_outlook‑2014‑5‑en.
Liebig, T and Tronstad, T. (2018). “Triple disadvantage? A first overview of the integration of refugee women,” OECD Social, Employment, and Migration Working Papers, (216), pp. 1‑37.
Lochmann, A., Rapoport, H., Speciale, B. (2019), “The Effect of Language Training on Immigrants’ Economic Integration Empirical Evidence from France,” European Economic Review, Vol. 113(C), pp. 265‑296.
Loewen, S., Isbell, D., Sporn, Z. (2020), “The Effectiveness of App-based Language Instruction for Developing Receptive Linguistic Knowledge and Oral Communicative Ability,” Foreign Language Annals, Vol. 53, pp. 209‑233.
Mayer, R. E. (2019), “Thirty years of research on online learning.” Applied Cognitive Psychology, Vol. 33(2), pp. 152‑159.
Ministry of Aliyah and Integration (2019), Guide to Ulpan Study, Tenth Ed.
New Zealand Government (2019), New Zealand Migrant Settlement and Integration Strategy: Outcomes Indicators 2018, Supplementary Report, New Zealand Immigration.
Nieuwboer, C., and van’t Rood, R. (2016), “Learning language that matters: A pedagogical method to support migrant mothers without formal education experience in their social integration in Western countries,” International Journal of Intercultural Relations, Vol. 51, 29‑40.
OECD (2007), Jobs for Immigrants (Vol. 1): Labour Market Integration in Australia, Denmark, Germany and Sweden, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264033603‑en.
OECD (2014), Jobs for Immigrants (Vol. 4): Labour Market Integration in Italy, OECD Publishing, Paris, https://doi.org/10.1787/9789264214712-en.
OECD (2016), OECD Review of Policies to Improve the Effectiveness of Resource use in Schools, School Resources Review, Country Background Report: Sweden, http://www.oecd.org/education/school/CBR_OECD_SRR_SE‑FINAL.pdf.
OECD (2017a), Making Integration Work: Family Migrants, Making Integration Work, OECD Publishing, Paris, https://doi.org/10.1787/9789264279520‑en.
OECD (2017b), Finding their Way: Labour Market Integration of Refugees in Germany,” OECD Publishing, Paris, https://www.oecd.org/els/mig/Finding-their-Way-Germany.pdf.
OECD (2018), International Migration Outlook 2018, OECD Publishing, Paris, https://doi.org/10.1787/migr_outlook‑2018‑en.
OECD (2019), International Migration Outlook 2019, OECD Publishing, Paris, https://doi.org/10.1787/c3e35eec-en.
OECD/European Union (2018), Settling In 2018: Indicators of Immigrant Integration, OECD Publishing, Paris/European Union, Brussels, https://doi.org/10.1787/9789264307216-en.
Ofsted (2008), “ESOL in the post-compulsory learning and skills sector: an evaluation,” Ofsted, London, the United Kingdom.
Pöyhönen, S., and Tarnanen, M. (2015), Integration policies and adult second language learning in Finland. Adult language education and migration: Challenging agendas in policy and practice, Routledge.
Ramboll (2007a), Evaluation of the Nationwide Integration Courses, Federal Ministry of the Interior, Berlin, Germany.
Ramboll (2007b), Med Moduler Som Motor: Evaluering af implementeringen af danskuddannelsesloven, Copenhagen, Denmark.
Ramboll (2017), Evaluering af danskuddannelsesreformen 2017, Copenhagen, Denmark.
Ramboll (2020), Evaluerig af puljen til virksomhedsorganiseret dansundervisning, Copenhagen, Denmark.
Rideout, V., and Katz, V. S. (2016). Opportunity for All? Technology and Learning in Lower-Income Families, In Joan Ganz Cooney Center at Sesame Workshop. Joan Ganz Cooney Center at Sesame Workshop, New York, NY.
Roberts, C. (2003), “English to Speakers of Other Languages (ESOL) in the workplace: review of research and related literature”, NRDC, London, the United Kingdom.
Rossner, R. (2014a), “Providers of courses for adult migrants: Self-assessment handbook,” Council of Europe, Strasbourg, France.
Rossner, R. (2014b), “Quality Assurance in the provision of language education and training for adult migrants: Guidelines and options,” Council of Europe, Strasbourg, France.
Sarvimäki, M. and Hämäläinen, K. (2012), “Assimilating Immigrants: the Impact of an integration programme”, Discussion Paper 306, Helsinki Center of Economic Research, Helsinki, Finland.
Sarvimäki, M. and Hämäläinen, K. (2016), “Integrating Immigrants: The Impact of Restructuring Active Labor Market Programmes,” Journal of Labor Economics, Vol 34, Issue 2, pp 479‑508.
Schuller, K. et al. (2011), “Integration Panel: Results of a longitudinal study on the effectiveness and sustainability of integration courses,” Federal Office for Migration and Refugees, Nürnberg, Germany.
Shadiev, R., Hwang, W. Y., and Liu, T. Y. (2018), “Investigating the effectiveness of a learning activity supported by a mobile multimedia learning system to enhance autonomous EFL learning in authentic contexts,” Educational Technology Research and Development, 66(4), 893‑912.
Significant (2010), Evaluatierapport Inburgering in Nederland, The Hague, The Netherlands.
Social Compass (2019), Evaluation of the Adult Migration English Program New Business Model, Commissioned by Department of Education and Training for the Department of Home Affairs, Australia.
Špačková, L. and Štefková, J. eds. (2006), “Libraries as Gateways to the Integration of Immigrants in the EU,” Multicultural Center Prague, Prague, Czech Republic. Asgary, R. and N. Seger (2011), “Barriers to Health Care Access among Refugee Asylum Seekers”, Journal of Health Care for the Poor and Underserved, Vol. 22, No. 2, pp. 506 522, May.
Tammelin-Laine, T., et al. (2018), “Predicting Placement Accuracy and Language Outcomes in Immigrants’ L2 Finnish Education,” in Davis, J.M., et al. (eds.), Useful Assessment and Evaluation in Language Education (p.149‑165), Washington: Georgetown University Press.
Tissot, A., et al. (2019), Zwischenbericht I zum Forschungsproject “Evaluation der Integrationskurse (EvIk),” Zentrale Ergebnisse, Nürnberg, Germany.
Tomlinson, C. A., and Imbeau, M. B. (2010), Leading and Managing a Differentiated Classroom, Alexandria, VA: ASCD.
UNESCO (2004), Study of best practices in education based on ICT: Mexico, UNESCO, Mexico City.
Wienberg, J., Dutz, G., and Grotlüschen, A. (2019, accepted), “Language Learning of Migrants: Empirical evidence from the German integration course system,” Germany.
Yates, L., et al. (2015), Adult Migrant English Program (AMEP) longitudinal study 2011‑14: Final report, Sydney, Australia.
Zorlu, A. and Hartog, J. (2018), “The Impact of Language on Socioeconomic Integration of Immigrants,” IZA Discussion Paper No. 11485, IZA, Bonn, Germany.
Notes
← 1. Over 100 projects have been funded under Canada’s Service Delivery Improvement Initiative to date, and a second intake process was launched in fall 2020 with a focus on supporting the adaptation and recovery of newcomers and the settlement service sector following the COVID‑19 pandemic. Funded projects under this second intake will begin in fall 2021.
← 2. A recent French study used random assignment to language treatment around a testing threshold, finding that two years after completion of classes, 100 hours of language training increased labour force participation by 15 to 27 percentage points (Lochmann, et al., 2019). A similar study of the decision to develop individual integration plans in Finland found that migrants were subsequently offered more language hours and obtained significant improvement in employment outcomes (Sarvimäki and Hämäläinen, 2016).