Assessing Canada’s System of Impact Evaluation of Active Labour Market Policies

3. Leveraging administrative data for analysis

Abstract

Rich and comprehensive data are essential to conduct high-quality impact evaluations. Over the years, Employment and Social Development Canada (ESDC) has moved from using expensively collected survey data to utilising its register data on employment insurance and Provinces and Territories (PTs) data on participation in active labour market policies. Integrating these data directly with Canada Revenue Agency data on income has been essential to assess high-quality data on outcomes. ESDC analysis could benefit even further by enriching these data with more socio‑economic information from other agencies and PTs. Facilitating greater access to data to make it easier for external researchers to conduct analysis would encourage further innovation and provide more evidence on how policies work to help individuals secure good jobs.

3.1. Introduction

Data are critical to the success of impact evaluations of policy. Without rich and robust data any inference from analysis is likely to be limited and estimates inaccurate. Analytical data requirements are different depending on the technique being used. When participation in programmes is randomised and the accuracy of the estimator is assured through the randomisation, little more is required than accurate data on programme participation and on the outcome variables in question to estimate programme impacts. Observational studies, such as those involving regression analysis or matching, rely on having a rich set of data on personal characteristics to ensure individuals compared are alike. There are nuances within this; if evaluators want to look at specific groups within the population, even a randomised study may require richer data on personal characteristics.

Typically, those studies that do not rely on the evaluator’s ability to control selection into the programme are more data hungry than those that do. ESDC’s primary methodology for impact assessment is non-experimental therefore placing a greater burden on data needs to ensure unbiased estimates. Over the years, ESDC has moved away from collection of these data using surveys of participants and non-participants, which were expensive, cumbersome to administer and did not inform income well, to make use of the rich set of information that they possess in their administrative data. Key in this process is linking the ESDC and PT administrative data to Canada Revenue Agency (CRA) data on income. This has permitted ESDC to make a thorough, comprehensive and accurate account of post-participation outcomes for individuals.

As technology progresses and data collection and storage is facilitated, countries are making strides in the collection and assimilation of different administrative data to aid policy analysis and make data more widely available (OECD, 2020[1]). Many countries are moving towards more open data access to allow external researchers secure access to data in order to benefit from more widespread access to the expertise needed to interrogate them. Statistics Canada, Canada’s federal statistics agency, acts as a repository for administrative and survey data for non-government researchers. However, at present it is not possible to access all of the data required to conduct impact evaluation of active labour market policies (ALMPs). Opening access further would democratise these evaluations, helping innovation and providing useful cross-validation of ESDC analysis.

3.2. The pathway towards linked administrative data

When the Labour Market Development Agreements (LMDAs) were launched in 1996, funding was transferred to provinces and territories (PTs) to deliver ALMPs. It came with a stipulation to conduct evaluation of their delivery. Twelve out of the thirteen PTs agreed to conduct this evaluation jointly with ESDC (and its predecessors). This change meant that data were now required to be collected separately from each PTs, who were responsible for their own administration of the LMDAs, so that ESDC could conduct the evaluation jointly with them. The biggest challenge here was the collection of high-quality data on outcomes.

At the same time, data linking between ministries was not routine at the inception of the LMDAs, which meant it was not possible for ESDC to observe earnings data for participants and non-participants, by integrating participants’ data with income data from Canada Revenue Agency (CRA). Therefore a decision was taken to utilise surveys to collect data on outcomes and programme participation. Because the evaluations were being delivered for each of the separate PTs, this meant a separate provincial survey was required for each jurisdiction. For the first cycle of evaluation, which took until 2012 to complete, information was gathered in this way. The nature of the collection meant that progress was slow, as it was a resource intensive process to manually interview individuals and record information from them. It meant that only two to three provincial studies could be conducted simultaneously and it took around ten years for the first cycle of analysis to be completed covering all participating PTs.

3.2.1. The shift towards evaluation driven by administrative data

The cumbersome nature and cost of individual surveys for PTs, led ESDC officials to explore the possibility of utilising administrative data to conduct the evaluations. Government administrative data offer several advantages in their use for impact assessment, relative to other forms of data such as survey, or privately held data:

• Universal Coverage – all recipients of government benefits are recorded as clients. They do not suffer from attrition – individuals have their details recorded for the duration of their claim. This contrasts to survey data where individuals may opt-out of follow-up data collection.

• Accuracy – they are not subject to recall errors. Individuals do not have to remember how much they are paid or when, or what programme they were participating in.

• Precision – they have the benefit of scale. Being population data, every individual is covered, so sample sizes are large. This aids statistical precision, which is beneficial where the outcomes of interest have a lot of natural variation (for example earnings), the expected impacts of the policy are small (for example in job counselling services), there are relatively few people participating in the programme or where the researcher wants to look at impacts for sub-groups. It means there can be greater confidence of detecting an impact where one exists.

• Timeliness – they are often timely, since they exist to support benefit administration, details need to be captured in real time.

• Cost – relative to other forms of data collection (such as surveys), they are cheap to collect because they are already collected for benefit administration purposes

These factors all contributed towards the move by ESDC to utilise their administrative data for evaluation. With appropriate privacy provisions in place, approval was granted in 2004 to integrate ESDC data to CRA data on income and tax. On this basis, ESDC started to evaluate the scope for administrative data to replace survey data for the impact analysis. This work started by comparing the CRA data to the survey data collected in one province. It revealed that ALMP participants systematically over reported their income and non-participants systematically under-reported it. The presence of this difference in “mis-reporting” meant that programme estimates based on these data would overstate the impact of the programme on earnings. These results paved the way for the systematic adoption of administrative data to replace survey data in the second round of LMDA evaluation, starting in 2010. CRA administrative data on income could replace the income data collected from the surveys and a combination of administrative data on past benefit receipt from ESDC and past income data from CRA could replace the socio‑economic data collected, which was used to compare similar participants and non-participants.

An additional benefit from the adoption of administrative data was the considerable cost saving to government on offer. The bilateral provincial surveys were expensive‑ around CAD 1 million per annum was spent on putting surveys into the field (Gingras et al., 2017[2]), with the associated data collection and assimilation. The use of CRA data combined with ALMP participation data was exploiting data already held for administrative purposes. Initial work by analysts to interrogate, assimilate and compile the data into a format conducive to analysis was completed and then ongoing maintenance and administration costs are minimal, compared to the costs associated with individual survey data collection at the PTs level.

3.3. Building a data platform for evaluation

In order to provide a platform to conduct their quantitative analysis, the ESDC evaluation directorate have developed the Labour Market Program Data Platform (LMPDP) (Table 3.1 (ESDC, 2020[3])). This platform consists of 11 separate but relationally integrated data entities which enable analysts to look at:

• Patterns of actual participation in ALMPs;

• Patterns of eligibility to participate in ALMPs;

• Patterns of claiming employment insurance benefits;

• Annual sources of income;

• Annual job patterns.

The process of compiling this platform takes three stages:

1. In the first phase, ESDC administrative data on ALMP participation are compiled. These data are taken from four separate sources, which come from different administrative systems within ESDC. For example, participants in programmes in the former Youth Employment Strategy are held in a different system than those participants in Labour Market Development Agreement (LMDA) programmes. Important data cleaning is conducted in this stage, to ensure that data are chronologically consistent and participation in different programmes is not in conflict (for example, individuals are not participating in two ALMPs at the same time where this is not possible).

2. In the second phase, the PTs data on programme participation are merged with ESDC data on employment insurance. This allows eligible non-participants to be identified. Because participation in the LMDA is contingent on qualifying contributions to employment insurance, this stage is important to reconcile how participation relates to periods of qualification. The employment insurance data also contain a number of personal characteristics (such as age, gender, marital status, disability) that are brought in, so they can be used for later analysis to compare participants with similar non-participants. In order to create a control group of individuals that did not participate in an ALMP, the detailed history of individuals must be analysed to check for underlying entitlement to participation. By doing this, ESDC can then compare individuals who did not participate in an ALMP, but who were eligible to do so, with those that did participate.

3. In the third phase, added to the ALMP participation (phase 1 dataset) and timing data (phase 2 dataset) are data on annual income and social assistance receipt from CRA and information on job spells. The CRA data are updated annually and transferred to ESDC. This step makes it possible to observe work outcomes of both participants and non-participants. An important step in this stage of the data preparation is the simulation of participation among eligible non-participants.

Constructing this platform provides ESDC several advantages. The first is that they have a consistent platform for evaluation. Evaluations of the LMDA, Youth and Indigenous programmes have been conducted using the same data platform. This creates:

• Efficiency – raw data are not being re‑processed for every new evaluation.

• Consistency – the participation data and related income and employment records are the same for individuals across evaluations.

• Institutional knowledge – having an enduring platform that analysts use also means that expertise that is built up on the data can easily be shared among ESDC analysts, meaning quality assurance is easier and analysis can be conducted quickly.

An extensive suite of data documentation and metadata has been amassed to ensure that ESDC has business continuity and that new staff are able to quickly assimilate themselves with the data. The existence of a data team within the evaluation directorate has helped to ensure that data are well documented, allowing for analysis to be conducted correctly and maintaining consistency between analysis on different projects.

Table 3.1. There are three stages to data processing to build the Labour Market Program Data Platform

Phase	Input Data	Data Processing	Output Data
1	-Input datasets on ALMP participation start and end date - Four separate datasets covering participation in ALMPs under different funding streams - Participation in ALMPs (start and end dates, programme type) related to eligibility through EI contributions	- Consolidate data to fix inconsistencies and incompatibilities - Imputation of end dates when missing or invalid - Remove duplicates/redundancies - Normalise coding and record structure	Integrated Intervention File
2	Integrated intervention File Employment Insurance (EI) data - Receipt/amount and spell data on EI -Individual Characteristics (for example, age, gender, marital status, disability)	- Calculate timing and duration of participation in ALMPs relative to EI qualifying periods (e.g. how many weeks after the start of EI receipt does someone participate in ALMPs) - Develop timing and duration models so that periods of eligibility are constructed to compare non-participants to participant	Timing and Duration File
3	Integrated intervention File Timing and Duration File Canada Revenue Agency annual tax return data – incorporates both income data and data on social assistance Receipt Record of Employment – records on job separation containing employment information	- Integrate Administrative Datasets - Simulate participation among eligible non-participants - Creation of analytical variables for data platform (for example, earnings pre‑ and post- participation and derived variables such as skill level)	LABOUR MARKET PROGRAM DATA PLATFORM

Source: ESDC (2021[4]), Labour Market Program Data Platform – Data Dictionary v3.05a; Handouyahia (2019[5]), “The creation of a rich data platform to support net impact evaluation of Labour Market Programmes”, https://www.oecd.org/employment/emp/S4.5.%20Handouyahia_CAN.pdf.

3.3.1. Securely linking administrative datasets enables a comprehensive analysis of outcomes

Linking together administrative datasets on Employment Insurance with CRA tax return data enables ESDC to look at a comprehensive suite of outcome information on individuals (rather than just patterns of employment insurance receipt), which is much richer than if ESDC administrative data were used alone.

Canada has a mixed ability to integrate its different administrative data together, driven in part by its federal structure. Similar to the majority of OECD countries, it is able to link its employment register with its unemployment register. However, it is one of only seven countries for which its social assistance register is held at a regional level (OECD, 2020[1]). Whilst these regional files are shared with ESDC, in principle this may cause additional costs to employ those data in the LMPDP as they would require inspection, standardisation and collation. However, the use of CRA tax returns facilitates the easy incorporation of this information since social assistance amounts received in a given calendar year are included on these returns (though these data are only available for employment insurance recipients and LMDA participants). The incorporation of the CRA data into the LMPDP then means that ESDC can evaluate not only the impact of its programmes on receipt of employment insurance, but also on social assistance and on earnings from work.

The principle drawback, relative to other OECD countries with employment register data, is the lack of information on employment spells in the employment register. It is one of only three countries for whom there is no information contained within that register (OECD, 2020[1]). These data are recorded in the Record of Employment dataset available in ESDC, but they are not always reliable and are only issued on separation for a job, so it is not possible to identify spells of employment that have not yet ended. However, they are a mandatory piece of information for individuals claiming Employment Insurance, so consideration could be given on whether to analyse them in the evaluation of current employment insurance claimants. Improving the coverage of these data would permit further insight into the outcomes of jobseekers.

Data are pseudo‑anonymised to comply with data protection regulation

Data are protected and integrated together using identifier keys. Social Insurance numbers are used to link participants, but have been encrypted by an algorithm so that the key is no longer the same as the original and is instead replaced by a unique “sequence key”. In this way, with the same encryption across datasets, it is possible to link individuals without ever disclosing their actual social security numbers. Names and addresses of individuals are removed so that no personal information remains.

Access and linking of these data sources is dependent on approval from the Privacy and Information Security Committee (PISC) review and Deputy Minister Approval. CRA data are updated annually and brought into ESDC’s secure data warehouse and are controlled using business cases, where named individuals have to specify their business reason for accessing the data and agree to abide by the security procedures in place for data access.

3.3.2. Socio-demographic and past outcomes data provide rich information on individuals

The data that ESDC integrates into the LMPDP provide them with rich information on socio-demographic characteristics on labour market participants, which are critical to conducting counterfactual impact evaluations (the significance of this is discussed in Chapter 4). One of the drawbacks to administrative data is that they are often relatively sparse. Ministries can be limited in their ability to collect only the data they need for administration of benefits. Data on personal characteristics of claimants can be lacking. When conducting impact assessments having rich data on socio‑economic status, educational history, marital status, motivations and aspirations, is pivotal to explaining an individual’s choices (Heckman, Lalonde and Smith, 1999[6]; Lechner and Wunsch, 2013[7]), particularly with respect to the labour market. Many of these variables will not be available to a ministry that deals with administration of unemployment insurance, as they are not necessary to discharge these duties. However, this sparseness of administrative data can be mitigated by using detailed historic information on outcomes. Things like education and motivation are highly correlated to earnings, so by using information on past outcomes and benefit receipt it is possible to proxy these variables, even where they are not recorded directly. ESDC records and uses a full five years of past employment outcomes and benefit receipt to help ensure that this information is rich. This permits ESDC to create a detailed typology of claimants, proxying for other unobserved variables.

The socio-demographic and historic earnings and benefit data contained in the LMPDP provide a strong basis for which to compare alike individuals that did and did not participate in ALMPs. ESDC makes use of up to 75 socio-demographic and labour market variables, which are observed over five years prior to the participation period (ESDC, 2017[8]). The data include information on a range of characteristics that are used for the evaluation work (Table 3.2). Particularly important within this set of data are “past outcomes” – looking at the earnings and receipt of employment insurance and social assistance for five years prior to the period in question.

The main additional socio‑economic variables captured in the LMPDP are age, gender, marital status, industry and occupation of previous job, and self-reported characteristics such as whether belonging to an identifiable “visible minority”, being Indigenous or having a disability. The majority of this information comes from the employment insurance administrative data and is collected as part of the administration of that benefit. Marital status is contained on the CRA income tax data. These variables are useful both to disaggregate programme impacts for different groups, and to construct groups of similar participants and non-participants.

Table 3.2. Variables used by ESDC in the statistical analysis

Variables
Age	Gender	Aboriginal origin
Visible minority	Disability	Marriage
Skill levels – Five groups based on occupation codes (managerial occupations where factors other than education is important, occupations requiring a university degree, college/vocational/apprenticeship, high school (one to four years of secondary schooling) or occupational (up to two years), on-the‑job training (up to two years secondary school and short work demonstration))	Province/territory of claim	Industry of previous job (NAICS 2‑digit code)
Number of hours of work contributing to employment insurance entitlement	Reason for previous job separation	Whether a new job market entrant
Year and quarter of ALMP participation (or eligible participation for non-participants)	Gap between start date of employment insurance receipt and start date of ALMP participation	Number of weeks with earnings between commencement of employment insurance receipt and start date (potential start date for non-participants) in ALMP
Participation in employment programmes in the previous five years	Annual earnings in the previous five years	Annual amount received in employment insurance (EI) or social assistance (SA) in the previous five years

Source: Authors summary of ESDC (2019[9]), Quantitative Methodology Report – Final; ESDC (2021[10]), “Analysis of Employment Benefits and Support Measures (EBSM) Profile and Medium-Term Incremental Impacts from 2010 to 2017, Technical Report, 2021”.

The absence of data on education and family status means some sub-group analysis is not possible

At the time that ESDC put the LMPDP together, it did not possess information on individuals who had children. Whilst CRA tax returns do record the presence of children for whom a non-refundable tax credit may be claimed, it appears only on the tax record of the parent with the highest income (ESDC, 2019[9]). In Canada, the propensity for women to provide childcare to children at home means these issues are likely to disproportionately affect them. They spend almost 26 hours per week more than men caring for children (Statistics Canada, 2021[11]). Details on the presence and age of children would allow a much more detailed understanding of how ALMPs may influence parents’ labour market participation decisions. This is particularly pertinent at different points in children’s ages, for example when they start or leave school. This may be achieved via the incorporation of extra data. For example CRA data on child benefit receipt might allow some of these issues to be surmounted. These data incorporate the number of dependent children and because records exist for every year of receipt, it would be possible to broadly proxy ages of children by looking at the start date of benefit receipt and changes to the number of dependent children over time.

Similarly data to accurately compare young ALMP participants to non-participants are sparse. ESDC does not hold information on educational attainment for all individuals. Information is held on self-reported educational attainment for participants in ALMPs, but none is held for non-participants, meaning it cannot be used for the evaluation analysis. The reliance on past labour market and benefit receipt to proxy unobserved information on socio‑economic status or education history, is unlikely to be completely sufficient for young people. As ESDC defines young people as aged 30 or under and given that relatively few of their customers have tertiary education (and so would have fewer schools years at older ages) (ESDC, 2021[10]), it is not likely that these issues dominate the results ESDC reports for this group. But, it does limit the ability to say with any certainty what the impact of these programmes may be for individuals that are relatively early on in their careers. This may have less impact on the LMDA evaluations because their catchment group requires some history of employment insurance (and therefore earnings) meaning it is unlikely to incorporate as many young people just leaving education, but may be more relevant to other programmes, such as the Youth Employment and Skills Strategy. Data on educational attainment can also be used as an outcome variable, which can be particularly useful for programmes for which a successful outcome might be that the participant then completes further training. New Zealand’s experience demonstrates that its administrative data on educational attainment is useful in both of these respects (de Boer and Ku, 2018[12]; 2018[13]), facilitating better matching but also providing useful information on how policies affect skills attainment. It links these data into its Integrated Data Infrastructure, making their use in analysis particularly easy. This data infrastructure is similar in design to the LMPDP employed in Canada.

As young individuals are particularly susceptible to the pernicious effects of shocks to the labour market, as has been demonstrated recently with the COVID‑19 crisis (OECD, 2021[14]), demonstrating which programmes are the most effective at helping them improve their outcomes is especially important. This is true not only for labour market outcomes but also for participation in education, which is another potentially common outcome for young individuals. It is for this group of individuals, for whom socio‑economic administrative data are most unlikely to remove heterogeneity between individuals that alternative data strategies would help to produce robust impact assessments. Randomising participation for young people, or using detailed survey data to collect rich socio‑economic data, could alleviate the current issues and allow young participants and non-participants to be better compared to one another.

Finland offers a good example of where extensive administrative data are available for use in analysis, facilitating more robust evaluations and permitting greater sub-sample analysis. The data held by Statistics Finland include a vast array of variables including educational level and qualification field, marital status size of household, tenure type, number and age groups of children, socio‑economic group, occupation, alongside employment and unemployment histories. (Statistics Finland, 2022[15]). Education data can be useful not only in the identification of impacts on young people, but also as a means to explore whether ALMPs offer differential impacts depending on the level of qualification of people (Aho et al., 2018[16]). The ability to control for family characteristics allow researchers to look at the dynamics of household formation on ALMP participation and outcomes.

3.3.3. Important choices need to be made on what is being assessed and for whom

ESDC does not assess the impacts of individual interventions (ALMPs) but rather combinations of them. This is because interventions are not always assigned on an individual basis but are co‑ordinated with other interventions to jointly achieve labour market objectives for the participant. The joint assignment of individual interventions is referred to as an “action plan”. However, because the “action plan” to which an intervention belongs is not consistently recorded in the various administrative data sources it is reconstructed for each participant by grouping interventions that take place within six months (specifically 183 days) of each other. Groups of interventions combined this way are referred to as “Action Plan Equivalents” (APEs). ESDC sought expert guidance and conducted a detailed data assessment to formulate the construction of these APE to reconcile for the missing administrative data.

ESDC then takes every APE and assign a principal ALMP to it, which is the longest intervention in that specific APE. For example, in an APE that contained both Skills Development and Job Creation Partnership, if the participant was in the former ALMP for the longest time, that APE would be labelled “Skills Development”.

Table 3.3 shows the combinations of ALMPs that are contained within APE for the cycle two evaluation conducted. For example, the APE with a principle ALMP of Targeted Wage Subsidy also contained an average of 1.66 programmes of Employment Assistance Support (EAS), 0.01 of Self-Employment and Job Creation Partnership. The high average number of EAS contained is due to its short duration and prominence in ALMP delivery- all jobseekers start with counselling before moving on to other ALMPs.

Table 3.3. APE contain a mixture of ALMPs in addition to the principle programme

The composition of Action Plan Equivalents by their underlying ALMPs, active claimants, cycle two evaluation

Principle ALMP	Share in total	Average number of programme occurrences per APE
		Skills Development	Targeted Wage Subsidy	Self-Employment	Job Creation Partnership	Employment Assistance Support
Skills Development	23%	1.00	0.03	0.01	0.01	1.00
Targeted Wage Subsidy	3%	0	1.00	0.01	0.01	1.66
Self-Employment	4%	0	0.01	1.00	0.01	1.44
Job Creation Partnership	1%	0	0.03	0.01	1.00	1.10
Employment Assistance Support- only	69%	0	0	0	0.00	1.00

Note: ALMPs: Active Labour Market Programmes. The table reports Action Plan Equivalents (APE) for Active Claimant.

Source: ESDC (2017), “Evaluation of the Labour Market Development Agreements – Synthesis Report” and ESDC (2016), Cost-Benefit Analysis of Employment Benefits and Support Measures.

Statlink

This is not problematic to the mechanics of an impact assessment – combining ALMPs in this way, ESDC will still be able to make robust estimates of an APE’s effect on outcomes, but it does complicate the interpretation of results somewhat. It becomes much harder to evaluate the different programmes next to each other, because the programmes are not defined in isolation, but rather in these combinations. The exception to this is the “Employment Assistance Support – only” category, which has been defined such that it is the only APE which contains no other ALMPs.

Estimating impacts per individual programme, rather than combining them together as is done now, would permit a more straightforward comparison of relative effects. Although the existence of ALMPs outside of the principal ALMP are minimal and it seems unlikely that they would make a big impact to the estimates of the APE, their presence confuse the presentation of the individual ALMP. It is hard to reconcile that a “co‑ordinated” action plan for an individual would have an element with, for example, both a wage subsidy and a job creation programme. More likely it seems this is an artefact of the data rule to categorise programmes within six months of each other as part of the same APE and therefore as part of the same evaluation package. The exception to this discussion is for EAS which one would expect most APE to contain given their nature as a gateway service and precursor to further support. An easy way to implement this, for presentational purposes, without having to change the underlying modelling, would be to subtract the weighted outcomes for the secondary ALMPs from the principle estimate (for example, if a Skills Development APE increased earnings by CAD 5 000 but also contained 0.5 Employment Assistance Support – a programme of which was estimated to increase earnings by CAD 1 000 – report earnings of CAD 4 500).

Participants are split into different groups according to their employment insurance status and underlying personal characteristics

There are two separate facets to how ESDC addresses potential differential impacts of ALMPs in the evaluation that merit discussion.

The first is, subsequent to the first cycle of evaluation, ESDC’s split of participants into two distinct groups:

• Current Claimants – An individual with a current open claim to employment insurance at the time of participation in an ALMP.

• Former Claimants – An individual with no current employment insurance claim but who qualifies for participation in a programme on the LMDA because they have had an open claim in the last five years or have paid a minimum level of EI contributions in the past ten years.

ESDC makes this distinction because of the problem of identifying a potential comparison group for “former claimant” and “non-insured” participants. Because available data are insufficient to identify programme-eligible non-participants, only those who show up for “limited treatment” (i.e. EAS) under the “former” and “non-insured” streams can demonstrate their eligibility under these streams. While this approach yields “relative” rather than “net” programme effects, it allows for the selection of a statistically equivalent comparison group using quasi‑experimental methodology.

In order to create a group of individuals to compare against for the former claimants, ESDC uses those former claimants accessing EAS only as a control group and compare them to those former claimants who participate in other ALMPs. In this way, this somewhat solves the motivation issue by comparing two sets of individuals that have both come forward for support.

Because of this, ESDC is unable to estimate the impact of EAS on former claimants. However, this re‑framing of the control population in order to make a comparison for the other ALMPs is a clever use of the data ESDC holds in order to provide inference on this group. This has the drawback that it then becomes more difficult to compare results for former claimants against current claimants. However, by proceeding with APE reporting as suggested in the previous section, this issue could also be surmounted.

The second consideration on participant type is on splitting estimates into different groups of individuals, who may respond differently to the ALMPs that they participate in. This is an area in which ESDC has progressively built evidence and expanded its ambition. In the second cycle of evaluation, beyond the split into current and former claimants, individuals were split into sub-groups based on whether they were:

• Youth (aged below 30 years)

• Older workers (aged 55 years and older)

• Long-tenured workers (employment insurance contributions in at least seven of the previous ten years)

In the third cycle of evaluation this sub-group analysis was extended further to other groups of interest, to look at:

• Gender

• Indigenous status

• Persons with disabilities

• Persons identifying as being a “visible minority”

• Immigration status

By defining sub-groups in this manner and separating out the analysis, it is possible to derive programme impacts for these specific groups. As the number of participants in a group becomes smaller, it can be harder to identify an impact statistically, because the precision of the statistical tests depends on the number of observations in the group (having more observations means more precise estimates). However, one of the advantages of having administrative data is that these sample size issues are much more easily avoided. In this respect, administrative data have allowed ESDC to be more ambitious.

ESDC is now moving towards the use of machine‑learning algorithms to extend this approach further. The sub-group analysis talked about previously was conducted by pre‑specifying groups of interest and then evaluating the impact on them. To this extent, it relies on user choice of these groups beforehand, based on some kind of expert knowledge that differences may occur and be meaningful to analyse. Machine‑learning automates this process and uses the data to discern whether differences in outcomes occur and for which groups. The advantage of this is that it is not reliant on a person to pre‑define the groups (risking incomplete or irrelevant sub-group choice), but it comes with the risk that the types of group chosen are less well qualitatively defined and for which it may be difficult to make operational delivery choices about (for example, if the algorithm chooses men, with three years of recent work in a managerial profession, interspersed with six months of unemployment prior to that, it may be difficult for counsellors to identify and serve those customers differently in practice).

3.3.4. A comprehensive suite of outcome variables are used in analysis

ESDC utilise the information contained in the LMPDP to look at a number of outcomes for individuals (participants and non-participants) (ESDC, 2021[10]). These outcome variables are:

• Annual employment earnings (A)

• The incidence of employment (denoted by whether an individual has had a spell of employment in a year)

• Annual amount of employment insurance benefits paid (B)

• Annual number of weeks of employment insurance benefit paid

• The incidence of social assistance receipt (denoted by whether an individual has had a spell of social assistance in a year)

• Annual amount of social assistance paid (C)

• Dependence on income support (defined as (B+C)/(A+B+C))

This set of outcome variables allows a comprehensive assessment of the impact of a programme on a participants subsequent labour market outcomes. ESDC’s administrative data on employment insurance and income tax data on social assistance allow a thorough assessment of the subsequent impact of the programme on the payment of benefits to participants. CRA data on income is essential in looking at how much individuals earn in their subsequent employment. The combination of both of these datasets included in LMPDP allows a thorough assessment of an individual’s post-participation income, including both work and non-work spells.

Because these outcomes are derived from administrative data they are high quality – there is no non-response bias or recall error. Errors can still occur in register data but their order of magnitude is typically lower than the aforementioned errors in survey data (see Meyer, C Mok and Sullivan (2015[17]) for a United States discussion or Bellemare, Kyui and Lacroix (2021[18]) for a discussion relating to Canadian immigration and earnings). This was also corroborated in ESDC’s past survey and administrative data comparison and gave ESDC a strong rationale to continue with administrative data as the main source of information for outcomes data.

But currently it is difficult to look at indicators of job quality such as job transitions, tenure length and contract type

One of the drawbacks of the current set of outcome variables is that they do not permit much insight into job quality, apart from earnings. The indicator for incidence of employment does not capture tenure – a job spell of one week would look identical to a job lasting for the full 52 weeks. Similarly there is not presently a way to identify the number of spells of employment, to look at job cycling. Some of this can be inferred by the impact on total employment earnings (an individual earning less in a year must either be working for less time or at a lower wage), but at present it is impossible to say which of these factors drives the result.

This limits the extent to which the analysis can identify low income individuals who frequently cycle into and out of work, compared to those with more stable employment history and with fewer but longer transitions between states. Given that participation in ALMPs is likely to be concentrated at the lower end of the income distribution, this could be an important distinction to make when looking at the selection pathways into employment programmes (see Andersson et al. (2013[19]) for an example of how data aggregation can change programme estimates). For example, a paper utilising Swiss unemployment register data and social security administration data on earnings analysed how benefit sanctions impacted upon subsequent earnings stability, by looking at spells of employment (Arni, Lalive and Van Ours, 2012[20]). The incorporation of such information would allow ESDC to investigate how their ALMPs influence job tenure. This could be done potentially using the Record of Employment data, at least for active employment insurance claimants, for whom there are better quality data.

Similarly without information on contract type, it is not possible to see whether the successful completion of an ALMP moves individuals towards securing jobs with more permanent employment contracts. A recent study from France demonstrates the value that having data on contract type can have on determining whether ALMPs impact the type and quality of job available (Algan, Crépon and Glover, 2020[21]). Twenty-four OECD countries can link these data directly from their employment register (OECD, 2020[1]), making the process much more routine for their incorporation into analysis. Having complete information on an individual’s occupation that would allow it to be used as an outcome variable, would allow an investigation into whether individuals moved up the job ladder as a result of participation. The current Record of Employment data contain some broad information on type of contract, an extension of these categories, to capture this information, may permit its incorporation into analysis.

Earnings data suffer from a lack of timeliness and aggregation which may inhibit policy makers

The use of CRA data for income and social assistance highlights the drawbacks of using administrative tax data, namely its timeliness and periodicity. The data held by ESDC on income and tax lag its ALMP programme data by two years. As individuals typically have to submit annual tax returns, the deadlines for which are some months after the end of the tax year, it means that there is a long lag to assimilate data. In the analysis of employment programmes, where the impacts of a programme can take some months to occur (for example, a training programme may last six months and then analysis should allow time for people to enter the labour market) this combination may mean that it is not practicable to evaluate policy until some years after its implementation. The lag in timely tax and income data is not a problem for policy analysis per se but, it does constrain policy makers in the shorter term, which may make a difference when budgets are being set, particularly in times of fiscal restraint. It is easier to justify cutting a programme in the absence of evidence of its benefit.

As real-time data become more widespread, there is a much greater opportunity to improve analytical turnaround times. Ireland provides an example of where this has added value to quickly provide insight on labour market outcomes following COVID‑19, using a real-time lookup of its Revenue Ireland data to analysis labour market outcomes of those using its Pandemic Unemployment Payment (Department of Social Protection, Ireland, 2021[22]).

The United Kingdom offers a comparable example where its use has sped up ALMP evaluation. The design of Universal Credit in the United Kingdom means that real time information on earnings is transferred from Her Majesty’s Revenue and Customs to the Department for Work and Pensions (DWP). This allows analysis to be conducted almost in real time, useful both to the operational monitoring of policy but also by reducing lead times on impact evaluations. The randomised control trial on counselling support services to employed individuals run by the DWP demonstrates the value of such real time data exchange. DWP recruited participants between March 2015 and March 2018. In September 2018 it was able to publish its preliminary impact assessment, looking at employment outcomes up to 52 weeks after trial enrolment. By October 2019, it was able to publish an extension to this analysis to 78 weeks (DWP, 2021[23]). In principle, given the real time nature of the data, the minimum amount of time between these two studies could have been just six months. Even so, the turnaround time was rapid for this type of assessment.

The periodicity of the income tax data in Canada also raises questions about its suitability to analyse the impact of EAS, whereby expected programme impacts are relatively short and smaller in scale. The small impact on earnings, of securing a job earlier, is easier to get lost in the noise of annual earnings data. There are also non-trivial questions to answer on the assignment of earnings to pre‑ and post- treatment periods. For programmes that begin in the middle of the year, it is particularly difficult to know whether annual earnings belong to the treatment or pre‑treatment period. ESDC circumvents some of these issues by specifying an “in-programme” year and then looking at years “post-programme”. When looking at programmes that might be expected to have longer-term impacts (such as Skills Development) this is unlikely to have substantial impacts on impact assessment. However, for EAS, this could very well mask the shorter-term impact that these services may have on employment, especially as they are often designed with improving jobseekers job search ability and may improve the job matching speed. Other studies looking at these types of programme, that have access to temporally disaggregated data, have demonstrated the impact they have on “in programme year” effects (for example Cheung et al. (2019[24]), DWP (2018[25])).

One of the drivers for ESDC to move to a unified, aggregate assessment was due to the long lead times posed by the bilateral survey-based evaluations. This suggests that timeliness is a dimension to policy making in Canada that has some salience. Therefore, any efforts made to reduce the lead time before data can become available is likely to be a welcome intervention to policy making.

3.4. Quality assurance

Quality assurance of results and exploration of the sensitivity of results to techniques and assumptions within them are critical to ensure accuracy of results, to evaluate risk and to convey the weight of evidence behind the conclusions. ALMP evaluation is complex especially so for programmes which are non-experimental and rely on the creation of a counterfactual group using statistical analysis. ESDC has implemented a range of processes to quality assure the data it uses, the methodologies it employs and the results that it produces, in order to ensure analysis is reliable.

The ESDC teams conducting the impact evaluation follow a series of steps and procedures to quality assure their analysis. The methodology team that sits within the evaluation directorate provides guidelines on the processes to follow in evaluations and separates out the task list into stages. Each of these stages has multiple checks to complete to ensure data accuracy. The four stages are broadly outlined as follows (ESDC, 2021[26]):

• Evaluation Strategy – to define strategy for evaluation with ESDC officials and programme teams. Checks are made on the validity and availability of all administrative data and potential outcome variables. The methodology team work with evaluators to check capacity of data to answer research questions and identify outcome indicators.

• Assessment of Evaluation Strategy – conducted internally within the project team, to cross-validate the administrative data, perform a literature review and submit data access requests.

• Data Analysis – checks with other ESDC branches on data collection and quality assessment. Internal project checks across all data verification (see Box 3.1 for more detail):

• Final validation – sharing results and methodology with peer reviewers, compare outcomes with other OECD countries, review against Statistics Canada census data, review code with external contractors.

Box 3.1. ESDC data analysis quality assurance

Extensive checks are conducted on data to ensure that the data used is reliable and up-to-date

Data type checks are conducted to investigate whether data are of the expected formats in meta data and checks are run to look for missing values or erroneous codes. An assessment of data quality is made by tabulating data to ensure its reliability, accuracy, relevance and completeness, following Statistics Canada’s data quality guidelines.

Ranges and frequencies are checked to ensure values lie within expected ranges (for example, incomes are not negative), and that frequencies remain consistent across the analysis. Any dropped cases are carefully documented with rationale.

Checks to external totals are conducted, to ascertain analysis is consistent with other reports (for example the annual ESDC employment insurance Monitoring and Assessment reports).

Code Checks are run throughout the code. Syntax and logical errors are checked. Mid-stage datasets are checked against their source files to ensure data has not been lost of inadvertently manipulated.

Post-Validation Checks compare results from past evaluations to assess whether observed differences are within expectation or exceed a tolerance threshold.

Literature reviews are conducted to take into account recent methodological developments (for example, the move towards using machine learning for sub-group analysis).

Software checks are conducted to validate results using different statistical packages (across SAS, Stata, R, and Python).

Robustness checks are conducted to validate the net impact results using alternative methods, to determine whether the results change dependent on the method used.

Sensitivity analysis is performed which changes key variable assumptions in the work, to determine the extent to which the results are affected by changing parameters.

The range of checks carried out allows ESDC to systematically assess all of the attributes to data quality in their evaluation and how these may impact on its results. Checks on the data used can be especially important with administrative data, particularly where data pertaining to the same individual or programme can be recorded on different systems and provide different answers. Similarly, whilst administrative data are usually high-quality and accurate, it can on occasion be inaccurate. Providing a strong and documented rationale for its inclusion or exclusion in analysis is necessary for transparency and scrutiny. Relationships with wider teams and colleagues can often be helpful in this domain- operational staff can often provide insight to data quality where statistical investigations alone may be insufficient. By having a clear and consistent set of metrics to compare against, it helps to ensure that checks are methodical, repeatable and comprehensive.

Source: ESDC (2021[26]), SSPB ED Quality assurance Methodology (unpublished internal guidance document); Statistics Canada quality guidelines, https://www150.statcan.gc.ca/n1/pub/12-539-x/2019001/ensuring-assurer-eng.htm.

This set of processes is both detailed and comprehensive. As an established set of procedures, it allows the quality assurance process to be recorded and documented. It also allows for an internal discussion of technique validation and checks on data and assumptions, prior to analysis being sent to external peer reviewers. Having this set of procedures in place helps to verify the analysis that has been conducted and ensure it is comprehensive, up-to-date and accurate. This is important because analysis is conducted internally. External quality assurers only advise on the outputs from the analysis and the methodology, they do undertake any detailed scrutiny of data used or the statistical coding that is performed. Having an internal set of processes ensures this work is completed. The methodology team works with the team conducting the evaluation to ensure all the processes are followed. Consultations with expert teams outside of evaluation directorate ensure that the correct expertise is employed to assess the analysis. For example, the data quality team within the Skills and Employment branch, is consulted on the data used in the evaluations, as they have responsibility for the data upload and quality assurance of the LMDA and WDA data and so have expertise on data quality and its suitability for use. Checks across both coding and data allow confidence that analysis is not subject to error which would undermine the reliability of the results. It adheres to the “four eyes principle” in having analysis cross-checked by different individuals. This principle is strengthened further by collaboration with staff in other ESDC units. On issues to do with data processing and extraction, expertise is sought from the Chief Data Officer’s team, to ensure that database extraction requests are valid. Results and coding checks are conducted within the evaluation directorate, so that teams that have not directly conducted the work scrutinise the code and the results to corroborate analysis. Once these checks have been completed, results are shared with external peer reviewers who provide expert judgement on them. Additionally contextual checks on the analysis are applied by cross-referencing to known international data and census data from Statistics Canada.

3.5. Increasing data availability

Currently, the availability of data for evaluation of ALMPs sits entirely within ESDC and specifically within the Evaluation Directorate. However, outside of the analysis that ESDC conducts, it is not possible for external researchers to conduct assessments of ALMPs. Improving the availability of data to external researchers could lead to more numerous studies, foster creativity and learning, and cross-validate the existing work produced by ESDC. It is no coincidence that in the meta‑analysis by (Card, Kluve and Weber, 2018[27]) the countries that feature with the most evaluations of their policies are the ones with open access to data. Germany, Austria and Switzerland contribute 52 studies. Denmark, Finland, Norway and Sweden contribute 48 studies. But the combined heft of Canada, the United States, the United Kingdom, Australia and New Zealand contribute just 24 (Card, Kluve and Weber, 2018[27]).

3.5.1. Statistics Canada are compiling and making data available for research

Statistics Canada, the national statistics agency, is leading Canada’s effort to increase data availability to researchers and does have an option to allow public access to microdata (OECD, 2020[1]). It offers access via two channels:

• Unrestricted access – Public Use Microdata Files (PUMFs) are available to institutions and individuals. They are non-aggregated data, which are carefully modified and then reviewed to ensure that no individual or business is directly or indirectly identified. There are some 145 PUMF available to individuals, either via download individually on the Statistics Canada website or via the use of an institutional subscription service that gives unlimited access to all of the datasets and metadata. Institutions that subscribe also sign a licence agreement.

• Restricted access – Data are available via the use of Research Data Centres, which are secure facilities located in government offices, universities or secure access points in approved locations. Around 150 data files are available for analysis. Researchers are deemed employees of Statistics Canada. Academic users are managed via the Canadian Research Data Centre Network, who provide 33 access points on campuses throughout Canada. For primary university users (including students or employees of the partner universities conducting self-directed research), there are no access fees. Access fees for secondary users vary. For academic researchers of other institutions or government and third sector users, they are CAD 6 250 for the first 200 hours of data access, with fees of CAD 3 250 for additional data access blocks of 100 hours. Private sector users are charged CAD 9 500 for the initial access and CAD 4 750 for 100 additional hours. They are also charged CAD 3 875 per data file requested, whereas academic, government and third sector users pay only CAD 700 per file after an initial allocation of 25 data files.1

Despite the drive towards greater open data access, there still exists no comprehensive way to interrogate administrative data on ALMPs, essential to any counterfactual impact assessment of ALMPs. Both the unrestricted and restricted access datasets contain mainly survey data, rather than administrative micro‑data. In addition, of the 145 files made available via PUMF, over 100 were released in 2015 or earlier (containing data relating to 2013 and earlier) – limiting the use of datasets for up-to-date policy analysis.

The Research Data Centre data files offer promise for ALMP evaluation but are as yet incomplete for proper evaluation of ALMPs. At present, its repository contains some but not all of the administrative data needed for evaluation:

• Employment insurance Status Vector – 1997‑2018: Weekly employment insurance records for claimants, covering the benefits enjoyed by participants and the earnings they report.

• Longitudinal Administrative Databank – 1982‑2018: Provides a 20% sample of income tax data from CRA.

• Record of Employment – 1987‑2019: Information on job separations, mandatory for employment insurance claimants, containing information on job tenure and other job characteristics related to insurance administration.

No data are available on ALMP participation. There is no facility via the current dataset access to link ALMP data to CRA data in the manner that ESDC uses for its evaluations. Making this a priority would open ESDC policies to the wider research community and would also allow PTs to take a more critical look at their own policy delivery.

3.5.2. Other countries provide examples of how institutions can be used to facilitate data access

Liberalising the use of data access can take different forms but usually revolves around there being a specified government institutes that warehouses register data from different ministries and links these data together. In addition to the examples of Institut für Arbeitsmarkt (IAB) in Germany, Stats NZ, Statistics Canada and Statistics the Netherlands (OECD, 2020[1]), there are several countries with innovative and extensive data collections for external researchers, organised around different access and warehousing protocols.

Some countries utilise statistics institutes to collate data and securely share it. Similar to Canada, Statistics Finland, the public institution with responsibility for statistics and data in Finland, collates, organises and links high-quality register data. It houses almost 160 sources of data which include ALMPs, income and tax, education, socio-economic status including detailed family status, occupation, health and education data, making it one of the most comprehensive sources of high-quality administrative data available. Researchers can apply for data and make use of its FIONA system to access the data securely via remote means. Bespoke datasets can also be created for researchers upon request, subject to extra charges and longer lead times. Users can also request that their own datasets be uploaded to the secure environment, so that they may make use of these in analysis.

The Federal Statistical Office (BFS) of Switzerland provides access to anonymised individual register data and provides data linking services between registers. As of 2016, BFS had 56 linkage agreements for statistical and longitudinal analysis with research institutions, federal and cantonal authorities and other organisations.

Stats NZ offers a range of linked administrative data using its Integrated Data Infrastructure and Longitudinal Business Dataset. Like Statistics Canada, researchers can use data in approved facilities. Costs for access and use of the datasets are low. There is an assessment fee of NZD 500 (waived for government and unsuccessful applications) and a fee of NZD 155 per hour for confidentiality checking of results (free for the first 15 hours) (Stats NZ, 2021[28]). Storage of data up to 200 GB is free and NZD 1.50 per GB per month after that. There is a six‑week cycle of application approvals, ensuring a quick turnaround to applications. As of November 2019, Stats NZ had over 600 researchers using its data, comprising over 250 projects. This has resulted in 116 evaluations of separate interventions on ALMPs (de Boer, 2019[29]).

Quasi-governmental autonomous bodies are used in other countries for making data accessible to researchers. Sweden’s Institute for Labour Market Policy Evaluation (IFAU) is a state‑owned research institute. It does not make any proposals or recommendations in its own reports. Its objective is to promote, support and carry out scientific evaluations on the labour market but this includes influencing the collection of data and making data easily available to researchers, both in Sweden and abroad.

France has sought to establish better public-private links with its external data centre. Its Centre d’Accès Sécurisé aux Données (CASD) is a public interest group bringing together public and private sector researchers, the State represented by INSEE, GENES, CNRS, École Polytechnique and HEC Paris and was created by ministerial decree in December 2018. Its main purpose is “to organise and implement secure access services for confidential data for non-profit research, study, evaluation or innovation, activities described as “research services”, mainly public. Its mission is also to promote the technology developed to secure access to data in the private sector”.2 Data are available from INSEE, the Ministries of Justice, National Education, Agriculture and Food, Economy and Finance. It has 400 data sources, over 3 000 users and has amassed some 400 publications and communications since its inception only three years ago.

3.6. Summary

The creation of a comprehensive data platform for analysis and the incorporation of CRA data into it has paved the way for ESDC to conduct high quality impact analysis of its ALMPs. This platform provides efficiency and stability as a basis for impact evaluations. Detailed information on participants’ characteristics and their outcomes in the labour market allows impacts of programmes to be thoroughly assessed. Information on past earnings and benefit receipt, alongside broader socio‑economic data, means that careful comparison is possible between participants and non-participants in ALMPs. ESDC makes a clever change to the comparison group for former employment insurance claimants, to ensure it is possible to construct a plausible counterfactual where the administrative data alone may not be sufficient. Individuals are disaggregated into sub-groups, which allows for a richer policy narrative to be developed on the impacts of ALMPs.

However there is still room for improvement to the data used that would permit even more colour to be given to policy assessment. Information on job type and tenure would allow more discussion of job quality. Income data that are more temporally disaggregated would allow better discussion of job counselling services. Better data on families and education would permit a more complete assessment of the impacts of ALMPs on parents and young people. Lags to the collection of income data also mean there is a constraint on feasible analytical timelines.

Availability of the data to conduct research is currently confined to internal ESDC analysts. Small changes to the availability of data via Statistics Canada would permit external researchers to conduct such research. This would allow ESDC to benefit from greater democratisation of its programme evaluation work, permitting greater innovation and cross-referencing of analysis, such as is already happening in countries which have made greater strides in this area.

References

[16] Aho, S. et al. (2018), Työvoimapalvelujen kohdistuminen ja niihin osallistuvien työllistyminen, https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/160639/19-2018-Tyovoimapalvelujen%20kohdistuminenpdf.pdf.

[21] Algan, Crépon and Glover (2020), “Are Active Labor Market Policies Directed at Firms Effective? Evidence from a Randomized Evaluation with Local Employment Agencies”, https://www.povertyactionlab.org/sites/default/files/research-paper/5484_Active-Labor-Markets-Evidence-from-Public-Service-France_January2020.pdf.

[19] Andersson, F. et al. (2013), “Does Federally-Funded Job Training Work? Nonexperimental Estimates of WIA Training Impacts Using Longitudinal Data on Workers and Firms”, NBER working paper series, http://www.nber.org/papers/w19446.

[20] Arni, P., R. Lalive and J. Van Ours (2012), “How effective are unemployment benefit sanctions? Looking beyond unemployment exit”, Journal of Applied Econometrics, Vol. 28/7, pp. 1153–1178, https://doi.org/10.1002/jae.2289.

[18] Bellemare, C., N. Kyui and G. Lacroix (2021), “Immigrants’ Economic Performance and Selective Outmigration: Diverging Predictions from Survey and Administrative Data”, SSRN Electronic Journal, https://doi.org/10.2139/ssrn.3811772.

[27] Card, D., J. Kluve and A. Weber (2018), “What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations”, Journal of the European Economic Association, Vol. 16/3, pp. 894-931, https://doi.org/10.1093/JEEA/JVX028.

[24] Cheung, M. et al. (2019), “Does Job Search Assistance Reduce Unemployment? Experimental Evidence on Displacement Effects and Mechanisms”, SSRN Electronic Journal, https://doi.org/10.2139/SSRN.3515935.

[12] de Boer and Ku (2018), Cost-effectiveness of MSD employment assistance, Ministry of Social Development, New Zealand, https://www.msd.govt.nz/documents/about-msd-and-our-work/publications-resources/research/employment-assistance-effectiveness/ea-effectivenes-report-07022019.pdf.

[13] de Boer and Ku (2018), Effectiveness of the Limited Service Volunteer programme in 2014/15, Ministry of Social Development, New Zealand, https://www.msd.govt.nz/documents/about-msd-and-our-work/publications-resources/evaluation/lsv-effectiveness/lsv-effectiveness.pdf.

[29] de Boer, M. (2019), Monitoring ALMP effectiveness using linked data: The New Zealand experience, https://www.oecd.org/employment/emp/S3.3.%20De%20Boer_NZL.pdf.

[22] Department of Social Protection, Ireland (2021), Working Paper: Employment Transitions of People Closing PUP Claims, https://assets.gov.ie/203173/a9f36c82-e218-464f-a464-897932ff8931.pdf.

[23] DWP (2021), Work and Health Progamme statistics: background information and methodology, Department for Work & Pensions, United Kingdom, https://www.gov.uk/government/publications/work-and-health-programme-statistics-background-information-and-methodology/work-and-health-progamme-statistics-background-information-and-methodology (accessed on 20 December 2021).

[25] DWP (2018), Weekly work search review trial, Department for Work & Pensions, United Kingdom, https://www.gov.uk/government/publications/jobseekers-allowance-weekly-work-search-review-trial (accessed on 26 November 2021).

[10] ESDC (2021), Analysis of Employment Benefits and Support Measures (EBSM) Profile and Medium-Term Incremental Impacts from 2010 to 2017, Employment and Social Development Canada (unpublished).

[4] ESDC (2021), Labour Market Program Data Platform - Data Dictionary v3.05a, Employment and Social Development Canada (unpublished).

[26] ESDC (2021), SSPB ED Quality assurance Methodology, Employment and Social Development Canada (unpublished).

[3] ESDC (2020), Horizontal Evaluation of the Youth Employment Strategy: Skills Link Stream, Employment and Social Development Canada, https://www.canada.ca/en/employment-social-development/corporate/reports/evaluations/horizontal-skills-link.html.

[9] ESDC (2019), Quantitative Methodology Report – Final, Employment and Social Development Canada (unpublished).

[8] ESDC (2017), Evaluation of the Labour Market Development Agreements: Synthesis Report, Employment and Social Development Canada, https://publications.gc.ca/site/eng/9.841271/publication.html.

[2] Gingras, Y. et al. (2017), “Making Evaluation More Responsive to Policy Needs: The Case of the Labour Market Development Agreements”, Canadian Journal of Program Evaluation, Vol. 32/2, https://doi.org/10.3138/cjpe.31119.

[5] Handouyahia, A. (2019), The creation of a rich data platform to support net impact evaluation of Labour Market Programmes, https://www.oecd.org/employment/emp/S4.5.%20Handouyahia_CAN.pdf.

[6] Heckman, J., R. Lalonde and J. Smith (1999), “The Economics and Econometrics of Active Labor Market Programs”, in Handbook of Labor Economics, Elsevier, https://doi.org/10.1016/s1573-4463(99)03012-6.

[7] Lechner, M. and C. Wunsch (2013), “Sensitivity of matching-based program evaluations to the availability of control variables”, Labour Economics, Vol. 21, pp. 111-121, https://doi.org/10.1016/J.LABECO.2013.01.004.

[17] Meyer, B., W. Mok and J. Sullivan (2015), “Household Surveys in Crisis”, Journal of Economic Perspectives, Vol. 29/4, pp. 199-226, https://doi.org/10.1257/jep.29.4.199.

[14] OECD (2021), OECD Employment Outlook 2021, OECD Publishing, Paris, https://doi.org/10.1787/5a700c4b-en.

[1] OECD (2020), “Impact evaluation of labour market policies through the use of linked administrative data”, OECD, Paris, https://www.oecd.org/els/emp/Impact_evaluation_of_LMP.pdf.

[11] Statistics Canada (2021), Time spent on unpaid care of a child in the household, by working arrangement and age of youngest child, Canada, 2010, https://www150.statcan.gc.ca/n1/pub/89-503-x/2010001/article/11546/tbl/tbl006-eng.htm.

[15] Statistics Finland (2022), Research Data Catalogue, https://taika.stat.fi/.

[28] Stats NZ (2021), Apply to use microdata for research, https://www.stats.govt.nz/integrated-data/apply-to-use-microdata-for-research/.

Notes

← 1.  https://www.statcan.gc.ca/en/microdata/data-centres/fees.

← 2.  https://www.casd.eu/en/le-centre-dacces-securise-aux-donnees-casd/gouvernance-et-missions/.