Getting Skills Right: Future-Ready Adult Learning Systems

Annex C. Methodology and data sources

Theoretical framework

The conceptual framework of the Priorities for Adult Learning (PAL) Dashboard was developed by the OECD based on a literature review and expert opinion. It aims to assess the readiness of adult learning systems to respond to the challenges of changing skill needs in OECD countries. It highlights key priorities to reduce skill imbalances while ensuring access to high-quality adult learning for everyone. To this effect, the Dashboard encompasses seven dimensions, 18 sub‑dimensions and 52 indicators. The dimensions reflect seven major aspects of the readiness of adult learning systems to address changing skill needs:

1. Urgency of training need, which summarises a range of contextual factors relevant to the skills development needs of the adult population. While adult learning is an important policy area for all countries, some countries face greater pressure to update the skills of their adult population based on their specific demographic, technological or educational context. This dimension includes indicators on population ageing, automation and structural change, adult skill levels, as well as data on globalisation.

2. Financing, which assesses the degree to which investments are made at individual, employer and public level, and to what extent costs of training constitute a limiting factor to employers’ provision and individuals’ participation. Sufficient levels of investment in adult learning are key to inclusive and high quality provision. This dimension includes the sub-dimensions government, employer and individual.

3. Coverage, which captures the level and intensity of participation in and provision of training activities by both individuals and firms. Adult learning systems can only address changing skill needs, where they involve significant parts of the adult population in updating their skills. The sub-dimension relative to individuals measures the incidence of participation, the number of average training hours, as well as time trends in participation. The sub-dimension employers measures the share of enterprises that provides training to workers, the training intensity and time trends.

4. Inclusiveness, which assesses the extent to which different groups of the population take part in adult learning to similar degrees. Research shows that those with greater need to update their skills, e.g. the low-skilled or mature-age adults, are less likely to take part in adult learning. To improve the readiness of countries to address changing skill needs, participation in adult learning must be inclusive and involve those most in need of training. This dimension analyses the gap in participation of disadvantaged groups, namely older workers, women, adults with low skills and those with low wages (sub-dimension socio-demographic characteristics); of the unemployed and long-term unemployed, temporary workers and workers in SMEs (sub-dimension employment and contract status).

5. Perceived impact, which includes some aspects of the perceived usefulness and effectiveness of training participation. There are a variety of aspects of impact of adult learning which are difficult to capture using quantitative data, and this dimension is therefore limited to measurable aspects of perceived impact of training. It assesses the self-reported usefulness and effectiveness of training as measured by the satisfaction of learners, the effectiveness of adult learning in terms of producing useful skills and improving labour market outcomes, as well as wage returns to participation in adult learning

6. Alignment with skills needs, which captures the extent to which the provided adult learning is directly relevant to address current and future skill needs. In particular, this dimension looks at labour market imbalances, whether firms assess future skill needs, the extent to which training is provided in response to the identified needs, and the participation in training by workers at risk of skills obsolescence.

7. Flexibility and guidance, which summarises in how far there is sufficient information on existing adult learning provision and the extent to which training is provided in a flexible manner. Many people face a variety of barriers to access adult learning opportunities, including a lack of information, time and distance constraints. Addressing these barriers can have important effects on participation levels in adult learning. This dimension includes indicators on the extent to which time and distance constitute a barrier to participation, the availability of distance learning and the availability and use of guidance on adult learning.

To the extent possible, the indicators in the dashboard focus on job-related adult learning activities. Job-related training activities are defined not only to refer to a specific job, but also to include training activities that improve employment chances more generally.

Data selection

The quality of any dashboard is crucially dependent on the availability of high quality data with appropriate country coverage. The data sources used to develop the dashboard are:

• Continuing Vocational Training Survey (CVTS), a long-running enterprise survey on continuing vocational training and other training in enterprises (excluding micro-enterprises). The survey is part of the EU statistics on lifelong learning and covers all EU Member States and Norway. Latest data available refers to 2015 (fifth wave).

• European Adult Education Survey (AES), a regular household survey covering persons between 25 and 64 years old and their participation in education and training. The survey is part of the EU statistics on lifelong learning and covers 35 countries, including all EU Member States, Albania, Bosnia-Herzegovina, Former Yugoslav Republic of Macedonia, Norway, Switzerland, Serbia and Turkey. Latest data available refers to 2016 (third wave).

• OECD and Eurostat data on public spending on active labour market policies. Data collection takes place yearly and the latest data available refers to 2015/2016.

• Survey of Adult Skills (PIAAC) data, a household survey covering adults aged 16‑65 and assessing their key cognitive and workplace skills, as well as skill use at work. This OECD survey has covered 38 OECD and partner countries in three rounds. Round one took place in 2008‑13 and collected data in Australia, Austria, Belgium (Flanders), Canada, Czech Republic, Denmark, Estonia, Finland, France, Germany, Ireland, Italy, Japan, Korea, Netherlands, Norway, Poland, Russian Federation, Slovak Republic, Spain, Sweden, United Kingdom (England and Northern Ireland) and the United States. Round two took place in 2012‑16 and collected data in Chile, Greece, Indonesia, Israel, Lithuania, New Zealand, Singapore, Slovenia and Turkey. Data collection for round three of the 1st cycle of PIAAC is currently on the way in Ecuador, Hungary, Kazakhstan, Mexico, Peru and the United States, however data is not yet available.

• UN world population prospects data, for the demographic indicators used in the ‘Urgency of training need’ dimension

In addition, country level surveys were used to fill some data gaps. These are the Australian Works-related Training and Learning survey (WRTAL), the Chilean Labour Survey (ENCLA), the Japanese Basic Survey of Human Resource Development, the New Zealand Business Operations Survey, and the Turkish Continuing Vocational Training Survey.

The data selection for the OECD Priorities for Adult Learning Dashboard respected four criteria:

• Coverage: Coverage of the Priorities for Adult Learning Dashboard was driven by considerations on data availability. Countries are included in the dashboard if data on them is available in at least one of the three major data sources AES, CVTS and PIAAC. All available data is presented using individual indicators. The aggregation of the indicators into sub-dimensions is only implemented where data is available for at least 50% of the indicators in a sub-dimension. Similarly, the aggregation of sub-dimensions into the overarching dimension is only done when at least 50% of sub-dimensions are non-missing. Countries covered by the dashboard include OECD countries, namely: Australia; Austria; Belgium; Canada; Chile; Czech Republic; Denmark; Estonia; Finland; France; Germany; Greece; Hungary; Ireland; Israel; Italy; Japan; Korea; Latvia; Lithuania; Luxembourg; Netherlands; New Zealand; Norway; Poland; Portugal; Slovak Republic; Slovenia; Spain; Sweden; Switzerland; Turkey; United Kingdom; United States.

• Relevance and quality: Data that was relevant to assess the performance of adult learning systems, as suggested in the relevant literature. Out of the relevant data, only data which had undergone rigorous quality control was selected for inclusion. All included data represents the best measure of a domain currently available.

• Accessibility: Data was publicly available or available upon request (microdata).

• Timeliness: The most up-to-date datasets available, with data selected from 2011 to 2016.

Imputation of missing data

It is likely that imputations are used in the underlying microdata (PIAAC, AES, CVTS). It was deemed inappropriate to impute values in the dashboard and the issue of missing observations was dealt with at the aggregation stage.

Normalisation

Normalisation was carried out in order to make the indicators comparable. The normalisation was implemented as a step-wise process:

1. Outliers were identified using criteria of both the skewness and kurtosis of the distribution. Identified outliers were replaced by the second largest (or smallest) observation in the sample to achieve a normal distribution (i.e. winsorising). Following this approach, only one outlier was identified and adjusted across all indicators.

2. Normalisation was implemented using the min-max method, i.e. y_i=(x_i-min⁡x)/(max⁡x-min⁡x). This resulted in indicators that were mapped onto a uniform scale, where 0 corresponds to the minimum and 1 to the maximum. This method can widen the distance of the observations compared to other normalisation methods, which gives more ‘power’ to the final composite index. Sensitivity tests were applied to test the impact of different normalisation methods (see below).

Weighting and aggregation

The weighting and aggregation was carried out according to the theoretical framework. Aggregation was implemented for each dimension separately and countries are ranked in each of the seven different domains. The domains highlight important aspects of the readiness of adult learning systems, in which better performance benefits society through better aligned skill demand and supply. No overall aggregation into a final index of “future-readiness” was made.

The approach to aggregation was driven by the fact that each dimension aims to capture complex, often multidimensional concepts (such as quality of training or alignment with skill needs). The multi-dimensionality was expressed through the introduction of sub-dimensions. Each sub-dimension has equal weight in the aggregation process, as they are considered equally important determinants of performance in a given dimension. Meanwhile almost all of the sub-dimensions consists of multiple (2‑4) indicators, in order to capture them in the most comprehensive way, given data and measurement constraints.

As mentioned above the dataset is not complete, i.e. not all of the indicators are available for all the countries and imputation is not possible. The issue of missing observations was treated at the aggregation stage.

Hence, weighting and aggregation is carried out as a two-step process:

1. Equal weights were assigned to each indicator and data was aggregated as the sub-dimension level. Addressing the issue of missing data, data had to be available for at least half of the indicators in a sub-dimension for a country to receive an aggregate score.

2. Equal weights were assigned to each sub-dimension and data was aggregated at the dimension level. Addressing the issue of missing data, data had to be available for at least half of the sub-dimensions in a dimension for a country to receive an aggregate score.

As a result the final score is a simple average of the sub-dimensions; but typically not the indicators themselves.

Using arithmetic averages allows some compensability between the components. This means that countries are not ‘punished’ if they have a very low score in a given indicator or sub-dimension (as opposed to using geometric average or the multi-criteria method). This acknowledges that there are various possible ways for countries to do well in a given dimension.

Uncertainty and sensitivity analysis

Various robustness tests were carried out to test the validity of the indicator choices, normalisation method, weighting and aggregation methods. It was found that overall, the final rankings of the countries in the various dimensions are robust to the methodological choices.

Indicator selection

Multivariate analysis was carried out to examine the underlying structure of the data and confirm that the indicator, sub-dimension and dimension choices made. This included factor analysis, Cronbach alpha coefficient (c-alpha) and pairwise correlations of the sub-dimensions as well as indicators within the dimensions. The analysis found that the framework of the dashboard was appropriate.

Looking at the relationship between dimensions and sub-dimensions, the analysis found that sub-dimensions were strongly and positively correlated within each dimension. Two exceptions to this were the urgency dimensions, which intends to describe a context of loosely connected forces, and financing, which intends funding sources which in many cases are complimentary (rather than correlated).

Turning to the relationship between dimensions and individual indicators, the analysis showed that individual indicators within the dimensions are not overly correlated. Only around 10% of the indicator-pairs have stronger than +/- 0.5 correlations (without the urgency dimension). The correlation analysis confirmed that indicators included in each dimensions were indeed measuring different aspects of an overall concept and double-counting was avoided.

Results of the Factor Analysis and Cronbach-Alpha indicate that some variables could be omitted from certain dimensions if internal consistency of each index was to be maximised. It was however decided to not omit any variables as the indicators concerned were considered important aspects of performance in their dimensions.

Normalisation method

Two further normalisation methods were tested, namely z-score normalisation and ranking aggregation. Different normalisation techniques produced highly similar results.

Compared to the chosen min-max method, z-score normalisation compresses the distribution of the data points, while using rankings further prevents outliers from influencing the results. Regardless of these differences country rankings achieved through different normalisation methods correlated in almost all cases higher than 0.9. Only in case of the financing dimension were the correlations slightly lower, 0.76 for the Z-score method and 0.78 for the ranking method.

Aggregation method

The chosen aggregation method (aggregation method A) assigns a rank to a country if:

• It has data for at least half of the sub-dimensions within a dimension; and

• It has data for a least half of the indicators within each sub-dimension.

Three alternative aggregation methods were examined to analyse the robustness of the results to the choice of aggregation method:

1. Aggregation method B: at least one indicator has data within the sub-dimension to receive an aggregated score per sub-dimension; at least half of the sub-dimensions have data to receive a final aggregated score per dimension.

2. Aggregation method C: at least half of the indicators have data within the sub-dimension to receive an aggregated score per sub-dimension; at least one of the sub-dimensions have data to receive a final aggregated score per dimension.

3. Aggregation rule D: more than half of the indicators within the sub-dimension have data to receive an aggregated score per sub-dimension; more than half of the sub-dimensions have data to receive a final aggregated score per dimension.

Aggregation rules B and C produce close to identical results to Aggregation rule A with regards to the ranks. Even the strictest criterion (Aggregation rule D) has a higher than 0.9 correlation with Aggregation rule A, except for the Flexibility and Guidance dimension (0.8). At the same time it should be noted, that when using this more demanding rule the number of countries included in the dashboard shrinks by a quarter.

Lastly, equal weighting of all the indicators within the different dimensions would result in highly similar country ranks in every case (above 0.9). This suggests that the final rankings are not subject to weighting decisions.