OECD assessed several case study interventions to promote integrated care. Together, the case studies cover several OECD and non-OECD European countries.
Selected case studies represent strategic, high-priority interventions among policy makers in the OECD and EU27.
Note: *The case study may operate across the country or in specific regions within that country. **Interventions evaluated using OECDs Strategic Public Health Planning for NCDs model.
This section outlines two complementary frameworks used to assess case studies, both of which were developed by the OECD – the Best Practice Framework and the Transferability Framework. Limitations associated with the analysis are also discussed.
The Best Practice Framework outlines five criteria to assess whether an intervention is “best practice” – namely Effectiveness, Efficiency, Equity, Evidence‑base, and Extent of coverage (Table A A.2). A review of the academic and grey literature, existing best practice frameworks and feedback from delegates to OECD’s expert Group on the Economics of Public Health informed the selection of criteria.
An intervention can be awarded a “stamp of approval” against one or multiple criteria if it performs particularly well relative to similar interventions.
Up and coming interventions (i.e. those that show promise but have not yet collected any of their own data) can be awarded a “promising best practice” stamp of approval for relevant criteria.
Public health interventions are complex given they involve multiple stakeholders, often target heterogeneous groups, and have outcomes affected by various direct and indirect factors. Therefore, positive outcomes achieved in one setting aren’t necessarily transferable to a different setting.
OECD has developed a Transferability Framework to assist policy makers assess whether a best practice intervention can be transferred from where it has been implemented (i.e. best practice “owner setting”) to a different country/region (i.e. the “target setting”). Specifically, whether the desired outcomes achieved in the owner setting are achievable in the target setting (Trompette et al., 2014[]; Burchett, Umoquit and Dobrow, 2011[]).
The Transferability Framework includes four contextual factors that affect transferability:
Population context: covers population characteristics such as sociodemographic factors as well as broader cultural considerations
Sector specific context: covers governance/regulation, financing, workforce, capital and access arrangements in the sector the intervention operates
Political context: political will from key decision-makers to implement the intervention
Economic context: the affordability of the intervention in the target setting.
In each case study, indicators to assess transferability are grouped under one of these four contextual factors. For the case studies presented in this document, countries are allocated into a group based on how far the indicator’s value is from the best practice owner setting. This method is referred to as the “distance from reference country” and is explained in Box A A.1. In addition, OECD developed a clustering methodology to group countries according to their potential to transfer a best practice intervention (Box A A.2).
Indicators were sourced from international databases to maximise coverage across OECD and non-OECD European countries (e.g. OECD Stat, Eurostat, World Bank Indicators, and the WHO). Relevant indicators were excluded if data was missing for the best practice owner setting and could not be identified through desktop research, or, if more than 50% of data was missing across countries.
By using international data, the scope of the analysis was inevitably limited – i.e. indicators from international sources are high-level and don’t cover all relevant information for assessing transferability. Therefore, each case study also includes a set of “new indicators” (i.e. those with no publicly available information) policy makers should consider before transferring the intervention.
Quantitative indicator values have been normalised using distance to a reference country, that is, the country in which the best practice intervention is currently implemented (also referred to as the best practice “owner” setting) (OECD/European Union/EC-JRC, 2008[]).
The normalisation equation is below:
(Equation 1)
Where:
= normalised value for target setting (country c) for indicator i
= original value for target setting (country c) for indicator i
= original value in the owner setting for indicator i.
Normalised values for equation (1) can be interpreted as percentage distance each country is from the best practice owner setting, whose value is centred on 0. Normalised values were used to allocate countries into one of five groups for each indicator, with a darker shade indicating greater transferability potential:
Value equal or greater than 0 = 
Value less than 0 but greater than ‑25% = 
(+25% when a lower value indicates better transferability)
Value less than ‑25% but greater than ‑50% = 
(>+25% but less than <+50%)
Value less than ‑50% but greater than ‑75% = 
(>+50% but less than <+75%)
Value less than ‑75% = 
(>75%)
For binary indicators, countries that respond ‘Yes’ to the indicator are allocated the darkest shade (
) while countries that respond ‘No’ are allocated the lightest shade (
).
For categorical indicators, any country that responds at least as well as the best practice owner are allocated the darkest shade (
), while the remaining countries are allocated a lighter shade based on the number of remaining categories.
OECD has developed a methodology to cluster countries and to make personalised recommendations on which member states and member countries are more likely to successfully transfer a recognised best practice intervention. A high-level summary of the clustering methodology is below.
Cluster analysis partitions data into homogenous groups, based on similarities in the data. In this case it was used to separate countries into groups with similar characteristics, based on how well adapted or suited they are for transfer of a best practice intervention from a host country. For each cluster, specific recommendations can then be made to address potential obstacles for implementation. This can help guide decision makers and potentially lead to the smoother implementation and increased success of interventions.
To select the best methodology, four different cluster methods were compared: k-means, k-medoids, hierarchical and DBSCAN (Density-Based Spatial Clustering of Applications with Noise). K-medoids using Gower distance was found to be the most effective method for clustering countries taking into account validation statistics, data characteristics, interpretability of the results and flexibility to use with other datasets. This is because it works with small, imbalanced datasets with missing data, and can accommodate categorical data as well as continuous data.
The k-medoids algorithm is based on the medoid: this is the most central observation (country in this case) in the cluster, where the total distance between it and all the other countries in the cluster is smallest. Distance is a quantitative measure of dissimilarity, where the larger the distance between two observations, the more different they are from each other. The number of clusters (k) must be chosen prior to running the algorithm.
The k-medoids algorithm has the following steps:
Randomly assign k countries as medoids
Repeat until there is no change in assignment of medoid
Assign each country to a cluster, based on distance to the closest medoid:
For each cluster, test whether selecting another country as the medoid decreases the total distance from the medoid to all other points in the cluster. If it does, reassign this country as the new medoid.
Gower distance was chosen because it is able to compute the difference between both categorical and continuous variables. Gower distance is calculated from the mean of the partial pairwise distances between observations (countries). The partial pairwise distance is the difference between two observations at a single variable and is calculated differently depending on whether the variable is continuous or categorical.
Continuous Variables: The partial pairwise distance, between two observations and , for variable is the difference between the two values and , divided by the maximal range () of all the values for variable , as follows:
Categorical Variables: If two countries have the same value for a categorical variable then the partial pairwise distance is 0 (identical). Otherwise, it is 1.
The Gower distance between two observations is then calculated as the mean of the partial pairwise distances. The partial pairwise distances can be weighted differently. Here, the variables were weighted so that each contextual factor had equal weighting and therefore equal influence on the Gower distance. The resulting value lies between 0 and 1, with values closer to 0 indicating greater similarity between countries and values closer to 1 indicating greater dissimilarity. If one or both values are missing for a given variable in a pair of countries, the partial distance for that variable will not be included in the Gower distance, meaning there is no need for data imputation. However, if a country had over 50% variables missing it led to inaccurate Gower distances and so these countries were removed.
The clusters were compared by calculating the difference between the mean of each cluster and the mean of the dataset, for each indicator. A positive difference meant a higher likelihood of successful transfer for that indicator, allowing the characteristics of each cluster to be identified. To more broadly compare clusters, identifying the contextual factors (or domains) where clusters were stronger or weaker, domain scores were created and used to compare cluster means. Domain scores were created using the following steps:
Assign categorical variables dummy values (0 = no, 1 = yes).
Normalise using min-max scaling.
Aggregate by the mean of the variables in each contextual factor.
In summary, the following steps are required:
Remove countries where >50% variables are missing.
Compute a Gower Distance Matrix, with each contextual factor having equal weighting.
Determine optimal value of clusters (k) between 3 and 5.
Run k-medoids clustering using the optimal number of clusters from step 3.
Create domain scores in order to compare cluster means with the dataset means, and identify strength and weakness of each cluster.
[2] Burchett, H., M. Umoquit and M. Dobrow (2011), How do we know when research from one setting can be useful in another? A review of external validity, applicability and transferability frameworks, https://doi.org/10.1258/jhsrp.2011.010124.
[3] OECD/European Union/EC-JRC (2008), Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Publishing, Paris, https://doi.org/10.1787/9789264043466-en.
[1] Trompette, J. et al. (2014), “Stakeholders’ perceptions of transferability criteria for health promotion interventions: A case study”, BMC Public Health, Vol. 14/1, pp. 1-11, https://doi.org/10.1186/1471-2458-14-1134.