Step 1 outlines the process for selecting best practice interventions. It is broken into two sequential sub-steps: Step 1a – identify best practice interventions; and Step 1b – assess transferability of a best practice intervention to a different context (Figure 1.1).
Identifying and assessing the transferability of a best practice intervention are key preliminary steps to the implementation process discussed in Step 2 of this guidebook.
Note: *Details on developing an evaluation study are however discussed in detail in Step 3 (“Evaluate” element of the cycle).
To assist policy makers identify best practice interventions in public health, OECD has developed a high-level multi-criteria decision analysis (MCDA) framework – the OECD Best Practice Identification Framework (hereafter, the OECD Framework). An MCDA framework incorporates a range of criteria and aligning indicators relevant for evaluating a suite of alternative interventions. MCDA frameworks are a useful evidence‑based decision-making tool given they provide a systematic approach to assessing the performance of interventions (Devlin and Sussex, 2011[3]; Thokala et al., 2016[4]).
The growing use of MCDAs in several sectors, including health, reflect the range of benefits this tool provides, as summarised in Box 1.1. MCDAs are particularly relevant for complex public health interventions as their impact cannot be measured by a single quantitative metric – i.e. a broader approach to assembling and analysing evidence is needed (Ng and de Colombani, 2015[2]; Rychetnik et al., 2002[5]).
The benefits associated with using MCDAs to evaluate interventions and improve the decision-making process include:
Transparency: criteria used to make decisions are explicit, as are the trade‑offs between criteria (if weights are applied)
Objectivity: by making the decision-making process explicit, decisions are made objectively and consistently
Complexity: multiple, competing criteria, better reflect real-world decisions
Decision convergence: engaging stakeholders with differing opinions to design the MCDA framework can facilitate decision convergence
Data collection: MCDAs highlight criteria considered of most importance, and thus areas where data collection efforts should be enhanced
Source: Mühlbacher and Kaczynski (2016[6]), “Making Good Decisions in Healthcare with Multi-Criteria Decision Analysis: The Use, Current Research and Future Development of MCDA”, https://doi.org/10.1007/s40258-015-0203-4; Phelps and Madhavan (2017[7]), “Using Multicriteria Approaches to Assess the Value of Health Care”, https://doi.org/10.1016/j.jval.2016.11.011; Mandelblatt et al. (2017[8]), “Evaluating Frameworks That Provide Value Measures for Health Care Interventions”, https://doi.org/10.1016/j.jval.2016.11.013.
The remainder of Step 1a outlines the process for identifying a best practice intervention using an MCDA approach. There are various methods for undertaking MCDAs, however, based on a review of the literature, the following high-level methodology was deemed most appropriate for public health interventions (Thokala et al., 2016[4]).
The first step in undertaking an MCDA is to define the decision problem (Frazão et al., 2018[9]; Thokala et al., 2016[4]). This involves confirming at the outset:
What is the objective of undertaking the MCDA assessment? For the purposes of this guidebook, the objective is to define public health interventions considered best practice. Policy makers may also wish to use the MCDA assessment to rank best practice public health interventions.
Which public health interventions are going to be assessed? The types of public health interventions under consideration should be confirmed upfront given this will determine the final design of the decision framework (e.g. the indicators used).
What are the practical constraints in undertaking the assessment? It is important to confirm the budget and timeline as these constraints will affect the design of the MCDA assessment (e.g. are there enough funds to collect new data? What is the timeframe for collecting participant data?)
Which stakeholders should be involved? It is important to identify stakeholders whose input into the decision-making process is considered relevant and of high-value (e.g. defining indicators and the relative importance of criteria). Engaging key stakeholders early on also enhances “buy in”, which is critical to success (see Box 1.2 for information on identifying stakeholders). It is important to note that the list of stakeholders in Step 1 can differ from those involved in implementing and evaluating a best practice intervention. Further details on stakeholder engagement can be found in Step 2 (Implement), Section 2.2 of the guidebook.
To identify stakeholders, start by “brainstorming” those whose value judgements are relevant to the decision problem (SELFIE 2020, 2017[10]). Types of stakeholders may include those responsible for implementing, participating or paying for the intervention. Further, given the technical nature of identifying best practice interventions, policy makers may also wish to include academics with experience undertaking similar analyses.
Engaging the right stakeholders from the outset is important given it will lead to a higher quality assessment. Further, it can enhance the legitimacy of decisions made by policy makers thereby boosting public and professional acceptability.
The final list of stakeholders should be diverse and, to the extent possible, focus on those who are underrepresented. Further, the list should avoid dominant stakeholder groups whose input may disproportionality shape decisions.
Further details on engaging stakeholders can be found in Step 2 (Implement), Section 2.2.
Once the decision problem has been confirmed, the next step is to define the criteria to assess whether an intervention is a best practice. Drawing upon a review of the literature, expert feedback as well as existing public health MCDA frameworks, the OECD Framework applies five criteria – the five E’s: Effectiveness, Efficiency, Equity, Evidence‑base and Extent of coverage as described in Table 1.1.
To measure performance against each criterion, a range of indicators have been proposed. The list of indicators is intentionally broad in order to capture a wide range of public health interventions. Therefore, policy makers are encouraged to only include indicators relevant to assessing the intervention(s) under examination. This will simplify the process and avoid unnecessary data collection and analysis.
For further methodological details on how criteria and indicators were chosen, see Annex A.
Note: Health Delegates to the OECD voted on the importance of criteria in 2019. Results from the vote can be found in Annex A.
In the OECD Framework, effectiveness is defined as the extent to which an intervention’s desired outcomes1 (objectives) were achieved in a real-world setting (RE-AIM, 2020[11]; National Institute for Health and Care Excellence, 2020[12]; Productivity Commission, 2020[13]). Outcomes refer to the tangible impact associated with an intervention and can be classified as either intermediate or final. As an example, an intervention targeting obesity may measure fruit and vegetable consumption as an intermediate outcome and changes in body-mass index (BMI) as a final outcome (discussed further in Step 3 (Evaluate), Section 3.2.2) (Productivity Commission, 2020[13]; Mandel, Derx and Ilzkovitz, 2008[14]).
When assessing effectiveness, it is important to examine whether the intervention led to any unintended negative consequences as these may reduce or outweigh benefits (RE-AIM, 2020[11]). For example, a cycling to work scheme may increase physical activity levels, as intended, but also the number of injuries and emergency hospital visits.
Policy makers shouldn’t necessarily disregard public health interventions with limited evidence given outcomes can take years to emerge. For this reason, policy makers are encouraged to consider interventions that have not yet been fully evaluated but which have a strong theoretical underpinning. Such interventions can be considered “promising” as they have the potential to become a best practice (Spencer et al., 2013[15]; Canadian Public Health Association, 2020[16]).
Finally, when assessing the effectiveness of different interventions, it is important to consider the timeframe in which outcomes are expected to be achieved. For example, school-based obesity interventions can look unfavourable against those targeting adults given diseases associated with obesity (e.g. cardiovascular diseases) don’t usually appear until middle adulthood and therefore are unlikely to be captured within the evaluation time period. This is supported by previous OECD analyses which showed school-based obesity interventions are effective once a sufficient number of children exposed to the intervention are adults who are susceptible to developing chronic diseases (OECD, 2019[1]). Conversely, interventions to prevent complications for patients with diabetes may produce benefits in a shorter timeline since they target older, sicker patients, but are likely to cover a smaller share of the population and, in the long-run, may have a lower effectiveness.
Indicators to measure effectiveness are split into universal health-related indicators and intervention specific indicators (Box 1.3). The former can be used to measure the relative effectiveness of interventions with different objectives by using a common outcome measure, such as quality-adjusted life years (QALYs). Although widely used, the validity and reliability of QALY measurements is often questioned – for example, alternate methods to measure QALYs can lead to markedly different results (e.g. standard gamble versus EQ‑5D questionnaire), further, methods to collect data are not necessarily straightforward for patients leading to “nonsensical” results (Pettitt and Raza, 2016[17]). Instead, intervention specific indicators can be collected, which may be more appropriate. Intervention specific indicators can also be used as inputs to measure universal health-related indicators. For example, a change in BMI can be translated into a reduction in the relative risk of developing diabetes, which together with disability weights (a weight factor representing the severity of a disease) can be used to measure the change in disability-adjusted life years (DALYs). The OECD has developed a tool – the OECD SPHeP-NCDs (Strategic Public Health Planning for NCDs) microsimulation model – which can be used to carry out this type of analyses (For further details from OECD’s SPHeP-NCDs model, see http://oecdpublichealthexplorer.org/ncd-doc/_2_1_Modelling_Principles.html).
Example effectiveness indicators are listed below. These indicators have been chosen given they are internationally accepted and routinely collected in a standardised way. Users of this guidebook, however, should add/remove indicators as necessary (as is the case with indicators for all other best practice criteria).
A non-exhaustive list of indicators measuring effectiveness according to an intervention’s objective(s) are in Annex B. This list covers interventions targeting public health issues such as key risks factors and chronic conditions.
Source: OECD Health Statistics (2017[18]), “Health Status”; Sassi (2006[19]), “Calculating QALYs, comparing QALY and DALY calculations”, https://doi.org/10.1093/heapol/czl018; European Commission (2021[20]), “Public Health Best Practice Portal”, https://webgate.ec.europa.eu/dyna/bp-portal/.
Efficiency in the OECD Framework refers to how well the mix of monetary and non-monetary inputs were used to achieve desired outcomes in a real-world setting (European Commission, n.d.[21]) (Productivity Commission, 2020[13]).
Efficiency indicators use effectiveness indicators and measure them in relation to inputs used. Therefore, efficiency indicators have also been broken into those that are universal or intervention specific (see Box 1.4).The economic evaluation methods used to measure efficiency indicators are discussed in Step 3 (Evaluate), Box 3.7.
Data for economic evaluations is often difficult to obtain for public health interventions (CHRODIS, 2015[22]). Therefore, interventions with limited efficiency data shouldn’t automatically be disregarded (e.g. they could instead be classified as a promising intervention).
Efficiency indicators measure how well resources were used to achieve desired outcomes. Universal indicators can be used when the outcome is expressed in a common unit (e.g. QALY), and are useful when comparing different types of interventions. Conversely, intervention efficiency indicators compare resources to intervention specific outcomes.
Incremental cost-effectiveness ratio – i.e. cost per QALY gained or cost per DALY averted
Cost-benefit ratio (where costs and benefits are both expressed in monetary terms)
Incremental cost-effectiveness ratio for specific health outcome – e.g. cost per 1‑point reduction in BMI / 1cm reduction in waist circumference
Equity measures whether an intervention reduces inequalities in society and is a basic requirement of any public health intervention (Ng and de Colombani, 2015[2]; European Commission Directorate General for Health and Food Safety, 2017[23]).
Measuring the impact of an intervention on inequality involves an assessment of two key concepts – access and outcomes – across different population groups (Lehne et al., 2019[24]). That is, did access to, and outcomes from, the intervention differ across population groups (NEJM, 2020[25]). Population groups of interest are those that are underserved and therefore at greater risk of adverse health outcomes – i.e. priority population groups (see Table 1.2 for examples) (Public Health Ontario, 2015[26]). In addition to priority population groups, it is important to analyse results by age and gender.
Source: Walker, Strom Williams and Egede (2016[27]), “Influence of Race, Ethnicity and Social Determinants of Health on Diabetes Outcomes”, https://doi.org/10.1016/j.amjms.2016.01.008; Snijder (2017[28]), “Cohort profile: Healthy Life in an Urban Setting (HELIUS) study in Amsterdam, the Netherlands”, http://dx.doi.org/10.1136/bmjopen-2017-017873; NEJM (2020[25]), “Social Determinants of Health (SDOH)”, https://catalyst.nejm.org/doi/full/10.1056/CAT.17.0312; Phillips (2005[29]), “Defining and measuring gender: A social determinant of health whose time has come”, https://doi.org/10.1186/1475-9276-4-11.
Interventions that reduce heath inequalities may not necessarily be the most efficient, that is, there may be an equity-efficiency trade‑off (Cookson, Drummond and Weatherly, 2009[30]). For example, it may be more efficient to implement a public health campaign in a highly populated area to maximise participation, however, this would be at the expense of those living in regional/rural areas. Consequently, the gap in access between the two groups would widen. Policy makers are encouraged to take this trade‑off into account when making decisions to ensure an equitable distribution of resources.
Indicators to measure an intervention’s impact on reducing societal inequalities are outlined in Box 1.5.
Differences in effectiveness indicators across population groups (see Box 1.3)
The needs of priority population groups were considered when designing the intervention
A process was used to identify priority populations who would benefit from the intervention
Source: European Commission (2021[20]), “Public Health Best Practice Portal”, https://webgate.ec.europa.eu/dyna/bp-portal/; Cookson et al. (2016[31]), “Healthy Equity Indicators for the English NHS: a longitudinal whole‑population study at the small-area level”, https://doi.org/10.3310/hsdr04260; CHRODIS, (2015[22])“Task 1: selecting JA-CHRODIS criteria to assess good practice in interventions related to chronic conditions”, http://chrodis.eu/wp-content/uploads/2016/03/Delphi-2_MULTIMORBID.pdf.
The evidence‑base criterion reflects the quality of evidence to develop or evaluate an intervention. Assessment of the former (i.e. to develop an intervention) is important for identifying promising interventions for which evidence is still limited. Conversely, assessment of the latter (i.e. to evaluate an intervention) is important for checking the robustness of outcome evaluations for established interventions. It is a cross-cutting criterion given it relates to the quality of evidence on outcomes which are used in effectiveness, efficiency and selected equity indicators.
Guides to assess the level of evidence in terms of outcomes classify types of evidence by their potential to eliminate bias and establish causality. Therefore, they are an important tool for assessing the strength and validity of outcome evaluation results.
Randomised controlled trials (RCTs), which primarily focus on individuals, are considered the “gold standard” (see Box 3.5 in Step 3 (Evaluate) for further details on RCTs) (Rychetnik et al., 2002[5]). RCTs, however, are often not applicable to public health interventions. For example, it may not be possible to create a control group if it is unethical or impossible to exclude people from accessing an intervention. RCTs can also be expensive to carry out and are associated with their own limitations (e.g. selection bias of those who receive treatment) (Deaton and Cartwright, 2018[32]). For these reasons, it is important to consider other forms of data collection and analyses better suited for public health such as observational studies (Barnish and Turner, 2017[33]).
High-quality evidence is more readily available for easy-to-reach populations such as those living in cities with access to the internet and mobile phones (Rychetnik et al., 2002[5]). Conversely, data from hard-to-reach or “hidden” populations (e.g. homeless, migrants, lower socio‑economic status) (Bonevski et al., 2014[34]), can be difficult to obtain thereby downgrading the quality of evidence for interventions targeting these populations (Rychetnik et al., 2002[5]). For this reason, flexibility when assessing the evidence‑base of an intervention is needed to ensure an equitable distribution of public health resources (Rychetnik et al., 2002[5]).
Evidence‑based indicators have been classified into a three‑tiered hierarchy (Box 1.6). The evidence‑base hierarchy is designed to provide policy makers with a rapid appraisal of a study’s internal validity (i.e. the extent to which outcomes achieved can be attributed to the intervention). For each level of evidence, systematic reviews or meta‑analyses combining multiple studies are preferable to a single intervention study.
Studies using the same design are not necessarily of the same quality. Therefore, a second assessment of a study’s internal validity is recommended (e.g. by examining whether the research design exhibits any signs of bias such as selection, performance, detection, attrition and/or reporting bias (Berkman et al., 2014[35])). The Quality Assessment Tool for Quantitative Studies developed as part of the Effective Public Health Practice Project provides a standardised method for assessing the quality of public health intervention studies (see https://merst.ca/wp-content/uploads/2018/02/quality-assessment-tool_2010.pdf). The tool is designed as a questionnaire (intended to take between 30‑60 minutes to complete) and consists of eight components to assess study quality. Results from the questionnaire allow users to rate the quality of evidence as either strong, moderate or weak (Effective Public Health Pratice Project, 1998[36]). Another option is to use the GRADE (Grading of Recommendations, Assessment, Development and Evaluations) approach, which outlines several domains that rate up or down the certainty of study results (Guyatt et al., 2011[37]). GRADE is one of the most widely used approaches to evaluate the quality of evidence, but its use on public health studies has been more limited and its applicability in this field is still debated (for example, see (Irving et al., 2016[38]; Rehfuess, Bruce and Pruss-Ustun, 2011[39])).
Evidence‑based indicators have been classified into a three‑tiered hierarchy. The hierarchy is high-level, therefore, it is important to examine the quality of the study design in further detail. For example, a well-designed cohort study may be of higher quality than a poorly designed RCT.
Study design hierarchy (see Table 1.3 for further explanation)
High-quality evidence
Randomised controlled trials (RCTs)
Medium-quality evidence
Non-randomised control trial
Pre‑post studies
Cohort studies (prospective or retrospective)
Case‑control studies
Cross-sectional studies
Ecological studies
Low-quality evidence
Patient experiences
Case studies
Expert opinion
Mechanism-based reasoning
Source: Mann (2003[40]), “Observational research methods. Research design II: Cohort, cross sectional, and case‑control studies”, http://dx.doi.org/10.1136/emj.20.1.54; Oxford University (2016[41]), “OCEBM Levels of Evidence”, https://www.cebm.net/2016/05/ocebm-levels-of-evidence/; U.S. Preventive Services Taskforce (2017[42]), “Section 4. Evidence Review Development”, https://www.uspreventiveservicestaskforce.org/uspstf/section-4-evidence-review-development; Rychetnik et al. (2002[5]), “Criteria for evaluating evidence on public health interventions”, http://dx.doi.org/10.1136/jech.56.2.119.
“Extent of coverage” refers to the extent to which the intervention reached the target population (RE-AIM, 2020[11]) Assessing this criterion is important for understanding the intervention’s success in recruiting eligible participants, that is, those who stand to benefit most (Spencer et al., 2013[15]; RE-AIM, 2020[11]).
Extent of coverage indicators are broken into two groups: individual and organisation level indicators (see Box 1.7). The former closely aligns with the “Reach” indicator in the RE‑AIM framework, which measures the proportion of the eligible population who accessed the intervention (RE-AIM, 2020[11]). Organisation-level indicators refer to participation of providers (e.g. primary care practices, schools, health clinics) responsible for delivering the intervention (CDC, 2011[43]).
Individual and organisation indicators are intertwined given a higher number of participating organisations will lead to a higher individual participation rate. Nevertheless, collecting organisational-level data is important as it may provide insights into why some population groups access an intervention and others do not.
Proportion of the eligible population who accessed the intervention (individual participation rate)
Proportion of individuals enrolled in the intervention who subsequently dropped out (individual dropout rate)
Proportion of providers responsible for delivering the intervention who participate (provider participation rate)
Proportion of providers enrolled in the intervention who subsequently dropped out (provider dropout rate)
Source: RE‑AIM (2020[11]), “Reach”, http://www.re-aim.org/about/what-is-re-aim/reach/; CDC (2011[43]), “Program Evaluation Tip Sheet: Reach and impact”, https://www.cdc.gov/dhdsp/programs/spha/docs/reach_impact_tip_sheet.pdf.
Once criteria and indicators are defined, the next step is to collect data to measure the performance of each intervention. Indicators proposed as part of OECD’s Framework are universally recognised, further, in regards to effectiveness indicators, they have been drawn from publically available databases (see Annex B for a list of these databases). This has two major benefits: 1) it reduces the burden associated with collecting data; and 2) it makes it easier for policy makers to compare the performance of different interventions. Where it has been determined that a specific indicator is of high importance but for which there is no available data, policy makers can rely on feedback from stakeholders/experts or the literature to measure performance.2
A “performance matrix” can be used to record results. An example performance matrix template focused on obesity prevention interventions is provided in Table 1.4. For a real-world example, see Marsh et al. (2013[44]) – Table 3 – who used an MCDA to assess a range of preventive health interventions (Marsh et al., 2013[44]).
To enhance the interpretability of results, policy makers may wish to convert “natural performance measures” in the performance matrix to quantitative scores. That is, convert measurements into units which are common across all criteria (such as from 0‑100). The quantitative scale should reflect the best and worst score that is realistically possible (SELFIE 2020, 2017[10]). This type of analysis is only appropriate when comparing interventions measuring the same indicators.
Further details on the varied ways of converting performance measurements into quantitative scores, see the OECD’s Handbook On Constructing Composite Indicators: methodology and user guide (OECD and European Commission, 2008[45]).
Note: This is an example performance matrix only. It is simplified and therefore should not be used a final framework for assessing obesity prevention interventions. *Based on (Marsh et al., 2013[44]), the distribution of benefit ratio represents the ratio of the proportion of the most disadvantaged 20% of the population eligible for the intervention to the proportion of the population as a whole eligible for the intervention.
The importance of criteria for assessing best practice will likely differ. Therefore, it may be appropriate to assign weights to each criteria (for example, using feedback from stakeholders).
There exist multiple methods to elicit weights, of most relevance to public health are discrete choice experiments and swing weighting given they force individuals to make trade‑offs between criteria:
Discrete choice experiments (DCE): in DCE, individuals are asked to compare various interventions which systematically differ on performance criteria. Individuals are then prompted to specify between two or more interventions, which they prefer. By doing so, individuals are forced to make trade‑offs which reveal criteria they deem most important.
Swing weighting: swing weighting begins with all interventions scoring the lowest possible score (worst-case scenario). From here, individuals are asked which criteria, if improved, would lead to the greatest overall gain. This process is then repeated for all remaining criteria to create a ranked scale, which are then transformed into weights.
Although considered methodologically more robust, the above two methods are relatively resource intensive. Therefore, policy makers may wish to consider straightforward weighting methods such as:
Visual analogue scale (VAS): VAS present individuals with a visual scale ranging from 0 (not important) to 100 (very important), for example, for each criteria. Individuals are then asked to pinpoint the importance of the criteria on the provided scale. If weights do not add up to 100, they must be re‑scaled. One possible downside of using this method is the potential for individuals to rank the importance of each criteria highly.
Point allocation: individuals are provided with the list of all criteria and asked to divide 100 points between them, with higher points reflecting greater importance.
For further details on each of the aforementioned weighting methods, and more, see the following European Commission (EC) report (SELFIE 2020, 2017[10]).
Based on feedback from delegates within OECD’s Expert Group for the Economics of Public Health and previous work undertaken by the EC, effectiveness, efficiency and equity are considered high priority indicators. Specifically, each of these are considered core criteria within EC’s tool for selecting best practices, further, between 72‑94% of OECD delegates voted they were either “important” or “necessary” (see Annex A for full results).
Two possible methods for determining best practice interventions are proposed in this section: 1) setting a minimum overall performance score threshold or 2) setting a minimum performance score threshold for each criterion.
Despite which approach is used, it is important that the methodology to determine best practice interventions is transparent. This involves reporting and justifying criteria and indicators used to measure performance; weights applied to criteria; and the process for developing a minimum threshold score.
This method sums the weighted quantitative scores for each criteria to obtain an overall score (often referred to as a “global score”). An example template using the weighted sums approach is provided in Table 1.5 (this is an example only; it does not reflect weights or scores previously defined in another study).
Next, based on feedback from policy makers and stakeholders, a minimum threshold overall score is defined. Interventions with an overall score at or above the threshold are considered best practice and not best practice, if below. This approach is currently used by the EC to identify good and best practice interventions (see (European Commission, n.d.[46])).
As a robustness check, policy makers may wish to undertake deterministic sensitivity analysis. This type of analysis involves changing parameters within the OECD Framework and assessing how this affects results. For example, how do results differ based on a change in weights allocated to criteria? (Marsh et al., 2013[44]).
Note: This is an example performance matrix only. It does not reflect weights or scores estimated during a systematic MCDA assessment. For example, school-based intervention: Overall score 69 = 80*0.2 + 40*03 + 50*0.1 + 90*0.3 + 80*0.1.
Using an overall performance score to determine best practice interventions can lead to misleading results as it is assumed that a low score for one criterion can be offset by a high score in another. For example, despite the use of weights, an intervention with the following characteristics could be wrongly assessed as best practice:
Targets a priority population group to advance health equity
Was evaluated using high quality evidence
Achieves a high participation rate and experiences low drop out
Does not lead to a significant improvement in outcomes.
To overcome this issue, minimum threshold scores for each criteria can be developed. Minimum threshold scores can be based on standardised quantitative score or natural performance scores. For example, for a public health intervention to be classified as “best” by the EC it is must meet a minimum score on “exclusion criteria” (e.g. whether the intervention is a public health political priority, and the quality of the evidence base) (European Commission, n.d.[46]).
The OECD Framework can assist policy makers identify public health interventions that are best practice based on performance within a specific setting. An intervention classified as best practice should not immediately be considered appropriate for implementation in another setting given the impact of external factors. For this reason, prior to implementation (Step 2), it is important to determine whether the intervention is transferable.
Public health interventions are complex given they involve several interacting components and multiple stakeholders in areas such as health, education, community and environment; often target heterogenous populations; and have outcomes influenced by various direct and indirect factors (Craig et al., 2008[47]; Norris et al., 2019[48]). Therefore, policy makers cannot assume positive outcomes achieved in one setting can automatically be transferred to a different setting (Schloemer and Schröder-Bäck, 2018[49]).
Step 1b assesses whether the identified best practice intervention is “transferable”. For the purpose of this guidebook, an intervention is defined as transferable if desired outcomes achieved in a given setting can be achieved in another well-defined setting (Trompette et al., 2014[50]; Burchett, Umoquit and Dobrow, 2011[51]). The assessment is broken into two components. First, a preliminary review of the best practice intervention in the setting where it has been proven successful (i.e. the best practice “owner setting”). And, second, comparing the owner setting with the “target setting”, that is, where policy makers intend to transfer the best practice (Wang, Moss and Hiller, 2006[52]; Schloemer and Schröder-Bäck, 2018[49]).
The process for assessing transferability addresses many of the same factors used to assess sustainability as summarised in Box 1.8.
There is no agreed term to define “sustainability”, in general, however, it refers to the ability to maintain an intervention and its aligning benefits in the long term. Sustainability is often viewed through the lens of funding (i.e. affordability), yet it is far more complex than this (Bodkin and Hakimi, 2020[53]).
Several frameworks to assess sustainability are available, however, no framework dominates, further, they are not validated. Nevertheless, there are factors common to several frameworks, many of which are covered in Step 1b – assess transferability – such as (Bodkin and Hakimi, 2020[53]):
Political support: assessing the political environment that determines which interventions are prioritised.
Affordability: whether there is a stable funding environment.
Organisational capacity: whether there are appropriate resources to deliver intervention activities (e.g. qualified staff).
“Champions”: whether there is a champion(s) to advocate for the intervention.
Fit/alignment: whether the intervention fits with existing interventions and aligns with community needs.
Adaption: whether the intervention is adapted to suit the needs of the target setting.
Sustainability is also considered in Step 2 – Implement – and Step 3 – Evaluate – of the guidebook. That is, an intervention is more likely to be sustainable if it is implemented following a thorough preparation phase involving key stakeholders, and evaluated in order to demonstrate its value.
This guidebook outlines four contextual factors affecting the successful transfer of an intervention. These four contextual factors form the structure of the transferability assessment:
Population context
Sector specific context
Political context
Economic context
Each contextual factor comprises several indicators to guide the decision-making process. Indicators are divided into two groups: 1) existing indicators – indicators with readily available sources of data; and 2) new indicators – indicators that require collection of new data for example through surveys, site visits and/or discussions with key contacts from the owner setting (Barnfield, Savolainen and Lounamaa, 2020[54]).
The transferability assessment framework is applicable to both intra- and inter-country transfers. The former relates to transfers within a country, for example, expanding an intervention to new schools within the same district. When compared to inter-country transfers, intra-country transfers are generally simpler given the owner and target setting are more likely to be similar (e.g. language, culture, education levels, and organisation of the health care system).
Given the attributes of transferability are specific to an intervention, policy makers should view the indicators outlined in this step as a reference point only (Wang, Moss and Hiller, 2006[52]). That is, policy makers should adapt the transferability assessment to suit their needs by adding/removing indicators as necessary. For example, indicators to measure health information system maturity are not relevant for assessing school-based obesity interventions. In addition, the extent of the transferability assessment will depend on available resources (both time and financial).
An understanding of the owner setting is an important first step within the transferral process (European Commission, 2015[55]; CHRODIS, 2017[56]). It involves obtaining information from those involved in designing, implementing and/or evaluating the intervention. For example, through study visits, special training sessions and/or short-term exchanges (European Commission, 2015[55]). If this is not possible (e.g. due to budgetary constraints), policy makers in the target setting can reach out using digital methods.
A preliminary review provides policy makers in the target setting with an opportunity to obtain information not readily available through public sources. Questions policy makers could ask include:
Has the intervention been transferred to another setting? If so, are there materials on how to transfer the intervention? (see Box 1.9 for two case studies)
What are the core components of the intervention? (i.e. components critical to achieve desired outcomes)3
Was there a local “champion” or “champions” responsible for driving change?
What partnerships and alliances did the intervention rely on for success?
A preliminary review may also be useful for collecting information on primary source indicators listed in Section 1.3.2.
The Prescription of Physical Activity (PAP) and Whole Grain Partnership have both been identified as best practice interventions in their owner setting – i.e. Sweden and Denmark, respectively. As a result, both interventions are in the process of being transferred to other EU Member States.
Prescription of Physical Activity (PAP) (Sweden)
PAP is a primary-care based intervention where general practitioners (GPs) provide patients with individualised written prescriptions to boost physical activity levels using latest scientific knowledge.
Following the success of PAP in Sweden, a three‑year project was initiated to facilitate the transfer and adoption of the intervention to ten EU member states (EUPAP). As part of the project a Feasibility Study Guide was developed to assist policy makers in the target setting assess their overall “preparedness of practice transfer”. The Guide is broken into two components:
Early diagnosis (macro level): identifies political priorities, current and past experiences with physical activity prescription programs, and legal and financial issues.
Preparedness for implementation (micro level): defines the stakeholders involved during implementation and the social, cultural and political context in which they operate. In addition, characteristics of the health care setting PAP will be implemented are identified, as well as who will implement (e.g. health care professionals) and receive PAP services.
A copy of the Feasibility Study Guide is available here: https://www.eupap.org/public-documents.
The Danish Whole Grain Partnership (Denmark)
The Danish Whole Grain Partnership is a public-private partnership between various organisations including government and food companies. The overall objective of the partnership is to increase whole grain intake, which it achieves by allowing partnering organisations to use a front-of-pack, easy-to‑understand whole grain logo.
The Whole Grain Partnership is considered one of the world’s most successful interventions to boost whole grain consumption (Suthers, Broom and Beck, 2018[57]). Consequently in 2019 the WholeEUGrain three‑year project was launched to facilitate the transfer of the intervention from Denmark to Romania, Slovenia, and Bosnia and Herzegovina. The project developed two outputs to assist countries transfer and implement the intervention:
Preparedness and feasibility study: a report to assist countries assess whether they are prepared to implement the intervention.
ToolBox: a guide to help policy makers successfully implement the Whole Grain Partnership in their local setting (available here: https://www.gzs.si/wholeugrain/vsebina/Publications#1150561284-reports).
In addition to the Feasibility Study and ToolBox, there is a “summer school” which offers training to those interested in transferring the Whole Grain Partnership to their local setting.
Developing a relationship between best practice owners is not only important at the start of the transfer process, but also during the transfer and implementation stage, as described in Box 1.10. If best practice owners and implementers agree to work together, at the outset, both parties should agree on the “nature of the transfer” (i.e. an exact or loose replication of the intervention) (Stegeman et al., 2020[58]).
In 2020, the Joint Action on Chronic Diseases (CHRODIS-PLUS Joint Action) produced a report outlining recommendations for transferring and implementing good practice interventions. The report outlines five recommendations including an investment between project “owners” and “implementers” in all phases of the intervention (Stegeman et al., 2020[58]). As noted by the authors, “even if a good practice is thoroughly described, regular support may be needed from the initial developers, to help maintain the integrity of the core elements that made it a success” (Stegeman et al., 2020[58]). Real-world examples from the Joint Action are summarised below.
JOGG is a Dutch public health intervention that aims to improve diet and physical activity amongst young people (0‑19 years). Activities within JOGG are decided at the local (municipal/city) level, for example, drinking water initiatives in kindergartens and creating playgrounds for children.
Iceland, in an effort to further develop its existing Health Promoting Community (HPC) programme, chose to transfer elements of the JOGG intervention. The transfer was seen as successful in part due to the “valuable support provided by the JOGG programme” (Gunnarsdóttir and Ingibjörg, 2020[59]). For example, the original site visit to the Netherlands allowed Icelandic officials to get an in-depth understanding of JOGG as well as develop casual ties with the JOGG programme. Further, throughout the transfer and implementation process, representatives from JOGG were generous in providing materials and time (e.g. regular phone meetings and emails) (Gunnarsdóttir and Ingibjörg, 2020[59]).
The Active Flag (ASF) is a school-based intervention that aims to improve levels of physical activity amongst children aged five to 18 years. ASF was originally developed in Ireland and was subsequently transferred to two schools in the Piedmont region of Italy and the Klaipeda District Municipality in Lithuania.
Two site visits were undertaken as part of the transfer process: 1) a visit to Ireland by Italian implementers to learn more about ASF before it was transferred; and 2) a visit by Irish good practice owners to Italy once the intervention was implemented to share experiences and receive feedback. In addition, throughout the implementation process, officials in Ireland and Italy communicated regularly through emails and a closed Facebook group. The regular communication between Irish and Italian officials was viewed as a key success factor (Stegeman et al., 2020[58]).
To assess transferability, this guidebook recommends policy makers compare the best practice owner setting and target setting against the following four contextual factors: population, sector specific, political and economic contexts.
The ideal setting for the best practice intervention isn’t necessarily that of the best practice owner. Further, indicator values may not be available for the owner setting. In these instances, policy makers should instead assess whether the target setting is receptive to the best practice intervention based on the indicator.
The “population context” refers to population characteristics and covers sociodemographic factors as well as broader cultural considerations (see Table 1.6).
Source: JA-CHRODIS (2017[60]), “Joint Action on Chronic Diseases and Promoting Healthy Ageing across the Life Cycle (JA-CHRODIS). Work Package 5: Good practices in the field of health promotion and chronic disease prevention across the life cycle. Recommendations report on applicability”, http://chrodis.eu/wp-content/uploads/2014October 170223_wp5-t5_report-successfactorstransf-scalability_wotable2.pdf; Schloemer and Schröder-Bäck (2018[49]), “Criteria for evaluating transferability of health interventions: a systematic review and thematic synthesis”, https://doi.org/10.1186/s13012‑018‑0751‑8; Wang, Moss and Hiller (2006[52]), “Applicability and transferability of interventions in evidence‑based public health”, https://doi.org/10.1093/heapro/dai025; WHO (2011[61]), “Identifying and addressing barriers to implementing policy options. In: SURE guides for preparing and using evidence‑based policy briefs”, https://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/SURE-Guides-v2.1/Collectedfiles/sure_guides.html; Kidholm et al. (2012[62]), “A model for assessment of telemedicine applications: Mast”, https://doi.org/10.1017/S0266462311000638; Cuijpers, De Graaf and Bohlmeijer (2005[63]), “Adapting and disseminating effective public health interventions in another country: Towards a systematic approach”, https://doi.org/10.1093/eurpub/cki124.
It is important to first understand whether the intervention is suitable for addressing population health needs in the target setting. For example, by analysing key population health trends across different population sub-groups, as well as levels of unmet need (Ng and de Colombani, 2015[2]; CHRODIS, 2017[56]; Schloemer and Schröder-Bäck, 2018[49]; Bonell et al., 2006[64]). This assessment should be undertaken early on given it is a pre‑requisite for evidence‑based decision-making (Yost et al., 2014[65]) and intervention effectiveness (Bonell et al., 2006[64]). Other population characteristics to consider may include levels of health literacy, education as well as the ethnic structure of the population as this can affect the success of an intervention, for example, intervention take‑up.
In addition to examining existing indicators on population characteristics, policy makers are encouraged to collect information to assess public acceptability of the intervention as well as cultural norms and expectations, for reasons described below.
Public acceptability: Public acceptability refers to perception among the public that an intervention is agreeable, palatable, or satisfactory. It therefore reflects several concepts including (Sekhon, Cartwright and Francis, 2017[66]; WHO, 2011[61]):
Whether the intervention is socially, culturally, philosophically and ethically accepted
Whether the intervention is seen to address key health needs within the population
Whether the intervention is seen as the most appropriate response to addressing health needs
The burden associated with participating in the intervention
The extent to which the public understand the intervention.
Policy makers are less likely to implement an intervention if it does not receive support from the public (or is met by public resistance) (Stok et al., 2016[67]; Diepeveen et al., 2013[68]).
Relative to indicators such as age/gender structure of the population and education levels, the level of public acceptability may be difficult to measure. Suggested methods to measure public acceptability include questionnaires and focus groups (Saracutu et al., 2018[69]; Stok et al., 2016[67]).
Public cultural norms and expectations: culture is a set of “distinctive spiritual, material, intellectual and emotional features” within a society which determine values, traditions and beliefs (Napier et al., 2017[70]). It is therefore not limited to national, race, ethnic or religious factors (Napier et al., 2017[70]; Napier et al., 2014[71]).
The impact of the cultural context is growing increasingly important in understanding the success or failure of health care interventions (WHO Regional Office for Europe, n.d.[72]). Therefore, it is important policy makers compare cultural differences between the owner and target setting (Iwelunmor, Newsome and Airhihenbuwa, 2014[73]). Failing to do so may result in wasted resources as well as widening existing health inequalities (Napier et al., 2014[71]).
OECD Statistics. OECD provides national level statistics across a range of topics including statistics on population structure. https://stats.oecd.org/.
Eurostat. Eurostat provides comparative data for European countries across a range of topics including “population and social conditions”. https://ec.europa.eu/eurostat/data/database.
Eurobarometer surveys. Eurobarometer surveys collect information from public opinion surveys across all EU Member States. https://www.europarl.europa.eu/at-your-service/en/be-heard/eurobarometer.
WHO Global Health Observatory. The WHO publishes data on a wide‑range of population health indicators for countries across the world. https://www.who.int/data/gho/data/indicators.
Institute for Health Metrics and Evaluation (IHME). IHME provides publically available data on the burden of disease across a range of countries, which is useful for measuring population health needs. http://www.healthdata.org/.
The “sector specific context” refers to governance/regulation, financing, workforce, capital (including technological infrastructure) and access arrangements in the sector the intervention will operate (see Table 1.7). Given the breadth of public health interventions, this may include sectors such as health, education, environment, transport and housing.
Source: Barnfield, Savolainen and Lounamaa (2020[54]), “Health Promotion Interventions: Lessons from the Transfer of Good Practices in CHRODIS-PLUS”, https://doi.org/10.3390/ijerph17041281; WHO (2011[61]), “Identifying and addressing barriers to implementing policy options. In: SURE guides for preparing and using evidence‑based policy briefs”, https://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/SURE-Guides-v2.1/Collectedfiles/sure_guides.html; Kidholm et al. (2012[62]), “A model for assessment of telemedicine applications: Mast”, https://doi.org/10.1017/S0266462311000638; Cuijpers, De Graaf and Bohlmeijer (2005[63]), “Adapting and disseminating effective public health interventions in another country: Towards a systematic approach”, https://doi.org/10.1093/eurpub/cki124; Schloemer and Schröder-Bäck (2018[49]), “Criteria for evaluating transferability of health interventions: a systematic review and thematic synthesis”, https://doi.org/10.1186/s13012-018-0751-8; Wang, Moss and Hiller (2006[52]), “Applicability and transferability of interventions in evidence‑based public health”, https://doi.org/10.1093/heapro/dai025; McCannn, Schall and Perla (2008[74]), “Planning for Scale: A Guide for Designing Large‑Scale Improvement Initiatives. IHI Innovation Series white paper”, www.IHI.org.
Governance: governance reflects key players who develop and implement key strategic policies to achieve policy objectives, as well regulate the behaviour of stakeholders (e.g. payers and providers) (WHO, n.d.[75]). An assessment of differences in governance arrangements may be relevant, for example, if the success of an intervention depends on local decision-making power. In the health sector, governance arrangements are particularly important when considering transferring an integrated care intervention (Box 1.11).
“Self Care Service” is an electronic online platform available to the citizens of the City of Oulu, Finland. The platforms offers a range of free online services to enable integrated care, such as: online appointment booking; online support for urgent health issues; ePrescriptions; primary care appointments; home monitoring; and information on health prevention, amongst other things.
The Self Care Service has been transferred to other municipalities in Finland, but not to other countries. Transferring the necessary IT is not seen as a barrier to implementing the intervention, rather, it is the organisation of the health care system in regard to how services are funded and reimbursed as well as how primary health care centres and hospitals are organised (Lupiañez-Villanueva, Sachinopoulou and Thebe, 2015[76]). Further, for the intervention to be successfully transferred, technological innovation should be aligned with the organisation of the health care system (Lupiañez-Villanueva, Sachinopoulou and Thebe, 2015[76]).
Financing (reimbursement mechanisms): reimbursement systems offer incentives and can therefore drive professional/provider behaviour (e.g. fee‑for-service, pay-for-performance, budgets, capitation salaries, or mixed models) (Tao, Agerholm and Burström, 2016[77]). A comparison of reimbursement mechanisms is important to see whether professional/provider motivation differs in the target setting (WHO, 2011[61]). For example, an intervention designed to boost services for multimorbid patients may work in a setting where providers receive risk-adjusted payments (which compensate for higher treatment costs) and fail in a setting where providers receive lump-sum payment for all services.
Workforce: comparing the volume (per capita) and skills of the workforce will determine whether the target setting has the capacity to operate the intervention or whether additional human resources and/or training are necessary (Schloemer and Schröder-Bäck, 2018[49]; Cuijpers, De Graaf and Bohlmeijer, 2005[63]; Wang, Moss and Hiller, 2006[52]). For example, a key barrier to implementing Active School Flag (a school-based physical activity intervention) from Ireland to Italy was teacher confidence due to their lack of training in physical education (Stegeman et al., 2020[58]).
It is also important to assess the workforce and broader stakeholder acceptability of the intervention as this will determine “buy-in” (e.g. is the intervention the most appropriate action for addressing a recognised health care need?) (WHO, 2011[61]; Cuijpers, De Graaf and Bohlmeijer, 2005[63]; Barnfield, Savolainen and Lounamaa, 2020[54]).4 For example, a chronic care wound management intervention developed as part of CHRODIS-PLUS Joint Action, in part, credited its success to the readiness for change amongst the medical profession who recognised the need to change the organisation of care. For another example, see Box 1.12.
ToyBox is a Kindergarten‑based intervention aimed at promoting healthy lifestyles to prevent overweight and obesity rates among children. The intervention is delivered by teachers over a 24‑week period, which concentrates on the four main energy balance‑related behaviours: water consumption; healthy snacking; physical activity; and sedentary behaviour.
To assess the acceptability of implementing ToyBox, feedback was collected from preschool practitioners and parents. For preschool practitioners, this involved focus groups where participants were invited to discuss barriers and facilitators to implementing the intervention and the use of an RCT for evaluation. Parents, conversely, participated in one‑on-one semi-structured interviews to discuss barriers/facilitators for delivering the home component of the intervention (each parent received a GBP 25 (USD PPP (purchasing power parity) 37) shopping voucher as an incentive) (Malden et al., 2020[78]).
Capital resources: capital resources are the physical assets required to implement and deliver an intervention. These cover long-term assets including land, facilities (usually stationary sites where services are provided such as buildings), equipment (machines, tools or objects), and digital and technical infrastructure, for example, to support eHealth (Box 1.13). Example capital resources include health information system technology, number of cooking facilities available to schoolchildren, gym facilities, and land space (e.g. for playgrounds or public transport). Understanding capital resources in both settings is important for determining the cost of implementing the intervention, as discussed under “economic context”.
Scaling Integrated Care in Context (SCIROCCO) is a project co-funded by the European Commission which aims to adopt and transfer good practices in integrated care. A key output of SCIRICCO is the Maturity Model which outlines 12 dimensions necessary for assessing areas of strength for implementing integrated care, as well as gaps (SCIROCCO, n.d.[79]).
One of the Maturity Model’s dimensions is “information & eHealth services” and is scored based on a six‑point scale (from least ready to most ready) (SCIROCCO, n.d.[79]):
There are no eHealth services to support integrated care in place (score: 0)
There is recognition of need but there is no strategy and/or plan on how to deploy eHealth services to support integrated care (score: 1)
There is a mandate and plan(s) to deploy regional/national eHealth services across the health care system but not yet implemented (score: 2)
eHealth services to support integrated care are piloted but there is no yet region wide coverage (score: 3)
eHealth services to support integrated care are deployed widely at a large scale (score: 4)
Universal, at-scale regional/national eHealth services used by all integrated care stakeholders (score: 5).
EHealth readiness scores from the owner setting are used to assist policy makers in the target setting determine whether their eHealth system is mature enough.
Access: key factors to consider when comparing the owner and target setting include differences in geographic, virtual and financial accessibility for eligible individuals. Geographical and virtual access refers to a population’s ability to physically and virtually (i.e. telemedicine) access services, respectively, with those living in regional/remote areas and/or on low-income particularly vulnerable (Rice and Smith, 2001[80]). Financial access refers to the absence of cost barriers to access an intervention. Costs barriers include out-of-pocket payments (OOPs) such as co-payments, co‑insurance and deductibles, as well as hidden costs such as transport (Brooks, Wilson and Amir, 2011[81]).
Existing interventions: existing interventions in the target setting may affect the success of a new best practice intervention (Cuijpers, De Graaf and Bohlmeijer, 2005[63]; Cambon et al., 2013[82]; Burchett, Umoquit and Dobrow, 2011[51]). Existing interventions with the same objective may overlap with the best practice intervention leading to duplication, or may be antagonistic and work against the intervention’s objectives. Conversely, they may have a synergistic effect leading to positive outcomes, for example, through existing infrastructure, networks and a workforce with relevant skills. In fact, findings from the CHRODIS-PLUS Joint Action recommend policy makers “build on existing good practice” (Stegeman et al., 2020[58]). Further, existing interventions with similar objectives can create a culture conducive to the success to the best practice intervention. For example, the more “established the culture of HPDP [health promotion and disease prevention], the greater the incentive for professionals at all levels to invest their time and energy in relevant activities” (Stegeman et al., 2020[58]). (See Box 1.14 for an example on how existing interventions in the target setting can have a positive effect.).
Active School Flag is a school-based intervention which aims to improve levels of physical activity amongst children aged between 4 and 12. The intervention was originally developed in Ireland where it is active in approximately 800 schools.
The success of Active School Flag in Ireland led to its transfer to the Piedmont region in Italy and the Klaipėda city and Klaipėda district municipalities in Lithuania. Barnfield et al. (2020) noted that implementation of the Active School Flag intervention in Italy and Lithuania benefited from previous interventions, which had created an environment supportive of health-promoting initiatives in school. For example, it was easier to identify qualified implementers as well as train new implementers (Barnfield, Savolainen and Lounamaa, 2020[54]).
“The use of a good practice to expand (or ‘mesh with’) a current programme aids transfer and implementation”, (Barnfield, Savolainen and Lounamaa, 2020[54]).
“It was easier for localities that already had a strong foundation in health promotion and disease prevention to trail the new initiatives.”, (Stegeman et al., 2020[58]).
OECD Health Statistics. OECD Health Statistics provides comparable data on topics such as health system financing, governance, workforce, utilisation and status. http://www.oecd.org/els/health-systems/health-data.htm. (Pre‑prepared statistics comparing countries across various health dimensions can be found in the following OECD publications: Health at a Glance: Europe 2018 and Health at a Glance 2019 (OECD/European Union, 2018[83]; OECD, 2019[84])).
OECD Health Systems Characteristic survey. OECD has created an interactive tool allowing users to compare health system characteristics (e.g. governance, delivery and resource allocation) across all 35 member countries, 2 key partnering countries and 21 Latin-American countries. The latest available information for OECD countries is from 2016. https://www.oecd.org/els/health-systems/characteristics.htm.
WHO Global Health Observatory: the WHO publishes data on a wide‑range of health indicators for countries across the world. https://www.who.int/data/gho/data/indicators.
Eurostat. Eurostat provides comparable information on a range of health indicators including care, expenditure and resources across European countries. https://ec.europa.eu/eurostat/data/database.
European Observatory for Health Systems and Policies. The Observatory publish a range of material on country health systems reviews and country health profiles. https://www.hspm.org/searchandcompare.aspx.
The Commonwealth Fund. The Fund is a privately run foundation designed to promote better performing health care systems. The Fund provides free‑to‑access international profiles of health care systems. https://www.commonwealthfund.org/international-health-policy-center/countries.
The “political context” refers to political will from key decision-makers to implement the intervention (see Table 1.8).
Source: Cuijpers, De Graaf and Bohlmeijer (2005[63]), “Adapting and disseminating effective public health interventions in another country: Towards a systematic approach”, https://doi.org/10.1093/eurpub/cki124; Wang, Moss and Hiller (2006[52]), “Applicability and transferability of interventions in evidence‑based public health”, https://doi.org/10.1093/heapro/dai025; JA-CHRODIS (2017[60]), “Joint Action on Chronic Diseases and Promoting Healthy Ageing across the Life Cycle (JA-CHRODIS). Work Package 5: Good practices in the field of health promotion and chronic disease prevention across the life cycle. Recommendations report on applicability”, http://chrodis.eu/wp-content/uploads/2014October 170223_wp5-t5_report-successfactorstransf-scalability_wotable2.pdf; Schloemer and Schröder-Bäck (2018[49]), “Criteria for evaluating transferability of health interventions: a systematic review and thematic synthesis”, https://doi.org/10.1186/s13012-018-0751-8;; WHO (2011[61]), “Identifying and addressing barriers to implementing policy options. In: SURE guides for preparing and using evidence‑based policy briefs”, https://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/SURE-Guides-v2.1/Collectedfiles/sure_guides.html; National Collaborating Centre for Methods and Tools (2007[85]), “Tool for Assessing Applicability and Transferability of Evidence”, https://www.nccmt.ca/uploads/media/media/0001/01/a13748201ebe5a6793cc641b109e21c70307dfdc.pdf.
Governments today face tight budgetary constraints concurrent with growing demand for services to meet population needs. The health sector, therefore, increasingly “competes” with itself and other sectors for political priority (WHO; ExpandNet, 2009[86]).
In order to secure political support, it is important that interventions align with overarching political priorities, strategies and action plans (Frieden, 2014[87]; McCannon, Schall and Perla, 2008[74]). These may include priorities/strategies/action plans at the international level, such as the United Nations Sustainable Development Goals (SDGs), or at the national level, for example, alcohol, obesity prevention or physical activity strategies. Political support is essential as it leads to the provision of resources needed to “co‑ordinate, implement and sustain” an intervention (see Box 1.15 for a case study on the benefits of having political support) (Frieden, 2014[87]).
Political support is particularly important for public health interventions given they affect multiple sectors, both public and private, and sometimes face opposition from certain interest groups (Frieden, 2014[87]). A prime example is opposition from industry, which often has access to significant financial resources to fund lobbying efforts, which can result in “undue influence” on public policy decisions (OECD, 2014[88]). Further, the benefits of public health interventions, although large on a population-wide scale, are generally small for each individual and not felt immediately (Rose, Khaw and Marmot, 2008[89]). This leads to a misperception over the benefits associated with the intervention thereby reducing public support (Rose, Khaw and Marmot, 2008[89]).
The Klaipėda district and Klaipėda city municipalities in Lithuania transferred Iceland’s Multimodal Training Intervention – a nutritional education and physical exercise programme targeting those aged 65 years and over.
The development of MTI in Lithuania required support from the Ministry of Health of the Republic of Lithuania, the Hygiene Institute, Public Health Offices and municipal leaders and administrative staff. In their evaluation report, officials from Lithuania noted that participation from these key political bodies was essential, therefore, any future transfers should involve political decision-makers from the outset (Stegeman et al., 2020[58]).
Positive outcomes achieved by MTI in Lithuania led the Klaipeda District Municipality to develop a physical activity Act making it easier for similar interventions to be implemented in the future (Stegeman et al., 2020[58]).
WHO Global Health Observatory: the WHO publishes data on countries with national action plans and strategies for different health issues (e.g. diabetes, unhealthy diet, physical activity). https://www.who.int/data/gho/data/indicators.
Eurostat. Eurostat provides comparable information for European countries on EU policy. https://ec.europa.eu/eurostat/data/database.
European Observatory for Health Systems and Policies. The Observatory includes a tool to readily access and compare information on current and future health reforms and policies across a range of countries. https://www.hspm.org/searchandcompare.aspx.
The “economic context” aims to assess whether the intervention is affordable in the target setting (Table 1.9).
Note: *Discussed further below.
Source: Kidholm et al. (2012[62]), “A model for assessment of telemedicine applications: Mast”, https://doi.org/10.1017/S0266462311000638; WHO (2011[61]), “Identifying and addressing barriers to implementing policy options. In: SURE guides for preparing and using evidence‑based policy briefs”, https://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/SURE-Guides-v2.1/Collectedfiles/sure_guides.html; Cambon et al. (2013[82]), “A tool to analyse the transferablity of health promotion interventions”, https://doi.org/10.1186/1471-2458-13-1184; OECD (2020[90]), “Main Economic Indicators (MEI)”, https://www.oecd.org/sdd/oecdmaineconomicindicatorsmei.htm; OECD (2019[91]), “Health expenditure and financing”, https://stats.oecd.org/Index.aspx?DataSetCode=SHA#.
Long-term affordability plays a critical role in determining whether an intervention can be successfully transferred. Several indicators to assess the economic context of the owner and target setting are availability such as GDP, GNI (gross national income) and health expenditure per capita (see Table 1.9, “existing indicators”). These indicators are recorded at the national level and therefore may be of limited use if assessing the transferability of a local intervention that is not representative of the entire country.
It is therefore important for policy makers to undertake a more in-depth evaluation of implementation and operating costs in the owner and target setting as these will likely differ. A high-level, three‑stage process for assessing the cost of an intervention is provided below:
First, develop an understanding of the financial costs associated with implementing and operating the best practice intervention in the owner setting (e.g. working time of health professionals and physical infrastructure needs). Information about these costs may be publically available (e.g. from an economic evaluation report) or may require discussions/visits with intervention leads in the owner setting (e.g. during the preliminary review stage).
Second, assess whether financial costs are likely to be similar in the owner and target setting. For example, the implementation cost of a digital health intervention will differ depending on whether necessary technical infrastructure already exists (Kidholm et al., 2012[92]).
Information from stages 1 and 2 are necessary for stage 3, which involves assessing whether the intervention is affordable in the target setting (e.g. by comparing costs with available budget) (RE-AIM, 2020[11]).
Further details on costing an intervention can be found in Step 2 (Implement) – page 52 (“Assessing resource needs against existing capacity and readiness”).
OECD Economy Statistics. OECD Statistics provides a wide range of comparable economic data across countries. https://data.oecd.org/economy.htm.
World Bank (World Development Indicators): the World Bank has developed a databank providing information on a range of economic indicators. https://data.worldbank.org/indicator (see “Economy and Growth”).
Eurostat. Eurostat provides a range of comparable economic statistics for European countries. https://ec.europa.eu/eurostat/data/database.
Indicators used to measure transferability will depend on various factors such as the intervention, the owner and target setting, and resources available to undertake the transferability assessment. Box 1.16 provides an example transferability assessment to demonstrate how the framework proposed by OECD can be adapted to suit the specific needs of policy makers. Specifically, it presents a list of indicators relevant for assessing whether an mHealth intervention offering personalised nutrition can be transferred to another setting.
The transferability assessment below refers to is a web-based and mobile app providing consumers with personalised nutrition (PN) advice – Let Food Be Your Medicine. Nutritional advice is based on a consumer’s dietary intake (and genetic tests, if provided) compared to recommended levels outlined by the European Food Safety Agency. The app is designed for both preventive and treatment purposes and can cost up between EUR 0‑5/month (USD PPP 0‑7.3), depending on application features.
Is this an appropriate intervention? Does the public accept this type of digital intervention? (e.g. are they an appropriate response high obesity rates? Does the public understand the intervention and its purpose?)
Does the population trust their personal health information will be used, stored and managed appropriately?
What proportion of the population have access to the internet?
What proportion of the population have access to a smartphone?
What are the rates of digital health literacy?
What, if any, existing web-based / mHealth apps are currently available to the population?
How accepted is the use mHealth apps to improve diet amongst health care professionals?
How accepted is the concept of personalised nutrition to improve diets amongst health care professionals?
Are there any other online nutrition tools available to the population?
What are the regulations/legislation regarding the use of genetic information?
Does the intervention align with policy strategies and/or action plans at the relevant level of government?
Is there political support and commitment from key decision-makers?
What is the cost of implementing the intervention in the target setting (e.g. how do infrastructure costs differ between the owner and target setting?)
What is the GDP per capita in the population?
What is the average disposable income of individuals/families?
The transferability assessment will lead to one of three possible outcomes:
1. Transfer the intervention in its original form
2. Transfer an adapted version of the intervention to suit the needs of the population
3. Do not transfer and return to Step 1a.
The owner and target setting do not have to be identical for an intervention to be transferred successfully. Changes to an intervention may be necessary to ensure it is sustainable, culturally appropriate or more widely accessible (Schloemer and Schröder-Bäck, 2018[49]; Movsisyan et al., 2019[93]; Escoffery et al., 2018[94]). Alternatively, changes to the target setting may be necessary for it to be receptive of the intervention (e.g. training for staff to ensure they have the skills to deliver the intervention) (Wang, Moss and Hiller, 2006[52]).
The more complex an intervention, the more likely it will need to be adapted to suit the target setting. Integrated care interventions, for instance, are often adapted given they bring together different health care payers and providers whose mutual relationships differ greatly across countries/regions (see Box 1.17 for a case study on the importance of adapting interventions to suit the target setting).
SELFIE (sustainable integrated care models for multimorbidity, delivery, financing and performance) is an EU-funded Horizon project designed to improve integrated care for people with multimorbidity. As part of the project, 13 practical transferability recommendations were developed, including recommendations regarding the “transferability of financing methods” (Kaló et al., 2020[95]).
Transferability recommendations related to financing clearly note that “transferability of financing methods is not a mandatory condition for transferring integrated care models from other jurisdictions” (Kaló et al., 2020[95]). As a first step, it is recommended policy makers explore whether the financing method used in the owner setting can be adapted to the target setting. If the financing method is neither transferable nor adaptable, a new financing methodology should be developed that aligns with the target setting (Kaló et al., 2020[95]).
By the end of Step 1, policy makers should have identified a best practice intervention suitable for transfer to the target setting. Prior to implementing the best practice intervention, it is important to design an evaluation study. Designing the evaluation study at an early stage is necessary as it specifies the indicators and therefore the data needed to undertake an evaluation. Importantly, this includes identifying effectiveness indicators (also referred to as outcome indicators), which may be universal (see Box 1.3) or intervention specific (see Annex B).
For the purposes of this guidebook, details on how to design an evaluation study are discussed in Step 3, alongside details on how to execute an evaluation.
The first step of the guidebook is designed to assist policy makers identify best practice public health interventions. Based on a review of the literature, previous MCDA frameworks and expert feedback, five criteria were deemed necessary for making this decision: effectiveness, efficiency, equity, evidence‑base, and extent of coverage. A best practice intervention shouldn’t automatically be considered appropriate for implementation in a new setting. Therefore, it is important to assess the transferability of a best practice intervention by comparing the best practice owner and target setting covering dimensions such as political and economic feasibility.
The framework for identifying and assessing the transferability of best practice interventions is designed to cover a range of public health interventions. For this reason, it is important that policy makers adapt the framework to suit their specific needs. This will involve adding/removing indicators from the best practice criteria as well as the transferability assessment.
Once a public health intervention has been identified as best practice and transferable, the next step is implementation, which is discussed in Step 2 of the guidebook.
← 1. Efficacy is a closely related concept to efficiency with the exception that it refers to outcomes achieved in a controlled setting.
← 2. For example, see article from Reddy et al. (2016[168]) who used this approach to measure the performance of various smoking cessation interventions.
← 3. Core components pivotal to the success of an intervention can be identified through a process evaluation, which is discussed steps 2 and 3 of the guidebook.
← 4. If the intervention is not delivered in a health setting, the same analysis can be undertaken for the relevant profession. For example, the skill-set of teachers delivering an intervention regarding the importance of healthy eating and exercise.