Guidebook on Best Practices in Public Health

Step 3. Evaluate

3.1. Introduction to evaluating an intervention

The last step within OECD’s guidebook is to evaluate the best practice intervention identified in Step 1 and implemented in Step 2. The structure of evaluating the best practice intervention is broken into two key components: designing and executing an evaluation study (Figure 3.1).

As outlined in Step 1, designing the evaluation study occurs in the early phase of the Plan-Do-Study-Act cycle, specifically, once an intervention has been classified as best practice and appropriate for transfer. Despite this, designing the evaluation study is discussed in Step 3 to cater to those who are solely interested in the evaluation component of the guidebook.

Evaluations are a necessary final step as they provide policy makers with insight into how the intervention is operating and whether or not it achieved (or is on the path to achieving) desired outcomes. Further details on the importance of evaluation are explored in Box 3.1.

It is important to determine at the outset of Step 3 whether people involved in the evaluation have the necessary technical skills. If not, it is possible to partner with an external organisation, such as a research organisation.

Step 3 of the guidebook is applicable to an individual evaluation as well as system-wide evaluations. System-wide evaluations incorporate a wider range of stakeholders from different sectors who undertake activities to meet high-level goals (e.g. United Nations Sustainable Development Goals). These types of evaluations, although more complex, are important for improving overall population health.

Figure 3.1. Guidebook overview – Step 3: Evaluate an intervention

Box 3.1. Why evaluate?

There are various reasons evaluations are not included a project plan, including (O’Connor-Fleming et al., 2006[130]; Lobo, Petrich and Burns, 2014[131]):

• Lack of expertise, time or resources

• Lack of recognition of the benefits of evaluation, especially in the presence of competing priorities (such as service delivery)

• The politics of evaluation, including the potential risk of unfavourable findings

• A “blind” belief in the success of the intervention.

Nevertheless, there are many more reasons that support a rigorous intervention evaluation. In particular, evaluation of a new implemented intervention is crucial to understand whether it is successful – and whether it should be continued or amended. Moreover, evidence from a robust evaluation can provide support for continued investment or expansion of the intervention. Other reasons supporting the incorporation of an evaluation in the project plan from the beginning include:

• To determine the best way to design, deliver and improve the intervention

• To ensure the optimal use of time and resources

• To justify continued funding

• To ensure services are equitable and accessible

• To satisfy questions of accountability, or to fulfil grant or contract requirements

• To advance health promotion by sharing knowledge about effective interventions.

Source: O’Connor-Fleming et al. (2006[130]), “A framework for evaluating health promotion programs”, https://doi.org/10.1071/he06061; Lobo, Petrich and Burns (2014[131]), “Supporting health promotion practitioners to undertake evaluation for programme development”, https://doi.org/10.1186/1471-2458-14-1315; Public Health Ontario (2016[132]), “Evaluating health promotion programs: introductory workbook”, https://www.publichealthontario.ca/-/media/documents/E/2016/evaluating-hp-programs-workbook.pdf?la=en.

3.2. Develop the evaluation study

The first component of Step 3 – Evaluation – is to develop the evaluation study. This involves developing a logic model, selecting evaluation indicators, and choosing a study design and data collection method.

It is not plausible to rigorously evaluate all public health interventions. Therefore an evaluation study should reflect the intervention’s size, significance and risk as discussed in Box 3.2.

Box 3.2. Prioritising evaluation resources

In countries such as Australia, governments offer guidance on how to prioritise evaluation efforts. Example criteria policy makers could consider when prioritising evaluation resources are summarised below:

• Priority: the overall strategic importance of the intervention in achieving high level objectives as well as the intervention’s importance against competing priority areas

• Funding: the overall funding the intervention receives

• Risk: risk associated with the intervention, such as difficulty in assessing intervention outcomes

• Impact: how likely the intervention will affect population health

• Evidence: the existing evidence‑base for the intervention to date.

Source: Information provided by the Australian delegation to the OECD Health Committee.

3.2.1. Develop a logic model

The first step in the evaluation process is to develop a logic model (The Better Care Fund, 2015[133]). To evaluate an intervention, there needs to be a clear definition of the intervention’s objectives. Generally, this would be an impact on population health, for example, reduction in the prevalence of obesity or the incidence of heart attacks (also referred to as “final outcomes”). It is also important to define the population of interest (Public Health Ontario, 2016[132]). A logic model can help to understand how the intervention will achieve the outcome within the population of interest. It defines the inputs needed for the intervention, the activities or processes that are associated with it and intervention’s outcomes (see Box 3.3).

Box 3.3. Logic models for evaluation

A logic model (or “programme logic”, “programme theory”, “causal model”, “conceptual map”) summarises the main elements of an intervention, and provides a visual overview of how it leads to the desired outcomes (Centers for Disease Control and Prevention, 2016[134]; Public Health Ontario, 2016[132]).

There is no one standard logic model in the literature, rather there are various terms to explain the same or similar concepts. The order of elements within the logic model may differ depending on how elements are defined. For the purpose of this guidebook, four programme logic elements are used:

• Inputs: the resources available to implement and deliver activities

• Activities: the essential activities required to produce intervention outputs

• Outputs: measure the products/good/services that result from implementing an intervention and are required to achieve desired outcomes

• Outcomes: describe the impact of the intervention and therefore whether it is achieving (or has achieved) its objectives.

Various templates and examples can be found online, but these should be used as a guideline only and adjusted to fit the intervention. The design and terminology can be changed to suit the local context.

An example programme logic for reducing the prevalence of adolescent smoking by the New South Wales (Australia) Ministry of Health is in Figure 3.2 (Centre for Epidemiology and Evidence, 2017[135]). For other examples, see WHO’s logic model for micronutrient interventions in public health or Public Health Ontario’s (Canada) logic model for diet related health problems (Public Health Ontario, 2016[136]).

Figure 3.2. Example programme logic for an intervention to reduce adolescent smoking

Source: Adapted from the NSW Ministry of Health Centre for Epidemiology and Evidence (2017[135]), “Developing and Using Program Logic: A Guide”, https://www.health.nsw.gov.au/research/Publications/developing-program-logic.pdf.

3.2.2. Select evaluation indicators

This section discusses the different types of indicators, how to select high-quality indicators as well as the importance of stratifying data collected from indicators.

Types of indicators

Once the programme logic is developed, indicators for each element within the logic model should be specified (see Table 3.1 for a high-level overview and Figure 3.3 for an example). When choosing indicators, it is important to be realistic by considering the data and resources available.

Inputs: the starting point of an evaluation is the inputs. Input indicators should cover items such as funding (financial resources); personnel required to deliver the intervention (e.g. in FTE); capital infrastructure (e.g. meetings rooms, exercise equipment, community kitchens); other materials and supplies; and in-kind resources. This information is important to put the outcomes in perspective in terms of return on investment of time and money.

Activities: activity indicators reflect the essential activities required to produce intervention outputs (i.e. what to do with intervention inputs). Example activity indicators include training for teachers to deliver obesity prevention curricula; healthy lifestyle workshops for parents and children; and developing advertising material to promote physical activity.

Outputs: output indicators reflect the products/goods/services produced from the intervention’s activities. Using the examples provided under “Activities”, aligning output indicators include: the number of teachers who received training; the number of healthy lifestyle workshops run; and the number of social media ads promoting physical activity. Output indicators, therefore, help measure how well the intervention was implemented. Without this information, it is possible, for example, to assume erroneously that an intervention has no impact, while in reality the intervention did not actually reach participants.

Outcomes: outcome indicators measure whether the intervention has achieved (or is on the path to achieving) its objectives and align with “effectiveness” indicators used in Step 1a of the guidebook. Outcomes can be broken into two categories – final and intermediate. Final outcomes reflect the ultimate objective of policy makers, and, in the field of public health typically refer to changes in population health (for example, a reduction in the incidence of cardiovascular disease (CVD)). Final outcomes, however, can be difficult to measure and can take many years to achieve. For this reason, intermediate outcomes (often referred to as short- and medium-term outcomes) directly related to the final outcome can be measured. Using CVD prevalence as an example, intermediate outcomes of interest may include a reduction in salt intake and improved knowledge on the importance of a healthy diet. Nevertheless, it is important to carefully consider this causal pathway. For example, a reduced calorie intake will only decrease body-mass index if all else stays equal. In reality physical activity may also decline, which could negate the impact on BMI. In this case, it would be useful to also measure any changes in physical activity.

Generally, indicators at the beginning of the logic model are easier to measure, while indicators at the end of the logic model are more relevant to the health impact of the intervention but more difficult to measure.

Table 3.1. Types of evaluation indicators

Type	Definition
Inputs	Monetary and non-monetary inputs necessary to deliver the intervention (e.g. funding and staffing)
Activities	The essential activities required to produce intervention outputs
Outputs	The products/goods/services produced from the intervention’s activities
Outcomes	Whether the intervention is achieving (or has achieved) its objectives. For the purposes of this guidebook, outcome indicators have been broken into two categories: Intermediate outcomes: outcomes that are known to relate directly to the final outcome* Final outcomes: outcomes of interest to policy makers and in the case of public heath generally refer to tangible changes in population heath (often referred to as long-term outcomes or impacts)

Note: *For the purposes of this guidebook, intermediate outcomes incorporates short- and medium-term outcomes, which is the terminology frequently used in the literature.

Source: CDC (2018[137]), “Program Evaluation Framework Checklist”, https://www.cdc.gov/eval/steps/step2/index.htm; Public Health Ontario (2016[132]), “Evaluating health promotion programs: introductory workbook”, https://www.publichealthontario.ca/-/media/documents/E/2016/evaluating-hp-programs-workbook.pdf?la=en.

Figure 3.3. Example evaluation indicators across a logic model

Note: *FTE = full-time equivalent. USD = US dollars. The logic model is a high-level example only, it does not present a logic model previously used to evaluate a school-based obesity prevention intervention.

Useful databases: Outcome indicators

A non-exhaustive list of universally recognised intermediate and final outcome indicators are listed in Annex B. The indicators cover a range of public health issues including risk factors and chronic disease. The Annex also includes links to databases which collect this information, including:

• OECD Health Statistics: http://www.oecd.org/els/health-systems/health-data.htm

• Eurostat: https://ec.europa.eu/eurostat

• WHO Global Health Observatory: https://www.who.int/data/gho/data/indicators

• Institute for Health Metrics and Evaluation: http://www.healthdata.org/

Where possible, ancillary outcomes may also be considered, for example the impact of the intervention on the workload of doctors, health care cost or productivity, as well as broader societal considerations such as those outlined within OECD’s Better Life Index (see Box 3.4). While not the primary aim of the intervention, these additional benefits can help make the case for investing in the intervention. Similarly, patient experience and well-being can show additional positive impacts of the intervention, or highlight issues with the design.

Box 3.4. Measuring well-being as an outcome: The OECD’s Better Life Index

Increasingly governments are moving beyond measures of economic activity to measure societal progress. It has been recognised that progress across a range of sectors is needed to improve people’s lives and therefore their well-being. Well-being is important for both the individual and society given its impact on economic growth, productivity, long-term sustainability and societal stability. For example, in 2019, the European Council adopted conclusions from the “The Economy of Well-being” report, which invites member states to put people and their well-being at the centre of policy design. Similarly, in Australia, the Australian Institute for Health and Welfare (the Federal Government’s health and welfare statistics agency) has developed a conceptual framework of welfare (and aligned indicators) to assist in evidence‑based decision making.

To assist countries in systematically measure societal progress (and therefore well-being), the OECD has developed a well-being framework. The framework consists of 11 key dimensions and aligned indicators (see Table 3.2) (OECD, 2020[138]).

Table 3.2. OECD Well-being Framework – headline indicators

Key dimension	Indicator
Income and wealth	Household net adjusted disposable income
	Household median net wealth
Housing	Disposable income after housing costs
Work and job quality	Employment rate
Health	Life expectancy at birth
Knowledge and skills	Cognitive skills of 15‑year‑old students in science
Environmental quality	Access to green space
Subjective well-being	Life satisfaction
Safety	Deaths due to assault
Work-life balance	Time allocated to leisure and personal care
Social connections	Time spent interacting with friends and family as primary activity
Civic engagement	Voter turnout

Source: OECD (2020[138]), “How’s Life? 2020: Measuring Well-being”, https://dx.doi.org/10.1787/9870c393-en.

Given the growing interest and importance of measuring well-being, consideration should be given to measuring outcome indicators beyond the health sector. For example, using an intervention to promote physical activity as an example, in addition to changes in BMI, it may be useful to measure the intervention’s impact on work-life balance (more time allocated to leisure and personal care) and life satisfaction.

In determining whether to include well-being indicators in an evaluation, it is important to remember the following:

• Well-being indicators should not replace intermediate and final outcome indicators, rather they should be complementary to improve evaluation results

• Additional resources will be needed to collect data for well-being indicators, which may be challenging due to budgetary constraints. Health leadership may consider engaging with other sectors to collect and analyse data of shared interest.

• Finally, similar to many final outcomes, changes in well-being indicators may take many years to realise and therefore fall outside the evaluation time period.

Choosing high-quality indicators

To assist in the selection of high-quality indicators across the logic model, the following criteria, based on the SMART1 framework, should be considered (The Better Care Fund, 2015[133]) (Victorian Government Department of Health, 2010[139]):

• Importance/relevance/utility: time and resources need to be invested to collect data on any indicator. They should therefore be important and meaningful to evaluate the intervention.

• Accessibility/feasibility: data needs to be obtainable, which depends on resource constraints, the availability of the data, and the frequency by which it is made available.

• Reliability: indicators should measure the issue or outcome consistently, to allow comparisons over time and between people or groups.

• Validity/accuracy/robustness: indicators should accurately measure the concept or event. However, in some cases this is not possible, and proxy measures should be considered (e.g. using income as a proxy for socio‑economic status).

• Timeframe: it is important to consider the timeframe in which a change in the indicator can reasonably be expected, and whether this falls within the evaluation study period.

Stratifying indicator measures

When studying the impact of public health measures on the population, it is important to look at their effect on inequalities (as discussed in Step 1a – “Equity”) (Victorian Government Department of Health, 2010[139]). Inequalities can exist across racial and ethnic groups, regions or neighbourhoods, rural or urban areas, socio‑economic groups (see Table 1.2 “Priority population groups” in Step 1a). Pertinent factors for inequalities in the implementation population should be identified, and included in the data collection process (for example, see Step 1 (Select), Table 1.2). This will allow results to be analysed across priority population groups, to see whether the intervention increases or decreases inequalities (e.g. did access and/or outcomes differ across population groups?).

3.2.3. Choose a study design

The two most common forms of evaluation are process and outcome (Centers for Disease Control and Prevention, 2011[140]):

• Process evaluation: process evaluations assess whether an intervention was implemented as intended (i.e. implementation fidelity). Process evaluations are therefore interested in examining indicators from the logic model’s input, activities and output elements.

• Outcome evaluation: outcome evaluations assess the impact of the intervention on desired outcomes. Outcome evaluations focus on the logic model’s outcome element, both intermediate and final.

Undertaking both a process and outcome evaluation leads to greater insights. Process evaluations facilitate understanding of the results from an outcome evaluation (i.e. the causes of performance) and therefore provide insight into how an intervention can be improved (Engbers and Sattelmair, 2008[141]). For example, using the case study from Figure 3.3 (school-based obesity prevention), if the outcome evaluation revealed no change in fruit and vegetable intake or a change in BMI, results from the process evaluation can provide insight into why this happened – e.g. what proportion of teachers attended training?

In regard to outcome evaluations, it is important to understand whether changes in outcomes are related to the intervention or external factors occurring simultaneously such as other policies or interventions (i.e. determining causal attribution). Again using the example from Figure 3.3, if the intervention was introduced at a similar time to a ban on junk-food advertisement, a simple examination of outcomes may reflect the impact of the advertisement ban as opposed to the school-based intervention.

The question of causal attribution can be addressed through the study design. There exist different approaches to design an evaluation study (see Box 3.5). A randomised controlled trial (RCT) is the gold standard for measuring the effectiveness of medical interventions in terms of outcomes. However, as discussed in Step 1a, this may not be feasible in a health promotion setting for various reasons: it may not be ethical to withhold the intervention from people; it may be difficult to achieve randomisation in participation; and it may not be possible to avoid “contamination” of the comparison group (O’Connor-Fleming et al., 2006[130]; Sanson-Fisher et al., 2007[142]). A non-randomised controlled trial does not withhold the intervention from willing participants or struggle with randomisation, but still requires access to information on some control group. A pre‑post study is often the easiest to achieve, but this requires researchers to pay close attention to potential confounding factors, and may not provide robust evidence.

For the reasons outlined above, it is important to carefully consider which study design is most appropriate. This may involve taking into account factors such as financial resources, available time to evaluate, logistics of collecting data, and the skills/capabilities of those leading the evaluation.

Box 3.5. Study design options for the evaluation of interventions

An evaluation can be carried using one of many study designs. Commonly study designs within the field of public health are listed below. This list is not comprehensive; for further evaluation study designs, see Table 1.3 in Step 1a.

Randomised controlled trial (RCT)

An RCT is the gold standard of study designs. A group of participants is randomly distributed between an intervention and a control group. If the randomisation is done correctly, the two groups will be similar in all respects except for the intervention. As such, any difference between the two groups in outcomes can be ascribed directly to the intervention.

Non-randomised controlled trial

A non-randomised controlled trial compares the outcomes of the participants of the intervention to another group of people who did not participate in the trial. For example, the neighbourhood where the intervention took place could be compared to another neighbourhood where there was no intervention. Alternatively, the control group can be selected from the wider population to match the intervention group (e.g. similar age, gender and baseline measurements). Either way, in this design there remains a risk that the intervention group is inherently different and that this difference somehow contributed to the observed results.

Cluster-randomised trial

This approach could be seen as a mid-way between randomised and non-randomised trials. In a cluster-randomised trial intact social units, such as households, primary care practices, hospital wards, classrooms, neighbourhoods or entire communities, are randomly allocated to an intervention or control arm. While the people in one such unit can be expected to be more similar to each other than if they were distributed individually, there is some degree of randomisation.

Pre‑post study

A pre‑post study compares the outcomes in the study population before and after the intervention – ascribing a change in the outcome to the intervention. However, this study design cannot account for other elements that may have changed over the time the intervention took place (e.g. seasonality, other interventions or policies).

Source: Thiese (2014[143]), “Observational and interventional study design types; an overview”, https://doi.org/10.11613/BM.2014.022; Weijer et al. (2011[144]), “Ethical issues posed by cluster randomized trials in health research”, https://doi.org/10.1186/1745-6215-12-100; Clark et al. (2019[145]), “Evaluating the impact of healthcare interventions using routine data”, https://doi.org/10.1136/bmj.l2239.

In addition to process and outcome evaluations, policy makers may also be interested in undertaking an economic evaluation. Economic evaluations use results from the outcome evaluation and compare these to input costs (see Box 3.6 for an overview of key cost items). This type of intervention is important given all policy makers face budget constraints. Types of economic evaluations are described in Box 3.7.

Box 3.6. Measuring intervention costs for an economic evaluation

In addition to process and outcome evaluations, policy makers may also be interested in results from an economic evaluation. An economic evaluation assesses costs in relation to benefits to assist policy makers make decisions that maximise outcomes from a limited set of resources.

Intervention costs can be broken into several categories including labour, capital, consumables, administrative and overhead costs (Tan-Torres Edejer et al., 2003[146]). Costs associated with implementing a nutrition and healthy lifestyle in-person intervention, for example, may include:

• The cost of training staff to deliver the intervention (labour)

• Wages paid to staff responsible for leading physical activity sessions and cooking classes (labour)

• The cost of renting out gym space to undertake physical activity sessions and kitchen space for cooking classes (capital)

• The cost of promoting the intervention to eligible participants (e.g. developing promotion material) (administrative)

• The cost of food used in healthy eating cooking classes (consumables).

When measuring costs it is important to also consider:

• Discounting: intervention costs occur at different stages therefore it is important to apply a discount rate to ensure future costs are in a common metric (i.e. their present value). Applying a discount rate means costs incurred today have a higher value than those in the future, which reflects that in general people prefer to receive benefits now but pay for them later. An annual discount rate of around 3% is considered the “norm” in global health amongst developed countries (this figure is higher for less advanced economies) (Haacker, Hallett and Atun, 2019[147]). If costs are discounted, it may also be important to discount outcomes using the same or a different discount rate.

• Itemisation: to the extent possible, costs should be itemised at the unit level. For example, it may be appropriate to assign only a proportion of wage costs to the intervention if wages cover costs to undertake unrelated activities.

Finally, unless considered essential for delivering the intervention, evaluation costs are to be excluded.

Box 3.7. Types of economic evaluations

Economic evaluations assess an intervention’s efficiency (see Step 1a, “Efficiency”, for further details). Commonly applied economic evaluations in the health sector include cost-utility, cost-benefit, cost-effectiveness analysis. The appropriate economic evaluation depends on how outcomes are measured.

• Cost-utility analysis (CUA): CUA provides an estimate of the additional monetary cost of obtaining one QALY (i.e. one year in full health). CUAs are appropriate when assessing multiple public health interventions with different objectives, all of which are converted into QALYs (York Health Economics Consortium, 2016[148]; Sassi, 2006[19]).

• Cost-benefit analysis (CBA): CBA transforms outcome measures into a monetary unit and compares this with the cost of the intervention (i.e. cost-benefit ratio). It provides a net monetary cost of achieving an additional unit of outcome. It is an appropriate technique when outcomes can be translated into monetary terms. CBAs are primarily used when assessing capital projects (such as building a new hospital) (York Health Economics Consortium, 2016[149]), however, they can also be used when measuring the value of a statistical life (i.e. willingness to pay for a reduction in the risk of death). CBAs identify interventions with the largest net benefit, these however, aren’t necessarily the most efficient, or the most effective.

• Cost-effectiveness analysis (CEA): CEA calculate estimated costs of an additional outcome unit across one or more interventions. The outcome unit is dependent on the intervention being evaluated (e.g. the cost per extra patient achieving a 10 mm Hg fall in blood pressure) (York Health Economics Consortium, 2016[150]). CEAs are useful when assessing interventions with the same outcome indicator of interest. For example, it is not intuitive to compare the cost of a one‑point reduction in BMI with the cost of a reduction in alcohol consumption by one standard drink per week.

Note: In relation to Box 1.4 (“Efficiency indicators”) CUA and CBA can be used for universal health-related efficiency indicators, while CEAs can be used for intervention specific efficiency indicators.

3.2.4. Choose a data collection method

Data for the indicators selected can come from a range of sources (see Box 3.8). These include primary datasets (which are collected specifically for the evaluation study) and secondary datasets (which already exist) (The Better Care Fund, 2015[133]). Healthcare systems generate large amounts of data during their routine operations, such as admissions data, registry data, claims data and electronic health record data. Using these routine data sources for an evaluation study requires less investment of time and resources, as the data is already collected. Moreover, data is often available for many years (which can be used to analyse the “before”) and for a large population (which can be used to create a control group) (Clarke et al., 2019[145]).

However, routine datasets are per definition not set up specifically to answer the questions of the evaluation study. Primary data collection on the other hand can be fully tailored to the needs of the study. A first step in choosing a data collection method is to identify any routine data sources that could potentially be used. These datasets should be compared to primary data collection in terms of investment needed and suitability of the data.

Box 3.8. Potential sources of data

Data for indicators comes from primary and/or secondary sources. The former involves collecting new information, while the latter relies on previously sourced information. Regarding secondary data, a list of recommended international, publically available datasets is provided in Annex B.

Primary data

• Participant data from the intervention (e.g. biomedical measurements, attendance)

• Process data from the intervention (e.g. records of activities, progress reports, minutes)

• Surveys (e.g. of participants, providers)

• Interviews

• Focus groups

• Observational data

Secondary data

• Population data (e.g. disease registries, census data, mortality data)

• Healthcare utilisation data (e.g. electronic health records, admissions data, claims data)

• Document review (e.g. medical records)

• Routine studies (e.g. provider-run patient experience surveys, national health questionnaires)

Source: Victorian Government Department of Health (2010[139]), “Evaluation framework for health promotion and disease prevention programs”, https://www.health.vic.gov.au/publications/evaluation-framework-for-health-promotion-and-disease-prevention-programs; CDC, (2012[151]), “Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide”, https://www.cdc.gov/eval/guide/index.htm.

It is recommended to collect both quantitative and qualitative data, to get a full picture of the performance of the programme (The Better Care Fund, 2015[133]; Centers for Disease Control and Prevention, 2012[151]). While quantitative data is often considered reliable and can help draw conclusions about the larger population, qualitative data can provide in-depth information on the functioning of the intervention, by identifying the factors or reasons affecting behaviour or outcomes. Qualitative data can be gathered from open-ended surveys, in-depth interviews, focus groups, narrative and participant observation (Victorian Government Department of Health, 2010[139]).

If primary data is collected, the approach taken needs to be standardised and tested. For interviews, an interview protocol needs to be developed. This can be done following four stages (Castillo-Montoya, 2016[152]):

• Developing interview questions that align with research questions. This also includes omitting any questions that are not necessary, to keep the length of the interview down.

• Constructing an inquiry-based conversation, which asks questions in a way that is understandable to participants, follows a logical conversational flow and may include specific prompts and follow-up questions.

• Receiving feedback on interview protocols (see Box 3.9).

• Piloting the interview protocol. During this try-out, in conditions similar to the real study situation, the interviewer provides feedback on the flow of the interview, any issues with responses and the timing of the interview.

Similar steps can be taken to develop and test survey questions, or a focus group script.

Box 3.9. Check-list for interview protocol review

Collecting information from interviews can be complex. To maximise the quality of objective and subjective information collected as well as ensure high levels of participation, it is important to refine the interview protocol. Drawing upon Castillo-Montoya (2016[152]), the following refinement framework may be used.

Structure

• Beginning questions are factual in nature

• Key questions represent the majority of the questions and are placed between beginning and ending questions

• Questions at the end of interview protocol are reflective and provide participant an opportunity to share closing comments

• A brief script throughout the interview protocol provides smooth transitions between topic areas

• Interviewer closes with expressed gratitude and any intents to stay connected or follow up

• Overall, interview is organised to promote conversational flow

Writing of Interview Questions & Statements

• Questions/statements are free from spelling and grammatical error(s)

• Only one question is asked at a time

• Most questions ask participants to describe experiences and feelings

• Questions are mostly open ended

• Questions are written in a non-judgmental manner

Length of Interview Protocol

• All questions are necessary

• Questions/statements are concise

Comprehension

• Questions/statements are devoid of academic language and jargon

• Questions/statements are easy to understand

Source: Extract from Castillo-Montoya, (2016[152]), “Preparing for Interview Research: The Interview Protocol Refinement Framework”, https://nsuworks.nova.edu/tqr/vol21/iss5/2.

As part of the data collection plan, it is important to determine your study population: the people from whom data will be collected. If the programme is relatively small, it may be possible to include all participants in the evaluation study (Olney and Barnes, 2013[153]; Public Health Ontario, 2016[132]). If this is not possible, the results will rely on a sample. This sample needs to be of adequate size and representative of the larger population (i.e. include people of all different sub-populations). The required sample size will depend on the expected size of the effect: if a small impact is expected, a larger sample is needed to achieve statistical significance in the results (Bhalerao and Kadam, 2010[154]).

3.3. Step 3a: Execute the evaluation study

A detailed execution plan is needed, which includes activities, timelines, resources needed and the individuals or entities responsible. In cases where resources or scientific research experience are scarce, it may be worth considering collaborating with a research organisation for data collection and analysis (e.g. academia, international organisation or private consultancy). Stakeholder engagement should be sought, to get buy-in for the plan and resources to execute the study – though it is not limited to this stage of the evaluation process (see Step 2, Section 2.2, for further details on engaging stakeholders).

The remainder of Step 3a provides details on executing an evaluation study – namely how to collect and analyse data.

3.3.1. Collect the data

Part of the execution plan should be the data collection procedures, which address issues such as logistics, consent, privacy, data security and other ethical considerations (see Box 3.10). Important questions to ask and answer in the data collection procedures (Centers for Disease Control and Prevention, 2012[151]; Olney and Barnes, 2013[153]; Public Health Ontario, 2016[132]).

• How is data collected, and how could this affect the results (e.g. would an email survey result in a higher response from computer-literate participants?)

• How do we recruit participants, in accordance with the desired study population previously defined?

• Who will be collecting the data, and how will they be trained for this task?

• How will we communicate the purpose of the study to potential participants?

• How will we obtain informed consent?

• How will data be stored safely?

• Is ethical approval required?

Box 3.10. Ethical considerations for evaluation studies

Informed consent

Informed consent is a process through which a prospective study participants voluntarily confirm their willingness to participate in the study, after having been fully informed about all the relevant aspects of the study (Manti and Licari, 2018[155]). Consent can be obtained in writing through a consent form, verbally prior to doing a telephone interview, or by clicking “next” on an online survey (Public Health Ontario, 2016[132]).

It is important to consider how informed consent can be obtained from people with intellectual disabilities (Horner-Johnson and Bailey, 2013[156]), underage persons (Vitiello, 2008[157]), vulnerable persons (Manti and Licari, 2018[155]) or people who are otherwise less able to understand the information presented or the implications of participation. It is possible to obtain consent from carers or family – called “informed consent by proxy” – though this may result in higher rates of refusal (Warren et al., 1986[158]) (Mason et al., 2006[159]).

Confidentiality

To encourage participants to share their personal information, confidentially needs to be assured (Public Health Ontario, 2016[132]). This can be done by full anonymisation of the data (where researchers or other staff cannot link responses to individuals), or by ensuring no personal data is available to anyone outside the study team. In addition, guidelines to ensure the safe storage of data need to be defined (Centers for Disease Control and Prevention, 2012[151]).

Sensitivity

The study needs to be sensitivity to participants’ background, values and situation. Cultural sensitivity is based on awareness of cultural diversity (for example how culture may influence a participants values, beliefs and attitudes) and requires acknowledging and respecting these differences (Brooks, Manias and Bloomer, 2019[160]). Researchers should also consider whether the condition or behaviour they are looking at may be stigmatised, and how this could influence participation, participant experience and results (Millum et al., 2019[161]).

When to collect data depends on the intervention, its duration and the time after which outcomes are expected to become measurable. The following time points should be considered for data collection (Public Health Ontario, 2016[132]):

• Start: collect data at the start of an intervention and before any activities take place to get an accurate baseline – for both the intervention and the control group

• During: collect data throughout the intervention period instead of only at the end, to understand when the intervention is effective and observe trends such as participation levels

• Completion and after: collect data at the end of the intervention as well as some time after, to observe the longer-term impact and determine whether there is a lasting effect once the intervention has ended (see Box 3.11 for a real-world case study)

Box 3.11. Case study: Multimodal training intervention (Iceland) – collecting data

The Multimodal Training Intervention (MTI) is designed to improve the overall health of older individuals in Iceland through nutritional education information sessions and exercise classes. To assess the impact of MTI on participant outcomes, intervention administrators collected data for several indicators including BMI, blood pressure, physical activity, as well as scores on a range of endurance and strength tests. For each outcome of interest, data was collected at the start of the intervention, three times during the intervention (6‑, 12‑ and 18‑months) and upon completing the intervention.

Follow-up data was not collected (e.g. six months after conclusion of the intervention), which limits the ability to draw conclusions on the intervention’s long-term impact. This is common amongst public health interventions given the lack of resources available to follow-up with intervention participants.

Source: Information provided by Janus Health Promotion.

Different timings may apply to different measures. For example, process measures should be monitored during the intervention, while the outcome indicator should at least be measured before and after the intervention.

For questionnaires or surveys, it is important to maximise the response rate (Olney and Barnes, 2013[153]). The response rate is the percentage of people targeted (those within the defined study population) that respond to the questionnaire. As this study population is defined with the aim of being of adequate size and representative, it is important that enough people within the study population participate to meet these two targets. A high overall rate of non-response could mean that the results are not statistically significant. If the non-response rate is higher in specific population groups, this could affect the accuracy of the results. For surveys that are administrated electronically, it is important to consider how the design and technology can affect the response rate (Box 3.12).

Box 3.12. Increasing the response rate for electronic surveys

Participation in electronic surveys and completion of questionnaires can be significantly increased using simple measures. Some examples are included below:

• Ensure that the survey loads quickly

• Allow participants to skip questions – to avoid them quitting the survey if they cannot answer one question

• Give participants an estimation of the time required to complete, and monitor progress with progress bars

• Scrolling may provide better completion rates than forcing respondents to click through pages with one or two questions

• Start with easier yet interesting questions – saving complicated questions that may overwhelm and “boring” questions such as demographics for the end

Source: Olney and Barnes (2013[153]), “Collecting and analyzing evaluation data – Planning and Evaluating Health Information Outreach Projects Booklet 3”, https://nnlm.gov/nec/evalmaterials/bookletThree508.

3.3.2. Analyse the data

A first step in quantitative data analysis is the descriptive statistics (Olney and Barnes, 2013[153]). These descriptive or summary statistics are often averages or frequency counts, and provide an overview of the participant characteristics and the intervention results. Demographic and evaluation indicators can be presented for different population groups (e.g. by gender, age group, socio‑economic status is highly recommended) or study arms (e.g. intervention and control group).

An important part of the initial descriptive analysis is to look at the pattern of missing data. Some participants may have dropped out of the study over time, or certain questions may have been left unanswered. If missing data are random (i.e. not related to the outcome of interest), it may not affect the results. However, it is possible that missing data is not random. For example, a sensitive question about alcohol consumption may be left blank by people who consume more alcohol and do not want to disclose this. Such a non-random pattern of missing data would change the results. It is therefore important to look at the degree and pattern of missing data, to try to understand whether the results could be affected. There are various techniques for dealing with missing data such as complete‑case analysis (deleting observations with missing data) and substituting missing values with the mean and multiple imputation. The right technique will depend on the nature of the missing data (e.g. missing completely at random, at random or not at random).

After the descriptive analysis, different statistical analyses can be used to identify whether the observed effects are “real”. There are statistical tests to compare outcomes in the intervention and control groups (or pre‑ and post-intervention), which determine whether a difference in outcome is statistically significant or whether it is likely to be due to chance. Where there are baseline differences between the intervention and control group that confound estimates of the effect of the intervention, regression-based methods can account for the confounders (such as age, gender and socio‑economic factors), to see whether the intervention effect persists after controlling for these differences.

For qualitative data (for example from interviews or focus groups) there exist tried-and-tested analysis methods. The following steps suggest how large amounts of raw text data may be analysed (Olney and Barnes, 2013[153]):

• Prepare text files: this can be verbatim transcripts or summary records, and these will form the raw data

• Create a codebook: after reading through the text files, identify themes or categories and assign them a code. These codes can have multiple layers. For example, under the “issues” category there can be “lack of statistical expertise” and “lack of funding”.

• Code the text: go through the text files and assign codes to sections of text. There are various software tools that can be used in this process (e.g. NVivo).

• Interpret the results: write summaries of the different categories. These summaries include a description of the broader theme and sub-themes, and can be supported with examples or quotes identified through the coded text.

This section represents a simplified overview of analysing data given it is not possible to provide a comprehensive overview of all statistical methods in this manual. The right statistical method will depend on many factors, including the type of indicators used, the structure of the data and the study design. For this reason, ensuring that statistical and analytical expertise is available for the analysis is crucial.

3.4. Step 3b: Act on evaluation results

Step 3b outlines possible follow-up actions after completing an evaluation as well as information on how to disseminate results.

3.4.1. Follow-up action

The results of the evaluation are not an end in and of themselves, rather they should be used as feedback to guide follow-up action. Specifically, evaluation results determine whether policy makers (Institute for Healthcare Improvement, n.d.[162]):

• Adopt: if the intervention is successful, it can be officially adopted and expanded to a wider population or for a longer time

• Adapt: if the evaluation identifies areas for improvement, the intervention should be adapted

• Abandon: if the intervention does not improve outcomes (or even worsens them) and no changes can be made to improve it, it may have to be abandoned.

To identify where improvements can be made, it is important to look at results across the entire logic model (Center for Community Health and Development, n.d.[163]). If issues were identified around inputs, activities and output (e.g. not enough participating practices, educational activities not completed), each should be reviewed with the aim of finding potential solutions or improvements. It is possible that the inputs, activities and outputs did not present clear issues, yet the intended outcomes were not met. In this case the basic design of intervention should be reviewed.

It is not always desirable to wait until the full results from the evaluation are available to make changes. If serious issues are identified mid-way through the evaluation period (for example a high drop-out rate), it would make sense to address this sooner rather than later (Center for Community Health and Development, n.d.[163]). However, this will affect the results of the study and should be taken note off during the analysis.

3.4.2. Disseminate results

Step 3, evaluate, culminates in the dissemination of evaluation results. Dissemination refers to the active spreading of evidence to a target audience via determined channels (Tabak et al., 2012[128]; Yuan et al., 2010[129]).

Disseminating evaluation results ensures lessons learnt reach interested stakeholders and can therefore be applied in future implementations. Where positive results ensue, disseminating evaluation findings can help gain stakeholder support for further scaling of the intervention (e.g. beyond the pilot study) or even transfer to a new target setting (see Box 3.13 for an example on disseminating results from Switzerland).

When disseminating results, it is important to keep in mind the target audience and tailor results accordingly. For example, when presenting to:

• Payers, it may be useful to compare the results of the evaluation with similar/competing interventions to highlight relative advantage

• The public, phrasing evaluation results in simple terms is appropriate, conversely, experts will likely be interested in the statistical analysis used to measure outcomes

• Patients, details on how the intervention affects patient outcomes will be of greater interest than potential cost savings, for example.

At this last stage of the evaluation, key political leaders and supported should be engaged to help promote and disseminate results.

Box 3.13. Case study: Sustainable spatial development (Switzerland) – disseminating results

The Swiss Federal Government supports a range of projects designed to test new approaches to implementing sustainable, social and exercise‑friendly spatial developments. Support not only refers to financial assistance but also networking, counselling and co‑ordination assistance. Over the period 2020‑24, the Swiss Government plans to support 31 projects implemented across the country. To encourage exchange of information across implementation sites and other stakeholders, four different types of meetings are held (see Table 3.3).

Table 3.3. Process for disseminating results

Type of meeting	Purpose	Attendees	Frequency
Kick-off meeting	To kick-start the intervention and for project implementers to receive input from other stakeholders, such as experts	All intervention members (e.g. NGOs, city planners), relevant government personnel and experts	One meeting per intervention undertaken at the outset (1/5 years)
“Experience exchange” meeting	Opportunity for people across all interventions to come together to exchange experiences (e.g. “lessons-learned”) and to discuss specific needs and experiences	Members from all interventions across a range of public health topics Experts with a strong relation to the ongoing interventions	Once per year, per intervention
Mid-term “knowledge exchange” meeting	Similar to above, this is an opportunity to share experiences and hear about new developments	As above + additional experts from other disciplines, and possibly the public	Once meeting per intervention run 1/5 years
Final meeting	Used to showcase evaluation results from the intervention and thus showcase best practice	As above	Once at the end of the funding cycle for all projects combined

Source: Feedback provided directly by Swiss officials.

3.5. Conclusion of Step 3

The final section of the guidebook describes the steps for undertaking an evaluation – the design and execution of an evaluation study. The former, designing an evaluation study, although discussed in Step 3 of the guidebook (to cater to those solely interested in the evaluation component of the guidebook), should be undertaken once an intervention has been assessed as best practice and suitable for transfer (i.e. the end of Step 1). Designing the evaluation study at an early stage is necessary as it specifies the indicators and therefore the data needed to undertake an evaluation.

Execution of an evaluation study occurs once an intervention has been implemented (end of Step 2). It involves collecting primary and/or secondary sources of data to assess whether the intervention achieved its objectives. Data should be collected at the beginning and end of an intervention, and, if resources permit, a period after the intervention has concluded. Once collected, researchers must analyse the data, which includes choosing appropriate statistical methods to assess whether observed effects are due to the intervention or chance.

Results from the evaluation should guide follow-up action on whether to adopt, adapt or abandon the intervention. They should also be used to ensure lessons learned are applied to future implementations. Where positive result ensue, promoting evaluation findings can help gain support for scaling-up and/or transferring the intervention.

Additional tools and resources

Developing a logic model: Logic Model Workbook from Innovation Network (Innovation Network, n.d.[164]).

Theory of Change Guidance from the United Nations Development Group (United Nations Development Group, n.d.[165]).

Stakeholder engagement and other steps in programme evaluation: CDC’s Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide (2011[140]).

Data collection methods and other steps in programme evaluation: Public Health Ontario’s Evaluating Health Promotion Programs: introductory workbook (Public Health Ontario, 2016[132]).

Developing an interview protocol: “Preparing for Interview Research: The Interview Protocol Refinement Framework” by Milagros Castillo-Montoya (2016[152]).

Collecting and analysing data: Collecting and analysing evaluation data – Planning and Evaluating Health Information Outreach Projects Booklet 3, by Olney and Barnes (2013[153]).

Acting on the results and other elements of community project evaluation: The Community Tool Box from the Center for Community Health and Development at the University of Kansas (Center for Community Health and Development, n.d.[163]).

Note

← 1.  SMART = specific, measurable, achievable, relevant and time‑bound.