Rethinking Health System Performance Assessment

Annex A. Three decades of health system performance assessment

A first glance at performance: OECD comparisons of goals of health system reforms (1992‑94)

The 1980s were marked by global macroeconomic shifts, mostly driven by sovereign debt crises, rising interest rates and declining government budgets. This decade proved to be a challenging period for health systems public financing. As trends in public policy making shifted towards values of efficiency, lean government, deregulation, and privatisation, the Donabedian model (an approach to healthcare quality developed in the 1970s that focuses on three main components: structure, process, and outcomes (Donabedian, 1972[61])) offered governments the possibility of conceptualizing health systems using similar frameworks (Van Olmen et al., 2012[62]). As national efforts to identify the bounds of their health systems continued, further models followed, including the framework by Evans to describe the structural organisation of health system actors, including providers, patients, payers, and government regulators (Evans, 1983[63]), and that of Kleczkowski et al. to elucidate the role of financing and other health system resources in sustaining these relationships (Kleczkowski, Roemer and Van Der Werff, 1984[64]).

In this context of knowledge development in health systems models, the relevance of relatively stable comparisons across health systems took hold. Governments interested in solutions to macroeconomic challenges in the health system sought to develop a suite of policy tools based in empirical evidence to reform them. As a technical space for knowledge sharing, the OECD served as a key venue of comparative analysis of public governance and service delivery, including in the health sector.

In 1992, the OECD published a first report comparing health systems of its member states: Reform of Health Care: Comparative Analysis of Seven OECD Countries. Through interviews with seven OECD member governments on their reforms conducted from 1989 to 1992, the OECD outlined key macroscopic differences between the structure of health systems, while establishing similarities between a set of “subsystems”, or structures of third-party payment and benefit provision. The OECD noted several issues faced by the seven countries at the time of publication, including remaining gaps in community access to services and in income protection when medical care was required. (OECD, 1992[65])

A second report followed in 1994 in which the OECD reviewed a total of 17 OECD member states. Building on the previous 1992 paper, the OECD outlined a distinct set of seven health system models characterizing specific relationships between payers, providers, patients, and governments. These were listed as:

• the voluntary out-of-pocket model

• the voluntary reimbursement-of-patients model

• the public reimbursement-of-patients model

• the voluntary contract model

• the public contract model

• the voluntary integrated model

• the public integrated model.

As the ability to compare countries grew through the expanded range of health systems considered, the OECD grouped health systems reforms into specific themes: equity, cost containment/micro‑efficiency, and choice (OECD, 1994[66]). This summary of reforms provided an early toolkit to governments for the adjustments required in a period of macroeconomic uncertainty and fiscal constraint. In addition, at an international level, the translation of core values in health system governance to models of health system assessment provided a blueprint for the development of new indicators to appraise the actual value of these reforms.

Measuring “performance”: A growing need for more and better data (2000‑02)

The growing recognition of the value of comparing national health system experiences drove the development of assessment programmes at the national and international levels. Central to this development was the focus on the “measurement” of health, where processes in the health system were linked theoretically to assessments of healthcare quality and population health outcomes.

At the turn of the century, in the vein of the Millennium Development Goals agreed to by the United Nations, the World Health Organization (WHO) published its landmark World Health Report 2000 aimed at providing the WHO’s first summary of the evidence regarding measurement of health system indicators. In introducing its inaugural HSPA Framework, the WHO intended to further comparative analysis of health systems goals and functions into mechanisms for “improving performance” across its member states. Most notably, it aimed to “[break] new ground” with the report in introducing a novel index of overall health system performance, presented as a ranking of national health systems in a league table based on performance according to three overall health system goals: “good health”, “responsiveness to the expectations of the population”, and “fairness of financial contribution” (World Health Organization., 2000[67]). These health system goals were operationalised through numerical indicators contributing to the calculated index of overall health system performance. These indicators included, inter alia: disability-adjusted life expectancy (DALE), health equality in terms of child survival, the level and distribution of responsiveness (employing a survey of key informants regarding elements of dignity, autonomy, confidentiality, prompt attention, access to social support networks during care, and choice of care provider), fairness of financial contribution, and performance on level of health.

While the World Health Report 2000 additionally contributed summaries of evidence on the vital functions of health systems, and the complex challenges of actor governance and resourcing, this translation of goals into a numerical league table generated significant attention and controversy. Some of the criticism was due to lack of face validity of the rankings as well as technical aspects such as using complex modelling where data was not available (McKee, 2010[68]),

Moving away from view of health system performance as a singular international process of comparison, governments and their multilateral partners shifted towards generating and capturing a more diverse range of data to contribute to national measurement processes. Accounting for the wide variation in health system structures, the OECD and Health Canada – the Canadian federal ministry of health – partnered to deliver Measuring Up in 2002, a report intended to standardise the collection and use of indicators at all levels of health service organisation and delivery (2002[69]).

“Together we can take great pride in what our varied health systems have achieved. But for all the reasons outlined above, every nation must now redouble its efforts to assess system performance at multiple levels. Assessment must go from appropriate use of medications in a single clinic, to costs per patient day across a set of acute care hospitals in a region, and on to broad markers of population health and system efficiency and equity. Without such information and indicators, health systems across the OECD will continue to vary more by institutional culture, political geography and historical happenstance, than by explicit evidence‑based design.” (OECD, 2002, p. 16[69])

Building on the development of the WHO and OECD international frameworks, and national health system performance assessment frameworks in Australia, Canada, the United Kingdom, and the United States, the report identifies four dimensions of health system performance. These include:

• health improvement/outcomes

• responsiveness

• equity

• efficiency.

In moving from conceptualisation of health system models toward a comparison of assessment activities common to national and international frameworks, Measuring Up established performance assessment as a process necessitating consideration of varying subsystems (as first outlined in the 1992 and 1994 OECD reports) while acknowledging common health system goals across these subsystems. This marked a significant shift towards comparing “models” to “indicators”: moving from fundamental indicators like life expectancy to discuss health outcomes, to disease‑specific clinical indicators aimed at understanding the efficiency of clinical care processes.

The move toward improved measurement of health systems inputs, processes, outputs, and outcomes coincided with the emerging challenge of conceptualizing health system performance. With a growing focus on indicator development and the expansion of data availability to health system stakeholders, new mechanisms for organizing these data were set to be considered.

Defining “performance”: Development of health system performance through the quality agenda (2000‑06)

The comparison of health system organisation in the 1990s raised questions regarding the linkage of health system goals with mechanisms for assessing achievement of such goals. As Murray and Frenk (2000[70]) posit in their 2000 WHO Bulletin article A framework for assessing the performance of health systems, an assessment of differences in health system “performance” requires a comprehension of differences in socially valued outcomes embedded in health system organisation, like “health improvement”, “responsiveness to expectations”, and “fairness in financial contribution”.

Understanding the achievement of these goals thus called for their translation into a “convincing and operational framework for assessing health system performance” by governments, development agencies, and multilateral institutions (Murray and Frenk, 2000[70]). Introduced earlier as the WHO HSPA framework, the Murray and Frenk framework provided the first summary of the functions and actions performed by health systems towards the achievement of common health system goals. For instance, the framework defines health systems by delimiting their bounds in terms of the “health actions” they perform (Murray and Frenk, 2000[70]). These included primary and clinical health services, as well as public health actions with a “primary intent” criterion of health in order to exclude other social services like education despite their role as a determinant of health outcomes.

From this global framework of the health system, the WHO framework defined performance as a process of assessment anchored in the four functions of health systems:

• financing (including revenue collection, fund pooling, and purchasing functions)

• provision (of personal and non-personal health services

• stewardship

• resource generation

Along with the WHO Framework, the OECD HSPA framework provided an alternative and complementary vision of the composition of health systems. In 2001, the OECD published a working paper by Hurst and Jee‑Hughes (2001[1]) defining performance assessment and its unique conceptualisation of the health system. Performance, as first defined by Hurst and Jee‑Hughes, is “the extent to which the system is meeting a set of objectives” in the measurement process (Hurst and Jee-Hughes, 2001, p. 8[1]). Centred on these objectives, the proposed OECD framework built on earlier developments by Murray and Frenk towards the WHO Framework by adapting the health system goals into components for assessment.

However, several differences were also noted. Social determinants of health, like education, were not excluded by the OECD Framework, but instead considered as part of frontier analyses while remaining external to accounting of health expenditures. Furthermore, the OECD framework incorporated several additional elements:

• The consideration of “access” as a component of responsiveness, and the incorporation of equity of health service access within the framework;

• The conversion of health expenditure as a goal integrated with fairness in financial contribution, and the specification of a desirable level of health spending (building on the value of macroeconomic efficiency raised in the 1992 OECD report) as part of the framework;

• The decision to not weight goals in an index, instead reporting performance on several separate dimensions; and

• A focus on healthcare system performance, to the exclusion of public health activities and other wider health-promoting activities.

The development of the WHO and OECD health system performance assessment frameworks provided a foundation for performance assessment activities at the national and international levels, while setting the stage for further developments in assessment at the subsystem level. In one instance, the WHO Regional Office for Europe (WHO EURO) – as part of its Performance Assessment Tool for quality improvement in Hospitals (PATH) project in 2005 – developed a framework specific to hospitals and the clinical treatment provided in tertiary settings. As part of its theoretical model, WHO strategic orientations – the former health systems goals – were adapted into four “dimensions” of performance in the hospital subsystem context, namely clinical effectiveness, efficiency, staff orientation, and responsive governance (Veillard et al., 2005[71]). These dimensions were additionally overlaid with two transversal perspectives: safety, and patient-centredness.

Heightened attention towards the healthcare quality improvement agenda defines the later years of this phase. With a greater number of health system stakeholders and care settings involved in performance assessment through internal quality improvement, more sources of data became increasingly available to national governments and multilateral institutions. Given the range of new ideas emerging across healthcare systems for indicator development, efforts at the international level to organise this growth began to take shape in the years following the publishing of the international performance assessment frameworks. At the OECD, the Health Care Quality Indicators (HCQI) Project comprised an expert group representing 23 member states and supported by The Commonwealth Fund to define a conceptual framework for dimensions of healthcare quality to be measured by national governments. Launched in 2001, the Project aimed to build on developments since the OECD first proposed its broader Health System Performance Assessment Framework. Kelley and Hurst (2006[2]), presenting on behalf of the OECD, outlined the sub-framework applicable to the “Quality” dimension of the Health System Performance Framework of concern to the HCQI project, see Figure A A.1.

Figure A A.1. Proposed conceptual framework for OECD’s HCQI project (2001)

Source: Kelley and Hurst (2006[2]) “Health Care Quality Indicators Project: Conceptual Framework Paper”, https://doi.org/10.1787/440134737301.

Of note is the introduction of “healthcare needs” by the Project. At the intersection of the performance management and quality improvement agendas, the proposal for indicators to address both goals of the health system (including effectiveness, safety, and responsiveness/patient-centredness) as well as patient needs throughout the life‑course (such as “staying healthy”, “getting better”, “living with illness or disability”, and “coping with end of life”) provided a novel integration of the patient perspective in health system planning.

Additionally, the HCQI Conceptual Framework presented a vision for the lifecycle of health system performance assessment and improvement. Beginning with health system design, policy and context at the foundation of health production, health system performance is presented as a subsequent tier, permitting an evaluation of health system design by balancing the tensions between healthcare quality, access to health services, and costs from health expenditure. Through its direct impacts on the health of patients and indirect improvements to determinants of health outside the care system, health system performance is crystallised by the HCQI Conceptual Framework in its intermediary role between health system design and health outcomes.

By 2006, the OECD expanded on the details of the HCQI Conceptual Framework, proposing a variety of subdomains for consideration in the other stages of health production. In a publication for the International Journal for Quality in Health Care, Arah et al. (2006[6]) – representing the OECD and academic partners – added guiding questions for each domain of the HCQI Conceptual Framework, and further specified dimensions to the tiers of health system design and context, non-healthcare determinants of health, and health status.

While the OECD included only a select group of dimensions to be represented in the framework of the many explored in national performance assessment frameworks, the HCQI Conceptual Framework, and the wider quality agenda, provided a clearer picture of the role of health system performance in between the processes of health system design and the outcomes of health status.

Practicing “performance”: Strengthening the link between indicators and health systems policy action (2015‑16)

Following nearly a decade of healthcare quality indicator development in the multilateral, national government, and academic contexts, the OECD Health System Performance Assessment Framework developed during the HCQI Project shifted its attention to the actionability of international comparisons of healthcare quality. By 2015, the OECD Health Care Quality Indicators Expert Group, led by Carinci et al. (2015[3]), aimed to revise the HCQI components of the broad OECD framework to suit evolving policy needs and to validate the reliability of indicators as part of decision support tools.

Through a modified Delphi approach and consensus meeting, Carinci et al. (2015[3]) applied a variety of criteria to score HCQI indicators for their effectiveness as part of healthcare quality improvement data collections and to identify gaps for further research and development.

The recommendations of the Delphi panel and consensus meeting identified revisions to the broader OECD conceptual framework, as well as a remapping of current indicators comprising the HCQI Project into new dimensions. Carinci et al. described the revised structure of the framework in an article published with the International Journal for Quality in Health Care as follows:

“Minor changes to the original structure were agreed upon: change the wording of ‘staying healthy’ to ‘primary/secondary prevention’ to provide a clearer distinction with ‘living with illness and disability –chronic care’ [and] include the categories of ‘individual patient experiences’ and ‘integrated care’ under the theme of ‘responsiveness’, to pave the ground for future indicator development” (Carinci et al., 2015, p. 3[3]).

The revised framework can be seen in Figure A.2. The results of the remapping exercise significantly reduced the number of HCQI indicators included in the HCQI Project’s data collection for the year 2015 and onwards. Although the OECD originally identified indicators in primary care, acute cardiac care, and mental health as key priorities in the 2005‑06 HCQI documents, several of these indicators were slated for omission from future data collection in the exercise. Additionally, indicators centred on communicable diseases (such as indicators for coverage of the diphtheria/tetanus/pertussis (DTP), measles, hepatitis B, and influenza vaccinations in recommended populations) and cancer care (regarding mortality from breast cancer, cervical cancer, and colorectal cancer) were omitted from the regular HCQI data collection process.

Figure A A.2. Revised OECD framework for performance measurement

Source: Adapted from Carinci et al. (2015[3]), Towards actionable international comparisons of health system performance: Expert revision of the OECD framework and quality indicators, http://dx.doi.org/10.1093/intqhc/mzv004.

Despite the reduction in indicators, the exercise offered a clear evaluation of indicators as health system assessment tools, towards broader national efforts to address key condition-specific health burdens. With these validated indicators integrated into the OECD HCQI project, collaborative partnerships between multilateral organisations, national governments, academia, and civil society stakeholders became increasingly possible to firm up data collection efforts and support the translation of data into policy action.

Rethinking “performance”: People‑centredness, shocks and megatrends (2017‑22)

The challenges facing health systems have continued to evolve. As member governments grapple with shifting health burdens amid demographic trends, changing political interests, and socio-economic and environmental concerns, the conceptualisation, measurement, and actionability of health system performance assessment continue to be evaluated.

One key development has centred on the integration of communities served by health systems into performance assessment. A novel framework introduced by Newton-Lewis et al. (2021[72])aimed to characterise health systems as complex adaptive systems – systems with multiple actors organised independently yet collaboratively towards the production of individual and community health. While conventional health system performance assessment largely centres on the final component by viewing measurement as an isolated tool for generating policy action, the complex adaptive system conceptual framework instead promotes linkages between measurement and “enabling” directives like incentives and policy support to strengthen individuals and organisations to act in health systems. In the shift away from the directive approach in governing health systems, performance assessment is increasingly viewed as a tool to support measures carefully targeted within health systems for community-driven quality improvement in healthcare.

In the international policy space, performance assessment has also adjusted to reflect the needs of the people comprising and participating in health systems. At the OECD’s 2017 Health Ministerial Meeting, health ministers of the member states issued a clear mandate for the OECD to support the development of people‑centred health systems across the Organisation. Despite efforts of health systems to increase healthcare quality towards improved health outcomes, health ministers noted persistent inequalities, a rise in chronic disease and growing socio-economic and environmental risk factors to health (OECD, 2017[14]). From the Ministerial meeting, three calls to action were issued in a joint statement: promoting high-value systems for all, adapting health systems to new technologies and innovation, and reorienting health systems to become more people‑centred, while encouraging dialogue and international co‑operation. In response to this third call to action, the OECD put forward a People Centred Health Systems Framework in its publication Health for the People, by the People: Building People‑Centred Health Systems in 2021. As part of its framework and pursuant scorecard, indicators generated from OECD Health Statistics, the European Quality of Life Surveys, and the Commonwealth Fund International Health Policy Surveys were collated under five dimensions of people‑centredness: “voice”, “choice”, “co-production”, “integration”, and “respectfulness” (OECD, 2021[4]).

To translate this framework into cross-country comparative indicators, the OECD launched the Patient-Reported Indicator Survey (PaRIS) initiative, a ground-breaking international survey comprised of Patient Reported Outcome Measures (PROMs) and Patient Reported Experience Measures (PREMs) aimed at measuring how health systems respond to patient needs (OECD, 2019[73]). While several PROMs and PREMs survey instruments had been previously developed and validated in clinical research settings and in use by some national governments, the goal to build capacity in people‑centred measurement across 20 participating countries was a milestone development. This shift in characterizing performance as dependent on the patient experience promises to generate new insights in the practice of care alongside existing macroeconomic and microeconomic indicators.

In recent years health systems have also faced significant shocks. As they were dealing with rapid and successive waves of COVID‑19 infections between 2020 and 2022, the mass disruption to supply chains and redirection of human and financial resources for emergency response stressed care pathways for chronic conditions in many countries. During this time, the OECD developed a health systems resilience programme, outlined in the 2023 publication of Ready for the Next Crisis? Investing in Health System Resilience. With major vulnerabilities identified in underprepared, understaffed, and underinvested health systems across member states, the OECD recommended significant policy action to promote population health, workforce retention and recruitment, data collection and use, international co‑operation, supply chain resilience, and governance and trust (OECD, 2023[5]). In addition to performance assessment strategies, the OECD cited the importance of preparedness through the use of frameworks for data-driven decision-making and co‑ordination for effective responses during the disruption of a shock to health systems. In this context, building on the lessons of the pandemic and considering further shocks, the OECD Resilience Shock Cycle Framework served as the basis for the OECD analysis of health system resilience (OECD, 2023[5]).

As multiple crises increasingly compete for attention from health system stakeholders, several “megatrends” are also shaping the architecture of performance assessment. The OECD and its national and multilateral partners are centred on indicator development for mitigation and adaptation strategies. Amid climate change, the development of new data governance and co‑operative structures as health systems digitalise, improvements in long-term care as populations age, and promoting gender-informed care to reduce gender-based inequalities in health outcomes.

In that note, the latest WHO Health System Performance Framework published in 2022, demonstrates this heightened role of whole of government and whole of society approaches in international performance assessment, drawing relationships between intersectoral and intersectoral actors across health system functions, intermediate objectives, and final goals (Papanicolas et al., 2022[12]).

Another widely used framework in the area of public health is the Dahlgren-Whitehead model, also known as the rainbow model (Dahlgren and Whitehead, 2021[11]). An important difference between this framework and those discussed above is that this model is not a health system framework but a framework on the determinants of health. This model provides a broader picture and reminds us that although health is the main outcome of most health systems frameworks, health systems as a whole are among a range of other factors that affect people’s health.