This section reviews the indicator selection process and criteria applied to populate the HSPA framework (Section 4.1). The list of indicators selected and suggested for the first the Czech HSPA implementation is provided in Section 4.2. More details on indicators within each domain and subdomain, including information on possible data disaggregation, indicator methodologies, data sources, data custodians, and indicator custodians is provided in Annex D.
Health System Performance Assessment Framework for the Czech Republic
4. Populating the HSPA framework with indicators
4.1. Indicator selection process and criteria
The selection of indicators for populating domains relies on two main concepts 1) the meaningfulness of the indicator for performance assessment and 2) quality of underlying data and feasibility of measurement. A balance must be reached between the conceptual relevance of indicators and the practical possibilities for measuring and reporting on the selected indicators (Figure 4.1). Not all selected indicators must be fully feasible and usable at the moment of the design of the HSPA – some may be aspirational – but conisation should be given to the commitment and capacity to develop and operationalise these indicators as part of the HSPA implementation plan.
For the purposes of the Czech HSPA, selection criteria for indicators also included an assessment of health system strategic priorities and goals dimensions (e.g. efficiency, resilience, access, person-centredness, and quality of care) and desired performance sub-dimensions. Other considerations included the feasibility of reporting on the indicators, the validity and reliability of selected indicators, and applicability for international benchmarking. In the final selection process, a third aspect was also considered – indicators selected for a specific domain or subdomain were assessed as a group. The criteria applied focused on whether the selected indicators as a group describe the (sub)domain in its entirety. In other words, a group of excellent, relevant indicators calculated from reliable data may not be enough if those indicators describe the given domain, or subdomain, only in part and not entirely. In such case, discussions were held in the principal working group on enlarging the indicator group by another existent or placeholder indicator to capture the full scope of a (sub)domain.
Figure 4.1. Trade‑off considered in HSPA indicator selection
Source: The Netherlands HSPA Development.
Box 4.1. International experience with selecting HSPA indicators
Countries have taken different approaches in the selection of indicators to populate their HSPAs during the development process. Malta scored indicators on three criteria: 1) importance 2) feasibility and 3) scientific soundness. The importance criteria accounted for 40% of the score, and the other two criteria accounted for 30% each. Indicators scoring above 60% were considered for inclusion. In Latvia, indicators were scored on value (usability to assess a benchmark/goal) and ability to assess trends. Ireland also assessed indicators on two main dimensions: fit for purpose (alignment to HSPA functions) and if they are fit for use (measurable and methodologically robust).
In Slovenia, indicator selection was achieved via a consensus development process. Indicator performance is then combined with 0‑3 “Importance score” in reporting. The Netherlands also used a consensus development process, looking at both the healthcare system’s objectives to determine the indicator domains and relevant indicators to be used, while at the bottom the data sources and scientific state of the art determine the data availability and reliability to populate indicators.
The discussion on potential indicators for the Czech HSPA was launched in the working group during its 5th workshop in April 2022, after the first draft of HSPA framework was developed. The discussion has benefitted from an overview of health data availability and data and information flow landscape in the Czech Republic, prepared as a meeting background document by the OECD team; an overview of the Czech health data custodians is provided in Annex E. First ideas on possible indicators were collected from the break-out group brainstorming on possible indicators during this workshop.
Following the first draft of HSPA report, multiple technical focus groups were held to discuss data availability and possible indicators in particular domains in detail, with the OECD team providing international expertise and knowledge inputs into the national experts’ discussion; for an overview of technical focus groups see Annex C.
As an initial step for the indicator selection process, a long list of more than 1 400 indicators was composed at the Ministry of Health. The list included indicators reported and published by the main health data custodians in the Czech Republic; indicators reported by the Czech Republic to international databases such as Eurostat and OECD; indicators published by international organisations and projects based on nationally submitted data (such as the OECD’s Potential years of life lost, and SHARE survey results); indicators featuring in the Czech health sector strategic documents; and indicators developed as placeholders from the various technical discussions held during this project.
Due to the nature and process of collecting indicators for the long list, some indicators were featuring more than once on the list (drawn from multiple sources) or provided different granularity and/or measuring units for the same data. Hence, the long list was shortlisted by the project management team to approx. 400 indicators. The shortlist was created mainly by removing duplications and by grouping similar and overlapping indicators together under one indicator name. Furthermore, priority was given to indicators reflecting national health system strategic priorities and goals, the availability and feasibility of indicator reporting, the validity and reliability of primary data, and the applicability of international benchmarking.
Majority of those indicators were already in use, regularly reported to Eurostat, OECD, or other international organisations.
The shortlist of approx. 400 indicators was sent to members of principal working group. The members were given several votes in each subdomain that they could use to select indicators that would – in their view – serve the best the purpose of HSPA (Table 4.1). They were asked to assess both the fit-for-use of each individual indicator and the fit-for-purpose of individual indicators as well as the group of indicators that they choose to vote for in each subdomain.
Table 4.1. Number of votes by subdomains assigned in the indicator scoring exercise
Note: The votes were assigned to domains and subdomains according to the draft the Czech HSPA framework as of February 2023. Small adjustments were yet made to the HSPA framework following the results of the indicator scoring exercise and the subsequent expert consultations.
Source: The Czech HSPA project, February 2023.
The total of 24 experts from 16 institutions participated in the indicator scoring exercise, representing healthcare providers (26%), patients (10%), the system view (42%), and health insurance funds (21%). The votes assigned by individual experts were weighted to account for the fact that various stakeholder groups (e.g. patients vs providers) were not equally represented. The choice of indicators across domains was surprisingly unanimous among stakeholder groups, which made the pre‑final selection of indicators largely consensual.
The results of stakeholder voting were presented back to the principal working group for feedback and further discussion during workshops 7 and 8 (held in March 2023), providing detailed discussion on each individual selected indicator as well as on the (sub)domain’s indicator entirety. A few ideas and suggestions came up during those workshops, which led to further refinement of the HSPA framework and to withholding some indicators and adding few others. The list of 119 indicators was sent for review to the main the Czech data custodians and individual consultations of HSPA project management team were then held with representatives of institutions that shall be the indicator custodians of most of the HSPA indicators. In-depth discussions were held with CZSO, National Institute for Public Health, and UZIS. Various data sources were considered for indicators with data source options, sometimes an indicator was replaced by a similar yet more relevant one; in some cases, an indicator was excluded for known problems with primary data validity. The final list of 122 indicators was sent to the principal working group for a final review and presented during the last principal working group meeting in April 2023.
Figure 4.2. Indicator selection process to populate the Czech HSPA framework
Source: The Czech HSPA project.
4.2. Indicators selected for the Czech HSPA
In total, 122 indicators were chosen to populate the Czech HSPA framework. The choice was made through the iterative process of consultations with the working group’s members. This process benefited from the scoping and the scoring exercise as described in Section 4.1 above. As the emphasis was put on data availability, most of the indicators come from existing sources and are ready to be implemented for the first round of the HSPA in the Czech Republic. However, some 32 other indicators are placeholders, where further development is needed. These are of two types. Primary data may be already existent (are collected), but there is not yet a clear consensus on the indicator methodology. Or, alternatively, an indicator has been inspired by an existing indicator methodology in another country or international organisation and its methodology needs to be localised to the Czech content and data availability requires checking and/or further development. Figure 4.3 presents the numbers of indicator in each framework domain and subdomain.
Figure 4.3. The final number of indicators per (sub)domain in the Czech HSPA framework
Note: Number of indicators for domains are in yellow circles, number of indicators for subdomains are in grey circle.
Source: The Czech HSPA project.
The Table 4.2 presents the list of selected indicators by the 12 HSPA framework domains and the 28 subdomains. For each indicator, information is provided on the domain, subdomain and indicator title. Detailed information on indicators is provided in Annex D. Indicators were selected to reflect the scope and depth of individual domains but also to reflect ongoing health-reforms or to shed light on existing problems – such as the state of mental health, financial affordability of dental care, progress in e‑health and others. The list of indicators selected during this project shall serve as an input for the next HSPA implementation phase. Further discussions among HSPA stakeholders, mainly the data custodians and suggested indicator custodians, are foreseen and deemed appropriate to develop detailed technical sheets for each individual indicator (see Section 6.2).
Annex D provides detailed information on indicators per framework domains. For each domain, a summary of selected indicators is provided together with a dedicated table. This table includes information on each indicator’s possible data disaggregation, possible benchmarking (international, regional, and time series, if available), indicator methodology, primary data source, data custodian, and indicator custodian. Primary data source can be health insurance administrative data (claim data from providers, custodied by the UZIS in the National Registry of Reimbursed Health Care, NRHZS), clinical or population registry data, and various reporting and survey data. In the last case, name of the survey is also provided. Data custodian indicates the institution, which is responsible for collecting the data and its custody, whereas indicator custodian is the institution responsible for contextualising and interpretation of the indicator within the Czech HSPA. In many cases, data custodian and indicator custodian is the same institution, while in other the data and indicator custodians differ.
In certain cases when alternatives exist, decision has yet to be taken on the choice of the data source, which will have a direct impact on the possible indicator disaggregation. Discussions during the HSPA implementation phase are foreseen for instance on the selection of survey data versus administrative (claim) data for particular indicators. While the former allows for socio‑economic status disaggregation, the latter are more accurate and allow for regional comparison. Similarly, in case of some other indicators further discussions are to be held on the use of international versus national survey results.
Table 4.2. List of proposed indicators for the Czech HSPA
The list of proposed indicators is the result of iterative consultation process held with the Principal Working Group members and other experts and the indicator scoping and the scoring exercise and shall serve as an input to the HSPA implementation phase.
Source: The Czech HSPA project, April 2023.
For additional information:
Annex D: List of indicators populating the Czech HSPA framework.
Overview tables of individual indicators by subdomains, with details on data disaggregation availabilities, methodology, data sources, data custodians, and indicator custodians.
Infographics showing selected existing indicator and their possible disaggregation, benchmarking, and visualisation.
Annex E: Overview of the Czech health data custodians and data sources