Sustainable and practical application of the HSPA framework, the governance structure, and other aspects of the project are considered essential for the HSPA to function as intended and meet the purpose and scope set for the effort. Given this, several aspects related to the implementation of the HSPA were considered throughout the project. Section 5.1 outlines the proposed frequency and implementation roadmap for the Estonian HSPA in relation to its defined purpose. Section 5.2 details out the related workflows for collection and analysis of HSPA health data. Section 5.3 then describes the importance of and proposed activities to support dissemination of the HSPA results. Section 5.4 details activities undertaken and proposes further actions to build capacity to ensure successful implementation and sustainability of the HSPA.
Health System Performance Assessment Framework for Estonia
5. Implementation of the HSPA
5.1. Frequency of reporting and implementation plan for the Estonian HSPA
During project meetings, discussions on the possible timing of consecutive HSPAs were held. The frequency of the assessment cycle can vary depending on the need. Drawing from experience in other countries, it was proposed that the need for (interim) evaluations of the NHP and elections of the Estonian Parliament could be considered in the timing of the consecutive HSPA cycles (Figure 5.1). Harmonising the HSPA with other needs for assessment will avoid duplication of work, allowing for more efficient use of resources. Aiming to publish the results ahead of elections will inform policy discussions and coalition agreements of consecutive governments, thus steering needed changes. The next interim evaluation of the objectives set in NHP 2020‑30 was planned to take place in 2025, while elections of the Estonian Parliament take place every four years (2027/2031/etc.). The importance of publishing results sufficiently ahead of elections, i.e. at least half a year, was considered to avoid discourse leading to politicisation of health issues. The first full report with the results of the Estonian HSPA is planned to be published in the second half of 2024, while the second HSPA report is planned for 2026 to be released before the election; the consecutive HSPA cycles are planned follow with a 4‑year interval.
Figure 5.1. Proposal for the timings of two consecutive HSPA cycles
Source: The Estonian HSPA framework project.
Figure 5.2 lays out the roadmap to implement the HSPA, including steps to be taken by each of the organisational entities in the governance model outlined in Section 4.1. As a preliminary estimate, it could take approximately one year from the confirmation of the definitive list of indicators for each cycle to publication of the results. During the assessment period, no further modifications to the framework should take place. After the assessment, there should be sufficient time between the cycles to disseminate the results, implement changes to the HSPA framework and indicators based on the results, and allow the HSPA structures to adapt to them. During the period between the assessment cycles, possible improvements to the HSPA, including inclusion of further indicators and changes to methodology as needed, can be discussed. The proposed governance model foresees regular meetings held between stakeholders to ensure continuous work on the dissemination of the results and improvement of the framework (see Section 5.3).
Figure 5.2. Implementation roadmap for the Estonian HSPA
Source: The Estonian HSPA framework project.
5.2. Workflows for the collection and analysis of HSPA health data
High-quality data delivered in a timely manner is crucial to conduct accurate and timely assessments of health system performance. Although Estonia collects a substantial amount of data across the health system, there is untapped potential to better leverage the existing data. Information on the heath data infrastructure and policy context in Estonia were thoroughly reviewed in the Situational Analysis Report developed as part of the Estonian HSPA framework project (OECD, 2022[2]).
Some of the main challenges related to health data collection and analysis in Estonia include the fact that data collection process is fragmented and a unified and co‑ordinated approach to national health data governance does not exist. The process to access data in different datasets is perceived as long and bureaucratic, and there are no standard data sharing agreements. Therefore, secondary use of data, such as generating reliable and timely statistics, can be complicated. Indeed, only a quarter of available health datasets in Estonia are regularly linked for research, statistics, or monitoring purposes (OECD, 2022[6]). Key challenges were identified in legislation, more precisely relating to data protection rules that make linking data complicated and time‑consuming. Nonetheless, the country performs well in terms of dataset availability, maturity, and utilisation (Oderkirk, 2021[7]). A unified and co‑ordinated approach to national data governance can enable smooth information exchange and use for a range of purposes without compromising privacy, security, and ownership of data.
5.3. Workstream for dissemination and policy engagement
Effective sharing, use and dissemination of data analysed for the HSPA are crucial to aid correct interpretation and lead to evidence‑based policy planning. Proposals for dissemination of the HSPA results were discussed by the Principal Working Group. As the Estonian HSPA is aimed to raise public awareness, increase transparency of the health system, and accountability of its stakeholders, dissemination of the HSPA results should reach various interest groups, including politicians, policy makers, hospital managers, healthcare providers, local authorities, other stakeholders throughout the health system, and the general population.
Firstly, to ensure wider awareness of the HSPA process and results early on, the HSPA must be implemented as a collaborative and inclusive process involving a wide range of stakeholders from the beginning. Thus to ensure engagement of all key stakeholders, the HSPA framework for Estonia has been developed through an inclusive process involving numerous rounds of consultations, meetings, focus groups, and workshops (Annex A). To ensure further continued engagement, the governance model (Section 4.1), HSPA implementation plan (Section 5.1), and assignment of indicator custodians (Section 4.3) of the HSPA process outline the structures and responsibilities of various organisations for continuous collaborative work, combining quantitative analysis with qualitative insights and contextual understanding. Active participation and continuous feedback will be encouraged to capture diverse perspectives and contextual insights.
Second, the process of implementing the HSPA should be transparent and straightforward. To do so, methodologies and assumptions made in the analysis of data must be clearly documented and communicated in all materials accompanying the publication of HSPA results. This will allow all stakeholders to understand and follow the process, fostering trust and credibility in its results. Results and recommendations developed throughout the analysis in the HSPA should be specific, actionable, and communicated in a clear and concise manner.
Thirdly, knowledge translation plays a critical role in bridging the gap between research findings and practical transformation into health policy. Engagement strategies should be developed with the aim of reaching a wide range of stakeholders. Thus, multiple channels should be used to amplify the outreach and impact of the HSPA. The primary publication of results will take place in the form of a report presenting each indicator alongside key results, while publication of targeted complementary reports will be considered to highlight domains or themes of particular interest. To make the HSPA more accessible and engaging, including to non-expert audience, a dedicated interactive online platform will be developed, which will include relevant results, methodologies, and dashboards. Annex F outlines some options for presenting the results visually.
Furthermore, dissemination activities were included in the governance model of the HSPA (Section 4.1) and the implementation roadmap (Section 5.1), developed by the core working group and approved by all key stakeholders. Specific responsibilities for disseminating the results were assigned to some organisations, arising from their specific roles and responsibilities in the health system. The NIHD will ensure public awareness. The EHIF will communicate the results to their contract partners while Health Board will use them in its proactive role in service quality assurance, safety, and surveillance (Section 4.2). The MoSA, within its leadership role of the HSPA Co‑ordination Board, will bring different parties together and ensure effective communication between them, including common understanding and objectives throughout the analysis and consecutive change management.
The Co‑ordination Board will additionally organise seminars and events on the HSPA and using information analysed in it with the aim of continuously engaging all stakeholders and interest groups, making the HSPA process more interactive, invite in-depth reflection from all parties, and raise awareness of the HSPA. Continuous feedback will be collected from stakeholders throughout each HSPA cycle.
5.4. Building national HSPA-related capacity
Building capacity entails developing and strengthening the skills, competencies and resources that help conduct necessary activities and adapt to changing priorities. As national capacity is vital for successful implementation and sustainability of the HSPA, activities to support its development were built into the HSPA framework development project, and concrete proposals were formulated to support future capacity building activities in different sectors across the health system. This component sets the Estonian HSPA project apart from other similar initiatives.
The special focus on capacity in this project aimed to ensure analytical capacity – abilities to perform analyses, draw analytical conclusions from emerging trends and effectively disseminate the information. Additionally, a wider conceptualisation of capacity was considered in the aim to increase awareness of the HSPA within the health system and build capacity to implement the results of the analyses for evidence‑based improvements of the health system.
5.4.1. Activities undertaken as part of the project
During the project, a broad range of participants took part in seven technical focus groups where domains of the HSPA framework, potential indicators, and their relevance to the areas of work of each participant were discussed (see Annex C). These indirectly contributed to increasing capacity in the Estonian health system by increasing the awareness of the interconnectivity between aspects of the health system and inviting in-depth consideration of individual domains in people working within the Estonian health system.
Additionally, a capacity building workshop was held on 6 February 2023, outlining the prerequisites of applying HSPA capacity building successfully. The workshop contributed to capacity building by providing key institutions an opportunity to discuss different ways of building capacity by highlighting the strengths and limitations of different parts of the health system. The ideas discussed led to suggestions of future capacity building activities (Section 5.4.2).
Finally, the University of Tartu enhanced educational activities by involving elements of the HSPA in the Master of Public Health curriculum. Some students may continue to work as analysts in any of the institutions identified as stakeholders of the HSPA, and thus creating opportunities for them to engage with the HSPA on a conceptual level during their studies can contribute to increasing capacity in analysis and implementation. The University has introduced the HSPA to the first-year Master of Public Health students, disseminated information about it via e‑mail, invited the students to participate in focus groups and made the recordings of the focus groups available to them alongside a study guide. Themes of the HSPA have additionally been proposed as a master’s thesis theme to Master of Public Health students.
5.4.2. Future capacity building activities
Given the importance of continuous capacity building for continuity and improvement of future HSPAs, future capacity building activities focusing on higher education, organisations involved in the HSPA and the whole of the health system were discussed. Target groups like medical students, students of nursing or midwifery, masters’ students in healthcare colleges and universities, working analysts and decision makers in other parts of the health system, including senior leadership of organisations, politicians, hospital quality managers and health promotion specialists were considered.
At universities, topics related to the HSPA should be further integrated into standard health‑related curricula. The teaching should cover how an assessment of a health system can be made, provide an overview of the framework, touch on indicator selection (using the focus groups in Annex C and indicator selection criteria in Section 3.2 as a source) and final indicator list chosen for the Estonian HSPA, as well as how aspects of the Estonian health system can be benchmarked both internally as well as against other countries. Teaching could include international and national experts who could give a high-level overview about certain domains of the HSPA, how indicators in those domains are validated, calculated, how conclusions are drawn from the results and how those indicators are used in international benchmarking. In the future, stakeholders with different roles in the implementation of the HSPA framework should be invited to discuss how they implement and use the HSPA framework.
Organisations involved in the implementation of the HSPA can make use of avenues to enhance their internal analytical capacity. This can be achieved by providing opportunities for further training and professional development for their analytical workforce. Drawing from previous experiences shared by participants in meetings conducted during the HSPA project, personal development plans could enhance the training and skillset of analysts and can increase job satisfaction and retention of workforce. It is also essential to ensure continuity of analytical capacity by incorporating it into organisational processes, including ensuring sufficient documentation and transfer of institutional knowledge, as well as engaging with the higher education sector to increase interest and generate awareness of their work. Inclusion of trainees and interns in the work of organisations working on implementing the HSPA should be considered. To benefit from analytical contributions from the academia in further developing and exploring key areas of the HSPA, organisations should continue to prioritise timely publication of key datasets through user-oriented open data portals that can be used for analysis in the academia.
Furthermore, close linkage and collaboration between institutions responsible for the collection and analysis of data in the field of health is essential to improve efficiency and reduce duplication of work across the health system (see Situational Analysis (OECD, 2022[2])). Processes for sharing data between institutions should be simplified while secure innovative data sharing solutions should be used. Regular meetings to discuss and improve the HSPA should be held in line with the implementation plan to ensure continuity of optimal implementation and use of the HSPA, and its adaptation to changing priorities over time.
Efforts to build capacity should consider capacity to adopt the results and make use of them within the whole health system, by stakeholders from hospital quality managers to policy makers, who could benefit from the disseminated results of the HSPA to guide improvements in the health system within the scope of their work. This line of priorities will rely on effective engagement and continuous bidirectional communication and dissemination of the results of the HSPA beyond the parties directly involved in its implementation. For policy, the timing of the publication of consecutive rounds of results (Section 5.1) can help ensure its optimal use.
Box 5.1. Recommendations for future HSPA-related capacity building
Healthcare colleges and universities should integrate topics related to the HSPA framework to their current curriculums and source out some teaching activities to external experts.
Organisations involved in the analysis of health data should invest in professional development of their analytical workforce by developing personalised training plans and continuously investing in opportunities for further staff training and professional development.
Organisations should link with academia by engaging in teaching, providing opportunities for traineeships and internships in fields related to the HSPA. This may also involve contributions from academic researchers to the analysis of health data by ensuring open access to key data.
Organisations should engage in continuous data linkage and collaboration, sharing data with each other through simplified access mechanisms and in-depth discussions on the interpretation and utility of the various conclusions drawn from the HSPA.
A strategy for effective, targeted, and continuous dissemination of the HSPA and its results should be developed to engage a broader audience of stakeholders.
For additional information:
For guides on utilisation of the focus groups for further HSPA capacity building see: