This chapter contains a review of national and transnational standards that have been proposed or adopted across OECD and EU countries to assure the quality of digital education. It also analyses the different options available to public authorities to support improvements in the quality of digital higher education. Finally, the chapter provides principles for institutions to consider when developing strategies to improve the quality of digital education, and recommendations for Croatian public authorities to reflect on in their policy deliberations.
Advancing Digital Maturity in Croatia’s Higher Education System
Higher Education
4. Enabling standards, supports and practices for high-quality digital higher education in Croatia
Abstract
Policy context
Traditionally, quality assurance processes and standards in higher education were primarily focused on face-to-face education. Before the COVID‑19 pandemic, digital technologies were increasingly integrated into higher education operations, and online education was gaining popularity in many countries. Consequently, quality assurance agencies and institutions began re-evaluating their quality standards. The pandemic significantly accelerated this shift by forcing a widespread transition to online higher education and driving emergency investments in digital infrastructure across OECD and EU countries.
In Croatia, as in other countries, it is now time to consider the reactive innovations required by the pandemic. The aim should be to build on what worked well and address issues requiring either further development or a change in direction. The Croatian higher education system has already made substantial progress on these deliberations, as evidenced by the 2022 amendments to the laws governing science and higher education (Box 4.1)
The revision of quality assurance criteria alone is unlikely to lead to quality improvements. There needs to be a simultaneous investment in people. One of the key urgent challenges across countries is the need for further development of digital competence and culture across higher education institutions. Digital competence and culture refer to the abilities of people within higher education institutions, both individually and collectively, to contribute to digital innovation within their organisation and to utilise digital technologies for enhancing teaching, research, operational, and administrative performance.
This chapter aims to support Croatian stakeholders and higher education institutions as they move forward with reforms and investment in high-quality digital education. It focuses on public supports, capacity developments and organisational culture shifts required to deepen digital competence. After some definitions, the chapter commences with an overview of recent transnational and national trends in quality assurance, reflecting on the balance quality assurance agencies are trying to achieve between taking an integrated approach and addressing the specifics of online and hybrid delivery modes.
The chapter then presents a categorisation of recent examples of public support for higher education institutions across OECD countries, which aim to improve the quality of digitally delivered or digitally enhanced higher education. The concluding section highlights institutional approaches that may improve the effectiveness of digital higher education and proposes a framework for institutions to build their own improvement strategies. The institution-specific framework is complemented by a set of institutional guidelines also developed as an output of this project (see Annex A).
Box 4.1. Croatia’s regulatory framework for quality assurance of digital higher education
Quality assurance of higher education institutions is the responsibility of the Agency for Science and Higher Education (ASHE) and the Ministry of Science and Education, as well as the institutions themselves. Croatia’s external quality assurance framework is co‑ordinated and implemented primarily by the ASHE. The responsibilities of the ASHE are prescribed by law. They encompass initial accreditation and re-accreditation of higher education institutions and study programmes, as well as external evaluation of higher education institutions internal quality assurance mechanisms. The ASHE also carries out a range of research activities and thematic analyses using data collected from higher education institutions and administrative data systems to identify strengths and areas where further action is needed to improve quality of provision.
In Croatia, as in other European countries, the ASHE increasingly co-ordinates its actions with other national quality assurance bodies. Croatia’s external quality assurance framework was established with regard for European best practice and seeks to inform its operations with best practices promoted by the European Network for Quality Assurance Agencies in the European Higher Education Area (ENQA) including the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG).
An amended Law on Quality Assurance in Higher Education and Science (Parliament of Croatia, 2022[1]) entered into force in December 2022. One provision of the amended law assigns primary responsibility for the development of criteria for the establishment of online higher education programmes to the ASHE. The Agency is responsible for authorising wholly online programmes, defined nationally as programmes where more than 50% of the subjects are delivered online (see Box 4.3).
Source: Agency for Science and Higher Education (n.d.[2]), Scope of Work, https://www.azvo.hr/en/about-ashe/scope-of-work/activities
Higher education, digital technologies, and modes of delivery
There are many different definitions describing the mode of delivery of education programmes, often with limited mutual understanding evident across – or even within – countries. “Distance education” is one of the oldest terms used to describe the delivery of education where the learner is at a physical distance from the provider. It is generally understood to refer to all modes of delivery of education other than in-person (e.g. correspondence courses, TV and radio delivery, and online delivery using digital technologies).
In recent decades, many other terms have been coined to categorise the mode of delivery of higher education (Finol, 2020[3]). Following review of the various definitions in use, recent OECD analysis (Staring et al., 2022[4]) defines three categories of online delivery of higher education as follows:
Fully online education: All instruction is delivered online, either synchronously or asynchronously, or in combination. While instruction is wholly delivered online, learners may still be provided with opportunities to interact with peers or instructors in person, or to access on-campus facilities.
Hybrid education: Instruction is delivered using a mix of online and on-campus instruction, with the online components taking place synchronously, asynchronously, or simultaneously with in-person instruction (the latter category being increasingly labelled as “hyflex” education).
Blended education: Instruction generally takes place in person and is supplemented, but not replaced, by online materials and activities such as a virtual learning environment, open educational resources, simulations, or gaming. (Siegelman, 2019[5]).
Various forms of online and hybrid education exist along a continuum of learner exposure to online material. This ranges from programmes with no online component to varying degrees of web-supported offers and eventually to programmes that are primarily or fully web-based (Guiney, 2016[6]).
A related group of definitions refer to the wider concept of digital education (often also called e-learning or digitally-enhanced teaching and learning), which includes not only using the internet to deliver education, but also using a range of digital tools and technologies to augment, streamline or enhance teaching and learning both in person and online (Qayyum, 2022[7]).
Digital education is, therefore, a broad term encompassing a wide range of digital tools, technologies, and resources to enhance learning. The extent to which a higher education programme integrates digital technologies will vary regardless of the mode of delivery of the programme. As digital technologies become more embedded in higher education systems it becomes more difficult to assign programmes to meaningful and mutually exclusive categories such as online, hybrid or blended. A more realistic approach, reflecting the complexity of today’s higher education offer, may be to map higher education programmes within a two-dimensional framework, considering both the intensity of use of digital tools and technologies and the proportion of the programme delivered through online channels (Figure 4.1).
Online or hybrid education necessarily employs digital technologies, but the extent to which it makes use of digital technologies can vary from simple recording and/or streaming of a live lecture (i.e. minimal technology use) to the employment of various evidence-based digital tools to promote engagement and learning (i.e. heavy technology use) (Figure 4.1). While this chapter focuses on improving the quality of online and hybrid learning, it recognises that successful high-quality online and hybrid learning necessarily depends on effective digital education. The analysis in the next sections proceeds on the principle that strategies for achieving high quality in online teaching and learning should be considered within, or be connected to, wider strategies for digitalisation in higher education.
Figure 4.1. A two-dimensional framework for mapping digital higher education programmes
An emerging principle from the deliberations of quality assurance bodies is that the fundamental conditions for high-quality education are, or should be, the same regardless of mode of delivery. Nevertheless, there are indications from international research evidence that further work is required before this vision can become a reality. While few rigorous studies of the impact of online education have taken place in Europe, available experimental and quasi-experimental evidence, primarily from North America, provides a mixed picture of the impact of online learning on student access, progress and achievement, and cost efficiency (Table 4.1).
As in most countries, the COVID‑19 pandemic forced Croatian higher education institutions to switch their entire education offer, almost overnight, to remote learning. The situation in Croatia was further exacerbated by the 2020 earthquakes. These caused considerable damage to buildings and necessitated prolonged periods of emergency remote instruction in some education institutions. Emergency remote teaching was generally considered to have been efficiently implemented in higher education institutions. During structured interviews conducted as part of this project many practitioners and institution leaders, however, expressed concerns to the OECD team about the potential adverse impacts of emergency remote instruction on quality, due to reliance on assessment procedures, educational materials and teaching methods not designed to be used in an online environment.
The studies presented in Table 4.1, and the experience of many higher education institutions during the emergency period of COVID‑19 indicate that caution is needed when moving from traditional face-to-face instruction to online modes of delivery.
There is also evidence, however, to suggest that many challenges related to providing higher education in online or hybrid mode could be surmounted by careful design of interventions tailored specifically to the online context, by iterative improvement of web-based or web-enhanced offers, or by more thoughtful use of digital tools. For example, a recent randomised controlled trial in Germany which offered remote, online peer tutoring to students during the pandemic led to significant improvements in remote students’ grades and credit acquisition (Hardt, Nagler and Rincke, 2022[8]). In New Zealand, a 2016 study showed that as web-enhanced and web-based courses became embedded and normalised in the tertiary education, gaps in completion rates between extramural (remote) and intramural (on-campus) disappeared for most groups of students (Guiney, 2016[6]).
Uncovering and sharing examples of good practice can help support better outcomes for students studying in online or hybrid modes. If higher education institutions are to make effective use of digital technologies in teaching and learning, they need support from quality assurance agencies and other public authorities to discover and assimilate emerging knowledge of good practices, and to design their own digitally enhanced teaching and learning innovations. They also need clear mechanisms to collaborate and exchange on their own best practice discoveries.
This chapter provides a review of emerging standards, supports and practices that can enable high-quality education to be partially or fully delivered online and argues such measures should be cultivated within a coherent and collaborative institution-wide strategy for digital education aligned to national and international policy developments.
Defining standards, supports and practices
High-quality digital education requires a range of enabling factors. Quality can be supported by a set of clear and cohesive standards, set by external quality assurance bodies, for the development and delivery of digital education. Such standards can help to set clear expectations for the measures that need to be taken to ensure high-quality, consistent delivery of teaching across national higher education systems (OECD, 2021[9]).
At the same time, the existence of standards alone is not sufficient to support widespread access to high-quality education. Institutions are likely to need a range of additional external supports to ensure they can achieve the standards required by external quality assurance bodies. Such supports may include, for example, public financing, staff training and mechanisms to incentivise co-ordination and collaboration among institutions.
Finally, institutions can promote quality through their own internal practices including internal policies, leadership on digitalisation, teaching and learning improvement, and mechanisms to support staff and student skills development.
This chapter reviews in turn emerging trends for each of the enabling factors supporting the delivery of high-quality digital education:
national and transnational standards as developed and evaluated by organisations external to higher education institutions;
supports, as provided by governments, public bodies, and supranational organisations to enable higher education institutions to improve quality standards;
practices, internal to higher education institutions, which can help to build the foundation of digital competence and culture to support successful online and hybrid learning.
Table 4.1. Selected studies on the impact of online, blended and hybrid higher education
Sources: Bowen et al (2014[10]), Interactive learning online at public universities: Evidence from a six-campus randomized trial, https://onlinelibrary.wiley.com/doi/10.1002/pam.21728; Bettinger et al. (2017[11]), Virtual classrooms: How online college courses affect student success, https://www.aeaweb.org/articles?id=10.1257/aer.20151193; Cellini and Grueso (2021[12]), Student Learning in Online College Programs, https://doi.org/10.1177/23328584211008105; Hemelt et al (2021[13]), Why is math cheaper than english? Understanding cost differences in higher education, https://www.journals.uchicago.edu/doi/full/10.1086/709535; Cacault et al. (2021[14]), Distance Learning in Higher Education: Evidence from a Randomized Experiment, https://doi.org/10.1093/jeea/jvaa060 and Kofoed et al (2021[15]), Zooming to Class?: Experimental Evidence on College Students' Online Learning during COVID-19, https://docs.iza.org/dp14356.pdf.
National and transnational standards for digital higher education
Prior to joining the European Union, Croatia had already committed to fully adopting the Bologna Process and restructuring its higher education system to align with European norms. While there remain significant differences across Europe in the ways higher education institutions are created, regulated, and funded (EHEA, 2022[16]) there is increasing convergence on quality assurance standards and guidelines being adopted to assure higher education. This convergence of norms is beginning, to some extent, to cover higher education provided online. This section provides an overview of current trends in the evolution of both transnational and national quality assurance standards for digital higher education, with a focus on online and hybrid delivery of education programmes.
Transnational approaches
Transnational quality assurance bodies work to ensure comparability of study programmes across countries. This comparability supports trust in the quality of higher education delivered across participating countries, permits recognition of qualifications gained abroad, and facilitates the mobility of learners and graduates. Transnational bodies may work on a regional basis (for example, ENQA) or global basis (for example, the International Network of Quality Assurance Agencies in Higher Education (INQAAHE). As well as transnational quality assurance agencies, other international bodies promoting international collaboration on improving quality of online and hybrid education include UNESCO, the European Commission, the OECD, and the International Council for Open and Distance Education.
A key point of reflection for transnational and international organisations in recent years has been how to approach quality assurance of online higher education. For the most part, these bodies have concluded that standards for assuring the quality of online education are best integrated into existing frameworks. In practice, this entails either clarifying that existing standards should be fully applied to higher education provided online, or deciding that specific standards for online higher education should be embedded within the overarching quality assurance framework (Ossiannilsson et al., 2015[17]).
Transnational organisations taking the former approach include UNESCO, whose Global Convention on the Recognition of Qualifications Concerning Higher Education states that qualifications delivered through emerging modes of delivery should be assessed “using the same criteria as those applied to similar qualifications acquired through traditional learning modes.” Similarly, the Asia-Pacific Economic Council (APEC) advocates integrating quality assurance of online education within existing frameworks, which ensures that standards and quality are equivalent for all modes of learning (Staring et al., 2022[4]).
At the same time, some transnational organisations have evolved slightly from fully advocating an integrated approach. For example,
INQAAHE’s base position is that its Guidelines of Good Practice (INQAAHE, 2018[18]) are equally applicable to online and in-person delivery. However, following a recent global review of trends (Karakhanyan and Stensaker, 2020[19]), INQAAHE acknowledged more specific guidance is needed for online education. Its 2022 release of international standards and guidelines for quality assurance includes a specific module on online and blended modalities (INQAAHE, 2022[20]).
ENQA’s European Standards and Guidelines for Quality Assurance 2015 state that the standards and guidelines apply to “all higher education offered in the EHEA regardless of the mode of study or place of delivery” (ENQA, 2015[21]). However, an ENQA Working Group subsequently recommended that external quality assurance “should include consideration of the characteristics of e-learning in regular procedures” (Huertas et al., 2018[22]).
A proliferation of frameworks and toolkits containing standards for quality of higher education digital provision have been developed by transnational organisations and other bodies (Volungevičienė et al., 2021[23]). In the European context, including Croatia, the most important are the European Standards and Guidelines for Quality Assurance 2015, and their proposed extension to specifically cover e-learning (Box 4.2). In other regions, emerging frameworks include UNESCO’s Blended Learning Self-Assessment Tool for the Asia-Pacific region (UNESCO, 2019[24]), and the Virtual Quality Seal developed in part by the Organisation of Ibero-American States for Education, Science and Culture (Organisation of Ibero-American States for Education, 2021[25]).
Box 4.2. The European approach to external quality assurance of digital education (ESG)
The ESG 2015 comprises a set of standards and guidelines grouped under ten key topics (ESG 1.1 to 1.10). These standards and guidelines can be used by external quality assurance agencies operating in the EHEA to guide their development of national standards. Building on ESG 2015, an ENQA Working Group created a set of non-binding considerations for e-learning to integrate with the existing standards relating to each ESG area (Huertas et al., 2018[22]). The Working Group adopted a broad definition of e-learning as, “encompassing every form, including blended learning… and that which is facilitated through the use of ICT.”
Selected indicators related to e-learning recommended for evaluation within the ESG
Source: Adapted from Staring et al (2022[4]), Digital higher education: Emerging quality standards, practices and supports, https://www.oecd.org/education/digital-higher-education-f622f257-en.htm.
Additional frameworks and standards have been developed by other bodies to evaluate and improve online and hybrid learning, including some with a focus on all forms of digital higher education. For example, the International Council for Open and Distance Education – a global association of more than 190 institutional members across 70 countries – developed a Benchmarking Framework for Online, Open, Smart, and Technology Enhanced Higher Education (Hassan, 2022[26]). Other prominent international standards covering digital education include the ACODE benchmarks for technology-enhanced learning (Sankey and Padró, 2016[27]) and the e-Excellence manual of the European Association of Distance Teaching Universities (EADTU, 2016[28]).
National approaches
National quality frameworks for higher education are often heavily informed by transnational frameworks and standards. In Croatia, for example, the national approach to quality assurance is heavily aligned with and informed by ENQA standards and guidelines. However, national quality assurance agencies are also guided by their own regulatory requirements and the extent to which national processes emphasise accountability and/or improvement as the main purposes of quality assurance (OECD, 2018[29]).
Thus, the specifics of national quality assurance frameworks will vary depending on whether the focus is on external evaluations (of programmes, institutions and/or individuals) or on building institutional capacity to improve digital competences and embed digital culture. Recent OECD research (Staring et al., 2022[4]) categorised national approaches to quality assurance of digital higher education provision in the OECD and European Union countries. A review of quality assurance documents across 43 higher education systems showed three distinct approaches:
The system has no specific standards for digital higher education. In 23 of the 43 systems, no specific standards or procedures for the external QA of digital higher education were identified. In these systems, common standards and processes for the accreditation and review of higher education providers and programmes are applied, regardless of delivery mode.
Common standards are in place for online and traditional study modes. In three systems, common standards are in place for all modes of delivery. However, in contrast with the previous category, common standards came about as the result of a conscious decision made by the quality assurance body to extend the application of standards for in-person provision to digital provision.
Specific standards for digital higher education. Specific standards or guidelines for digital higher education were identified in 17 systems, although further variation within this category is evident, depending on whether the standards are optional and the categories of institution to which they apply.
Table 4.2 presents a more detailed breakdown of the systems in each category.
Table 4.2. Approaches for the external quality assurance of digital higher education
Source: Adapted from Staring et al (2022[4]) Digital higher education: Emerging quality standards, practices and supports, https://www.oecd.org/education/digital-higher-education-f622f257-en.htm.
Some national standards for online or blended higher education are provided on an optional basis, although public authorities may offer incentives to institutions to undergo external review using the standards. For example, the Estonian Quality Assurance Agency for Higher and Vocational Education (HAKA) offers instructors the opportunity to apply for an external review of their digital courses and to obtain an “E-Course Quality Label” (HAKA, 2022[30]). Take up of such incentives may vary depending on the attractiveness of the award and/or the administrative burden on the institution. In Spain, while institutions can apply to have a degree programme receive a Quality Label for Distance Learning from the national quality assurance body (ANECA), as of 2022 just 12 degrees in total had received the label (Staring et al., 2022[4]).
In countries where specific quality assurance criteria exist for online programmes, including Croatia, standards are applied only to programmes that meet specific national definitions of an “online” or “blended” programme offer. For example, in Hungary, Japan, Portugal and Spain, specific standards exist only for the ex-ante accreditation of programmes defined as fully online, while in Ireland and Romania specific quality considerations have been established for hybrid education (Table 4.2). Croatia’s specific criteria apply to education programmes where more than 50% of the contained subjects are to be offered online (Box 4.3).
Conversely, some countries target their specific standards or guidelines at entire institutions. In Australia, the Tertiary Education Quality and Standards Agency has developed a list of “risks to quality” for technology-enhanced learning (TEQSA, 2019[31]). The risks are assessed within the legislative basis provided in Higher Education Standards (HES) Framework (Threshold Standards). In Malta, providers wishing to offer digital education have to apply to the Further and Higher Education Authority for necessary approval before undergoing an evaluation by independent digital education experts (Malta Further and Higher Education Authority, 2021[32]).
Croatia’s approach falls between the two models described above. Institutions must have accreditation at the programme level for online learning, but there is also a comprehensive set of standards requiring institutions to have certain infrastructures and supports in place. Croatia’s approach to quality assurance for online learning is currently evolving from a situation with separated definition and evaluation of criteria for online programmes to a more integrated approach located within the ASHE (Box 4.3).
Box 4.3. Quality assurance of online programmes in Croatia’s higher education system
In Croatia, institutions wishing to offer a programme of study in which at least 50% of subjects or 50% of teaching hours are provided online must meet several additional quality criteria, established in 2016. These requirements are checked by the ASHE as part of ex-ante programme accreditation procedures. In addition, students pursuing such programs are automatically classified as part-time students.
The requirements on institutions to meet the criteria for online programmes are comprehensive, and include:
clear articulation of programme purpose, including the modes of delivery, and how it relates to the institution’s strategy, based on robust research;
infrastructural requirements, including implementation of a VLE, interoperability with other technologies and programme offers, embedded use of learning analytics, financial plans for infrastructure procurement and maintenance, and detailed information provided to students about the technologies and associated policies;
defined programme learning outcomes;
a detailed weekly schedule of online activities; a list of quality online learning materials, specifically prepared for the independent use by students; a list of other resources that will be used in the online course; clear and precise instructions for students’ online work; and a clear description of how online assessment will be carried out;
a range of specific technical, administrative, and academic supports to be made available to students studying online.
The extensive criteria were perceived in some cases to be too stringent for institutions to meet. Prior to the pandemic, fewer than 20 online higher education programmes had been established which meet these criteria.
With the entry into force of the revised Law on the Quality of Higher Education in December 2022, ASHE became the competent body for the establishment of criteria for accreditation of online programmes. It plans to set criteria to align with the considerations for quality assurance of e-learning developed by the ENQA Working Group for Quality Assurance and e-learning (Box 4.2).
Sources: Republika Hrvatska (2016[33]), Kriterije i postupke za vrednovanje online studija, https://www.nvzvotr.hr/images/documents/Kriteriji%C2%A0i%20postupci%20za%20vrednovanje%20online%20studija.pdf; Parliament of Croatia (2022[1]), Law on Quality Assurance in Higher Education and Science, https://narodne-novine.nn.hr/clanci/sluzbeni/2022_12_151_2330.html
Public supports for improving digital higher education
Quality standards alone are unlikely to lead to improvements in higher education delivered in online, hybrid or blended mode. Increasingly, quality assurance bodies are playing a more active supporting role – aiming to guide and collaborate with institutions on improving their internal quality assurance systems. Potential drivers of this trend include the need to respond to the largely unregulated mass movement to online education during the COVID‑19 pandemic, and calls from transnational organisations such as INQAAHE and ENQA to move towards institution-based quality assurance and permit wider self-accreditation of programmes (Council of the European Union, 2022[34]). When external quality assurance agencies devolve more trust and responsibility to individual higher education institutions, they also free up agency resources to become a more supportive and collaborative partner for institutions, even if they maintain some external regulatory functions (Staring et al., 2022[4]).
Other public bodies may also support higher education institutions as they seek to improve the quality of their digital education provision. This section develops a taxonomy of five support types that have been introduced or strengthened across OECD and EU countries in recent years with an explicit objective or implicit possibility to assist in the improvement of digital higher education. These supports include:
advice on strategic alignment and development;
collecting, and sharing good practices for digital education provision;
providing funding and financial incentives;
building skills, expertise, and capacity;
taking actions to stimulate collaboration (Figure 4.2).
While institutions are primarily responsible for building digital competence and culture within their organisations, they are more likely to be successful with a strongly supportive policy framework.
Support type 1: Supporting strategic development and alignment
In some countries, external quality standards for digital higher education are accompanied by guidance to higher education institutions to help interpret the formal language of standards and support their implementation. As an example, the United Kingdom Quality Assurance Authority has developed a Taxonomy for Digital Learning, providing definitions and classifications of terms commonly used by education providers in the UK to describe their digital offerings (e.g. blended, hybrid, online, virtual, distance, remote, face-to-face, in-person) (QAA, 2020[35]).
Public authorities may also provide more comprehensive guidance to institutions seeking to adapt their own strategies to align with national and international quality standards. In this regard, governments seek to take into account the autonomy and academic freedom of higher education institutions. They provide information, resources, and capacity-building opportunities that institutions may not have easy access to from other sources. For example, while higher education institutions across the OECD tend to have autonomy to set their own strategies for digital education, many governments and other actors still provide a statement of national policy and/or other guidance to support institutions’ strategic development. This helps them understand and assess their current capacity to develop high-quality digital education and chart a course for its improvement (OECD, 2023 (forthcoming)[36]).
Figure 4.2. Options for public supports to improve the quality of online and hybrid education
Advice for strategic alignment and institution self-evaluation of its digital higher education provision is becoming widespread across OECD countries. In New Zealand, Ako Aotearoa first established strategic advice for institutions on e-learning in 2014 (Coolbear, 2014[37]). Norway’s Standing Quality Committee of Flexible Education Norway (FuN) also has long-established guidelines to support institutions with the development of strategy for digital higher education quality (Flexible Education Norway, 2018[38]).
Transnational and supranational organisations also provide a wide range of advice to support institutions in strategy development with respect to digital education. Prominent examples in Europe are the European Commission’s frameworks of best practice statements to support reflection and improvement of digital practices in education institutions. These include the European Framework for the Digital Competence of Organisations (DigCompOrg) and the European Framework for the Digital Competence of Educators (DigCompEdu). These frameworks cover a range of thematic areas including leadership and governance practices, collaboration, and networking (Redecker and Punie, 2020[39]).
Institutions can also increasingly access more precise guidance and advice on specific topics related to, for example, the improvement of teaching and learning, and the provision of open educational resources. A recent report by the European University Association found that institutional advice for improving digital teaching and learning is offered not only by quality assurance agencies but also by a range of different organisations operating at a national or regional level. These include National Research and Education Networks, foundations, sectoral associations, or networks of institution leaders. A wealth of self-assessment instruments are also now available to higher education institutions to help guide their self-evaluation and strategic development with respect to digitalisation (Volungevičienė et al., 2021[23]).
The provision of advice and guidelines to institutions can support their improvement. However, as the number of toolkits and guidelines multiply, institutions may become overloaded with efforts to integrate and assimilate them. This is a particular risk where multiple guidelines and sets of advice available from various national and international actors do not align or share a common vision, or where they duplicate each other. Depending on the context, institutions may require external support to curate and interpret knowledge and guidance from many diverse sources and adapt it to their local context.
One promising development in a small number of countries (including Ireland, Germany, Norway and the United Kingdom) has been the establishment of specific national centres whose role is to support higher education institutions as they work to improve their education function (Ehlers and Zhang, 2022[40]). Such centres (for example, the Irish National Forum for the Enhancement of Teaching and Learning, Box 4.4) can play a pivotal role, helping institutions to integrate and reflect on guidance and advice provided by various sources related to digital education.
Box 4.4. National Forum for the Enhancement of Teaching and Learning, Ireland
Established in 2012, Ireland’s National Forum for the Enhancement of Teaching and Learning has responsibility for leading and advising on the enhancement of teaching and learning in Irish higher education. Funded by the Irish Higher Education Authority (HEA), the National Forum co‑ordinates many different initiatives, all focusing on the goal of enhancing teaching and learning. From its inception, building digital capacity was one of five principal areas of action. Following an external review and a sector-wide consultation process, in 2018 the National Forum Strategy 2019-21 was launched, with four key strategic priorities, one of which is “Teaching and Learning in a Digital World”. This priority has the objective of “supporting those who learn, teach, and support learning to embrace and harness the potential of digital technologies”.
The National Forum for the Enhancement of Teaching and Learning has published a range of strategic advice for higher education institutions, developed through research, collective discussion, and co-creation among higher education institutions in Ireland. For example, a Guide to Developing Enabling Policies for Digital and Open Teaching and Learning was published in 2021.
Also in 2021, a project on “'Next Steps for Teaching and Learning, moving forward together” was established, to bring together a range of stakeholders from across the higher education sector (including Quality Qualifications Ireland (the Irish quality assurance agency), institution representatives, students and other stakeholders) to answer one shared question: 'In the context of Covid, what have we learnt and what does it mean for the future of teaching and learning in higher education. The project represented a new departure in the higher education sector in Ireland in terms of the scale of and the approach to national collaboration. The recommendations of the report attracted targeted public funding to support their implementation through a specific funding instrument - the Strategic Alignment of Teaching and Learning Enhancement Funding in Higher Education in Ireland. The funding is targeted specifically to teaching and learning and is non-competitive (with funding allocated on a pro rata basis based on student numbers), thereby encouraging cross-institution collaboration.
Source: Adapted from National Forum (n.d.[41]), National Forum for the Enhancement of Teaching and Learning in Higher Education: Resource Hub, https://www.teachingandlearning.ie/resourcehub/.
Support type 2: Researching and sharing good digital practices
The COVID‑19 pandemic led to an acceleration of digitalisation in teaching and learning and in some cases forced educators to adopt digital practices for the first time, many of whom characterised the rapid shift to online education necessitated by the pandemic as challenging or very challenging (OECD, 2021[42]). Many who integrated digital tools into their teaching did so with little awareness of how technology can support teaching and, in a crisis, made limited use of evidence from the research literature on good practices (Martin et al., 2020[43]). There is now an imperative across countries to increase digital competence of staff in higher education institutions, including building the pedagogical knowledge of teachers to produce high-quality content intended for online delivery. Governments and other national bodies can play a role in researching and sharing good organisational and pedagogical practices that drive improvement of digital education provision, helping to inform and orient educators and institution leaders.
Strategies for collecting information on promising practices vary across countries. Some public bodies collect information directly from institutions to gain an understanding of practices than can be shared more widely. For example, in Hungary, the Ministry for Innovation and Technology commissioned two surveys on digital higher education in 2020, one of which sought information on digital practices and institutional leaders’ views on factors determining the extent of digitalisation in their organisations, including external factors (e.g. students’ digital skills) and internal factors (e.g. access to digital infrastructure, teachers’ digital skills) (OECD, 2021[9]).
Quality assurance agencies in many countries also actively collect and disseminate national and international good practices and resources for the quality improvement of digital provision. In Australia, for instance, TEQSA published a series of key considerations for providers of online delivery in 2020 which was subsequently broadened into an Online Learning Good Practice website (TEQSA, 2022[44]). In the UK, QAA offers training sessions and instructional videos to institutions on best practices in digital education, and published “Hallmarks of Success” covering student-centred teaching; assessment in digital and blended pedagogy; programme design and management; and empowering teaching staff (QAA, 2022[45])
Some national quality assurance bodies have also carried out specific thematic surveys to elicit information on best practices among the institutions for which they bear responsibility for supporting. While such surveys became far more common during and following the pandemic (as in Croatia), in some systems, thematic reviews of digital learning were carried out years beforehand. In 2017, for instance, UKÄ in Sweden conducted a survey among Swedish institutions on “strategies and range of distance learning courses and degree programmes, and the support they offer distance students” (UKÄ, 2018[46]).
Some governments and public bodies also conduct or fund regular research reviews on organisational practices associated with improved quality in higher education and curate their results to share with institutions. For example, the Digital Higher Education Consortium of Texas commissioned a comprehensive meta-analysis of available data and research relevant to online and blended higher education, to synthesize messages about good practices from the existing evidence and identify important gaps in the research (Box 4.5).
Box 4.5. Meta-analysis to surface good practices in digital higher education in Texas
In 2020, the Digital Higher Education Consortium of Texas carried out a meta-analysis of existing data and research on digital (including distance and online) higher education in Texas to identify gaps in the evidence base. It also worked on strategies to improve the body of knowledge in areas with less evidence of good practice available. The review found that there is a pressing need for more quantitative, empirical, and experimental research about digital education. At the same time, digital education produces substantial amounts of data about teaching and learning previously difficult to capture at scale in face-to-face courses, which, if harnessed effectively, can support a better understanding of the impacts of digital education.
Some of the studies reviewed in the meta-analysis highlighted positive student outcomes, notably in institutions with an extensive experience and history in using online education to support successful outcomes for diverse learner populations. At the same time, it was noted that few of the reviewed studies were able to disaggregate data for minorities and disadvantaged populations, limiting ability to focus on equity issues.
The review also highlighted gaps in research on impact and outcomes, including student retention, progress, completion, and success, related to digital education. Comparative studies of online and face-to-face courses and programs, even though they are complex in design, are needed. Some studies highlighted in the meta-analysis offer a template for experimental and quasi-experimental designs.
Source: Sebesta et al, (2020[47]). Digital Higher Education in Texas: A Meta-Analysis of Data and Research, https://tacc.org/sites/default/files/documents/2020-06/digitex_meta-analysis.pdf.
A small number of countries have developed specific portals that can be used to widely disseminate the results of evidence reviews of different practices and approaches across education organisations, including reviews of the impact of digital technologies. Such initiatives include the Teaching and Learning Toolkit of the United Kingdom’s Educational Endowment Foundation’s “What Works” centre (EEF, n.d.[48]), and the Danish Clearinghouse for Educational Research (DPU, 2022[49]).
Support type 3: Providing funding and financial incentives
Many higher education institutions, including in Croatia, struggle to secure financial and human resources to develop their capacity for high-quality digital education. This is particularly true for smaller institutions, with few staff available to lead on building digital capability across the institutions and often not representing a lucrative customer or partner for educational technology companies. Public authorities have a role to play in supporting institutions to mobilise resources to improve digital education, either through direct provision of funding or through indirect financial incentives. In many countries, governments also are a key funder or provider of the vital digital infrastructure needed to support effective digital education.
Few public authorities systematically award earmarked funding for digitalisation to institutions, although some countries, such as Germany, are considering it (Box 4.6). Where funding has been provided for digital innovation, it tends to be targeted for specific purposes and time limited (OECD, 2023 (forthcoming)[36]). One of the most common ways that governments financially support institutions to develop online provision is through the financing of digital infrastructure, either directly or funnelled through NRENS. In many systems, including in Croatia, NRENs play a vital role in supporting institutions with procurement, provision, or joint purchase of a wide range of technologies and services and building capacity among user groups, as well as their traditional role of providing Internet connectivity (see Chapter 5).
OECD research shows that one of the main sources of revenue for higher education institutions in many systems – government-funded tuition fees – is often not applicable to students studying online. The 2020 edition of the OECD Higher Education Policy Survey found that OECD countries typically limit eligibility to public student financial aid programmes for students enrolled in online and hybrid programmes. For example, while all 28 responding systems indicated that full-time students in recognised on-campus bachelor’s programmes were eligible to apply for student financial aid, this was the case for online bachelor’s degrees in only 15 jurisdictions. Additionally, just 9 of the 28 systems provide financial aid to students taking online short-cycle programmes and only 5 provide support for students taking non-degree short online programmes (Golden, Troy and Weko, 2021[50]).
Nevertheless, there are indications that governments in some countries may be starting to adapt funding mechanisms and financial incentives to support the provision of high-quality digital education. An increasing number of countries in Europe operate institutional performance agreements as a steering and accountability tool for publicly funded higher education institutions, some of which have targets related to the quality of their digital teaching and learning offer. In the Netherlands, for example, the government and publicly funded higher education sector have agreed on six priority themes for their 2019-24 performance agreements, and some institutions have formulated digitalisation targets under the themes of “education infrastructure” and “professional development of educators” (Rijksoverheid, 2018[51]). In Austria institutional performance agreements for 2022 to 2024 include digitalisation as a key priority, aiming to achieve a significant expansion and development of digital education (BMBWF, 2022[52]).
Box 4.6. Funding digital innovation in higher education in Germany
Germany is an example of a country working to establish a recurrent public funding line for digitalisation in higher education, in a context where it has been found to be lagging on many general measures of digitalisation. A 2019 report by the National Commission of Experts for Research and Innovation (EFI) argued the “digitalisation of Germany’s structurally under-financed tertiary education system is an ongoing task which requires long-term financing” and proposed a per-capita public funding allocation specifically for digitalisation. This funding could be used to invest in digital infrastructure and build capacity for the creation of new and improved digital education offers (EFI, 2019[53]). The German Rector’s Conference adopted the proposal in 2021 (HRK, 2021[54]) but no final agreement has yet been reached.
Germany also has several targeted funds to support improvements in digital higher education. For example, the Foundation for Innovation in Higher Education Teaching receives EUR 150 million annually from the federal and state governments to award to higher education institutions in competitive calls for learning innovation projects, including projects aimed at improving teaching through the use of digitalisation (Stiftung Hochschullehre, 2022[55]).
Source: OECD (2022[56]), OECD Reviews of Innovation Policy: Germany 2022, https://doi.org/10.1787/50b32331-en.
Support type 4: Building skills, expertise, and capacity
Several concepts have emerged to describe factors supporting effective assimilation of digital technologies by education institutions, including e-capacity (Vanderlinde and van Braak, 2010[57]), and digital capacity (Costa, Castaño-Muñoz and Kampylis, 2021[58]), In Croatia, the concept currently used widely in both schools and higher education institutions is that of digital maturity (Balaban, Redjep and Čalopa, 2018[59]). Regardless of the specific language used, these concepts emphasise broadly similar dimensions in institutions’ capacity for digital education. There is a strong focus on technical and pedagogical knowledge of educators, their access to professional development; institutional leadership capacity and a conducive organisation structure and culture (Castaño Muñoz, Pokropek and Weikert García, 2022[60]).
Across OECD and EU countries, governments and other actors have established initiatives aimed at building skills, capacity, and expertise in digital education. Governments can support staff development in several different ways, through:
direct provision of high-quality training and other opportunities for building skills;
support for communities of practice;
adaptation of human resources policies; and
recognition and reward of teaching excellence.
Efforts to incentivise staff to improve digital course materials form part of wider efforts to attach a higher value to teaching activity within academic careers, in a context where research has been widely perceived to be (from a career perspective) the most rewarding activity (Janger, Campbell and Strauss, 2019[61]).
National networks have emerged to build communities of practice around digital learning. These include SURF in the Netherlands, Flexible Learning Norway, and the Online Learning Consortium in the United States. While these networks are managed by a range of different organisations, they often receive recurrent or project-based funding from governments. For example, SURF received funding from the Dutch Ministry for Education, Research and Culture between 2019 and 2022 to support 40 higher education institutions, collaborating on the development of resources, practices, and guidelines to advance the quality of digital higher education (SURF, 2022[62]). International networks and capacity building projects also exist at European level, such as the European Distance and E-Learning Network (EDEN, n.d.[63]), and the European Digital Learning Network (DLEARN, n.d.[64]).
Another area where capacity-building efforts are strengthening is the use of learning analytics to monitor and improve the education experience of learners. For example, the UK NREN Jisc has created a Senior Managers’ Guide to Learning Analytics (Jisc, 2020[65]) with six key factors to target for the effective implementation of learning analytics. The Japanese government has also embarked on an ambitious project to design and implement learning analytics across its education system. Its first stage will establish an infrastructure and align software across the system so it is ready for learning analytics, while the second stage will collect learner log data that can feed into monitoring and evaluation mechanisms. A third stage envisages personalisation: using the data collected to optimise learning to the individual and to evaluate academic progress. (Digital Agency et al., 2022[66]).
Developing capacity and building skills for digital course design and development takes time. Yet most academics do not have time for these activities available in their existing workloads. While most national governments are not directly involved in prescribing workload models for higher education institution staff, they could consider creating incentives for investment of instructional time in the effective use of digital technologies. This could include providing financial assistance with training fees and/or time off to focus on digital education improvement. It could also include augmenting administrative supports to free up capacity for leadership in instructional design and delivery (OECD, 2023 (forthcoming)[36]). As yet, there are few examples available of governments attempting to incentivise the creation of such supportive human resources policies.
Capacity can also be built by reconsidering the range of staff roles that can be involved in the development of online and blended learning. Emerging professional roles with job titles including “digital learning technologist,” “educational technologist” and “instructional designer” are well-established in higher education systems in the United States and are becoming more prominent in Europe. Learning technologists can support and advise academic staff on the effective development of online and blended learning material. They are expected to have good pedagogic knowledge as well as technical knowledge.
Support type 5: Stimulating collaboration and resource sharing
Encouraging collaboration and resource sharing can be an effective way to build capacity among academic staff and institution leaders with responsibilities for developing high-quality digital education (OECD, 2017[67]). Beneficial and impactful collaboration can take place both within and between institutions. This section examines collaboration instigated or supported by government authorities while the next section will explore institution-led efforts to support collaboration. Given the elevated levels of autonomy and academic freedom enjoyed by higher education institutions, public authorities tend to focus on encouraging higher education instructors, leaders, and IT support staff to engage in peer learning and collaboration on digital teaching and learning through funding incentives.
One of the more innovative recent public funding schemes aimed at supporting peer learning among higher education institutions is the French “digital demonstrator programme” (Box 4.7).
Box 4.7. The French “digital demonstrator programme”
In 2021, the French government awarded EUR 110 million to 17 “digital demonstrator projects” (Démonstrateurs numériques dans l’enseignement supérieur – DemoES) to fund strategy development, infrastructure, and pedagogical innovation in public higher education institutions across France. The intention is for these demonstrators to enable the state and other organisations to draw inspiration from emerging good practices and embed them in the digital transformation of the entire higher education system.
Each of the 17 projects was chosen to provide next generation showcases of technology integrated into pedagogy in digital higher education. For example, one demonstrator project chosen, a collaboration between Arts et Métiers, CEA Tech, the National Conservatory of Arts and Crafts (CNAM) and the Center for Higher Industrial Studies (CESI) creates “digital twins” – mechanisms to allow students to do practical training on equipment using remote access. The project also focuses on the creation of immersive virtual and augmented reality experiences for students to learn in environments that realistically replicate professional or industrial contexts.
Technologies developed will serve as a showcase example for the rest of the higher education system and support more efficient transfers of the technologies to other higher education institutions.
Source: ANR (2021[68]), Démonstrateurs numériques dans l'enseignement supérieur, https://anr.fr/fileadmin/aap/2021/ia-ami-demoes-2021-jo.pdf
As mentioned previously, collaboration can also be effectively stimulated by public support and funding for the establishment of permanent national organisations with a remit to support teaching and learning in higher education, such as the Irish National Forum for Teaching and Learning in Higher Education (Box 4.4). Few organisations of this type exist, but existing models exhibit promising signs of improving and systematising collaboration across institutions.
The European Commission is also working to address perceived fragmentation in the institutional, national, and international landscape of collaboration and exchange related to digital learning, through the launch of a Digital Education Hub. The Hub aims to establish a European community of practice, engaging a wide variety of stakeholders and supporting cross-sector collaboration on digital education in Europe (European Commission, 2022[69]). Finally, support is coalescing around the greater use of Open Education Resources (OER), driven in part by recent UNESCO Recommendation on OER, adopted in 2019 (UNESCO, 2020[70]). Governments can also support resource-sharing between institutions by funding or otherwise supporting the creation of national platforms for sharing of e-learning material.
The next section moves from the level of transnational organisations and public authorities to the level of institutions, discussing some of the practices that institutions can adopt to improve their digital competence and culture. This in turn can improve the effectiveness of digitally enhanced teaching and learning.
Practices to support high-quality digital education
Higher education institutions are likely to benefit most from integrated strategy and joined-up implementation
Assuring the quality of digital teaching and learning is, first and foremost, the responsibility of higher education institutions, given their organisational autonomy and academic freedom. As with external quality assurance standards, some evidence indicates institutions may be best served by integrating specific considerations for digital education in their existing internal quality mechanisms (Jung, 2022[71]). Institutions have been active in this regard: a recent EUA survey of 368 institutions across Europe found that 51% were already integrating digitally enhanced teaching and learning in their internal QA strategies, while integration was under development in a further 41% (Gaebel et al., 2021[72]).
While many institutions have developed internal digitalisation strategies, few have developed written guidance or training for institution staff to support the implementation of internal strategies. For example, a review of institution practices in Commonwealth countries (Latchem, 2016[73]) found that only 36% of institutions surveyed had access to internal quality assurance manuals when working on quality assurance or improvement activities. Recent reviews of practice indicate that implementation guidance through manuals and similar documents is becoming more widespread (OECD, 2023 (forthcoming)[36]).
However, written internal guidance represents only one narrow avenue for progressing improvements. Institutions should take a whole-of-institution approach to developing high-quality digital education, encompassing all modes of delivery. A whole-of-institution approach entails building a shared and aligned vision, strong leadership on both digitalisation and teaching and learning issues, improving infrastructure, equipment, and skills of staff, decision makers and students.
There are three important principles higher education institutions can strive towards embedding throughout their organisation to maximise their capacity to implement high-quality digital education:
building a shared vision and strong culture for the improvement of digital teaching and learning;
emphasising innovative and engaging digital course content, design, delivery, and assessment;
using an evidence-informed framework to guide institutional improvement processes.
The approaches presented below are informed by a comprehensive recent OECD literature review on enabling factors for high-quality digital education (OECD, 2023 (forthcoming)[36]) and external expert contributions to the project.
Principles for institutions to achieve high-quality digital education
A shared vision and strong culture
Adopting a coherent approach towards improving digitally enhanced teaching and learning enhances the chances of an institution improving its overall education offer. Evidence shows the quality of co-ordination and relationships within a higher education institution can positively influence its performance (De Carmen et al., 2013[74]). Co‑ordination and alignment of strategy for online and blended education is likely to lead to better results for the institution as a whole compared to a top-down policy or fragmented “pockets of excellence” (Tømte et al., 2019[75]).
Some higher education systems are beginning to shift their position on digital education from control-based governance to culture creation (Jung, 2022[71]). Recent EUA research stresses the need for educational leaders to build both positive shared beliefs and an enabling climate and atmosphere when designing and implementing major learning innovations, including digital innovations. (EUA, 2022[76]). Digital tools can themselves stimulate and streamline co-ordination, creating superior relationship management and more effective communications (Mendoza and Heredero, 2016[77]). Finally, co‑ordination between institutions on strategy development can potentially lead to more efficient use of individual institution resources and promote widespread peer learning and practice sharing (Lerstad, 2019[78]).
A fundamental first step in supporting high-quality online and blended learning within a higher education institution is the development of a shared, institution-wide vision for improving the quality of digital higher education. This should be co-created with stakeholders from across the institution. Co-creation within the institution supports internal alignment of objectives, while external alignment of the vision with wider national and international policy objectives can be helpful when mobilising resources for implementing reforms.
Focus on innovative and engaging digital course content, design, delivery and assessment
There is a growing expectation, enshrined in many digital education quality standards and frameworks, that courses designed for online or blended delivery should reflect up-to-date theory and practice in the subject area, as well as best pedagogical practices for engaging students in a digital environment.
Ensuring the active participation of, and interactivity with, students has long been reported as one of the main challenges of teaching online (Garrison, Anderson and Archer, 2010[79]). This challenge was underscored by the negative reactions of many learners during the pandemic period to emergency remote instruction, which often attempted to replicate the face-to-face experience in terms of both duration of interaction with instructors and presentation of learning materials. When talking to the OECD review visit team, instructors in Croatia also reported difficulties engaging students in an online environment. These difficulties were often attributed to lack of time and competences needed to prepare a high-quality online learning experience (see Chapter 3).
There is an emerging consensus that educational materials intended for online consumption should be designed to stimulate interactivity, and be usable, accessible, and engaging (OECD, 2023 (forthcoming)[36]). Conversely, it has long been recognised that digitalisation can offer solutions to challenges encountered when teaching in-person, and that some digital tools used in online delivery can provide greater engagement between students and their instructors (Oblinger, 2004[80]). Therefore, development of high-quality digital teaching and learning can entail both designing content for online delivery and harnessing digital technologies to augment in-person education.
Simunich et. al. (2022[81]), provides an overview of general good practice principles for improving the quality of institutional course design as follows (Staring et al., 2022[4]):
Choosing the right people – senior but not remote – so they can lead but also act as a bridge between different groups.
It is essential to achieve the buy-in of multiple stakeholders to the proposed action.
New processes must ease, rather than increase, administrative burden, and workload.
Promote widespread conversation and exploration of the envisaged goals and be willing to accept incremental change rather than expecting major impact overnight.
Individual institutions are pursuing innovations in the use of digital tools for course design and delivery across several fronts. Examples of emerging practices include establishing partnerships with online learning platforms and course or programme managers. The number of such partnerships has been increasing rapidly in both Europe and the United States (HolonIQ, 2021[82]). Institutions pursuing course development also have increasing recourse to evidence-based models and frameworks for digital course design. For example, Universal Design for Learning (UDL) is emerging as an important pedagogical principle that should underpin the development of any course, regardless of delivery mode, to support greater inclusion and equity in student outcomes (Ehlers and Zhang, 2022[40]).
A major challenge reported by institutions and instructors is how to ensure authentic and trusted online assessment, and in particular mitigating quality risks when students take tests online (Manoharan and Ye, 2020[83]). While much of the policy and practice responses to date have focused on enforcement measures such as student authentication and proctoring (Lee-Post and Hapke, 2017[84]), there are persistent concerns about the intrusiveness of some measures, and their overall effectiveness in ensuring quality and integrity of assessments. Partly as a reaction to these concerns, there is an emerging focus on prevention, including the redesign of examinations – such as structuring examinations as open-book/open-web tasks or making greater use of oral examination – or shifting towards more authentic, problem-based assessment. Many institutions have ethics and academic misconduct policies to make students aware of and accountable to their responsibilities in terms of academic integrity (OECD, 2020[85]).
Harness the power of collaboration to progress more efficiently and increase influence on public policy
The previous section outlined some ways public authorities can support and incentivise institutions to collaborate on teaching and learning improvement. Institutions can also build capacity for collaboration from the bottom up. While collaboration and connection between diverse individuals and organisation is considered beneficial for innovation, the smaller scale of the higher education system in Croatia and the country’s ongoing demographic changes make it even more prudent for institutions to co-operate on digital education issues. Inter-institutional collaboration creates opportunities for the limited funding that is available to be put to the best use possible, through sharing the time and cost involved in building capacity for digital education across institutions.
Furthermore, when institutions and other stakeholders do collaborate effectively, they provide a more influential force to inform national policy development than would be possible with each institution acting alone. Small scale collaborative initiatives can provide powerful use cases to serve as a basis for scaling up. This can happen with the aid of public support or by incentivising more institutions to get involved and contribute to the initiative. A recent example of bottom-up collaboration in Portugal provides a useful example of how collaboration can start small and scale up in a relatively brief period of time (Box 4.8).
Box 4.8. Scaling up teaching and learning innovation in Portugal
In Portugal, the Universities of Minho and Aveiro have long co-operated on a small scale for specific topics. In 2019 the two universities entered a strategic bilateral partnership with the goal of transforming teaching and learning in their respective institutions. A retreat on the topic organised in 2019 was so successful that three additional retreats were subsequently organised, with staff participation increasing each time, stimulating the formation of networks and a practice community across the two institutions.
The success of the bilateral initiative led to enquiries from staff in other institutions about participating. The two universities created an online inter-institutional conference on pedagogical development (Jornadas Interinstitucionais de Desenvolvimento Pedagógico), as a means of expanding innovative practice in teaching development to other institutions. The movement has grown since the first edition was held in 2020, and, consistently, in between sessions, more institutions express interest in being involved. While 9 universities engaged in the first session held in September 2020, by the time of the sixth session just 18 months later 17 higher education institutions contributed, including public and private universities and polytechnic institutes.
Source: Adapted from the more detailed summary of this case study contained in Annex A.
A framework for institutional-level transformation of digital education
The institution-wide vision and culture promoted above remains rare among higher education institutions. Quality improvement initiatives often only attract a limited number of already highly motivated instructors, while engaging students and institutional leadership in such activities remains challenging for many institutions (Tømte et al., 2019[86]). Furthermore, developing design principles for online education during the initial creation of a course requires a lot of planning and preparation by teaching staff, and these efforts often go unappreciated (Barquero, 2022[87]). Similar points were made multiple times to the OECD review team by higher education academic staff during the OECD project team visit to Croatia in 2022.
The improvement efforts of institutions may be more impactful if they are guided by an evidence-informed framework which sets out steps and important principles for transforming online and blended learning. Box 4.9 proposes such a framework, summarising the key points of the more detailed institution guidelines developed as part of this project (see Annex A).
Box 4.9. A framework for institutional-level transformation of digital education
A framework for institution-level transformation of online and blended learning should support a co-creative process that builds a collective vision and allows for gradual, iterative progress. An 8-step cyclical approach is proposed for Croatian institutions to develop an inclusive, consultative strategy:
Step 1: Find and understand the drivers for strategy development.
Step 2: Identify resources for strategy development and implementation.
Step 3: Ensure a whole-of-institution strategic approach.
Step 4: Assess the starting point.
Step 5: Build consensus on high-level priorities and success indicators.
Step 6: Translate strategic priorities into transformative actions.
Step 7: Launch, monitor and review.
Step 8: Capture, celebrate and build on success.
Discussions should address what works well in quality digital education and what needs improvement. Each step incorporates questions and considerations intended to stimulate conversation and consensus-building while formulating a whole-of-institution digital education strategy and associated implementation plan. Specific topics to consider and discuss include academic integrity, active learning, staff and student digital capabilities, infrastructure, attitudes towards technology, learning spaces, agile curriculum design, good practice sharing, and designing inclusive learning experiences. The development level and strategic priority for each topic will vary across institutions.
Source: Adapted from the set of institution guidelines presented in Annex A.
How are Croatian institutions prepared for digital transformation? Key findings from a workshop with institutions
On January 27, 2023, the OECD project team held a workshop with representatives from Croatian higher education institutions and stakeholders, to test the draft guidelines for institutions prepared during the project activities. These draft guidelines provide advice to institutions on developing their capacity for strategic planning and collaboration to improve digital teaching and learning, and advice on investing in digital infrastructure.
In total 20 representatives from educational institutions as well as the Ministry for Science and Education, CARNET and the Agency for Science and Higher Education participated in the workshop. The selection of participants considered institutional sector, legal status, and levels of current digital maturity (as assessed by the OECD-CARNET survey of digital maturity in Croatian higher education institutions). Participants, with the support of the workshop facilitators, considered current challenges and potential strategies for moving forward with improving institutional and system-wide digital maturity in terms of leadership, infrastructure, competence, and culture.
Key findings of the workshop included:
Many participants recognised the potential benefits of digitalisation for engaging both local and international students, and even to contribute to the long-term survival of institutions – a concern for some participants in the Croatian context.
At the same time, participants repeatedly raised the urgent need to focus on people – developing not only their capacity but also their willingness to engage in strategic development processes. The most frequently cited barrier to widespread staff engagement in contributing towards quality digital education was lack of time, followed by lack of motivation (due to a sense that the time and effort spent would not be valued or rewarded).
The workshop hosts and facilitators concluded that there is limited history in Croatia of the institution-wide strategic development and inter-institutional collaboration being promoted during the workshop, and that there is a substantial “implementation gap” between the current position of Croatian institutions and the point where whole-of-institution approaches to collaboration can be developed. On the other hand, Croatian institutions are often highly engaged in European collaborative initiatives, including in digital education committees. This demonstrates that collaboration could be stimulated if the right incentives are in place.
While each of the participants was interested and engaged in the discussions, there was a sense of lack of empowerment for driving wider change from their individual positions. There was concern that some teachers in Croatia will be hesitant to share practices and collaborate. In many cases there is no support for teachers to share or develop digital education resources.
The group concluded that change can only be achieved incrementally and that creating a new culture will take time. In the meantime, even small successes should be highlighted and celebrated.
There was support among participants for establishing a system-wide group or forum that can meet regularly to share practices and build momentum for improvement. System-level supports are needed to solidify inter-institutional connections, including practical guidance and advice for the group establishment and operation, targeted funding from government, and agreements in principle for collaboration between institution managers.
Recommendations for Croatian authorities
The analysis contained in this chapter, and the discussions at the related stakeholder seminar held in January 2023 (see Annex C), result in the following recommendations for Croatia to consider in future strategic deliberations related to digitally enhanced teaching and learning.
Continue to strengthen quality enhancement as well as quality assurance
Quality assurance agencies are aiming to balance two competing priorities when considering online and hybrid education. On one hand, there is a need to ensure standards and guidelines for online education are coherent with existing quality assurance frameworks. Integrated standards minimise administrative and operational burden and reinforce the principle that quality standards (and, consequently, learner outcomes) should be the same regardless of how an education programme is designed and delivered. On the other hand, a large body of evidence indicates that online education may in some cases lead to poorer outcomes for students and may not always be cost-effective or efficient for higher education providers. Quality assurance agencies, therefore, remain invested in ensuring that students are not disadvantaged by studying online.
Many of the international quality assurance frameworks developed in recent years have not yet been fully embedded in the standards and practices of quality assurance agencies and higher education institutions (Volungevičienė et al., 2021[23]). Croatia’s quality assurance system is designed to be aligned with best practices in the European Higher Education Area. Croatia also intends to align its criteria for the accreditation of online programmes with proposed European best practice. However, even where standards have nominally been adopted, coherent implementation may not always follow. This effectively reduces the quality assurance process to a “box ticking” exercise with limited real impact. As a result, transnational organisations are beginning to put a greater emphasis on improving implementation of the guidelines within member organisations and higher education institutions (Blackstock, 2022[88]).
National quality assurance agencies are also increasingly positioning themselves as partners to, and supporters of, quality enhancement efforts by institutions, rather than only concentrating on their regulatory functions. In Croatia, ASHE conducts a range of quality enhancement related activities in addition to its role in setting and evaluating standards and accreditation of institutions. For example, ASHE carried out extensive research into the experience of institutions and their staff during the COVID‑19 pandemic to help inform future policies and practice. It conducts regular training for institutions to build their capacity for self-evaluation and organises webinars to share good practices. ASHE also collects data from higher education institutions and publishes a range of institution-level statistics and analytics.
Going forward, and aligning with best international practice, ASHE could continue to strengthen its role as a supportive partner for higher education institutions in quality enhancement processes, in tandem with its central position within the regulatory framework. For example, in the Croatian context, where there is not yet a strong endemic culture of institution collaboration at the national level, ASHE is uniquely positioned to gather information on good internal quality practices relating to digital education and find ways to communicate them for wider benefit.
Develop and share a long-term vision for supporting digitalisation in the Croatian higher education system.
Croatian public authorities have prioritised the development of digital infrastructure and competence in their current strategic investment plans for higher education, notably through the major investment foreseen in the e-Universities project. The e-Universities project is expected to run until 2025 and should lead to an improvement in the quality of digital infrastructure and digital competence across Croatian higher education institutions. However, institutions currently lack a sense of what comes next in terms of the long-term system level objectives for digitalisation in higher education and the public supports being considered to achieve it.
Institutions themselves are responsible for setting strategy for digitalisation, including investment strategy. At the same time, strategic development in higher education institutions is at least informed, if not driven, by the national and international policy agenda. In this regard, a statement of long-term goals and intentions with respect to integrating digital technologies in higher education system can act as a powerful driver for institutions to develop their own objectives for digitalisation aligned to national goals.
Create space for institutions to collaborate sustainably on teaching and learning improvement
Higher education institution staff in Croatia gave a clear and consistent message to the OECD team during interviews and workshops: they do not have the time or space in their current workload to innovate or collaborate on issues related to teaching and learning, including digital innovation. Institution leaders have the leading role in making that space, but governments can also support and accelerate the process by providing resources and incentives for institutions to collaborate.
Furthermore, as is also the case in many other countries, there is a belief among Croatian academic staff that research activity and production is valued more highly than efforts made to develop teaching practice. Research performance is considered the key to career advancement rather than excellence in teaching and learning. Without efforts to change that perception, motivation for participating in innovative processes is likely to remain low, and it will be difficult to build momentum in transforming teaching practices.
Public resources devoted to collaboration can be allocated within the context of specific programmes and projects, as is the case with the current e-Universities project in Croatia. However, as discussed earlier in this chapter and in Annex A, experience from Ireland shows that the establishment of a national-level forum can drive substantial change in institutional culture over a relatively brief period of time and at a reasonably low cost.
Croatian higher education institutions appear to have limited experience with collaborating on a large scale on teaching and learning topics. The establishment of a national-level funded space for teaching and learning development in higher education can create a focal point for institutions to gather and discuss challenges and experiences on teaching and learning topics of common interest, including the development of digital education. Croatia should consider establishing such a body not only to stimulate collaboration and innovation, but as a public statement of the value and worth of teaching and learning in the higher education system. The Irish model (the National Forum for the Enhancement of Teaching and Learning in Higher Education) offers a relevant case study and blueprint for how this can be achieved in a country similar to Croatia in size and, until recently, institutional structure.
References
[2] Agency for Science and Higher Education (n.d.), Scope of Work, https://www.azvo.hr/en/about-ashe/scope-of-work/activities.
[68] ANR (2021), Démonstrateurs numériques dans l’enseignement supérieur (DemoES), https://anr.fr/fr/detail/call/demonstrateurs-numeriques-dans-lenseignement-superieur-demoes-appel-a-manifestations-dinteret/.
[59] Balaban, I., N. Redjep and M. Čalopa (2018), “The analysis of digital maturity of schools in Croatia”, International Journal of Emerging Technologies in Learning (iJET), Vol. 13/06, pp. 4-15, https://doi.org/10.3991/IJET.V13I06.7844.
[87] Barquero, J. (2022), The importance of instructional design for e-learning education, CAE Innovative Learning Solutions, Miami, https://www.cae.net/why-is-instructional-design-important-for-e-learning-education/ (accessed on 22 June 2022).
[11] Bettinger, E. et al. (2017), “Virtual classrooms: How online college courses affect student success”, American Economic Review, Vol. 107/9, pp. 2855-2875, https://doi.org/10.1257/AER.20151193.
[88] Blackstock, D. (2022), Presentation to the OECD Group of National Experts in Higher Education, September 2022 meeting.
[52] BMBWF (2022), Rekordbudget an den Universitäten: Leistungsvereinbarungen 2022-24 [Budget for university performance agreements 2022-24], Bundesministerium für Bildung, Wissenschaft und Forschung, Vienna, https://www.bmbwf.gv.at/Ministerium/Presse/20220104.html (accessed on 29 June 2022).
[10] Bowen, W. et al. (2014), “Interactive learning online at public universities: Evidence from a six-campus randomized trial”, Journal of Policy Analysis and Management, Vol. 33/1, pp. 94-111, https://doi.org/10.1002/PAM.21728.
[14] Cacault, M. et al. (2021), “Distance Learning in Higher Education: Evidence from a Randomized Experiment”, Journal of the European Economic Association, Vol. 19/4, pp. 2322-2372, https://doi.org/10.1093/JEEA/JVAA060.
[60] Castaño Muñoz, J., A. Pokropek and L. Weikert García (2022), “For to all those who have, will more be given? Evidence from the adoption of the SELFIE tool for the digital capacity of schools in Spain”, British Journal of Educational Technology, https://doi.org/10.1111/BJET.13222.
[12] Cellini, S. and H. Grueso (2021), “Student Learning in Online College Programs”, AERA Open, Vol. 7, https://doi.org/10.1177/23328584211008105.
[37] Coolbear, P. (2014), The eLearning guidelines, Ako Aotearoa, Wellington, http://elg.ac.nz/ (accessed on 25 July 2022).
[58] Costa, P., J. Castaño-Muñoz and P. Kampylis (2021), “Capturing schools’ digital capacity: Psychometric analyses of the SELFIE self-reflection tool”, Computers & Education, Vol. 162, p. 104080, https://doi.org/10.1016/J.COMPEDU.2020.104080.
[34] Council of the European Union (2022), Council Recommendation on building bridges for effective European higher education cooperation, Official Journal of the European Union, Brussels, https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32022H0413(01)&from=EN (accessed on 9 February 2022).
[74] De Carmen, P. et al. (2013), “Coordination and university results: an evaluation model”, Pensee Journal, Vol. 75/12.
[66] Digital Agency et al. (2022), Roadmap on the Utilization of Data in Education, https://www.digital.go.jp/assets/contents/node/basic_page/field_ref_resources/0f321c23-517f-439e-9076-5804f0a24b59/20220307_en_education_outline_01.pdf.
[64] DLEARN (n.d.), European Digital Learning Network, http://dlearn.eu/ (accessed on 27 July 2022).
[49] DPU (2022), “Dansk Clearinghouse for Uddannelsesforskning”.
[28] EADTU (2016), Manual - E-xcellence label, https://e-xcellencelabel.eadtu.eu/e-xcellence-review/manual (accessed on 27 March 2023).
[63] EDEN (n.d.), European Distance and E-Learning Network, https://www.eden-online.org/ (accessed on 21 June 2022).
[48] EEF (n.d.), Teaching and Learning Toolkit, https://educationendowmentfoundation.org.uk/education-evidence/teaching-learning-toolkit (accessed on 12 August 2022).
[53] EFI (2019), Gutachten zu Forschung, Innovation und technologischer Leistungsfähigkeit Deutschlands 2019 (Report on Research, Innovation and technological capacity in Germany 2019), Expertenkommission Forschung und Innovation (EFI), https://www.e-fi.de/publikationen/gutachten (accessed on 24 June 2022).
[16] EHEA (2022), Full members, http://www.ehea.info/page-full_members (accessed on 16 April 2022).
[40] Ehlers, U. and T. Zhang (2022), Curriculum and assessment Thematic Peer Group Report, European University Association (EUA), Brussels, https://eua.eu/resources/publications/1009:learning-teaching-thematic-peer-groups-2021.html (accessed on 4 August 2022).
[21] ENQA (2015), “ESG”, in Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG), European Association for Quality Assurance in Higher Education; European Students’ Union; European University Association; European Association of Institutions in Higher Education, http://www.enqa.eu/wp-content/uploads/2015/11/ESG_2015.pdf.
[76] EUA (2022), Learning and teaching thematic peer groups: Strategy and organisational culture group report, European Universities Association (EUA), Brussels, https://www.eua.eu/resources/publications/1010:strategy-and-organisational-culture-group-report.html (accessed on 25 July 2022).
[69] European Commission (2022), Work starts on the Digital Education Hub Community of Practice, https://education.ec.europa.eu/news/work-starts-on-the-digital-education-hub-community-of-practice (accessed on 27 July 2022).
[3] Finol, M. (2020), Asynchronous vs. Synchronous Learning: A Quick Overview, Bryn Mawr College, https://www.brynmawr.edu/blendedlearning/asynchronous-vs-synchronous-learning-quick-overview (accessed on 14 December 2021).
[38] Flexible Education Norway (2018), A Guide to Quality in Online Teaching and Learning FLEXIBLE EDUCATION NORWAY, Flexible Education Norway, Oslo, https://www.eadl.org/wp-content/uploads/2020/03/A-guide-to-quality-in-online-teaching-and-learning-for-digital-sharing-Norway-Nov-2018.pdf (accessed on 25 July 2022).
[72] Gaebel, M. et al. (2021), Digitally enhanced learning and teaching in European higher education institutions, European University Association (EUA), Brussels, https://eua.eu/resources/publications/954:digitally-enhanced-learning-and-teaching-in-european-higher-education-institutions.html (accessed on 7 December 2021).
[79] Garrison, D., T. Anderson and W. Archer (2010), “The first decade of the community of inquiry framework: A retrospective.”, The internet and higher education, Vol. 13/1-2, pp. 5-9, https://doi.org/10.1016/j.iheduc.2009.10.003.
[50] Golden, G., L. Troy and T. Weko (2021), “How are higher education systems in OECD countries resourced?: Evidence from an OECD Policy Survey”, OECD Education Working Papers, No. 259, OECD Publishing, Paris, https://doi.org/10.1787/0ac1fbad-en.
[6] Guiney, P. (2016), E-learning Provision, Participation and Performance, https://www.educationcounts.govt.nz/publications/tertiary_education/e-learning/e-learning-provision,-participation-and-performance.
[30] HAKA (2022), Juhend kvaliteetse e-kursuse loomiseks – Eesti Kõrg- ja Kutsehariduse Kvaliteediagentuur, 2021 [A guide to creating a quality e-course], Estonian Quality Agency for Vocational and Higher Education (HAKA), Tallinn, https://oppevara.edu.ee/ekursus/ (accessed on 25 July 2022).
[8] Hardt, D., M. Nagler and J. Rincke (2022), “Tutoring in (online) higher education: Experimental evidence”, CESifo Working Paper, No. 9555, Center of Economic Studies, Munich, https://www.cesifo.org/DocDL/cesifo1_wp9555.pdf (accessed on 17 June 2022).
[26] Hassan, M. (2022), “Middle East region: Benchmarking framework and data set for online, open, smart and technology-enhanced higher education”, in Report from the ICDE quality network: Global quality perspectives on open, flexible and distance learning, International Council for Open and Distance Education (ICDE), Oslo, https://static1.squarespace.com/static/5b99664675f9eea7a3ecee82/t/6220bfa167590f615a454786/1646313378469/ICDE+Quality+Network+Report+2021.pdf (accessed on 1 July 2022).
[13] Hemelt, S. et al. (2021), “Why is math cheaper than english? Understanding cost differences in higher education”, Journal of Labor Economics, Vol. 39/2, pp. 397-435, https://doi.org/10.1086/709535.
[82] HolonIQ (2021), Education Intelligence Unit Report: University Partnerships in 2021, https://www.holoniq.com/notes/450-university-partnerships-in-the-first-three-quarters-of-2021 (accessed on 19 February 2023).
[54] HRK (2021), Forderungen an Bund und Länder zur Weiterentwicklung der digitalen Lehrinfrastrukturen - (Demands to the federal and state governments for the development of digital teaching infrastructures), Hochschulrektorenkonferenz (German Rectors’ Conference) - HRK, https://www.hrk.de/positionen/beschluss/detail/forderungen-an-bund-und-laender-zur-weiterentwicklung-der-digitalen-lehrinfrastrukturen/ (accessed on 24 June 2022).
[22] Huertas, E. et al. (2018), Considerations for quality assurance of e-learning provision. Report from the ENQA Working Group VIII on Quality Assurance and E-Learning., European Association for Quality Assurance in Higher Education (ENQA), Brussels, https://www.enqa.eu/wp-content/uploads/Considerations-for-QA-of-e-learning-provision.pdf (accessed on 21 July 2021).
[20] INQAAHE (2022), “International Standards and Guidelines for Quality Assurance in Tertiary Education 2022 Edition © International Network for Quality Assurance Agencies in Higher Education”, https://www.inqaahe.org/sites/default/files/INQAAHE-International-Standards-and-Guidelines-ISG.pdf (accessed on 19 February 2023).
[18] INQAAHE (2018), Guidelines of Good Practice, International Network for Quality Assurance Agencies in Higher Education (INQAAHE), Barcelona, https://www.inqaahe.org/guidelines-good-practice (accessed on 10 January 2022).
[61] Janger, J., D. Campbell and A. Strauss (2019), “Attractiveness of jobs in academia: a cross-country perspective”, Higher Education, Vol. 78/6, pp. 991-1010, https://doi.org/10.1007/S10734-019-00383-7.
[65] Jisc (2020), Senior managers’ guide to learning analytics, Jisc, https://repository.jisc.ac.uk/7782/1/senior-managers-guide-to-learning-analytics.pdf (accessed on 2 August 2022).
[71] Jung, I. (2022), “Quality Assurance in Online, Open, and Distance Education”, in Handbook of Open, Distance and Digital Education, Springer Singapore, Singapore, https://doi.org/10.1007/978-981-19-0351-9_39-1.
[19] Karakhanyan, S. and B. Stensaker (eds.) (2020), Global Trends in Higher Education Quality Assurance, BRILL, Leiden, https://doi.org/10.1163/9789004440326 (accessed on 21 July 2022).
[15] Kofoed, M. et al. (2021), “Zooming to Class?: Experimental Evidence on College Students’ Online Learning during COVID-19”, https://docs.iza.org/dp14356.pdf (accessed on 14 December 2022).
[73] Latchem, C. (2016), Open and Distance Learning Quality Assurance in Commonwealth Universities: A Report and Recommendations for QA and Accreditation Agencies and Higher Education Institutions, Commonwealth of Learning, Vancouver, http://oasis.col.org/handle/11599/2046 (accessed on 23 June 2022).
[84] Lee-Post, A. and H. Hapke (2017), Online learning integrity approaches: current practices and future solutions, University of Kentucky, Lexington, KY, https://doi.org/10.24059/olj.v21i1.843.
[78] Lerstad, K. (2019), “Action Plan for Digitalisation in Higher Education and Research”.
[89] Lowenthal, P. et al. (2021), “Accessible Online Learning: An Analysis of Online Quality Assurance Frameworks”, Quarterly Review of Distance Education, Vol. 22/2, pp. 15-29, https://www.researchgate.net/publication/354765146_Accessible_Online_Learning_An_Analysis_of_Online_Quality_Assurance_Frameworks (accessed on 16 February 2022).
[32] Malta Further and Higher Education Authority (2021), Guidelines for Quality Assurance - For Online Learning Providers in Malta, Malta Further and Higher Education Authority, Valletta, https://mfhea.mt/wp-content/uploads/2021/10/Guidelines-for-FHEI-V1.pdf (accessed on 2 August 2022).
[83] Manoharan, S. and X. Ye (2020), On upholding academic integrity in online examinations, IEE, Piscataway, NJ, https://doi.org/10.1109/ic3e50159.2020.9288468.
[43] Martin, F. et al. (2020), Examining higher education faculty use of current digital technologies: importance, competence, and motivation, http://www.isetl.org/ijtlhe/.
[77] Mendoza, A. and C. Heredero (2016), Managing relationships and communications in higher education efficiently through digital social networks: The importance of the relational coordination model, https://revistas.unal.edu.co/index.php/dyna/article/download/49296/59617.
[41] National Forum (n.d.), National Forum for the Enhancement of Teaching and Learning in Higher Education: Resource Hub, https://www.teachingandlearning.ie/resourcehub/.
[80] Oblinger, D. (2004), “The Next Generation of Educational Engagement”, Journal of Interactive Media in Education, Vol. 2004/1, p. 10, https://doi.org/10.5334/2004-8-OBLINGER/GALLEY/199/DOWNLOAD/.
[56] OECD (2022), OECD Reviews of Innovation Policy: Germany 2022: Building Agility for Successful Transitions, OECD Reviews of Innovation Policy, OECD Publishing, Paris, https://doi.org/10.1787/50b32331-en.
[9] OECD (2021), Supporting the Digital Transformation of Higher Education in Hungary, Higher Education, OECD Publishing, Paris, https://doi.org/10.1787/d30ab43f-en.
[42] OECD (2021), The state of higher education One year into the COVID-19 pandemic, OECD Publishing.
[85] OECD (2020), Remote online exams in higher education during the COVID-19 crisis, OECD Publishing, Paris, https://www.oecd.org/education/remote-online-exams-in-higher-education-during-the-covid-19-crisis-f53e2177-en.htm.
[29] OECD (2018), Rethinking Quality Assurance for Higher Education in Brazil, Reviews of National Policies for Education, OECD Publishing, Paris, https://doi.org/10.1787/9789264309050-en.
[67] OECD (2017), The Funding of School Education: Connecting Resources and Learning, OECD Publishing, Paris.
[36] OECD (2023 (forthcoming)), Shaping the future of digital education: enabling factors for quality, equity and efficiency.
[25] Organisation of Ibero-American States for Education, S. (2021), Guía Iberoamericana de Evaluación de la Calidad de la Educación a Distancia [Ibero-America guide for quality assessment in distance education], Organización de Estados Iberoamericanos para la Educación, la Ciencia y la Cultura (OEI), Madrid, https://oei.int/oficinas/secretaria-general/publicaciones/guia-iberoamericana-de-evaluacion-de-la-calidad-educacion-a-distancia (accessed on 3 August 2022).
[17] Ossiannilsson, E. et al. (2015), The ICDE reports series Quality models in online and open education around the globe: State of the art and recommendations, International Council for Open and Distance Education (ICDE), Oslo, http://www.icde.orgCompletereport,ExecutivesummaryandAppendices:http://icde.typepad.com/quality_models/ (accessed on 21 July 2022).
[1] Parliament of Croatia (2022), Law on Quality Assurance in Higher Education and Science, https://narodne-novine.nn.hr/clanci/sluzbeni/2022_12_151_2330.html.
[45] QAA (2022), Hallmarks of Success, Quality Assurance Agency for Higher Education (QAA), London, https://www.qaa.ac.uk/membership/about-qaa-membership/current-year-membership-themes-and-topics/beyond-covid-19-the-future-of-digital-and-blended-delivery/hallmarks-of-success (accessed on 21 July 2022).
[35] QAA (2020), Guidance Building a Taxonomy for Digital Learning, Quality Assurance Agency for Higher Education (QAA), London, https://www.qaa.ac.uk/docs/qaa/guidance/building-a-taxonomy-for-digital-learning.pdf (accessed on 31 July 2020).
[7] Qayyum, A. (2022), “Assessing the Digital Transformation of Education Systems”, Handbook of Open, Distance and Digital Education, pp. 1-18, https://doi.org/10.1007/978-981-19-0351-9_14-1.
[39] Redecker, C. and Y. Punie (2020), European Framework for the Digital Competence of Educators. DigCompEdu., Publications Office of the European Union, Luxembourg, https://publications.jrc.ec.europa.eu/repository/handle/JRC107466.
[33] Republika Hrvatska (2016), Kriterije i postupke za vrednovanje online studija, https://www.nvzvotr.hr/images/documents/Kriteriji%C2%A0i%20postupci%20za%20vrednovanje%20online%20studija.pdf.
[51] Rijksoverheid (2018), Investeren in Onderwijskwaliteit - Kwaliteitsafspraken 2019-2024, Ministerie van Onderwijs en Wetenschap, The Hague, https://www.rijksoverheid.nl/documenten/rapporten/2018/04/09/investeren-in-onderwijskwaliteit-kwaliteitsafspraken-2019-2024 (accessed on 26 July 2022).
[27] Sankey, M. and F. Padró (2016), “ACODE Benchmarks for technology enhanced learning (TEL)”, International Journal of Quality and Service Sciences, Vol. 8/3, pp. 345-362, https://doi.org/10.1108/IJQSS-04-2016-0033.
[47] Sebesta, J. et al. (2020), “Digital Higher Education in Texas: A Meta-Analysis of Data and Research”, Digital Higher Education Consortium of Texas (DigiTex).
[5] Siegelman, A. (2019), Blended, Hybrid, and Flipped Courses: What’s the Difference?, Temple University, Philadelphia, https://teaching.temple.edu/edvice-exchange/2019/11/blended-hybrid-and-flipped-courses-what%E2%80%99s-difference (accessed on 14 December 2021).
[81] Simunich, B. et al. (2022), “Creating a Culture of Online Quality: The People, Policies, and Processes that Facilitate Institutional Change for Online Course Quality Assurance”, American Journal of Distance Education, Vol. 36/1, pp. 36-52, https://doi.org/10.1080/08923647.2021.2010021.
[4] Staring et al. (2022), Digital higher education: Emerging quality standards, practices and supports, https://www.oecd.org/education/digital-higher-education-f622f257-en.htm (accessed on 14 December 2022).
[55] Stiftung Hochschullehre (2022), Stiftung Innovation in der Hochschullehre (Foundation for innovation in higher education teaching), Stiftung Hochschullehre, https://stiftung-hochschullehre.de/ueber-uns/ (accessed on 24 June 2022).
[62] SURF (2022), Acceleration plan for educational innovation with IT, SURF, Utrecht, https://www.surf.nl/en/about-surf/acceleration-plan-for-educational-innovation-with-it (accessed on 28 July 2022).
[44] TEQSA (2022), Online learning good practice | Tertiary Education Quality and Standards Agency, Tertiary Education Quality and Standard Agency (TEQSA), Melbourne, https://www.teqsa.gov.au/online-learning-good-practice (accessed on 25 July 2022).
[31] TEQSA (2019), Guidance note: technology-enhanced learning. V.1, Tertiary Education Quality and Standard Agency (TEQSA), Melbourne, https://www.teqsa.gov.au/latest-news/publications/guidance-note-technology-enhanced-learning (accessed on 25 July 2022).
[75] Tømte, C. et al. (2019), “Digitalisation in higher education: mapping institutional approaches for teaching and learning”, https://doi.org/10.1080/13538322.2019.1603611, Vol. 25/1, pp. 98-114, https://doi.org/10.1080/13538322.2019.1603611.
[86] Tømte, C. et al. (2019), Digitalisation in higher education: mapping institutional approaches for teaching and learning, DOI: 10.1080/13538322.2019.1603611.
[46] UKÄ (2018), Distance Learning in Swedish Higher Education - A Short Version in English, Swedish Higher Education Authority (UKÄ), Stockholm, https://english.uka.se/download/18.357a35b0160df7fbfcd7486/1518179049446/report-distance-learning-in-swedish-higher-education.pdf (accessed on 9 August 2022).
[70] UNESCO (2020), UNESCO Recommendation on OER, https://en.unesco.org/themes/building-knowledge-societies/oer/recommendation (accessed on 12 August 2021).
[24] UNESCO (2019), Blended Learning Self-Assessment Tool, UNESCO Office Bangkok and Regional Bureau for Education in Asia and the Pacific, Bangkok, https://blendedlearning.bangkok.unesco.org/ (accessed on 25 July 2022).
[57] Vanderlinde, R. and J. van Braak (2010), “The e-capacity of primary schools: Development of a conceptual model and scale construction from a school improvement perspective”, Computers & Education, Vol. 55/2, pp. 541-553, https://doi.org/10.1016/J.COMPEDU.2010.02.016.
[23] Volungevičienė, A. et al. (2021), Developing a High-Performance Digital Education Ecosystem: Institutional Self-Assessment Instruments, European University Association (EUA), Brussels, https://eua.eu/downloads/publications/digi-he%20desk%20research%20report.pdf (accessed on 24 June 2022).