Ensuring Quality Digital Higher Education in Hungary

4. Supporting institutions to enhance the quality of digital higher education

Abstract

This project, led and implemented by the Organisation for Economic Co-Operation and Development (OECD), was carried out with financial support provided by the European Commission’s Directorate-General for Structural Reform Support (DG REFORM), in close collaboration with the Hungarian Ministry of Culture and Innovation (KIM) and the Hungarian Accreditation Committee (MAB).

This chapter analyses Hungary’s institutional support landscape for digital higher education and provides recommendations on how they can be strengthened to build the capacity of higher education institutions (HEIs) to assure the quality of their (digital) education offerings.

4.1 Institutional support landscape for digital higher education in Hungary

This section examines how higher education institutions (HEIs) in Hungary are being supported to build their capacity to effectively manage the quality of their (digital) education offerings, and presents three key challenges facing institutions as they seek to assure the quality of digital teaching and learning.

Who is supporting the quality enhancement of digital higher education in Hungary?

In systems where the responsibility for quality assurance (QA) rests primarily with HEIs, additional supports and guidance are often provided to institutions to help them enhance the quality and effectiveness of their internal QA policies and practices, including for digital education (Staring et al., 2022[1]). In addition to the (financial) incentives and supports offered by national governments and international organisations (such as the European Commission) for the development of digital higher education, or the strategic guidance, advice and recommendations provided by national governments or national QA bodies, a wide range of actors can play a part in building the capacity of HEIs.

Depending on a country’s cultural history and policy traditions, institutional supports for the quality enhancement of (digital) teaching and learning may be provided by – among others – the National Research and Education Network (NREN),1 the National Rectors’ Conference, stakeholder associations (e.g. national students’ union, academies of science), non-profit and private actors providing specialised support services, the national statistics office (in its capacity to provide sector-wide data and statistics on the performance of (digital) courses and study programmes), and HEIs themselves (engaging in sectoral co-operatives and partnerships to take ownership for quality enhancement) (Zhang, 2022[2]).

A wide range of actors can support institutional quality enhancement

While responsibility for the formal QA of higher education in Hungary is shared between the Hungarian Accreditation Committee (MAB), the Educational Authority (OH) and the Ministry of Culture and Innovation (KIM), a wide range of organisations can (and do) play a role in the quality enhancement (QE) of (digital) higher education in Hungary. Table 4.1 provides an overview of some of the most important consultative and implementing bodies in Hungarian higher education, including their primary function and activities in QE.

Table 4.1. Overview of consultative and implementing bodies in Hungarian higher education

Organisation	Who does it represent?	Primary function	Activities in QE
Tempus Public Foundation	Ministry	Support the internationalisation and quality enhancement of HEIs	Co-ordinate and organise international mobility, projects and training for HEIs
Governmental Agency for Information Technology Development (KIFÜ)	Ministry	Provide ICT services to institutions in Hungary	Provide ad hoc support to institutions with digitalisation matters
Hungarian Rectors’ Conference (MRK)	HEI leadership	Provide a discussion and advocacy forum for higher education leadership	Forum for exchange on internal QA practices; advisory function to the Ministry in developing QA regulation
Hungarian National Doctoral Council	Chairs of HEI Doctoral Councils	Provide a discussion forum and advocacy platform for doctoral students	Advisory function to the Ministry in developing QA regulation
National Union of Students (HÖOK)	Representatives of HEI student unions	Provide a discussion and advocacy platform for university students, and co-ordinate the efforts of student unions	Advisory and advocacy function to the Ministry; ad hoc surveys on students’ experience with teaching and learning
Association of Hungarian PhD and DLA Candidates (DOSZ)	Doctoral/PhD students	Provide a discussion and advocacy platform for doctoral candidates	Advisory and advocacy function to the Ministry; ad hoc surveys on students’ experience with teaching and learning
Hungarian Chamber of Commerce and Industry	Businesses/Private sector, typically industrial or service sector employers	Represent the interests of businesses in Hungarian higher education	Promote the introduction of dual vocational training in higher education; provide training and quality control of businesses in dual training programmes
Hungarian Academy of Arts	Responsible for overseeing and supporting artistic activities in higher education	Provide a co-ordination and funding body for artistic activities in Hungary	Consults on university professor applications in (performing) arts
Hungarian Academy of Sciences	Scientific community	Supporting scientific research and disseminating scientific knowledge	Advisory function to the Ministry in developing regulation regarding scientific research
Hungarian Olympic Committee	Elite and professional sports education	Co-ordinate Hungary’s participation in the Olympics and other international sports tournaments	Consults on university professor applications in sports and sports sciences

Source: Government of Hungary (2011[3]), Act CCIV of 2011 on National Higher Education, Government of Hungary, Budapest, https://net.jogtar.hu/jogszabaly?docid=A1100204.TV.

Establishment of specialised centres for digital higher education

To monitor and support the digital transformation of higher education in Hungary, KIM has recently set up two new bodies: the Digital Higher Education Competence Centre/Digital Success Nonprofit Ltd. (DSN/DHECC) and the Digital Government Development and Project Management Ltd. (DKFKT). However, interviews with higher education stakeholders conducted by the OECD review team reveal that, to date, these bodies have played a limited role in building HEIs’ capacity to effectively manage the quality of their digital course offers and internal QA systems.

• Digital Higher Education Competence Centre/Digital Success Nonprofit Ltd. (DSN/DHECC). The DSN/DHECC was set up by KIM in 2020. In September 2020, the centre carried out its first national-level survey to collect institutional leaders’ views on factors influencing HEIs’ level of digitalisation, with the aim of identifying ways to monitor digitalisation in Hungarian higher education. Participating institutions were also asked to share their digitalisation practices (e.g. creation of digital content, support services for digital education, updating of pedagogical methods, digital dissemination of research outputs etc.). A second survey was conducted in November 2020 to collect data on access to digital infrastructure at Hungarian HEIs, such as high-speed internet and the availability of digital tools (OECD, 2021[4]).

• Digital Government Development and Project Management Ltd (DKFKT). DKFKT is a fully state-owned organisation that has been established for an indefinite period of time (Government of Hungary, 2011[3]). It participates as a consortium leader or partner in EU-funded digitalisation projects for the 2021-2027 programming period (e.g. the Digital Europe Programme, the Multiannual Financial Framework and the Recovery and Resilience Fund). DKFKT oversees the implementation of various EU-funded projects to support the development of e-government practices and information and communications technology (ICT) across society.

How is the quality enhancement of digital higher education ensured?

In Hungary, efforts to support institutions with the quality enhancement of their digital higher education offerings and internal QA practices have primarily focused on national strategy setting and guidance to incentivise institutions to embed digitalisation as a strategic priority in institutional policy, and on strengthening the digital infrastructure of HEIs and society more broadly through targeted funding programmes. More specific support for staff professional development programmes and effective internal QA policies and procedures has, however, remained limited. The collection and use of system-level data on the quality and performance of digital higher education is also still developing in Hungary.

National guidance and steering for the development of digital higher education

In most higher education systems across the OECD, institutions have a high degree of autonomy over their organisational, financial, staffing, and educational matters. For example, according to the University Autonomy Scorecard, developed by the European University Association (EUA), the recruitment of senior academic staff is carried out by universities themselves in 18 out of 29 surveyed jurisdictions,2 and the recruitment of senior administrative staff in 21 jurisdictions3 (EUA, 2021a[5]). As a result, rather than strictly regulating how institutions should organise their financial and human resources, and educational offerings, governments across the OECD have typically tried to carefully balance institutional autonomy with national guidance or priority setting in governments strategies for higher education, or through the introduction of external quality assurance. In some jurisdictions, such as Austria, Finland, the Netherlands or the United States, performance-based funding models are being introduced to more actively steer institutional action (Staring et al., 2022[1]).

National strategy setting to guide institutional strategy development

In Hungary, HEIs are required to submit institutional development plans to KIM, setting out their strategic goals and priorities for the five years ahead (OECD, 2021, p. 92[4]). To guide the priorities included in institutional development plans, the government has adopted two strategic documents. Each of these includes a set of strategic goals and priorities that seek to incentivise HEIs to increase the flexibility, pedagogical innovation, and digitalisation of their higher education course offerings.

• Digital Education Strategy. The Digital Education Strategy was adopted in 2016 and “covers all parts of the Hungarian education system and aims to enable students at all levels in the education system to use digital tools and experience a digital study environment” (OECD, 2021, p. 49[4]). It includes a range of recommendations and associated action plans, including “the creation of the Digital Higher Education Competence Centre (DHECC)” (now established) and “changes to quality assurance, teacher performance review and other aspects of the current regulatory regime for higher education that currently impede the adoption of digitalisation to encourage instructors to use online channels” (ongoing) (OECD, 2021, p. 51[4]).

• Shifting of Gears Strategy in Higher Education. The Shifting of Gears in Higher Education strategy presents a set of goals and actions for 2016-30 consisting of three key components: better support for students, instructors and innovative (and digital) programme development, to support the delivery of high-quality and student-centred teaching and learning in higher education to improve student retention and completion rates. Together, these have the aim of making HEIs the primary centre for lifelong learning in Hungary, and to improve the attractiveness and competitiveness of Hungarian higher education internationally (KIM, 2016[6]). Among the 56 objectives listed in the strategy, nine relate directly to the digitalisation of higher education.

Performance-based funding to steer institutional action

In line with international practice across the OECD, Hungary is introducing a performance-based funding model for foundation status institutions. This is based on based on performance against a set of national key performance indicators (KPIs) covering education, research, infrastructure, and sectoral objectives (see Table 2.3, Chapter 2). Moreover, in a presentation delivered on 9 October 2020 during a roundtable discussion organised as part of an OECD on the digital transformation of higher education in Hungary (OECD, 2021[4]), a representative from the OH said that, while not explicitly included in the current strategy, a key objective is that “in five years 30% of all study programmes should be fully online and 50% should be hybrid (where possible)” (KIM, 2020[7]).

Specific financial support for the development of digital higher education

Strategic guidance for the digital transformation of higher education in Hungary is underpinned by financial support for the development of HEIs’ digital infrastructure and network connectivity. The National Digitalisation Strategy (KIM, 2020[8]) includes a specific “Digital Higher Education, Research and Public Infrastructure Development Strategy”, consisting of four separate programmes (Mohácsi, 2018[9]): the Hungarian backBONE++ (HBONE++) programme, the Digital Welfare Services (DJP) programme, investments to support the renovation of institutional infrastructure, and a National Super-Computing Programme (NSZP). As a result of these investments, HEIs’ access to the necessary basic digital infrastructure and network connectivity to support digital education has improved significantly in recent years, including for learners from socio-economically disadvantaged backgrounds (OECD, 2021[4]). Table 4.2 presents the four main dimensions of the strategy, which are currently in the early stages of implementation.

Table 4.2. Digital higher education, research and public infrastructure development strategy

Dimension	Actions
Hungarian backBONE++ (HBONE++)	Serving data network needs for the next 10-15 years Development of HBONE ++ backbone, application and access capacities By 2023, all institutions should be connected to the Worldwide Web on an anonym or separate wavelength Providing networking, research and public collections service on the HBONE++ infrastructure Strengthening network services as part of the Digital Success Programme (DJP) General Wi-Fi access, hospital education-research Wi-Fi network, public education digital network etc.
Further development of Digital Welfare Services (DJP)	Providing public education, higher education, research and public collection services Cloud services for education, research and public collections, including data storage (long-term data storage) Authentication services (eduID, eduroam, synchronisation) Provision of multimedia services (Voice over Internet Protocol (VoIP), Video conferencing, videotorium) Integrated e-learning services (e.g. MOOC, Big Data Analysis, Machine learning, VLE/LMS) Programmes to support service development Content and curriculum development applications
Renovation of the built infrastructure of institutions	Development of university infrastructure required for the use of digital services Modernisation of IT rooms and educational spaces (Classroom 4.0) Physical and virtual educational spaces, laboratories Upgrading audio-visual active and passive infrastructure (lighting, acoustics, etc.) Modernisation of institutional networks, Wi-Fi and IT systems
National Super-Computing Programme (NSZP)	Momentum-1 High Performance Computing (HPC) construction Support for national research objectives Permanent presence among the world's TOP 100 machines and among the TOP 10 machines in Europe NSZP Mentor Program Complex knowledge HPC supporting network Development of supporting software Involvement of SMEs in RDI activities Machine time Mentor programme

Source: Mohácsi, J. (2018[9]), Digitális Felsőoktatási, Kutatási és Közgyűjteményi Infrastruktúra- fejlesztési Stratégia (Digital Higher Education, Research and Public Library Infrastructure Development Strategy), KIFÜ, https://www.niif.hu/sites/default/files/niif_program_strategia_20180124v1_short.pdf.

However, interviews with higher education stakeholders carried out by the OECD review team reveal that more specific support for HEIs to purchase, maintain, upgrade, and effectively use digital technologies is limited. For instance, no nationally shared standards or guidelines exist for HEIs to help them navigate the highly diverse and fast-evolving educational technology (EdTech) landscape and make informed investment decisions or digital technology upgrades. Support staff responsible for the maintenance of information and communications technology (ICT) also do not have access to training, guidance, or support on how to effectively maintain their institution’s digital infrastructure, and especially the interoperability, security and data privacy of different ICT systems. The same is true for instructors who – other than the often limited pedagogical or ICT support offered by their own institution – do not have access to any shared guidelines, platform or institution that can provide them with external expert advice or guidance on how to use different educational technologies for pedagogical enhancement. The Shifting of Gears in Higher Education Strategy only includes nine broad objectives related to digitalisation (KIM, 2016[6]), and the ESG (ENQA, 2015[10]) also only refer to digital technology in a very broad sense (e.g. under ESG 1.6: Learning resources and student support), without identifying more specific standards or guidelines to support their effective use or maintenance. Hungary’s NREN, the Governmental Agency for Information Technology Development (KIFÜ), is also more active in school level capacity building for effective technology purchase, maintenance, and use, than at the higher education level, where support is more ad hoc (Digital Success Programme, 2016[11]; KIFÜ, 2021[12]; OECD, 2021[4]).

Supporting staff professional development for digital teaching and learning

Supports and incentives for the professional development of academic staff’s digital and pedagogical skills are less extensive than is support for the adoption of digital infrastructure. Higher education institutions do not have obligations for the professional development academic staff and assessment of their digital and pedagogical skills; there is a lack of nationally shared guidance and training on high-quality digital teaching and learning; and academic staff have limited opportunities, time, and incentives to engage in inter-institutional collaboration and peer learning on the topic of digital higher education and QA.

The professional development of academic staff

As is the case in many other higher education systems across the OECD, HEIs in Hungary are not required by law to organise professional development for their instructors, or to assess instructors’ pedagogical skills as part of staff appraisals or performance assessments. However, such practices are slowly emerging in many Hungarian HEIs, including for digital education (see Chapter 3). National regulation (Government of Hungary, 2011[3]) only specifies the minimum number of weekly teaching hours staff are required to deliver (as an average across two consecutive academic semesters) and the qualifications staff should hold. National regulation also distinguishes between staff employed on a “teaching track” and those employed on a “research track”. While both tracks include teaching and research duties, staff employed on a “teaching track” are expected to spend at least 80% of their time on teaching activities; for staff employed on a “research track” this is only 20%. Table 4.3 presents an overview of the minimum requirements for staff employed on a “teaching track” in Hungary. National regulation also includes some details on assessment practices. These should be made public to students before the course starts, but instructors can choose freely between diagnostic (i.e. pre-assessment, to test students’ knowledge), formative (i.e. as part of the course) and summative (i.e. at the end of the course) assessment. However, the most common form of assessment used by instructors in Hungary is summative, end-of-course assessment (Kálmán, Tynjälä and Skaniakos, 2020[13]).

Higher education stakeholders interviewed by the OECD review team mentioned that the lack of an explicit acknowledgement or minimum requirement for instructors to engage in staff professional development, or for HEIs to organise staff training and performance assessment, including around skills for digital teaching and learning, constituted a major barrier to the further development of the pedagogical skills of Hungarian academics and the quality of teaching and learning in Hungarian higher education more generally.

Table 4.3. Minimum requirements for staff employed on a “teaching track”

Status	Teaching activities	Research activities	Requirements
Assistant lecturer (tanársegéd)	Minimum 12 hours per week (the teaching time may be increased by not more than 40% and reduced by not more than 25%)	At least 20% of total working time allocated to research or artistic activities	Be enrolled in a doctoral programme Skills to teach subjects in foreign languages
Senior lecturer (adjunktus)	Minimum 12 hours per week (the teaching time may be increased by not more than 40% and reduced by not more than 25%)	At least 20% of total working time allocated to research or artistic activities	Holder of a doctoral degree
Master lecturer (mesteroktató)	Minimum 12 hours per week (the teaching time may be increased by not more than 40% and reduced by not more than 25%)	At least 20% of total working time allocated to research or artistic activities	Holder of master’s degree Minimum 10 years of working experience Skills to provide practical training to students
College / university associate professor (főiskolai / egyetemi docens)	Minimum 10 hours per week (the teaching time may be increased by not more than 40% and reduced by not more than 25%)	At least 20% of total working time allocated to research or artistic activities	Holder of a doctoral degree Skills to supervise the academic and/or artistic work of students
College / university professor (főiskolai / egyetemi tanár)	Minimum 8 hours per week (the teaching time may be increased by not more than 40% and reduced by not more than 25%)	At least 20% of total working time allocated to research or artistic activities	Appointment by Ministry following MAB evaluation of application for university professor title

Source: Government of Hungary (2011[3]), Act CCIV of 2011 on National Higher Education, Government of Hungary, Budapest, https://net.jogtar.hu/jogszabaly?docid=A1100204.TV.

National guidance and training on digital teaching and learning

To support the professional development of students and staff for digital higher education, the Hungarian government is planning to fund digital competence development programmes and the further development of the Digital Textbook Library (Digitális Tankönyvtár), managed by the OH (Educational Authority, n.d.[14]). Higher education stakeholders interviewed by the OECD review team mentioned that the library currently consists of a repository of Word and PDF documents, which does not capitalise on the full potential offered by digital technologies to create an interactive, open and collaborative database for the sharing and development of best practices in (digital) teaching and learning. As a result, the national resource bank is felt to be of limited value in the development of academic staff’s pedagogical practices. To address this challenge, the Ministry is planning to populate the library with modernised and digitised educational content, starting with priority study fields, as well as a professional development module for instructors.

Several organisations – both publicly and privately funded – as well as HEIs themselves have also started to engage in projects focused on supporting the development of shared national guidance, training and peer learning on digital teaching and learning. For example, as mentioned in Chapter 3, digital education experts from four HEIs in Hungary – Károli Gáspár University of the Reformed Church, Budapest Business School, the University of Pécs and the Hungarian Dance Academy – have developed a handbook to promote and support the effective use of digital technology among Hungarian higher education instructors (Dringó-Horváth et al., 2020[15]). The Faculty of Education and Psychology at Eötvös Loránd University has also developed a Faculty Distance Education Handbook during the COVID-19 pandemic (Bereczki et al., 2020[16]). In 2020, the Hungarian Association for Counselling in Higher Education (FETA), which represents all higher education counselling centres in Hungary (FETA, n.d.[17]), has published Tips for coping with stress and anxiety during the COVID-19 pandemic (Kiss et al., 2020[18]). The Hungarian Academy of Sciences manages a Virtual Collaboration Area (VirCA, n.d.[19]) and Electronic Information Service National Programme (EISZ, n.d.[20]), providing member institutions with a platform for the exchange of digital education content and resources, and to collaborate virtually.

Tempus Public Foundation also regularly organises seminars and workshops to support the professional development of academic staff – including on the topic of digital teaching and learning – which, according to estimates by Tempus Public Foundation, have so far attracted between 1 000 and 2 000 higher education instructors. This however only represents a small share of Hungary’s total higher education teaching population and means that the reach of these workshops remains limited to date. With financial support from KIM, Tempus Public Foundation has also developed a tool to support the self-assessment, peer review, student assessment and appraisal of staff’s pedagogical skills by HEIs. It also runs an international teaching award and maintains a database of international best practice in higher education teaching and learning (PROFFORMANCE, 2022[21]).

International collaboration and peer learning

Box 4.1. Recent European initiatives driving institutional quality enhancement in Hungary

Digital Education Hub

On 14 February 2022, the European Commission launched the Digital Education Hub, which is intended to strengthen cooperation and exchange in digital education and responds to the need for greater dialogue between stakeholders across Europe. It also seeks to address weak spots and perceived fragmentation of digital education policy, research, and implementation practices at the European level. The Hub aims to develop a European “community of practice” to engage a wide variety of stakeholders and support cross-sector collaboration on digital education in Europe.

European Universities Initiative

The European Universities Initiative (EUI) is one of the flagship programmes of the EU's European Education Area with the ambition to build European universities of the future, promote European values and identity, and improve the quality and competitiveness of European higher education. Within the European University Alliances funded as part of the programme, different cooperation models are explored that are based on a common long-term mission and strategy. Joint degree and micro-credential programmes as well as staff and student mobility are at the heart of EUI projects. These partnerships create new challenges in the digitisation of higher education. To date, 24 Hungarian institutions participate in European University Alliances.

Sources: European Commission (2022[22]), Work starts on the Digital Education Hub Community of Practice, https://education.ec.europa.eu/news/work-starts-on-the-digital-education-hub-community-of-practice; and European Commission (2020[23]), European Universities Initiative, https://ec.europa.eu/education/education-in-the-eu/european-education-area/european-universities-initiative_en.

Participation in international projects, including student and staff exchanges, was mentioned by almost all higher education stakeholders interviewed by the OECD review team as highly beneficial to the development of internal QA systems, as well as staff and student (digital) skills and competencies. Stakeholders were, therefore, positive about existing government supports and incentives for the internationalisation of higher education in Hungary (KIM, 2016[6]). The development of joint programmes as part of EU-funded initiatives such as the European Universities Initiative (EUI) and the Digital Education Hub (see Box 4.1) was mentioned as an important driver for the modernisation and digitalisation of institutional quality management practices. Stakeholders explained that participation in such programmes requires institutions, instructors and QA agencies to collaborate in the joint development and delivery of programmes and QA, often in digitally enhanced formats. However, institutions and instructors currently have limited time and opportunities to engage in inter-institutional collaboration.

National performance monitoring of digital higher education

During the COVID-19 pandemic, several organisations in Hungary started to collect feedback from higher education students on their experience of fully online remote learning to inform the institutional quality enhancement of digital teaching and learning. Hungary has also started to take steps to adapt its higher education national administrative data and graduate tracking systems to digitalisation..

National surveys on the quality of digital teaching and learning

In recent years, several organisations in Hungary have started to carry out student feedback surveys on the quality of digital higher education. For example, in 2020 the National Union of Students (HÖOK) conducted a student survey shortly after the transition to emergency remote learning during the COVID-19 pandemic. More than 17 000 students participated (12 000 student responses were used in the analysis), with mostly undergraduate students responding (HÖOK, 2020[24]). KIM also commissioned two surveys on digital higher education in the fall of 2020, administered by the DHECC (DSN/DHECC, 2021[25]). In addition, the OECD conducted a higher education stakeholder consultation survey in February–March 2021 as part of the project Supporting the Digital Transformation of Higher Education in Hungary (OECD, 2021[4]). In 2022, the OH included questions related to students’ experience with online learning during the COVID-19 pandemic in its annual graduate career tracking survey (DPR) (Educational Authority, 2020[26]). Stakeholder interviews also revealed that the Association of PhD and DLA Candidates (DOSZ) and Rectors’ Conference (MRK) conduct annual surveys among their members. In the case of DOSZ, the 2021 paper-based student feedback survey includes questions related to students’ experience with digital learning. Finally, as part of the current project, the OECD has collected qualitative feedback from national and institution-level higher education stakeholders on the quality and quality management practices for digital teaching and learning in Hungarian higher education institutions.

Higher education stakeholders interviewed by the OECD review team noted that there is no regular and national student survey on the quality of (digital) higher education in Hungary. Moreover, stakeholders make limited use of available national (and institutional) survey data on the performance of digital higher education.

System-level administrative data collection on digital higher education

System-level administrative data collection, analysis and dissemination of information on the performance of higher education students, programmes, and institutions is managed by the OH, through three main tools: the administrative Higher Education Database and Information System (Felsőoktatási Információs Rendszer – FIR), which compiles institutional student enrolment, progression and completion data, collected through the national student information system NEPTUN (DSN/DHECC, 2021[25]); the annual graduate tracking career survey, which collects information on graduate employment outcomes and in which most HEIs in Hungary participate (Educational Authority, 2020[26]); and the national student information and admissions platform, Felvi.hu, which provides information to prospective students on the content and learning outcomes, as well as the admission, application and enrolment requirements for all higher education study programmes in Hungary (Educational Authority, n.d.[27]).

The data collection for these systems, however, is still based on the existing legal categories of full-time, part-time and distance education. As a result, it is not possible for higher education stakeholders to compare information on the performance of study programmes based on study mode (i.e. online, hybrid, in person/blended) (OECD, 2021[4]). As part of the project Supporting the Digital Transformation in Hungary, the OECD has developed a list of potential indicators for Hungary to include in its national data collection systems to adapt them to digitalisation (OECD, 2021, pp. 99-102[4]). However, as recommended in Chapter 2 of this report, an important first step for Hungary will be to revise the existing categorisation of study formats in higher education to reflect digital education.

Higher education stakeholders interviewed by the OECD review team also mentioned that the information included on national platforms is insufficiently detailed to inform prospective students. For example, while a link to the national graduate career tacking survey is included on the Felvi.hu website, the information on study programmes is – like the accreditation system – highly input-oriented. The website mainly includes information on the content and learning outcomes of programmes, as well as application and enrolment requirements (see Box 4.2). There is no comparable information on programme outcomes (e.g. drop-out, completion or employment rates of students) as can be found on national higher education information platforms in other OECD jurisdictions For more detailed information on the content, learning outcomes and quality of study programmes, students need to consult the website of individual HEIs and faculties, which often vary in terms of the amount and type of information provided and makes comparison between institutions and programmes challenging. Moreover, Felvi.hu is only available in Hungarian, which does not encourage applications from international students. Recently, however, it has been made possible for students to apply online through Felvi.hu (Educational Authority, n.d.[28]).and further developments to the Felvi.hu system are planned as part of the ongoing OH-MRK-MAB project (see Chapter 2).

Box 4.2. Study programme information included on Felvi.hu

The Felvi.hu database provides the following information on Hungarian higher education programmes:

• Faculty name

• Training level (higher VET/bachelor’s/master’s/doctoral/post-graduate/single cycle)

• Delivery mode (full-time/part-time/distance/correspondence)

• Form of financing (self-funded/state-funded)

• Cost (per semester)

• Maximum number of students admitted to the study programme

• Secondary school final exam requirements (language competency)

• Major

• Credit limit

Source: Educational Authority (n.d.[28]), Felvi.hu – E-admission, https://www.felvi.hu/felveteli/efelveteli.

What are the key challenges facing higher education institutions (HEIs)?

The stakeholder consultations carried out by the OECD review team as part of this project revealed that institutions face several challenges in enhancing the quality of their digital provision. Three main areas of support can be identified: developing, maintaining, upgrading, and supporting the effective use of digital technology; supporting and incentivising the professional development and assessment of staff for digital teaching and learning; and developing effective processes for the collection, monitoring and use of data on the performance of digital higher education.

Developing, maintaining, upgrading, and supporting the effective use of digital technology

As evidenced by the speed with which institutions and instructors were able to switch to remote online instruction during the COVID-19 pandemic (DSN/DHECC, 2020[29]), the infrastructure of HEIs in Hungary is relatively well-developed and funded to support digital teaching and learning. However, several stakeholders interviewed by the OECD review team highlighted gaps in the capacity of HEI leadership, instructors, and (IT) support staff to develop, maintain, upgrade, and effectively use educational technology that keeps up with the latest developments in an increasingly fast-developing EdTech landscape.

Apart from a few large and well-resourced institutions (such as the University of Debrecen (University of Debrecen, 2020[30])), institutions in Hungary often lack the financial resources, capacity, and internal expertise to regularly review the quality and suitability of their digital infrastructure to ensure it incorporates the latest technological developments and standards. HEIs also noted challenges related to the maintenance of their digital education infrastructure, in particular ensuring the interoperability, data privacy and security of the multitude of digital technologies and systems used by instructors, learners, and (IT) support staff – for example, many institutions face challenges in linking their institutional software or virtual learning environment/learning management system (VLE/LMS) to the government’s NEPTUN system (Tolnai, 2021, p. 172[31]). Finally, institutions highlighted challenges related to ensuring the effective use of digital technologies by students and instructors, especially those technologies that have the potential to transform and enhance quality, inclusion, learner success and flexibility in higher education (e.g. learning analytics, Open Educational Resources (OER), Massive Open Online Courses (MOOCs), hybrid course design, trusted and authentic online assessment).

Supporting and incentivising the professional development and assessment of staff for digital teaching and learning

It is widely acknowledged by academic staff that teaching innovation and quality are not top priorities for most instructors in Hungary, and that this is the result of a career system in which academics are rewarded (with advancement and preferment) for measurable research and publication activities, not teaching performance. This is also reflected in the absence of a legal requirement for HEIs in Hungary to provide instructors with professional or pedagogical development. Stakeholders interviewed by the OECD review team highlighted that few institutions have either the required in-house expertise or the capacity to develop training and guidance for staff around the wide range of professional development areas for effective digital delivery. Where targeted institutional support for digital pedagogy exists, this has often emerged only recently in response to the COVID-19 pandemic and is therefore still in the very early stages of development. While an increasing number of institutions has started to conduct performance assessments of instructors’ pedagogical skills and include these in appraisal procedures (see Chapter 3), few have focused specifically on assessing instructors’ digital skills. Institutions also struggle to mainstream best practice in digital pedagogy across the institution, and to incentivise instructors to focus on actively enhancing their pedagogical practices. The main barriers cited are the heavy workload for academic staff operating in a competitive research environment combined with a high number of administrative duties and societal engagement activities on top of ensuring quality teaching, for which they receive little recognition. In addition to this, negative attitudes remain towards the potential added value of digitalisation for teaching and learning, especially among senior academics and staff from more practically oriented disciplines.

The “isolated development” of quality digital pedagogy within institutions also affects more “general digital developments” across Hungarian higher education (Tolnai, 2021, p. 173[31]). Higher education instructors interviewed by the OECD review team mentioned that they would benefit from widely discussed, shared, and agreed guidance on what good practice in digital instruction and QA looks like. The ESG (ENQA, 2015, p. 8[10]) and accompanying evaluation sheets developed by MAB for institutional accreditation do not provide sufficiently detailed guidance for institutions on how to manage the quality of their programmes. Stakeholders also highlighted the need for additional opportunities to engage in regular training, peer learning and collaboration around digital teaching and learning and QA, both at national and international level. Institutional support staff and students underlined the importance of building the capacity of student counselling and advice centres to effectively support students in online settings. This was highlighted as particularly important to mitigate the risk of students from rural and socio-economically disadvantaged backgrounds falling behind even further in fully online and hybrid courses (due to potential inequalities in access to and effective use of educational technologies), and to tackle students’ declining mental health and wellbeing.

Developing effective processes for the collection, monitoring and use of data on the performance of digital higher education

Hungarian higher education law requires institutions to collect feedback from students on the quality of their courses. However, no further specifications, guidance or recommendations are provided on the regularity, methodology, scope or target groups of feedback and data collection practices. As a result, institutions (and within those, individual faculties) have taken varied, and often limited, approaches towards collecting and using institution- and faculty-level data on the performance of their (digital) course offers. In most cases, the collection of feedback on the quality of instruction is limited to a student or staff feedback survey (in some cases still paper-based) distributed at the end of the semester or academic year, and often not adapted to the specificities of digital learning. While ad hoc surveys on the quality of fully remote online instruction have emerged in response to the COVID-19 pandemic, they are often not embedded in regular institution- or faculty-wide feedback processes. More regular feedback loops are still developing in Hungary – for example, the collection of end-of-lecture feedback, as at Semmelweis University (Kiss, 2022[32]), or including student representatives in strategic discussions. While most Hungarian HEIs participate in the national graduate career tracking survey (DPR), carried out by the OH (Educational Authority, 2020[26]), the collection and use feedback from employers or other civil society actors on the labour market relevance of courses and programmes is not common (OECD; European Commission; DGES, 2022_[33]). Qualitative stakeholder feedback and learning analytics data generated through the VLE/LMS were highlighted as two important additional data sources for HEIs to triangulate with administrative and survey data to obtain an in-depth and real-time picture on the quality of instruction and support needs of students.

More comprehensive and institution-wide self-assessments or inter-institutional benchmarking of digital learning, such as the institution-wide digital readiness assessment of instructors conducted by Károli Gáspár University of the Reformed Church, using the EU’s DigCompEdu framework (Dringó-Horváth et al., 2020[15]), are also rare in Hungary. One of the main reasons for this is that there are, at present, few tools adapted to the Hungarian higher education context to support such self-assessments. Another reason is that critical self-assessment and peer learning in itself is still a relatively new concept for many institutions and instructors in Hungary. Prior to the introduction of institutional accreditation based on the ESG, the accreditation of HEIs in Hungary was primarily focused on technical requirements and paid limited attention to teaching and learning processes, student outcomes or internal QA practices (see Chapter 2). In addition to this, the ESG – which are used by MAB as a basis for their institutional and doctoral schools’ accreditation procedures – do not include any specific digital education indicators.

4.2 International practice and recommendations to support higher education institutions to build their capacity for the effective quality management of digital teaching and learning

While several organisations have stepped up to support institutions with the quality enhancement of their digital education infrastructure (e.g. funding provided by the government) or staff professional development (e.g. development and dissemination of self-assessment tools, guidance and best practices by Tempus Public Foundation and HEIs), the current support landscape for HEIs in Hungary has been unable to meet the (in some cases highly specific) support needs of institutions, staff and students. National-level data collection and monitoring of the performance of digital higher education has primarily been carried out in the form of ad hoc student surveys undertaken by various organisations operating at sectoral level (e.g. the OH, HÖOK, DOSZ or the Ministry), but there is a lack of more granular and up-to-date information on the performance of online, hybrid and in-person/blended study programmes. There is also limited effective use of system- and institution-level data on the performance of (digital) higher education by HEIs to inform programme review and development.

The following section presents international examples of best practice for the effective quality management of digital teaching and learning from which Hungary could take inspiration. It also sets out four proposed policy recommendations and associated policy options on how to implement them, as Hungary seeks to strengthen its institutional support system for digital higher education and build the much-needed bridge between national policy ambitions and institutional practices for the QA of digital higher education.

Support the purchase, maintenance, upgrading and effective use of digital technology

In several OECD jurisdictions, in addition to strengthening the general digital infrastructure and connectivity of HEIs as part of wider digital society strategies, governments have also started to fund the development of specific digital technologies in HEIs. Austria and Germany, for example, have issued strategies to support the integration of OER in higher education (Ebner et al., 2016[34]; BMBF, 2022[35]). In some systems, the NREN or other sectoral organisations support institutions to make joint investments to help reduce the costs associated with purchasing new digital technologies independently. In the United Kingdom, for example, APUC (Advanced Procurement for Universities and Colleges) is a joint procurement service for Scotland's universities and colleges (APUC, 2022[36]). It is one of the eight procurement consortia in UK higher education, six regionally based and two specialist national consortia, supporting collaborative procurement within higher and further education in the UK (UKUPC, 2022[37]).

In addition to national guidance and financial support, NRENs play a key role in many OECD jurisdictions by supporting HEIs to navigate the highly diverse, complex and fast-evolving EdTech landscape. Sectoral organisations, as well as private and non-profit (international) organisations, have also developed a variety of guidance materials, tools and resources to support institutions with the maintenance and effective use of digital technologies (Staring et al., 2022[1]). In the Netherlands, Croatia, Lithuania and Norway, NRENs provide centralised hosting services on top of central network connectivity, and rely on a combination of peer learning activities and expert advice to guide digital infrastructure choices and the use of digital technologies by HEIs (SURF, 2022[38]; LieDM, 2022[39]; CARNET, 2022[40]; Sikt, n.d.[41]), (LieDM, 2022[39]). Guidance and training focus on both specific technologies (e.g., learning analytics, OER, MOOCs) or dimensions related to their maintenance or use (e.g. interoperability, data privacy and security). Table 4.4 presents examples of resources developed by a variety of both national and international organisations.

Table 4.4. Examples of guidance and resources to inform the purchase, maintenance, and effective use of digital technologies

Digital technology (dimensions)	Examples	Organisation (and type)	Description
Open Educational Resources (OER)	Recommendation on the integration of OER (Ebner et al., 2016[34])	Austrian New Media Forum in Higher Education (sectoral organisation)	A recommendation to support the integration of OER in higher education has been developed by the Austrian New Media Forum in Higher Education.
Massive Open Online Courses (MOOCs)	Framework for the QA of MOOCs (OpenupEd, 2022[42])	OpenUpEd (international network of MOOC providers)	OpenUpEdu, one of the largest networks of MOOC providers for higher education in Europe – has developed a checklist to support HEIs with the quality enhancement of MOOCs.
Hybrid course design and delivery	Methods and tools for blended education (SURF, n.d.[43])	SURF (NREN)	SURF in the Netherlands coordinates a specific working group for the collection of best practices and the development of guidance for hybrid course design and delivery.
Inclusive course design	Framework for inclusive course design (CAST, 2018[44])	CAST (non-profit organisation)	CAST, a non-profit educational research and development organization, has developed a framework to support inclusive course design, including 31 indicators.
Trusted and authentic online assessment	Framework for the quality assurance of e-assessment (2019[45]),	TeSLA (European Commission funded co-operation project)	A European consortium of universities has developed a framework including eight standards and associated “indicators” and “minimum evidence” for the quality assurance of e-assessment practices.
Learning analytics	Jisc framework for assessing institutional readiness for learning analytics (2020[46])	Jisc (NREN)	Jisc in the United Kingdom has developed a framework to review and support the effective use of learning analytics by institutions.
Interoperability	IT Interoperability Framework (Department of State Information Systems, 2005[47])	Estonian Department of State Information Systems (government)	Estonia has had a national interoperability framework since 2005, which provides a set of standards and guidelines to ensure interoperability between the systems used by public administration institutions, enterprises, and individual users at both national and European level.
Data security and privacy	Guidance for a safe and reliable learning analytics and student data infrastructure (SURF, 2021[48])	SURF (NREN)	SURF, the Netherlands NREN, has developed specific guidance for institutions to help them meet privacy, ethics, and functionality standards when using learning analytics and student data.

Source: Based on a review of international policy and practice included in Staring et al. (2022[1]), “Digital Higher Education: Emerging Quality Standards, Practices and Supports”, OECD Education Working Papers, No. 281, OECD Publishing, Paris, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en.

Recommendation 6: Support the development of shared national standards and guidance for the purchase, maintenance, upgrading and effective use of digital technology

To assist HEIs with the purchase, maintenance, upgrading and effective use of digital technology, Hungary could consider implementing a combination of guidance and financial support mechanisms (to guide and support institutional investment in digital infrastructure), centralised IT support (to support IT staff with central network maintenance and ensure the security and interoperability of different IT systems) as well as supporting the development of shared sectoral guidance on the effective use of different digital technologies in higher education. Table 4.5 presents an overview of the options and potential international models from which Hungary could take inspiration.

Table 4.5. Options for Hungary to support the purchase, maintenance, and effective use of digital technology in higher education

Options	Description	Potential model(s) for Hungary
1. Steering and targeted funding for specific digital technologies	Through targeted and competitive funding calls, the government (KIM) supports and incentivises HEIs to make investments in digital technologies and resources that have demonstrated potential to enhance the quality of digital higher education (e.g. OER or learning analytics), without prescribing which providers HEIs should choose.	OER strategy in Germany and Austria
2. IT maintenance support	The NREN (KIFÜ) strengthens its role in supporting institutions with central network management and hosting services, as well as a central student assessment (or proctoring) platform to free up the capacity of institutional IT support staff to help instructors with the effective use of digital technology in their pedagogical practice.	Centralised hosting and networking services offered by NRENs in Croatia, Lithuania and Norway
3. Guidance and training for effective technology purchase, maintenance and use	The NREN (KIFÜ), MAB or a sectoral (stakeholder) organisation is supported to take on the responsibility for co-ordinating the development of shared sectoral guidance and training – led by digital education and quality assurance experts and practitioners from the higher education sector – to support HEIs with the purchase, maintenance and effective use of digital technologies.	Shared sectoral guidance developed under the co-ordination of NRENs in the Netherlands and United Kingdom

Source: Based on a review of international policy and practice included in Staring et al. (2022[1]), “Digital Higher Education: Emerging Quality Standards, Practices and Supports”, OECD Education Working Papers, No. 281, OECD Publishing, Paris, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en.

Support and incentivise the quality enhancement of staff professional development

A 2022 review of national initiatives in teaching and learning in higher education in the European Higher Education Area (EHEA) found that eight jurisdictions4 have national regulation in place to direct the professional development of staff offered by HEIs (Zhang, 2022[2]). In these systems, higher education providers are required to offer professional development to their staff and/or evaluate instructors’ pedagogical competencies as part of institutional employment and appraisal arrangements. While institutions are mostly free to decide how to organise professional development and which criteria to use for performance assessment, in some cases (e.g. Norway) national regulation specifies the minimum number of hours of pedagogical training instructors are required to complete (e.g. Denmark, Estonia and France).

In some systems, national-level sector organisations have developed standards or guidelines to support the quality enhancement of staff professional development. In the United Kingdom and Ireland, for example, Advance HE (the United Kingdom) and the National Forum for the Enhancement of Teaching and Learning in Higher Education (Ireland) have developed national frameworks for the quality enhancement of teaching and learning in higher education (Advance HE, 2019[49]), (National Forum for the Enhancement of Teaching and Learning in Higher Education, 2016[50]). In the United Kingdom, AdvanceHE uses the framework to review and accredit staff professional development programmes. In Denmark and the Netherlands, the National Rectors’ Conferences have developed national qualifications frameworks for staff professional development programmes, which are reviewed on a cyclical basis. Denmark’s national qualifications framework was adopted as recently as 2021 (Universities Denmark, 2021[51]), while the Universities of the Netherlands (Universiteiten van Nederland) already had a framework in place in 2008 (VSNU, 2008[52]). Following a review of all university teacher qualifications in 2018, the Universities of the Netherlands concluded that “education and ICT and blended learning” should be one of four key competences for institutions to focus on in staff professional development (VSNU, 2018, p. 16[53]).

Some systems have supported the development of national (online) platforms to promote virtual collaboration and exchange among higher education practitioners (e.g. Austria (iMoox, 2022[54]) and France (FUN, 2022[55])). Others have funded dedicated national centres to co-ordinate and support the organisation of national guidance, training, and peer learning activities for the quality enhancement of (digital) teaching and learning. Such centres exist in Ireland (Education, 2022[56]), New Zealand (Coolbear, 2014[57]), Norway (Flexible Education Norway, 2018[58]) and Germany (Stiftung Innovation in der Hochschullehre, n.d.[59]). In the United Kingdom, support is provided by a range of organisations operating at national level, including Jisc, AdvanceHE and the Association for Learning Technology.

Recommendation 7: Introduce national regulation and support for the quality enhancement of staff professional development

To support the professional development of instructional staff, in addition to introducing requirements in national regulation (e.g. that HEIs are required to organise staff professional development or that instructors should complete a minimum number of training hours), Hungary could consider a range of other institutional support options to promote the development of a national “community of practice” around higher education teaching and learning. These options include: supporting HEIs to collaboratively develop shared standards for staff professional development (as in the Netherlands, the United Kingdom and Ireland); developing guidance, tools and training for the (self-)assessment of instructors’ (digital) skills and competencies (building on existing tools such as PROFFORMANCE); or establishing a national centre for teaching and learning in higher education, with dedicated responsibility for organising training, guidance and capacity building activities for staff in HEIs (as in Norway, the United Kingdom, Ireland and Germany). Table 4.6 presents an overview of the options and potential international models from which Hungary could take inspiration.

Table 4.6. Options for Hungary to support the quality enhancement of staff professional development for (digital) teaching and learning

Options	Description	Potential model(s) for Hungary
1. Introduce national regulation on staff professional development	The government (KIM) introduces a requirement in national higher education law, which specifies that institutions are required to offer professional development opportunities to instructional staff and regularly assess their competencies, including digital skills. However, institutions have full autonomy to decide how to organise professional development, as well as which criteria to use for performance assessments.	Regulation on staff professional development in Estonia, France and Norway
2. Develop national standards for staff professional development	The National Rectors’ Conference, or another sectoral (stakeholder) association, is given responsibility and resources to co-ordinate the development of institutionally shared national standards and guidelines for staff professional development programmes offered by HEIs in Hungary. Based on these standards, the quality and effectiveness of institutionally-based staff professional development programmes could be carried out.	Standards for staff professional development in Denmark, the Netherlands, the United Kingdom and Ireland
3. Provide guidance and training for the (self-)assessment of instructors’ digital skills by HEIs	Building on the PROFFORMANCE self-assessment tool, Tempus Public Foundation, or another sectoral (stakeholder) organisation, is given the responsibility and resources to co-ordinate the organisation of training, guidance and capacity-building activities for the performance assessment of staff’s (digital) skills and competencies by HEIs.	PROFFORMANCE self-assessment tool
4. Support the establishment of a national centre for teaching and learning	Establish a national centre for teaching and learning, with dedicated responsibility for co-ordinating and developing quality enhancement activities for teaching and learning in higher education, including for digital education (e.g., guidance, training, best practice collection and dissemination).	National centres for teaching and learning in higher education in Germany, Ireland, New Zealand, Norway and the United Kingdom
5. Support the development of an (online) national digital content sharing platform	Building on existing national platforms developed by the OH (the Digital Textbook Library) or the Hungarian Academy of Sciences (Virtual Collaboration Area, Electronic Information Service National Programme), Hungary develops a national digital education content sharing platform for higher education instructors.	Online digital content sharing platforms developed in Austria, France, Ireland, New Zealand, and Norway

Source: Based on a review of international policy and practice included in Staring et al. (2022[1]), “Digital Higher Education: Emerging Quality Standards, Practices and Supports”, OECD Education Working Papers, No. 281, OECD Publishing, Paris, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en.

Strengthen the collection of system-level data and the dissemination of comparable information on the performance of digital higher education

There has been a slow but steady emergence of national student and staff surveys on digital learning across OECD and EHEA systems. Most of these surveys, however, are ad hoc and rarely conducted on a cyclical or regular basis (Staring et al., 2022[1]). Examples of regular national surveys of students’ experiences of digital learning can be found in the United Kingdom, New Zealand, and Australia. First piloted in 2016 and based on the Digital Experience Insights (DEI) survey in Australia and New Zealand (Beetham, Newman and Knight, 2019[60]), Jisc in the United Kingdom has been collecting students’ views on the quality of digital teaching and learning through an annual student survey across all UK HEIs since 2017–18. The latest edition of the survey analyses the responses from 62 658 students in the 2020–21 academic year (Jisc, 2021[61]), which is almost three times more than the 27 069 students who responded in the 2019–20 survey (Jisc, 2020[62]). The 2019 Irish National Digital Experience (INDEx) survey is based on the UK example and was completed by 2 484 students and 4 445 staff at 32 HEIs (National Forum for the Enhancement of Teaching and Learning in Higher Education, 2020[63]). Several HEIs have used analysis of students’ responses to the survey to shape their response to the pandemic, inform the purchase of additional laptops for their laptop rental schemes or develop and disseminate guidance materials for students and instructors (National Forum for the Enhancement of Teaching and Learning in Higher Education, 2021b[64]).

A few systems have adapted administrative and graduate tracking systems to assess the performance of digital and traditional programmes and courses. The United States Integrated Postsecondary Data System (IPEDS) (NCES, 2021[65]) and the New Zealand Single Data Return (SDR) system (New Zealand Ministry of Education, 2021[66]) are examples of systems that provide detailed information on student enrolment, time-to-completion and drop-out rates based on demographic characteristics, level and field of study.

In response to the COVID-19 pandemic, several organisations – including QA agencies – have started to engage in thematic reviews of institutional best practices and challenges for digital teaching and learning. For example, QAA in the United Kingdom has supported HEIs in England and Wales by collecting and disseminating best practices for fully online and remote instruction through a thematic review of HEI practices and challenges in both nations (QAA, 2020[67]; QAA, 2021[68]). In Estonia, too, the national QA body HAKA carried out a thematic review of institutions’ experiences with fully online and remote learning during the COVID-19 pandemic (HAKA, 2020[69]). In Ireland, with the Ministry of Education’s endorsement, a National Institute for Digital Learning (NIDL) was established as a dedicated national centre to strengthen and disseminate scholarship around digital education (Brown and Keogh, 2021[70]). Similarly, in the United Kingdom the Association for Learning Technology aims to raise the profile of research and scholarship in digital teaching and learning to inform best practice through its open access journal Research in Learning Technology (Association for Learning Technology, n.d.[71]). In Norway, the Nordic Institute for Studies, Innovation and Education has published a report on the use of digital technology in higher education (Korseberg et al., 2022[72]), and the Norwegian Ministry of Education and Research also monitors the “digital status” (Digital tilstand) of higher education every two years. So far, five of such reports have been prepared, and the latest was published in 2021 (DIKU, 2021[73]). Until 30 June 2021, this responsibility fell under the Norwegian Agency for International Cooperation and Quality Enhancement in Higher Education. Since then, it has become part of the Directorate for Higher Education and Skills.

Some OECD jurisdictions have invested in the development of websites to provide prospective students with comparable information on the content and quality of the (digital) higher education programmes on offer in their country. One example is the “Study Choice 123” (Studiekeuze 123) platform in the Netherlands. Developed with funding provided by the Dutch Ministry of Education, Culture and Science (OCW), the website provides clear and comparable information for prospective students on the content and learning outcomes of higher education programmes on offer in the Netherlands, including admission and selection criteria, student satisfaction rates (e.g. on the atmosphere, study facilities, and content of the programme), employment outcomes (e.g. information on professional orientation of programmes, feedback from alumni), and enrolment, drop-out and completion rates (e.g. drop-out rates among first-year students, gender balance of student population). The information is linked to data collected in national administrative data registers, as well as national student surveys on the quality of higher education (Studiekeuze123, 2022[74]).

Recommendation 8: Embed digitalisation in existing national data collection and monitoring instruments for higher education

A recommendation for Hungary is to firmly embed digitalisation in national data collection and monitoring instruments for higher education to strengthen the system-level evidence base on digital higher education and inform institutional decision making and pedagogical practice. This can be done in different ways, including: further developing or implementing the list of 30 potential indicators for Hungary, developed by the OECD (OECD, 2021, pp. 96-102[4]) and embedding them in national administrative data systems for higher education; introducing an annual or bi-annual student, staff and/or institutional survey on digital teaching and learning, building on various already existing national and international surveys instruments; or supporting thematic reviews of (digital) pedagogy and institutional QA practice. Table 4.7. presents an overview of the options and potential international models from which Hungary could take inspiration.

Table 4.7. Options for Hungary to monitor and evaluate the quality of digital higher education at system level

Options	Description	Regularity of data collection	Potential model(s) for Hungary
1. Adapt national administrative data collection and information systems to digitalisation	Add a “digitalisation lens” to the Graduate Career Tracking Survey (DPR), Higher Education Database and Information System (FIR) and Felvi.hu student admission and application website, to collect and publish up-to-date data on the performance of digital study programmes. This will not only enhance institutional decision making and pedagogical practice, but also allow prospective students to make more informed study choices.	Ongoing	Further develop or implement the list of 30 potential indicators for Hungary, developed by the OECD (OECD, 2021, pp. 96-102[4]); the Netherlands “Study Choice 123” (Studiekeuze 123) platform
2. Conduct a regular national survey on digital teaching and learning	Building on the ad hoc surveys on digital learning conducted by DOSZ, HÖOK, the DHECC and the OH, Hungary introduces a regular survey of students’ and/or staff’s experience with digital teaching and learning, informed by a careful methodological analysis of the existing survey instruments.	Every 1 or 2 years	Digital Experience Insights (DEI) surveys in Australia, Ireland, New Zealand and the United Kingdom
3. Carry out thematic reviews of digital higher education	Through competitive funding calls, Hungary funds thematic reviews of challenges and best practices in digital higher education in Hungary, focused on specific areas of focus or priority.	Every 3 to 5 years	National centres responsible strengthening scholarship and evidence on digital learning in Ireland and the United Kingdom
4. Carry out thematic reviews of institutional quality assurance	As part of its accreditation reviews, MAB asks experts to collect best practices identified as part of reviews, for dissemination through MAB’s communication channels with the sector.	Every 3 to 5 years	Thematic reviews of digital learning and QA carried out by QA agencies in Estonia and the United Kingdom

Source: Based on a review of international policy and practice included in Staring et al. (2022[1]), “Digital Higher Education: Emerging Quality Standards, Practices and Supports”, OECD Education Working Papers, No. 281, OECD Publishing, Paris, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en.

Foster an institutional culture of self-assessment and continuous improvement for digital higher education

As mentioned in Chapter 2, a wide range of quality frameworks, self-assessment and benchmarking tools has been developed to support the promotion of an institutional culture of continuous quality enhancement and reflection for digital teaching and learning. These tools and frameworks aim to incentivise HEIs to engage in specific self-assessments and/or external reviews of their digital practices, in addition to the (often more general) reviews required by law and carried out by the national QA body.

A selection of 20 frameworks with specific relevance to the European context is included in Volungevičienė et al (2021[75]). They include the DigCompOrg, DigCompEdu and E-xcellence frameworks (Kampylis et al., 2015[76]; Redecker and Punie, 2017[77]; EADTU, 2016[78]), with funding provided by the European Commission, as well as several frameworks developed with national funding. In Germany, for example, the Leibniz Institute for Knowledge Media has developed a Digital Benchmarking Toolkit in collaboration with several German universities for application in the German context (Leibniz Institute for Knowledge Media, 2022[79]). In New Zealand, funding from Ako Aotearoa (via two major grants) and later the Tertiary Education Commission (one grant) has supported the development of the E-Learning Maturity Model, led by experts from HEIs across New Zealand (Marshall, 2012[80]). Another example is the Distance Learning Benchmarking Club, which was established at the initiative of the University of Leicester and involved institutions from Australia, New Zealand, Sweden, Canada, and the United Kingdom. Using the Pick&Mix framework developed by Maltic Media, the institutions identified 17 critical success factors to be used in the benchmarking exercise (Bacsich, 2011[81]).

Recommendation 9: Support and coordinate the development of an institutional self-assessment or benchmarking tool for digital higher education

Hungarian educators would benefit from the creation of a collaborative Working Group that takes responsibility for the development of a self-assessment or benchmarking toolkit for digital higher education, adapted to the specific needs and challenges of the Hungarian higher education sector. The Working Group could consist of both national and international digital education experts and practitioners, and build on existing national and international frameworks and tools developed for the QA of digital higher education (cf. Volungevičienė et al (2021[75])), the ESG (ENQA, 2015[10]), and MAB’s assessment frameworks.

The sector-led collaborative process underpinning this type of initiative has the potential to build a community of practice around digital teaching and learning in Hungary and support the development of institutional quality management policies and practices in line with national and international best practice. Volungevičienė et al. (2021[75]) provide a key lesson in this context (Volungevičienė et al., 2021, p. 5[75]):

Institutions cannot just take one of these instruments off the shelf. Rather, searching for one all-encompassing instrument for DELT [i.e. digitally enhanced leaning and teaching] reflection, self-assessment and capability development would require what Paul Bacsich describes “as a ‘pick n mix’ approach to institutional benchmarking for eLearning” […] This enables the institutions to use and repurpose a range of existing instruments to engage in rich conversations, ask the right questions, identify gaps and areas for development, and collect and assess relevant data against key performance indicators as part of a wider institutional commitment to quality enhancement.

Table 4.8 presents eight common quality domains included in 12 international quality frameworks for digital higher education that feature prominently in the literature, are being used by HEIs, and have been analysed by Staring et al. (2022, pp. 28-29[1]). The quality domains are structured along the “Plan-Do-Check-Adjust” cycle developed by Tague (2005[82]). Table 4.9 provides an overview of self-assessment frameworks, tools and guidance that have been developed with specific reference to the European and Hungarian higher education contexts. Both could be relevant to inform the development of an institutional self-assessment framework for digital higher education in Hungary.

Table 4.8. Eight common quality domains included in 12 institutional self-assessment instruments for digital higher education

Domains for institutional self-assessment of digital higher education
Plan and adjust	Implement	Monitor
Institutional strategy, quality culture and infrastructure for digital teaching and learning	Implementation of quality assurance processes and supports for digital teaching and learning	Feedback and performance monitoring of digital teaching and learning quality
Common quality domains	Common quality domains	Common quality domains
1. Vision, mission and strategy for digitalisation and innovation 2. Organisational quality culture for digitalisation, innovation and collaboration 3. Digital education infrastructure	4. Digital course content, design, delivery and assessment 5. Supporting and incentivising staff professional development 6. Preparing and supporting students for digital learning	7. Monitoring the quality of digital teaching and learning 8. Strengthening feedback and monitoring practices

Source: Based on a review of international policy and practice included in Staring et al. (2022[1]), “Digital Higher Education: Emerging Quality Standards, Practices and Supports”, OECD Education Working Papers, No. 281, OECD Publishing, Paris, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en.

Table 4.9. Selection of self-assessment or benchmarking tool for digital higher education designed for the European and Hungarian higher education context

Quality framework	Focus	Level covered by quality framework
Institution	Programme	Course	Instructor/learner
International tools and frameworks
1. DigCompOrg (Kampylis et al., 2015[76])	All types	✓
2. European Maturity Model for Blended Learning (Goeman, Poelmans and Van Rompaey, 2018[83])	Hybrid education		✓	✓
3. E-xcellence (EADTU, 2016[78])	All types	✓	✓	✓
4. Considerations for the quality assurance of e-learning provision (Huertas et al., 2018[84])	All types	✓
5. DigCompEdu (Redecker and Punie, 2017[77])	All types				✓
Hungarian tools and frameworks
6. Methodological considerations for digital technology in higher education (Dringó-Horváth et al., 2020[15])	All types	✓	✓
7. PROFFORMANCE (PROFFORMANCE, 2022[21])	All types				✓
8. Faculty Distance Education Handbook (Bereczki et al., 2020[16])	Fully online			✓	✓

Source: Based on a review of international policy and practice included in Staring et al. (2022[1]), “Digital Higher Education: Emerging Quality Standards, Practices and Supports”, OECD Education Working Papers, No. 281, OECD Publishing, Paris, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en.

References

[49] Advance HE (2019), UK Professional Standards Framework for Teaching and Supporting Learning in Higher Education, Advance HE, https://www.advance-he.ac.uk/guidance/teaching-and-learning/ukpsf (accessed on 19 December 2022).

[36] APUC (2022), Advanced Procurement for Universities and Colleges, Advanced Procurement for Universities and Colleges, Stirling, https://www.apuc-scot.ac.uk/ (accessed on 19 December 2022).

[71] Association for Learning Technology (n.d.), Research in Learning Technology, https://journal.alt.ac.uk/index.php/rlt/index (accessed on 19 December 2022).

[81] Bacsich, P. (2011), The Distance Learning Benchmarking Club: Final Summary Report, https://www.academia.edu/13029695/The_Distance_Learning_Benchmarking_Club_Final_Summary_Report (accessed on 19 December 2022).

[60] Beetham, H., T. Newman and S. Knight (2019), Digital Experience Insights Survey 2018: Findings from Australian and New Zealand University Students, Jisc, Bristol, https://www.jisc.ac.uk/reports/digital-experience-insights-survey-2018-students-anz (accessed on 19 December 2022).

[16] Bereczki, E. et al. (2020), Faculty Distance Education Handbook, ELTE Faculty of Education and Psychology, Budapest, https://ppk.elte.hu/media/06/d6/d99b696c409129c8bc8aa7f1b26d2acc8a383263224a062fed20d001ee77/Faculty%20Distance%20Education%20Handbook.pdf (accessed on 19 December 2022).

[35] BMBF (2022), OER-Strategie - Freie Bildingsmaterialien für die Entwicklung digitaler Bildung [OER Strategy - Open Educational Resources for the Development of Digital Education], Bildungsminiterium für Bildung und Forschung (Ministry of Education and Research), Bonn, https://www.bmbf.de/SharedDocs/Publikationen/de/bmbf/3/691288_OER-Strategie.html (accessed on 19 December 2022).

[70] Brown, M. and D. Keogh (2021), “Beyond Distance Education Cutting-Edge Perspectives on the Future of Global Open Universities”, in Degang, J. (ed.), .), Beyond distance education: Cutting-edge perspectives on the future of global open universities., Open University of China, Beijing, https://www.london.ac.uk/sites/default/files/cde/beyond-distance-education.pdf (accessed on 19 December 2022).

[40] CARNET (2022), CARNET - Linking knowledge, CARNET, Zagreb, https://www.carnet.hr/en/ (accessed on 19 December 2022).

[44] CAST (2018), The UDL Guidelines, Center for Applied Special Technology (CAST), Wakefield, MA, https://udlguidelines.cast.org/?utm_source=castsite&lutm_medium=web&utm_campaign=none&utm_content=aboutudl (accessed on 19 December 2022).

[57] Coolbear, P. (2014), The eLearning guidelines, Ako Aotearoa, Wellington, http://elg.ac.nz/ (accessed on 19 December 2022).

[47] Department of State Information Systems (2005), Estonian IT Interoperability Framework, Ministry of Economic Affairs and Communications, https://www.digar.ee/arhiiv/en/download/21943 (accessed on 19 December 2022).

[11] Digital Success Programme (2016), Digital Education Strategy of Hungary, Digital Success Programme, Budapest, https://digitalisjoletprogram.hu/files/0a/6b/0a6bfcd72ccbf12c909b329149ae2537.pdf (accessed on 19 December 2022).

[73] DIKU (2021), Digital tilstand 2021 Støttemiljøers arbeid med digitalisering og utdanningskvalitet ved universiteter og høyskoler [Digital state 2021 Support environments’ work with digitization and educational quality at universities and colleges], Directorate for Higher Education and Skills (DIKU), Oslo, https://diku.no/rapporter/digital-tilstand-2021-stoettemiljoeers-arbeid-med-digitalisering-og-utdanningskvalitet-ved-universiteter-og-hoeyskoler (accessed on 19 December 2022).

[15] Dringó-Horváth, I. et al. (2020), Az oktatásinformatika módszertana a felsőoktatásban (Educational Technology in Higher Education – Methodological Considerations), Károli Gáspár Református Egyetem IKT Kutatóközpontja, Budapest, https://btk.kre.hu/images/ikt/oktatasinformatika_a_felsooktatasban.pdf (accessed on 13 July 2022).

[25] DSN/DHECC (2021), An Analysis of Current Higher Education Data Collected in Hungary and the Value of This Data to Assess Digital Readiness and Digital Practices, Digital Success Nonprofit Ltd. (DSN)/Digital Higher Education Competence Centre (DHECC), Budapest, document provided to OECD for the project "Supporting the Digital Transformation of Higher Education in Hungary".

[29] DSN/DHECC (2020), Position Paper on Digitalisation of Hungarian Higher Education, Digital Success Nonprofit Ltd. (DSN)/Digital Higher Education Competence Centre (DHECC), Budapest, document provided to OECD for the project "Supporting the Digital Transformation of Higher Education in Hungary".

[78] EADTU (2016), E-xcellence. Quality Assessment for E-Learning: A Benchmarking Approach., European Association of Distance Teaching Universities (EADTU), Brussels, https://e-xcellencelabel.eadtu.eu/e-xcellence-review/manual (accessed on 19 December 2022).

[34] Ebner, M. et al. (2016), Empfehlungen für die Integration von Open Educational Resources an Hochschulen in Österreich [Recommendation on the integration of Open Educational Resources in Austrian higher education institutions], Forum Neue Medien in der Lehre, https://www.researchgate.net/publication/303298777_Empfehlungen_fur_die_Integration_von_Open_Educational_Resources_an_Hochschulen_in_Osterreich (accessed on 19 December 2022).

[26] Educational Authority (2020), Graduate Career Tracking System, https://www.diplomantul.hu/ (accessed on 19 December 2022).

[14] Educational Authority (n.d.), Digitális Tankönyvtár [Digital Textbook Library], https://dtk.tankonyvtar.hu/ (accessed on 19 December 2022).

[27] Educational Authority (n.d.), Felvi.hu, https://www.felvi.hu/ (accessed on 19 December 2022).

[28] Educational Authority (n.d.), Felvi.hu - E-admission, https://www.felvi.hu/felveteli/efelveteli (accessed on 19 December 2022).

[56] Education, N. (2022), National Resource Hub, National Forum for the Enhancement of Teaching and Learning in Higher Education, Dublin, https://hub.teachingandlearning.ie/ (accessed on 19 December 2022).

[20] EISZ (n.d.), Electronic Information Service National Programme, https://eisz.mtak.hu/index.php/en/ (accessed on 19 December 2022).

[10] ENQA (2015), “ESG”, in Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG), European Association of Quality Assurance in Higher Education (ENQA), Brussels, http://www.enqa.eu/wp-content/uploads/2015/11/ESG_2015.pdf (accessed on 19 December 2022).

[5] EUA (2021a), Hungary: EUA University Autonomy in Europe, European University Association (EUA), Brussels, https://www.university-autonomy.eu/countries/hungary/ (accessed on 19 December 2022).

[85] European Commission (2022), GÉANT Partnership projects (b), https://cordis.europa.eu/programme/id/H2020_GEANT-2018-b (accessed on 11 August 2022).

[22] European Commission (2022), Work starts on the Digital Education Hub Community of Practice, https://education.ec.europa.eu/news/work-starts-on-the-digital-education-hub-community-of-practice (accessed on 19 December 2022).

[23] European Commission (2020), European Universities Initiative, https://ec.europa.eu/education/education-in-the-eu/european-education-area/european-universities-initiative_en (accessed on 19 December 2022).

[17] FETA (n.d.), Hungarian Association for Counselling in Higher Education, https://www.feta.hu/en (accessed on 19 December 2022).

[58] Flexible Education Norway (2018), A Guide to Quality in Online Teaching and Learning FLEXIBLE EDUCATION NORWAY, Flexible Education Norway, Oslo, https://www.eadl.org/wp-content/uploads/2020/03/A-guide-to-quality-in-online-teaching-and-learning-for-digital-sharing-Norway-Nov-2018.pdf (accessed on 19 December 2022).

[45] Foerster, M. (2019), D4.7 – Framework of e-assessment, CORDIS, Brussels, https://cordis.europa.eu/project/id/688520/results (accessed on 19 December 2022).

[55] FUN (2022), France Université Numérique (FUN), https://www.fun-mooc.fr/fr/ (accessed on 19 December 2022).

[83] Goeman, K., S. Poelmans and V. Van Rompaey (2018), Research report on state of the art in blended learning and innovation. European Maturity Model for Blended Education (EMBED), The Netherlands: European Association of Distance Teaching Universities (EADTU), Maastricht, https://www.researchgate.net/project/EMBED-European-Maturity-model-for-Blended-Education-EMBED (accessed on 19 December 2022).

[3] Government of Hungary (2011), Act CCIV of 2011 on National Higher Education, Government of Hungary, Budapest, https://net.jogtar.hu/jogszabaly?docid=A1100204.TV (accessed on 19 December 2022).

[69] HAKA (2020), How did higher education institutions cope with the forced distance learning, Estonian Quality Agency for Vocational and Higher Education (HAKA), Tallinn, https://ekka.edu.ee/wp-content/uploads/Digio%CC%83ppe-analu%CC%88u%CC%88s-Final-EN-8.1.21.pdf (accessed on 19 December 2022).

[24] HÖOK (2020), Távoktatási jelentés [E-learning report], National Union of Students’ in Hungary (HÖOK), Budapest, https://hook.hu/hu/felsooktatas/tavoktatas-jelentes-2851 (accessed on 19 December 2022).

[84] Huertas, E. et al. (2018), Considerations for quality assurance of e-learning provision. Report from the ENQA Working Group VIII on Quality Assurance and E-Learning., European Association for Quality Assurance in Higher Education (ENQA), Brussels, https://www.enqa.eu/wp-content/uploads/Considerations-for-QA-of-e-learning-provision.pdf (accessed on 19 December 2022).

[54] iMoox (2022), iMooX, https://imoox.at/mooc/?lang=en (accessed on 19 December 2022).

[61] Jisc (2021), Learner digital experience insights survey 2020/21 - UK further education (FE) survey findings, Jisc, September 2021, https://repository.jisc.ac.uk/8488/1/Student%20DEI%20FE%20report%202021%20final.pdf (accessed on 19 December 2022).

[46] Jisc (2020), Senior managers’ guide to learning analytics, Jisc, https://repository.jisc.ac.uk/7782/1/senior-managers-guide-to-learning-analytics.pdf (accessed on 19 December 2022).

[62] Jisc (2020), Student Digital Experience Insights Survey 2020: Question by Question Analysis of Findings from Students in UK Further and Higher Education, Jisc, Bristol, https://www.jisc.ac.uk/sites/default/files/dei-2020-student-survey-question-by-question-analysis.pdf (accessed on 19 December 2022).

[13] Kálmán, O., P. Tynjälä and T. Skaniakos (2020), “Patterns of university teachers’ approaches to teaching, professional development and perceived departmental cultures”, Teaching in Higher Education, Vol. 25/5, pp. 595-614, https://doi.org/10.1080/13562517.2019.1586667 (accessed on 19 December 2022).

[76] Kampylis, P. et al. (2015), Promoting effective digital-age learning: a European framework for digitally-competent educational organisations, Joint Research Centre (JRC), Brussels, https://joint-research-centre.ec.europa.eu/european-framework-digitally-competent-educational-organisations-digcomporg_en (accessed on 19 December 2022).

[12] KIFÜ (2021), Governmental Agency for IT Development (KIFÜ), https://kifu.gov.hu/en/ (accessed on 19 December 2022).

[8] KIM (2020), National Digitalisation Strategy 2021-2031, Minsitry of Culture and Innovation (KIM), Budapest, https://2015-2019.kormany.hu/download/f/58/d1000/NDS.pdf (accessed on 19 December 2022).

[7] KIM (2020), Support for the Digital Transformation of Hungarian Higher Education - Stakeholder Roundtable, YouTube, https://www.youtube.com/watch?v=o79Hd_J3Fs8 (accessed on 19 December 2022).

[6] KIM (2016), Shifting of Gears in Higher Education Policy Strategy Action Plan, Ministry of Culture and Innovation (KIM), Budapest, https://www.niif.hu/sites/default/files/niif_program_strategia_20180124v1_short.pdf (accessed on 19 December 2022).

[18] Kiss, M. et al. (2020), Tips for coping with stress and anxiety during the COVID-19 pandemic, Hungarian Association for Counselling in Higher Education (FETA), Budapest, https://www.feta.hu/sites/default/files/FETA%20F%C3%BCzetek_COVID_Tips%20for%20coping%20with%20stress%20and%20anxiety%20during%20the%20COVID-19%20pandemic.pdf (accessed on 19 December 2022).

[32] Kiss, V. (2022), “QR code system helps student feedback on teaching at Semmelweis University”, Semmelweis News, https://semmelweis.hu/english/2022/01/qr-code-system-helps-student-feedback-on-teaching-at-semmelweis-university/ (accessed on 19 December 2022).

[72] Korseberg, L. et al. (2022), Pedagogisk bruk av digital teknologi i høyere utdanning [Pedagogical use of digital technology in higher education], Nordisk institutt for studier av innovasjon, forskning og utdanning (NIFU), https://www.nifu.no/publications/1984117/ (accessed on 19 December 2022).

[79] Leibniz Institute for Knowledge Media (2022), HFD Strategie-Benchmark - Home, https://benchmark.hfd.digital/ (accessed on 19 December 2022).

[39] LieDM (2022), LieDM consortium, LieDM consortium, Vilnius, https://liedm.lt/ (accessed on 19 December 2022).

[80] Marshall, S. (2012), “E-learning and Higher Education: Understanding and Supporting Organisational Change in New Zealand”, Journal of Open, Flexible and Distance Learning, Vol. 16/1, pp. 141-155, https://files.eric.ed.gov/fulltext/EJ1079905.pdf (accessed on 19 December 2022).

[9] Mohácsi, J. (2018), Digitális Felsőoktatási, Kutatási és Közgyűjteményi Infrastruktúra- fejlesztési Stratégia [Digital Higher Education, Research and Public Library Infrastructure Development Strategy], KIFÜ, https://www.niif.hu/sites/default/files/niif_program_strategia_20180124v1_short.pdf (accessed on 19 December 2022).

[63] National Forum for the Enhancement of Teaching and Learning in Higher Education (2020), INDEx Findings from Students and Staff Who Teach in Higher Education, National Forum for the Enhancement of Teaching and Learning in Higher Education, Dublin, https://www.teachingandlearning.ie/index/ (accessed on 19 December 2022).

[50] National Forum for the Enhancement of Teaching and Learning in Higher Education (2016), National Professional Development Framework for all Staff Who Teach in Higher Education, National Forum for the Enhancement of Teaching and Learning in Higher Education, Dublin, https://hub.teachingandlearning.ie/resource/national-professional-development-framework-for-all-staff-who-teach-in-higher-education/ (accessed on 24 October 2022).

[64] National Forum for the Enhancement of Teaching and Learning in Higher Education (2021b), INDEx survey: final summary report, National Forum for the Enhancement of Teaching and Learning in Higher Education, Dublin, https://hub.teachingandlearning.ie/resource/index-survey-final-summary-report/ (accessed on 19 December 2022).

[65] NCES (2021), 2020-2021 Data Collection System, https://surveys.nces.ed.gov/IPEDS/ (accessed on 19 December 2022).

[66] New Zealand Ministry of Education (2021), National Student Index (NSI) Web Application, https://applications.education.govt.nz/national-student-index-nsi-web-application (accessed on 19 December 2022).

[4] OECD (2021), Supporting the Digital Transformation of Higher Education in Hungary, Higher Education, OECD Publishing, Paris, https://doi.org/10.1787/d30ab43f-en (accessed on 19 December 2022).

[33] OECD; European Commission; DGES (2022), LMRO Partnership Initiative Enhancing Labour Market Relevance and Outcomes of Higher Education, OECD, Paris, https://www.wpz-research.com/wp-content/uploads/2022/05/Improve_teach-learn_Sem5Brochure.pdf (accessed on 19 December 2022).

[42] OpenupEd (2022), MOOC Quality Assurance Checklist, European Association of Distance Teaching Universities (EADTU), Maastricht, https://www.openuped.eu/quality-label/223-mooc-checklists (accessed on 19 December 2022).

[21] PROFFORMANCE (2022), PROFFORMANCE - Assessment Tool and Incentive Systems for Developing Higher Education Teachers’ Performance, https://profformance.eu/ (accessed on 19 December 2022).

[68] QAA (2021), Review of Digital Learning (Wales), Quality Assurance Agency for Higher Education (QAA), Gloucester, http://www.qaa.ac.uk/docs/qaa/guidance/building-a-taxonomy-for-digital-learning.pdf (accessed on 19 December 2022).

[67] QAA (2020), How UK Higher Education Providers Managed the Shift to Digital Delivery During the COVID-19 Pandemic, Quality Assurance Agency for Higher Education (QAA), Gloucester, https://www.qaa.ac.uk/docs/qaa/guidance/how-uk-higher-education-providers-managed-the-shift-to-digital-delivery-during-the-covid-19-pandemic.pdf (accessed on 19 December 2022).

[77] Redecker, C. and Y. Punie (2017), European Framework for the Digital Competence of Educators. DigCompEdu., Publications Office of the European Union, Luxembourg, https://publications.jrc.ec.europa.eu/repository/handle/JRC107466 (accessed on 19 December 2022).

[41] Sikt (n.d.), Norwegian Agency for Shared Services in Education and Research, https://sikt.no/about-sikt (accessed on 19 December 2022).

[1] Staring, F. et al. (2022), “Digital Higher Education: Emerging Standards, Practices, and Supports”, OECD Education Working Papers , No. 281, OECD Publishing, Paris, p. 97, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en (accessed on 19 December 2022).

[59] Stiftung Innovation in der Hochschullehre (n.d.), Geförderte Projekte - Hochschullehre durch Digitalisierung stärken [Funded projects - Strengthening university teaching through digitalisation], https://stiftung-hochschullehre.de/foerderung/hochschullehre-durch-digitalisierung-staerken/gefoerderte-projekte/ (accessed on 19 December 2022).

[74] Studiekeuze123 (2022), Studiekeuze123 [Study Choice 123], https://www.studiekeuze123.nl/ (accessed on 19 December 2020).

[38] SURF (2022), SURF is the collaborative organisation for IT in Dutch education and research, SURF, Utrecht, https://www.surf.nl/en (accessed on 19 December 2022).

[48] SURF (2021), Aandachtspunten voor een veilige en betrouwbare infrastructuur voor Learning Analytics en Studiedata [Attention points for safe and reliable infrastructure for Learning Analytics and Study Data], SURF, https://www.surf.nl/files/2021-05/surf-aandachtspunten-voor-een-veilige-en-betrouwbare-infrastructuur-voor-learning-analytics-en-studiedata.pdf (accessed on 19 December 2022).

[43] SURF (n.d.), Overzicht methodieken en tools voor blended onderwijs [Overview of methods and tools for blended education], https://www.surf.nl/overzicht-methodieken-en-tools-voor-blended-onderwijs (accessed on 19 December 2022).

[82] Tague, N. (2005), “Plan–Do–Study–Act cycle”, in The quality toolbox, ASQ Quality Press, Milwaukee Wis., https://www.worldcat.org/title/quality-toolbox/oclc/57251077 (accessed on 19 December 2022).

[31] Tolnai, Á. (2021), Quality assurance issues of blended learning courses, J. Selye University, Komárno, Slovakia, pp. 171-177, https://doi.org/10.36007/4133.2022.171 (accessed on 19 December 2022).

[37] UKUPC (2022), UK Universities Purchasing Consortia, UK Universities Purchasing Consortia, London, https://www.ukupc.ac.uk/ (accessed on 19 December 2022).

[51] Universities Denmark (2021), Danish framework for advancing university pedagogy, Universities Denmark, https://dkuni.dk/wp-content/uploads/2021/03/danish-framework-for-advancing-university-pedagogy-1.pdf (accessed on 19 December 2022).

[30] University of Debrecen (2020), Online felmérés az online oktatásról [Online research about online education], University of Debrecen (DE), Debrecen, https://btk.unideb.hu/hu/online-felmeres-az-online-oktatasrol (accessed on 19 December 2022).

[19] VirCA (n.d.), Virtual Collaboration Area, http://www.virca.hu/ (accessed on 19 December 2022).

[75] Volungevičienė, A. et al. (2021), Developing a High-Performance Digital Education Ecosystem: Institutional Self-Assessment Instruments, European University Association (EUA), Brussels, https://eua.eu/downloads/publications/digi-he%20desk%20research%20report.pdf (accessed on 19 December 2022).

[53] VSNU (2018), Professionalisation of university lecturers The UTQ and beyond, VSNU, May 2018, https://www.universiteitenvannederland.nl/bko.html (accessed on 19 December 2022).

[52] VSNU (2008), Overeenkomst inzake wederzijdse erkenning basiskwalificatie onderwijs [Agreement on mutual recognition of basic qualification], VSNU, January 2008, https://www.universiteitenvannederland.nl/files/documenten/BKO-overeenkomst%202018%20BKO%20overeenkomst.pdf (accessed on 19 December 2022).

[2] Zhang, T. (2022), National Developments in Learning and Teaching in Europe, European University Association asbl, Brussels, https://eua.eu/downloads/publications/lotus%20report_2022_fin2.pdf (accessed on 19 December 2022).

Notes

← 1.  NRENs are a specialised internet service providers dedicated to supporting the needs of the research and education communities within a country. While they are known to support a high-speed backbone network, they also have a mandate to provide seamless and secure access to digital education resource (European Commission, 2022[85]).

← 2.  Austria, Denmark, Estonia, Finland, Flanders, Hesse, Iceland, Lithuania, Luxembourg, The Netherlands, North Rhine-Westphalia, Norway, Serbia, Slovakia, Slovenia, Sweden, Switzerland, United Kingdom.

← 3.  Austria, Brandenburg, Estonia, Finland, Flanders, Wallonia, Hesse, Hungary, Iceland, Latvia, Lithuania, Luxembourg, The Netherlands, North Rhine-Westphalia, Norway, Poland, Slovakia, Slovenia, Sweden, Switzerland, United Kingdom.

← 4.  Denmark, Estonia, France, Georgia, Kazakhstan, Latvia, Lithuania, and Norway.