This chapter proposes a framework for the assessment of digital maturity in Croatia’s higher education institutions, comprising three main elements – digital leadership, digital infrastructure and digital competence and culture. It then provides quantitative and qualitative assessment of each element, based on data from a survey of Croatian higher education institutions and interviews with institution leaders, staff and students.
Advancing Digital Maturity in Croatia’s Higher Education System
Higher Education
3. Digital maturity in Croatian higher education institutions
Abstract
Introduction
The concept of digital maturity, as used in this project, refers to the extent of development of digitalisation at the organisation level – in this case, individual higher education institutions. Evaluating the digital maturity of an organisation is a complex, multidimensional process. Digital maturity may be influenced by both external and internal factors, including digital readiness in the system in which the institution operates and its internal capacity for effective integration of digital technologies.
Much of the previous development of digital maturity frameworks was carried out with business organisations in mind or was focused on the development of specific organisation-wide tools such as management information systems (Proença and Borbinha, 2018[1]). Still, many frameworks, whether for commercial organisations or higher education institutions, focus on similar thematic areas, encompassing both organisation-level and individual-level factors that might drive or enable digital transformation. These factors include strategic leadership, governance, technology adoption, culture and expertise. (Ifenthaler and Egloffstein, 2020[2]; Rossman, 2018[3]).
Digital maturity measurement frameworks have been proposed specifically for education organisations, although there have been few attempts to measure digital maturity of higher education institutions systematically. Croatia is one of the more experienced jurisdictions with respect to attempting to measure digital maturity directly, having developed a framework for evaluating schools’ digital maturity in its e-Schools project. The project entailed triangulating internal and external school evaluations to arrive at a “starting level” of maturity for each school, and then aimed for each school to advance by at least one level (Box 3.1). Many of the indicators used in the e-Schools Digital Maturity Framework found to be significant predictors of digital maturity are also applicable to higher education institutions (Table 3.1).
Table 3.1. Significant predictors of digital maturity in the e-Schools framework
Notes: The instrument evaluation focused on the digital maturity level of 151 primary and secondary schools in Croatia. 21 significant indicators, found to directly influence the maturity level of a school, are included above.
Source: Balaban, Redjep and Calopa (2018[4]) The Analysis of Digital Maturity of Schools in Croatia; https://doi.org/10.3991/ijet.v13i06.7844.
Box 3.1. The e-Schools project in Croatia
At the school level, the e-Schools programme is Croatia’s flagship example of bolstering digital maturity. Launched over two phases in 2015 and 2018, the programme equipped school campuses with reliable and modern infrastructure, delivered up-to-date ICT devices to classrooms for teaching and learning, and provided e-content and digital platforms for schools, as well as training and support for teachers. The project was co‑ordinated by CARNET and funded by European Structural Funds (ESF), European Regional Development Funds (ERDF) and the state.
Digitally mature schools were defined as schools where ICT integration was high and a systematic approach is taken towards ICT in school management and educational processes. Teachers in digitally mature schools were better able to develop their own digital content, to use technology effectively to enhance teaching, and support independent learning and critical thinking skills in their students. Evidence shows that digital education initiatives have more success with teachers’ buy-in, participation, engagement and eventually ownership of the process (Conrads et al., 2017[5]).
Measurement was carried out using a Framework for Digital Maturity of Schools (FDMS), designed to identify the level of digital maturity of schools in Croatia, their progress towards integration and efficient use of ICT and areas for improvement. Its creators reviewed 15 existing maturity frameworks and selected the European Framework for Digitally Competent Educational Organisations (DigCompOrg) and eLearning Roadmaps as the foundation for the construction of the FDMS rubric. The rubric, a “maturity matrix”, has five evaluation domains: 1. Planning, Management and Leadership; 2. ICT in Learning and Teaching; 3. Development of Digital Competence; 4. ICT Culture; 5. ICT Infrastructure, with 38 individual elements assessed across five maturity levels (basic, initial, e-enabled, e-Confident and e-Mature).
The matrix, along with the self-evaluation responses and software, evaluated the digital maturity level of 151 primary and secondary schools in Croatia. To triangulate results, an external evaluation was also conducted. This revealed a visible gap between the self-evaluation and external findings, with more schools identifying themselves as further along in their digital maturity level (e.g. e-Confident or e‑enabled) than was found in the external evaluation. The largest deviation was in Domain 1, Planning, Management and Leadership. The adaptation of strategic documents and upgrading schools’ documents with ICT strategy is considered to positively affect digital maturity.
Source: Balaban, Redjep and Calopa (2018[4]) , Analysis of Digital Maturity of Schools in Croatia; https://doi.org/10.3991/ijet.v13i06.7844.
The e-Schools approach to evaluating digital maturity may not translate directly to use in higher education institutions for several reasons. Firstly, the e-Schools framework was designed for schools and teachers, and as a result many of its indicators are heavily focused on the application of ICT in teaching. Higher education institutions have a much wider mission than schools, encompassing not only teaching but also research, innovative and entrepreneurial activity, the provision of wraparound services to students, and serving the wider population and local community. From the perspective of higher education institutions, digital maturity frameworks need to permit holistic evaluation and reflection across all their activities.
Higher education institutions also have more complicated leadership and organisational structures than schools, with roles and responsibilities for digitalisation spread across individual faculties and departments. As a result, self-evaluation (and improvement) of digital maturity in higher education institutions is likely to require much greater levels of co‑ordination among personnel than it does in schools. Moreover, unlike with e-Schools, most higher education institutions will not enter a maturity evaluation process as “digital beginners”. Many higher education institutions in Croatia have a long history of independent development of digitalisation strategies and processes to increase digital maturity, tailored to their specific needs and circumstances (Box 3.2).
Finally, it is difficult to apply the concept of “levels” of maturity coherently across an entire higher education institution, given the wide range of staff, and student categories with varied digital competence, while the extent of leadership and infrastructure may vary across departments and activities. One-size-fits-all approaches to measuring digital maturity are unlikely to bear similar fruit in higher education institutions as they do in schools. For example, subject-specific e-content repositories would have limited utility for higher education institutions compared with schools, and the development of standardised infrastructure allocation models that cover diverse institution circumstances is more challenging than for schools.
Box 3.2. Examples of previous higher education institution strategies for digitalisation in Croatia
A 2016 survey administrated to 119 Croatian higher education institutions revealed that 84% of survey respondents did not have a dedicated fund for applying e-learning and developing e-learning resources.
At the same time, many higher education institutions across Croatia have developed initiatives aimed at improving various aspects of digital maturity.
The University of Zagreb introduced a strategic initiative (the Zagreb E-learning strategy for 2007-2010) to strengthen e-learning. The university e-learning committee evaluated the program in 2012 and subsequently incorporated it into the university strategy for teaching and learning 2014-25. The initiative establishes a National Centre for e-Learning in Higher Education, a university committee for E-learning, and the University Office for E-learning (Bralić, 2016[6]; University of Zagreb, 2021[7]).
The University of Rijeka implemented a multi-year strategy, Strategy for the introduction of e-learning, from 2006-10 and 2011-15 and founded a centre for e-learning in 2009. Wave 1 of the strategy built a foundation for increasing delivery of e-learning content and supported teachers in moving from traditional to active learning methodologies. Wave 2 continued to build on high-quality e-learning, supported students in these endeavours and developed distance learning programs. The institution also created the E-day of the UNIRI, established annual awards for the best e-courses and provided grants for e-course development.
The University of Split also created a multi-year strategic plan to boost e-learning content in courses by 25% and to work towards a university-wide joint e-learning system. Similarly, the University of Osijek’s 2011-20 plan focused primarily on building a joint information and communication system on campus.
Source: Adapted from Bralić (2016[6]), ICT and e-learning in HE in Croatia: strategies and current state, https://www.bib.irb.hr/846915.
One of the most detailed and comprehensive framework development processes available, specific to higher education, was carried by Durek et al (2018[8]). Based on an exhaustive review of existing frameworks, the researchers concluded that, as well as a systematic approach towards the use of technology in teaching and learning, a digital maturity framework for higher education institutions should also include other dimensions such as leadership, planning and management; quality assurance; scientific research work; technology transfer and service to society, ICT culture and ICT resources and infrastructure.
Durek et. al. provides a comprehensive set of indicators for assessing digital maturity that builds on previous frameworks in a holistic way and reflects the diverse set of activities carried out in higher education institutions (Table 3.2).
Table 3.2. Digital Maturity Framework for Higher Education Institutions (DMFHEI) indicators
Source: Durek, Kadoic and Begicevic Redep (2018[8]), Assessing the digital maturity level of higher education institutions, https://ieeexplore.ieee.org/document/8400126.
Despite the completeness of the framework and the rigorous approach taken to its development, several important limitations arise with its application. Most of the proposed indicators cannot be measured in a structured way by extant data systems. A demanding and expensive data collection workload would be required to collect all data required for the evaluation framework. The model developers envisaged that the involvement of field experts would be necessary to collect and analyse data and make subsequent judgements about the levels of maturity (starting with ‘basic, followed by ‘initial’, ‘e-Enabled’, ‘e-Confident’ and ‘e-Mature’) indicated by each element of the framework. The selection process for field experts and how they would be funded and governed in their operations adds additional complexity to the measurement process. In addition, the model developers acknowledge that further validation of the assessment rubric would be needed (Durek, Kadoic and Begicevic Redep, 2018[8]).
Taking the considerations above into account, one can conclude that a framework for digital maturity for higher education institutions in Croatia and elsewhere needs to be more comprehensive than for schools. At the same time, the framework cannot be too prescriptive or excessively burdensome for institutions and stakeholders working with it. Considering these factors, the following section proposes a model to evaluate digital maturity in Croatian higher education institutions.
A framework for evaluating digital maturity of higher education institutions
Recent OECD research identifies several enabling factors for supporting high-quality digitalised education. These encompass adequate infrastructure and equipment, strong leadership, a digitally competent and capable human infrastructure, and capacity for monitoring and evaluation (OECD, forthcoming[9]). Similarly, the DigiCompOrg framework for digitally competent educational organisations stresses the importance of human capacities and organisational culture, as well as access to appropriate and sufficient digital technologies. Taking these results, and various other available digital maturity approaches proposed for Croatia into account (as detailed in the previous section), Table 3.3 proposes a simple conceptual framework for digital maturity evaluation of higher education institutions, summarised also in Figure 3.1.
Table 3.3. A conceptual framework for digital maturity evaluation of higher education institutions
Source: Author’s elaboration.
The framework identifies three key enabling elements – digital leadership, digital infrastructure, and digital competence and culture. The stronger the presence of each of the enabling factors in the framework, the more likely the organisation will advance towards full digital maturity. Each of the three constructs in the proposed digital maturity framework can be measured in many ways. Specific evaluation and measurement strategies will depend on several factors, including whether the maturity framework is being used for accountability purposes or formative assessment and development; the level of available resources for data collection and assessment, and future national and institutional goals and priorities related to digital maturity.
Figure 3.1. A conceptual framework for digital maturity evaluation and improvement
Source: Author’s elaboration.
Trade-offs must also be made between what is ideal and what is possible – balancing the effort required for measurement with the value of the information provided. Evaluation measures for the framework should be formulated considering information that may already be available and the ease of collecting additional data. Scarce resources are best focused on measuring areas where there is the greatest uncertainty, and where the measurement is crucial to the success of an initiative.
Assessments of digital maturity using the framework can be carried out on a qualitative basis by defining specific indicators and approaches that aim to measure important attributes of each enabling element. The framework can also serve as a useful inception point for qualitative evaluation and as a basis for conversations about needed improvements (at institution or national level). The following sections apply the framework to conduct both a quantitative and qualitative evaluation of digital maturity in Croatia’s higher education institutions, drawing on a digital maturity survey conducted as a joint exercise between the OECD and CARNET in February 2022, and interviews of higher education institution staff and students carried out in Croatia by the OECD team during a country visit in May 2022.
Quantitative assessment of digital maturity in Croatian higher education institutions
As part of the activities of this project the OECD team, in collaboration with CARNET, launched a digital maturity survey of higher education institutions. The intention of the survey was to collect a baseline set of data on institution activities, internal policies and practices related to digital transformation. The CARNET-OECD survey was fielded in February 2022 and received 95 responses from public and private institutions (Table 3.4), together covering about 77% of student enrolments in Croatia in 2020/21.
Table 3.4. Respondents to the 2022 CARNET-OECD survey, by type of institution.
Source: CARNET-OECD survey of digital maturity of higher education institutions.
The items covered in the survey included many variables that could be used to measure the elements of digital maturity, as laid out in the framework in Figure 3.1. Table 3.5 proposes a set of ten binary indicators that can be populated from the CARNET-OECD survey data for each of the three elements of digital maturity proposed in the conceptual framework. The indicators proposed are similar to many of those appearing in other frameworks for benchmarking and measuring the extent and impact of digitalisation in education, including the Australasian Council on Open, Distance and e-Learning framework for Technology-Enhanced Teaching and Learning, the E-xcellence framework of the European Association of Distance Teaching Universities and the Commonwealth of Learning e‑learning benchmarks (DIGI-HE, 2021[10]). They are, however, presented only as an example and a starting point for collaborative deliberation and potential development of national indicators of digital maturity.
Table 3.5. Example indicators of digital maturity using the CARNET-OECD survey data
Notes: The indicators are provided as an example and starting point only. Agreement on a final set of indicators of digital maturity for Croatian higher education institutions is best progressed as a national collaborative exercise.
Aggregation of the binary indicators for each element of the framework into a score shows that, on average, institutions appear to have the highest levels of maturity in the digital competence and culture element (average indicator score of 5.3 out of 10) and the lowest rates of maturity on digital infrastructure (average indicator score of 4.5 out of 10). Few institutions achieved an indicator score greater than 7 out of 10 on the maturity measure for digital infrastructure (Figure 3.2).
Figure 3.2. Digital maturity indicator scores for each element of the maturity framework
Source: OECD analysis of the 2022 CARNET-OECD survey data. Based on survey responses from 95 higher education institutions.
When data for each of the three elements of the framework are combined into an overall measure of digital maturity (out of a total possible score of 30), most institutions exhibit scores in the range between 12 to 18 (Figure 3.3). Slightly fewer institutions reach the highest levels of maturity according to this measurement than achieve the lowest scores: in total 14 institutions exhibit a combined indicator score of greater than 21, while 17 institutions show a combined indicator score of less than 9.
Figure 3.3. Distribution of digital maturity scores in Croatian higher education institutions
Source: OECD analysis of the 2022 CARNET-OECD survey data. Based on survey responses from 95 higher education institutions.
These results serve only as a very preliminary indication of digital maturity in Croatian higher education institutions. More comprehensive validation of the proposed indicators and underlying data as appropriate measures of digital maturity would be required before integration of any quantitative framework into national and institutional deliberations on digitalisation. Nevertheless, the results provide a crude indication that, while some institutions have well-established digital leadership, high-quality infrastructure and an embedded digital culture, many other institutions have made few steps towards digital transformation of their activities. The detail provided in the CARNET-OECD survey can allow for more comprehensive institution-level comparisons to be carried out, if required, to support conversations about future strategies for digital transformation.
Qualitative assessment of digital maturity in Croatian higher education institutions
The final diagnostic element of this report is a qualitative evaluation of the digital maturity of Croatian higher education institutions, organised according to the framework proposed above. It presents the key findings of the OECD review team related to digital maturity in Croatia, based on interviews and discussions the OECD conducted with institution staff and other stakeholders in Croatia, and illustrated in some cases by aggregate results from the CARNET-OECD survey.
Digital leadership
Digital leadership in higher education institutions is defined as an ability to coherently develop and co‑ordinate organisation-level digitalisation strategy and mobilise resources for its implementation. It can also be indicated by robust institution-level strategies to monitor and improve the quality of its digital operations. Evidence from the CARNET-OECD survey and from interviews with institution staff and students is summarised in the following key messages about digital leadership in Croatian higher education institutions.
Institution leaders view digitalisation as a concept that goes far beyond online education
In Croatia, few fully online programs have been accredited, and many stakeholders reported to the OECD team that appetite for fully online higher education in Croatia is limited, especially after the experience of emergency remote instruction. Some interviewees noted that during the emergency remote period, much of the online delivery was made up of uneven ad hoc solutions which are unsustainable in the long run. Going forward, public authorities have a role to play in regularising the current situation with online and hybrid programs and reviewing national criteria for digitally delivered education.
OECD interviews with higher education institution leaders also indicated that most consider online tools and content as a valuable complement to face-to-face teaching, rather than as a primary objective. There is a growing desire among students to have access to online learning material in addition to face-to-face instruction to support, reinforce and, at times, replace their on-campus lectures. Some institutions have maintained online access to digital learning content developed during the pandemic, but differences are evident among institution leaders in terms of the extent to which they are willing and able to provide such complementary digital content in the future.
It is also clear that institution leaders in Croatia do not view digitalisation simply in the context of education provision – there is an imperative to integrate digital tools and technologies into all the activities of higher education institutions. Specific requirements mentioned multiple times include the need for digital tools and software to streamline administrative processes, a need for wider access to digitalised content (including e-books, research articles and databases, qualification and occupational standards), and requirements for specialised software licences and high-performance computing needed in many fields of study.
Stakeholder involvement in strategies for digitalisation appears limited.
Data from the CARNET-OECD survey of digital maturity in higher education institutions showed that more than half (53%) of responding institutions have a digital strategy in place, either as a standalone strategy or (more commonly) as part of an institution-wide general strategy. However, the results also show that, in most cases, important stakeholder groups affected by digitalisation were not consulted during the development process (Figure 3.4). Students were the least consulted of any group, while teachers were only consulted in the development of approximately 60% of institution strategies. Leaving students and staff out of the consultation process creates the risk that institution leaders develop strategies irrelevant to the actual needs or previous experiences of staff and learners.
The results indicate that more efforts may be needed to embed coherent institution-level digital leadership across Croatian higher education institutions. In OECD site visits, some interviewees acknowledged a need to improve leadership and strategic co‑ordination. Within independent constituents of non-integrated universities, for example, it was noted that change efforts at the faculty level are often not sufficient but must rather be embedded across the entire university structure. However, even within an integrated university, strategic approaches to digitalisation can vary across departments and specialisms. It was also noted in interviews that an overly narrow view of digital strategy is sometimes taken both at central and institutional level, with a focus mainly on providing equipment rather than a more holistic capacity-building approach.
Figure 3.4. Development of institutional strategies for digitalisation in Croatian institutions
Source: 2022 CARNET-OECD survey of digital maturity of 95 higher education institutions.
A lack of public investment is considered a key barrier to successful digitalisation
In the CARNET-OECD survey of digital maturity, a lack of access to public funds for digitalisation was identified as the greater barrier to digitalisation, followed by limited physical infrastructure and a lack of available staff to support digital operations (Figure 3.5). In interviews, many institution leaders and senior staff expressed gratitude and appreciation for the range of services provided by CARNET and other bodies such as SRCE, but also noted that available public funding was not sufficient to meet national needs. Some interviewees also perceived a greater tendency to fund digital projects in STEM fields, even though there are emerging needs for access to digital technologies in all fields.
Figure 3.5. Barriers to digitalisation identified by respondents to the CARNET-OECD survey
Source: 2022 CARNET-OECD survey of digital maturity of 95 higher education institutions.
Capacity for strategy development can also vary according to the resources available to the institution (Figure 3.6). Stakeholder interviews with the OECD team highlighted that Croatia’s polytechnics in general are smaller than universities and tend to have fewer resources available for digital transformation. In addition, many institutions’ current digital development strategies were produced before the pandemic and do not reflect the latest developments.
Figure 3.6. Sources of funding available to institutions for digital transformation
Source: 2022 CARNET-OECD survey of digital maturity of 95 higher education institutions.
Institutions are not fully dependent on public sources for investment in digitalisation. In fact, institutions are most likely to fund digital tools from their own budgets rather than rely on public funds (Figure 3.6). At the same time, 15 survey respondents, almost all of which were faculties of non-integrated universities, reported that centralised public funding was their only source of funds for digitalisation.
There is a lack of clarity about responsibility for digital transformation in some institutions.
Data from the CARNET-OECD digital maturity survey highlights disparities in the way that institutional responsibility is allocated for digital transformation. In total, about half the respondents could not highlight a specific appointed person or unit within their institution who has responsibility for digital transformation, while approximately one-third of respondents reported that responsibility for decisions lay at overall institution level (Figure 3.7). On the other hand, decisions about the acquisition of specific tools and technologies are most often made at the faculty level (Figure 3.8). Taken together, these results indicate potential fragmentation in institution decision making and responsibilities in some institutions, which may create inefficiencies and limit knowledge flows within the organisation.
This conclusion is supported, to some extent, by evidence gathered during the OECD team’s site visits. Some stakeholders felt institutions themselves could take on greater leadership and responsibility for their transformation efforts by playing a greater role in encouraging cultural change within their institution. On the other hand, some institutions and their representative bodies reported the need for a stronger legislative backbone for digital education and were awaiting clarity from public authorities on matters such as the revision of criteria for online and hybrid learning and the parameters of upcoming reforms (for example, the updated Law on Science and Higher Education). The imminent finalisation of reforms in Croatia may create a clearer understanding within institutions of the emerging future policy framework in which they will operate and support the development of appropriate institution-level policies and practices for digitalisation.
Figure 3.7. Institution-level allocation of responsibility for digital transformation
Source: 2022 CARNET-OECD survey of digital maturity of 95 higher education institutions,
Figure 3.8. Locus of decision making about the acquisition of digital tools and software
Source: 2022 CARNET-OECD survey of digital maturity of 95 higher education institutions.
Digital Infrastructure
Investment in digital infrastructure is fundamental to enhancing the ability of higher education institutions to achieve their digitalisation objectives. The following sections provide a selection of key findings from the institution digital maturity survey and interviews conducted during the OECD review visit. Additional perspectives on digital infrastructure are available in the technical advice on investment in digital infrastructure developed during this project (see Chapter 5).
Higher education institutions are satisfied with the internet connection provided by CARNET, but most require at least some upgrades to their on-campus networks
The CARNET-OECD survey results show most responding institutions were satisfied with the internet connection provided by CARNET. Average satisfaction among institutions on a scale from 1 (least satisfied) to 5 (most satisfied) was 4.6. On the other hand, institutions overall reported much lower satisfaction with the quality of their wireless connectivity (Table 3.6). More than one-quarter of all respondents to the survey provided a ranking of 1 or 2 for the quality of their on-campus wireless network.
Table 3.6. Average institution quality ranking of different types of network connectivity
Source: CARNET-OECD survey of digital maturity of 95 higher education institutions.
A lack of reliable network connectivity can have an adverse impact on the ability of institutions to conduct their day-to-day operations, and their capacity to improve their overall digital maturity. During interviews, many institution staff mentioned their satisfaction with the services and connectivity provided by CARNET. However, a few institutions reported that their existing internet connection was either no longer sufficient or was soon likely to become insufficient due to increased demand on-campus for network connectivity.
Conversely, and reflecting the results of the survey, many institution staff raised concerns during interviews about the state of their on-campus network equipment, which in some cases was reported as being up to 20 years old and in need of urgent upgrades. This has various impacts on operational efficiency. For example, one institution reported a need to outsource support for hybrid meetings as the local network could not manage the connectivity required, while others noted they had bandwidth issues when delivering online learning or large online meetings. These issues limited the extent to which participants could engage with meetings (e.g. needing to keep cameras off) or created lags when accessing Merlin (the centrally provided installation of Moodle). In other cases, institutions reported uneven access to connectivity in different parts of the campus, with some older buildings and buildings with thicker walls presenting more challenges for network upgrades, along with a lack of documentation on the current network structure.
Most institutions rely on in-house support staff, and some experience severe difficulties in maintaining ICT support services
Most institutions rely heavily on in-house specialist staff or their own IT departments to maintain their local networks and computer hardware, as well as to provide technical support to students and staff (Table 3.7).
Almost all the institution leaders interviewed by the OECD team during site visits raised the challenge of recruiting skilled IT staff to support their existing digital infrastructure. Higher education institutions are limited in the level of salary they can offer their staff. This constraint results in difficulties finding and retaining qualified staff, as the private sector often offers more attractive terms and conditions and is perceived by candidates as offering a more stimulating job environment. Many institution leaders said that achieving stable IT support staffing is at least as high a priority for their institution as upgrading their equipment.
Existing staff members are often overworked and have a heavy burden of responsibility. This was especially true during the period of emergency remote instruction. While institutions express their high appreciation for the work the staff members do, current regulations limit how they can be explicitly rewarded. While institutions have made efforts to come up with innovative ways to reward and retain IT staff, few of them have been able to arrive at a comfortable long-term solution for this difficulty, and are overdependent on existing staff members.
Table 3.7. IT support available within institutions for different functions.
Number of institutions reporting the existence of each type of support
Source: CARNET-OECD survey of digital maturity of 95 higher education institutions.
Strong concerns are evident in many institutions about the quality of digital infrastructure
Following its fact-finding visit, the OECD review team concluded that many higher education institutions had been notably proactive and diligent in identifying infrastructure gaps and finding the means to plug them. For example, some institutions have a policy of applying systematically for any available grant funding from national and international sources to improve digital infrastructure, including the use of research funds to purchase needed equipment. Others detailed their efforts to attract partners nationally and internationally to work with on digitalisation initiatives.
At the same time, many institution staff indicated persistent concerns about the quality of some of their infrastructure, equipment and technical services. In the CARNET-OECD survey, when asked to rate the quality of various services on a scale of 1 to 5, the vast majority of institutions reported the medium score (3). While in general institutions had a more positive perception of the quality of their own support services, almost one-quarter of institutions rated the quality of the servers and computing infrastructure as poor (a score of either 1 or 2) (Figure 3.9).
Croatian institutions would benefit from more centralised provision of software and services
Institution staff interviewed by the OECD team expressed appreciation for the software services provided to them by public authorities (directly or through organisations such as SRCE and CARNET). Frequently cited examples of software in heavy everyday use included Merlin (a SRCE-provided and supported Moodle-based virtual learning environment) and the Microsoft Office 365 licence provided by the Ministry of Science and Education. There is a clear demand from institutions and their representatives for more centralised software solutions. Examples of services in high demand include the implementation of persistent academic identifiers for students and staff that would be operational up to European level, research tools and access to databases for research students and staff, and software that can support assessment and detect plagiarism.
Figure 3.9. Institutions’ quality rating of different elements of digital infrastructure
Source: 2022 CARNET-OECD survey of digital maturity of 95 higher education institutions.
Institution staff and stakeholders said cybersecurity protection was also essential, and uncertainty was expressed by some interviewees about the vulnerability of some software and open-source platforms to cyber-attacks. Smaller institutions and those with less capacity have reached out to resource providers such as SRCE for assistance, while larger and more equipped institutions have developed and implemented their own solutions. The survey results also highlighted a general demand for central provision of services related to cyber and data security. For example, 73 of the 88 respondents from public institutions expressed interest in having an advanced firewall service provided, while 56 expressed interest in traffic encryption/decryption services.
Finally, institutions expressed a need for strong central support and guidance in relation to online exams and assessments. Ensuring the integrity of assessment processes carried out online is a persistent challenge. Many institution staff perceive a lack of central support, tools or guidance on this issue. During the pandemic, some institutions held exams online but required candidates to be located on the university campus, with restricted Wi-Fi access. Others aimed to redesign exam questions to be less fact-based and more analytical in nature. While some institutions concluded that in-person examinations were the only option to ensure integrity and quality, others noted that digital assessment tools offered the ability to automate some grading processes (e.g. for quizzes and some formative assessments) and expressed interest in building further capacity for high-quality digital assessment, with central support.
Institutions expressed a pressing need for certain types of audio-visual equipment
Croatian higher education institutions expressed a need to improve their on-campus audio-visual equipment, both during the site visits carried out by the OECD team and in their responses to the CARNET-OECD digital maturity survey. In particular, the survey results indicate a pressing need for audio-visual equipment to support the use of digital resources during lectures, as well as equipment that can be used for recording live lectures and hybrid teaching. Notably, these needs are ranked much more highly, on average, than the need for end-user equipment stocks for students and staff (Table 3.8).
Table 3.8. Croatian higher education institutions’ perception of on-campus equipment needs.
Source: 2022 CARNET-OECD survey of digital maturity of 95 higher education institutions.
Evidence from interviews indicates that funding equipment for lecture recording may be a particular challenge, as it generally cannot be financed by project-based funding (e.g. from research funding). Recordings carried out during the pandemic were often not of professional quality and were considered useful for emergency purposes only. Some institution staff highlighted the need not only for the audio-visual equipment itself, but also for suitable room environments in which to conduct the recordings, with proper acoustics and lighting.
Digital competence and culture
Digital learning in higher education is most successful, “when the socio-environment, supporting digital infrastructure, host institutions, and human participants are well prepared” (Blayone, 2018[11]). This requires digital competence of individual participants in higher education activities. Digital competence has been defined as “sets of knowledge, skills and attitudes relating to the purposeful and effective use of digital technologies” (Ala-Mutka, 2011[12]). At the European level, this is considered one of the core competencies necessary for learners both in current and future society (European Commission, 2019[13]). Broad evidence indicates that academic staff often lack the necessary competences to progress beyond using basic digital technology (e.g. slide decks) in the classroom (Englund, Olofsson and Price, 2017[14]); (Schneckenerg, 2009[15]).
Croatia has a long history of supporting development of competence in e-learning in its higher education system (Kupres and Pašić, 2005[16]). As in other countries, there is limited evidence available regarding the extent of digital competence in higher education teaching staff. A recent study did, however, highlight differences in competences across both fields of study and generations (Box 3.3).
A teacher’s use of digital technology to create quality learning for their students can depend on their own knowledge, skills and attitudes towards digital technology (Hofer, Nistor and Scheibenzuber, 2021[17]). But it can also depend on the existence of supportive and like-minded colleagues with a similarly high commitment to maximising the value of digital technologies. This can be considered the “digital culture” of the organisation. A strong digital culture within an organisation promotes collaboration above individual efforts, and aims to create a supportive, innovative environment where individuals feel empowered to use and experiment with digital technologies (BCG, 2018[18]).
The development of digital competence and culture also needs to consider the needs of learners. Today’s young entrants to higher education grew up surrounded by digital devices (Creighton, 2018[19]). But they still may not automatically be proficient with using common digital tools and software found in higher education institutions, particularly as digital technologies become increasingly integrated into all fields of education.
With these considerations in mind, this section presents some findings from the CARNET-OECD digital maturity survey and interviews conducted during the OECD’s site visit that are relevant to the development of digital competence and culture in Croatia’s higher education institutions.
Box 3.3. Assessing digital competence of higher education lecturers in Croatia
Few cross-national studies have been carried out addressing digital competence in higher education lecturers. A 2019 research study in Croatia aimed to fill the evidence gap. The study in Croatia was part of the wider 2015 DFGP study, “E-learning and Multimediakompetenz der Deutschen Initiative für Netzwerk Information” Research was also conducted in Germany and France. A representative sample (n=1800) of teachers and associates from institutions in Osijek, Split and Zagreb were polled through an online questionnaire regarding their level of digital competence and usage of modern technologies in their teaching practices.
One significant finding was the difference in digital competencies between faculties. Those working in technical faculties reported being “more competent” in conducting scientific research and using modern technologies than their counterparts in social sciences and humanities faculties. On the other hand, respondents from the humanities reported using technology in teaching more often and for their own scientific work than those in the technical and social sciences fields.
More than 70% of respondents noted they are ‘competent’ in the use of computers and new technologies. However, there is an apparent generational divide - 65% of assistant professors, teaching assistants and lecturers consider themselves part of the “digital generation” while only 33% of full-time professors felt the same, citing insufficient training in ICT technologies. There was also a difference in fields, with those in the technical field (56.8%) reporting a higher usage of e-learning technology (e.g. Moodle) than those in social sciences (39.6%).
The study authors also highlight that flexibility and a critical approach to information is necessary to address the growing obsolesce of competences caused by the speed of advancement in technology. A clearer definition of digital competences is required (defined as “complex integration of cognitive processes and dimensions”) along with a need to identify and develop different dimensions of digital competences, so teachers can respond to learner needs with the appropriate pedagogy and technology. The authors proposed a Multimedia Training and e-Learning programme tailored to education institutions offering teacher training. The programme would focus on honing basic levels of digital competence and would include analysis and implementation of e-learning; methodology and didactics in e-learning, models of e-learning platforms and techniques for using multimedia.
The authors also proposed 15 specific key competencies all educators should acquire, including acquisition of basic digital knowledge and literacy, knowledge of the Microsoft Office productivity suite, ability to search, collect and process information and data, ability to independently design an education programme for use in an online environment, and capacity to understand and use different e-learning tools (e.g. SCORUM, Open Content, MOOCs).
Source: Muller and Varga (2020[20]), Digital Competences of Teachers and Associates at Higher Education Institutions in the Republic of Croatia, https://doi.org/10.32914/i.52.1-2.4.
Some initiatives are in place to develop staff digital competence, but a more systematic and joined-up approach would be beneficial
During the OECD team review visit, many interviewees stressed the need to further improve digital competence in its staff members and outlined their efforts to do so. Some institutions have developed their own initiatives for digital competence development, while others encourage the use of CPD among their staff. The period of emergency remote instruction was also mentioned as indirectly providing general development of pedagogical skills – the large-scale upload and digitalisation of learning material was beneficial not only for students, but also for teachers to engage in peer-based learning; allowing younger professors to review the teaching practices of older and more experienced colleagues in a way that wasn’t possible previously.
Regardless of progress made during the pandemic, many staff expressed a need for a more systematic approach to the development of digital competence and culture that requires a rethinking of existing academic staff terms, conditions, and reward structures. In Croatia, as in most OECD countries, academic staff begin teaching in higher education institutions without having or needing a formal teaching qualification or training, and there are no standardised requirements for professional development of staff to enhance their teaching skills (OECD, 2020[21]). As was pointed out multiple times to the OECD review team, this renders the efforts and progress of staff and organisations in improving digital competence largely invisible, with no clear channels available to either recognise or reward high performance.
Some interviewees felt centrally provided training on digital competence could move towards systematically imparting knowledge and evidence on the best pedagogical practices for engaging and supporting students in a digitalised environment. In some cases, current training offers and competence-building initiatives were perceived to focus more on the use of technical tools and sharing of content, rather than how to apply these resources effectively. More training is needed to teach staff how to design and construct courses fit for a digital environment, rather than simply duplicating practices used in face-to-face instruction.
Many existing pedagogical practices for online and hybrid education lead to deficits in the social element of learning and lower student engagement
As in other countries, the period of emergency remote instruction in Croatia exposed many limitations in current pedagogical practices for delivery of online learning. Teaching staff reported several difficulties with supporting student learning in an online environment. Frequently mentioned challenges included:
A lack of eye-contact and inability to read non-verbal feedback of students (i.e. facial expressions and body language), which provide vital signals to teachers about students’ level of understanding of the material being taught, and cues for which concepts and knowledge need ad hoc reinforcement.
An inability to provide adaptive instruction, particularly with asynchronously provided material. Each student is required to consume the same content, with no ability to reinforce learning by asking questions of the instructor or of peers in the classroom.
The attention span of students in an online environment appears shorter. Online study provides easy access to a range of distractions, leading to reduction in student engagement.
Lack of engagement of students in an online environment leads to greater inefficiencies in contact between instructors and students (e.g. instructors needing to respond to the same query multiple times to individual students, rather than dealing with it once during a live class).
The online environment also created new pressures for teaching staff. Many staff made increased efforts to support students in a remote environment, becoming more available and responsive over a range of communication channels. This led in some cases to increased student expectations for immediate access to staff and quick responses to ad hoc queries, even following the return of in-person education. Some teaching staff also reported that students increasingly expect digitalised content to be made available as a complement to their in-person lecturers, which has led to a greater workload for staff.
Many of the issues raised, it is acknowledged, were heightened by the emergency context in which mass online learning was introduced and staff had to adapt their content very quickly to an online environment. Nevertheless, the OECD review team concluded that successful mainstreaming of online education in Croatia would only be achieved with greater emphasis on engaging students through pedagogical principles and practices specifically designed to be effective in an online environment.
Students enjoy a range of digitalised services but could be more heavily included in institution plans for the development of digital competence and culture.
Students in many higher education institutions in Croatia enjoy access to a range of digitalised administrative support and wraparound services that can simplify and streamline their educational pathways. Students in most reporting institutions have access to a common identity authentication service, online library services and online enrolment and records services, while about half of institutions report that digital assessment and grading and online academic support or tutoring is available to their students (Figure 3.10).
On the other hand, far fewer institutions offer counselling online or orientation to students for online study or study skills support. The CARNET-OECD survey also indicated that in general, teachers have far more access to formalised opportunities to improve their digital competence than students (Figure 3.11). Teaching staff in almost all institutions that responded have access to occasional formal training aimed at improving digital competence, while many also avail of informal learning through communities of practice and other forms of knowledge sharing. On the other hand, students are generally expected to build digital competence through their regular coursework, or via self-directed learning. A few institutions do offer targeting training and certification for students to acquire digital skills (Figure 3.11).
Figure 3.10. Digitalised student services available in Croatian higher education institutions
Source: 2022 CARNET-OECD survey of digital maturity of 95 higher education institutions.
National studies and reviews of practice (e.g. from the Agency for Science and Higher Education) have indicated a need for greater efforts to build digital competence among students. Student representatives reported to the OECD team in interviews that while institution staff made substantial efforts to support them during emergency remote instruction, many classmates struggled with stress and mental health issues – including before and during online assessments – and did not always feel able to ask for assistance. Students also reported difficulties communicating with some professors, particularly those with more limited experience in working online.
Overall, however, it was clear students highly appreciated the efforts made by staff during the emergency period and were more likely to have encountered problems with connectivity and lack of access to on-campus equipment (e.g. for practical study) than difficulties in using the digital tools and content provided during the pandemic. Many students report that they have been able to improve their digital competence using resources located outside their institution (e.g. YouTube videos or asking friends).
Some student representatives highlighted that, despite being the target audience for much of the digital content being developed by instructors, they were rarely consulted on issues of pedagogical design or user experience. They perceived substantial differences in the approach to online learning taken by their institutions, compared to professional education technology companies, such as edX or Coursera. Greater student involvement in digital learning design could improve the relevance of content for students and increase their engagement.
Figure 3.11. Available means for teachers and students to improve digital competence
Source: 2022 CARNET-OECD survey of digital maturity of 95 higher education institutions.
Some elements of innovative digital culture appear to be more developed than others in Croatian institutions
Digital culture entails building a comprehensively supportive environment for digital innovation and improvement across the entire organisation. Some items included in the CARNET-OECD survey provide an indication of the extent to which there is a culture for digital innovation embedded in Croatian higher education institutions. For the most part, Croatian institutions tend to focus on raising the general awareness of the need to innovate, at staff gatherings and events. Less emphasis is placed on more concrete steps, such as the development of an innovation strategy, providing incentives for innovation, or recognising and awarding innovative projects (Figure 3.12).
Figure 3.12. Supports for digital innovation in Croatian higher education institutions
Source: 2022 CARNET-OECD survey of digital maturity of 95 higher education institutions.
Challenges associated with innovation were also raised during site visits with the OECD team. While many institution staff mentioned examples of recent innovative digital initiatives, others reflected that innovation was dependent on the motivation and capacity of individual academic or ICT staff, rather than a product of the general culture of the institution. While some staff are enthusiastic and eager to become involved in experimentation with digital technologies, others are less engaged or simply do not have the capacity to do so. Specific incentives and additional staff supports are necessary to establish a broader culture for digital innovation across Croatian higher education institutions.
References
[12] Ala-Mutka, K. (2011), Mapping digital competence: Towards a conceptual understanding, European Commission, Joint Research Centre (JRC), https://doi.org/10.13140/RG.2.2.18046.00322.
[4] Balaban, I., N. Redjep and M. Calopa (2018), “The Analysis of Digital Maturity of Schools in Croatia”, International Journal of Emerging Technologies in Learning (iJET). 13 (06):4, doi: 10.3991/ijet.v13i06.7844.
[18] BCG (2018), It’s Not a Digital Transformation Without a Digital Culture | BCG, https://www.bcg.com/publications/2018/not-digital-transformation-without-digital-culture (accessed on 12 July 2022).
[11] Blayone, T. (2018), “Reexamining digital-learning readiness in higher education: Positioning digital comptencies as key factors and a profile applicaiton as a readiness tool.”, International Journal on E-Learning, pp. 425-451.
[6] Bralić, A. (2016), “ICT and e-learning in HE in Croatia: strategies and current state”, Conference: Central European Conference on Information and Intelligent Systems, CECIIS, Vol. University of Zagreb, pp. p. 91-98;, https://www.bib.irb.hr/846915.
[5] Conrads, J. et al. (2017), Digital Education Policies in Europe and Beyond: Key Design Principles for More Effective Policies, Joint Research Centre.
[19] Creighton, T. (2018), “Digital Natives, Digital Immigrants, Digital Learners: An International Empirical Integrative Review of the Literature”, Education Leadership Review, Vol. 19/n1, pp. 132-140; https://eric.ed.gov/?id=EJ1200802.
[10] DIGI-HE (2021), Developing a high performance digital education ecosystem-Institutional self-assessment instruments, https://eua.eu/resources/publications/953:developing-a-high-performance-digital-education-ecosystem.html (accessed on 26 March 2023).
[8] Durek, V., N. Kadoic and N. Begicevic Redep (2018), “Assessing the digital maturity level of higher education institutions”, 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 0671-0676, https://doi.org/10.23919/MIPRO.2018.8400126.
[14] Englund, C., A. Olofsson and L. Price (2017), “Teaching with technology in higher education: Understanding conceptual change and develompent in practice.”, Higher Education Research and Development 26 (1), pp. 73-87; https://doi.org/10.1080/07294360.2016.1171300.
[13] European Commission (2019), Key Competencies for Lifelong Learning, https://op.europa.eu/en/publication-detail/-/publication/297a33c8-a1f3-11e9-9d01-01aa75ed71a1/language-en.
[17] Hofer, S., N. Nistor and C. Scheibenzuber (2021), “Online teaching and learning in higher education: Lessons learned in crisis situation”, Computers in Human Behavior, pp. https://doi.org/10.1016/j.chb.2021.106789.
[2] Ifenthaler, D. and M. Egloffstein (2020), “Development and Implementation of a Maturity Model of Digital Transformation”, TechTrends, Vol. 64/2, pp. 302-309, https://doi.org/10.1007/S11528-019-00457-4.
[16] Kupres, D. and D. Pašić (2005), E-learning in Croatia, https://www.e-mentor.edu.pl/_xml/wydania/8/127.pdf.
[20] Müller, M. and M. Varga (2020), “Digital Competences of Teachers and Associates at Higher Education Institutions in the Republic of Croatia”, Informatol, pp. 8-23; DOI:, https://doi.org/10.32914/i.52.1-2.4.
[21] OECD (2020), Resourcing Higher Education: Challenges, Choices and Consequences, Higher Education, OECD Publishing, Paris, https://doi.org/10.1787/735e1f44-en.
[9] OECD (forthcoming), Towards Effective and Equitable Digital Education- State of Play and Promising Policies, OECD Publishing.
[1] Proença, D. and Borbinha (2018), “Maturity models for data and information management”, Springer, Vol. 11057 LNCS, pp. 81-93, https://doi.org/10.1007/978-3-030-00066-0_7.
[3] Rossman (2018), Digital Maturity: Conceptualization and Measurement Model, https://www.researchgate.net/publication/345760193_Digital_Maturity_Conceptualization_and_Measurement_Model (accessed on 11 July 2022).
[15] Schneckenerg, D. (2009), “Understanding the real barriers to technology-enhanced innvoation in higher education.”, Educational Research, pp. 411-424; https://doi.org/10.1080/00131880903354741.
[7] University of Zagreb (2021), About the University, http://www.unizg.hr/homepage/about-university/.