Measuring the Digital Transformation

A Measurement Roadmap for the Future

Measuring the Digital Transformation maps existing indicators drawn from a wide range of areas including education, innovation, trade, economic and social outcomes against current digital policy issues, as presented in Going Digital: Shaping Policies, Improving Lives (OECD, 2019). By so doing, it identifies gaps in the current measurement framework and assesses progress made by several initiatives towards filling these gaps, as documented in the 19 roadmap sections of the publication. The overarching objective of Measuring the Digital Transformation is to advance the measurement agenda by building on these roadmaps and a wide body of ongoing work in national and other international organisations, as well as areas already identified in Measuring the Digital Economy: A New Perspective (OECD, 2014) and in the G20 Toolkit for Measuring the Digital Economy (G20, 2018).1

This is a challenge. Existing metrics and measurement tools struggle to keep up with the rapid pace of the digital transformation. The range of questions that can be asked about its impacts is daunting. How can digital transformations be measured and tracked in all sectors of the economy, including the public sector? How to measure the disruption of existing business models and the emergence of new ones, the reorganisation of work or the size of the sharing economy? How can the value of data, both private and public, be captured in standardised statistics? How can international transactions of digitised goods and services be traced? How should the impact of policies on the digital economy be monitored and assessed? What are the economic activities and jobs of the future? What are the impacts of digital transformations on the well-being of citizens and society at large?

Much of the information required to respond to these questions already exists or is being developed, but not all. There is a recognition that statistical information systems need to adapt, and in some cases expand, to capitalise on their ability to provide more granular insights. There is also a need for new, complementary, data infrastructures capable of tracking the emergence of new activities and monitoring their substitution for traditional ones, on a timely basis wherever these occur. Such information systems must also adapt to newly emerging digital footprints (i.e. the enormous flows of information generated by digital technologies and digitally enabled activities, such as e-commerce, cloud services and the Internet of Things) that are now being generated.

In the shorter term, the challenge is to improve the international comparability of current indicators and make statistical systems more flexible and responsive to the introduction of new and rapidly evolving concepts driven by the digital transformation.

“Even in areas where international standards to guide statistical collection exist, countries may lack the capabilities and resources to implement them systematically, disseminate the resulting information openly or make efforts to ensure that data are comparable. There is a clear lack of coverage in developing countries compared to developed countries due to differences in statistical capacity among countries, or user needs and priorities for statistical collection” (G20, 2018).

Even among OECD countries, ensuring the international comparability of indicators used to monitor the digital transformation can present challenges. Only a limited number of indicators can be compiled for monitoring across countries, and these are usually fairly standard and not sufficiently granular to capture the changing dynamics of the digital transformation. Efforts to exploit official statistics at the micro level (e.g. enterprise/establishment/organisation, worker or household/individual) in an internationally co-ordinated fashion, including the use of administrative data and the exchange of micro-data among national statistical offices (NSOs), should be supported, especially with respect to data-linking opportunities. This will mean continuing to encourage the development of tools and mechanisms to access micro-data while ensuring data confidentiality.

A number of options exist and have begun to be explored and developed to increase the flexibility of current statistical frameworks. These include developing and populating satellite accounts, exploiting the potential of existing micro-data, adding questions to existing surveys, periodically augmenting existing surveys with topic-specific modules and developing high-frequency surveys to meet specific needs. Remaining gaps could be addressed through new and experimental approaches developed to meet the specific priorities and resources of countries (OECD, 2014).

In the longer term, the challenge for the statistical community will be to design new and interdisciplinary approaches to data collection and to leverage the information captured by digital systems.

As the digital transformation spreads across every sector and affects every aspect of society, measuring its distinct features and dynamics will become increasingly challenging. New approaches will be needed – and the digital tools and footprints created by digital activities can form part of the solution. The digital transformation is also being felt across all dimensions of data production and use. For example, qualitative information is increasingly becoming a source of quantitative evidence. Text-mining tools (e.g. natural language processing) underscore the potential to alleviate some of the common challenges facing statistical collection (e.g. survey fatigue and classification systems that are applied differently by human coders) and offer opportunities for generating adaptable indicators. In this context, policies promoting (open) access to data collected for administrative purposes by the public and private sectors represent an important means to facilitate new forms of analysis.

The next generation of data infrastructure for policy making in the digital era needs to build partnerships with the private sector and engage with stakeholders to bring publicly available, reliable data into the policy-making process.

The proposed measurement roadmap will have to be discussed and implemented gradually through close co-operation between the statistical community and other stakeholders. Policy makers will need to define user needs, while researchers contribute insights essential for the development of appropriate metrics and data infrastructures. Engagement with organisations, businesses, universities and the public sector will be indispensable, as the statistical system can only collect data that can feasibly be measured inside such organisations. In particular, private source data can open new opportunities for monitoring the digital transformation and its impacts. They can help track data flows and uses on a continuous basis across actors, sectors and locations. For example, these data can provide insights into job vacancies and the emergence of new jobs profiles or the new services and business models enabled by online platforms. However, the use of private source data for measurement and analysis raises new challenges that need to be overcome by working together both on statistical quality frameworks and viable economic models of data sharing.

The OECD and the other international organisations actively contributing to the digital measurement agenda will need to continue to improve co-ordination, in order to avoid fragmented efforts and initiatives and ensure that the international community takes up the challenge to further build the evidence base for more robust policies for growth and well-being in the digital era.

Nine actions – if prioritised and implemented – would substantially advance the capacity of countries to monitor the digital transformation and its impacts. The first four overarching actions (1 to 4) are directed towards building the next generation of data and indicators capable of dealing with the challenges of the digital transformation. An additional five actions (5 to 9) target specific areas identified as requiring priority attention.

Action 1: Make the digital economy visible in economic statistics

Measuring the digital transformation and its impacts requires the development of indicators that complement the views provided by traditional measurement frameworks, such as those used to measure GDP and trade flows. But even within these current frameworks the way that firms, products and indeed transactions are classified and identified requires adaptation. In addition, it is essential to accelerate efforts to capture relevant phenomena outside the current production boundary of national accounts, for example, those concerning the consumption (and value) of online services provided to consumers free of charge, such as online search, social networking sites and so on. At the same time, work on tackling the challenges of globalisation and the measurement of services not physically fixed to a single location (e.g. cloud services and services provided by online platforms) should be further encouraged. Platforms in particular raise new policy challenges, but little is yet known about the actors operating on them, their characteristics, the types of activities in which they engage, the services they provide, the value they create and the locations from which they operate. In addition, by providing easy access to customers for transport services, accommodation, food delivery and many others, online platforms are increasing the importance of household production, blurring the lines between different institutional sectors within the economy and changing the nature of work.

NSOs, research communities and international organisations are encouraged to continue to work together to:

• Populate the OECD digital supply-use tables (Mitchell, 2018) and measure transactions in line with the Handbook on Measuring Digital Trade (forthcoming), in particular for those areas that supplement national accounts and trade statistics, in order to obtain new details and perspectives.

• Within the framework of the digital supply-use tables and the Handbook on Measuring Digital Trade:

  • Identify transactions based on their “digital nature” (i.e. digitally ordered, digitally delivered and/or digital intermediary platform enabled) and new actors relevant for the digital economy (e.g. digital intermediary platforms, e-sellers and firms dependent on intermediary platforms).

  • Develop new aggregations of firms, products and transactions that provide more granular insights into the actors, including households and the products involved.

  • Better capture digitally enabled production by households and continue to develop estimates of unpaid household activities in economic statistics and tackle the challenge of understanding and estimating the value generated by services provided to users free of charge (though often involving an implicit transaction related to personal data).

• Improve the quality and breadth of information on e-commerce transactions through enhancements to surveys on ICT use by businesses and individuals, the incorporation of e-commerce questions into other appropriate surveys (particularly e-commerce revenues in structural business surveys and online spending in household expenditure surveys), and the use of alternative data (e.g. anonymised information on transactions from banks and credit card companies).

• Support the development of common definitions and taxonomies of different types of platforms, formulate standard questions on platform work for inclusion in relevant surveys (e.g. labour force, ICT usage and time-use surveys) in order to derive robust estimates of the numbers of platform workers, and explore the role of administrative data and alternative data sources (e.g. web-scraped data) to gain insights into platform-intermediated transactions.

Action 2: Understand the economic impacts of digital transformation

Digital technologies are implemented as a part of business processes, together with labour, capital and knowledge capital assets, in order to drive performance. The initial and strongest evidence of their economic impact will likely surface in micro-data (data about firms, workers or consumers) before showing up in macro-data. To this end, it is important to be able to link together existing datasets, exploit the potential of administrative records, and develop measures of digital maturity in business that can then be used to analyse the impacts of digital technologies on firm performance. Robust measures of changes in prices and quality are also crucial to analysing the contribution of digital technologies to economic performance. For example, measures of the actual performance of broadband connections (i.e. broadband quality) are critical for consumers to make informed choices, and for policy makers and regulators alike to ensure that the services provided are of optimal quality. However, they are also key to measuring productivity and assessing the contribution of ICTs to economic growth. Digitalisation may also further complicate the measurement of prices and volumes more generally, as it increases the pace of quality change, leads to changes in the outlets through which products are sold and may involve new price differentiation practices, among others.

Quality of service provision should also be considered in the context of “divides”, such as between businesses of different sizes, or households with different compositions, incomes or locations. To this end, business and household surveys on the adoption of digital technologies should continue to be reviewed regularly to fully account for emerging phenomena, such as high-speed broadband, cloud computing services, data assets and other technologies, both as enablers of innovation and as contributors to business performance and consumer welfare. At the same time, opportunities to further capitalise on administrative data through linking existing datasets should continue to be exploited. In addition, surveys of technology adoption and administrative data need to be aligned with aggregate economic measures.

The broader statistical community is encouraged to:

• Improve the measurement of ICT investment to arrive at internationally comparable deflators for hardware, software and communication infrastructure, including the pricing of broadband services bundles, and analyse the impacts and opportunities digitalisation creates in relation to the measurement of prices and volumes more generally.

• Improve the measurement of broadband quality (performance), including experienced speeds, latency, reliability and robustness of broadband services in both rural and urban areas.

• Regularly review the framework for measuring ICT usage to identify and prioritise areas in which surveys can improve and evolve in line with ongoing developments and policy priorities; this includes delivering sufficiently granular detail for the differentiated analysis of impacts of the digital transformation on individuals, firms and places.

• Exploit the statistical potential of administrative data sources and review existing data collections to maximise data-linking opportunities for research.

• Improve access to these datasets while ensuring data confidentiality.

Action 3: Encourage measurement of the digital transformation’s impacts on social goals and people’s well-being

The digital transformation is impacting many aspects of people’s lives. Accordingly, measurement frameworks are required to capture these aspects including emerging impacts. In this respect, frameworks play a key role in measuring the extent to which digital technologies and new business models can help address societal goals, including those associated with health, ageing populations and climate change. At present, evidence of the impacts of the digital transformation on well-being is scarce in many areas. For example, relevant data on how the use of digital technologies affects people’s experiences of mental health or their social lives are not collected frequently or in a harmonised manner. Survey vehicles are an important source of self-reported objective and subjective data. They can be used to collect data on people’s life experiences in the context of the digital transformation, as well as to attempt to establish causal relationships (e.g. between the diffusion of digital technologies and various well-being outcomes).

The broader statistical community is encouraged to:

• Promote wider implementation of the OECD Model Survey on ICT Access and Usage by Households and Individuals (OECD, 2015) to develop subjective well-being and mental health questions for inclusion.

• Include detailed ICT-use variables in household surveys (e.g. general social surveys and labour force surveys), and especially in longitudinal surveys, in order to better understand the causal relationships between Internet use and well-being outcomes over time.

• Develop new statistical tools including surveys to monitor the impact of ICT use on adults and children, such as exposure to disinformation or hate speech.

• Improve measurement of the impact of the digital transformation on the environment, by enhancing statistical linkages among ICT-use surveys, consumer expenditure surveys, supply-use tables and industry-level data.

Action 4: Design new and interdisciplinary approaches to data collection

Given the pace of technological change, it is understandable that current frameworks are not yet able to reveal the full magnitude and scope of the digital transformation. However, digital technologies can be part of the solution as they generate enormous flows of information. Numerous online actions leave digital “footprints” that can be observed using tools that scan, interpret, filter, gather and organise information from across the Internet. While they offer great opportunities for statistics, Internet-based data also raise a number of issues regarding statistical quality, security and privacy that must be addressed. The Internet also enables the creation of non-physical organisations and the flexible outsourcing of business activities, within sectors of activity and across locations, thus blurring the boundaries between firms and markets and between work and social life. This creates challenges for current methods of collecting statistics. New interdisciplinary methods of analysis are therefore necessary to understand innovative behaviour, its determinants and its impacts at the level of the individual and the organisation.

NSOs, regulators, Internet Service Providers (ISPs), the research community, the Internet community and international organisations are invited to work together to:

• Further develop international statistical standards for the collection of Internet-based data and their compilation into statistical indicators (e.g. treatment of web search results).

• Assess alternative models of co-operation among businesses, Internet intermediaries and NSOs for the collection and treatment of Internet-based data; and promote the development of an associated regulatory framework, including technical and regulatory solutions, to preserve user security and privacy.

• Develop interdisciplinary approaches to data collection and new units of data collection.

• Improve the measurement of digital activities in complex business structures, organisations and networks.

Action 5: Monitor technologies underpinning the digital transformation, notably the Internet of Things, AI and Blockchain

A range of rapidly developing technologies are set to drive the next phase of the digital transformation. The Internet of Things (IoT), an ecosystem in which applications and services are driven by data collected from devices that act as sensors and interface with the physical world, is expected to grow exponentially, connecting many billions of devices within a relatively short time. IoT applications span economic sectors including: health, education, agriculture, transportation, manufacturing, electric grids and many more. Meanwhile, artificial intelligence (AI) has the potential to revolutionise production as well as contribute to tackling global challenges related to health, transport and the environment. Blockchain likewise has the potential to transform the functioning of a wide range of industries and applications such as finance, health, transportation, agriculture, environment and supply chain management. The general purpose and interdisciplinary nature of these digital technologies underscores the need for a consistent framework to define them, identify their emergence, monitor their development and diffusion, and quantify their economic and social impacts.

Policy makers, regulatory authorities, business, statistical and research communities are encouraged to:

• Develop internationally harmonised definitions and taxonomies for AI and Blockchain, fit for the purpose of monitoring the development of these technologies and their applications, including defining the key policy needs for measurement.

• Build on the OECD definition of IoT (OECD, 2018) and related taxonomies for its application domains (e.g. massive machine communications such as sensors for smart cities, critical IoT requiring ultra-fast and highly reliable connections such as automated vehicles); and provide clear prioritisation for the measurement of those IoT elements and indicators of most relevance to policy makers, beyond simple counts of machine-to-machine connected devices, in order to measure the potential demands IoT might create for communication infrastructures due to the flow of large amounts of data generated.

• Engage with stakeholders within the IoT ecosystem (e.g. different connectivity providers, IoT platform providers, etc.) for the benefit of data collection and policy and regulatory analysis.

• Develop tools to monitor the adoption of IoT, AI and Blockchain technologies by businesses and the impact of their diffusion on performance and productivity.

Action 6: Improve the measurement of data and data flows

In recent years, both the scale of data usage and its importance for many business models and processes has increased exponentially. However, there are significant challenges involved in evaluating data as an input to production and their “asset-like qualities”. Data flows between organisations in particular can take place quickly and at low cost. Moreover, different organisations can derive value from the same data, at the same time, without diminishing what others can do with them. Finally, the value of data is heavily context-dependent (e.g. on the information contained and how it is used). The combination of these factors results in many conceptual and practical measurement challenges. These are further amplified by the fact – linked in part to the proliferation of cloud computing services – that these flows and interactions commonly occur across national boundaries.

The statistical, business and research communities, and international organisations are encouraged to work together to:

• Develop pertinent taxonomies and classifications of data for statistical measurement purposes.

• Further study the role and nature of data in business models and processes.

• Explore methods for measuring data flows and stocks.

• Improve the measurement of knowledge-based assets including data and their role for production, productivity and competitiveness.

Action 7: Define and measure skills needs for the digital transformation

The development of the digital economy and its applications, such as “Big data” analytics, cloud computing and mobile applications, increases the demand for certain skills that are often in short supply. At work, a shortage of ICT specialists may be compounded by managerial obstacles to the development of new business models, new organisational structures and new working methods. At the same time, demand is rising for complementary skills, such as the capability to compile and analyse information, communicate on social networks, brand products on e-commerce platforms and so on. This trend also heightens the need for users to learn how to search and choose among a myriad of mobile applications and to know how to protect themselves against digital security risks (“digital hygiene”).

Traditionally, official statistics have used educational attainment, vocational training with standardised content, or occupational categories with codified and predictable tasks as a proxy for skills. New insights could be gained by exploiting and harmonising detailed national surveys on tasks and skills and by working with the business community to define new metrics of skill shortages.

The statistical, business and research communities, and international organisations are encouraged to work together to:

• Exploit the potential of existing public and private statistics on skills, and occupation and industry classifications, and to promote the harmonisation of national job tasks surveys.

• Better exploit existing cross-country surveys (e.g. the European Survey of Working Conditions and the OECD Programme for the International Assessment of Adult Competencies), and promote the linking of employer-employee datasets containing information on skills, jobs and activities at the individual level.

• Improve access to and the use of online vacancy datasets to measure vacancies in digital-related jobs, their duration and rate of filling.

• Encourage the systematic use of expert assessment to identify emerging skills needs at a detailed level of tasks and occupations, and across different countries.

Action 8: Measure trust in online environments

Management of security, privacy and consumer protection risk online, as well as the general level of trust of the population in online environments, have become key policy issues as individuals, businesses and governments shift large parts of their daily activities to the Internet. While efforts have been made to improve the measurement of trust, such as the harmonisation of statistics from Computer Security Incidents Response Teams (CSIRTs) and a consumer survey of attitudes to trust in peer platform markets, other avenues should be explored further. For instance, the OECD has developed an analytical framework for measuring digital security risk management practices in businesses, based on the Principles contained in its 2015 Council Recommendation on Digital Security Risk Management for Economic and Social Prosperity. This framework has led to the identification of a set of potential core indicators. Work is also ongoing to improve the international comparability of personal data breach notification statistics, which are produced by Privacy Enforcement Authorities (PEAs). Despite the broadly acknowledged importance of trust between partners in online exchanges, measurement of these aspects of trust is not a longstanding practice, especially within official statistics. Alternative approaches currently underway utilise behavioural insights from experiments, for example, to ascertain how disclosures impact consumer trust in the context of personalised pricing in e-commerce. Internet-based data (e.g. malware activities recorded by a firewall, use of sentiment analysis on social media to measure people’s trust, cookie statistics, browser settings or statistics on downloads of security/privacy-related software) could also be used to measure various aspects of trust.

The statistical community, regulators and other stakeholders, such as Internet intermediaries, business and consumer associations, and international organisations are invited to work together to:

• Develop guidance for PEAs to produce and report internationally comparable statistics on data breach notification.

• Develop a more reliable and comprehensive dataset on digital security incidents and digital risk management practice, key elements of which include reaching a consensus on typology and taxonomy, the creation of a trusted public-private digital security incident repository, and incentives to promote the reporting of incidents and data sharing by organisations.

• Test and improve the quality and rate of response of digital security-related surveys.

• Further study consumer attitudes and behaviour to highlight contexts where trust in online interactions increases or decreases, with a view to improving survey methodology in this area.

• Develop a framework for measuring individuals’ trust in online environments, and explore survey-based and experimental approaches to test the feasibility of measuring this trust.

• Explore the use of Internet-based statistics to measure trust-related aspects and promote a statistical quality framework for Internet-based data.

Action 9: Establish an impact assessment framework for digital governments

Governments are progressively adopting digital technologies to encourage innovation in service design, operation and delivery. The move from using digital technologies to improve efficiency (e-government) to using them to influence and shape public governance outcomes (digital government) should enable governments to better respond to broader policy imperatives such as public trust, social well-being and civic engagement. To address the challenges and seize the opportunities of the digital age, governments should prioritise the establishment of an impact assessment framework to measure the concrete contribution of digital government to broader policy outcomes.

Policy makers, the statistical and research communities, and international organisations are encouraged to work together to:

Develop new statistical tools to assess the effects of digital technologies on the relationship between governments and citizens and businesses, taking into special consideration the extent to which key groups are impacted (e.g. seniors, low-income households, single parents, those with disabilities or mental health issues and so on), and providing evidence on the overall level of public trust in government.

• Define metrics to assess the effect of digital technologies on driving more efficient, inclusive and tailored public service delivery.

• Develop metrics to measure the impacts of existing practices and policies to promote public sector data sharing, access and re-use, including with regard to citizens’ trust in the ability of governments to handle personal data.

• Establish guidelines for public sector organisations to measure the scope and impact of data re-use in public administrations and on public policy-making processes.

• Measure the diffusion of emerging technologies such as AI and Blockchain within government processes and services.

• Evaluate potential barriers to the full integration of digital technologies within government.

References

OECD (2019), Going Digital: Shaping Policies, Improving Lives, OECD Publishing, Paris, https://doi.org/10.1787/9789264312012-en.

G20 (2018), Toolkit for Measuring the Digital Economy, G20 Digital Economy Task Force, G20 Argentine Presidency 2018, Buenos Aires.

OECD (2018), OECD (2018), “IoT measurement and applications”, OECD Digital Economy Papers, No. 271, OECD Publishing, Paris, https://doi.org/10.1787/35209dbf-en.

OECD (2015), “Model Survey on ICT Access and Usage by Households and Individuals”, OECD publishing, Paris, https://www.oecd.org/sti/ieconomy/ICT-Model-Survey-Access-Usage-Households-Individuals.pdf.

OECD (2014), Measuring the Digital Economy: A New Perspective, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264221796-en

Mitchell, J. (2018), A proposed framework for digital supply-use tables, (OECD, forthcoming).

Note

← 1. The 2018 Argentine G20 Presidency, in collaboration with a steering committee of international organisations (IOs) led by the Organisation for Economic Co-operation and Development (OECD) and comprising the International Telecommunication Union (ITU), the United Nations Conference on Trade and Development (UNCTAD), the European Union, the World Bank Group (WBG), the International Monetary Fund (IMF), and the International Labour Organization (ILO), has produced a G20 Toolkit for Measuring the Digital Economy. The toolkit highlights methodological approaches and indicators used to monitor the digital economy, and key gaps and challenges regarding digital economy measurement for further study. See Annex 3 of the G20 Digital Economy Ministerial Declaration, 24 August 2018, Salta, Argentina.

Several international organisations are contributing to the measurement of the digital transformation through initiatives, some of which are described in the G20 Toolkit for Measuring the Digital Economy. These include, but are not limited to, work on key ICT indicators within the Partnership on Measuring ICT for Development led by the ITU, UNCTAD and the United Nations Educational, Scientific and Cultural Organization (UNESCO) Institute for Statistics (UIS). The OECD works closely with several of these organisations, including the World Trade Organisation (WTO) on the issue of measuring digital trade, and the IMF on measuring the implications of the digital economy for macroeconomic statistics.