OECD Science, Technology and Innovation Outlook 2018

Chapter 14. Blue Sky perspectives towards the next generation of data and indicators on science and innovation

Fernando Galindo-Rueda

Abstract

Just like its subject matter, the world of data, indicators and analysis on science, technology and innovation (STI) is experiencing profound transformations that call for co-ordinated action by users and producers of STI data and statistics. This chapter brings together discussions and perspectives shared at the OECD Blue Sky Forum 2016. It presents several key trends presenting both challenges and opportunities. It first considers major developments, starting with the latest trends in policy uses of STI data and statistics. It then assesses how digitalisation has transformed the production of STI data, and examines the major issues facing traditional and new producers of STI data and statistics. It concludes by discussing the future outlook and key governance perspectives in STI measurement and analysis, including the potential role of the OECD.

I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be

William Thompson, Lord Kelvin (1883)

Introduction

Just like the field of science, technology and innovation (STI) itself, the data, statistics, indicators and empirical analysis on the structure and dynamics of science and innovation systems have changed significantly in recent years. Many of these trends are expected to intensify over the next decade. This chapter examines expert views on the way digitalisation and the aftermath of the global financial crisis will shape the production and use of STI data and indicators. It builds on the discussions and outcomes of the OECD Blue Sky Forum 2016, a major global conference organised by the OECD on the future of STI data and indicators in Ghent, Belgium (Box 14.1). It also examines work carried out since. Science ministers of the OECD countries endorsed this initiative as a vehicle to “continue improving statistics and measurement systems to better capture the key features of science, technology and innovation” (OECD, 2015). This chapter reviews key messages from the Blue Sky Forum 2016 and considers the outlook for STI statistics production and policy use in the coming years.

Box 14.1. The OECD Blue Sky Forum on STI data and indicators

Every ten years, the OECD convenes and engages the policy community, data users and data providers in an open dialogue to review and develop the OECD long-term agenda on STI data and indicators. This event is part of the OECD Committee for Scientific and Technological Policy’s Programme of Work; its organisation is entrusted to its Working Party of National Experts on Science and Technology Indicators (NESTI). Known as the “OECD Blue Sky Forum”, the title reflects the intention to provide a long-term, unconstrained discussion on evidence gaps in STI, as well as on initiatives the international community can take to identify and address related data needs. Previous editions held in Paris (1996) and Ottawa (2006) have been influential in setting the path for a series of national programmes supporting research aiming to advance the scientific basis for science and innovation policy (Marburger, 2007, 2011). They have also informed the 2010 OECD Innovation Strategy (OECD, 2010a) and its measurement agenda (OECD, 2010b).

Among its objectives, the Blue Sky Forum 2016 set out to:

• review the main conceptual underpinnings and use of current frameworks for STI indicators and data infrastructure initiatives

• explore the role of digital infrastructures in creating new opportunities for measurement and analysis, as well as the challenges inherent in existing collection standards and STI indicator quality

• provide new opportunities for collaboration and strengthen the dialogue between policy makers, data users and providers

• lead to a forward-looking and policy-relevant roadmap on STI measurement the OECD could implement in collaboration with its membership, other international organisations and experts.

More detailed information on the 2016 OECD Blue Sky Forum, including papers, presentations and videos, can be found at: http://oe.cd/blue-sky.

In defining the scope of this chapter, it bears noting that the STI field possesses unique features as a domain for policy analysis that have to do with the nature of knowledge and how it is created and diffused, leaving a limited number of traces. These features make it particularly difficult not only to measure STI concepts, but also often to demonstrate how the different elements are connected through cause and effect. The empirical study of innovation (defined in Box 14.2) and innovation policy faces the challenge of seeking to measure how activities that are themselves difficult to measure affect other outcomes that are also difficult to measure (Krugman, 2013). Besides, there are few instances in which experiment-like conditions arise naturally or can be reproduced in the empirical analysis of STI. The scope for experimentation in innovation policy has increased in recent years (Nesta, 2016), but remains limited.

Understanding the nature, outcomes and eventual impacts of science and innovation activities requires the ability to observe and understand action at multiple levels of analysis across the entire system (Lane et al., 2015). Over the least decades, the main objective of key actors in the world of STI data and statistics has been to ensure that the frameworks and tools used in this area align with the broadly espoused view of innovation as a highly interconnected and dynamic system (Soete, 2016).

Box 14.2. Defining and measuring innovation

Although firmly grounded in knowledge development, innovation requires implementation: innovations derive from the creation and application of ideas. As defined in the 2018 edition of the Oslo Manual (OECD/Eurostat, 2018): “An innovation is a new or improved product or process, or combination thereof, that differs significantly from the unit’s previous products or processes and that has been made available to potential users (product) or brought into use by the unit (process).”

This neutral, measurement-oriented definition can be applied to the economy and society as a whole. Innovations may succeed or fail in meeting their objectives. Over time, a diverse range of features and unforeseen effects may appear. From a policy perspective, understanding innovation is critical to understanding how policies can harness it to drive growth and well-being, while managing the potential downsides of individual innovations and broad innovation systems.

While these substantive conceptual and practical challenges may have resulted in lagging evidence for empirical analysis of STI compared to other data-rich areas (Bakshi and Mateos-García, 2016), they have not prevented producer and user communities from creating and applying conceptual frameworks, data infrastructures, indicators and analytical efforts to shed light on the functioning of STI systems. This chapter considers major developments, starting with the latest trends in policy uses of STI data and statistics. It then assesses how digitalisation has transformed the production of STI data, and examines the major issues facing traditional and new producers of STI data and statistics. It concludes by discussing the future outlook and key governance perspectives in STI measurement and analysis, including the potential role of the OECD.

New perspectives on the policy use of STI data, statistics and analysis

The STI measurement infrastructure has developed over time to serve the needs of science users and innovation policy makers for statistics and related analysis, often for advocacy purposes. As indicated in Chapter 6 on the digitalisation of science and innovation policy (DSIP), policy demand for data is broader, covering not only inputs to policy definition, but also the operational delivery and management of public services. In addition to policy makers, businesses and institutional managers increasingly rely on science and innovation data to support both their day-to-day and strategic decisions, e.g. when assessing the intellectual-property landscape before deciding to enter a market. However, published statistics and related research are generally too broad for management purposes; hence, their overall relevance is largely determined by how they influence public policy.

Changes in policy user interests following the crisis

In the immediate response to the 2008 global financial and economic crisis, most countries recognised the importance of sustaining economy-wide research and innovation capabilities. However, the crisis had a profound impact on the STI policy questions that guide decisions on data collection and analysis. Its very roots challenged the implicit assumption that innovation is inherently good and should be promoted, generating a heightened sense of responsibility about its outcomes and impacts on society. The ensuing wave of government financial austerity measures led to a tighter and more competitive budgetary environment, triggering requests for evidence that would allow STI policy makers to present the best possible case for economy and finance ministers to provide public support for STI. The raised bar for quality evidence across all competing policy areas entails demonstrating the economic return of investment in STI compared to other government investments.

The debates about the origins of innovation and the government’s role in relation to companies have intensified (Mazzucato, 2015). The generalised productivity slowdown has also renewed mainstream policy analysts’ interest in measuring the link between innovation and growth. They are questioning the tools used to measure both (Brynjolfsson and MacAfee, 2011; Gordon, 2016; Coyle, 2017) and ascertaining whether research productivity has been declining (Bloom et al., 2017).

In this context, conceptual models need empirical evidence to allow testing hypotheses about the operation of STI systems and the role of policies. Data development requires new models and concepts. Dosi (2016) humorously described the limits of theories and value measurement in STI as searching for “the marginal impact of an extra gram of butter to the taste of the cake”. However, major policy discussions still focus on explicit or implicit value and credit attribution. As econometric analysis and value-for-money assessments have entered the debate on science and innovation policy, and results are used to justify policies, a very significant change in the data and analysis landscape is taking place. Some question whether this could be the start of an “arms race” in reported impacts, and whether studies that report massive impacts could be generating unrealistic expectations of returns from STI investment. If true, this may well affect STI programmes with modest returns, as well as programmes that are harder to evaluate.

The struggle to communicate and act upon STI data and indicators

Communicating complex messages for policy making presents significant challenges. Policy makers demand simple indicators to monitor and benchmark STI systems that directly relate and can inform their key decisions. Experts meeting at Blue Sky described the role of STI policy makers as managing uncertainty. This requires an “options-thinking approach” – i.e. considering a wide range of small seed investments, which open opportunities for more decisive investments in specific areas that are chosen when uncertainty is resolved. Both quantitative data and narratives (“story telling”) are necessary to address uncertainty, and communicate the results from research on science and innovation to policy makers (Feldman, 2016).

Ministers participating in the Blue Sky Forum 2016 welcomed the opportunity to reflect on the data they use, and how they use them (Box 14.3). Some even reported being struck by the extent to which science and innovation can be driven by targets and composite indicators. They recognised the need to identify insightful indicators on science and innovation actors and their linkages, and examine the enabling infrastructure.

Box 14.3. Selected senior policy makers’ perspectives from the Blue Sky Forum 2016

There is a need to open and enrich the production of indicators towards a diversified set of skills, together with theoretical advancements regarding the use of indicators and the navigation through data sets. The next generation of data should guide new forms of international cooperation giving priority to science, education and mobility. Data is also needed to foster the collective action of governments, public institutions and the private sector to promote the diversification of education and research systems leading to technological change, as well as a participatory approach to science and innovation.

Manuel Heitor, Minister for Science, Technology and Higher Education, Portugal

The quality of our measurement needs to keep pace with our societies. The ability to compare ourselves internationally helps us immensely in our domestic policy decision-making. We must recognise the interaction between fundamental knowledge and innovation. We need to collaborate internationally to harmonise open data standards and use evidence to increase transparency, accountability and citizen engagement.

Kirsty Duncan, Minister of Science, Canada

When I became minister, I was struck by how much science and innovation policy is driven by indicators and targets. Researchers and universities change their behaviour in response to them. We need to keep under review how indicators are produced and used. Indicators can become less useful as they become more widely used. Good evidence is more important for policy makers at times of budgetary constraints.

Elke Sleurs, State Secretary with responsibility for Science Policy, Belgium

To achieve a real impact on our society, over the next ten years we should strive to maximise the value of the massive quantity of data available to us […] I firmly believe the next ten years will not be about producing data, as much as it will be about understanding data. […] If we are serious about the growth enhancing and job creating role of research and innovation, then we need to be able to demonstrate and prove those.

Carlos Moedas, European Commissioner for Research, Science and Innovation

Source: OECD (2016), Blue Sky Forum 2016, http://oe.cd/blue-sky.

Problems with indicator-driven science and innovation policy

The purpose of innovation data and indicators is to be used by policy makers – without that, the whole enterprise would be a failure. But could there be too much of a good thing? The potential misuse of indicators – which are commonly used to set targets – is one concern, and experts argue that indicator-driven policy should not be viewed as equivalent to evidence-based policy (Polt, 2016).

Targets without data are a common problem. Several domestic and international initiatives focusing on quantitative targets keep on being launched. In many cases it has soon transpired that suitable data for monitoring target fulfilment were not available for many of these initiatives, and no significant resources were devoted to addressing these gaps. In other instances, setting unrealistic targets without adequate analysis may undermine interest in measurement. It may also divert resources from activities that do not directly respond to the indicator – e.g. innovation efforts beyond research and development (R&D) that may be just as important to the objective. When R&D investment targets are difficult to meet, incentives may exist to blame the measurement effort and “shift the goalposts” (Bakshi et al., 2017), rather than question the choice of indicator.

A key concern is the potential abuse and misuse of STI indicators that oversimplify reality on the sole basis of what can easily be measured, obfuscating their interpretation and generating some complacency about what is and is not important (Martin, 2016). A majority of Blue Sky participants intensely criticised the use of composite science and innovation indexes, which combine multiple widely available indicators and rank countries’ or regions’ performance –implying that a higher value or rank is preferable from a societal perspective. Although they recognised the value of simplicity in communicating a high-level message, they saw considerable conceptual and practical problems in current practice.

Changing demands for STI data from a more sophisticated user base

The STI policy-making community has been increasingly attracting officials with backgrounds in various analytical disciplines and a good understanding of empirical evaluation tools and the importance of good data and hypothesis building to support them. Such an expert community, embedded within policy ministries and agencies, can ensure that data are considered at the appropriate stages of policy planning and evaluation. As part of building a culture for evaluating everything, policy makers are asked to impose the integration of data and evaluation requirements into programmes supporting science and innovation (Jaffe, 2016). In several instances – such as the disbursement of public funds – public interest and accountability for policy actions may override privacy concerns.

A greater focus on the human perspective

The policy community appears to agree on the need to place individuals and communities at the centre of science and innovation-related policy design and analysis. Citizen-based analytics are considered as enabling better targeting of public services (Gluckman, 2017). At the Blue Sky Forum 2016, ministers and other policy makers and experts strongly advocated for “human-centred policy design”, and called for systematic collection of data about individuals (Heitor, 2016). This represents a shift in the collection and use of evidence on STI systems from an organisation-based perspective towards considering the “human factor”, such as scientists’ decisions to return to their home country or work in industry, or their motivation for developing a new solution to a particular community problem. Information on career decisions can contribute to STI supply-side policies, but the dimension of individual and collective demand for innovation should also be considered. Policy questions and measurement efforts need to be framed in terms of society’s engagement in innovation systems and preparedness for the changes brought about by innovations.

The policy community is also underscoring the need to better characterise participatory R&D processes and agenda-setting on innovation policy. The goal is to help engage scientific institutions and actors with civil society – highlighting the need for collaboration with scientists, engineers and users, to understand the process and impact of knowledge production. The development of data toolkits can help explore choices in research landscapes and spur citizen participation in decision-making (Rafols et al., 2016), as well as raise popular buy-in for science and innovation.

Demand for more accessible, granular and linked data

Growing requests from general users of statistical data for more complex solutions, tailored to their needs and relatively simple to use, are another major trend (Peltola et al., 2017). Research and policy users of STI data and statistics with more advanced analytical capabilities expect data to be micro-based, with the finest possible degree of granularity, to support aggregate statements not only about a country or sector, but also about relatively small geographical areas, organisations, teams and even individuals. They demand infrastructures that allow accurately linking and using data for a range of statistical and research purposes (Hicks, 2010). Data linking has been significantly facilitated by improvements in information and communication technology – including advances in machine learning – but remains a significant challenge in the absence of universally adopted identifiers.

A move to more granular data raises potential issues: measurement can have a direct impact on individuals’ incentives and behaviour, potentially undermining its ultimate utility. Aggregate indicators serve as a guide on the functioning of entire systems and inform the policy direction. But what happens when disaggregated data are used to base more targeted action? A more sophisticated discussion needs to take place about the relationship between data, indicators and policy uses. Questions are raised about whether the statistical use of certain indicators at an aggregate level (as at the OECD and other international organisations) represents an implicit endorsement of the same indicators’ use at a far more disaggregated level, for non-statistical purposes. This is a common concern regarding specific bibliometric indicators, which are also often used for research assessment; this practice is criticised by signatories to the San Francisco Declaration on Research Assessment (DORA, 2012) and the Leiden Manifesto on Research Metrics (Hicks et al., 2015). It can be difficult to explain that the same indicator may be appropriate for one purpose, and not for the other. However, the consensus is that the underlying microdata need to be available, to facilitate the production of more complex and nuanced aggregate indicators and analyses, which in turn help break down and comprehend the various components of headline indicators.

Data beyond jurisdiction boundaries: Towards truly global STI statistics

Previous trends pointing to the increased globalisation of STI systems have continued – and possibly accelerated – over the past decade. Without the ability to map the creation and circulation of knowledge and related financial flows across countries, it is impossible to characterise science and innovation, or measure their drivers and impacts. However, generating STI statistics still relies on a largely national approach. Privileging the nation as the natural scale of analysis is a built-in bias of statistics, which years of economic change have progressively eroded (Davies, 2017).

What is preventing the adoption of truly global standards and measurement practices? Path dependencies imply substantial adjustment costs to measure STI phenomena similarly across countries; rendering jurisdiction-specific data truly interoperable and global is still a long way off. Born-global data require a more significant co-ordination effort, including reaching agreements on data sharing and standards. Global companies are partly occupying this space, because their cross-border activities are less constrained than those of official organisations. Policy makers should ask themselves whether commercial databases represent an appropriate basis for sustainable data infrastructures – at least to meet their global statistical needs.

Thanks to their global nature and the replication of best practices, science and research activities are becoming increasingly similar, implying that measurement methods should also converge. This may, however, not be the case for the innovation culture, which retains a strong local component (Bauer and Suerdem, 2016). Even in the age of global brands and hyper-connected individuals, innovation critically depends on social, spatial and historical contexts that are essentially local. Hence, integrated local approaches to measurement are needed to support common policy learning at the global level.

Innovation for the SDGs

One key recent driver of global statistical data on STI has been the world leaders’ definition and endorsement of the 17 Sustainable Development Goals (SDGs) in the context of the 2030 Sustainable Development Agenda. Improving the data is key to attaining these goals. As part of Target 9.5, countries have pledged to “enhance scientific research [and] upgrade the technological capabilities of industrial sectors in all countries” (United Nations, 2015). A key issue for STI policy makers is the ability to monitor the link between science and innovation on the one hand, and the entire range of global sustainability objectives – from poverty and hunger eradication, to equality and climate action – on the other. Those links are not easily traced or exposed solely by using indicators. Given the multidimensional nature of the SDGs, monitoring the overall role of science and innovation requires accumulating the findings from empirical studies of policy experiments around the world. Enhanced use of meta-analysis in this area should be further encouraged.

Timelier data

At the Blue Sky Forum 2016, EU Commissioner for Research, Science and Innovation Carlos Moedas worried that most of the data he used to take policy decisions was outdated (Moedas, 2016). The timing of STI evidence shapes the types of decisions it can support. Most STI indicators are fundamentally structural and exhibit limited variation over small periods; they can help inform the development of long-term plans to attain strategic goals. However, in times of rapidly disruptive change, timelier and frequent data become more critical, owing to the risks of basing decisions on information that is no longer relevant. Timeliness is also critical when measuring possibly short-lived processes, such as entrepreneurship and business dynamics.

In this context, many organisations consider applying nowcasting methods using complementary sources, instead of relying solely on models. Today, many surveys include questions about respondents’ intentions to invest in R&D or innovation over the current and coming year; this has become part of OECD statistical guidance (OECD, 2015; OECD/Eurostat, 2018). Data from quarterly and annual reports to investors and regulators informing about investment and product launches as well as hiring campaigns with new job descriptions can be obtained from online sources. This more frequent information creates the need to filter higher levels of “noise”, and identify the optimal balance between structural and conjunctural data. More active nowcasting also requires more tolerance from policy makers regarding revised statistical data, as commonly occurs when measuring key macroeconomic indicator (e.g. gross domestic product).

Digitalisation: The expanding frontier for STI data and statistics

Exploiting the digital trace of science and innovation

Digital technologies and data are transforming business processes, the economy and society. Digitalisation represents a major force for change in the generation and use of STI data and statistics. STI systems have become remarkably data-rich: information on innovation inputs and outputs that was only recorded in highly scattered paper-based sources is now much easier to retrieve, process and analyse. The use of digital tools by researchers and administrators leaves digital traces that can be used to develop new databases and applied to indicators and analysis. The digitalisation of the patent application and scientific publication processes has already provided rich and widely used data resources for statistical analysis. Digitalisation is rapidly extending to other types of administrative and corporate data, e.g. transactions (billing and payroll data); website content and use metadata; and generic and specialised social media, in which STI actors interact with their peers and society.

Data practitioners view these new “big data” as “uncomfortable data”, i.e. datasets that are too large to be handled by conventional tools and techniques (Alexander et al., 2015). The fuzzy boundary between qualitative and quantitative data is a striking example. Methodologies (e.g. user testing or interviews) traditionally considered as qualitative can now be conducted on a very large scale and quantified – text, images, sound and video can all be “read” by machines. Natural language-processing tools automate the processing of text data from thousands of survey responses or social media posts into quantifiable data. These techniques can help alleviate some of the common challenges facing STI statistics (e.g. survey fatigue and unfit-for-purpose classification systems applied differently by human coders) and generate adaptable indicators. Effective application of these methods relies on fit-for-purpose, high-quality systems to collect qualitative information consistently and avoid potential manipulation. Administrative database managers become important gatekeepers of data quality, but the information providers still need incentives.

Big data implies risks in exploiting datasets with possible defects and biases not recognised by the researchers; difficulties in evaluating big-data techniques and analysis, especially using conventional criteria (such as falsifiability); and complexities in explaining these techniques – and their value as evidence for policy evaluation – to decision-makers and the public. One case in point is altmetrics – where “alt” stands not only for “alternative”, but also “article-level” (Priem et al., 2011) – which offer great promise and attract plentiful attention. However, altmetric indicators are essentially citation-based and do not reflect actual use or impact; more concerningly, they can be easily gamed, e.g. by bots. Research shows that altmetric indicators do not broaden the geographic or cross-disciplinary dissemination of science, but they do help heighten the profile of authors whose work, for good or bad reasons, generates attention. Sugimoto et al. (2016) conclude that altmetrics have not been the expected panacea, and recommended: a) providing disaggregated (rather than composite) altmetric indicators; b) accounting for outliers and gaming; c) expanding both evidence and scholarship sources; and d) reframing the conversation around the meaning of “broader impacts”.

Moving progressively away from fixed scales of analysis (such as the nation) towards variable categories and dealing with vast new databases requires a different way of searching for patterns, trends, correlations and emergent discourses. Visualisations “mapping” interdependencies between individuals, teams, institutions and research domains can support the interactive exploration of large amounts of abstract data (Börner, 2010). These map-like representations – comprising nodes representing objects like researchers and their work with various labels, their positions and the highlighted edges connecting them – approximate a complex reality. However, their effective interpretation requires a qualitative narrative that is consistent with the underlying data, as well as visualisations of counterfactual scenarios. A rich research agenda is assessing how users assimilate statistical data, using various forms of experimentation and tests.

Surveys in the era of big data

Surveys are the cornerstone of statistical data – especially official data – on STI. Compared to data arising “organically” from administrative or commercial processes, survey data are “designed”. Two elements constrain their potential: asking questions of individuals and organisations relies both on respondents’ memories and formal records, and on their willingness to collaborate and provide truthful answers. The increased complexity of measurement constructs can sometimes exceed respondents’ attention span and reporting capacity; they may also lack incentives to keep records of the information that surveys (and the policy makers promoting them) aim to retrieve.

Some question whether the shift to big data is the precursor to the demise of surveys, while others, paraphrasing Mark Twain, will argue that reports of the death of surveys “are greatly exaggerated”. The manner in which surveys are carried out has indeed changed, as online surveys have largely displaced more expensive non-digital methods. Surveys are also more targeted towards areas where other data sources are less effective (Callegaro and Yang, 2018). The ease with which surveys can be conducted using electronic tools (including do-it-yourself survey platforms) has resulted in an explosion of surveys both in general and in the area of STI studies. A downside of this apparent “democratisation” is that these surveys often fail to meet basic statistical quality requirements, including for safeguarding privacy and confidentiality. This surge represents a growing source of fatigue for respondents; it results in lower expected response rates to non-compulsory (and compulsory, but difficult-to-enforce) surveys and may undermine trust. STI policy makers should co-ordinate and apply standards to their sponsored survey efforts.

Some of the transformational power of new STI data sources stems from their multidimensionality and possibilities to interconnect the different types of subjects and objects that are covered in them. The strengths of these organic data sources are hard to reproduce in surveys, which are traditionally conceived to identify key actors and the presence of pre-defined types of interactions, rather than trace those linkages. Digital solutions applied to survey tools can help address this gap. They are viewed as key components of the move towards “rich data” and are crucial to validating and augmenting the quality of big-data sources (Callegaro and Yang, 2018). Rather than competing with alternative sources, surveys will increasingly focus on the crucial information that cannot be obtained otherwise. Recent experience shows that trust and credibility will be the most crucial factors determining the success of survey efforts in the digital era.

A digitally enabled decentralisation of value measurement?

Building on the themes of trust and the new possibilities deriving from digital technologies, the STI community should think beyond current tools and data sources. It could explore, for example, the transformative potential of distributed ledger technology like blockchain in science and innovation systems (Soete, 2016). These technologies present alternatives to trusted third-party intermediary models aiming to assert “quality” or simply “truth” (a key function for government agencies), by shifting to tamper-proof models where that responsibility becomes distributed. It has been argued that countries or communities with weaker “rule-of-law” enforcement systems may find this particularly relevant. While the use of blockchain in STI and STI measurement remain more of an aspiration than a reality, it is worth reflecting on the potential role of collective intelligence when asserting the veracity and importance of the growing information contained in statistical data or indicators. STI indicators need to move beyond just counting items (i.e. papers, patents or even citations) so that they are able to distinguish contributions of different value.

Perspectives for producers of STI data and statistics

Data scientists and STI statistics

The great achievement of statisticians has been to reduce the complexity and fluidity of national populations into manageable and comprehensible facts and figures (Davies, 2017). Today, statistical skills are in high demand. Google’s chief economist, Hal Varian, foresaw this trend when he argued in 2009 that the “sexy job in the next 10 years” would be “statistician” (Varian, 2009). This prediction has come true for data scientists, a category of statisticians at the junction of software programming and decision sciences, who are equipped to marshal the current data-driven boom in artificial intelligence capabilities. Data scientists thrive on access to data and cumulative algorithmic knowledge. Much of the basic expertise in this area is accessible to anyone by tapping into online courses and other instructional material, but true expertise requires using data towards a defined goal. Data scientists are powering major developments across the entire STI analysis community, handling extremely challenging tasks such as data extraction, disambiguation, data linking and topic analysis. Many now work for companies that are changing the landscape of STI data and statistics, offering a combination of services to meet the needs of scientists, administrator, firms and policy makers. It has become increasingly common for policy organisations to tap into this community’s expertise, not only through conventional contracts, but also by organising problem-oriented prizes and challenges, like the disambiguation and visualisation challenges (e.g. the 2016 Cancer Moonshot Challenge) organised by the United States Patent and Trademark Office.

Official STI statistics at a crossroads

Following up on the pioneering efforts by academics and research organisations, national statistical organisations (NSOs) are now the backbone of many STI statistics. Working within a legal framework and applying minimum professional standards, their independence and objectivity has allowed them to attain official statistics status. Preserving privacy and confidentiality has been a longstanding concern and driver of official STI statistics. The highly concentrated nature of many STI activities is one of the main causes of the lack of granularity in published statistics as data gets aggregate to avoid disclosing information about individual organisations to preserve confidentiality.

A measure of the success of STI statistics is that economic statistics now incorporate several key dimensions of the “knowledge economy” in the System of National Accounts. A key lesson is that economic statistics can reflect STI issues, as long as tested data sources are available to address potential gaps. Accounting for innovation in the digital and knowledge-based economy is part of the new frontier.

However, NSOs have been experiencing major disruptions to their business models. Competition from alternative data sources is one factor: for example, many users seem willing to forego the limitations of publicly disclosed business data on R&D or other activities. NSOs struggle to hire young employees with high qualifications in data and statistics, as they are lured towards better-paying, cutting-edge companies. With few exceptions, governmental fiscal austerity has reduced the resources to maintain – let alone develop – new statistical data infrastructures. Process and product innovation within NSOs is constrained by day-to-day operational requirements, which absorb resources and leave little room for adapting business models to changing demands and new opportunities (Rozkrut, 2016). Too often, the role of NSOs is restricted to investigating their activities at the national level, making it difficult to address inherently global phenomena. NSOs need to exchange information and co-ordinate their activities to pursue their mission. They may not always consider customising national data for international comparison as a top priority, leading to many gaps in international statistics (such as those produced by OECD). A misguided drive for efficiency may leave little room for triangulating between different data sources that examine issues from different yet complementary perspectives (Bean, 2016).

As they struggle to capture more fluid identities, attitudes and economic pathways, traditional forms of statistical definition and classification are under strain. For example, what is a firm, an employee or a research field in the digital era? Efforts to represent socio-economic changes in terms of simple, well-recognised indicators risk losing legitimacy if users do not recognise themselves in them (Davies, 2017). As STI measurement moves increasingly into capturing individual behaviours and attitudes, respondents will also demand to define themselves in their own terms.

Nevertheless, the role of NSOs is largely considered critical to STI policy evidence. No other existing entities are endowed with the adequate formal authority and responsibilities to become viable alternatives to NSOs. NSOs are uniquely placed to objectively assess the reliability of new data sources and methods, and conduct representative statistical surveys. They can provide the clearing houses requested by researchers, combining information sources in ways other organisations cannot (National Research Council, 2014). It is therefore relevant for research-funding organisations supporting STI analysis to consider the role of NSOs, facilitating their work by providing the basic data infrastructure, and making it linkable and usable. While this represents a slight change from the traditional model, entirely centred on releasing aggregate statistics, many NSOs use the full potential of microdata infrastructures and partnerships to serve their core business needs, e.g. assess their own data quality. Lack of access lowers users’ interest in promising initiatives; it may lead to terminating otherwise promising developments requiring further national and global consolidation. For example, innovation surveys partly owe their survival and diffusion to conscious efforts to make microdata available for analysis. Researchers need to understand better how NSOs operate; they need to present their research proposals so as to deliver operational benefits to NSOs, which may otherwise struggle to support access infrastructures. Many participants at the Blue Sky Forum 2016 demonstrated how such partnerships can be built over time.

Engaging STI researchers and administrators in data development

Since its emergence as a recognisable form in the late 1950s, practised by a handful of people interested in the subject, science policy research is now an established discipline practised by thousands of researchers (Martin, 2016). A common question asked by the more data-oriented group within this community – a major source of ideas for developing and using data sources – is what it takes to scale up to the national level and compel NSOs to adopt these ideas in their statistical enquiries, at home and internationally. While the absorptive capacity of NSOs is limited, many academics have succeeded in putting forward persuasive cases for NSOs to experiment with new questions, often by demonstrating their feasibility and policy relevance.

Empirical researchers are often faced with either buying or making their own data. They find the increasingly commercial control over many STI sources challenging, since they have to secure resources to pay for licences covering proprietary data. Commercial control is also a potential hindrance for public-sector organisations, which often end up paying to access information about research they themselves funded, or that data companies have secured from public registers over the years. Researchers are also actively creating data, using different approaches and gaining more recognition for their efforts. Around the world, initiatives such as the Science of Science and Innovation Policy in the United States; Japan’s Science for RE-designing Science, Technology and Innovation Policy; and the EU Framework Programmes involve researchers in collaborative undertakings aiming to link the available data sources on STI and make them broadly accessible. The long-term sustainability of these databases after an initial wave of papers have been written is, however, still an open question, which may require deeper interaction with NSOs.

Administrative requirements and procedures are powerful enablers of statistical data in STI. Current research information systems (CRIS) contain a growing trove of valuable data that serve several purposes, including allowing universities to complete R&D surveys. Meanwhile, librarian collection-management needs have contributed to the emergence of bibliometrics. The social and policy drive for publicly funded institutions to manage “open science”, as well as ensure appropriate management of knowledge resources, positions CRIS managers as critical providers of research metadata – i.e. data about research and innovation (Chapter 6). Concurrently, institutional librarians, repository managers and administrators have been particularly active in developing and promoting common standards for CRIS (Chapter 12).

Infrastructures and tracing and interoperability standards have increasingly emerged not from traditional standard-development organisations, but from ad-hoc organisations and consortia formed for specific purposes. This broad movement stems from dissatisfaction with data that are inconsistently specified, country-specific, and prevent researchers and policy makers from sharing or linking cleaned datasets. Access to these data has been limited by a patchwork of laws, regulations and practices, which are unevenly applied and interpreted (Haak et al., 2012). The creation of open digital object identifiers for individuals, organisations, STI outputs (e.g. documents) and relational information (e.g. ownership and citation) is a key element of this new infrastructure. These efforts need to address strong incentives among actors to keep key information proprietary and attain market power. While outside the business sector database developments have been in most cases initially propped up by government (such as the platforms VIVO in the United States and Lattes in Brazil or the OpenAire or RISIS infrastructure projects), some initiatives are supported by philanthropic sources which have been increasingly promoting the creation of data and metadata commons that can help map and understand STI systems. Examples include the ORCID persistent researcher identifier (Chapter 12), opencorporates.com, a database to retrieve and map information on companies and their complex ownership-based interconnections (Tett, 2018) or lens.org, a tool for combined patent and scholarly search. Although generating statistical data and analysis is not the primary driver for these initiatives, they present considerable potential for applied statistical work. However, they are yet to be fully tested, and statistical representativeness is a key issue to consider for such purpose.

These new initiatives should aim for greater consistency with existing and new standards related to STI statistics. Statisticians should in turn take greater notice of them and major changes that are transforming what – and how – individuals and organisations can report on their STI activities. A starting point for convergence could involve NSOs testing these new resources within their regular internal business processes to assess their comprehensiveness and consistency, e.g. in terms of classifications. NSOs have been using all sorts of available registers and information sources for decades to keep their own sampling frames up to date.

Policy and governance perspectives for STI measurement

A call for action for the STI indicator and policy analysis community

The STI indicator community exists to create data and metrics to gain shared understanding, evaluate policy alternatives and identify gaps (Stern, 2016). Like certain crafts, evidence-based policy making is a mix of art and science (Harayama, 2016). While digitalisation brings closer the utopia of integrating evidence processes into decision-making, it also brings about significant changes in information markets and consumer behaviours. Today, digital platforms build relations, establish truth or falsity, and prioritise events (Baldacci and Pelagalli, 2017). In this new world, data are captured first and research questions come later, often leaving little room for high quality statistics and experts. Without good governance and data interpretation skills at all levels, this may in turn reduce statistics to the role of attention-grabbing “clickbait”.

The idea of a common public good is also worth defending. Like many other strands of statistics, innovation statistics were created as tools through which policy makers could view society, gradually developing into something in which academics, businesses and civil society also have a stake. As the business of innovation analytics grows, secrecy surrounding data methods and sources can be a competitive advantage, not to be given up voluntarily. In what Davies (2017) described as a move to a post-statistical society, policy makers need to consider that statistical “facts” may become privatised or diluted in the surrounding “noise”.

Long-term perspectives to move beyond what is easy to measure

The nature of data, statistical and quantitative analysis in the relatively young and multidisciplinary domain of innovation is such that significant time lags exist between the formulation of a new user need and the provision of a solution. In the meantime, priorities may shift to other subtopics, resulting in a misalignment of statistical evidence with user demands. It is therefore important not only to anticipate future user needs in order to develop new data sources, but also to secure and fully utilise the available data sources and infrastructures, in order to deal with unexpected and time-sensitive questions as they arise. This agenda requires considerable transparency and accountability in public policy (García, 2016), highlighting once more the interdependence between good data and statistics, and good policy.

Considering the entire value chain of STI data generation and use

Arguably, it is time to stop focusing on single indicators and consider instead the entire STI data value chain. The objective should be to consider both interdependencies that span the full data cycle and data reusability in different settings, possibly for initially unintended purposes and applications. Data play different roles, from feeding into agenda-setting and policy design, to supporting implementation and policy evaluation. The business case for data can be more easily articulated if all their uses and implications, including confidentiality and privacy, can be examined holistically. The cost of developing new sources to answer specific questions may be prohibitive, but linking different data sources can provide insights that could not be derived from working separately with the different components. Thus, several policy questions can be addressed not by collecting new data, but by meaningfully connecting existing sources.

Reaping the opportunities of digitalisation

The transformational potential of digitalisation is being felt in all dimensions of data production and use. The data revolution that facilitated the emergence of new STI evidence communities and actors has brought considerable dynamism and change to the field. However (as was already noted at the Blue Sky Forum 2006), much work is still being carried out in silos, which are difficult to connect. New tools and data sources need to be viewed as toolkits, rather than as silver bullets for policy makers. Looking ahead, the measurement community will need to utilise all types of data sources and methods to meet their objectives, and engage in partnerships to this end. Since Blue Sky 2016, Germany and several other countries have embarked on their own “Blue Sky initiatives” (expanding into measurement areas inspired by the experiences shared in Ghent), and Australia is currently embarking on a major review of its STI data and indicators.

The STI evidence community needs to address the persistent and significant disconnect between users and producers of STI data, statistics and analysis. Building capabilities and encouraging co-ordination among different actors will be necessary to allow new data infrastructures to emerge. Most solutions aiming to build infrastructures that transform evidence capabilities rely on social change rather than technology, underpinned by community engagement and trust building. Driving progress requires identifying the major obstacles to developing infrastructures – often starting within public administrations, where data are fragmented and synergies foregone. Lack of policy awareness can block improvements to the legal framework for data exchange and re-use. It can also hinder the implementation of sustainable “business models” for data, which consider the intended statistical use by policy makers.

Conclusions and future outlook

Progress in STI measurement is expected to continue to be incremental, based on refinement of existing tools and experimentation. Indeed, most of the issues and solutions identified at the Blue Sky Forum 2006 (see OECD, 2007; Gault, 2013) are still relevant more than ten years later, and will likely remain relevant in the future. As in many other areas of research on socio-economic systems characterised by reflexivity, it is difficult to dispel the perception that STI policy questions formulated several decades ago still lack conclusive evidence, even as new questions continue to emerge. However, the landscape in which data and statistics on STI are produced and used has undergone major transformations, creating new opportunities to match the growing challenges. Policy awareness and understanding of specific areas will be transformed by the availability of new data sources, and new opportunities to combine them with existing data. These data will often come from unanticipated sources. However, traditional organisations – such as NSOs – will continue to play major roles, albeit informed by new practices and methods.

By reflecting on its own achievements and shortcomings, including the role of the OECD (Box 14.4), the STI community is reaching growing consensus on the need to challenge assumptions and move beyond sterile debates (including debates opposing traditional data sources to new data sources; economic-impact measures to social-impact measures; and narratives to hard numbers).

In a desirable future scenario, an appropriate mix of instruments and disciplines will be used to address specific evidence needs and develop solutions that can globally scaled up to achieve international comparability and synergies in highly interconnected STI systems. Defining such needs requires a higher level of policy engagement with data producers, as well as a higher degree of literacy in empirical methods. A sign of engagement is that STI policy makers are increasingly able to use quantitative arguments in their discussions with peers from other policy areas.

Box 14.4. Blue Sky messages for future OECD work on STI data and indicators

Discussions at the Blue Sky Forum 2016 offered a series of reflections on the historical contribution and future role of the OECD, highlighting its leading role in promoting evidence-based STI policy both domestically and internationally. Some argued this could be achieved by reaching out to new actors presenting the interest and potential to transform STI data and statistics. The OECD could also contribute to national efforts to develop an evidence culture by: 1) empowering NSOs to access and use relevant commercial and administrative data; 2) enhancing the availability and interoperability of administrative data on science and research funding, not only to benefit statistical evidence, but also to enhance the governance of science and innovation systems; and 3) providing more hands-on guidance to practitioners. The OECD DSIP project (Chapter 12) brings together policy and data perspectives as a direct outcome of these Blue Sky discussions.

Blue Sky 2016 participants also recommended that the OECD consolidate and extend its work on defining standards, compiling statistical information, building a global infrastructure and instructing data users worldwide. They advised the OECD to focus on areas where it is uniquely placed to do so, and to prioritise global policy relevance and international comparability. In particular, they recommended that the OECD:

• prioritise collecting evidence on the role of individuals in STI systems

• secure statistical information directly from key STI actors worldwide, to identify key emerging challenges and possible responses in more timely fashion

• extend the framework for measuring innovation beyond business

• promote secure international infrastructures and institutional agreements that make it easier to link and analyse microdata sources, making intensified use of projects based on distributed analysis across countries when common infrastructures are not possible

• map public efforts to support research and innovation geared towards a range of societal objectives and challenges, and identify global funding gaps

• provide evidence on the incidence and impact of known and hidden forms of public support for innovation

• integrate research and innovation elements in economic statistics, and develop frameworks that help account for the contribution of investment in knowledge and its diffusion to economic performance within and across countries

• ensure that STI statistics capture globalisation and digitalisation phenomena, and demonstrate the vast interconnectedness and major interdependencies of global STI systems.

Realising a vision in which data and statistics on science and innovation become part of the mainstream requires concerted action to make this future possible and define its trajectory. The pressure will increase to displace, control or appropriate valuable public information to serve private interests. As public information goods, statistics will be used to assert or dispute facts about how innovation occurs and changes our societies. Aligning private and public interests in data and evidence will be a major test for the future governance of STI data and statistics. The Blue Sky process represents an extremely valuable vehicle for the international community to engage more broadly and sustainably in the run-up to the next Blue Sky Forum in 2026.

References

Alexander, J., M. Blackburn and D. Legan (2015), “Digitalization in R&D management: Big Data. Research”, Research-Technology Management, Vol. 58/6, pp. 45-49, Innovation Research Interchange and Routledge, Arlington, VA and Abingdon, United Kingdom.

Baldacci, E. and F. Pelagalli (2017), “Communication of statistics in post-truth society: The good, the bad and the ugly”, Statistical Working Paper, Eurostat, Publications Office of the European Union, Luxembourg, http://ec.europa.eu/eurostat/documents/3888793/8223142/KS-TC-17-005-EN-N.pdf/deae4e95-c2b0-43bd-b963-a362002db02c.

Bauer, M. and A. Suerdem (2016), “Relating ‘Science Culture’ and Innovation”, paper submitted to the OECD Blue Sky meeting on Science and Innovation Indicators – Theme: “Trust, culture, and citizens' engagement in science and innovation”, Ghent, https://www.oecd.org/sti/097%20-%20OECD%20Paper%20attitudes%20and%20innovation_v4.0_MB.pdf.

Bakshi, H. et al. (2017), “The new UK R&D target: Why it’s now time to move the measurement goalposts”, Nesta blog, London, https://www.nesta.org.uk/blog/the-new-uk-rd-target-why-its-now-time-to-move-the-measurement-goalposts/.

Bakshi, H. and J. Mateos-García (2016), “New Data for Innovation Policy”, paper submitted to the OECD Blue Sky Forum 2016, Ghent, http://www.oecd.org/sti/106%20-%20Bakhshi%20and%20Mateos-Garcia%202016%20-%20New%20Data%20for%20Innovation%20Policy.pdf.

Bean, C. (2016), “Independent Review of UK Economic Statistics: Final report”, UK Government, London, www.gov.uk/government/publications/independent-review-of-uk-economic-statistics-final-report.

Bloom, N. et al. (2017), “Are Ideas Getting Harder to Find?”, NBER Working Paper, No. 23782, National Bureau of Economic Research, Cambridge, MA, https://doi.org/10.3386/w23782.

Börner, K. (2010), Atlas of science. Visualizing what we know, MIT Press, Cambridge, MA.

Brynjolfsson, E. and A. McAfee (2011), Race against the Machine – How the Digital Revolution Is Accelerating Innovation, Driving Productivity, and Irreversibly Transforming Employment and the Economy, Digital Frontier Press, Lexington, MA.

Callegaro, M. and Y. Yang (2018). “The Role of Surveys in the Era of ‘Big Data’”, in Vannette, D. and J. Krosnick (eds.), The Palgrave Handbook of Survey Research, Palgrave Macmillan, Cham, https://doi.org/10.1007/978-3-319-54395-6_23.

Coyle, D. (2017), “Do-It-Youself digital: The Production Boundary and the Productivity Puzzle”, ESCoE Discussion Paper, Vol. 2017/1, 13 June 2017, Economic Statistics Centre of Excellence, London, http://dx.doi.org/10.2139/ssrn.2986725.

Davies, W. (2017), “How statistics lost their power – and why we should fear what comes next. The long read”, The Guardian, London, https://www.theguardian.com/politics/2017/jan/19/crisis-of-statistics-big-data-democracy.

DORA (2012), San Francisco Declaration on Research Assessment. Web version accessed from https://sfdora.org/read/.

Dosi, G. (2016), contribution to Blue Sky panel on “New models and tools for measuring science and innovation impacts”, OECD Blue Sky Forum, Ghent, http://ocde.streamakaci.com/blueskyagenda/.

Feldman, M. (2016), keynote address at Blue Sky panel on “Science and Innovation policy-making today: what big questions are begging for an answer?”, OECD Blue Sky Forum, Ghent, http://ocde.streamakaci.com/blueskyagenda/.

García, C. (2016), contribution to Blue Sky panel on “Looking forward: what data infrastructures and partnerships?”, OECD Blue Sky Forum, Ghent, http://ocde.streamakaci.com/blueskyagenda/.

Gault, F. (2013), Innovation indicators and measurement challenges, in: Fred Gault, (ed.), Handbook of Innovation Indicators and Measurement, Edward Elgar, Cheltenham, UK and Northampton, MA, USA, 441-464.

Gluckman, P. (2017), “Using evidence to inform social policy: the role of citizen-based analytics: A discussion paper”, Government of New Zealand and European Commission, Auckland and Brussels, www.pmcsa.org.nz/wp-content/uploads/17-06-19-Citizen-based-analytics.pdf.

Gordon, R.J. (2016), The Rise and Fall of American Growth: The U.S. Standard of Living since the Civil War, Princeton University Press, Princeton.

Haak, L. et al. (2012), “Standards and infrastructure for innovation data exchange”, Science, Vol. 338/6104, pp. 196-197, American Association for the Advancement of Science, Washington, DC, https://doi.org/10.1126/science.1221840.

Harayama, Y. (2016), contribution to Blue Sky panel on “The Blue Sky Agenda”, OECD Blue Sky Forum, Ghent, http://ocde.streamakaci.com/blueskyagenda/.

Heitor, M. (2016), “What do we need to measure to foster ‘Knowledge as Our Common Future?’”, position paper contributed to the OECD Blue Sky Forum, Ghent, http://oe.cd/heitorbluesky.

Hicks, D. (2010). “Systemic data infrastructure for innovation policy”, NSTC’s interagency task group on Science of Science Policy Science of Science Measurement Workshop, https://www.nsf.gov/sbe/sosp/tech/hicks.pdf.

Hicks, D. et al. (2015). “Bibliometrics: The Leiden Manifesto for research metrics” Nature News, Springer Nature. https://www.nature.com/news/bibliometrics-the-leiden-manifesto-for-research-metrics-1.17351.

Jaffe, A. (2016), “A Framework for Evaluating the Beneficial Impacts of Publicly Funded Research”, Motu Note 15, Motu Publications, Wellington, https://motu.nz/our-work/productivity-and-innovation/science-and-innovation-policy/a-framework-for-evaluating-the-beneficial-impacts-of-publicly-funded-research/.

Krugman, P. (2013). “The New Growth Fizzle”, The New York Times, New York, 18 August 2013, https://krugman.blogs.nytimes.com/2013/08/18/the-new-growth-fizzle/.

Lane, J et al. (2015), “New Linked Data on Research Investments: Scientific Workforce, Productivity, and Public Value”, NBER Working Paper, No. 20683, National Bureau of Economic Research, Cambridge, MA, http://www.nber.org/papers/w20683.

Marburger, J. (2011), “Why Policy Implementation Needs a Science of Science Policy”, in Fealing, K.H., J. Lane and S. Shipp (eds.), The Science of Science Policy: A Handbook, Stanford University Press, Redwood City, CA.

Marburger, J. (2007), “The Science of Science and Innovation Policy”, in Science, Technology and Innovation Indicators in a Changing World, Responding to Policy Needs, OECD Publishing, Paris, https://doi.org/10.1787/9789264039667-en.

Martin, B. (2016), “Twenty challenges for innovation studies”, Science and Public Policy, Vol. 43/3, pp. 432-450, Oxford University Press, Oxford, https://doi.org/10.1093/scipol/scv077.

Mazzucato, M. (2015), The Entrepreneurial State: Debunking Public vs. Private Sector Myths, Anthem Press, London and New York.

Moedas, C. (2016), “What new models and tools for measuring science and innovation impact?”, keynote speech provided at OECD Blue Sky Forum, Ghent, https://ec.europa.eu/commission/commissioners/2014-2019/moedas/announcements/what-new-models-and-tools-measuring-science-and-innovation-impact_en.

National Research Council (2014), Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy, The National Academies Press, Washington, DC, https://doi.org/10.17226/18606.

Nesta (2016), “Experimental innovation and growth policy: Why do we need it?”, Nesta, London, https://www.nesta.org.uk/report/experimental-innovation-and-growth-policy-why-do-we-need-it/.

OECD/Eurostat (2018), Oslo Manual 2018: Guidelines for Collecting, Reporting and Using Data on Innovation, 4th Edition, The Measurement of Scientific, Technological and Innovation Activities, OECD Publishing, Paris/Eurostat, Luxembourg, https://doi.org/10.1787/9789264304604-en.

OECD (2016), Blue Sky Forum 2016, Ghent, general website, http://oe.cd/blue-sky.

OECD (2015), Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development, The Measurement of Scientific, Technological and Innovation Activities, OECD Publishing, Paris, https://doi.org/10.1787/9789264239012-en.

OECD (2010a), The OECD Innovation Strategy: Getting a Head Start on Tomorrow, OECD Publishing, Paris, https://doi.org/10.1787/9789264083479-en.

OECD (2010b), Measuring Innovation: A New Perspective, OECD Publishing, Paris, https://doi.org/10.1787/9789264059474-en.

OECD (2007), Science, Technology and Innovation Indicators in a Changing World: Responding to Policy Needs, OECD Publishing, Paris, https://doi.org/10.1787/9789264039667-7-en.

Peltola, M.J. et al. (2016) “Value of official statistics”, OECD Statistics Newsletter, Issue 65, November 2016, https://issuu.com/oecd-stat-newsletter/docs/oecd-statistics-newsletter-11-2016/2.

Polt, W (2016), “Scope and limits of indicator use by STI policy”, contribution to OECD Blue Sky Forum panel discussion, Ghent.

Priem, J et al. (2011), Altmetrics: A manifesto (v 1.01), http://altmetrics.org/manifesto.

Rafols, I. et al. (2016), “Towards more inclusive S&T indicators: A review on efforts to improve STI measurements in ‘peripheral’ spaces”, paper submitted to the OECD Blue Sky Forum, Ghent, www.oecd.org/sti/086-2016-Rafols-OECDb.pdf.

Rozkrut, D. (2016), “New data and frontier tools: The challenge for official statistics in science and innovation”, contribution to the OECD Blue Sky Forum panel discussion, Ghent.

Soete, L. (2016), “A sky without horizons. Reflections: 10 years after”, keynote presentation at the OECD Blue Sky Forum, Ghent, https://www.slideshare.net/innovationoecd/soete-a-sky-without-horizons.

Stern, S. (2016), “Innovation-driven entrepreneurial ecosystems: A new agenda for measurement, policy, and action”, keynote presentation at the OECD Blue Sky Forum, Ghent, https://www.slideshare.net/innovationoecd/stern-innovation-driven-entrepreneurial-ecosystems-a-new-agenda-for-measurement-policy-and-action.

Sugimoto, C. et al. (2016), “Social Media Metrics as Indicators of Broader Impact”, paper submitted to the OECD Blue Sky Forum, Ghent, https://www.oecd.org/sti/172%20-%20SugimotoOECDaltmetrics.pdf.

Tett, G. (2018). “How to untangle the murky corporate web”, Financial Times, September 12.

Varian, H. (2009), “How the Web challenges managers?”, The McKinsey Quarterly, January 2009, McKinsey & Company, New York, https://www.mckinsey.com/industries/high-tech/our-insights/hal-varian-on-how-the-web-challenges-managers.

Thompson, W. (Lord Kelvin) (1883), “Electrical Units of Measurement” (3 May 1883), Popular Lectures, Vol. I, p. 73, https://archive.org/stream/popularlecturesa01kelvuoft#page/73/mode/1up%7C.

United Nations (2015), Sustainable Development Goals, United Nations, New York, https://sustainabledevelopment.un.org/?menu=1300.