Tackling Policy Challenges Through Public Sector Innovation

Annex A. Research methodologies

Research on enhancement-oriented innovation

The sources for the chapter on enhancement-oriented innovation were identified via a semi-systemic literature review made up of three phases. Firstly, the Web of Science and Scopus databases were explored by using key search terms related to enhancement-oriented innovation. In a second phase, the reference snowballing technique was adopted to find additional sources related to the relevant topics. Lastly, specific sources were obtained via conversations with experts in public sector productivity, workshops with practitioners from over 30 countries, and internal consultations with OECD innovation experts.

The initial search was conducted with the following search strings on both the Web of Science and Scopus databases:

• “public sector” +”innovation” +”efficien*”

• “public sector” +”innovation” +”productiv*”

• “public sector” +”innovation” +(“performance” AND “efficien*”)

To lower the number of results and increase their relevance, the keywords had to appear in either the title, the abstract or the article keywords. Eligible results were then further selected via the following criteria:

Year of Publication

Studies were included that were published during 2010 and 2021.

Language

Only studies written in English were included.

Publication status

Peer-reviewed articles, book chapters, books, or proceedings papers were counted as academic texts. Moreover, reports and grey literature were included.

Availability

Studies had to be available in full-text format (PDF).

The results were then manually screened and selected based on their relevance to the enhancement-oriented innovation facet. Together with the sources obtained from the additional expert inputs, a total of 197 sources were analysed for the review.

Research on adaptive innovation

The research on adaptive innovation was conducted based on the PRISMA method.

Identification of sources

Three main search strategies were used in the systematic literature review: First, Web of Science and ProQuest were used as the main data bases. Second, senior researchers in the agile and adaptive innovation field suggested further publications. To identify practical applications of the topic, a snowball-approach led to searching for grey literature published in the following data banks: OECD, IMF, World Bank. The initial search through the databases contained two search strings:

1. ((agil* OR adapt* OR "agil* innovat*" OR "adapt* innovat*") AND ("change" OR "change management" OR "reform" OR "change process" OR "upscal*" OR "implement*") AND (“public sector” OR “government” OR “public organi*ation”))

2. ((“citizen-led” OR “human-centered design”, OR “user-centric”) AND (“public sector” OR “government” OR “public organi*ation”))

Eligibility criteria

Studies from the original search were included in the systematic review if they met the following inclusion criteria: Abstract, title or keywords. In order to reduce the number of articles, the keywords had to appear in the abstract or title.

Year of publication

Studies were included that were published during 2000 and 2021. The Agile Manifesto (2001) later adopted into public administration research was published in 2001.

Field of study

Studies had to deal with agile and adaptive innovation and change in the context of the public sector. “Public sector” was defined as “those parts of the economy that are either in state ownership or under contract to the state, plus those parts that are regulated or subsidised in the public context” (Flynn, 2007: 2). Furthermore, within the Web of Science Core Collection data bank studies had to be published in journals related to the following categories: Public Administration, Management, Computer Science Software Engineering, Business, Information Science Library Science. Journals from management and computer science were included to cover the history of agile approaches with strong roots in software development and later adopted in organisational studies. Within the ProQuest data bank, results were filtered by theme (public sector/public administration).

Language

Only studies written in English were included.

Publication status

Peer-reviewed articles, book chapters, books, or proceedings papers were counted as academic texts. Moreover, reports and grey literature were included.

Availability

Studies had to be available in full-text format (PDF).

Figure A A.1.  PRISMA flow chart: research on adaptive innovation

Research on mission-oriented innovation

The research on mission-oriented innovation is based on the PRISMA method.

Identification of sources

The review is based on the PRISMA reporting flow. In summary, the study screening and selection identified a total of 619 references, excluded 389 references, yielding a corpus of 230 references for inclusion in qualitative synthesis (see Figure A A.2).

Figure A A.2.  PRISMA flow diagram of study's screening and selection

The initial search was carried on titles and abstracts for Boolean search strings. The search strings were clustered around two sets of issues: first, articles discussing innovation and innovation policy and missions, societal challenges, SDGs, transformation and transition; second, articles discussing STI organisations, new ways of working (e.g., service design) and capabilities.

The aim of using multiple keywords was to ensure the broad inclusion of literature over the past two decades and across various disciplines. The data sources used in this study include the following electronic databases and manual sources:

Core electronic databases: Scopus, Web of Science, EBSCO, ProQuest, Wiley

Manual searching: Google Scholar, hand scanning of related websites, e.g. OECD, European Union, Nesta UK, etc.

Search results were exported to a citation format and imported into a common repository hosted on EPPI Web Reviewer. The first parse of records screening on the EPPI platform identified and removed duplicates automatically. The remaining record set was manually screened by two independent reviewers and quality checked by the method of spot sampling and differences in resolution by a third reviewer. The study protocol for the screening activity was defined with three exclusion criteria (see Table A A.1) to prune records from the record set that did not fulfil the basic scope of the study. Where it was not possible to assess a criterion, the record remained was not excluded. The criterion would be assessed later based on the review of the reference’s full text.

Table A A.1.  Screening exclusion criteria for public procurement and innovation

Exclusion criteria	Description
Duplicates	A duplicated reference from multiple search results and cases where both book chapter and book exist, or a conference proceeding was also a journal publication. In such cases, only one is included in the record set.
Not mission-oriented innovation	Does not focus specifically on mission-oriented innovation, broadly defined.
Not STI agencies	Does not feature a discussion of STI agencies.

Year of Publication

The scope of the systematic review consisted of academic papers and book chapters and publications from policy organisations (e.g., the OECD) between 1990 and 2020.

Publication status

The eligible set of records were imported into a Zotero shared reference collection for further processing. A full-text assessment of each record identified a further 28 records for exclusion. Reasons for exclusion were either due to screening exclusion criteria (see Table A A.1) or the full-text document was unavailable in English.

A final total of 230 records satisfied the eligibility criteria in the study’s PRISMA protocol for inclusion in the quantitative and qualitative analysis.

The 230 full-texts were then coded for keywords based on the research questions above and summarised in Table A A.2.

Table A A.2.  Summary of keyword coding taxonomy

Policy targets	Policy logics	Policy tools and methods	Policy capabilities
Attaining SDGs Tackling societal challenges Enabling smart specialisation Systems transformation Social innovation	Creating missions Shaping / creating markets Complexity	Stakeholder engagement Using policy mixes Cross-sectoral coordination Experimentation	Public (dynamic) capabilities Governance Labs Entrepreneurial funders and change agents New agencies

Assumptions applied to coding

Under policy targets, the guiding question is what kind of new policy targets are emerging that go beyond growth and competitiveness narratives (Schot and Steinmueller, 2018[1]). In order words, under this category, we should find the key drivers of mission-oriented innovation in the public sector. Closely associated with policy targets is the category of policy logics or epistemic frames. Market failure based STI policies that dominate the current policy landscape are often criticised for their shortcomings and indeed closeness to neoclassical economics (Lundvall, 2007[2]; Mazzucato, 2016[3]; Fagerberg, 2018[4]). Accordingly, we could expect an emergence of alternative policy logic underlying missions-oriented policies. For policy tools and methods we were interested in support structures, tools and methods associated with new policy targets (such as societal challenges) and policy logics (such as market-shaping). We were also interested in understanding whether new policy targets and logics and accompanying support structures, methods and tools, lead to new capabilities in public organisations. These were categorise these as policy capacities, typically defined as the set of skills, capabilities and resources necessary to perform policy functions, from the provision of public services to policy design and implementation (Karo and Kattel, 2018[5]; Wu, Ramesh and Howlett, 2018[6]).

The overall set of sources is characterised by two key features: first, out of 230 sources, 161 are academic articles or book chapters; and second, most sources were published in or after 2015. As many European countries have started to develop explicitly mission-oriented policies in the aftermath of the European Commission’s new Horizon Europe R&D programme published in 2020, it is to be expected that many sources in this review discuss missions from either a conceptual or normative standpoint. We would expect relatively few studies detailing more applied aspects of missions. Indeed, the academic sources tend to be mostly conceptual studies rather than empirical investigations of actual missions practices.

Research on anticipatory innovation

The sources for the chapter on anticipatory innovation were identified via a semi-systematic literature review made up of three phases. Firstly, existing seminal secondary and tertiary sources on anticipation and strategic foresight were consulted for the sources of their own analysis. This allowed for a comprehensive overview of the knowledge that has been created already in the field of anticipation and public sector innovation. In a second phase, specific sources were obtained via consultation of academic programmes and syllabi on anticipation, foresight, and futures literacy; as well as via conversations with experts in foresight, workshops with practitioners from over 30 countries, and internal consultations with OECD experts in multiple domains. Lastly, the reference snowballing technique was adopted to find additional sources related to the relevant topics.

Language

Only studies written in English were included.

Publication status

Studies included journal articles, as well as books and book chapters.

Availability

Studies had to be available in full-text format (PDF).

The results were then manually screened of and selected based on their relevance to the anticipatory innovation facet. Together with the sources obtained from the additional expert inputs, a total of 148 sources were analysed for the review.

Research on portfolios of innovation

The sources for the chapter on innovation portfolio management were identified via a semi-systematic literature review made up of three phases. Firstly, the Web of Science and Scopus databases were explored by using key search terms related to innovation portfolio management. In a second phase, the reference snowballing technique was adopted to find additional sources related to the relevant topics. Lastly, specific sources were obtained via conversations with experts in public sector productivity, workshops with practitioners from over 30 countries, and internal consultations with OECD innovation experts.

The initial search was conducted with the following search strings on both the Web of Science and Scopus databases:

• “public sector” +”innovation” +”portfolio”

• “public sector” +”innovation” +”manage*”

• “public sector” +”innovation” + (“portfolio” AND ”tool”).

To lower the number of results and increase their relevance, the keywords had to appear in either the title, the abstract or the article keywords. Eligible results were then further selected via the following criteria:

Year of Publication 

Studies were included that were published during 1957 and 2021. 

Language

Only studies written in English were included.

Publication status 

Peer-reviewed articles, book chapters, books, or proceedings papers were counted as academic texts. Moreover, reports and grey literature were included.

Availability 

Studies had to be available in full-text format (PDF).

The results were then manually screened and selected based on their relevance to the innovation portfolio management topic. Together with the sources obtained from the additional expert inputs, a total of 182 sources were analysed for the review.

References

[4] Fagerberg, J. (2018), “Mobilizing innovation for sustainability transitions: A comment on transformative innovation policy”, Research Policy, Vol. 47/9, pp. 1568-1576.

[5] Karo, E. and R. Kattel (2018), “Innovation and the state: towards an evolutionary theory of policy capacity”, in Policy Capacity and Governance, Palgrave Macmillan, Cham.

[7] Kaur, M. et al. (2022), “Innovative capacity of governments: A systemic framework”, OECD Working Papers on Public Governance, No. 51, OECD Publishing, Paris, https://doi.org/10.1787/52389006-en.

[2] Lundvall, B. (2007), “National innovation systems - Analytical concept and development tool”, Industry and Innovation, Vol. 14/1, pp. 95-119.

[3] Mazzucato, M. (2016), “From market fixing to market-creating: A new framework for innovation policy”, Industry and Innovation, Vol. 23/2, pp. 140-156.

[1] Schot, J. and W. Steinmueller (2018), “Three frames for innovation policy: R&D, systems of innovation and transformative change”, Research Policy, Vol. 47/9, pp. 1554-1567.

[6] Wu, X., M. Ramesh and M. Howlett (2018), “Policy capacity: Conceptual framework and essential components”, in Policy Capacity and Governance, Palgrave Macmillan, Cham.