This chapter describes the significant role that public procurement can have in leveraging secondary policy objectives. It assesses the extent to which Adherents are balancing the pursuit of such secondary policy objectives against the primary procurement objective of achieving value for money. The analysis focuses on the trade-offs that Adherents are making throughout the procurement lifecycle to realise the desired outcomes from the application of broad strategic policy objective. The chapter includes an assessment on the existence of strategies on Green public procurement, innovation goods and services, support to small-to-medium enterprises (SMES), responsible business conduct, and women owned enterprises.
Reforming Public Procurement
1. Strategic uses of public procurement to achieve broader policy outcomes
Abstract
The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law.
The OECD Recommendation calls on Adherents to “recognise that any use of the public procurement system to pursue secondary policy objectives should be balanced against the primary procurement objective” (Principle on balance, paragraph V). The Recommendation contains guiding principles to assist Adherents in achieving the right balance between the primary procurement objective and secondary policy objectives, so that public procurement systems support the achievement of broader outcomes.
In order to realise the desired outcomes from application of broad strategic policy objectives, public procurement regulatory frameworks are themselves being implemented in more strategic ways. New public procurement methods are being tested and there is increasing use of digital technologies throughout the public procurement life cycle.
Helping public procurement organisations succeed in their pursuit of strategic procurement objectives is a key aim of the Recommendation. The challenges of implementing strategic public procurement are many: reducing risk aversion, setting up new forms of co-ordination and collaboration, improving skills and capacity, encouraging procurement officials to dialogue with suppliers, and enhancing data collection and monitoring of results.
In addition, other initiatives and policies are being implemented in regions with goals similar or complementary to those of the OECD Recommendation, and these contribute to the effective implementation of the Recommendation. For example, the European Commission (EC) is developing policies to support implementation of the European Union (EU) legal framework for public procurement as well as providing tools to make the procurement process more efficient (e.g. e-procurement tools, guidelines and templates). The EC also provides assistance and support to non-EU Member States in pre‑accession phase or in the context of neighbourhood co-operation.
Box 1.1. The European Commission’s Communication on Public Procurement Strategy
In 2017 the European Commission (EC) adopted a Communication on Public Procurement Strategy that called for a wide partnership with the Member States. The intent was to make procurement work on the ground in six priority areas:
1. greater incorporation of innovative, Green and social criteria in awarding public contracts, through publishing specific guidance (such as the Buying Green! handbook, “Guidance on innovation procurement” (European Commission, 2018[1]) and the upcoming guide on taking account of social considerations in public procurement aspects
2. professionalisation of public buyers with the 2017 Recommendation on to the EU Member States highlighting the steps they should take to ensure that public buyers have the business skills, technical knowledge and procedural understanding needed to comply with the rules and develop a more strategic approach to procurement
3. improving SME access to procurement markets in the EU and by EU companies in third countries, including facilitation of access to procurement by social enterprises
4. increasing transparency, integrity and quality of procurement data, notably by promoting national contract registers
5. digitisation of procurement processes with the development of e-procurement tools
6. more co-operation among public buyers across the EU, notably with high-level training for central purchasing bodies.
As part of the public procurement strategy, the Commission also set up a voluntary mechanism to assess and support large infrastructure projects.
Source: (European Commission, n.d.[2]).
Findings in the 2018 Survey (OECD, 2018[3]) demonstrate the Recommendation’s relevance in providing a vision for the holistic and inclusive use of public procurement, and confirms the value of investing in strategic public procurement as a key means to address societal challenges (OECD, 2018[3]).
The 2018 Survey shows that the majority of respondent countries have developed policies at some level regarding Green and innovative public procurement and SMEs (Figure 1.1). The comparison between the 2014/16 and 2018 Surveys shows that there is an upwards trend in the development of strategies and polices addressing Green public procurement and particularly responsible business conduct. The increase in policies to pursue responsible business conduct has been mainly at the central level, which could reflect the increasing need to co-ordinate this strategy from a centralised governance position (OECD, 2016[4]), (OECD, 2018[3]).
Many Adherents rely on their body of law to enforce the implementation of strategic public procurement. Some Adherents, such as Sweden, have also moved to define the strategic direction further by setting out a policy framework that includes specific objectives. Sweden in fact developed a National Public Procurement Strategy in response to the challenges faced by its public sector (Box 1.2) – challenges that are common to most public sector organisations around the world. These included changes in the world such as climate, the environment, demographic trends and migration flows, and citizens’ expectations of service. The strategy is designed to be a catalyst for new ideas in the public sector.
Box 1.2. Sweden – Sound public affairs for a sustainable future
The National Public Procurement Strategy in Sweden includes six policy objectives:
1. public procurement as a strategic tool for doing good business
2. effective public purchasing
3. a multiplicity of suppliers and well-functioning competition
4. legally certain public procurement
5. public procurement that drives innovation and promotes alternative solutions
6. public procurement that is environmentally responsible.
1.1. Supporting implementation of public procurement to encourage innovation
In line with the provisions of the Recommendation (Principle on balance, paragraph V), public procurement should be used as a strategic instrument to promote innovation. Many countries are taking measures to support that principle by developing action plans, either contained within broader innovative or procurement strategies or as stand-alone initiatives (OECD, 2018[3]). In line with the 2030 Agenda for Sustainable Development Goals (OECD, 2016[7]), public procurement’s strategic potential for achieving government policy goals such as innovation is increasingly recognised (OECD, 2017[8]).
As noted earlier, a large percentage of respondents (81%, including 69% at a central level) have strategies and policies in place to support the objective of procuring innovative goods and services (OECD, 2018[3]) . A report containing an overview of the findings from the 2015 OECD Survey on Strategic Procurement for Innovation (OECD, 2017[8]) noted that public procurement continues to direct efforts in the traditional area of the supply side by ensuring that the private sector operates in an environment conducive to innovation. Increasingly however, public procurement for innovation is at the forefront of developing policies for the demand side. Innovations materialise when there is a demand for innovation -- effective policies to support innovation are therefore coming from the demand side as well as the supply side.
Using public procurement to meet societal needs can be seen in targeted, demand-side innovation policies, such as anticipating future investments to address existing or future societal challenges; this allows potential vendors to enter the market with new innovative goods or services. Indeed, public procurement has the potential to be a catalyst for innovative solutions to pressing challenges. In a world of digital transformation, public sector investment decisions are becoming increasingly complex. Digital services must respond to citizens' fast-changing expectations, which requires crosscutting actions and integrated decisions. “On the supply side, improved Internet access and speed mean that governments have access to ever cheaper and more modular, usually cloud-based, services. But emerging technologies introduce also new uncertainties (dominant standards) and new issues that need to be managed by governments (e.g. data ownership and sovereignty, tendering and management of contractual relations, exit strategies, transitioning from legacy systems)” (OECD, 2018[9])
In New Zealand, innovation to solve public sector challenges has been realised through a contract for government banking services. This contract builds on recent innovations in the banking sector. A unique public and private sector collaboration in New Zealand is delivering innovative solutions that arose out of a public procurement procedure underpinned by a set of strategic objectives including the digital transformation of the New Zealand Government (Box 1.3).
Box 1.3. Improving existing public services in New Zealand through innovation
The Innovation Fund in New Zealand, born out of the All-of-Government banking tender, set up a joint initiative involving one of the winning bidders, Westpac New Zealand, and the New Zealand Government. The Innovation Fund invests in proposals that are expected to drive value for both the public sector and Westpac. The aim of the fund is to help create services and experiences that help grow a better and more innovative country. The fund can be accessed by contracting authorities across the New Zealand government sector, and has encouraged collaborative approaches in the form of innovation labs, hackathons and accelerators.
The fund has been used to help several projects, including:
An artificial intelligence bot to assist Ministry for Primary Industries staff with simple biometric questions. The Virtual Assistant Interface (Vai) has been initially stationed at the Auckland International Airport’s biosecurity arrivals area to answer visiting passengers’ questions that do not require human interaction.
To help the New Zealand Government deliver on its Cyber Security Strategy, a Cyber Security Safety Audit service, designed using innovation methodology with public procurement woven in to address the problem of security breaches in the country’s small businesses.
The expansion of smart customer onboarding software to help small business in New Zealand by making compliance with anti-money laundering measures easier through co-funding.
Enabling a team from the University of Auckland to investigate historical and anonymised bank transaction data, to see if they can identify spending patterns in a community following a disease outbreak.
Source: (Innovation Fund, 2018[10])]; (ZDNet, 2018[11]).
The 2015 OECD Survey on Strategic Procurement for Innovation found that comprehensive programmes at the national level are the second most used instrument to support strategic procurement for innovation behind policy instruments (OECD, 2017[8]). One such collaborative project is the PPI2Innovate project, which aims to build regional capacities in public procurement of innovative solutions (PPI) by directly targeting public procurers at all administrative levels in central Europe (Box 1.4).
Box 1.4. The PPI2Innovate project
The PPI2Innovate project is designed to use capacity building to boost usage of public procurement innovation in Central Europe. The project began on 1 June 2016 and continued until 31 May 2019 within the European Union-funded programme “Interreg CENTRAL EUROPE”.
It consists of a consortium of 10 partners from six central European countries (Hungary, the Czech Republic, Poland, Italy, Slovenia and Croatia), with eight associated partners. The project is aimed at building regional capacities in public procurement innovation, changing attitudes towards public procurement innovation, strengthening linkages among relevant stakeholders in regional innovation systems, and consequently boosting usage of public procurement innovation in Central Europe.
The outputs of the project are three thematic PPI2Innovate tools (Smart Health, Smart Energy and Smart ICT), which will be fully customised to the six national institutional frameworks and translated into each national language.
In addition, there will be six action plans for the operation of Competence Centres to be established by networking partners covering the regional level in Poland (RARR), Italy (University of Torino – UNITO) and Hungary (CTRIA- Central Transdanubian Regional Innovation Agency), and the national level in Slovenia (ICT Technology Network Institute), Croatia (BICRO) and the Czech Republic (DEX Innovation Centre). Other activities include:
a Central Europe network of PPI2Innovate competence centres
training of new members of the PPI network
PPI pilots in the energy, health and ICT sectors in Hungary (Somogy County), Italy (Piedmont Region), Poland (Lubin) and Slovenia (Ministry of Public Administration).
various workshops for contracting authorities organised by project partners.
Source: (OECD, 2018[3]).
Innovation procurement can be defined in many different ways, and it is a challenge for public procurement procedures and the professionals delivering them to keep up. The whole public procurement cycle is developing and adapting to the future needs of stakeholders that are seeking out innovative approaches to solving problems. Procurement of innovation is often related to a concrete need or demand. In the 2015 OECD Survey on Strategic Procurement for Innovation, the reasons that countries chose to implement procurement for innovation fell into two categories:
(most commonly) the need for goods or services that were not yet available to match demand, and therefore required a specialised, new good or service as opposed to an improved good or service
improving the performance of existing products or services such as producing total cost savings and/or energy efficiency and risk reduction (OECD, 2017[8]).
In Mexico the Ministry of Economy created a programme to drive innovation through public procurement. The programme was designed to promote innovation especially within micro, small and medium-sized firms (Box 1.5). It was also intended to improve public services through innovative products and solutions (OECD, 2017[8]).
Box 1.5. Innovative digital solutions in Mexico
In 2015 the General Directorate of Information Technology of the President’s Office in Mexico, in collaboration with other agencies of the Federal Public Administration, launched the project “Public Challenges”. Through an online platform, Mexican companies were invited to compete by offering innovative digital solutions to problems related to the environment, health education, transportation, food, connectivity and prevention. In total, 15 Public Challenges were launched and 341 proposals were received. In each challenge, five finalists selected by a non-governmental committee received a grant to develop a functional prototype, and the best one was awarded a contract to fully develop the selected project.
Following the Public Challenges, a Working Group on Innovation Procurement was created in order to strengthen the implementation of innovation procurement policy. The main objective of this working group is to generate policies to mitigate the risk of adopting innovation, and to propose modifications to the current legal framework for procurement to facilitate innovation.
Source: (OECD, 2017[8]).
1.1.1. Using procurement procedures to encourage competition in the area of innovation
One of the largest obstacles that countries must overcome when aiming to increase innovation is the tendency to use the award criterion of lowest price (OECD, 2017[8]). The EU Directive 2014/24 stipulates that contracting authorities should apply award criteria corresponding to the most economically advantageous tender (MEAT) (European Union, 2014[12]). Using this approach, weighted criteria within the best price-quality ratio (BPQR) can be included for tenders. The method allows contracting authorities to consider criteria that can include qualitative, environmental and/or social aspects. Examples of such aspects could include quality, technical merit, social and environmental characteristics, qualification and experience of supplier staff, after-sales service, and technical assistance and delivery conditions.
Encouraging the inclusion of secondary policy objectives as part of the award criteria as opposed to drafting descriptive technical specifications is a way to stimulate the market to offer innovative solutions. By describing a detailed technical solution, economic operators are unlikely to submit tenders that substantially exceed the minimum requirements as they will be aware that a cheaper solution – one that is less innovative but still within the minimum requirements – may be more likely to succeed. In such cases the competition is restricted to the price-quality ratio and secondary policy objectives will usually be a small component of the overall weighting.
Using the MEAT criteria along with the life cycle cost (LCC) method can support innovation outcomes. Life cycle costing tends to broaden the scope and can include the cost of externalities (the shadow price) related to upstream and downstream activities (Baron, 2016[13]).
In the 2018 Survey, 48% of respondents include award criteria related to innovative goods and services about 25% of the time (Table 1.1). The percentage of respondents in this category has increased between 2016 and 2018. More respondents seem to be integrating such award criteria into procurement procedures but only for specific procurement processes. Most of the respondents are from the countries’ CPBs, and so the data relate mainly to them rather than contracting authorities (OECD, 2018[3]).
Table 1.1. How often do CPBs use award criteria related to Green public procurement and to innovative goods and services?
|
Green public procurement |
Innovative goods and services |
||
---|---|---|---|---|
|
2016 |
2018 |
2016 |
2018 |
Australia |
".." |
".." |
".." |
".." |
Austria |
○ |
○ |
● |
● |
Belgium |
".." |
♦ |
".." |
● |
Canada |
♦ |
♦ |
● |
● |
Chile |
■ |
○ |
● |
● |
Czech Republic |
".." |
".." |
".." |
".." |
Denmark |
■ |
■ |
♦ |
♦ |
Estonia |
♦ |
♦ |
♦ |
♦ |
Finland |
□ |
□ |
○ |
○ |
France |
".." |
♦ |
".." |
♦ |
Germany |
○ |
○ |
♦ |
♦ |
Greece |
♦ |
♦ |
● |
● |
Hungary |
♦ |
♦ |
♦ |
♦ |
Iceland |
♦ |
♦ |
♦ |
♦ |
Ireland |
".." |
".." |
".." |
".." |
Israel |
■ |
■ |
♦ |
♦ |
Italy |
■ |
■ |
■ |
♦ |
Japan |
Non applicable: No CPB |
|||
Korea |
♦ |
♦ |
■ |
■ |
Latvia |
♦ |
♦ |
● |
● |
Lithuania |
".." |
♦ |
".." |
♦ |
Luxembourg |
".." |
".." |
".." |
".." |
Mexico |
● |
● |
● |
● |
Netherlands |
Non applicable: No CPB |
|||
New Zealand |
○ |
○ |
○ |
○ |
Norway |
■ |
■ |
● |
♦ |
Poland |
".." |
".." |
".." |
".." |
Portugal |
■ |
■ |
● |
● |
Slovak Republic |
♦ |
♦ |
● |
♦ |
Slovenia |
○ |
□ |
● |
♦ |
Spain |
■ |
■ |
○ |
○ |
Sweden |
".." |
".." |
".." |
".." |
Switzerland |
".." |
".." |
".." |
".." |
Turkey |
♦ |
♦ |
♦ |
♦ |
United Kingdom |
○ |
".." |
○ |
".." |
United States |
".." |
".." |
".." |
".." |
OECD Total |
||||
● Never |
1 |
1 |
10 |
9 |
♦ Seldom (About 25% of the time) |
9 |
12 |
7 |
13 |
○ Sometimes (About 50% of the time) |
5 |
4 |
4 |
2 |
■ Often (About 75% of the time) |
7 |
6 |
2 |
1 |
□ Always |
1 |
2 |
0 |
0 |
".." No information available |
11 |
9 |
11 |
9 |
Non applicable: No CPB |
1 |
1 |
2 |
9 |
Costa Rica |
○ |
■ |
♦ |
♦ |
Morocco |
".." |
♦ |
".." |
♦ |
Peru |
".." |
♦ |
".." |
● |
Source: (OECD, 2016[4]), (OECD, 2018[3]).
1.1.2. Digital transformation in public procurement – the path to greater effectiveness, efficiency and transformation
The Recommendation calls upon Adherents to use information and communication technologies to “drive cost savings and integrate public procurement and public finance information” and to “employ recent digital technology developments that allow integrated e-procurement solutions covering the public procurement cycle” (Principle on e-procurement, paragraph VIII, i). E-procurement systems collecting consistent, up-to-date and reliable data on procurement processes can feed into other government information technology (IT) systems through automated data exchanges, reducing risks of mistakes, errors and duplication. Meanwhile, integration with other digital government systems such as digital invoicing is essential to make e-procurement systems fully functional during all phases of the procurement cycle.
OECD survey data suggest that Adherents are increasingly integrating their e-procurement systems with other government IT systems, such as budgeting interfaces, business and tax registries, social security databases, public financial systems and enterprise resource planning (ERP) systems. While only 40% of respondents reported some kind of integration with other government IT systems in the 2016 Survey, this percentage increased to 69% in the 2018 Survey. The vast majority (63% of respondents) had their e-procurement system integrated with other central government IT systems, while three respondents (Belgium, Australia and New Zealand, corresponding to around 9% of respondents) have e-procurement systems integrated to the ERP of contracting authorities (Figure 1.2).
Overall, 63% of respondents have integrated their e-procurement systems with a variety of central government IT systems. Most common among the latter are budgeting systems, business and tax registries, social security databases, financial systems for payment, and the ERPs of contracting authorities. However, Estonia and Latvia have integrated their e-procurement system with the local tax register (Tallin’s tax register in the case of Estonia). Many countries have integrated e-procurement systems with e-signature and e-invoicing systems. Turkey built automated data exchanges between its e-procurement system and its National Statistical Agency.
Overall, the Korean KONEPS (Korea ON-line E-Procurement System) provides for the highest connectivity to external databases in the OECD area, as it is interconnected to over 200 external database systems, out of which only 65 are from public entities, including the public finance system D-brain. For instance, KONEPS has interfaces with databases from 12 private sector business associations, 9 credit rating companies and the payment systems of 15 commercial banks. In addition to collecting information from external sources, information from KONEPS is also shared with many public entities and private sector information systems (OECD, 2016[14]).
1.2. Using Green public procurement to address environmental challenges
One secondary policy objective Adherents have sought to address in implementing paragraph V of the Recommendation, is addressing environmental challenges. Green public procurement (GPP) is the public purchasing of products and services that are less environmentally damaging when taking into account their whole life cycle. Countries increasingly use GPP to achieve secondary policy objectives in the area of environmental protection ( (OECD, 2015[15]; 2002[16]). The OECD has been a forerunner in encouraging the development of Green public purchasing policies. Indeed, as early as 2002 the OECD Council adopted its Recommendation on Improving the Environmental Performance of Public Procurement [OECD/LEGAL/0311]. It urges countries to incorporate “environmental criteria into public procurement of products and services including, when appropriate, environmental impact criteria throughout the life cycle”.
Take-up of LCC methodology, now in its earliest phase, is used most often by EU Member States in GPP. Public institutions can use LCC to calculate all the costs of a product or service through the whole lifespan. LCC can be applied in different ways and typically includes three types of assessment: conventional, societal and environmental. Conventional LCC refers to traditional financial assessment, including organisation costs. Societal LCC also considers externalities through internalisation of social and environmental costs. Environmental LCC is a life cycle assessment approach that considers costs ascribed to different stakeholders, including future generations and evaluation of “environmental externalities” (De Giacomo et al., 2018[17]).
LCC enables cost efficiencies by allowing contracting authorities to compare alternatives based on an overall cost that includes accurate quantification of any “hidden costs” that can occur later, when the product or service is used. Public authorities can then achieve cost savings and efficiency gains, leading to a situation where a Greener product can also turn out to be cheaper if the cost across the whole life cycle is considered (De Giacomo et al., 2018[17]).
The 2018 Survey shows that respondents are employing good practice in a number of areas to encourage GPP. These include:
laws, regulations and policies in support of GPP
plans for GPP with regard to the market (solutions on offer, capacity, etc.) and cost/benefit assessments
use of environmental standards in technical specifications, such as materials; recycled content; production methods; allowing for submission of alternative solutions; and exclusion criteria for non-compliance
use of environmental standards in award criteria and contract performance clauses (weighted environmental criteria; eco-labels as a criterion; environmental management systems as a criterion)
systems to monitor GPP
professionalisation activities
awareness raising (OECD, 2016[18]).
GPP strategies and policies have been developing for some time now, as can be seen from data collected by the OECD in 2016. The 2018 Survey shows that 64% of surveyed countries are now integrating award criteria relating to GPP into public procurement procedures, at least “seldom” (about 25% of the time) or “sometimes” (about 50% of the time).
There is potential for the environmental benefits of GPP to be improved if GPP award criteria are included in procurement procedures (De Giacomo et al., 2018[17]). Training and guidance for public procurers on the inclusion of technical specifications that take into account secondary policy objectives can result in improved outcomes, as can raising the awareness of stakeholders regarding these performance requirements and standards (OECD, 2018[19]).
1.3. Improving economic results through public procurement
Public procurement is increasingly recognised as a lever for driving economic growth. Employing appropriate impact assessment methodology to measure the effectiveness of procurement in achieving secondary policy objectives is a growing practice (Principle on balance, paragraph V, iii). As the complexity of public procurement activity increases, so does the difficulty of measuring its impact. The Secretariat has found that while public procurement impacts are widespread, measurement frameworks are unable to systematically demonstrate the benefits or drawbacks of procurement policies. High-level indicators can be used for measuring progress against objectives. Data availability and complexity mean that centralised activity can support development of a broader measurement framework that takes into account the procurement system more fully (OECD, 2019[20]).
The link between secondary policy objectives and economic growth is clearly acknowledged by respondents. Several respondents collect data on the degree to which secondary policy objectives are met, and some provide reports to various levels of government (OECD, 2018[3]). This is particularly the case regarding GPP and support to SMEs: around 70% of the countries surveyed collect data or measure the result of procurement processes around these two dimensions.
Some respondents use the data in a way that is intended to drive better outcomes – for instance, publicising that a good or service has achieved a certain status and encouraging further consumption. For example, in Finland a complex methodology is used for measuring the results of GPP. There is a long list of criteria and any framework agreement of the CPB is evaluated against a set of criteria. If at least two of the criteria are fulfilled, the framework agreement is given a “Green label”, meaning all purchases under it are officially labelled Green (OECD, 2018[3]).
Respondents also described procurement data being used to boost consumption of desirable products, such as Green products. Benchmarking of the volume of purchases enabled such strategies to be developed in a thoughtful manner. In Japan the Ministry of Environment collects procurement data from the central government’s agencies and ministries. Based on these data the Ministry calculates the share of GPP for each designated item every year. Based on the results it considers the policies and plans to enlarge the share of GPP for the following year. The number of government purchases from SMEs for each fiscal year is also collated (OECD, 2018[3]).
Use of e-procurement systems is a prerequisite for effective measurement. Data availability is improved by widespread and coherent use of these systems. Using robust and open data standards can also help in collecting data in a usable format. Centralised procurement organisations and framework agreements can also assist in enabling the collection of data that can be used to demonstrate efficiency and effectiveness (OECD, 2019[20]; 2018[21]; 2016[22]). In the area of transparency and data collection, the European Commission provides e-procurement tools such as the Tenders Electronic Daily (TED) database, which plays a key role at European level in centralising the mandatory publication of contract notices and contract award notices above the EU thresholds.
Overall, respondents confirm that the use of procurement data is common for reporting and visibility (with a few exceptions). Once a baseline can be established through the collection of public procurement data, improvements can be measured. Measuring improvements requires being able to demonstrate better economic results through the use of public procurement data (OECD, 2018[3]).
1.4. Public procurement as a tool for inclusive growth
Inequality in many Adherents is at its highest in 30 years. The average disposable income of the richest 10% of the population is around ten times that of the poorest across the OECD – a striking increase from the mid-1980s, when it was seven times. In terms of wealth, the richest 10% owns about 50% of all household assets, while the bottom 40% owns barely 3% (OECD, 2018[23]).
Although poor people and places bore the worst of the financial crisis, the increase in labour productivity has not always led to higher incomes for middle class people. Disadvantages can have a compounding effect, as low household income leads to low-quality education and jobs being at risk, which in turn hinders socio-economic and inter-generational mobility. These inequalities negatively influence economic performance and trust in public institutions, and threaten social cohesion. This means that more and more people are rejecting the established political and economic order and globalisation (OECD, 2018[23]).
In order to meet this challenge, policy makers and private sector leaders need to promote inclusive growth models that prioritise well-being – for the poorest in particular – and the preservation of the planet (OECD, 2018[23]). Policy makers increasingly recognise that public procurement has the potential to contribute to socio-economic development. The changes in how public procurement is perceived has led to reforms in public procurement systems in many countries. It is recognised that procurement enables delivery of more effective public services for better lives (OECD, n.d.[24]).
The Recommendation calls upon Adherents to “facilitate access to procurement opportunities for potential competitors of all sizes” and to “have in place coherent and stable institutional, legal and regulatory frameworks, which are essential to increase participation in doing business with the public sector and are key starting points to assure sustainable and efficient public procurement systems” (Principle on access, paragraph IV).
In the European public procurement system, the most important regulations refer to non-discrimination and equal treatment of all enterprises. These rules, along with the fundamental freedoms of the common market, are the pillars of the EU legal system. In regard to SMEs for example, rather than stipulate a clear preference for SMEs in public procurement procedures, consideration is given to the reduction of financial burden on participants and the implementation of training measures for SMEs (Thai, 2015[25]).
Some Adherents specifically consider social inclusiveness in the tendering process itself. For example, Australia has an Indigenous Procurement Policy that requires Australian Commonwealth entities to award 3% of Australian Commonwealth contracts to indigenous businesses. The policy also requires that certain contracts be set aside for indigenous businesses and that a number of other contracts include minimum indigenous employment or supplier use requirements. Consideration is given at various stages of public procurement procedures that are designed to support social outcomes and inclusion (OECD, 2018[3]).
Respondents provided the following examples of methods being used to increase social inclusion through public procurement:
certification of economic operators’ social objectives
certification of adherence to laws relating to social objectives
consideration of work/life balance in the criteria for procurement awards procedures
creation of jobs included in criteria
criteria include support for SMEs, gender equality and vulnerable groups
requirements to procure from social enterprises
human rights included in award criteria and technical specifications
minimum employment of marginalised groups required of suppliers.
The public procurement system in Chile uses an e-marketplace that encourages use by SMEs and women. There are specialised training programmes for women, and in 2015 Chile revised relevant regulations and guidelines to help officials include gender considerations in their decisions by incorporating gender-specific evaluation criteria. Women represent 36% of the contracts awarded by the government, and the figure is increasing even though women account for a smaller share of aggregated procurement value. Many of the women who have participated are from rural areas, and 64% are the main breadwinner for their family. The main issue with increasing women’s participation is identifying which companies are truly women-led or women-owned, and certification and identification can be a barrier to entry as it can be expensive.
Chile has remedied this problem by introducing an electronic registry that certifies “female enterprise” and is linked with the civil registry (for sole proprietorships). As far as more complex corporations are concerned, women must own the majority of company shares and the CEO must be a woman for the company to be labelled “female enterprise” in the registry. Chile measures the average amount traded by suppliers who are women and noted that it increased from 2013 to 2017 by USD 1 500. Participation in the total amounts traded by suppliers who are women has increased by 6 percentage points from 2013 to 2017 (from 21% to 27%) (Trinidad Inostroza, n.d.[26]).
Engaging SMEs in public procurement helps governments to better meet the public sector’s procurement needs. Increasing their participation in the public procurement market ensures a more competitive bidding process and affords access to a wider choice of available and innovative solutions (Flynn and Davis, 2017[27]). An OECD survey on the strategic use of public procurement to support SMEs found that in 44% of the countries surveyed, public procurement strategies for GPP and public procurement for innovation commonly reflect the objective of supporting SMEs (OECD, 2017[28]).
Complex public procurement systems and processes are a major hurdle to SME participation. SMEs are disproportionately affected due to internal constraints relating to their financial, technical and administrative capacities to access procurement opportunities, prepare tender documents, apply procedures and execute contracts. Countries are addressing this issue through a variety of measures (Figure 1.3), including encouraging the division of contracts into lots; simplifying processes, for instance by standardising procurement documents; encouraging the use of e-procurement; and promoting joint bidding of SMEs with larger companies (OECD, 2018[29]).
The most widely adopted approach to supporting SMEs in public procurement is to ensure that they are aware of tender opportunities, and that competent SMEs have a fair chance of competing for government contracts. By contrast, only a few OECD countries have legislative provisions for bid preference (e.g. Korea and Mexico) or set-asides (e.g. Canada, Korea and the United States), often targeting specific categories of small businesses (aboriginal small businesses in Canada, small businesses from disadvantaged districts in the United States, etc.) (OECD, 2018[29]).
Regarding the simplification of procurement processes, the European Single Procurement Document (ESPD), provided for in Article 59 of the 2014 European Directive on public procurement, is a self-declaration tool for suppliers. Based on a standard form, it offers preliminary evidence regarding exclusion criteria (e.g. criminal convictions, grave professional misconduct) and selection criteria (financial, economic and technical capacity). The full set of supporting documentation, including attestations and certificates, needs only to be presented by the winning economic operators (OECD, 2018[29]). Such simplification measures will ultimately enable more SMEs to participate in public procurement.
1.5. Assuring global supply chains by using public procurement to support responsible business conduct
Enterprises operating in global supply chains can generate growth, employment and skills through their operations and sourcing. However, when they fail to operate responsibly they risk being a contributor to adverse human rights, labour and environmental impacts in their operations or supply chains. The OECD Guidelines for Multinational Enterprises (the OECD MNE Guidelines) – part of the 1976 OECD Declaration on International Investment and Multinational Enterprises [OECD/LEGAL/0144] is a legal instrument that sets out a range of recommendations from governments to multinational enterprises on responsible business conduct. The Guidelines commit adhering governments to providing an open and transparent environment for international investment; in so doing, they encourage the positive contributions that multinational enterprises can make to economic and social progress (OECD, 2011[30]).
Through the online database eCertis, the European Commission has enabled access to information on the evidence documenting associated with the exclusion grounds and eligibility criteria in public procurement. Furthermore, at European level there is a growing trend towards public procurement data disclosure in line with the recommendation of the European EXEP working group (European Commission, 2014[31]) on the development of Contract Registers (CR). This tool aims at centralising information with a single point of access, as well as its publication according to the rules defined by each country.
The OECD Investment Committee, building on the Declaration on International Investment and Multinational Enterprises and the OECD MNE Guidelines, has developed due diligence guidance for responsible business conduct that provides enterprises with practical support for implementing the OECD Guidelines. There are plain language explanations of the due diligence recommendations and associated provisions, both in general as well as in relation to specific sectors. The guidance is intended to help enterprises avoid and address adverse impacts on workers, human rights, the environment, consumers and corporate governance that are associated with their supply chains and other business relationships (OECD, 2018[32]).
Box 1.6. Poland using public procurement to pursue secondary policy objectives
In Poland provisions relating to responsible business conduct are contained within the Public Procurement law and are the result of transposition of very similar provisions contained in the European Union Directives. There is provision for reserved contracts, where the contracting authority may limit competition for contracts to sheltered workshops and other economic operators whose activities include social and professional integration of people belonging to socially marginalised groups. Particular attention is accorded disabled and unemployed people, people with mental disorders or belonging to disadvantaged minorities, the homeless, and refugees Persons from socially marginalised groups must comprise a minimum of 30% of those employed in the organisations. The law also contains exclusion criteria that provide that in certain circumstances, economic operators cannot be awarded contracts. There are various stages of public procurement procedures during which consideration of secondary objectives is encouraged.
Poland has overarching policy instruments that touch upon secondary policies in public procurement. There is ongoing work to have the national purchasing policy take into account strategic use of public procurement. The National Action Plan on Sustainable Public Procurement 2017-20 is in the process of implementation, and a project titled “Effective Public Procurement – Strengthening Administrative Capacity” -- co-financed by the European Social Fund – is under way. The latter project, directed at contracting authorities at all levels of national administration, contains components that address specific secondary policies. A number of measures are additionally being taken to support contracting authorities, including training, conferences and the creation of model documents.
Source: (OECD, 2018[3]).
In a session on Supply Chain Transparency at the 2018 OECD Global Forum on Responsible Business Conduct, it was concluded that supply chain disclosure is a key component of due diligence; it enables workers to play a critical role in providing information to companies about their human rights risks and impacts. There is growing momentum towards greater disclosure of supplier information, but each sector is at a different stage in this journey in terms of the scope and type of information they disclose publicly (OECD, 2018[33]).
The 2018 Survey shows that some respondents are using public procurement levers to increase responsible business conduct on the part of suppliers. Around a third of respondents are measuring the results of procurement processes in relation to some dimension of responsible business conduct, and around half of respondents include responsible business conduct in award and selection criteria for at least some purchases. This is for instance the case in Norway, whose Agency for Public Management and eGovernment (Difi) issued several guidance documents on the topic (Box 1.7).
Box 1.7. The approach to responsible business conduct in Norway
The Norwegian legal framework allows human rights considerations to be taken into account in public procurement processes as award criteria, and in technical specifications. There are several guidance documents and other information on responsible business conduct available on the website of the Agency for Public Management and eGovernment (Difi).
In Norway, measures including new procurement rules have been put in place to prevent work-related crime, which has a negative effect on competition in markets. Norway has identified that one way to prevent work-related crime and social dumping is to impose more stringent requirements on public officials who order, as well as on suppliers. Difi has prepared a guide to best practice for compliance with the regulations concerning pay and working conditions in public contracts.
Also now in place are new provisions for contracting authorities requiring that there be a maximum of three subcontractor levels in the supply chain, i.e. a main contractor with a maximum of two subcontractor levels in contracts for construction works and cleaning services. Furthermore, contracting authorities must require the use of apprentices in contracts over a certain size and duration. Guidelines are available for these provisions.
Responsible business conduct by suppliers is promoted in Canada by certification as part of the bidding process by apparel suppliers. The Canadian federal CPB, Public Services and Procurement Canada (PSPC), is supporting research on ethical sourcing of apparel in other jurisdictions as well as the practices of apparel suppliers in Canada with offshore production. On 7 September 2018 the PSPC launched the Ethical Procurement of Apparel Initiative. This initiative requires suppliers and their first-tier subcontractors to certify that they comply with a list of eight fundamental human and labour rights principles each time they bid. Suppliers that are found to be in violation of their certification risk having their contract terminated for default. The certification is coupled with a requirement to provide information regarding the location of the factories that produce the products being purchased. When the contract is awarded, the certification and information about the factory is made publicly available.
The federal government In Canada also launched an initiative to develop a new government-wide vendor relationships and performance management regime. It will ensure that suppliers’ past performance record is used to inform future contract awards. Ethical sourcing will be assured during the regime in supplier contracts. It will also be included in transparent evaluation criteria and rating scales for assessing vendor performance.
The private sector is using responsible business conduct to assure supply chains, sometimes due to the reputational damage and associated loss of shareholder value associated with supplier failures in this area. Public organisations also have incentives to influence the behaviour of suppliers and encourage responsible business conduct to ensure citizen outcomes are optimised as well as to comply with legal requirements. There is currently interest in using technologies such as blockchain to solve the problems of scale and trust inherent in responsible business conduct (OECD, 2018[35]).
1.6. Understanding the trade-offs for informed decisions
An area of concern in many OECD countries is the possibility of being overwhelmed by a large number of policy objectives through public procurement – the risk being that the system of mandates and preferences becomes unmanageable or impossible to satisfy. There are many different technology solutions and project management techniques that can support the collection of and reporting on various objectives that have strategic importance for various stakeholders (Sumiani, Haslinda and Lehman, 2007[36]) (Amran and Ooi, 2014[37]).
Many jurisdictions are using methods to prioritise the various strategic initiatives being adopted through procurement functions (Table 1.2). There is recognition in some countries that it is not possible to do everything, especially where resources are limited. The various approaches range from leaving the decision up to the contracting authorities to prioritising the initiatives in an annual work plan.
Table 1.2. Prioritisation of strategic procurement initiatives
.Prioritisation method |
Countries |
---|---|
Contracting authority decision |
Germany, New Zealand, Poland |
Legislation sets out priorities in terms of secondary policy objectives |
Hungary, Mexico |
Policy sets out primary objectives |
Iceland, Portugal (in the future) |
Annual work plan prioritisation/Action Plan |
Korea |
Each project is considered separately by individual procurement entities on a case-by-case basis underpinned by the suitability of a particular procurement process to further a secondary policy goal |
France, Israel, Netherlands, Costa Rica |
All initiatives are promoted equally |
Greece, Sweden |
Prioritised against in accordance with societal goals |
Slovak Republic |
Source: (OECD, 2018[3]).
It seems that there is no “one size fits all”, and governance over strategic procurement goals is configured according to the prevailing need in each country. Sometimes a top-down approach is taken, as in New Zealand where a recent procurement reform programme was directly linked to government policy direction (Box 1.8).
Box 1.8. Achieving wider benefits from government procurement in New Zealand
In New Zealand it is recognised that public procurement can and should be used to support wider social, economic and environmental outcomes that go beyond the immediate purchase of goods and services.
On 23 October 2018, the New Zealand Government agreed to a set of priority outcomes for contracting authorities to draw from their procurement activities, and identified specific contracts or sectors for initial focus.
Contracting authorities are expected to collectively focus on four priority outcomes of the greatest benefit to New Zealand:
1. Access for New Zealand businesses – Increase access to government contracts for New Zealand businesses, particularly those less able to access opportunities and those working in priority sectors (such as ICT, Maori and Pasifika businesses, and businesses in the regions).
2. Construction sector skills and training – Increase the size and skill level of the domestic construction sector workforce and provide employment opportunities to targeted groups.
3. Employment standards – Improve conditions for workers and future-proof the ability of New Zealand business to trade.
4. Reducing emissions and waste – Support transition to a zero net emissions economy and reduce waste from industry by supporting innovation.
To implement this work, the central purchasing body, New Zealand Government Procurement and Property, will undertake the following steps:
Work with agencies and stakeholders to identify the best approach to operationalise each outcome.
Update the Government Rules of Sourcing, and develop guidance and support for agencies.
Develop a monitoring and reporting framework to track agency adoption and outcomes achieved. This will be reported to the Cabinet annually and findings will inform practice improvements.
In the 2018 Survey respondents described the reform programmes they have embarked on for public procurement. Some have defined the priorities for procurement in terms of the policy objectives identified as part of the reforms. In Sweden the reform programme has outlined policy objectives and ultimately results in partnerships for co-creation between suppliers and procurers (Box 1.9) (OECD, 2018[3]). In Norway a White Paper on Public Procurement (presented to the Norwegian Parliament in 2019) contained measures related to GPP, innovation, pay and working conditions, and human rights considerations (OECD, 2018[3]).
Box 1.9. The National Public Procurement Strategy in Sweden
September 2015 saw the establishment of the National Agency for Public Procurement (NAPP). The government chose to place support for public procurement with this independent authority to raise its profile and to provide organisational conditions that would make support from NAPP as effective as possible. NAPP has overall responsibility for developing, managing and supporting public procurement carried out by contracting authorities and entities. A National Public Procurement Strategy was developed to demonstrate the benefits that can be achieved through a strategic approach to purchasing. The objectives set for public procurement are that it is efficient, is legally certain, and takes advantage of the competition in the market. It also promotes innovative solutions and takes environmental and social concerns into account. Strategic implementation of public procurement is an effective means of achieving the objectives while at the same time bringing about positive effects for society. These include driving increased growth and employment, sustainable development and other social and ethical considerations. The government formulated seven policy objectives for its overall procurement strategy:
1. public procurement as a strategic tool for doing good business
2. effective public purchasing
3. a multiplicity of well-functioning supplier competition
4. legally certain public procurement
5. public procurement that drives innovation and promotes alternative solutions
6. public procurement that is environmentally responsible
7. public procurement that contributes to a socially sustainable society.
The stated intent in the policy document is that it will be applied to all public purchases.
In Korea a survey report was developed to support understanding of the various requirements and prioritise their satisfaction within an organisation. The report identified all the various secondary policy requirements that applied to public procurement: up to 46 “recommended” procurement priorities and 8-10 mandatory requirements (OECD, 2016[14]).
The 2018 Survey results show that some jurisdictions are balancing primary and secondary procurement objectives. Methods differ, but there is recognition by some respondents that trying to address all objectives in a balanced way is necessary. Some use policy objectives and goals, as in Denmark where areas related to primary policy objectives are identified in thorough studies that compare the buyer’s needs with what the market can offer. The goal of public procurement in Denmark is to lower costs but keep the quality and efficiency sufficiently high in services that are delivered to contracting authorities. Similarly, the National Public Procurement Strategy in Sweden states that all public procurement shall be efficient, be legally certain, and take advantage of the competition in the market. It also states that public procurement shall promote innovative solutions and take environmental and social concerns into account. In some countries such as Israel, the balance between primary and secondary policy objectives is maintained through the use of tools such as a sensitivity analysis to forecast the cost of integrating the policy objective and to decide how to weight it in the procurement procedure.
Other jurisdictions, such as that of Hungary, rely on the law to support prioritisation. In Hungary the legal provisions relating to public procurement state that the law was put in place partly to enhance the access of local small and medium-sized enterprises, promote environmental protection, and promote the social considerations of the state. The Ministerio de Hacienda (Ministry of Finance) in Costa Rica is constantly working on identifying priority areas to balance the primary objective of public procurement – which is to procure goods and services – and other objectives that are part of the national agenda. Including these secondary objectives as part of evaluation criteria has helped to achieve that balance. Balancing primary and secondary objectives is done on a case-by-case basis taking into account the objective itself as well as national priorities. For example, guideline DGABCA-002-2018 provides that no entity shall buy single-use plastics and so contracting authorities must consider this before their procurement procedures begin.
References
[37] Amran, A. and S. Ooi (2014), “Sustainability reporting: Meeting stakeholder demands”, Strategic Direction, http://dx.doi.org/10.1108/SD-03-2014-0035.
[13] Baron, R. (2016), The Role of Public Procurement in Low-carbon Innovation, OECD, https://www.oecd.org/sd-roundtable/papersandpublications/The%20Role%20of%20Public%20Procurement%20in%20Low-carbon%20Innovation.pdf.
[6] Brannstrom, F. (2018), Public Procurement in Sweden.
[17] De Giacomo, M. et al. (2018), “Does Green Public Procurement lead to Life Cycle Costing (LCC) adoption?”, Journal of Purchasing and Supply Management, http://dx.doi.org/10.1016/j.pursup.2018.05.001.
[1] European Commission (2018), Commission notice Guidance on Innovation Procurement, https://ec.europa.eu/transparency/regdoc/rep/3/2018/EN/C-2018-3051-F1-EN-MAIN-PART-1.PDF.
[31] European Commission (2014), Multi-Stakeholders Expert Group on eProcurement, http://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetail&groupID=3142.
[2] European Commission (n.d.), Public Procurement Strategy.
[12] European Union (2014), Directive 2014/24/EU of the European Parliament and the Council, https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32014L0024&from=EN (accessed on 13 February 2019).
[27] Flynn, A. and P. Davis (2017), “Explaining sme participation and success in public procurement using a capability-based model of tendering”, Journal of Public Procurement, http://dx.doi.org/10.1108/JOPP-17-03-2017-B003.
[10] Innovation Fund (2018), New Zealand Government Innovation Fund, https://innovationfund.co.nz/how-it-works.
[5] Ministry of Finance Sweden (n.d.), National Public Procurement Strategy, https://www.upphandlingsmyndigheten.se/globalassets/english/procurement/national_public_procurement_strategy_english_web.pdf (accessed on 1 July 2019).
[38] New Zealand Government Procurement and Property (2018), Broader Outcomes.
[20] OECD (2019), Productivity in Public Procurement: A Case Study of Finland: Measuring the Efficiency and Effectiveness of Public Procurement, OECD, Paris, http://www.oecd.org/gov/public-procurement/publications/productivity-public-procurement.pdf.
[3] OECD (2018), Data from the 2018 OECD Survey on the Implementation of the 2015 OECD Recommendation on Public Procurement.
[21] OECD (2018), Mexico’s e-Procurement System: Redesigning CompraNet through Stakeholder Engagement, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264287426-en.
[35] OECD (2018), OECD Blockchain Policy Forum - OECD, http://www.oecd.org/finance/oecd-blockchain-policy-forum-2018.htm.
[32] OECD (2018), OECD Due Diligence Guidance for Responsible Business Conduct, http://mneguidelines.oecd.org/OECD-Due-Diligence-Guidance-for-Responsible-Business-Conduct.pdf.
[23] OECD (2018), Policy Brief What’s the issue? Opportunities for all: OECD Framework for Policy Action on Inclusive Growth, http://www.oecd.org/policy-briefs.
[19] OECD (2018), Public Procurement in Nuevo León, Mexico: Promoting Efficiency through Centralisation and Professionalisation, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264288225-en.
[9] OECD (2018), Public Procurement Week: Investing in strategic public procurement to maximise benefits for all.
[29] OECD (2018), SMEs in Public Procurement: Practices and Strategies for Shared Benefits, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264307476-en.
[33] OECD (2018), Supply Chain Transparency: A Growing Norm for Responsible Business Conduct, http://www.businesswire.com/news/home/20080403005469/en/HP-Technology-Sector-Release-List-Top-.
[28] OECD (2017), “OECD Survey on strategic use of public procurement to support SMEs”.
[8] OECD (2017), Public Procurement for Innovation: Good Practices and Strategies, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264265820-en.
[18] OECD (2016), Checklist for Supporting the Implementation of the OECD Recommendation of the Council on Public Procurement, http://www.oecd.org/governance/procurement/toolbox/search/checklist-implementation-oecd-recommendation.pdf (accessed on 1 July 2019).
[4] OECD (2016), Data from the 2016 OECD Survey on public procurement (partially published), https://qdd.oecd.org/subject.aspx?Subject=GOV_PUBPRO_2016.
[14] OECD (2016), The Korean Public Procurement Service: Innovating for Effectiveness, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264249431-en.
[7] OECD (2016), Towards an OECD Strategic Response to the Sustainable Development Goals: Consultation draft, https://www.oecd.org/dac/Consultation-draft--Outline-OECD-strategic-response-to-the-SDGs.pdf.
[22] OECD (2016), Towards Efficient Public Procurement in Colombia: Making the Difference, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264252103-en.
[15] OECD (2015), Going Green: Best Practices for Sustainable Procurement, http://www.oecd.org/gov/public-procurement/publications/Going_Green_Best_Practices_for_Sustainable_Procurement.pdf.
[30] OECD (2011), OECD Guidelines for Multinational Enterprises, 2011 Edition, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264115415-en.
[16] OECD (2002), Recommendation of the Council on Improving the Environmental Performance of Public Procurement OECD/LEGAL/0311.
[24] OECD (n.d.), Public Procurement for Sustainable and Inclusive Growth Enabling Reform through Evidence and Peer Reviews, OECD, https://www.oecd.org/gov/ethics/Public-Procurement-for%20Sustainable-and-Inclusive-Growth_Brochure.pdf (accessed on 3 December 2018).
[34] Solberg Prime Minister Anniken Hauglie, E. et al. (2017), Revised Strategy for Combating Work-related Crime, https://www.regjeringen.no/contentassets/18425e5b42f84340b02fbb74bdb90b19/revised_strategy_for_combating_work_related_crime.pdf.
[36] Sumiani, Y., Y. Haslinda and G. Lehman (2007), “Environmental reporting in a developing country: a case study on status and implementation in Malaysia”, Journal of Cleaner Production, http://dx.doi.org/10.1016/j.jclepro.2006.01.012.
[25] Thai, K. (ed.) (2015), International Public Procurement: Innovation and Knowledge Sharing, Springer International Publishing, Cham, http://dx.doi.org/10.1007/978-3-319-13434-5.
[26] Trinidad Inostroza (n.d.), Una herramienta estratégica para la eficiencia del Estado, https://www.wto.org/english/tratop_e/womenandtrade_e/s02_01_trinidad_inostroza.pdf (accessed on 12 January 2019).
[11] ZDNet (2018), Auckland airport gets ’digital’ biosecurity officer | ZDNet, https://www.zdnet.com/article/auckland-airport-gets-digital-biosecurity-officer/ (accessed on 12 January 2019).