3. Assessment of the Hungarian measurement framework
3.1. Description of the framework
The development of the Hungarian public procurement performance measurement framework (hereafter the Hungarian framework) was linked to the commitments of the country as part of the conditionality procedure. Indicators part of the framework were listed in a document published on the 30th of November 2022 (Hungarian Government, 2022[6]). However, some indicators mentioned in this list, lack clarity in particular for indicators related to centralisation and capacity of the procurement workforce. For instance, on centralisation, one indicator of the published list refers to “Results of a survey on the efficiency of central purchasing bodies” which cannot be considered as an indicator per se. This lack of clarity of some indicators could be explained by the limited time the Hungarian authorities had to develop the framework.
The first report of the results of the Hungarian fraemwork (hereinafter the report) that was published on the 28th of February 2023 (Hungarian Government, 2023[7]) provides more clarity on certain indicators. Therefore, indicators mentioned in this document were based mainly on the indicators identified in the report and discussions with the PMO.
The indicator framework includes 77 indicators divided into three categories like the ones identified in the OECD public procurement measurement framework (compliance, efficiency and strategic objectives). The Hungarian measurement framework includes 20.8% of indicators in the compliance category and respectively 72.7 % 6.5% and for the efficiency and strategic objectives categories. It also includes 57 sub indicators covering different aspects such as “Effectiveness of the remedies”, “Legal compliance” and “Transparency” for Compliance indicators. Table 3.1 provides a summary of the framework and Annex A provides a detailed description of the Hungarian framework. Furthermore, in line with the OECD measurement framework, the indicators of the Hungarian framework are covering all procurement phases.
3.2. Improving existing indicators
While the existing indicators of the Hungarian framework represent already a great achievement, some indicators of the framework could be improved to further support the government in achieving its objectives (see Section 2.2.2). The improvement could be related to the scope, the methodology or to the possibility to provide further sub-indicators to have a better understanding of the system in line with policy priorities. After some general considerations to improve the framework, the below sections assess individual indicators following the current organisation of the framework by “categories” and “sub-categories”. While some recommendations on the improvement of the existing indicators can be implemented in the short term, other recommendations can be implemented in the medium and long term (see Annex B).
3.2.1. General considerations
The implementation of a performance indicator requires ensuring that the different users and stakeholders of the public procurement system understand what this indicator is aiming to achieve. As mentioned in the OECD public procurement measurement framework, this is usually achieved by providing different information, including the i) metric description which is a detailed description of the indicator and ii) the calculation/ data requirements (OECD, 2023[5]).
The Hungarian framework published in November 2022 does not provide such information for all indicators. For indicators based on EKR data, Annex 3 of the first report provides the methodology used to calculate these indicators. For the other indicators, no methodology has been provided. However, some methodological aspects could be identified in the analysis of some indicators provided in the report. Independently from the data source, it is recommended to add in the updated Hungarian framework a detailed methodology to calculate each single indicator and sub-indicator.
Furthermore, while most indicators included in the Hungarian framework are relevant, it could be important for the Hungarian government to explain in the framework the reason for focusing on certain sub-categories of the framework and for choosing certain indicators.
In addition, the Hungarian government could also clarify the methodology used for some sub indicators such as those that mention “CPV divisions” or “CPV groups”. Indeed, it is not clear if these sub-indicators include all CPVs or the Top CPV divisions or groups.
3.2.2. Compliance indicators – effectiveness of the remedies
Remedies systems are key for the well-functioning of the public procurement system. The OECD Recommendation on public procurement highlights the need to handle complaints in a fair, timely and transparent way through the establishment of effective courses of action for challenging procurement decisions to correct defects, prevent wrong-doing and build confidence of bidders, including foreign competitors, in the integrity and fairness of the public procurement system (OECD, 2015[8]). The section below assesses the indicators included in the Hungarian measurement framework.
1 Number of public procurement challenges
This indicator provides an overview of the number of public procurement challenges. It includes four sub-indicators providing relevant information to understand key aspects of the remedies system: i) by ex officio and on-demand procedures, ii) by EU and national procedures, iii) by source of funding (EU vs domestic funding) and iv) breakdown by appeals decided on the merits and appeals not decided on the merit. It could be relevant to consider adding a sub-indicator on the most important sectors or CPV divisions where challenges are filled.
Furthermore, in terms of methodology, the breakdown by year is the year of publication of the contract award notice. However, in some cases, some procedures might be cancelled and therefore not accounted for. It is therefore recommended to use a breakdown the year where the challenge has been filled or to clarify this point in the methodology.
Different institutions or their representative can launch an ex officio procedure before the PPAB. Article 152 mentions of the PPL provides their list: a) the President of the Public Procurement Council; b) the State Audit Office; c) the government body responsible for control; d) the body responsible for the legal supervision of local governments; e) the Hungarian State Treasury; f) the Commissioner for Fundamental Rights; g) the entity granting support for the public procurement, or the entity co-operating pursuant to law in the use of the support; h) the central purchasing body appointed by the Government; i) the Hungarian Competition Authority; j) the body auditing European Union supports; k) the minister competent in the supervision of the national property; l) the public prosecutor; m) the minister competent in public procurements; n) the minister responsible for the use of EU fund (Hungarian government, 2021[12]).
This indicator aims at understanding to which extent this procedure is used by these different bodies and their representatives.
3 Share of public procurement challenges out of the total number of public procurement procedures
To have a real understanding of the functioning of the remedies system, the indicator on the number of challenges which is an “absolute value” needs to be complemented with the “share of public procurement challenges out of the total number of public procurements procedures” (relative value). Therefore, ensuring also the availability of the same sub-indicators as for the” Number of public procurement challenges” would further improve the assessment of challenges.
4 Share of successful public procurement challenges
Providing information on the share of successful public procurement challenges is key to build trust in the system and can impact the level of competition in the public procurement market. The indicator on the share of successful public procurement challenges is complemented by a sub-indicator on the source of funding (EU vs national funding).
To calculate this indicator, the PMO relies on data provided by the PPAB. However, the exact methodology for calculating this indicator has not been described. According to discussions with the PMO, this indicator refers to review procedures where in its decision the PPAB establishes that an infringement occurred. The Hungarian government would benefit from ensuring the availability of the methodology used in the performance measurement framework.
5 Share of public procurement procedures with a request for a preliminary dispute settlement as a percentage of all public procurement procedures
Understanding the use of the different means available to settle disputes is key to understand the functioning of the remedies system. To calculate this indicator, the PMO relies on data provided by contracting authorities in EKR as they are mandated to publish in EKR requests for preliminary dispute settlement (Hungarian government, 2021[12]).
6 Number of notifications of public interest received by the Public Procurement Authority
According to Act XXV of 2023 on complaints, disclosures in public interest, and related rules on reporting abuses “A public interest disclosure calls attention to a circumstance the remedying or discontinuation of which is in the interest of the community or the whole society.” (Hungarian government, 2023[47]). Based on notifications of public interest, the PPA can launch controls or a legal review before the PPAB.
7 Number of launched review procedures based on notifications of public interest received by the Public Procurement Authority
As mentioned in the previous section, the PPA can launch legal reviews before the PPAB based on notifications of public interest (ex-officio). To further assess the performance of the public procurement remedies system, the Hungarian government could consider adding a sub-indicator on the share of the reviews before the PPAB based on notifications of public interest received by the PPA out of the total ex officio reviews launched by the PPA.
8 Cost of public procurement challenges
This indicator aims at assessing the cost of public procurement challenges for bidders and contracting authorities. From a methodological perspective, the cost for bidders and for contracting authorities should be considered as two separate indicators rather than sub-indicators.
The legal framework mentions that it is mandatory to be represented by a responsible accredited public procurement consultant, legal adviser registered with the chamber of lawyers or a solicitor for challenges before the PPAB (Hungarian government, 2021[12]). The survey only includes a specific question to contracting authorities on the fee for the legal representation of the organisation when using an external legal representative in an appeal procedure (before the PPAB) with up to three elements of complaint. It is therefore recommended to also add a specific question for economic operators as challenging public procurement decisions represents also a cost for them.
Furthermore, from a methodological perspective, it is important to acknowledge that costs of public procurement challenges include also internal costs that are not accounted for in this indicator (e.g. costs related to the analysis performed by the legal department, etc.)
9 Average duration of redress procedures
In line with good principles of remedies system, the review must be as rapid as possible (OECD-SIGMA, 2011[48]). This indicator is key to understand the effectiveness of the remedies system. By law, the PPAB has a maximum of 15 days, if there is no hearing; 25 days if there is a hearing; and 60 days in case of an amendment or performance violating the PPL, or the bypass of the procurement procedure (Hungarian government, 2021[12]).
The PMO would benefit from detailing the methodology used for this indicator in terms of time/ date to calculate this indicator. For instance, for a specific year, does it include only cases opened that year or cases ruled that year.
10 Number and share of judicial reviews in the number of appeals
Effective courses of action for challenging procurement decisions also include the possibility to appeal the PPAB decision before the relevant court. It is therefore pertinent to understand to which extent this course of action is used by both contracting authorities and bidders.
The PMO would benefit from detailing the methodology used for this indicator in terms of time/ date to calculate this indicator. For instance, for a specific year, does it include only cases opened that year or cases ruled that year.
This indicator also includes a sub indicator aiming at refining the information by judicial outcome. The different judicial outcomes could also be detailed in the methodology.
11 Number and percentage of challenges by contracting authorities against themselves out of the total number of challenges
In Hungary, before the 1st of September of 2023 a contracting authority that identifies or detects an infringement to the public procurement law in its procurement procedures had limited possibilities to remedy this infringement. The contracting authority could only challenge its own decisions before the PPAB which highlighted the lack of flexibility of the Hungarian public procurement system in this area (Hungarian government, 2021[12]). An amendment to the PPL was introduced to remedy this situation.
Having a dedicated indicator shows that the Hungarian government is aware of this issue. The PMO would benefit from detailing the methodology used for this indicator in terms of time/ date to calculate this indicator.
3.2.3. Compliance indicators – Legal compliance
Enhancement of public procurement systems requires understanding the main infringements identified by competent authorities. This sub-category within the compliance indicators includes only one indicator “Number of infringements broken down by the legal provision infringed”.
12 Number of infringements broken down by the legal provision infringed
This indicator is particularly relevant to identify key public procurement areas to further monitor and potentially to improve, however several aspects of this indicator could be improved.
First, to calculate this indicator, the PMO relies on “raw” data provided by the PPAB. As mentioned in Section 2.3 the PMO had to assess all the documents and data provided by the PPAB to identify the specific provisions that were infringed. It is worth enhancing the collaboration with the PPAB on the data classification to ease the process of calculating this indicator and thus to enhance the monitoring of the public procurement system. For instance, after ruling a case, the PPAB commissioner could directly fill a form with the relevant information required by the PMO.
In addition, similarly to other indicators, there is no clear description of the methodology used to calculate this indicator in the framework. It is worth mentioning that the report includes some relevant information which are not comprehensive. The PMO should therefore consider detailing the methodology used for this indicator.
Moreover, the data is provided in absolute value (number of infringements), it is worth considering also providing the data in relative value (as a share) to “neutralise” the potential variations of number of infringements per year.
3.2.4. Compliance indicators – Transparency
Competitive procedures should be the standard method for conducting procurement as a means of driving efficiencies, fighting corruption, obtaining fair and reasonable pricing and ensuring competitive outcomes (OECD, 2015[8]). Transparency in terms of publishing procurement opportunities contributes to fair and equitable treatment for potential suppliers (OECD, 2016[49]). The strong linkages between transparency and access to public procurement opportunities are reflected in four indicators of the Hungarian measurement framework that are described below.
13 Share of lots (in numbers) of public procurement procedures launched by public notice among the lots awarded
This indicator is relevant to assess the share of procedures that are competitive (open for competition). Given that this indicator relies on data provided by EKR, Annex 3 of the report provides a detailed description of the methodology used. It could be relevant to consider sub-indicators that may reveal specific trends (e.g. by EU vs national procedures threshold, and source of funding).
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
14 Share of negotiated procedures without prior publication of a contract notice
Article 49 of the PPL provides the possibility to use negotiated procedure without the publication of a contract notice. To use this procedure, contracting authorities are required to inform the PPA on the day of its launch, with referring to the specific circumstances justifying its application and submitting the invitation to tender. As part of the compliance control, the PPA checks whether the contracting authority’s decision to use this procedure complies with the PPL and publishes a reasoned decision concerning the legal grounds on its website. If the PPA is unable to determine that the legal ground exists, it shall initiate a remedy procedure before the PPAB. Contracting authorities must publish all documents of the negotiated procedure without prior publication, including data regarding the invited and contracted economic operators (Hungarian government, 2021[12]). Given the non-competitive nature of this procedure, it is relevant to have it covered by a dedicated indicator.
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
15 Share of lots (in numbers) of public procurement procedures conducted under Article 115 of the PPL
The procedure under Article 115 of the PPL is applicable to public works contracts with an estimated value of less than HUF 300 million (approximately EUR 78°1540). Through this procedure, contracting authorities should directly send a written invitation to tender to at least five economic operators instead of publishing a contract notice to announce the procedure, and only the economic operators invited to tender may submit a bid. Since February 2021, following an amendment of the PPL, this type of procedure cannot be used when using EU funds (even partly) (Hungarian government, 2021[12]).
This indicator aims at understanding to which extent this procedure is used by contracting authorities in the country. It is complemented by two sub-indicators:
1. the share of public procurement lots conducted under Article 115 of the Public Procurement Act compared to the number of public works successful lots (contracts),
2. the share of public procurement lots conducted under Article 115 of the Public Procurement Act compared to the number of public works successful lots of works contracts conducted below-EU-thresholds.
According to several stakeholders, including the Integrity authority (Integity Authority, 2023[23]), this provision should be repealed as it does not ensure a fair competition between bidders given that contracting authorities decide on the list of economic operators who will receive an invitation to tender, breaching the principles of “equal opportunities” and “equal treatment” mentioned in the PPL. This argument is relevant to consider even if this procedure represents only 2% of the total procurement volume in 2022 and 4% of contract value in the construction field. Furthermore, public procurement principles and rules should consider the procedure that achieve the greatest value for money independently of the source of financing. These findings have been also highlighted in the responses to the survey that were reflected in the first report of the measurement framework.
16 Share of lots (in values) of public procurement procedures conducted under Article 115 of the PPL
This indicator is similar to the previous one but provides information in terms of value of lots which is key to understand the impact of the use of procedures conducted under Article 115 of the PPL on the public works market.
3.2.5. Efficiency indicators – Efficiency of public procurement procedures
Assessing the efficiency of procurement processes goes beyond procurement outcomes, it also includes the time to undertake and complete specific processes, the use of specific tools and mechanisms by contracting authorities. In this context, the Hungarian framework includes a set of ten indicators aiming at assessing the efficiency of public procurement procedures.
17 Average time, in days, between the publication of the call for tender and the publication of the report on the award of the contract
This indicator aims at assessing the average duration of a procurement process from the publication of the call for tender to the publication of the award of the contract. It includes four sub-indicators:
i) by EU and national procedures for the first summary (evaluation)
ii) by EU and national procedures for the last summary
iii) by procurement object for the first summary
iv) by procurement object for the last summary
To support the analysis of the sub-indicator on “procurement object” it could be interesting to also look at CPV divisions when relevant. This analysis cannot be included as a sub-indicator as it would entail adding 45 additional sub-indicators.
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
18 Average time in days between the publication of the call for tender and the bid submission deadline
This indicator is key to assess the access to the public procurement market as the time allotted for economic operators to respond to a call for tender should be proportionate to the size and complexity of the procurement (OECD, 2015[8]).
It includes two sub-indicators i) by EU and national procedures and ii) by procurement object. To support the analysis of the sub-indicator on “procurement object” it could be interesting to also look at CPV divisions when relevant. This analysis cannot be included as a sub-indicator as it would entail adding 45 additional sub-indicators.
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
19 Average time in days between the bid submission deadline and the publication of the summary
This indicator is key to assess the average time taken by contracting authorities for the tender evaluation which can also impact the competition in the public procurement market. Indeed, lengthy evaluations and publications of summaries could have an impact on the level of participation of economic operators in procurement opportunities as they represent a risk in terms of the financial viability of their bids (e.g. inflation, etc.). The indicator includes two sub-indicators i) by EU and national procedures, and ii) by procurement object. To support the analysis of the sub-indicator on “procurement object” it could be interesting to also look at CPV divisions when relevant. This analysis cannot be included as a sub-indicator as it would entail adding 45 additional sub-indicators.
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
20 Share of unsuccessful lots (in numbers)
This indicator aims at assessing the share of lots not awarded. It includes a sub-indicator distinguishing EU and national procedures. This indicator could be complemented by other sub-indicators such as “procurement object” and the type of procedure to provide a real picture of the situation. To support the analysis of the sub-indicator on “procurement object” it could be interesting to also look at CPV divisions when relevant. This analysis cannot be included as a sub-indicator as it would entail adding 45 additional sub-indicators.
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
21 Share of each reason of unsuccessful lots
This indicator provides further information on the reasons for declaring a lot unsuccessful which is a source of inefficiency of the public procurement system which can be found in the PPL. Understanding the exact reasons could support the government in taking the necessary actions to decrease the rate of unsuccessful lots.
The methodology in Annex 3 of the report does not list all the different reasons for unsuccessful lot. However, these reasons are found in Annex 2 which present the results of some indicators: The Hungarian government could further detail the different reasons in the methodology.
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
22 Number and share of lots using efficient procurement techniques
The use of tools or techniques such as framework agreements of dynamic purchasing systems (DPS) have usually a positive impact on the efficiency of the public procurement system as they aim at improving procurement procedures, reducing duplication and achieving greater value for money (OECD, 2015[8]).
The Hungarian measurement framework includes an indicator on the number and share of lots using efficient procurement tools or techniques. The “number” and “share” should be considered as two distinct indicators.
Moreover, the indicator includes two sub-indicators, one in relation to framework agreements and one in relation to DPSs. However, from a statistical perspective the two sub indicators should be considered as two separate indicators as they don’t add up because they have a different denominator (For DPS: the number of awarded lots + the number of lots of dynamic purchasing systems already established, and for FAs: the total number of lots). The Hungarian government should consider addressing this issue in the updated public procurement measurement framework.
23 Average duration of framework agreements
A framework agreement may generally not exceed four years; The duration of framework agreements, in the case of centralisation can have a significant impact on competition for a fixed period. Therefore, a good understanding of the nature of the market and of the purchases to be made is essential when deciding on the contract duration. Careful consideration needs to be given to balancing the benefits of longer-term arrangements – such as reduced procurement costs and speedier procurement procedures – with the effects of potentially limiting competition (OECD-SIGMA, 2016[50]).
Therefore, it is relevant for the Hungarian government to understand the average duration of framework agreements. It could consider adding a sub-indicator in relation to the procurement object. To support the analysis of the sub-indicator on “procurement object” it could be interesting to also look at CPV divisions when relevant. This analysis cannot be included as a sub-indicator as it would entail adding 45 additional sub-indicators.
Furthermore, as CPBs are implementing framework agreements, it is recommended to add a specific sub-indicator on framework agreements managed by CPBs. This would further enrich the analysis on the efficiency of centralised procurement schemes. This sub-indicator could be integrated when advancing the data integration between EKR and CPB systems as recommended in Section 2.3.3.
Last, from a methodological perspective, framework agreements can be usually extended up to four years, it could be relevant to also assess the effective duration of framework agreements and not the one mentioned initially.
24 Share of framework agreements concluded with one or multiple operators
One feature of framework agreements is the number of economic operators that are part of it: one supplier or multi-supplier. This feature could have an impact on the level of competition in a given market where a framework agreement is used. However, it is important to note that a multi-supplier framework agreements, does not imply that all suppliers will get a share of the contract (OECD-SIGMA, 2016[50]).
From a methodological perspective, this indicator should be divided into two indicators: “Share of framework agreements concluded with one operator” and “Share of framework agreements concluded with multiple operators”. The latest could be complemented by sub-indicators aiming at assessing the average number of economic operators’ part of the framework and if suppliers’ part of a multi-supplier framework agreement are performing a share of the contract or not.
Furthermore, as CPBs are implementing framework agreements, it is recommended to add a specific sub-indicator on framework agreements of CPBs (depending on data integration between EKR and CPB systems).
25 Share of lots awarded in public procurement procedures using each evaluation criteria
The evaluation of the submitted bids is a key milestone of the procurement process as it allows public buyers to assess the relative merits of proposals received against institutions’ objectives and for this reason, care must be taken to ensure that the outcome is satisfactory and that it has been decided in a fair and transparent manner (OECD, 2020[51]).
Therefore, it is relevant for the different public procurement stakeholders to have information on the different evaluation criteria used to assess bids. In line with EU regulation, in Hungary, article 76 of the PPL provides for the possibility to use the following evaluation methods: i) the lowest price criterion, ii) the lowest cost criterion and iii) the best price-quality ratio. However, the use of the lowest price criteria is only allowed in justified cases.
The Hungarian measurement framework mentions as an indicator the “Share of lots awarded in public procurement procedures using each evaluation criteria” and three sub-indicators reflecting the different evaluation criteria provided in Article 76. However, as the three sub-indicators cannot be added up, it is recommended to consider them as three separate indicators.
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
26 Share of procurement procedures broken down into lots (as a percentage of the number of successful procurement procedures)
When preparing a procurement procedure, contracting authorities should consider if their needs are to be responded by using one contract or by using a number of separate contracts or “lots”, which may be awarded and performed by different economic operators. The decision is not an easy one, as savings derived from economies of scale may promote the use of a single contract, while the diversity resulting from multiple contracts, or lots can enhance competition and increase efficiency and sometime allows to better respond to the contracting authority’s needs (OECD-SIGMA, 2016[52]).
It is relevant for the Hungarian government to understand the share of public procurement procedures broken down into lots. It could be relevant for the Hungarian government to consider introducing a break down by procurement object to have a better understanding of the use of allotment strategies in specific sectors or procurement categories. To support the analysis of the sub-indicator on “procurement object” it could be interesting to also look at CPV divisions when relevant. This analysis cannot be included as a sub-indicator as it would entail adding 45 additional sub-indicators.
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
3.2.6. Efficiency indicators – Cost-effectiveness /administrative costs
As mentioned earlier, assessing the efficiency of procurement processes goes beyond procurement outcomes, it also includes the time to undertake and complete specific processes and should consider the related costs. Furthermore, efficient processes require understanding the functioning of the market and its characteristics to be able to correctly estimate the value of the contract.
27 Cost for contracting authorities of running a public procurement procedure
Running a procurement procedure can be costly for contracting authorities. In addition to internal costs such as the salaries of the officials involved, the cost of facilities, contracting authorities need to account for the cost of external procurement experts and administrative fees such as the fees for the use of the EKR system and the administrative service fees for the various controls carried out by the PPA. Understanding the cost of procurement procedures could help contracting authorities and governments to take informed decisions in relation to procurement strategies such as the use of centralised procurement schemes.
The Hungarian government introduced the indicator on “cost for contracting authorities of running a public procurement procedure” which is calculated from data provided by contracting authorities who responded to the survey. This indicator includes a sub-indicator capturing the costs for EU procedures vs national procedures.
As mentioned in Section 2.3.4, the question addressed to contracting authorities to assess their costs lacks clarity and could be improved. The exact methodology to calculate the costs should be described in the measurement framework and reflected in the question of the survey.
28 Cost for economic operators of bidding in a public procurement procedure
The cost for economic operators to bid in a procurement process can have a significant impact on the level of competition (OECD, 2022[53]). Therefore, understanding these costs could support policy actions to enhance the participation of economic operators in the public procurement market.
The Hungarian government introduced an indicator on the cost for economic operators. Similarly, to the indicator on cost for contracting authorities, the question addressed to economic operators lacks clarity and could be improved. Therefore, the exact methodology calculates the costs should be described in the measurement framework and reflected in the question of the survey.
29 Ratio of the estimated value of lots to the contract price for the awarded lots in public procurement procedures
This indicator enables to understand if the estimated values of lots are close to the real contract values. To further refine the analysis, it is complemented by two sub-indicators. The first one considers the contracts where the estimated value is larger than the contract price and the second considers the reverse case. In addition to further clarifying the methodology used, it could be beneficial for the Hungarian government to consider adding as a specific sub-indicator of the “median” values.
30 Share of number of lots in which a tendering security has been lodged in respect of the number of lots awarded
Bid bonds, guarantees or other financial commitments can represent a barrier to participate in the public procurement market. In Hungary, Article 64 of the PPL provides contracting authority with the possibility to require a tender guarantee from bidders. It is relevant to understand to what extent this provision is used by contracting authorities.
Some sectors might be more prone to use bid bonds, therefore considering adding sub-indicators by procurement object could bring a value added to the measurement framework and the underlying analysis.
3.2.7. Efficiency indicators – Effectiveness of contract performance
Public procurement can be impacted by a wide range of risks that can affect the procurement process itself, as well as broader risks to projects or service delivery. Risks do not occur only during the tendering process, but over the life of the contract or the life of the procured asset (OECD, 2023[54]).
The contract management phase of the procurement cycle is often overlooked when it comes to generating added value from procurement processes, but actually it can play a significant role (OECD, 2023[5]). Aware of this strategic role, the Hungarian government introduced four efficiency indicators dedicated to the contract management phase.
31 Share of the number of contracts that were fully terminated in the course of contract performance
A contract can be terminated for several reasons. It is then relevant to understand the number and share of contracts that were terminated during the contract management phase. This indicator is one of the seven indicators part of the Hungarian commitments under the conditionality procedure.
However, some issues leading to the termination of a contract might be specific to some sectors. Therefore, considering adding sub-indicators by procurement object could bring a value added to the measurement framework and the underlying analysis. To support the analysis of the sub-indicator on “procurement object” it could be interesting to also look at CPV divisions when relevant. This analysis cannot be included as a sub-indicator as it would entail adding 45 additional sub-indicators.
32 Share in value of contracts that were fully terminated in the course of contract performance
In addition to the indicator on the contracts that were terminated during the contract management phase in terms of numbers, it is relevant to also have the information in terms of value. Like the indicator in numbers, considering adding sub-indicators by procurement object could bring a value added to the measurement framework and the underlying analysis.
33 Number and share of contracts affected by contract modification
When a public contract needs to be modified, the starting assumption is that the modification will trigger the requirement for a new competitive public tender process. It is not generally permitted for a contracting authority and an economic operator to agree to change an existing contract. In practice, however, the limited modification of an existing public contract can be necessary. The agreement to change the contract should not represent a breach of the principles of transparency and equal treatment. Contracting authorities and economic operators might be faced with legitimate situations that require changes in the contract (OECD-SIGMA, 2016[55]).
In Hungary, modification of contracts is subject to PPL rules and PPA monitoring. The PPA carries out this monitoring task according to an action plan (set up for each budgetary year and published on the PPA’s webpage). If the PPA finds that the amendment or the execution of the procurement breached the law, it is obliged to initiate the review procedure of the PPAB. It is also required to submit a motion in court to declare the modification of a contract illegal.
It is relevant to understand to which extent contracting authorities are modifying their contracts. Here again, considering adding sub-indicators by EU vs national procedures, EU vs domestic funds and procurement object could bring a value added to the measurement framework and the underlying analysis.
Furthermore, in specific cases, a contract could be modified more than once, it could be relevant to add a sub indicator on the average number of modifications.
Last, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
34 Share of contracts (in numbers) where there was a delay in the performance of the contract (compared to the originally planned deadline)
Goods, services and public works should be delivered at the time mentioned in the contract. A delay in contract performance could have significant impact on public service delivery. However, in practice, different circumstances could explain the need to change the initial final day for the performance of a contract.
It is relevant to understand the share of contracts with delays. This indicator is one of the seven indicators part of the Hungarian commitments under the conditionality procedure. Considering adding sub-indicators by EU vs national procedures, EU vs domestic funds and procurement object could bring a value added to the measurement framework and the underlying analysis.
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
35 Share of contracts affected by cost overruns
Like delays in the performance of contracts, different circumstances could explain cost overruns. For instance, wrong estimation of the value of the contract, but also foreseen external shocks, such as Russia’s war of aggression against Ukraine, that are increasingly affecting our globalised economies, and thus the prices of goods and services (OECD, 2023[44]). The Hungarian measurement framework mentions the share of contract affected by cost overruns as an indicator which is one of the seven indicators part of the Hungarian commitments under the conditionality procedure.
This indicator is complemented by two sub-indicators: ii) the volume of contracts affected by cost overruns in HUF and ii) cost overrun expressed as a percentage. These two sub-indicators could be considered as separate indicators as they do not represent a breakdown of the main indicator. Like the indicators on delays, considering adding sub-indicators by EU vs national procedures, EU vs domestic funds and procurement object n could bring a value added to the measurement framework and the underlying analysis.
Furthermore, the methodology mentions that this indicator does not include contracts concluded under framework agreements and design contests. The methodology should clarify if contracts concluded under a DPS are included.
3.2.8. Efficiency indicators – Competition
As mentioned in Section 2.1, competition issues and more particularly single bids (contracts awarded in procedures where there was only one bidder) triggered the development of the measurement framework. While different indicators of the Hungarian measurement framework can have an impact on competition (OECD, forthcoming[11]), some specific indicators are captured in a dedicated subcategory of the framework.
36 Number of preliminary market consultations
Buyers should undertake market analysis prior to tendering in order to better understand the range of goods and services on offer, market developments and innovation, what commercial models are available, the competitive landscape, and the specific capabilities of SMEs, etc. (OECD, 2018[56]). Engaging with market actors is key to have a clear understanding of the market. This could be performed through different ways including through the “preliminary market consultation” (PMC).
In Hungary, Article 28 of the PPL provides contracting authorities with the possibility before launching a procurement procedure, to hold a PMC with independent experts, public authorities and market participants in order to prepare the procurement procedure and to provide economic operators with information on the planned public procurement opportunity and the requirements thereof (Hungarian government, 2021[12]). This obligation is at the procedure level and not at the “lot” or “contract” level. The methodology mentions that the breakdown by year is the year of publication of the contract award notice. However, in a specific procedure where a PMC is used it is possible to have contracts awarded in different years (for instance if one lot has been challenged and then awarded the next year). The methodology should therefore clarify this point.
37 Number of comments received during preliminary market consultations
To understand if PMCs are effective in engaging the market, it is relevant to assess the number of comments received. However, this number in absolute value cannot give alone a real sense of how active economic operators are in providing comments. It could be interesting to explore the average number of comments received in a specific procedure and to also undertake this analysis per procurement object.
38 Share in number of lots awarded to one bidder
As described in Figure 2.1, the proportion of single bid procurement remains among the highest in the European Union. Therefore, it is particularly relevant to have a dedicated indicator covering this issue and following the methodology used in the Single market scoreboard. This indicator is one of the seven indicators part of the Hungarian commitments under the conditionality procedure.
The exact methodology used for this indicator is not described in Annex 3 of the report although the data is based on EKR; information on the methodology is available in Annex 1 of the report which focuses on single bids. The Hungarian government would benefit from including the detailed methodology used for this indicator in the measurement framework.
The indicator includes several sub-indicators that are necessary to further understand the extent of the issue and that can support the government to take informed decisions: i) EU vs national procedures, ii) EU vs domestic funding, iii) procurement object, iv) CPV divisions and v) place of performance of the contract (county). Although not formally considered as sub-indicators, Annex 1 of the report also provides for a combination of the different sub-indicators (i.e. EU vs national procedures and procurement object).
Furthermore, as mentioned earlier the PPA and the PMO are using different methodologies to calculate this indicator. Indeed, the PPA bases its calculations on the number of bids submitted at the procedure level (and not at the lot level) while the PMO calculates this indicator based on the number of bids submitted per lot for contracts awarded. As such, for instance, if one procurement procedure includes10 lots, and for each lot only one bid is submitted, the PPA will account this as a procurement procedure with 10 bids (thus not considered as a single bid procurement), while the PMO will account 10 lots or contracts awarded to one supplier. This discrepancy can drastically change the findings regarding single bid procurement and impacts the efficiency and outcomes of public procurement procedures. Although PPA committed to publish results using the two methodologies, the PPA and PMO should initiate discussions to ensure the use of one methodology at the national level capturing the reality of the level of competition in the public procurement market.
39 Share in value of lots awarded to one bidder
In addition to understanding the share of single bids in terms of numbers, it is relevant to have a similar analysis in terms of value.
40 Median number of bids received per lots awarded
Another indicator that is relevant to assess the level of competition in the performance measurement framework is the median number of bids received per lot in public procurement procedures. The median is sometimes used as opposed to the average when there are outliers in the sequence that might skew the average of the values. Through this indicator, the Hungarian government could have a good understating of the practices in terms of bidding. Considering adding sub-indicators by EU vs domestic funds, procurement object and main CPV division could bring a value added to the measurement framework and the underlying analysis.
41 Mode number of bids received per lots awarded
Not initially foreseen, the Hungarian government added the” mode number of bids received” as an indicator. The mode is the value that occurs most frequently in a data series, this could help to understand what the most frequent number of bidders in the public procurement market is. Considering adding sub-indicators by EU vs domestic funds, procurement object and main CPV division could bring a value added to the measurement framework and the underlying analysis.
42 Number of decisions establishing a breach by the Competition Authority in public procurement cartel cases
As mentioned in Section 2.2.2, the competition authority could play a key role in enhancing competition in the public procurement market as it is responsible for safeguarding competition and it has the mandate to investigate public procurement cartels and take actions against anti-competitive behaviours. The number of decisions in relation to public procurement where the competition authority issued a decision establishing a breach is relevant but should be compared to the number of cases submitted/investigated by the competition authority. This additional analysis could be introduced as a sub-indicator.
3.2.9. Efficiency indicators – Capacity
The capacity of the public procurement workforce is a crucial element of a sound procurement system that delivers efficiency and value for money in the use of public funds (OECD, 2023[57]). However, public procurement through its strategic nature requires a more complex skillset for public procurement officials that implement public procurement procedures. Aware of the key role of capacity, the Hungarian government included eight dedicated indicators covering this area that are calculated using data collected through the general survey.
43 The organisation has its own methodology for measuring its procurement performance
It is relevant to understand if contracting authorities have developed a methodology to measure public procurement performance. This indicator could signal the level of maturity of the entity in this area.
44 Use of services of an external consultant with expertise in the subject matter of the procurement have been used by the organisation
Efficient public procurement requires a specific” technical” expertise in relation to the subject matter of the procurement. This lack of expertise may impact public procurement outcomes. Aware of the potential impact of the knowledge and expertise on public procurement outcomes, the Hungarian framework includes a dedicated indicator to understand whether contracting authorities are using external consultants to provide technical expertise.
This indicator is complemented by a sub-indicator providing a breakdown by procurement object. In addition to tissues related to the design of the survey (in particular regarding the need to clarify the time period considered), the extract methodology requires to be added to the measurement framework.
45 Employment of an Accredited Public Procurement consultant or a procurement expert to perform procurement consultancy tasks.
Contracting authorities may employ accredited public procurement experts or non-accredited ones to perform procurement tasks throughout the procurement cycle. This indicator aims understanding if contracting authorities employ these experts.
46 Reasons for the organisation to use the services of an external consultant
Contracting authorities can use the services of external consultants for different reasons. It is relevant to understand these reasons to take informed decisions. The methodology should list all the potential reasons as listed in the question survey.
47 Percentage of procedures involving external experts
In addition to understanding if contracting authorities rely on external consultants to conduct or manage their procurement activities, it is relevant to understand the share of procedures where external experts or consultants are involved. The extract methodology used to calculate this indicator should be specified in the measurement framework and in the survey.
48 Number of staff involved in public procurement.
It is important to understand the public procurement capacity within contracting authorities in terms of number of staff. As mentioned in Section 2.3.4, the question in the survey could be improved to bring more clarity (by mentioning "full staff equivalent”) and this should be reflected in the methodology that should be part of the measurement framework. Furthermore, a sub indicator capturing the number of accredited public procurement experts could further enrich the analysis.
49 Number of staff with higher vocational training in public procurement
While it is relevant to know the number of employees that benefited from higher vocational training, it is however important to define “vocational training” in the survey and in the methodology to ensure a common understanding of the concept.
50 Number of staff involved in the management of public procurement procedures in the institution who have received external or internal training in public procurement in a specific year
Like the previous indicator, it could be relevant to provide a clear definition of internal and external trainings and to avoid overlaps in responses with vocational trainings.
3.2.10. Efficiency indicators – Centralisation
When used adequately, centralisation is a key tool to achieve efficiency gains (OECD, 2015[8]). The Hungarian measurement includes a dedicated sub-category to centralisation that includes 20 indicators. These indicators were calculated based on responses to the survey that was sent to the four main CPBs in the country.
51 Number of contracting authorities using CPB services per CPB in a given year
Evaluating the efficiency of CPBs requires understanding to which extent contracting authorities are using their services. However, it is particularly relevant to understand those who are using CPB services on a voluntary basis or on a mandatory basis. The Hungarian measurement framework includes an indicator on the “Number of contracting authorities using CPB services per CPB in a given year” with a sub-indicator aiming at capturing the number of contracting authorities using CPB services on a mandatory and voluntary basis. It could be interesting for the Hungarian government to undertake this analysis by main procurement categories such as ITC (Information and communication technology), office furniture, medical goods, services, etc. The methodology to calculate this indicator should be described in the measurement framework.
52 Number of contracts concluded under framework agreements or dynamic purchasing systems of CPBs per CPB in a given year
While the number of contracting authorities using CPB services could be an interesting indicator to capture the activities of CPBs, it is relevant to also have an indicator on the number of contracts concluded under framework agreements or DPSs. However, it is not clear if this indicator also includes dedicated framework agreements concluded by a CPB on behalf of a specific contracting authority. The methodology to calculate this indicator should clarify this point and be reflected in the measurement framework.
The indicator also includes a sub-indicator on the number of contracts awarded under framework agreements, it could be beneficial to also have contracts derived from DPSs.
Furthermore, it could be interesting to explore as a sub-indicator contracts that were concluded by the CPB on a mandatory and voluntary basis.
53 Number of contracts concluded with SMEs/CPB in a given year
One of the main criticisms of CPBs is related to the size of their contracts as they might not be suitable for SMEs to participate and win the contract. Therefore, it is relevant to have an indicator on the number of contracts concluded with SMEs. However, it is also not clear in this indicator if it includes dedicated contracts conducted by CPBs on behalf of a specific contracting authority. This point should be clarified in the methodology that should be reflected in the measurement framework.
54 Number of requests received by CPB from contracting authorities to conduct procurement procedures by themselves.
As mentioned in the first report of the measurement framework (Hungarian Government, 2023[7]), when it is mandatory to use CPB services, the regulatory framework provides the possibility for contracting authorities to conduct their procurement procedures by themselves in specific circumstances (for KEF) or based on a decision of the central purchasing body. There can be different reasons for contracting authorities for requesting to conduct the procurement procedures on their own, including the lack of competitiveness of the CPB offer. This indicator aims at measuring the number of requests received by CPBs from contracting authorities to conduct procurement procedures.
55 Number of procurement procedures that have been referred back for own responsibility by the CPB
CPBs receiving requests from contracting authorities to conduct procurement procedures by themselves may accept or reject these requests. Therefore, this indicator aims at measuring the number of procurement procedures where the request has been accepted by CPBs. Instead of numbers, it could be more relevant to present the data in terms of share of procurement procedures that have been referred back for own responsibility by the CPB (compared to the number of requests).
56 Share of the value of successful purchases conducted in a specific year under the framework agreement to the total value of the framework agreement concluded in a given year/ CPB
This indicator aims at understanding the use of the framework agreement in terms of value. It could also enable to assess if the estimated value of the FA is correct (in case of overuse or underuse). It could be interesting to have a specific indicator by procurement category.
57 Procurement capacity in each CPB in a specific year (in number)
CPBs are usually considered as centre of excellence with a capable procurement workforce. As mentioned earlier, capacity is key for a sound procurement system and to achieve value for money. Therefore, it is relevant to understand the capacity available within contracting authorities. This indicator includes a sub-indicator that aims to capture if the experts employed are accredited or not.
In terms of methodology, it is particularly important to ensure the comparability of data and thus to mention in the survey and in the methodology to be added to the measurement framework that this data should be provided “full staff equivalent”.
58 Staff with a postgraduate certificate on public procurement/ CPB
For internal staff of the CPB, it is interesting to have data on their educational background and to understand if some of them have a postgraduate certificate on public procurement.
59 Number and percentage of staff trained externally or internally in public procurement in a given year /CPB
Supporting a learning culture in the public sector will ensure that skills are reinforced and regularly updated. It allows public officers to keep up with the fast-changing nature of work (OECD, 2023[57]). Therefore, it is important to regularly monitor the number public procurement officials trained externally or internally.
60 Use of CPBs of external consultants for the implementation of public procurement procedure (Yes/no)
While CPBs are often considered as centre of expertise, they may also rely on external consultants to support them in their procurement activities. Therefore, it is relevant to understand if CPBs are using external consultant services. However, the indicator and therefore the survey should also mention a specific timeframe considered as some CPBs were created many years ago. A detailed description of this indicator should also be added to the measurement framework.
Furthermore, it is possible to rely on external consultants with or without accreditation, a sub-indicator providing this information could further enrich the measurement framework.
61 Average use by a CPB of accredited procurement consultants’ services for procurement consultancy tasks per 10 procedures.
While some CPBs may be using accredited procurement experts, it is relevant to understand how often, in terms of number of procedures those services are used. The terminology “consultancy tasks” should be defined in the methodology that has to be included in the measurement framework.
62 Share of procurement procedures of CPBs involving the services of external consultants for public works per CPB
This indicator aims at measuring the use of external consultants involved in public works procedures. It is not clear if this indicator covers all external consultants or only the accredited ones. This clarification could be brought in the methodology that should be included in the measurement framework.
63 Share of Procurement procedures of CPBs involving the services of external consultants for services per CPB
This indicator aims at measuring the use of external consultants involved in the procedures aiming at procuring services. It is not clear if this indicator covers all external consultants or only the accredited ones. This clarification could be brought in the methodology that should be included in the measurement framework.
64 Share of procurement procedures of CPBs involving the services of external consultants for goods per CPB
This indicator aims at measuring the use of external consultants involved in the procedures aiming at procuring goods. Like the two previous indicators, it is not clear if this indicator covers all external consultants or only the accredited ones. This clarification could be brought in the methodology that should be included in the measurement framework.
65 Existence of a system to monitor market price evolution of procurement items covered by FAs/ CPB
Public procurement is an essential economic activity. Given the role of CPBs, it is particularly relevant for them to monitor the market price evolution of the procurement categories covered in the FAs and DPSs they manage. Therefore, it is relevant to understand if they have a dedicated system to monitor market prices. However, given the activities of CPBs, this system should not be limited to FAs it could also include DPSs. Indeed, even though DPSs by their nature are more competitive as they ensure an open competition throughout their duration, it is still important to ensure that procurement outcomes are competitive compared to market prices.
66 Percentage of price reduction compared to the original price as a result of reopening competition for a given year
This indicator aims at measuring price reductions that were achieved in a given year as a result of reopening competition. This indicator lacks clarity as it is not clear what is meant by several terminologies used: “reopening competition” and “original price”. Discussions with the PMO highlighted that this indicator is a “perception indicator” covering only framework agreements. It is therefore suggested to clarify it in the methodology and in the survey and to define these terminologies to ensure consistent and comparable data. Furthermore, this indicator may be procurement category specific, it would be recommended to add a sub-indicator by main CPV divisions managed by CPBs.
67 Percentage of savings achieved through centralisation in a given year (compared to market price)
While understanding how the prices paid by contracting authorities through the use of CPB services are positioned compared to market prices is relevant, in practice the calculation of this indicator and in particular the assessment of “market price” could be challenging. Each CPB may have its own methodology to assess savings achieved which impacts the consistency and comparability of data. Furthermore, there is no possibility for the PMO to assess the “veracity” of the data provided by CPBs.
The PMO is collaboration with CPBs should develop jointly a methodology that will be used regularly by CPBs to calculate their savings.
68 Percentage of time savings achieved through centralisation
One of the arguments for using CPB services is the time saved by contracting authorities. Therefore, it is interesting to have information on this area. However, it is not clear how CPBs will have information on time saved by contracting authorities as this may depend on the internal process and capacities of each of them. The relevance of this indicator is therefore questioned. If the PMO intends to keep this question, a detailed methodology should be provided in the survey and in the measurement framework to ensure a common understanding of the question.
69 Operation of an online marketplace or catalogue specifically for procurement below the threshold/CPB
E-catalogues or marketplaces are considered as a tool to improve procurement procedures, reduce duplication and achieve greater value for money This indicator aims at understanding if CPBs are operating an online marketplace or catalogues for below threshold procurement.
70 Frequency of conducting a preliminary market analysis before launching public procurement procedures
Given the size of contracts launched by CPBs, it is particularly relevant for them to conduct a market analysis prior to launching a procurement process. This indicator aims at understanding if conducting a preliminary market analysis is a common practice in CPBs. This indicator is based on a survey question that provides four possibilities: i) Yes, regularly, ii) Yes, but only occasionally, iii) Not common, but has happened, iv) Not at all common and has never been performed. However, no definition is provided for each possibility (e.g. how to differentiate between “Yes, occasionally” and “not common, but has happened”). In addition, the question, and thus the indicator does not mention the specific year or time considered. The question in the survey should be reviewed to provide the necessary clarity and the exact methodology, including definitions, should be described in the measurement framework.
71 Existence of a methodology for the preliminary assessment of procurement needs
Given the role of CPBs of aggregating needs of different contracting authorities, it is relevant to understand if they have a methodology to assess the needs of the different users of their services.
72 Existence of a framework for measuring the effectiveness of the public procurement procedures carried out (indicators for monitoring the effectiveness of the CPB and of each procurement procedure and their evaluation)
CPBs can largely contribute to enhance the efficiency of public procurement processes. It is therefore relevant to understand if they have in place a framework for measuring the effectiveness of the procedures they carry out.
3.2.11. Strategic indicators
Public procurement is increasingly recognised as a strategic instrument for achieving government policy goals aligned with the 2030 Agenda for Sustainable Development (SDGs), including promoting a circular and green economy, stimulating innovation, supporting small and medium-sized enterprises (SMEs), and promoting ethical behaviour and responsible business conduct (OECD, 2023[57]).
The Hungarian public procurement measurement framework includes five indicators to capture the use of public procurement to achieve some policy objectives.
73 Share of lots with environmental aspects (in numbers)
Many economies are using public procurement to advance the environmental agenda in line with national policies and international commitments (OECD, 2023[2]). It is therefore relevant to include a dedicated indicator to understand to which extent public procurement contracts include environmental aspects and thus contribute to national and international commitments.
This indicator is complemented by a sub-indicator covering CPV divisions. Considering adding sub-indicators by EU vs domestic funds, procurement object could bring a value added to the measurement framework and the underlying analysis.
Moreover, the methodology used to calculate this indicator is described in Annex 3. However, the methodology could be improved by providing a detailed explanation of what is meant by environmental aspects and how they can be included in the tender documentation (e.g. award criteria, qualification criteria, technical specifications, contract performance clauses, etc.).
74 Share of lots with social aspects (in numbers)
Understanding how public procurement contributes to enhance social conditions in a relevant indicator to monitor. This indicator is complemented by a sub-indicator covering CPV divisions However, similarly to the indicator on the environmental aspects, it could be beneficial to consider; i) adding sub-indicators by EU vs domestic funds, procurement object could bring a value added to the measurement framework and the underlying analysis, ii) improving the methodology by providing a detailed explanation of what is meant by social aspects and how they can be included in the tender documentation (e.g. award criteria, qualification criteria, technical specifications, contract performance clauses, etc.) and, iii) adding a dedicated indicator in terms of value of lots.
75 Share in number of lots awarded to micro and SMEs
SMEs represent most businesses in the country, understanding to which extent they access public procurement market could help policy makers to take the necessary actions. The indicator on the share of lots awarded to SMEs is one of the seven indicators part of the Hungarian commitments under the conditionality procedure. It is complemented by two sub-indicators, one in relation to the CPV divisions and one another one in relation to CPV groups.
Considering adding sub-indicators by EU vs domestic funds, foreign vs. domestic SMEs, could bring a value added to the measurement framework and the underlying analysis. The share of lots awarded to different categories of SMEs (micro vs small vs medium enterprises) were already calculated in the first report of the measurement framework but could be added formally as sub-indicators. Furthermore, in addition to the “number of lots”, it could be relevant to have a dedicated indicator in terms of value of lots. As framework agreements are sometimes considered an impediment to SME participation due to their large value, it could be useful to add a sub-indicator on the number and value of framework agreements awarded to SMEs.
The methodology mentions that in case of consortium, data considered is the one of the lead bidder, which might impact the representativity of SMEs and in particular of micro enterprises. Therefore, it is recommended to explore also this indicator without integrating consortia.
76 Share of bids submitted by micro and SMEs (among the valid bids)
In addition to understand the share of contracts awarded to SMEs, it is also relevant to assess their level of participation in the public procurement market which is the purpose of this indicator. This indicator is also complemented by two sub-indicators one in relation to the CPV divisions and one another one in relation to CPV groups. Considering adding a sub-indicator on the categories of SMEs (micro vs small vs medium enterprises) could bring a value added to the measurement framework and the underlying analysis.
The methodology mentions that in case of consortium, data considered is the one of the lead bidder, which might impact the representativity of micro enterprises. Therefore, it is recommended to also explore this indicator without integrating consortia.
77 Share of contracts awarded (in numbers) where the full life cycle of the contract or life cycle costing was taken into account
The LCC approach moves beyond the initial purchase price and evaluates all other significant cost flows over the entire life period of works, supplies or services, such as installation, operation, maintenance and end of-life (disposal) costs (OECD, 2022[19]).
The Hungarian regulatory framework for public procurement provides ample room for the use of green public procurement approaches and LCC methodologies. However, in practice the use of LCC is limited. To enhance the use of LCC, the Hungarian government included specific measures to promote LCC as part of its green public procurement strategy (source). It is therefore relevant to monitor the use of LCC through a dedicated indicator. This indicator is one of the seven indicators part of the Hungarian commitments under the conditionality procedure. It should be linked with the indicator on the use of different award criteria.
3.3. Additional indicators to consider for improving the measurement framework
As mentioned earlier, the measurement framework developed by the Hungarian authorities goes beyond their initial commitments. While the previous section provides recommendations on how to improve the existing indicators, this section aims at providing guidance to the Hungarian authorities on the areas that are not currently in the framework and that could be explored to further enrich the framework in light of the issues faced by the Hungarian public procurement system. However, it is first recommended to strengthen the existing framework before enriching it.
3.3.1. Providing general data on the main breakdowns envisaged in indicators to understand their relevance
Many indicators include sub-indicators representing a breakdown by for instance “EU and national procedures”, procurement object, CPV divisions or place of performance. However, there are no specific indicators in the framework covering these areas which does not enable to understand the relevance of these breakdowns and thus of the sub indicators. For instance, PMO data shows that between 2021 and 2023, procurement procedures using EU funds may represent between 25 to 38% of the total procurement volume in Hungary which highlights the relevance of integrating this data in sub indicators.
3.3.2. Including indicators on “Values” of public procurement spent
Most of the indicators of the Hungarian framework are based on “number of lots" or “procurement procedures”. The Hungarian framework includes only 4 indicators based on values (see Table 3.2). While having the indicator in terms of numbers of lots or procedures in relevant, it is pivotal to also have indicators in terms of “value” to have a real picture of the practices and to draw the necessary conclusions. When relevant, the Hungarian government should consider integrating in the measurement framework indicators on “values” alongside indicators on “numbers”.
3.3.3. Compliance: Considering integrity and control issues
As mentioned in Section 2.1, the Hungarian public procurement system faces issues in relation to integrity and the monitoring/control of the public procurement. Introducing indicators related to these issues could further enhance the measurement framework system.
On the remedies system, it could be relevant to explore, the duration of the appeals procedures before the relevant court and the share of preliminary dispute resolutions that were upheld or rejected (either on the substance or procedural issues).
3.3.4. Efficiency: further indicators to assess the level of competition in the public procurement market
For competition purposes and to assess the efficiency of procurement outcomes, it is relevant to include several indicators listed in Table 3.3. With regards to access to procurement opportunities, it is relevant to have data on sub-contracting and consortia formed by economic operators and to have an additional indicator on the use of PMC (complementing indicators 36 and 37 on PMCs)
The framework includes dedicated indicators on framework agreements, it could be relevant to have more information on DPSs, including the average number of economic operators’ part of the DPS, the average number of economic operators to whom a contract under a DPS is awarded and the average number of extra suppliers joining the DPS during its execution.
In addition to the average value of lots, it is relevant to include several indicators and sub-indicators on the number of bids, more specifically on the “average number of bids”, “the share of bids submitted by domestic vs foreign suppliers” (to understand how is defined the relevant market), and the “average number of bids” when i) using BPQR, ii) using the lowest price criterion, iii) including environmental aspects, and iv) including social aspects. These last indicators aim at understanding if including quality criteria or aspects in relation to policy objectives have an impact on competition.
In addition, some indicators are recommended to assess the market concentration aiming at assessing the extent of domination of some economic operators in a particular market. These indicators are not the traditional ones used in the calculation of market concentration (like the Herfindahl index); however, they can give an idea about market concentration and domination. Therefore, it is suggested to assess the number of contracts awarded to the top X suppliers and the value of contracts awarded to the top X suppliers (“X” to be decided by the Hungarian government). These indicators could be developed together with relevant stakeholders including the competition authority and the integrity authority.
Given that payment delays impact also the access to procurement opportunities, it is relevant to have a dedicated indicator to understand what the real practice in light of the legal requirement is. While this data is not available centrally, PMO should work first towards accessing this data before including a dedicated indicator in a later stage.
3.3.5. Strategic public procurement: providing data on “values” and monitoring the implementation of innovation procurement
Indicator on strategic objectives could be further developed. Indeed, while some existing indicators can be considered as relatively advanced compared to other countries such as “the share of contracts for which the full life cycle of the contract or life cycle costing was taken into account”, other key aspects of strategic procurement such as innovation are not covered. Furthermore, all strategic indicators are considered only in terms of “numbers of lots”, adding indicators on the value of lots could further enrich the measurement framework. With the implementation of eForms, data on innovation with be available starting from 2025.
