Strengthening Oversight of the Court of Auditors for Effective Public Procurement in Portugal

3. The data landscape in Portugal and opportunities for enhancing the TdC's audits of public procurement

3.1. Introduction

Public procurement is a complex process that involves multiple actors and institutions, from which several sources of information are essential for audit activities. As with all data-related endeavours, when a task requires assessing data from multiple sources at different levels of granularity, data mapping is a critical first step. The main challenge in data mapping lies in the data landscape, which involves a variety of potential data owners, influencing data accessibility.

This chapter presents a comprehensive overview of the data landscape in Portugal, specifically identifying data opportunities to enhance the Tribunal de Contas (TdC) - Portuguese Court of Auditors' - audit tasks related to public procurement activities. The primary objective is to identify key variables from stakeholder databases that can serve as valuable sources to identify core risks and irregularities in public procurement. Specifically, the chapter highlights crucial features of these databases, particularly data quality and appropriate methods to identify risks/irregularities, delving into the first steps for implementing a data-driven framework.

By focusing on these essential aspects, this chapter lays the groundwork for developing a robust data-driven approach that empowers the TdC to enhance its capacity to assess risks related to public procurement, thereby strengthening its oversight of public procurement activities in Portugal. The approach to assessing the data landscape involved desk research, meetings with important stakeholders, online questionnaires, and analysis of data sources. Specifically, this task included analysing existing databases, government platforms, legal documents, and other relevant sources. This data mapping exercise established a holistic view of the data ecosystem, facilitating a more accurate and informed analysis of opportunities for TdC's audits.

3.2. Overview of data mapping and activities undertaken

The data landscape encompasses all systems and processes involved in creating, collecting, processing, storing, and general use of data (internal analysis, sharing with third parties, etc.). Generically, a data mapping process involves defining the scope of the project, identifying relevant databases and data sources, and assessing the overall data landscape. The data mapping process also includes a quality assessment of the mapped data to ensure its accuracy and reliability. In other words, data mapping consists of surveying existing databases that are potentially relevant, profiling the available datasets (e.g., data quality, available features, and their data types, completeness of the dataset, and record key identifiers), and identifying opportunities/challenges to combine different data sources.

Thus, a data map serves as a comprehensive inventory of the available sources within the range of organisations related to public procurement in Portugal. Chapter 1 identified and described relevant databases for the project's scope, and Chapter 2 addressed opportunities and challenges for data integration.

This chapter addresses the downstream tasks related to data mapping, specifically identifying relevant methodologies and variables to assess risks and irregularities. Additionally, this chapter covers a quality assessment of the relevant available data sources. By undertaking this data mapping exercise, it is possible to identify the analytical requirements, allowing TdC to effectively leverage available data resources and optimise their audit activities to achieve more robust and informed outcomes.

Figure ‎3.1. Data mapping process

Source: Author’ elaboration.

3.3. Exploring approaches for analytical requirements

Identifying the analytical requirements is an important step in data analysis and decision-making since it is the cornerstone for aligning data collection and analysis. Specifically, understanding the analytical requirements allows us to effectively determine what data is needed, how it should be collected, and which analytical methods are most suitable. This is relevant to ensure that the data analysis process is efficient, relevant, and tailored to the needs of TdC’s audit activities.

Therefore, this task considered relevant data-driven risk/irregularity indicators compiled by the Court (see Chapter 1, Section 1.4.2). These indicators can be segmented on whether the analytical requirements are based on rules, statistical inference, or machine learning/statistical models. These categories are related to the complexity of the risk/irregularity indicator. Particularly, indicators assessed through rule-based methods are less complex and more intuitive than indicators assessed through machine learning models.

3.3.1. Rule-based

Rule-based approaches are methods that rely on a set of predefined rules to make decisions or solve problems. In a rule-based approach, knowledge is encoded in the form of if-then rules or condition-action rules. These rules consist of the antecedent (if part) and the consequent (then part). The antecedent specifies the conditions or criteria that must be satisfied, while the consequent defines the actions or conclusions to be taken if the conditions are met. When applied to a specific situation or problem, the rules are evaluated based on the available information or data, and the system follows the actions or conclusions specified by the rules that match the given conditions.

Rule-based approaches are particularly useful when well-defined rules or guidelines can be explicitly stated, and the decision-making process can be structured deterministically. Rule-based systems are often designed to handle complex sets of rules and can provide transparent and explainable reasoning, as the decision-making process is based on explicit rules. However, rule-based approaches may have limitations when faced with uncertainty or situations that require reasoning beyond simple if-then conditions. In such cases, other procedures may be more appropriate.

Table ‎3.1 displays a risk/irregularity assessment through rule-based methods. Specifically, the risk indicator that a call for tender was not published in the official journal. The failure to publish a call for tender can lead to legal and reputational risks, undermine fair competition, and compromise the efficiency and transparency of the procurement process. Thus, to determine whether a call for tender is published in official journal, an if-then condition that checks if the public procurement procedure is published in the official journal is enough to flag possible bids. Since not all call for tenders are required to be published in the official journal, this approach will contain additional conditions to restrict the indicator to the appropriate calls.

Table ‎3.1. Risk/irregularity considered in the rule-based approach

ID	Indicator Type	Indicator Group	Name of Risk Indicator	Approach for risk/irregularity assessment
1	Irregularities/Risks	Procurement procedure	Call for tender not published in official journal	If the fields related to the official journal are empty, then the bid will be flagged.

3.3.2. Inference-based

Inference-based approaches use statistical methods in decision-making or inference tasks. In inference-based processes, uncertainty is represented using probability distributions or probabilistic models. Instead of relying on strict rules or deterministic relationships, these approaches assign probabilities to different events or outcomes based on available evidence. In that sense, indicators compare the features in contracts against the expected (estimated through a sample), and deviations typically highlight cases that should be relevant to study further.

Table ‎3.2 exemplifies a risk assessment through inference-based methods. Specifically, the underestimation or overestimation of the contract value in relation to market prices. A rule-based method is not adequate, when it is required to obtain an estimation. An estimation adds complexity to the approach since it is necessary to account for the uncertainty of the sample. Thus, we can compare the contract value of the procedure with the estimated market price (obtained, for example, from a sample of similar bids, within the same type of procedure and/or within the same type of industry/sector).

The contract value is defined by the contracting body and it represents the economic value of the contract. The base price (upper limit of cost) is the maximum value the contracting body is willing to pay and it could be equal or lower than the contract value. The contractual price is the price of the winning bid and it should be less than the upper limit of cost. Thus, if the economic value of the contract (which determines the upper limit of cost, which in turn determines the contractual price) is over or underestimated, it represents a risk and it should be flagged.

The comparison between the contract value and the market price can be assessed through a statistical test or a confidence interval (Hyari, Tarawneh and Katkhuda, 2016[88]).

Figure ‎3.2. Inference-based approach

Source: Author’s elaboration.

This approach is suitable to identify more complex risk/irregularities than the rule-based methods. Additionally, these approaches are appropriate when the market prices can be estimated accurately. If that is not possible, alternative models (Hyari, Tarawneh and Katkhuda, 2016[88]) should be considered, placing these indicators in the model-based approaches.

Table ‎3.2. Risk considered in the inference-based approach

ID	Indicator Type	Indicator Group	Name of Risk Indicator	Approach for risk/irregularity assessment
1	Risks	Evaluation of bidders, tenders and award procedure	Underestimation/overestimation of the contract value in relation to market prices	Statistical Test Confidence Interval

3.3.3. Model-based

Model-based approaches refer to methods that utilise models to represent and understand a given system. These approaches involve constructing a representation or abstraction of the system, capturing its key components, relationships, and dynamics.

In model-based approaches, a model serves as a conceptual or mathematical/computational representation of the system under study. The model can take different forms depending on the context and requirements of the problem.

Table ‎3.3 contains a risk indicator that can be assessed through a model-based approach. Several approaches could be considered in this case, depending on the data (i.e., percentage of missing values) and the overall goodness-of-fit of the models.

Random Forest is an ensemble machine-learning method that combines multiple decision trees to make predictions. This approach can be used to identify bid-rigging cartels (García Rodríguez et al., 2022[89]; Huber and Imhof, 2019[90]; OECD, 2021[40]), identify connections between public and private entities (Popa, 2019[91]) and detect single bidding (Fazekas, Sberna and Vannucci, 2022[92]). Neural networks are a class of machine-learning models inspired by the human brain, which can also be used for bid-rigging cartels (Anysz, 2019[93]; Huber and Imhof, 2019[90]; García Rodríguez et al., 2022[89]). Support Vector Machines are a type of machine learning model that finds a hyperplane to separate data into different classes with the maximum margin. SVM can be used to detect single-bid in public procurement (Rabuzin and Modrušan, 2019[94]), collusion (Dadfarnia et al., 2020[95]; García Rodríguez et al., 2022[89]) and favouritism (Torres-Berru and López Batista, 2021[96]). Finally, logistic regression is a statistical model used for binary classification tasks, estimating the probability of an input belonging to a particular class and it has been used to detect single bidders and collusion (Fazekas, Sberna and Vannucci, 2022[92]; Rabuzin and Modrušan, 2019[94]).

In general, there is not a singular, universally superior method for identifying risks in public procurement (Lyra et al., 2022[97]). Numerous machine learning models can prove efficient and suitable, allowing for the implementation of various approaches.

Table ‎3.3. Risk considered in the model-based approach

ID	Indicator Type	Indicator Group	Name of Risk Indicator	Approach for risk/irregularity assessment
1	Risks	Conflicts of interest, fraud, and corruption	Bid rotation (price matching)	Random Forest Model Neural Network Models Support Vector Machines Logistic Regression

3.4. Identification of data requirements

Data requirements refer to the specific attributes and characteristics of the relevant data. Understanding the data requirements is important for several reasons. Firstly, it ensures that the data analysis process is accurate, as misaligned or irrelevant variables can lead to flawed results. Secondly, it contributes to the efficiency of the data analysis process by focusing resources and efforts on the most pertinent variables, thus saving time and reducing complexity. Moreover, a clear grasp of data requirements helps in identifying potential issues, such as missing or incomplete data, which can be addressed proactively.

The structure of data existing in TdC’s database management systems (DBMS) is broad, namely unstructured data (emails, images, videos, other formats), semi-structured (text published on the web), and structured data that comply with classification and formal logic. Currently, the information relating to the TdC's activity and residing in the DBMS is mostly structured, although there is a lot of unstructured, relevant information subject to processing. Moreover, TdC classifies the nature of the variables as usual: quantitative and qualitative (numeric vs. categorical). The first is presented in the form of numerical data, which may be discrete numerical data or continuous numerical data. Qualitative variables integrate data into categories, which can be ordered categories or unordered categories. The variables are also characterised according to their functional nature with characteristics such as transactional, master, reference, and metadata. Additionally, the origin of the data is classified as primary, secondary and tertiary. Primary data is obtained from the domain of the TdC itself. Secondary data is from an external source to the TdC and it is not publicly available. Finally, tertiary data is from an external source, but is considered public.

Considering the available data sources and the broad data landscape, the selection of relevant variables should consider the analytical requirements. Table ‎3.4 displays variable matching for the analytical requirements presented in the aforementioned tables (see Table ‎3.1, Table ‎3.2 and Table ‎3.3). For data-driven approaches, this process is indispensable for ensuring that all the relevant variables are included in the analysis and to increase efficiency and reliability.

Table ‎3.4. Data requirements from available sources

Risk/Irregularity indicator	Variables	Type	Source
Call for tender not published in official journal	Number of public announcement	Numeric	Secondary - IMPIC's Database
Underestimation/Overestimation of the estimated value and base price in relation to market prices	1. Winning bid price 2. Bid prices 3. Contract value 4. Type of procedure 5. Sector/Industry	1. Numeric 2. Numeric 3. Numeric 4. Categorical 5. Categorical	Secondary – IMPICS’s Database
Bid rotation (price matching)	1. Tender code 2. Type of procedure 3. Bid values 4. Winning bid value 5. Number of bids per auction 6. Date of tender 7. Location	1. Numeric 2. Categorical 3. Numeric 4. Numeric 5. Numeric 6. Date 7. Categorical	Primary – Public Contracting Files (econtas), GENT Secondary – IMPIC’s Database

Source: Author’s elaboration.

Despite the richness of the available data sources, it is critical to highlight the risks that the TdC will not be able to measure with the existing data sources, as well as the additional sources that could further improve the risk assessment methodology in the future. Specifically, IRN’s database is relevant to obtain data on the companies’ ultimate beneficiary, which is an important feature to determine conflicts of interests’ risks, such as if the owners/shareholders of the tenderer(s) are the same . In addition, AT’s data sources, such as invoice data (from the e-fatura database), would also be relevant in the context of the TdC’s audit activities to identify risks related to the payment or financial obligations. Specifically, for the former it would be relevant to determine if the amount paid is higher than the invoiced values declared by suppliers. Additionally, these external data sources could include features that would improve the machine-learning approaches by adding more complex attributes related to the tender in the model, thereby increasing their accuracy. Thus, although there are several available data sources that would allow the TdC to implement data-driven approaches, there are opportunities to build on and improve the initial risk assessment methodology by accessing additional data that is essential for assessing specific risks in public procurement. Table ‎3.5. Data requirements from unavailable sourcesmaps the data requirements concerning data sources that were unavailable at the time of developing the risk methodology during the project.

Table ‎3.5. Data requirements from unavailable sources

Risk/Irregularity indicator	Variables	Type	Source
Supplier partners are also procurement officials	→Supplier partners’ name Supplier partners’ VAT number	→Nominal Numeric	Primary – Public Contracting Files (econtas), GENT Secondary – IRN database
Owners/shareholders of the tenderer(s) are the same	Owners/shareholders VAT number	Numeric
Amount paid is higher than the invoiced values declared by the suppliers	→Payment amount (in tender) →Invoice value (declared to AT) →VAT number of supplier Date of payment	→Numeric →Numeric →Numeric Date	Primary – Public Contracting Files (econtas), GENT Secondary – AT invoice database (efatura)
Artificial split of works/services/supplies across several tenders	→Payment amount (in tender) →Invoice value (declared to AT) →VAT number of supplier →Date of payment →Date of tender Tender code	→Numeric →Numeric →Numeric →Date →Date Numeric

Source: Author’s elaboration.

Understanding data requirements in sources that are available requires a comprehensive analysis. This includes an examination of the indicators for risks and irregularities, first developed by the TdC, an evaluation of international examples outlined in the analytical requirements section, and a scrutiny of available data sources. This led to the identification of 15 primary variables (see Table ‎3.6). These variables were identified for their inherent significance and their widespread applicability for a data-driven risk assessment and enhancing the TdC’s audit activities.

Table ‎3.6. Relevant variables for the TdC’s data-driven risk assessment

Variables	Type	Source
Contracting authority	Categorical	GENT, Compliance Reporting System (eContas), IMPIC’s Database
Contract object	Categorical	Compliance Reporting System (eContas), IMPIC’s Database
ID Contract	Numeric	Compliance Reporting System (eContas) only from 2023, IMPIC’s Database
Type of contract	Categorical	Compliance Reporting System (eContas), IMPIC’s Database
Type of procedure	Categorical	Compliance Reporting System (eContas), IMPIC’s Database
Base price	Numeric	Compliance Reporting System (eContas), IMPIC’s Database
Contract price	Numeric	Compliance Reporting System (eContas), IMPIC’s Database
Award date	Date	IMPIC’s Database
Visa date	Date	Compliance Reporting System (eContas), IMPIC’s Database
Date of authorisation of the expenditure	Date	IMPIC’s Database
Due date	Date	Compliance Reporting System (eContas), IMPIC’s Database
Appointment date	Date	Compliance Reporting System (eContas), IMPIC’s Database
Payment day	Date	Compliance Reporting System (eContas) (only first day of payment), IMPIC’s Database
Payment amount	Numeric	Compliance Reporting System (eContas), IMPIC’s Database
Sum of payments made in the last 3 years to the successful tenderer in each type of procedure	Numeric	Compliance Reporting System (eContas), IMPIC’s Database
Number of interactions with the prior control department before obtaining visa	Numeric	GDOC

Source: Author’s elaboration.

The data requirements identified above consider mostly structured data. However, the TdC could benefit from using unstructured data as well to further refine its methodology for assessing risks in public procurement, as discussed in Chapter 2. For example, methodologies that use unstructured data can help the TdC to detect whether the tender had an insufficient definition of the object of the contract. This type of risk is relevant because the opacity of the object of the contract can restrict competition, thereby creating inefficiencies and undermining principles of legality, transparency and the public interest. In addition, the uncertainty in the object of the contract can contribute to encumbering the contract with the performance of complementary works, all of which have an impact on the financial result of the contract and the efficiency of public expenditure. By analysing the unstructured data contained within the tender through machine learning and AI methods, it would be possible to detect the opacity of the object of the contract. Additionally, PORTAL BASE's content can be validated against the information available in unstructured data (documents related to tenders and contracts). This approach would allow for content verification of the databases, which would lead to more accurate data, while taking benefit of the unstructured data sources.

3.5. Quality assessment

The final task of the data mapping consisted of quality assessment. The following criteria were considered:

• Relevancy: Ensuring that the data meets the needs of the Court, i.e., to what extent the data is pertinent for the user.

• Accuracy: Verifying the degree of validity, i.e., to what extent the data values are in conformance with the actual values and free of error.

• Completeness: Evaluating if there are any gaps or missing data, i.e., to what extent the data are sufficient for the task.

• Timeliness: Evaluating the speed of data retrieval, i.e., to what extent the data is up-to-date.

• Accessibility: Assessing the ease of data access, i.e., to what extent the data is available.

• Coherence: Ensuring data consistency, i.e., to what extent the data is aligned logically within the dataset.

Through to these guiding principles, it is possible to effectively prioritise the relevant databases to enhance the efficiency and effectiveness of TdC's activities. The tasks undertaken at this stage were mainly focused on data profiling and data auditing. Regarding the profiling, the goal was to analyse the structure and metadata. Regarding auditing, the purpose was to validate the data against expected values (in terms of data format) and find potential errors. The data profiling and audit were aligned, since the first allowed to identify the type of variable and the second allowed to check if the data values were according to the metadata definition. This quality assessment did not involve verification of whether the data was correctly inputted in the database, in terms of content, but if the format of the variables is according to the expected. The databases considered were GENT, Compliance Reporting System (eContas) and IMPIC’s Database.

The principles were evaluated quantitatively and/or qualitatively. Relevance was ensured in the previous task of the data mapping. Specifically, the relevance was assessed qualitatively, according to the content and the retrieved variables of the database, whether these were relevant or not for audit activities and for implementing the data science framework (see Chapter 1). Accessibility was also evaluated qualitatively, according to whether TdC had access to the database or not. Table ‎3.7 contains the metrics considered for the quantitative assessment of the remaining principles.

Table ‎3.7. Data Quality Metrics

Principle	Data Quality Metric	Formula
Accuracy	Percentage of data values that match expected or reference values.	1-Number of incorrect data valuesTotal number of data values
Completeness	Percentage of records with all required fields complete.	1-Number of incomplete/null data valuesTotal number of data values
Timeliness	Average number of days between date published in the database and the date the record was available to be imputed	∑i=1# recordsPublished datei-Available dateiNumber of records
Coherence	Percentage of data values with consistent values.	1-Number of inconsistent data valuesNumber of checked data values

The initial phase of the data quality procedures involved importing the provided sample data, which was predominantly available in either CSV or XLSX formats.

The accuracy assessment included ensuring the uniformity of data values, with a specific emphasis on verifying the appropriateness of data types throughout the dataset's columns and assessing the proper formatting of variables, particularly those of date types. An examination of anomalous values, such as the presence of negative figures in price-related variables or the uniformity of VAT numbers (9 digits for Portuguese contracting bodies/hired entities), was also an integral part of this accuracy measurement.

The assessment of completeness similarly entailed a column-based approach, subsequently aggregated to provide an overall perspective on the dataset's completeness. This assessment approach recognises that certain variables may exhibit substantial incompleteness but might not be relevant for subsequent analyses. It is important to note that the completeness assessment considered only the sampled data, i.e., the percentage of records that are complete. An alternative strategy, which was not applied in this assessment, involves evaluating potentially absent records by comparing against the comprehensive scope the database should encompass.

Given that the publication dates of records were unavailable for most of the databases analysed, the assessment of timeliness was predominantly through a qualitative approach, considering the insights from the data landscape.

The coherence measurement was initiated with an observation of the datasets, culminating in the formulation of defined rules and coherence checks between columns. An illustrative example of such a check could involve the validation that the total price corresponds precisely to the sum of the decomposed price variables.

The quality assessment considered samples of the identified databases. Using sample data instead of production-level data in quality assessment serves several important purposes. Firstly, it is more efficient. Working with a smaller sample set streamlines the assessment process. This approach requires fewer computational resources and reduces the time needed for analysis, which is especially beneficial for initial exploratory analysis and quick quality checks. Secondly, sharing smaller sample datasets for assessment purposes simplifies collaboration among different teams or stakeholders. It is easier to transfer, analyse, and discuss findings based on manageable sample data compared to handling large production datasets. Finally, by using sampled data, the data owners can mask or anonymise sensitive information while retaining the essential characteristics needed for quality assessment. However, it is crucial to ensure that the sample data is representative and accurately reflects the characteristics of the entire production dataset. Given the diversity and variability of public procurement data and considering that the selection process of observations was through random sampling, the following conclusions should be interpreted with caution.

3.5.1. GENT: TdC’s entity management system

TdC provided a random sample of the database of 1 000 observations and a support file that contained the row counts, total space used and completeness of each table. This file also included a description of the variables with the corresponding data type and a flag indicating if that given variable is a key variable for identification within the table.

The analysis of the data quality was focused on DB_Organismo, which is the most important table in this relational database. It holds critical information on the contracting body, and it connects to the other tables through the organism code ID variable. Table ‎3.8 contains the organism code, the date of creation and ending (it will be null if the entity is still in vigour), activity code, country, juridic form, codes to connect with other entities, such as the statistical body, the number and place of registry.

Table ‎3.8. GENT's quality assessment

Principle	Assessment Summary
Relevancy	Highly relevant database since it contains the universe of entities the Court plans to control (see Chapter 1).
Accuracy	For date variables, the date format was appropriate, and it did not show any errors. The VAT number of the entity presented some issues, namely 325 observations had this value field as zero and one observation had a value that was less than 9 digits. For this field, the accuracy measure is around 67.4%. This is particularly important because this field can be used to connect to different sources and normalise entity values, such as the name.
Completeness	DB_Organismo presented around 67% of complete records, considering the complete 20 626 records and not the sample of 1 000 records.
Timeliness	The data entries are imputed manually by a specially assigned team, responsible for managing GENT’s database. The sources from this imputation are IRN communications to TdC and the Portuguese National Gazette. Thus, a lot of the work involved in the data entry of this database involves a thorough analysis of the Portuguese National Gazette. This work is done periodically, thus fields available in this Gazette (or other more direct sources) are easily up to date. Some entry fields are not easy to update due to the lack of access to their corresponding sources and might not be regularly monitored. However, these fields are deemed more complementary attributes. The regularly reviewed are characterised by being structural fields of the entity and their corresponding standard accounting. There are some differences regarding timeliness between mainland Portugal and the islands, specifically, there is longer timeliness in the latter.
Accessibility	Very accessible, since it is managed by TdC.
Coherence	The main coherence check was regarding the dates, specifically the date of creation should be before the date of ending. The DB_Organismo table included two pairs of date-related variables, with start and ending date. The columns “Data_Inicio” (date of creation) and “Data_Fim” (ending date) presented 675 records where the date of creation was complete, but the ending date was null. Excluding these records, from the assessed sample, all the records presented the date of creation before the date of ending. The second pair of date-related variables is “Data_dr_Inicio” (date of creation, published in the Portuguese National Gazette) and “Data_dr_Fim” (ending date, published in the Portuguse National Gazette). For these variables, around 337 records have the date of creation complete and the ending date null and around 131 records had the creation date set to null and a complete value for the ending date. Excluding these records, all the records had the creation date before the ending date. Thus, excluding missing values the coherence measure is 100%. An additional and very relevant coherence check would be to verify whether the relationships and integrity between tables are correct. For example, to check whether an entity that has a designation but does not exist in the DB_Organismo table. However, to conduct this assessment, it is necessary to include all the records of the database instead of a sample.

3.5.2. Compliance Reporting System (eContas)

The compliance reporting system contains information regarding administrative hiring of public bodies. The information is submitted by the public entities through a reporting map which includes additional information to the budget data about all the celebrated contracts that were subjected to financial execution during the period. TdC provided a sample of the five different reporting maps,1 for five different accounting standards relevant for this purpose,2 within the compliance reporting system. Some of these reporting maps shared the same columns and thus were joined, leading to three different reporting maps: reporting map #1, reporting map #2 and reporting map #3.

There is not a specific variable to join this reporting maps with other sources. Currently, TdC joins through the VAT number of the contracting entity, the bidder and the amount with IMPIC’s database information. This could lead to some issues when the amount is wrongly inserted in IMPIC’s database, which then requires manual correction. These matching procedures are currently conducted on an ad hoc basis, handled individually by auditors for specific purposes. At present, the Court doesn't execute this matching as part of a comprehensive, organisation-wide process.

Table ‎3.9. eContas' quality assessment

Principle	Assessment Summary
Relevancy	Highly relevant database, containing important information regarding company data (see Chapter 1).
Accuracy	In the reporting map #1, the deadline variable is in text format and does not present a unique measure of time. Some entities report as “1 year”, others as “365 days”. The column related to VAT numbers for the contracting bodies did not present any issues. However, for the hired entities, the VAT number presented some issues. Specifically, 33 cases had a character value, such as the name of the company and 5 cases had a number with less than 9 digits (number of digits of the VAT in Portugal). However, since the suppliers of goods and services, i.e., the companies being contracted by public entities, can be foreign entities (not registered in Portugal), it is expected that the fiscal ID is distinct from Portugal’s 9-digit VAT Field. Finally, regarding the dates, a lot of accuracy errors arise, specifically, the observations were imputed with different date formats, which is an issue when analysing or applying operations to these columns. Regarding the reporting map #2, for the date variables, no issues related to formatting appear in this map. The VAT number of the public body had 201 observations having a value of more or less than 9 digits. Similarly, to reporting map #1, the deadline variable had inconsistent formats. Overall, the accuracy measure of the assessed reporting maps is around 96.10%
Completeness	The reporting map #1 presented a completeness measure of 88.46%, reporting map #2 of 87.33% and reporting map #3 of 95.46%. In reporting map #2, relevant variables such as the deadline and VAT number of the hirer had around 24% and 5% of missing values, respectively. The remaining variables of the reporting maps that presented missing values are related to observations columns or similar variables, which can be classified as irrelevant.
Timeliness	Yearly until April, however, the entities may ask for deadline extensions to deliver the information.
Accessibility	Very accessible, since it is managed by TdC.
Coherence	For the coherence assessment only the reporting map #1 and #3 were considered since no coherence checks were needed for the reporting map #2. The coherence checks assessed were (1) ensuring the contractual price without VAT is less than or equal to the contractual price with VAT and (2) ensuring that the total value of payments is equal to the sum of the decomposition payment variables. Additional coherence checks related to the date variables could be included, however since the formatting differs among columns, it would require extensive pre-processing, exiting the scope of this analysis. Overall, the coherence measure for map #1 and map#3 is, respectively, 99.29% and 98.98%. Similarly, to GENT database, an additional and very relevant coherence check would be to extend the coherence checks to include external sources, specifically verify whether the values from this reporting system are aligned with the ones provided to IMPIC’s obligatory contracting portal.

3.5.3. IMPICS’s Database

IMPIC provided 18 individual datasets in 27 physical files. The names of the files are defined according to the Annexes defined by Order No. 284/2019 (for more details, see Chapter 1). Around 9 of these files contained information regarding same Annexes, but with two different versions: one regarding the procedures and another regarding the contracts. The same information needs to be provided at different stages of the public procedure, hence these two versions. In a sense, the procedures files are a super set of the contract files, since they contain information on the procedures that were fulfilled as a contracts and those that did not had any bids or answers to the call. Currently, the Court only considers the tables regarding the contracts, thus the quality assessment was focused on those files only.

Table ‎3.10. IMPIC’s Database tables

Table	Observations per column (with duplicated records)	Variables	Duplicated Records	Completeness Measure
Annex II (for contracts only)	142378	31	0	55.98%
Annex III	2301	16	0	90.45%
Annex IV (for contracts only)	481767	9	0	99.71%
Annex IX (for contracts only)	5	7	0	100.00%
Annex V	572	7	0	85.71%
Annex VI	75	12	0	32.78%
Annex VII	354411	11	0	91.11%
Annex VIII	57868	10	0	63.77%
Annex XI	79814	5	0	98.39%
Annex XII (for contracts only)	6806	12	0	84.63%
Annex XIII	3	5	0	73.33%
Annex XIV (for contracts only)	30116	14	0	58.47%
Annex XV (for contracts only)	681	16	0	55.93%
Annex XVII	92	13	0	82.94%
Contracts (for contracts only)	602930	56	31171	49.77%
Lotes (for contracts only)	201347	9	0	92.92%
Portal Base (for contracts only)	189958	13	19294	49.15%
Procedures	142360	19	0	62.50%

Table ‎3.11. IMPIC's Database quality assessment

Principle	Assessment Summary
Relevancy	Highly relevant database, containing important information regarding public procurement data (see Chapter 1).
Accuracy	The “Lotes”, “Contracts”, “Annex II” and “Annex IV” tables presented accuracy issues. Specifically, “Lotes” and “Contractos” table presented 1 record, where the contract price was negative. Additionally, “Annex II” table had 2 observations with negative bid prices and the variable “Deadline” presented inconsistent formats (in some cases was depicted has “365 days”, in others as “1 year”). Overall, the accuracy measure for “Lotes”, “Contracts” and “Annex IV” table was 100% (the number of inaccurate observations was very small to have an impact on the measure), 96.77% for “Annex II”.
Completeness	The completeness measure for each table is displayed in Table 3.9. Although some tables present low values of completeness, such as “Annex IV”, “Annex II”, “Contracts” and “Portal Base”, it is important to note that these tables contain empty columns that are supposed to be non-empty only in specific cases, such information and/or description columns. Thus, are not generic relevant variables, that apply to all types of procedures/contracts. Overall, the tables assessed present a completeness measure of 96.29%.
Timeliness	The timeliness of the announcements depends of the type of procedures. If the process is published in the National Gazette is more automated, hence it is faster to publish in the database. However, if it depends on the entities, it could take longer and in some cases the announcement is only published by the entity when the contract is already signed.
Accessibility	Very accessible, publicly available and TdC has access to the private area.
Coherence	The coherence checks performed in the “Contracts” table were to check consistency in date-related variables (i.e., check if ending date of the contract is after starting date) and ensuring that if the procedure type was a direct award, then the flag column corresponding to that procedure was equal to one. For the “Contracts” table the coherence metric was around 100%. For “Annex IV” the date-related variables assessed presented a consistency of 99.97%.

3.6. Strengths and weaknesses of available data sources

The quality of any data endeavour depends on the availability and quality of relevant data sources. Moreover, features regarding award, tender, hired and contracting bodies are crucial for enhancing the Court’s audit activities through a data-science framework.

The universe of available data sources includes several relevant variables which can be used to implement simple methods, such as rule-based methods and more complex methods, for example machine learning algorithms. In addition, the available data sources are not managed by a single entity, allowing to enrich the records available, which in turn reduces potential biases in the data. Finally, the Court has detailed knowledge of the data sources, which is beneficial for contextual understanding and interpretation.

Nevertheless, there are some weaknesses that should be considered. Firstly, since the available data has different sources, there are different variable formats and structures, which adds complexity to the data integration. Secondly, another challenge is the presence of missing values. For example, the contracting body VAT number in the GENT database presented some completeness issues. This variable is important to identify an entity and, in some cases, join different data sources. Thus, missing values associated with the VAT number could lead to information loss. In general, considering the variables identified in Table ‎3.6 and the quality assessment, there were no major issues identified. With the exception of some date-related variables, that had different formats within the same database. To mitigate these weaknesses, it is essential to thoroughly assess and preprocess the data to ensure its suitability for the intended analytical purposes. Finally, the timeliness of data can be a concern, the data is not updated in real-time and the timeliness varies widely among data sources. This could cause potential inaccuracies and the results should be analysed considering this particularity.

3.7. Conclusion

This chapter focused on the follow-up tasks related to data mapping, specifically the identification of analytical and data requirements and quality assessment of available data sources, with the goal to identify opportunities to enhance the TdC’s audit activities.

The risk/irregularities indicators matrix developed by the Court allowed to identify of three main analytical requirements categories: rule-based, inference-based and model-based. Rule-based indicators can be of high gain with a low cost of implementation and would already have a considerable impact on improving the TdC activities. Additionally, considering the rich universe of available data sources, model-based methods are relevant to capture more complex indicators.

From the quality assessment, the main challenge lies in optimising the processes to efficiently extract value from semi-structured and unstructured information, which can be found both within and outside the TdC. The available data has different sources, which leads to different formats across variables. Additionally, missing values on relevant features could be a potential issue. A thorough data preprocessing is essential to overcome some of these challenges.

Finally, with the data and methods identified, the auditors may apply analytical procedures for their activities within the Court and enhance the identification of risks and irregularities. However, it is important to note that, despite the potential of the available data sources, these are limited since they do not allow for the identification of all the risks. Thus, this work should be seen as the beginning of a data-driven journey within the Court, where the methods and data sources should be expanded, leading to a constant and iterative improvement of these approaches.

Notes

← 1.  The map 3000012 for entities that report in SNC AP is applicable to the entities in the full regime, in the small entities regime or in the micro entities regime.

← 2.  Local Business Sector (map 3000011), SNC Local Business Sector (map 300003), SNC-AP, Local Administration (map 3000013) and SNC-AP MNE Services (map 3000014).