This chapter presents the opportunities and challenges of the implementation of the new preventive and concomitant control, particularly in relation to its internal governance, the role played by the different dependencies of the Office of the Comptroller General and the cultural change needed for the development of this new tool. In particular, it highlights the challenges of internal co-ordination, data use and analytics as well as opportunities to leverage processes such as performance audits and sectorial studies.
Preventive and Concomitant Control at Colombia's Supreme Audit Institution
New Strategies for Modern Challenges
OECD Public Governance Reviews
2. Strengthening the Office of the Comptroller General for implementing preventive and concomitant control in Colombia
Abstract
From theory to practice: challenges in the internal governance of preventive and concomitant control in the CGR and recommendations for its institutionalisation and appropriation
There are no regulatory reforms which, alone, respond to all of the issues related to the implementation of a new project or mechanism. In essence, the institutional arrangements associated with any regulatory reform are perhaps equally or even more important to its success than the regulatory reform itself. For this reason, this chapter focuses on the successes and opportunities of the current arrangements and practices, as well as the imperative need for the appropriation of the mechanism and its various elements by CGR officials.
The legal changes to the reform introduced by Legislative Act 04 of 2019 and the new approach to preventive and concomitant control required adjustments to the internal institutional framework of the CGR. The CGR is divided into different dependencies. Among them, the Sectorial Delegate Comptrollers (Contralorías Delegadas Sectoriales) with macro and micro areas and the General Delegate Comptrollers (Contralorias Delegadas Generales), as well as the units and support offices (such as the DIARI or the Legal Office) and the Departmental Management Offices.
The CGR has four General Delegate Comptrollers (Figure 2.1). Under the preventive and concomitant control, the Delegate for Citizen Participation intervenes in relation to the observations of the citizens and the administration of the Internal Control Early Warning System (Sistema de Alertas de Control Interno, SACI).
Figure 2.1. General Delegate Comptrollers
In addition, in accordance with Law 267 of 2000, the Sectorial Delegate Comptrollers have two directorates: one for fiscal surveillance, whose function is to direct the implementation of policies, plans, programmes and projects related to micro fiscal surveillance and control. Equally, they have a macro fiscal control area, which includes a Direction for Sectorial Studies in charge of analysing policies, plans and programmes in the respective sector. Today there are 11 Sectorial Delegate Comptrollers Offices Table 2.1, who carry out special audits or investigations related to events of national impact that require the immediate intervention of the entity, due to the imminent risk of loss or affectation of public assets or to establish the occurrence of events constituting liability and to collect and store evidence for the corresponding processes.
Table 2.1. Sectorial Delegate Comptrollers
The Departmental Management Offices are branches of the Comptroller's Office located in the country’s departments in charge of controlling and surveillance of resources in the country's territories, under the direction of the Comptroller General. They were created in order for the CGR to have a presence throughout the national territory, strengthening fiscal control and efficiently carrying out prevention, investigation and punishment of acts of corruption and control of public management in the assigned territory. There are 31 Departmental Management Offices made up of a Departmental manager and no less than two provincial comptrollers (collegiate decision-making structure). Given that the preventive mechanism now includes 3 966 new control subjects at the territorial level, the Departmental Management Offices are central in the collection and monitoring of information related to preventive and concomitant control (DNP, 2021[1]). The implementation of this control over the new subjects is largely leveraged on the information compiled and provided by the Departmental Management Offices.
The fiscal surveillance directorates of the Sectorial Comptrollers' Offices and the fiscal surveillance groups of the Departmental Management Offices are in charge of the surveillance and control of fiscal management as developed by each of the control subjects forming part of the administration and of the individuals who handle state funds or assets. This work is carried out through different control actions, such as all types of audits and through special actions constituting of brief surveillance and control actions, in which an interdisciplinary working team addresses the investigation of a fact or matter brought to the attention of the CGR, by any means of information or by a citizen complaint (Contraloría General de la República de Colombia, 2021[2]).
Finally, through Decree 2037 of 2018, the Directorate of Information, Analysis and Immediate Reaction (DIARI) was created. The DIARI is a key element for preventive and concomitant control and, in general, in the digital transformation process of the CGR. The DIARI comprises the following units for Information, Information Analysis and Immediate Reaction:
The Information Unit is in charge of managing the access, storage, security and administration of the information sources of the different public entities or individuals that administer public resources or carry out public functions. It also receives the source identification information required by the analysis unit and the Sectorial and General Delegate Comptrollers Offices. The information unit also conducts technical meetings with the entities to define and implement the technical mechanisms for connecting information sources and ensures improvements in data quality.
The Information Analysis Unit designs and applies predictive and descriptive analytics models of the data provided by the information unit and carries out data analytics reports with alerts on risks of presumed detriment to public assets using, amongst others, “big data” techniques.
The Immediate Reaction Unit acts according to the alerts raised by the Information Analysis Unit and carries out any special actions that may be required.
Promoting a strategy that integrates and leverages the work of the DIARI in relation to preventive and concomitant control
The use of artificial intelligence, analytics and data mining is becoming a fundamental tool for risk management and preventive control (OECD, 2019[3]). Making effective use of data and analysis requires more than simply introducing new tools, technologies or data sources into the work of audit institutions. It requires a strategy, with clear goals and objectives at all levels of the SAIs and in particular, a strategy that line managers and those in charge of implementation are aware of and can support. A data analytics strategy should be able to serve the public administration in guiding investments and other public decisions. A strategy also provides incentives for continuous learning and aligning data and analytics with long-term goals. Data and analytics serve institutional goals. Defining these goals and articulating them is a critical step in an organisation's digital transformation and instilling a culture that promotes decision-based analytics rather than data-driven decision making (Langhe and Puntoni, 2020[4]).
INTOSAI promotes the modernisation of SAIs by issuing standards and guidance that emphasise the critical role of data in supporting SAIs missions. Specifically, INTOSAI asks SAIs to take a strategic approach to how they use data. Several INTOSAI working groups and partners, such as the Information Technology Audit Working Group and the INTOSAI Development Initiative, have issued practical guidance on how SAIs can achieve this. The Guide to Combating Fraud and Corruption also highlights the critical role that data and analytics and suggests policies, practices and tools for leveraging data in support of this specific goal. There is no single definition of "analytics" (used as an abbreviation for "data analytics" in this report). Data analysis is a process of inspecting, cleaning, transforming and modelling data with the aim of discovering useful information, informing conclusions and supporting decision making (INTOSAI, 2019[5]).
The DIARI was created for this purpose. To this end, it comprises the aforementioned Information, Information Analysis and Immediate Reaction units to which functions related to preventive and concomitant control, among others, were assigned. In particular, Decree 403 of 2020 established that “fiscal surveillance and control activities must be supported by information management and the efficient use of all available technological capabilities.”
Analytical models used by the DIARI to identify risks based on data
The DIARI uses a methodology for monitoring public resources based on information processing, the purpose of which is to produce data analytics reports and other reports and inputs on facts constituting alleged fiscal damage (Box 2.1). In the last year, the Analysis Unit deployed eight descriptive and predictive analytics initiatives to strengthen fiscal surveillance and control functions. Among them is a risk model for the processes in their contractual stage that involves descriptive data visualisation components allowing the characterisation of contracts, contractors and contracting parties. As such, dashboards were designed allowing the identification of the distribution patterns of the most prominent indicators, such as the frequency or value of contracts (Contraloría General de la República de Colombia, 2021[2]).
Box 2.1. Novell fiscal surveillance techniques in Colombia
The Directorate of Information, Analysis and Immediate Reaction (DIARI) of the Comptroller General of the Republic (CGR) of Colombia is made up of a multidisciplinary team of more than 130 people, trained in different disciplines. These people, according to their professional career and the entity's own needs, perform one or more roles with the following responsibilities:
Connecting information sources, carrying out data quality processes, guaranteeing information security and managing connected sources.
Quickly identifying the predictive value of information through exploratory analysis techniques, data processing and preparation.
Managing structured or unstructured, geographic or alphanumeric data sets.
Developing analytical models.
The DIARI makes intensive use of information technologies and related artificial intelligence to leverage the strategies and tactics of fiscal surveillance and control, whether preventive and concomitant, or ex post, supported by a robust physical infrastructure and logical architecture.
Among the products to be highlighted is the development of analytical models conceived as an abstract and reduced representation of phenomena resulting from the application of mathematical, statistical and computational techniques to associated data sets, in order to describe and explain their behaviour. Today, the DIARI has 22 analytical models.
For example, with the use of these technologies in the Public Works of Infrastructure Model, the CGR has managed to locate more than 2 000 infrastructure projects, including roads, educational institutions, hospitals, ports, airports and housing projects, among others, which are subject to surveillance and fiscal control.
Source: Information provided by CGR Colombia
Capitalising on the work of DIARI’s Information Unit in strengthening data governance in Colombia
According to the interviews carried out for this report, both in the public sector and in civil society, one of the factors that most hinders the correct implementation of preventive and concomitant control is the disarticulation of public data. Indeed, there is currently no unified, homogenised and open information system in Colombia. This disarticulation implies that public entities and control bodies often have their own isolated information system and the availability and quality of data may vary significantly.
There are issues related to the construction and development of databases in the public administration. Human data-entry results in errors and inconsistencies (for example, in the use of names of entities or even numerical inconsistencies in the data itself). Added to this is the difficulty caused by the lack of harmonisation of databases and their interconnection (interoperability).
Thus, Decree 403 of 2020 generated tools for the CGR to advance data governance in Colombia. In particular, it establishes in Art. 57 (a) a mandate to the Information Unit of the DIARI to overcome barriers related to data quality.
To this end, this Unit leverages its data quality strategy in the “Data and Information Management and Governance System of the Office of the Comptroller General of the Republic”, which aims to establish an appropriate orientation for the organisation, facilitating the efficient use of data and ultimately enabling business efficiency. To do this, it has organised its work into focuses, the main one being “data quality”. To this end, it has assigned a “data quality manager” in charge of supporting the chief data officer (CDO) in establishing the data quality strategy, as well as supporting the dependencies and domain leaders in the execution of processes, design of controls, monitoring of plans, monitoring of compliance and activities aimed at improving data quality. This policy has been established in order to improve the use of data within public entities and to reduce the time and expenses associated with reconciling data.
In case of any need to remedy the quality of data produced internally, the Information Unit will determine the timescale for such remediation by the CDO depending on the impact on the CGR (reputational, operational and legal risks). In this way, domain leaders will be able to propose updates to the quality rules and quality thresholds for critical data elements based on business needs and historical information. These updates must be shared with the CGR's Data and Information Governance Committee.
The CGR reported that this process has been essential in the definition of a strategy for the implementation of technological tools that provide support, among others, related to the requirements of performing data cleaning, the production of data quality reports and feedback to the public administration regarding corresponding remedial actions. As such, the Information Unit holds workshops with the public administration, which allow them to communicate quality findings identified and thus have better response times for adjustments to information sources.
The CGR has made efforts to establish criteria for the presentation of information by financial managers, in compliance with Art. 57 (a) of the aforementioned Decree. In particular, through Circular 09 of 2020, a scheme was defined for working with all entities. The Circular includes information related to the form of access and disposition of the sources of information, as well as the possibility of holding workshops to define connection mechanisms, the information required by the CGR, the form of presentation and the transfer mechanism. In summary, the Information Unit currently has 5 420 sources of information from more than 683 entities.
One of the most noteworthy aspects of the DIARI and in particular of the Information Unit, is the role it plays in providing feedback to the public administration on the improvement of data. This feedback allows the entity to know which data are indeed relevant for preventive and concomitant control. In particular, data clean-up and feedback work, feedback sessions through workshops and data quality reports are a valuable by-product. In this sense, the CGR has set the goal of closing agreements for the consolidation of the 5 420 sources of information.
Consequently, the Information Unit has developed a unique vision on data management in Colombia, with significant contributions to data quality and management. This work has the potential to nurture data governance policy in Colombia on a larger scale.
To this end, the CGR could consider various actions as a producer and consumer of data:
Measuring the progress of the quality of the data it receives and incorporating this component into the methodology of the Data Governance Model currently being designed.
Promoting, via workshops with the public administration and based on the technologies used by DIARI, the standardisation of a database model that allows the interoperability of high volumes of information, in such a way as to facilitate its automation.
Promoting, through the current spaces of interaction (workshops and feedback reports), the automation of all processes by the entities, in order to have structured and better quality data.
Working on the cleaning of data and feedback with other actors - including at the territorial level - private administrators and parafiscal funds. To this end, it could demand high quality standards in data reported via government mechanisms.
Communicating internally and externally the role of the "Data and Information Governance Committee" in such a way that it helps in the unification of the technological standard throughout the Comptrollers' Offices, facilitating the interaction of internal data.
Based on the model of the CGR's Data Governance Committee, ensuring that the public administration defines the design and implementation of an intersectorial committee (governing body) for the management and quality of data for all entities, in such a way that it helps in the unification of the technological standard throughout the entire Colombian public administration.
Strengthening the role of the Analysis Unit in risk analysis and giving it improved tools for its management
The data compiled by the Information Unit is communicated to the Analysis Unit for the construction of analytical models. These reports generally contain alerts about suspected fiscal damages. The Analysis Unit produces this information in relation to its own criteria and upon any necessary request by the General, Sectorial and Territorial Delegate Comptrollers. The Analysis Unit also provides support to these offices on the use of the model.
This is done based on advanced analytics models, which address the most recurrent business questions or general management issues with the greatest impact. To do this, it designs and executes the most appropriate analytical strategies to address working hypotheses, defines descriptive or predictive objectives, traces the data preparation route, information modelling and architecture and other analytical functions. Finally, it evaluates the results of the data mining process and contrasts them with the proposed objectives, carrying out a statistical and quality-based validation in accordance with the requirements of the end users of each of the models (Contraloría General de la República de Colombia, 2021[2]).
To this end, the Analysis Unit has various roles within the CGR. On the one hand, it provides support to the Sectorial, General and Territorial Delegate Comptrollers in the crossing of data for the support of audits. On the other hand, it produces reports where imminent risk of loss of public resources and/or negative impact on patrimonial assets is detected, which are delivered to the Sectorial, General and Territorial Delegate Comptrollers and the DIARI Immediate Reaction Unit. Finally, it produces dashboards, which help resolve recurring questions or work hypotheses prioritised by the CGR mission teams and provides support and technical assistance to the end users of the models.
Specialised teams in data analytics within SAI to reinforce risk identification
Specialised groups for data analytics, such as DIARI’s Analysis Unit, have created new opportunities to assess the risks of fraud and corruption in various countries. For example, the Korea Fair Trade Commission uses the Bidding Manipulation Indicator Analysis System (BRIAS) to analyse large volumes of data from Korean public entities and create a probability score for bid rigging. In Chile, the government uses data mining from the electronic procurement system to prevent collusion and favouritism. In the United Kingdom, Italy and Turkey there are also examples of the use of analytics groups for risk identification, as well as the design of methodologies for the use of analytics in the development of audits (Box 2.2). Such efforts have developed rapidly in recent decades, as governments embrace digital strategies and take advantage of open data, "big data" tools and data analytics (OECD, 2019[3]).
Box 2.2. Examples of using analytics in SAIs
United Kingdom
The United Kingdom’s National Audit Office (NAO) established a Data Service to meet the demands of auditors that routinely need access to large volumes of data. This team maintains a number of large datasets, stores them in NAO’s data warehouse and merges them for auditors to use and interpret. The Data Service also provides guidance for audit teams that are using the data, which can be accessed through a common Share Point site. The Methods, Economics and Statistics Hub (MESH) complements the Data Service. This community of practice leads the NAO’s work on analytics and big data, and it co‑ordinates across a range of specialist areas to provide training and financial support for audits and wider assurance work. In addition to data analysis and analytics, MESH’s areas of expertise include economics, statistics, modelling, mapping, and qualitative analysis.
Italy
The Italian Court of Audit (Corte dei Conti, CdC) developed a “Data Analysis Competency Centre,” which became a cross-functional team and brings together business and technical competencies to support the effective implementation of SIDIF-ConosCo. The Centre supports users of SIDIF-ConosCo to make better decisions using machine learning, analytics, predictive analysis and other data analytics techniques. This Centre is in the early stages of its development and intends to be a multi-disciplinary team with knowledge and skills that span levels of government (i.e. national and regional) as well as technologies. According to CdC officials, this effort signals a recognition that any data-driven tool is not static, and requires a capacity-building strategy to support its development and evolution.
Turkey
In 2017, the Turkish Court of Accounts (TCA) created a “Data Analysis Group” to design methodologies for using computer-assisted audit techniques (CAAT) and enhance the capability of the TCA to assess risks in municipalities. The group had other aims, including decreasing auditors’ workload, analysing big data, identifying mistakes and errors in data processing, and automation of analyses to facilitate continuous monitoring. Their efforts resulted in “VERA”, TCA’s Data Analysis and Business Intelligence System, which automates risk analysis for over 1 400 municipalities to inform audit programming and planning.
Source: (U.K. National Audit Office, 2019[6]), (House of Commons, United Kingdom, 2017[7]), (Court of Auditors, 2019[8]), OECD Interviews with public officials from the CdC.
Strengthening the external and internal relationship of the DIARI through the use of tools produced by the Analysis Unit
The reliability and accuracy of risk assessments is a fundamental challenge for both the audit body and the auditees. In theory, data analytics should be able to enable financial managers to identify risky transactions as well as adapt control activities throughout the cycle of the project, including predicting high-risk transactions before committing funds (OECD, 2019[3]).
To this end, and as observed in the interviews carried out for this report, the Analysis Unit could still have great potential to leverage co-operation processes both internally and externally, but this merits reinforcement. Firstly, during the interviews, it was observed that the main focus of the Analysis Unit was identifying shortcomings related to the monitoring of the budget chain (cost overruns, execution time and other procedural aspects). However, the identification of risks in public administration, the feedback of early warnings information to entities, as well as the predictive analytics exercises are still in a primary stage. Secondly, communication with the General, Sectorial and Territorial Delegate Comptrollers deserves special attention, to the extent that there is reprocessing in requests for information to public entities, alongside scant feedback between the parties. Finally, even though the DIARI has carried out information cross-checks and obtained the meshes requested by the Departmental Management Offices, the interviews conducted for this report found evidence of lack of communication with the territorial level, where products received were said to be basic or difficult to understand (such as highly complex contracting meshes without any prior analysis). It is in this sense that the Analysis Unit should work on strengthening and better communicate the delivery of products in accordance with territorial needs.
As such, strengthening the data analytics strategy should be considered to yield the results required by the audit teams. To do this, an internal and external client identification exercise could be used. This inventory could contain internal and external allies that promote a balanced increase of capacities for the use of data and analytics. As such, the CGR could adopt a unified, cohesive and strategic approach on data analytics to maximise resources, improve decision making and achieve the objectives of the organisation, as seen in Brazil or Australia (Box 2.3).
Box 2.3. Development of policy and systems for the use of analytics in SAIs
Brazil: Information-Based Governance and Solutions
Brazil's supreme audit institution (Tribunal de Contas da União, TCU) developed its own analytical capacity with a strategic decision to decentralise its analytical capacities to the audit teams. Departments trained auditors in data analytics, encouraged them to seek data and technology support tools that would be useful for their audits and formed communities of practice.
Analytics can facilitate decision making related to the selection of audits, but the increasing amount of information and the seemingly endless possibilities for analysis can often hamper efforts toward efficiency and effectiveness. Therefore, institutions must formalise and implement work processes to ensure that the analyses actually improve the body's work, that the information used is necessary and sufficient to fulfil its purpose.
Brazil's supreme audit institution did this by focusing on three pillars:
Governance: the guidance, support and direction from top management, including a cohesive organisational strategy and action plan. This can help create a culture within the organisation that encourages the use of analytics for control activities.
Platform: the tangible tools necessary to execute analytics-related tasks, to obtain useful information from raw data. For example, TCU established and manages a virtual environment called “Labcontas” that allows auditors to access easily and independently information from dozens of databases as a benefit of agreements signed between public institutions. This facilitates co-ordination of efforts and reduces the time and resources often required to obtain information during audits.
Information-based solutions: The tools and high-level structures in place must result in actionable, timely and reliable insights. This type of information is used in Brazil for a variety of audit-related tasks, from holding managers accountable to flagging potential high-risk grants or procurement proposals submitted to the government.
Australia: a unified approach to data analytics
Australia has adopted a unified, cohesive and strategic approach to data analytics for audit institutions hoping to maximise resources, improve decision making and achieve organisational goals. With this in mind, the Australian National Audit Office (ANAO) annual corporate plan sets out a clear and concise framework on how technology and data analytics will be used in the organisation, the specific and relevant processes in place and which teams across the organisation are responsible for each aspect.
More specifically, the document identifies a stand-alone team dedicated to analytics, key areas in which the Office would like to use data to better meet its objectives, and ways in which it would like to improve data governance practices. The importance of data analytics to the organisation’s work is also highlighted by its mention in other related sections of the corporate plan, including those on human capital, budgeting and productivity.
Similarly, in Turkey for example, the VERA system provides auditees with a standard and automated tool for the risk-based classification of more than 1 400 municipalities (Box 2.4).
Box 2.4. Data analytics and organisational intelligence system in Turkey
The Turkish Court of Accounts (TCA) created “VERA”, a Data Analysis and Business Intelligence System, which automates risk analysis for over 1 400 municipalities to inform audit programming and planning. VERA provides auditees a standard, automated tool for risk-based ranking of over 1 400 municipalities. VERA allows management to take into account risks before the TCA’s annual audit programming and supports the creation of the audit strategy.
In addition, auditors use the results of the risk analyses to plan audits, as well as identify possible material misstatements in financial reports that could represent errors and fraud. All auditors have access to VERA, and are able to assess the results of VERA’s automated analyses related to risks and financial indicators in a dashboard or automatically generated reports.
Source: (OECD, forthcoming[11])
Making use of new techniques for data analytics that contribute to strengthening preventive and concomitant control
Developing an action plan to monitor new and existing analytical initiatives is critical to the success of SAI. The CGR is primarily a consumer of data and the models and systems it develops to run analytics are subject to change over time. This may include changes in the quality or relevance of the data depending on the methodology. Changes in context, data reliability or data access can affect the accuracy and usefulness of any model. For this reason, continuous monitoring of the performance of data and analytics initiatives should be a critical component.
The use of new data analytics techniques, in which the DIARI is specialised, is essential for the success of preventive and concomitant control. For example, in relation to “big data” techniques, these can be a valuable addition to the analysis of audit information, particularly when rigorous analytical procedures are combined with appropriate audit techniques and experts (Gepp et al., 2018[12]). The value of integrating "big data" into audit analysis enables the development of new skills. In particular, in instructing analysts on which data to obtain from the information and its use by mission areas (General, Sectorial and Territorial Delegate Comptrollers Offices), as well as the ability to permeate the entire audit process, allowing auditors to get used to its functionalities (Gepp et al., 2018[12]; EY, 2015[13]). Figure 2.2 shows the experience of the UK National Audit Office (NAO) in relation to its data analytics strategy. In particular, it focuses on data mining processes and on reducing costs to administrations by detecting patterns and anomalies.
For this reason, the CGR could consider adopting tools aimed at preventive work in the future, through predictive analytical models to identify risks. This work, in turn, can be leveraged on the use of prospective models based on data analytics and incorporate the available information within the different areas of the CGR, as well as that produced, for example, by performance audits and sector studies. For this, it could also use international experiences as references, such as those of the United Kingdom shown in Figure 2.2 and could establish and maintain a quantitative risk assessment framework that is efficient and can be used effectively in risk management.
Figure 2.2. Data analytics at the UK National Audit Office (NAO)
In summary, the potential of the Analysis Unit could be explored through the following considerations:
Develop a unified vision for data and analytics, including one that reflects audit and governance trends related to big data. This policy vision can be included and aligned with the next National Development Plan (PND, Plan Nacional de Desarrollo), providing broader objectives for digital transformation, including delineating a theory of change to contribute to good governance and accountability in the public sector, as well as positioning the CGR as a national and international example of high quality technical expertise (Chapter 4).
Establish indicators and a theory of change with respect to data and analysis, focusing on results and not simply outputs (for example, indicators that reflect the prevention, mitigation or recovery of losses as a result of data and analysis).
Develop an action plan to ensure the relevance of its analytical initiatives and the reliability of results, particularly for the new methodologies and systems that are under development. This action plan may focus on corporate objectives to leverage data and analysis to improve audit work. Using the data governance framework outlined above to help frame key areas to focus on and identify priorities, where it is important to consider the implications and relevance of data analytics to the entire audit cycle, as well as considering the internal context and the fact that activities are tailored to the purpose of individual teams within the CGR.
Adopt tools aimed at preventive work, including the identification of risks in public administration, as well as the adequate communication of them to public managers. This work can in turn leverage the use of prospective models and incorporate information available within the different areas of the CGR (such as that produced, for example, by performance audits and sector studies).
Promote learning, inclusion and experimentation when using data and new techniques, including the development of a baseline to measure the impact of the CGR in relation to the new data analytics strategies and in relation to its results and advantages.
Raise awareness among CGR officials about data analytics and how it contributes to risk analysis in order to develop technical capacities and promote the specialisation of personnel.
Generate internal co-ordination through meetings between the DIARI and the General, Sectorial and Territorial Delegate Comptrollers' Offices in order to identify and make use of the internal information of the CGR, including audit reports from previous years (Box 2.5), in such a way that these audits are integrated as data input into the system.
Articulate with the territorial level so that they have access so the same information shared with other dependencies of the CGR, in particular through the establishment of links and the strengthening of capacities in the Departmental Management Offices.
Box 2.5. Learning from history: Using previous reports and sectorial studies as sources of information
Some Supreme Audit Institutions (SAI) use the evidence in their reports as input to develop good practice guides.
The UK National Audit Office (NAO) published its good practice guide for commercial contract management in 2021 based on findings and recommendations from 209 reports published over the last 20 years. The guide incorporates recent audits of commercial contracts related to COVID-19.
The Comptroller General of the Republic of Peru identified the main causes of delays in ten public works projects, executed with the Public-Private Partnerships (PPP) model. The study concludes with recommendations to prevent delays in future projects, which could serve as criteria applicable to future PPPs. The methodology was based on a PPP literature review, on interviews with PPP officials and specialists, on the selection of a sample of PPP contracts, using criteria of accessibility, heterogeneity, amount and an analysis of contracts additions to identify causes of renegotiation. The findings show that close to 50% of the additions were signed before three years, compared to the average term of the concessions of 27 years. In addition, about 20% of the additions were the product of non-compliance by the State (the grantor) with the deadlines for land reclamation and lack of access to water. Also, at the time of the contract design, the basic engineering, environmental and geological studies were often not available, which affected the allocation of risks. Thus, the study recommends avoiding delays in the delivery of land by the State, ensuring that before the beginning of the process, the land is properly cleaned up and that there are updated and reliable studies to avoid cost increases.
Optimising the functions of DIARI’s Immediate Reaction Unit (URI) in such a way that it contributes more than special control actions to strengthen inter-institutional relations for preventive and concomitant control
The Immediate Reaction Unit (URI) is in charge of closing the cycle of review, analysis and data use. In particular, it can issue alerts based on the reports of the Analysis Unit, as well as advise and guide immediate reaction activities. As such, the URI executes on-site verifications of unfinished public works through the use of monitoring technologies (drones, satellites, among others).
For the purposes of preventive and concomitant control, the URI is in charge of the "special control action", which is a brief fiscal control action, constituting a rapid response to a fact or matter that may come to the attention of the CGR through the SACI, once the system will be in operation. This action can also be started through citizen’s complaints or by identifying concerning issues through social media.
Therefore, the URI has received 302 alerts on contracts with alleged cost overruns, within the framework of the health emergency caused by COVID‑19 and issued by the Information Analysis Unit (Contraloría General de la República de Colombia, 2021[2]). These alerts were distributed to internal dependencies of the CGR and to other control entities for their knowledge and processing. As such, the URI has strong links with the public administration and, in particular, provides a fundamental feedback service to the public administration through these alerts. It does this through risk reports, as well as workshops with the public administration and citizens.
However, the URI has not yet reached the level of maturity needed to generate more preventive alerts. According to the CGR, this is because there are still no criteria to determine whether the cases made available to them and brought to its knowledge are considered sensitive or if they have some other classification. That is, the alerts that are raised with the URI are made under criteria of importance or relevance for the region, project value and the similar alerted situations, but not based on a risk analysis.
Supporting transformation processes which allow the internal appropriation of preventive and concomitant control
Preventive and concomitant control, as a new and developing tool, still has opportunities for improvement and appropriation among CGR officials. An example of this was given during the interviews conducted for this report where several actors referred to the “alerts” as part of the ex post control or where several officials expressed their concern about the lack of training on this new tool. The need for clarification and a truly differentiated approach became evident in the call by various officials to set up special groups for the preventive control or to include the preventive perspective into the Sectorial Delegate Comptrollers. In particular, several of the interviewees mentioned that a large part of the CGR officials involved in preventive and concomitant control have a vision focused on the culture of “findings” (hallazgos), where “alerts” or “warnings” are perceived as a process focused on detecting cases and requiring investigative work to eventually implement a fiscal responsibility process.
In addition, there was evidence of a tendency to analyse complaints and information on issues related exclusively to acts of corruption, including the so-called “white elephants” or processes where cost overruns are evident in public procurement. Even though these actions are essential for the CGR’s work in the fight against corruption, as already emphasised, preventive control has the potential to generate more structural positive changes within the public administration. In this sense, its usefulness should be strengthened in matters such as risk management and improvement of processes and procedures associated with fiscal management.
Resolution 762 of 2020, Art. 38 had already highlighted the need to establish working groups for concomitant and preventive control. In particular, it assigned to the General and Sectorial Delegate Comptrollers a Concomitant and Preventive Control Working Group attached to the office of the respective Delegate and made up of a minimum of three public servants with an exclusive mandate for the development of the functions of the group. As such, the general functions of the Concomitant and Preventive Control Working Group would be supported by the Directorates of Fiscal Surveillance or Sectorial Studies or the Directorate of Regional Surveillance, Promotion and Development of Citizen Control. Currently, there are some officials who are in charge of monitoring the "alerts" as well as fiscal surveillance and permanent monitoring to identify situations that should be communicated to other CGR dependencies, other control entities or to the public. Additionally, the constitution of these working groups has not prevented the existing organisational structure from advancing or being able to carry out preventive and concomitant control exercises.
To this end, and taking into consideration the provisions of Legislative Act 4 of 2019, Law Decree 403 of 2020, Resolution 762 of 2020 and the internal procedure adopted, the CGR should strive to develop actions aimed at:
Overcoming silos to strengthen preventive oversight and improve communication and internal exchange.
Taking advantage of the new preventive and concomitant control to strengthen sectorial studies and performance audits.
Promoting behavioural and cultural change, leveraged on the current institutional framework of the CGR, which helps to understand and internalise the preventive function among the CGR, the public administration and the citizens.
Strengthening internal exchange and communication to break “silos” and ensure the effectiveness and efficiency of preventive and concomitant control
During the interviews, there was clarity about the need to increase internal communication between the different areas of the CGR related to preventive and concomitant control to ensure that the new function is properly understood and appropriated and misunderstandings with other mechanisms avoided. These challenges in internal co-ordination are undoubtedly based on a cultural change and the appropriation of the mechanism, but also from an appropriate understanding of the conceptual and practical differences between "alerts", which in most cases are generated correctively and "warnings" that seek, as stated by the Law and the Constitutional Court, to have more of a preventive approach (Chapter 1). In addition, the analysis of the information provided by the CGR and the interviews carried out, evidenced the fragmentation of criteria and the existence of silos in relation to preventive and concomitant control within the CGR.
The silo mentality is a recurring problem in the implementation of new processes across all organisations and is characterised by individuals or divisions that hide information from others in the organisation for one or more reasons. These may include power struggles, fear, organisational inefficiency, or simply may be due to the fact that no effort is made to update shared information. Also, historical labour relations ("it has always been like this here") and organisational micro cultures ("this unit doesn't work like us") can contribute to exacerbate the problem. The silo mind-set destroys trust, cuts off communications and encourages complacency. An isolated organisation cannot act quickly or seize opportunities. When information is not freely shared, informed and evidence-based decisions cannot be made (Froy and Giguère, 2010[15]). Breaking these resistances saves time and allows the entity's objectives to be achieved.
These silos refer to the refinement of internal processes and procedures. In particular, the interaction between the DIARI and the Sectorial Delegate Comptrollers (and with the Delegate of Citizen Participation) should be strengthened and designed with more clarity. In addition, the interviews conducted indicated the need for greater integration with the Departmental Management Offices, who have the need to access not only the information produced by the DIARI, but also to improve their interaction with the Sectorial Delegate Comptrollers.
A positive aspect observed during the interviews refers to the informal arrangements between the General and Sectorial Delegate Comptrollers that currently guide the process. Several of the interviewees established that this informal co-ordination has allowed them to refine criteria and learn from previous decisions made by other Delegates in similar processes.
However, this informal co-ordination could benefit from some additional formal arrangements. In particular, such an institutionalisation would be key for the generation of institutional memory within the entity and for the unification of performance criteria.
In this regard, the CGR could consider incorporating the following recommendations:
Creating a unified vision of preventive and concomitant control that overcomes divisions within the CGR, both at the national and territorial levels, establishing goals that benefit multiple agencies and territorial levels. Among others, ensure that each General, Sectorial and Territorial Delegate is not only focused on achieving their specific objectives with isolated information, but on the contrary, this information must be shared between teams. For this, it is essential to strengthen the relationship between decision makers and area leaders, as well as to encourage work groups with officials from various agencies.
Educate, work and train. One way to break down silos is to educate, work and train together on exercises that include different dependencies. Collaborative training across divisions is a way to combine required training with collaborative silo-breaking practices.
Frequent internal communication. Another way to break silos is to generate constant communication spaces, formally and informally, where information is made available, as well as the generation of feedback exercises between different units of the CGR. For example, this could be achieved by encouraging communication exercises with Departmental Management Offices and providing case studies that differentiate the preventive function from ex post control.
To ensure good co-ordination between the different dependencies of the CGR, the following could be analysed:
Include in the procedure for the implementation of concomitant and preventive control (Process VIG 01 PR 001) casuistic guides from previous activity and relevant international practices.
Improve understanding of the new control by conducting seminars or workshops on specific cases that serve to illustrate concomitant and preventive control. In these seminars or workshops, the Sectorial Delegate Comptrollers will be able to share information and experiences.
Analyse the potential for an internal, non-binding committee, where some of the most relevant cases, risks identified and best ways to understand the different situations are discussed. In particular, this committee could count on input not only from the areas directly involved, but also from other relevant actors where applicable.
Continue with the feedback exercises of the General, Sectorial and Territorial Delegate Comptrollers to the DIARI in relation to the usefulness, relevance and usage of the information the DIARI produces. For example, it could be very useful for the DIARI to receive feedback on whether this information is useful for scheduling audits, whether it should be more focused on x or y aspect or on whether the reports should more clearly indicate risks). As such, the DIARI could have closer communication with different areas of the CGR, including the Departmental Management Offices, by sharing the analysis reports they produce, as well as providing training on how to use them and on how to interpret the information produced.
Analyse the possibility of creating a specialised unit to allow the intersectorial analysis of the information collected by the Sectorial Delegate Comptrollers for the purposes of preventive and concomitant control. This will make it possible to homogenise the type of information received by the Comptroller General for the purposes of the warning mechanism and to differentiate the interaction of the Sectorial Delegate Comptrollers with the public administration for purposes other than fiscal control. Ultimately, this will make it possible to differentiate the sanction-related from the preventive work in the relationships with public entities.
Consider establishing concomitant and preventive control working groups, as provided in Resolution 762 of 2020, Art. 38 and in accordance with the needs of the CGR and the Sectorial Comptrollers' Offices.
Taking advantage of the new preventive and concomitant control to strengthen sectorial studies and performance audits
Performance audits seek to provide information, analysis and recommendations for improvements in public policy planning (INTOSAI, 2019[16]). In particular, they aim to improve the management of the public sector and report on how many of its recommendations are accepted and applied by the public administration. Thus, performance audits follow a different logic than the more traditional role of fiscal control bodies, since they evaluate the policy design from its conception, identifying design flaws and whether it is at odds with macro policy objectives. In this way, performance audits are clearly framed within the concept of preventive control as they seek the same objectives. Furthermore, the information produced by these performance audits may eventually also inform preventive and concomitant control work.
The CGR has a mandate to carry out performance audits through Article 119 and 267 of the Political Constitution, which establishes that "the Office of the Comptroller General of the Republic is in charge of the surveillance and control of fiscal management of the administration”. As such, Art. 124 of Law 1474 of 2011 states that "the regulation of the methodology of the auditing process by the CGR and the other comptrollers will take into account the instrumental condition of regular audits with respect to performance audits, with a view to guaranteeing a comprehensive exercise of the auditing function". In addition, Decree 1499 of 2017 establishes the guidelines of the Quality Management System as a systematic and transparent management tool to allow the management and evaluation of institutional performance in terms of quality and social satisfaction in the provision of services. In interviews conducted for this report, it became clear that there is still scope for improvement. This is, because the system cannot really determine if institutional policies being analysed in the framework of performance audits, operate in accordance with the principles of economy, efficiency and effectiveness or if there are areas of improvement of these public policies within public institutions.
This situation is exacerbated due to several circumstances within the CGR. Performance audits are located in the micro areas of the Sectorial Delegate Comptrollers that is, in the area of fiscal surveillance. Several experts have already mentioned that a weakness of the current system is that the fiscal surveillance departments follow the logic of “sanction” and “compliance” rather than being result oriented (Inter-American Development Bank, 2019[17]). Multiple experts interviewed for this report have also identified several shortcomings in the current institutional arrangement. Among others, the areas in charge do not have the horizontal vision for the implementation of public policies and this, in turn, is reflected in the profile of the officials in charge. This distorts the preventive work and feedback to the entity. This vision was confirmed by the evaluation of the Performance Measurement Framework (Marco Medición del Desempeño, MMD) that was carried out in the CGR between 2017 and 2018, as well as during the update of guidelines to implement part of the recommendations for closing performance gaps (Inter-American Development Bank, 2019[17]).
In particular, these documents recommended giving the Sectorial Studies Directorates (Direccciones de Estudios Sectoriales, DES) authority over performance audits. This, under the understanding that the DES have the experience and specific capacity to carry out evaluations of public policies. The Constitutional reform brought by Legislative Act 04 of 2019 included the modification of some of the mechanisms to deepen the analysis of public policies and sectorial behaviour of the DES, bringing a unique opportunity to transfer this function to these areas, as they have the necessary skills to carry out performance audits. In addition, this mechanism could be strengthened through the information produced by the DIARI.
Furthermore, performance audits can support citizen participation strategies that in turn generate institutional benefits and strengthen public trust. Social organisations can be valuable allies of the CGR, using their experience and networks throughout the performance audit process and becoming valuable stakeholders. This is especially the case for policies aiming, for example, to achieving the sustainable development objectives (SDG), since the audits will ultimately reflect a citizen vision for control and therefore contribute to the sustainability of the public policies being audited.
The CGR has taken important steps to improve the institutional framework related to performance audits. In compliance with the CGR improvement plan, the CGR advanced the final verification of the update of the Performance Audit Guide, focused on auditing government programmes, projects, systems, operations, activities or organisations, to deem whether they operate in accordance with the principles of economy, efficiency and effectiveness. In addition to these changes, in the new version of the Performance Audits Guide, the criteria and standards of ISSAI 300 (3000, 3910 and 3920) and Decree Law 403 of 2020 have been incorporated. Once approved, the guide may impact the growth of performance audits considerably in the 2022 period.
To this end, the CGR could consider the following actions to strengthen performance audits, to complement and leverage preventive and concomitant control:
Give the mandate for performance audits to the DES, as part of the evaluation process of public policies, which today are in charge of the DES, ensuring co-ordination with the fiscal surveillance areas as appropriate, given that DES are more familiar with the development of analysis and research in sectorial studies.
Advance the updating of the new Performance Audit Guide, where the criteria and standards of ISSAIS 300 (3000, 3910 and 3920) and Decree Law 403 of 2020 are incorporated.
Adjust the profiles required by the DES and in particular, the personal in charge of performance audits as well as provide the available positions on the areas of performance audits.
Consolidate a team of mentors to carry out training programmes in performance audits, ensuring that those who receive the training have the appropriate profile.
Develop a strategy for citizen participation that in turn contributes to the SDG through mechanisms of dialogue, participation and accountability with citizens, which reflect a citizen vision for control and provide sustainability to audited public policies.
As such, as recommended in Chapter 1, one substantial advance would be for the CGR to be able to issue recommendations to the public administration to contribute to improve performance-related results. Such recommendations should not to be equated with co-administration. Currently, as mentioned before, the CGR does not issue recommendations of any kind due to the perception that this would be co-administration. However, the CGR could consider regulating this mechanism. As mentioned, many SAIs around the world use the evidence found in performance audits to formulate recommendations. Box 2.6 shows the experience of the United States Government Accountability Office (GAO) in this area.
Box 2.6. Effective use of recommendations in performance audits
The United States Government Accountability Office (GAO) was able to significantly reduce fraud in the COVID-19 relief programme by providing recommendations to the entities managing the emergency. The report, prepared in less than 90 days, presents relevant, sufficient and reliable evidence that supports the recommendations, including the recovery of incorrect payments to deceased persons and new legislation to prevent fraud.
The report to Congress describes (1) the government's activities to support individuals; (2) estimates of the aid distributed; (3) the responsible ministries; and (4) the effectiveness of internal control in the ministries to prevent and detect errors and fraud in payments. The methodology included: interviews with ministries; analysis of previous audits by the GAO and internal control offices in the responsible ministries; analysis of expenses and people who received the aid and a review of the corresponding documents.
Among the recommendations issued are the following:
1. That the Internal Revenue Service (IRS), the US tax agency, send notifications to the beneficiaries, requesting that they return payments designated for deceased persons.
2. That Congress amend the law so that the IRS can obtain early and expeditious access to the Social Security Administration (SSA) file of persons deceased.
To this end, there was a follow-up to recommendations, taking into account that the IRS reported that it recovered 57% of the payments sent to deceased persons and that, in effect, Congress approved the legislation that requires the Ministry of the Treasury to access the file of persons deceased for a period of three years.
Promote cultural change within the CGR and the public administration for a collaborative, preventive and transparent vision of fiscal management
Part of the success of the new preventive and concomitant control will depend on the appropriation of the mechanism, which requires an understanding of the substantial differences between a supervisory task and a preventive task, as well as the generation of internal capacities that point to preventive work, not only to a more timely detection of problems or to sanctioning. As within all organisations, it is the officials who are the final decision makers, who implement the regulatory framework and therefore are the ones who determine its success. Cultural change then becomes one of the most important engines of change for the success of the new preventive and concomitant mechanism.
The CGR has made a significant effort to respond to this challenge. During 2020 and 2021, socialisation activities have been carried out through seminars, guaranteeing that 90% of the entity will be trained in various relevant subjects. However, the CGR's senior management recognises that it is important to continue the efforts to raise the awareness of officials on all the operational details related to preventive and concomitant control. To this end, a dedicated strategy has been designed within the CGR aimed at strengthening the preventive and concomitant control so that CGR officials know, master, understand and apply the tools provided by the Constitution, the law and the regulations on the new model of fiscal control in Colombia.
However, despite these efforts, in the interviews conducted for this report, several of the interviewees spoke not only of the need to generate capacities, through training and capacity building for the preventive perspective, but also of the need to promote larger-scale cultural change in the CGR. The informal standard that a good audit is one which produces many “findings and observations” still seems to dominate within the CGR and influences the behaviour of auditors.
As such, it is important for the CGR to continue its training efforts with an effective awareness-raising and capacity-building programme around risk management in general, with a specific module on fraud and corruption risk management, taking advantage of the variety of already existing programmes. For example, training should be delivered as part of induction or orientation programmes. In addition, the CGR could consider:
Continuing training and education for the Sectorial, General and Territorial Delegate Comptrollers on the scope of preventive and concomitant control, as well as on the use of the information produced by the DIARI. The training could even be extended to include risk analysis and management of public administration, examining the capacity of the latter to achieve expected results.
Making use of the Centre for Fiscal Studies (Centro de Estudios Fiscales, CEF), which is in charge of high-quality training in matters of surveillance and control in the management of public resources, to carry out internal training processes on the tasks of preventive control, including the identification and management of fraud and corruption risks.
However, training is not enough to generate changes in the behaviour and culture of organisations (OECD, 2018[18]). In fact, auditors are part of social groups, which may be their audit unit or the SAI in which they work, or even the auditing profession as a whole. What auditors believe most auditors in their reference group are doing (empirical expectation) or what they believe most auditors in their group expect them to do (normative expectation) may explain their patterns of behaviour. These beliefs are called "social norms" and can be extremely powerful in shaping behaviours (Bicchieri, 2017[19]; Bicchieri, 2005[20]).
This is why changing social norms so that they impact organisational cultures is complex, takes time and is a gradual process, but it is also key to generating sustainable changes (Yamin et al., 2019[21]). The strategies that can be used to generate changes in the organisational culture associated with preventive and concomitant control are various and the CGR could consider applying the BASIC methodology (Behaviour, Analysis, Strategy, Intervention, Change), developed by the OECD to apply a behavioural perspective to public policy. Thus, the CGR could identify and analyse behaviours of its public officials that prevent the appropriation of new preventive control and develop strategies to induce change (OECD, 2019[22]).
The application of behavioural insights to organisations demonstrates the importance of influencing specific individuals to achieve changes throughout the organisation and to intervene directly in the organisational routines, policies and procedures (OECD, 2020[23]). Box 2.7 provides an overview of some key theoretical foundations and insights from organisational psychology when it comes to influencing organisational behaviour.
Box 2.7. Applying Behavioural Insights to Organisations: Theoretical Foundations
When enough people are nudged toward behavioural change, those new behaviours have the potential to become habit, switching from deliberate choices and actions otherwise known as controlled processing, to less deliberate, less effortful, more habitual actions known as automatic processing. Whether deliberate or effortful, choice or habit, when enough people in a work group or entire organisation behave in a certain way, that behaviour has the potential to become a norm. Norms are rules for expected and accepted behaviour. As humans, violating norms tends to make us uncomfortable. We are likely to conform to the norms of our work group and organisation. This is especially true of cohesive groups who feel a degree of attraction to their work group.
Nudging supervisors or other powerful or influential people within an organisation can have a multiplying effect such that the behaviours exhibited and endorsed by influential individuals have a better chance of being adopted en masse, nudging a whole organisation in the process. Indeed, charismatic and transformational leaders are believed to possess qualities that inspire followers to behave in desired ways in service of a larger goal. Nudging such leaders can effect largescale behavioural change.
Of course, those in formal leadership roles toward the top of the organisational hierarchy are also in a good position to effect widespread behavioural change by altering organisational policies and procedures. Nudges that help high-level decision makers (leaders, boards, etc.) optimise organisational policy decisions in the face of their own biases and irrationalities can have an effect. Thus, helping decision makers see the connection between policies, procedures and behaviour on the ground is another way to nudge whole organisations.
Source: (OECD, 2020[23])
For example, an internal cultural change strategy within the CGR might include:
The implementation of measures that aim at promoting day-to-day learning through superiors and peers, identifying and raising awareness among internal leaders who can become vectors of change.
Internal communication and awareness raising about preventive control mechanisms.
Establishing the internal performance objectives of preventive control. As mentioned above, these objectives ideally may not be associated with the number of "findings" or "alerts" issued, but rather with results in generating change in risk management within the public administration.
Strengthening the preventive control groups within the Sectorial Delegate Comptrollers. In particular, establishing management of preventive control at the second level of the Delegate. This will mean that even when the auditor can act in both directions, the preventive function can be clearly differentiated both internally and externally.
The interviews conducted for this project also highlighted that preventive control does not only require a cultural change in the CGR. In addition, it will be very important to work, from the CGR and from the public administration, to achieve change among financial managers so that they understand and appropriate the support that the CGR can provide them without perceiving this to be co-administration. Continuing and nurturing the Bank of Good Fiscal Management Practices, not only to evaluate the management and results of the control subjects, but also to provide guidelines for proper fiscal management, will be essential for this purpose.
Preventive and concomitant control, through its potential to help identify risks in public entities, can also leverage cultural changes within public entities and help improve public governance. This is because preventive controls have the potential to promote transparency and strengthen accountability. Of course preventive controls can create tensions and critical reactions, especially when the consequences involve more work for the auditees. However, these progressive tensions may eventually be accepted if their recipients understand and see their work rewarded throughout the chain of public management, including the gradual improvement of public governance (Kimi Makwetu, 2020[24]).
To this end, it is recommended that the CGR and the Administrative Department of the Public Service (Departamento Administrativo de la Función Pública, DAFP) join forces to develop and implement measures that seek to explain and promote the acceptance of preventive control as an ally for the strengthening of public management. In addition, the CGR could analyse how, on its side, it can contribute to improving the relationship between auditors and auditees. For example, audit reports written in simpler and less technical-legal language could facilitate both ownership by the public administration and monitoring by citizens. Simpler language will make it easier to understand, increase motivation to act and eventually improve the impact of preventive control mechanisms.
References
[10] Australian National Audit Office (2020), Corporate Plan 2020-21, https://www.anao.gov.au/work/corporate/anao-corporate-plan-2020-21.
[19] Bicchieri, C. (2017), Norms in the wild: How to diagnose, measure, and change social norms, http://dx.doi.org/10.1093/acprof:oso/9780190622046.001.0001.
[20] Bicchieri, C. (2005), The grammar of society: The nature and dynamics of social norms, http://dx.doi.org/10.1017/CBO9780511616037.
[14] CGR Perú (2015), Estudio sobre las Causas y efectos de las renegociaciones contractuales de las Asociaciones Público - Privadas en el Perú, Contraloría General de la República del Perú, Lima, https://library.pppknowledgelab.org/documents/2406/download (accessed on 24 September 2021).
[2] Contraloría General de la República de Colombia (2021), Una Contraloría para Todos: informe de Gestión 2020-2021.
[8] Court of Auditors (2019), Public Audit in the European Union 2019 Edition, https://op.europa.eu/webpub/eca/book-state-audit/en/ (accessed on 23 November 2021).
[1] DNP (2021), CONPES 4045, https://colaboracion.dnp.gov.co/CDT/Conpes/Econ%C3%B3micos/4045.pdf (accessed on 8 September 2021).
[13] EY (2015), How big data and analytics are transforming the audit, https://www.ey.com/en_lb/assurance/how-big-data-and-analytics-are-transforming-the-audit (accessed on 25 August 2021).
[15] Froy, F. and S. Giguère (2010), Breaking Out of Policy Silos, OECD, http://dx.doi.org/10.1787/9789264094987-en.
[12] Gepp, A. et al. (2018), “Big data techniques in auditing research and practice: Current trends and future opportunities”, Journal of Accounting Literature, Vol. 40, pp. 102-115, http://dx.doi.org/10.1016/J.ACCLIT.2017.05.003.
[7] House of Commons, United Kingdom (2017), Value for Money Study, Delivery of Benefits From the NAO’s IT Enabled Change Program, https://www.parliament.uk/globalassets/documents/public-accounts-commission/15-NAO-VFM-report-on-IT-enabled-change-programme.pdf.
[17] Inter-American Development Bank (2019), Informe de Evaluación del Desempeño, https://www.contraloria.gov.co/documents/20181/449782/Informe+Final+MMD+EFS+CGR+Colombia+2019_unlocked.pdf/611096c8-6eef-4de0-924e-2839eab84552 (accessed on 24 August 2021).
[5] INTOSAI (2019), “Data Analytics Guideline”.
[16] INTOSAI (2019), ISSAI 300.
[24] Kimi Makwetu (2020), “Re-establishing accountability”.
[4] Langhe, B. and S. Puntoni (2020), Leading With Decision-Driven Data Analytics, MIT Sloan Management Review, https://sloanreview.mit.edu/article/leading-with-decision-driven-data-analytics/ (accessed on 9 September 2021).
[23] OECD (2020), Behavioural Insights and Organisations: Fostering Safety Culture, OECD Publishing, Paris, https://dx.doi.org/10.1787/e6ef217d-en.
[3] OECD (2019), Analytics for Integrity: Data-driven Approaches for Enhancing Corruption and Fraud Assessments, OECD Publishing, Paris, http://www.oecd.org/gov/ethics/analytics-for-integrity.pdf.
[22] OECD (2019), Tools and Ethics for Applied Behavioural Insights: The BASIC Toolkit, OECD Publishing, Paris, https://dx.doi.org/10.1787/9ea76a8f-en.
[18] OECD (2018), Behavioural Insights for Public Integrity: Harnessing the Human Factor to Counter Corruption, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264297067-en.
[11] OECD (forthcoming), Analytics at Mexico’s Supreme Audit Institution: Considerations and priorities for assessing integrity risks, OECD Publishing, Paris.
[9] Revista do TCU (2016), “Evolution of Control in the”, Federal Court of Accounts Journal, http://See PDF (accessed on 3 August 2021).
[6] U.K. National Audit Office (2019), Transparency Report, https://www.nao.org.uk/wp-content/uploads/2020/07/National-Audit-Office-Transparency-Report-2019-20.pdf.
[21] Yamin, P. et al. (2019), “sustainability Using Social Norms to Change Behavior and Increase Sustainability in the Real World: a Systematic Review of the Literature”, Sustainability, Vol. 11/20, p. 5847, http://dx.doi.org/10.3390/su11205847.