Data and analytics have the potential to transform the work of SAIs, but significant investment in capacity, skills and infrastructure is critical for this to happen. The ASF has already invested heavily in these areas in an effort to modernise its approach to using data and analytics. This chapter explores these issues from the perspective of the ASF’s operational challenges and priorities, and in particular, issues related to improving co-ordination and analytics capacity, enhancing analytics for detecting integrity risks and nurturing a data-centric culture.
Strengthening Analytics in Mexico’s Supreme Audit Institution
Considerations and Priorities for Assessing Integrity Risks
OECD Public Governance Reviews
2. Operational priorities for Mexico’s supreme audit institution to enhance analytics
Abstract
2.1. Introduction
Supreme audit institutions (SAIs) are usually data consumers that rely on other government entities for data to fulfil their mandate, which translates into the need for a broad range of expertise and knowledge about many different contexts in government and a variety of data sources. Moreover, SAIs operate in highly technological environments in which data and the means to extract value from it are constantly evolving. In this context, big data and small data alike can pose challenges for SAIs. Data and analytics have the potential to transform the work of SAIs. Examples from across the SAI community demonstrate the value they can bring to performance audits, compliance audits, financial audits and investigations. Moreover, studies suggest that investing in analytics reduces fraud. The Association of Certified Fraud Examiner’s Report to the Nations: 2020 Global Study on Occupational Fraud and Abuse found that organisations with “proactive data monitoring and analysis” have 33 percent less fraud loss than those without it (ACFE, 2020[1]).
Nonetheless, data on its own does not have intrinsic value. It becomes an asset only when applied effectively, and this requires people, critical thinking and a learning mindset, not to mention robust information and technology (IT) infrastructure and architecture. Moreover, as described in Chapter 1, SAIs consideration of various strategic components is critical, including of their data governance framework in order to effectively take advantage of data and analytics for assessing integrity risks. In the case of Mexico’s supreme audit institution, the Superior Audit of the Federation (Auditoria Superior de la Federación, ASF), this includes improvements to its data strategy, co-ordination and plans for continuous improvement, including assessing the impact of its efforts. These activities are also relevant outside of the integrity context. In addition to these elements, SAIs also need the capacity, expertise and infrastructure to ensure effective and efficient investment of taxpayer money in analytics. In interviews and workshops with ASF officials, addressing capacity was considered the top priority for the organisation going forward, both in terms of analysing data as well as following up on results. Officials also recognised the need for improvements to the ASF’s infrastructure and availability of analytical tools or new techniques to analyse data. This chapter explores ways for the ASF to strengthen some of these operational aspects of working with data and leveraging analytics, including actions to improve co-ordination and capacity, enhance analytics for detecting integrity risks and nurture a data-centric culture.
2.2. Key systems and databases that support the ASF’s analytics
As described in Chapter 1, the ASF’s analytics and related processes for data governance are decentralised across different departments and teams. The Special Audit of Financial Compliance (Auditoría Especial de Cumplimiento Financiero, AECF) and the General Directorate of Forensic Audit (Dirección General de Auditoría Forense, DGAF), including DGAF’s Forensic Laboratory (Laboratorio Forense), are key drivers of the ASF’s analytics capacity. In addition, the Special Audit of Federal Spending (Auditoría Especial de Gasto Federalizado, AEGF) has developed its own capacity for managing data and carrying out analytics. The AEGF is responsible for auditing federal expenditures that are transferred to states and municipalities. Like the AECF, it consists of General Directorates (Direcciones Generales, DG), which have their own mandates, strategies and audit universes. As noted in Chapter 1, ASF recently introduced a new DG of Forensic Audit and Federal Spending (Dirección General de Auditoría Forense del Gasto Federalizado, DGAFGF) under the AEGF. The AEGF developed the System for the Control, Administration and Audit of Federal Expenditure Resources (Sistema de Control, Administración y Fiscalización de los Recursos del Gasto Federalizado, SiCAF).
The SiCAF is an online platform for the administration, management, monitoring and control of public works and acquisitions in states and municipalities that are financed with federalised expenditures. It facilitates auditing tenders, different phases of the procurement process and payments. It will also have geo-referenced maps of the location of public works. With the SiCAF, the AEGF also aims to enhance audit planning, promote real-time auditing and increase its territorial coverage of resources spent in states and municipalities. The AEGF will establish a permanent team to monitor registered projects and the quality of information and data inputted into the SiCAF (ASF, 2021[2]).
The COVID-19 pandemic forced the ASF to accelerate the implementation of virtual audit processes, including the SiCAF. The pandemic revealed the ASF’s vulnerabilities and limitations in terms of auditors’ abilities to access information and government systems in a remote environment. The SiCAF is meant to address these vulnerabilities, with the goal of allowing the ASF to audit 100 percent of public works virtually, thereby reducing the need to deploy auditors across the country. Moreover, the SiCAF will help the ASF to improve oversight throughout all phases of the project cycle. The AEGF partnered with INFOTEC, a public Mexican research centre that is part of the National Council of Science and Technology (Consejo Nacional de Ciencia y Tecnología, CONACYT) and specialised in the development and innovation of technological products and services. The following are key sources of data for the system to process and store information developed with INFOTEC, although not all of this information is available yet:
The Treasury of the Federation (Tesorería de la Federación, TESOFE), an Administrative Unit of the Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público, SHCP) in charge of the financial management of the resources and values of the Federal Government, including: receipt of income, execution of payments charged to the expenditure budget and administration of the available resources of the TESOFE. As noted, ASF’s data sharing agreement with TESOFE was cancelled in December 2020 and had not been renewed as of March 2022.
The National Banking and Securities Commission (Comisión Nacional Bancaria y de Valores, CNBV), a decentralised body of the SHCP with powers of authorisation, regulation, supervision and sanction on the various sectors and entities that make up the financial system in Mexico, as well as on those individuals and legal entities that carry out activities provided for in the laws relating to the financial system. The CNBV’s databases offer names of account holders and data on movements of bank accounts. As of March 2022, there was still no data sharing agreement between ASF and the CNBV.
The Ministry of Economy (Secretaría de Economía, SE), a cabinet-level body responsible for economic policies and overseeing the economy, and it maintains a hotline for citizens to report suspected fraud.
The Tax Administration Service (Servicio de Administración Tributaria, SAT), which maintains taxpayer records for individuals, tax vouchers, information on government suppliers, among other data.
Unstructured data, such as audit reports and contracts. This information is not available in INFOTEC as of March 2022.
Social media outlets (e.g. Twitter, Facebook, Instagram, YouTube and LinkedIn). This information is not available in INFOTEC as of March 2022.
According to ASF officials, the AEGF intends to complement available databases with additional information from other public institutions and that is relevant for the identification of new cases. It also intends to promote the implementation and use of software and technologies that allow the processing of unstructured information. At the time of writing this report, the lack of data sharing agreements for key databases created delays for uploading the information to servers, as noted above. Figure 2.1 provides an illustration of the system to process and store information developed with INFOTEC.
Figure 2.1. Overview of the System to process and store information developed with INFOTEC
Source: ASF.
The AECF developed its own analytics capacities and an “intelligence system” (sistema de inteligencia) to support auditors within its department, drawing from many of the same sources as the SiCAF. The General Directorate of Audit of Information and Communications Technology (Dirección General de Auditoría de Tecnologías de Información y Comunicaciones, DGATIC) within the AECF is responsible for processing, storing and maintaining information, as well as setting the policies for improving the quality of data it manages. A team of three experts within the DGATIC manages the central data repository, cleaning data at the request of auditors in other DGs within the AECF, maintaining data integrity and setting data policies. The current system incorporates data from various sources, including open data sources, the ASF’s own databases (e.g. data collected on government contractors), government contract databases and development bank databases.1
The DGATIC effectively acts as data service provider for other teams within the AECF, but it conducts some analytics on its own, such as analysis of trends or suspicious transactions in public procurement data, or analyses of sanctioned businesses. According to ASF officials, the AECF has also been developing prototypes that will allow it to add unstructured information to its database, increase its analytical capabilities and build predictive models for the integration of unstructured sources and detection of suspicious behaviour. Analytics is decentralised further to other DGs which have the subject matter expertise and use tools like Excel and ACL to support audits. The DGATIC officials described future plans for the AECF to build on its intelligence systems towards a more integrated, cloud-based system that takes advantage of other data sources, analytic techniques and outputs (e.g. visualisations and risk profiles). Like the SiCAF, this system would improve how the ASF conducts machine learning and network analysis. Figure 2.2 is ASF’s own illustration of the AECF’s plans for a new intelligence system, bringing together structured and unstructured data, including various forms of administrative data, into “master catalogues” of data that can ultimately be used by auditors.
Figure 2.2. Proposed new Intelligence System for the AECF
The ASF can access databases from other entities across the Mexican government, per national laws but this authority has its limits.2 In particular, the ASF does not have the authority to access Platform Mexico (Plataforma Mexico), which is maintained by the Ministry of Public Security (Secretaría de Seguridad y Protección Ciudadana) and includes a number of databases consisting of police records, criminal records, biometric data, prison records and vehicle records, among other data. The ASF officials said they are working to address access issues, including limitations to other data sources, but they are often restricted due to confidentiality and national security provisions. The ASF is also exploring possibilities to license additional databases.
2.3. Improving co-ordination and building capacity
2.3.1. Strengthening internal co‑ordination around data processes and analytics
There are numerous avenues for the AECF and the AEGF to enhance co-ordination, building on the existing communication as the foundation, as shown in Table 2.1. Co‑ordination is an iterative process and can vary depending on the objective. As shown in the table below, higher degrees of co-ordination requires a higher level of institutional commitment on behalf of senior leadership and staff within the AECF and the AEGF, in particular, as they are responsible for much of the ASF’s core analytics capacities. Given its institution-wide mandate and existing role in leading the ASF’s digital transformation work programmes, the Unit of Regulation and Legislative Liaison (Unidad de Normatividad y Enlace Legislativo, UNEL) would have a key role to play.
Table 2.1. The co-ordination spectrum based on defined objectives and level of intensity
Note: The intensity of co-ordination can be viewed on a continuum of low to high from left (starting with “Communication”) to right (ending with “Integrated action and decision-making”).
Source: OECD adaptation of (Strimling, 2006[4]).
In interviews with ASF officials, the AECF and the AEGF said they are aware of each other’s initiatives; however, substantive co-ordination on common policies, practices or the development of tools remains limited. For instance, each department independently carries out activities related to data management, analytics and strategic planning. Moreover, the AEGF and the AECF, along with their respective DGs, have their own set of policies and processes for managing data for their audits, some of which includes databases with the same structure, even if the data fields have different information. Co-ordination occurs primarily within departments between the DGs, with limited co-ordination and co-operation between the AECF and the AEGF concerning their data and analytics efforts. According to ASF officials, part of the issue with regards data sharing is that regulations prohibit one team from accessing information of the other. Insufficient co-ordination and co-operation between the AECF and the AEGF in particular increases the risk of inefficiencies.
On the spectrum of co-ordination described above, there are several ways for the ASF to enhance internal co-ordination for improving its analytics and the data management that underlies it. Based on workshops with ASF officials, the current intensity of co-ordination is low and mostly reflects the “communication” end of the spectrum, including giving presentations on each other’s initiatives and taking part in institution-wide committees. The development of an action plan for analytics would provide a constructive vehicle for advancing co-ordination beyond basic communication towards integrated action and decision making. In addition, several of the sources in the AECF’s intelligence system appear to overlap with the AEGF’s SiCAF, including data from SAT, SHCP, SE and TESOFE. A joint review of the extent of overlap, considering these entities have different databases, could provide assurances that there are no duplicative efforts in terms of data processing in particular. This phase, which can involve extensive data cleaning, is typically the most resource intensive, while the actual analytics makes up a smaller percentage of the time required of auditors and data experts. Figure 2.3 illustrates a general process for what is commonly referred to as “data analytics,” which is often more focused on processing the “data” than doing the “analytics.” Finally, according to AECF officials, many of the open source databases at its disposal are not useful, because the quality of data is poor. As a result, they prefer to organise direct access to data with the relevant authority. Improved internal co-ordination within the ASF also has the potential to reduce the burden on data owners and auditees to the extent there is a risk that multiple teams within the ASF request the same data.
Figure 2.3. Analytics as an exercise in pre-processing data
2.3.2. Considering data sharing pilots for breaking down siloes
In interviews with the OECD, the DGATIC and the DGAF officials noted it would be useful to be able to share systems and developments in order to facilitate better co-ordination within the ASF. In the current situation, DGATIC and DGAF have limited knowledge of the databases the AEGF uses, officials said, and they recognised the possibility that the AEGF could be using a database that would help their work and vice versa. Even though the AEGF and the AECF are not able to access each other’s databases by law and they have different auditees, many of the databases they use share similar structures, as noted. Moreover, the introduction of a new forensic team within AEGF (the DG of Forensic Audit and Federal Spending, or Dirección General de Auditoría Forense del Gasto Federalizado), suggests the possibility of furthering creating siloes of forensic activities at the directorate level. This development presents opportunities, as well as risks, in terms of data sharing or lack thereof. The ASF could consider a data sharing pilot with a discrete objective as an efficient, simple means to test collaboration. The phases of a data pilot focusing on data sharing for detecting fraud risks is depicted in Figure 2.4 and could include not only teams within the ASF, but also stakeholders and data owners outside of the ASF, such as members of the NACS, as described in the previous section.
Figure 2.4. Five phases of a data pilot for fraud detection
Each of these phases breaks down into a series of steps to complete a data sharing pilot. In the first phase, a critical step for the ASF, there are considerations as to whether the pilot involves internal or external partners, as well as privacy and security concerns of the relevant data sources. In general, the ASF adheres to a privacy law in Mexico, the General Law on the Protection of Personal Data in Possession of Obliged Subjects (Ley General de Protección de Datos Personales en Posesión de Sujetos Obligados). The law governs the data sources that ASF can access, and its officials take mandatory trainings and certifications to ensure they understand the requirements. Given its mandate, the ASF has a broad authority to access data across government directly from the administration, with the exception of specific data sources. There are legal constraints for the sharing of some data (e.g. census data, data related to national security interests and private sector data). However, as illustrated by the experience of the PDN and limitations it has faced, other challenges could remain that a data sharing pilot would help to uncover:
Cultural – this might be expressed as “we do not share data”.
Risk appetite – this might be expressed as “it is just too risky to release our data”.
Familiarity – this might be expressed as “we have never done it before, we wouldn’t know where to start”.
Capability – this might be expressed as “we do not have the technical or legal expertise that we would need.”
Resources – this might be expressed as “we do not have enough resources to devote to a data sharing project” (Commonwealth Fraud Prevention Centre, Government of Australia, 2020[6]).
A data sharing pilot could help to first identify and then address these challenges constructively and incrementally, using fewer resources to test concepts before the ASF commits to more sustained forms of collaboration, such as an automated data sharing arrangement. Even if the stakeholders decide not to move ahead following the pilot, the process itself can provide insights to enhance data quality and ultimately improve fraud detection. For instance, sharing information about data sources, and if relevant, sharing responsibilities for data management, cleaning and other common activities could help to break down or prevent siloes at the auditor level. The pilot can facilitate informal channels for auditors to collaborate on data quality issues, as well as sharing data dictionaries, methodologies and techniques used, coding and even analyses to the extent they are relevant.
2.3.3. Institutionalising a cross-departmental and cross-functional capacity
Enhancing internal co-ordination between existing departments and DGs is a critical, but insufficient, step for the ASF to fulfill its own plans for developing the aforementioned IT systems, as well as advance its digital transformation work programme. Moreover, existing co-ordination mechanisms with regards to the ASF’s data management and analytics are largely ad hoc, and as noted, focus largely on communication between respective departments and DGs. At a minimum, the ASF could establish a cross-functional group to formalise the current ad hoc communication and promote consistent exchange of knowledge, expertise and data across departments and DGs. The Italian Court of Audit’s (Corte dei Conti, CdC) Data Analysis Competency Centre offers an example of this model. Box 2.1 illustrates other models from the SAIs of the United Kingdom and Turkey, both of which have recognised the need for dedicated entities with institution-wide support and responsibilities to enhance data processes and analytics.
Box 2.1. Examples of analytics communities of practice in SAIs
The United Kingdom
The United Kingdom’s National Audit Office (NAO) established a Data Service to meet the demands of auditors that routinely need access to large volumes of data. This team maintains a number of large datasets, stores them in NAO’s data warehouse and merges them for auditors to use and interpret. The Data Service also provides guidance for audit teams that are using the data, which can be accessed through a common Share Point site. The Methods, Economics and Statistics Hub (MESH) complements the Data Service. This community of practice leads the NAO’s work on analytics and big data, and it co‑ordinates across a range of specialist areas to provide training and financial support for audits and wider assurance work. In addition to data analysis and analytics, MESH’s areas of expertise include economics, statistics, modelling, mapping, and qualitative analysis.
Italy
The Italian Court of Audit (Corte dei Conti, CdC) developed a “Data Analysis Competency Centre,” which became a cross-functional team and brings together business and technical competencies to support the effective implementation of ConosCo. The Centre supports users of ConosCo to make better decisions using machine learning, analytics, predictive analysis and other data analytics techniques. This Centre is in the early stages of its development and intends to be a multi-disciplinary team with knowledge and skills that span levels of government (i.e. national and regional) as well as technologies. According to CdC officials, this effort signals a recognition that any data-driven tool is not static, and requires a capacity-building strategy to support its development and evolution.
Turkey
In 2017, the Turkish Court of Accounts (TCA) created a “Data Analysis Group” to design methodologies for using computer-assisted audit techniques (CAAT) and enhance the capability of the TCA to assess risks in municipalities. The group had other aims, including decreasing auditors’ workload, analysing big data, identifying mistakes and errors in data processing, and automation of analyses to facilitate continuous monitoring. Their efforts resulted in “VERA”, TCA’s Data Analysis and Business Intelligence System, which automates risk analysis for over 1 400 municipalities to inform audit programming and planning.
The examples in Box 2.1 and the experience of other SAIs suggest that the degree of the formality of the group (e.g. working group, community of practice or unit) and its place in the ASF’s hierarchy can vary based on strategic and institutional factors, including the evolution of the ASF’s current analytics capacity. Decentralisation of analytics functions, as is the case in the ASF, comes with benefits. For instance, it allows teams to build expertise around specific databases and methodologies that are most relevant for their audit universe. In the context of carrying out integrity risk assessments, auditors who know the business processes of auditees can have sharper insights about the vulnerabilities in internal control systems and sources of potential integrity risks. Centralising of data management or analytics functions would not be able to replace this type of knowledge that accumulates over time.
As described in the next section, a formal data capability assessment would help the ASF to further target key issues and prioritise next steps; however, input from ASF officials already suggests that capacity gaps exist at all levels of the organisation, even though ASF has carried out trainings for a small group of auditors on big data. A community of practice that operates as a network for information exchange and knowledge sharing would be a conservative start. However, there are other ways for the ASF to go beyond communication as a form of collaboration to enhance its use of data and analytics. For instance, one model would be for the ASF to create a centralised data service or analytics function that would focus on specific cross-cutting areas of the ASF’s analytics processes, while leaving the analysis to the teams. This model would be similar to the UK model of having a Data Service and the Methods, Economics and Statistics Hub, which supports auditors with training on analytics. To some extent, this approach also reflects what the AECF is already doing at the department level. Regardless of the model, given the cross-cutting nature of the ASF’s data and analytics and its relevance for institution-wide goals, an effective group would likely need to be above the level of a DG and have direct reporting lines to senior leadership. It could represent a cross-section of the ASF’s existing strategic and technical functions to make it distinct from existing teams, such as the UNEL and the Special Audits (i.e. the AEGF and the AECF).
The ASF could also consider establishing a formal role, such as a Chief Data Officer (CDO) or Chief Technology Officer (CTO), to act as a steward for institution-wide data policies and processes. The precise title is less important than the definition of the duties and position within the ASF hierarchy. In the current organisational structure, the UNEL exists to provide high-level advice, planning and co‑ordination on the ASF’s strategies for implementing IT policies and systems. While this unit provides “political governance,” it is not designed to take on the operational data governance that affects the day-to-day success of the ASF’s use of data, analytics or new technologies3 that could be envisioned for a CDO- or CTO-like role. For instance, CTOs can help organisational leaders to navigate different technological options, such as clarifying specific options, trade-offs and implications, as these considerations increase in number and complexity (OECD, 2020[9]). The CDO can act as a general caretaker of data, responsible and accountable for all of the ASF’s information assets, including processes around generating data and ensuring their quality and security (Stockpoll, 2021[10]). In some SAIs, an Innovation Lab fulfils some of these roles, as described in the section on experimentation.
The CDO or the CTO role is not always filled by the same person. However, the entity requires the authority and autonomy to provide vision and visibility across the ASF, as well as authority to make strategic investments in architecture, software and tools to address institution-wide needs and priorities. Direct reporting lines to the Auditor General facilitates this role. The individual does not necessarily have to have a technical background in auditing, but would have a grounding in data management and new technologies to play an operational role within the ASF. The Office of the Comptroller and Auditor General of India describes a similar role for its Centre for Data Management and Analytics (CDMA) as follows:
CDMA will play an advisory and supporting role for the overall use of data analytics…CDMA will facilitate through capacity building, collecting third party data at the central level, identifying new software, assessing applicability of different analytic techniques/analytic models, and disseminating them in IA&AD. CDMA will provide technical support to the field offices in their data analytic efforts wherever necessary. The Data Analytic models will be vetted and approved by CDMA, in consultation with functional wings in headquarters (Office of the Comptroller and Auditor General of India, 2017[11]).
Hiring a CDO, CTO, or data scientists does not automatically translate into an ability to extract value from data, or leverage analytics to enhance detection of integrity risks. Digital transformation from an operational perspective relies on a team of individuals that bring the right mix of skills and knowledge. As discussed, this includes individuals with expertise in fraud and corruption to the extent the objectives of the analytics function is to enhance detection of these risks. Given the rapid rate at which fraud detection practices are evolving, the roles of audit institutions are shifting beyond just conventional audits, especially as a result of the COVID-19 pandemic, The onboarding of individuals with a strong understanding of data and analytics is critical, but many SAIs have turned to co-sourcing, contracting, or outsourcing models, which can provide additional expertise to the department or its projects. Regardless of the approach, the ASF can enhance the cross-functionality of its teams as it further develops its capacities for using data, analytics and new technologies. Figure 2.5 illustrates the key elements of a cross-functional team from the perspective of the European Court of Auditors.
Figure 2.5. Key elements of a cross-functional and data-driven audit team
2.3.4. Conducting an internal assessment to further explore capacity gaps and data capabilities
The proposals for action above reflect some of the key priorities for the ASF to enhance its current approach to data management and analytics, drawing primarily from responses to a questionnaire, as well as interviews and workshops with ASF officials. These inputs offer a useful starting point; however, they focused on the analytics led by select departments and by design were not meant to cover the broad scope of issues facing the ASF concerning data and analytics. The ASF could take additional steps to elaborate on its internal capacity challenges within all departments and teams, including an institution-wide assessment of capacity gaps. According to the International Organisation of Supreme Audit Institution (INTOSAI) Development Initiative’s Strategic Management Handbook for SAIs, assessments can be carried out as a step in strategy development, so that capacity gaps are determined in relation to defined objectives and outputs (INTOSAI, 2020[13]). For example, the ASF could start with its objectives for enhancing the use of data and analytics in its audits and investigations, and addressing issues of operational data governance, as described above. The linkage to concrete objectives will help the ASF to nuance the assessment so that it targets gaps that are relevant for what the ASF wants to do in the future, while recognising the diversity of needs across the organisation. As discussed, the ASF’s current analytics capacity is highly decentralised and operates in siloes, so any capacity assessment would need high level stewardship to ensure collaboration between departments, particularly the AECF and the AEGF.
There are numerous frameworks available to support the ASF in carrying out an assessment of its internal capacity for data and analytics. Effective assessments map the key elements of data governance particularly capacity for coherent implementation, as described in Chapter 1. Assessments often provide a holistic view of gaps and strengths as a basis for establishing development priorities. In New Zealand, the government developed a data capability framework that defines 25 capabilities for effective data use, based on seven categories of the data lifecycle (see Figure 2.6). The ASF could reference this framework as a template for identifying potential areas of improvement with respect to strategic planning, performance development, recruitment and on boarding (Government of New Zealand, 2020[14]).
Figure 2.6. Data capability framework of the New Zealand government
New Zealand’s data capability assessment focuses on breadth over depth, but it will not necessarily offer a greater understanding of root causes of those challenges. For a more comprehensive and nuanced picture, the ASF could conduct a root cause analysis that would provide further insights about not only the technical challenges facing auditors, but also the human and cultural elements that influence the ability of the ASF to adopt analytics and fulfil broader goals of digital transformation. As part of this analysis, the ASF could also look at specific challenges facing individual teams and processes, including those related to the application of analytics for detecting irregularities and integrity risks.
SAIs use root cause analysis for their own audits in an effort to go beyond the identification of deficiencies and understand the key challenges and features of an issue. For instance, the Auditor-General of South Africa in its Consolidated General Report on National and Provincial Audit Outcomes provides an overview of how auditees have addressed the root causes of audit findings (Auditor General South Africa, 2020[15]). In addition, the INTOSAI Development Initiative, a not-for-profit organisation that supports SAIs to enhance their performance and capacity, promotes the use of root cause analysis in its implementation handbooks for International Standards of Supreme Audit Institutions (ISSAIs) for performance and compliance auditing, and provides guidance on different approaches.4 Box 2.2 offers additional insights and a resource for conducting root cause analyses from the Canadian Audit and Accountability Foundation. These references could support the ASF in applying a root cause analysis internally to obtain a fuller understanding of its capacity challenges for using data and analytics. This analysis can complement the proposals for action in Chapter 1 to enhance the ASF’s strategic approach to analytics and create an action plan, with performance monitoring, so that further assessment of capacity and resource issues are tied to actual institution-wide objectives.
Box 2.2. Guidance for conducting root cause analysis
The Canadian Audit and Accountability Foundation (CAAF) is a not-for-profit organisation dedicated to promoting and strengthening public sector performance audit, oversight, and accountability in Canada and abroad. Per the CAAF, root cause analysis can be an effective approach for helping government entities understand complex challenges and fundamental areas of concern. By focusing on the principle question – “why?” we may better be able to identify systemic deep-seeded issues faced by the organisation. Root cause analysis can be integrated into every step of the auditing process—planning, examining and reporting, as shown in Figure 2.7.
Figure 2.7. Canadian Audit and Accountability Foundation’s Use of Root Cause Analysis and Audits
Root causes are often governance-related or operations-related. The first pertains to overarching structures, strategy and oversight. The second is more concerned with the daily workings of the organisation. Increasingly as well, auditors also see broader organisational culture as a third potential category of root causes, and have begun to develop more rigorous methods by which to monitor this phenomenon. Figure 2.8 shows some of the most frequently observed root causes.
Figure 2.8. Canadian Audit and Accountability Foundation’s Main Areas of Potential Root Causes
One simple technique for conducting a root cause analysis is known as the “five whys” method, in which the auditor(s) repeatedly asks the question “why” for each subsequent response in order to determine the true underlying reason behind a finding. Another way is to employ a fishbone diagram (Figure 2.9). By including nudges of potential root cause categories, these diagrams can mitigate human biases and push auditors to think of novel topics they may not otherwise have considered.
Figure 2.9. A Fishbone Diagram
2.4. Enhancing analytics for detecting integrity risks
2.4.1. Improving analysis of risk trends and use of dashboards
Many of the strategic considerations and operational priorities discussed above, while having broader implications for the ASF’s digital transformation, influence its ability to leverage data for detecting integrity risks. Responses from ASF officials in questionnaires and interviews highlighted several specific priorities to enhance the tools and processes in place for applying analytics to the detection of integrity risks, including the development of a risk dashboard to improve how the ASF tracks, visualises and communicates risks across the organisation. According to ASF officials, while the systems envisioned by the AEGF and AECF incorporate dashboards, the ASF has yet to develop a dashboard for supporting its risk analytics for irregularities. SAIs have long used dashboards to support risk identification and tracking. Developing a dashboard, incorporating insights and data from the DGAF in particular, would be a low cost and high return approach to facilitate sharing of risk data and facilitate auditors’ analyses.
As noted in interviews with ASF officials, within the AECF, the DGAF and the Forensic Laboratory support other teams in identifying irregularities and potential fraud, and they maintain a risk registry with red flags. The registry is effectively a database for uploading findings and information corresponding to specific audits with an explanation of the irregularity detected. It includes a brief description of the evidence for the irregularity or potential fraud. Currently, the risk information is communicated ad hoc during meetings among a group of DGs and relevant work teams. The meetings cover a range of issues, including red flags. Risks are also shared across the organisation in the context of specific audits. For instance, DGs may detect an irregularity during the course of their audits, in which case they would engage the DGAF to conduct forensic analyses or investigations, as needed. Among other databases at its disposal, officials said the DGAF is also developing a database that includes the companies flagged for irregularities in prior audits as a resource for future audit teams to identify past issues.
As discussed in meetings with the OECD, the DGAF officials noted the use of the registry and informal database of risks could be enhanced, for instance, by analysing trends and patterns of risks in the data. A risk dashboard offers a vehicle for disseminating such analyses, while allowing auditors themselves to access and explore the information that the DGAF maintains to support audits. Moreover, the use of dashboards can be useful for continuous monitoring and providing auditors with off-the-shelf or automated tools to conduct analyses and prioritise risks. Visualisations incorporated into dashboards can also help auditors to analyse entire datasets for outliers and potential irregularities. The new systems of the AEGF and AECF both envision such functionalities. Box 2.3 shows how the Turkish Court of Account made use of risk dashboards and automated trend analysis to support its annual audit programming.
Box 2.3. Automating risk analyses at the Turkish Court of Accounts
The Turkish Court of Accounts (TCA) created “VERA”, a Data Analysis and Business Intelligence System, which automates risk analysis for over 1 400 municipalities to inform audit programming and planning. VERA provides auditees a standard, automated tool for risk-based ranking of over 1 400 municipalities. VERA allows management to take into account risks before the TCA’s annual audit programming and supports the creation of the audit strategy. In addition, auditors use the results of the risk analyses to plan audits, as well as identify possible material misstatements in financial reports that could represent errors and fraud. All auditors have access to VERA, and are able to assess the results of VERA’s automated analyses related to risks and financial indicators in a dashboard or automatically generated reports.
Source: Interview with the OECD.
In addition, the experience of the UK National Audit Office (NAO) demonstrates that investing in dashboards and off-the shelf tools for auditors can also have benefits for reporting. The NAO’s Data Service has developed various tools for its auditors, such as web-scraping of inspection reports to harvest data on school funding or to assess the readability of tax guidance, which automate phases of the analytic process. This allows auditors to spend more time analysing information and data, and less time collecting it. The visualisations offered on the NAO’s dashboard not only support analyses, but they can also be integrated into reports to raise attention about issues and support key messages. Some visualisations can attract as much attention as the report itself (UK National Audit Office, 2018[17]).
2.4.2. Enhancing follow-up on findings and creating feedback loops to improve analytics
At the conclusion of its audits, the DGAF lays out its “determination of the facts” for the auditee, which has 30 working days to resolve the findings before the ASF issues a report or presents a complaint to the National Prosecutor’s Office (Fiscalía General de la República, FGR). Only in cases when it is clear a crime has been committed can ASF issue a report to relevant authorities, before the end of this 30-day period. If the DGAF identifies evidence of fraud or corruption, in accordance with the Law on Auditing and Accountability (Ley de Fiscalización y Rendición de Cuentas de la Federación) and in compliance with the ASF's internal regulations, it must prepare technical reports, which are sent to the Legal General Directorate for referring to relevant authorities. The DGAF relies on co-ordination with the Special Audit of Monitoring, Reporting and Investigation (Auditoría Especial de Seguimiento, Informes e Investigación, AESII) and the AECF for follow-up of audits, as its authority ends with the issuing of its findings.5
Follow-up is a fundamental phase of the audit process, reflected in various INTOSAI standards and guidance.6 SAIs can evaluate impact in different ways, including assessing the impact and the uptake of its recommendations by auditees (EUROSAI, 2019[18]). ASF institutionalised a follow-up mechanism in the AESII; however, according to officials, the team is under-resourced and follow up can be lengthy. Knowing the status and the outcome of audits is a critical step in the feedback loop for the DGAF and other DGs. For example, feedback loops—knowing the results of audits and how the DGAF’s findings supported outcomes—act as a control for the DGAF’s and Forensic Laboratory’s own analytics functions. The DGAF can fine tune its forensic methodologies and analytics based on the ultimate results of the audits and whether findings led to concrete actions. Optimisation of methodologies helps to reduce false positives and false negatives, and enhance the logic that underlies algorithms and indicators for detecting irregularities.
2.4.3. Strengthening analysis of unstructured and semi-structured data
Improving the management, processing and analyses of unstructured data has become a key priority for many SAIs to enhance its analytics in the digital age. By some estimates, including a 2016 study on text mining, unstructured or semi-structured data accounts for over 80 percent of all data (Talib et al., 2016[19]). Unstructured and semi-structured data accounts for large amounts of “big data” and will be an ongoing challenge for the ASF in the future. The AEGF and the AECF both envision improvements over the coming years in terms of the ASF’s architecture, methodologies and tools (e.g. machine learning) to better analyse “big data.” In interviews, ASF officials emphasised the need to build capacity to achieve its ambitious goals in this area, which will necessarily require improvements to how the ASF manages, processes and analyses unstructured and semi-structured data.7 They also highlighted the need for improving capacity for managing and analysing unstructured data as one of their top priorities in the coming years.
Other initiatives have the potential to lead to the systematic collection of more unstructured data than the ASF has ever had to manage in the past. For instance, the ASF recently established a Digital Mailbox (Buzón Digital) to enhance the bilateral communication between auditors and auditees. This platform allows the ASF and audited entities to manage the audit process electronically, such as by sending requests and certifying documents. It will also facilitate the auditees’ submission of documentation for audits, allowing the ASF to collect text files and supporting evidence for audit easier than it has ever been able to in the past. Moreover, as noted, the ASF has also developed prototypes to add unstructured information to existing databases, which in turn would help building predictive models, detecting suspect behaviour and increasing analytics capabilities.
The systematisation and digitalisation of this process makes auditing easier and promotes efficiencies, particularly in a remote environment, but it comes with risk. One risk is that auditees will submit more documentation, even if it is irrelevant for the audit, which would have the potential to overwhelm the audit team unless they have the appropriate tools and skills to analyse the text quickly. Text mining and other analytic techniques can be helpful in such situations, depending on the objectives of the audit and the format of the evidence submitted. There are several examples of SAIs that have progressed in recent years in their capacity to process and analyse unstructured data. Many of these initiatives focus on one type of analytic technique, and it is common to see examples that focus on text data. For instance, the SAI of Germany, the Bundesrechnungshof, analysed how federal government entities communicated to the public and its impact on public perception and the readability of messages. To do this, the SAI explored the use of various analytic techniques, including web-scraping, text mining, natural language processing and sentiment analysis of publicly-available sources (e.g. press releases, social media posts and news articles) (EUROSAI, 2021[20]).
Similar processes can be used in the context of assessing corruption risks in infrastructure. For example, a line ministry could assess internal risks of fraud or corruption by scraping emails or social media to identify red flags, like key words or evidence of procurement officials spending beyond their means. To maximise the value of text analytics, entities may use the fraud triangle as a reference to develop a list of keywords based on the industry, relevant fraud risks, and data set (OECD, 2019[21]). Social network analysis is also commonly applied to unstructured data related to infrastructure and public procurement in order to identify collusion amongst actors in the procurement cycle. Applying network analysis in this context can help to raise red flags and identify corruption risks. Moreover, data visualisations can be used to present the results of network analysis to identify “hot spots” of potential fraudulent activity.
The ASF could further develop its own capacity to analyse unstructured and semi-structured data, building on current initiatives Figure 2.10 provides a broader framework for the ASF to take into consideration when thinking about a strategic approach to unstructured/semi-structured data that goes beyond text analytics, and accounts for the different types of unstructured data sources it encounters, including audio, images and video. Going beyond text data, the framework could be useful for the DGAF and teams that collect other types of unstructured and semi-structured data.
Figure 2.10. A framework for capturing and analysing unstructured data
The various processing and analytic techniques described in the figure above are beyond the scope of this report; however, the diversity of techniques and their underlying tools highlight the need for the ASF to consider strategically how to approach unstructured data. As noted above, this starts with defining clear objectives and priorities of auditors, while building capacities based on further assessment in the gaps in capabilities. Many of the analytics described are those that the ASF, particularly DGAF, may already be carrying out. However, as shown in Figure 2.10, the process of integrating findings and results from the analysis of unstructured data into the traditional systems of the ASF, as well as into dashboards for auditors to reference, still largely remains an ambition for future work.
2.5. Nurturing a data-centric culture
2.5.1. Promoting digital skills and ethical use of data through trainings
Introducing new systems or tools is insufficient; developing skills, motivation and interest in analytic approaches is vital for sustaining future analytics initiatives. Leading practices from other SAIs consistently highlight the development of auditors’ skills and capabilities as a key enabler of digital transformation. For instance, the National Audit Office of Finland’s (NAOF) maturity in terms of data and analytics reflects the ASF’s own path, as it advances with its digital transformation work programme and updates its architecture and tools to better support auditors. Officials from the NAOF described the next phase of their digital transformation as one in which data and analytics becomes more systematised and integrated across the NAOF’s audit work. In discussions with the OECD and the ASF, NAOF officials highlighted people, skills and organisational culture as key enablers on its digital journey. Officials also highlighted the need to focus on building a culture and models for continuous process development, driven by the audit expertise, the availability of data and opportunities of new technologies (Kärki and Saarteinen, 2020[23]).
Data literacy is often highlighted as a key requirement of modern auditors’ skillset, as described, and is the focus of trainings, workshops and guidance for SAIs. While critical, data literacy—the ability to read, interpret, create and communicate data as information (OECD, 2020[24])—is just one component of a broader set of competencies that the ASF could focus on in developing its workforce to meet the demands of auditing in a digital age. In addition to data literacy, the ASF could promote the development of digital skills, defined as the broader range of abilities to use digital devices, communication applications, and networks to access and manage information. For auditors, these skills include an understanding of software, tools and data (OECD, 2020[24]).
The distinction between data literacy and having digital skills reflects the notion that auditors have different specialities and require varying levels of specialisation when it comes to managing and using data; however, all auditors can benefit from having an understanding and fluency with a range of digital tools and technologies that are critical for the modern auditing profession. Auditors with digital skills are data literate, but they are also equipped to ask strategic questions, understand limitations of techniques and tools and maintain realistic expectations about time and resources when planning the use of data and deciding on methodological trade-offs. At the time of drafting this report, officials said ASF had trained 50 auditors on the use of “big data,” but without elaborating on the details of the content or target audience for the trainings.
Nonetheless, when thinking about the competencies needed for its auditors, the ASF could draw inspiration from the European Union’s Digital Competence Framework (DigComp), which is a tool to improve citizen’s digital competence. In its report, Building digital workforce capacity and skills for data-intensive science, the OECD assessed the relevance and adequacy of DigComp for the academic science community. As a type of evaluator, external auditors in the public sector share many of the same requirements as academics in terms of digital competencies. Moreover, the ASF could follow many of the same principles of the science community reflected below, including the promotion of transparency and leading by example (i.e. protecting one’s reputation). The criteria below, which include both the OECD’s additions to the DigComp’s original framework as well as elements of the original framework itself, can provide a useful categorisation for the ASF as it considers the types of digital skills its auditors need in addition to digital literacy:
Information and digital literacy: Browsing, searching and filtering data; critically evaluating credibility and reliability of data sources; organising and storing data. Understanding of statistics to help evaluation and analysis of data; understanding of requirements for reproducibility.
Communication and collaboration: Sharing data; knowing about referencing and attribution practices; using digital tools and technologies for collaborative processes; protecting one’s reputation. Following open science principles to share data, information and content, engage in good digital citizenship, and improve collaboration; extend knowledge of referencing and attribution practices to research data and software citation/referencing; protecting academic reputation, both of one’s own organisation and that of academic research more generally.
Digital content creation: Creating new, original and relevant content and knowledge; understanding copyrights and licenses; programming and software development, visualisation of data and information to convey knowledge.
Safety: Protecting personal data, protection of sensitive data, understanding of tools and techniques such as delinking, anonymisation and safe heavens.
Problem solving: Customising digital environments to personal needs; using digital tools to create knowledge and innovate processes; identifying digital competence gaps and seeking opportunities for self-improvement (OECD, 2020[25]).
The “safety” competency touches on a critical issue for the ASF and SAIs that goes beyond the competencies described above. This involves the ethical implications of data use, including auditors’ own use of data. For this purpose, and depending on their position and level of responsibility, the ASF could consider this competency beyond what is described in the framework above. There are several ways the ASF can raise awareness and promote the ethical use of data. Box 2.4 provides examples from the OECD’s Good Practice Principles for Data Ethics in the Public Sector.
Box 2.4. Good practices for promoting the ethical use of data
The Good Practice Principles for Data Ethics in the Public Sector shed light on the value and practical implications of data ethics in the public sector. They aim to support public officials in the implementation of data ethics in digital government projects, products, and services so that: i) trust is placed at the core of their design and delivery; and ii) public integrity is upheld through specific actions taken by governments, public organisations and, at a more granular level, public officials.
The Thematic Group on Data-driven Public Sector, meeting under the aegis of the OECD Working Party of Senior Digital Government Officials (E-leaders), drew together Good Practice Principles for Data Ethics in the Public Sector. They emerge from observed practices in digital government and data-driven public sectors across OECD Member and non-Member countries. The following good practices provide insights as to how organisations can promote the ethical use of data:
Ensure the availability of multi-faceted and diverse teams working on or collaborating around specific projects. Diversity in the workplace can help to mitigate biases by offering multiple perspectives on a policy issue and fostering inclusive and informed decisions in terms of the data informing or resulting from a project (e.g. selection of data sources, data availability issues, data access restrictions, data’s reflection of reality).
Publish data governance and management policies, practices, and procedures, especially around the use of personal data.
Engage in social dialogue with relevant actors inside and outside the public sector. These include actors whose data is being used, or their representatives, and secondary stakeholders who can be affected or harmed by data use. Multi-stakeholder and multi-faceted approaches can help in identifying risks, defining boundaries and channelling actions prior, during and after the deployment of projects, policies and decisions involving the access to, sharing and use of data.
Communicate to relevant stakeholders, or their representatives, in a clear and understandable way about the role of data (e.g. expected benefits and trade-offs), and its primary purpose – including in the context of training algorithms. Intention and use beyond the original purpose and the impact of not consenting to data use should also be communicated (e.g. delays due to slower decision‑making procedures to grant access to or deliver public services).
Acknowledge the social context, including factors such as the presence of indigenous communities and native nonofficial languages to foster inclusion.
Educate relevant stakeholders (e.g. data subjects and their representatives, and those from vulnerable, underrepresented, or marginalised groups in society) on data governance, including its meaning and implications for them. Confront scenarios in which only privileged and educated segments of the population have a voice and say in how their data is being used. This includes the capacity to contest certain uses of data.
Source: (OECD, 2020[24]).
2.5.2. Creating room for experimentation and small wins
Among SAIs with successful initiatives to incorporate data and analytics into their audit work, an openness to experimentation is a consistent theme, even when other aspects of the SAI’s work and culture remain risk averse. ASF has demonstrated a willingness to experiment. For instance, ASF officials said the AEGF launched a pilot exercise whereby the audit areas were provided with cases of suspicious suppliers and contractors, so that they could be reviewed in greater detail during audits. The auditors’ feedback from this effort will be used to improve ASF’s analytics and establish guidelines to extend the use of data in more audits.
As noted in Chapter 1, leadership can make its support for strategic experimentation explicit in its strategy and action plan for analytics, for instance. The freedom for auditors to experiment creates opportunities for both small wins and small losses, meaning a SAI can pilot new methodologies, tools and data sources in a controlled and cost-efficient way before deciding whether to scale up or avoid developing further. For SAIs with “Innovation Labs,” experimentation has become a strategic objective. One benefit of an innovation lab is that it helps to institutionalise knowledge and expertise, and for the ASF, it could help to advance new methodologies it is already considering that may benefit multiple departments. This would be a key difference from the ASF’s existing analytics efforts, including the DGAF’s Forensic Laboratory, which focuses more on supporting investigative processes for a specific directorate, rather than promoting institution-wide innovation as a priority with benefits for integrity risk detection and beyond. The Office of the Auditor General of Norway (OAGN) established an innovation lab to promote data science within the OAGN and support auditors with a range of tools and functions (see Box 2.5).
Box 2.5. The Innovation Lab at the Office of the Auditor General of Norway
The Office of the Auditor General of Norway created the Innovation Lab in 2019 as a semi-autonomous body to advance the use of data science, machine learning and provide the country’s audit work with more computing power. The lab conducts a broad scope of work, including:
gathering and preparing data for audits
conducting analytics work on demand
creating applications to make the work of auditors more efficient
educating auditors about the use of machine learning
experimenting with new analytics techniques
promoting a culture of data science across the organisation.
The Innovation Lab has found success as a result of being given the freedom to experiment, receiving full support from management, and using free open-source technology to reduce costs. The group hires auditors rather than individuals with technology backgrounds and for most of its work, focuses on finding solutions to long-standing root cause concerns. By solving some of the concrete problems faced by auditors, they have built credibility and trust across the OAGN, and by managing the data sciences, the office gives auditors more time to focus on analysis.
Establishing an Innovation Lab or adding a permanent team to the ASF’s organisational chart is not the only approach. Moreover, the ASF’s existing teams demonstrate a high level of ambition to innovate, as illustrated by some of the examples described in this report. Nonetheless, in conversations with ASF officials, the notion of experimentation and investing resources in pilots before investing in the overhaul of architecture or introduction of new tools was not part of the strategic approach. Considering the ASF’s current structure and initiatives, the ASF could also consider temporary models to tap into the skills and innovative energy of its staff. For instance, the Auditor General of Wales developed a 9-month project called the “Cutting Edge Audit Office,” which aimed to transform how the Wales Audit Office used data and technology. The team consisted of six junior staff that reported directly to the Auditor General (see Box 2.6).
Box 2.6. The Cutting Edge Audit Office Project of the Wales Audit Office
The Cutting Edge Audit Office in Wales was a temporary creation of the Auditor General as a means of transforming the supreme audit institution. Specifically, the office’s mandate included the following:
data acquisition
data analytics
the use of data in day-to-day activities
audit reporting such as data visualisation
building long-term skills and strategies around audit innovation.
The success of the office was a result of different factors. For instance, by reporting directly to the AG, the Office’s work was tangible and left a lasting legacy. The Cutting Edge Audit Office developed and implemented a three year strategic plan on data use, and data was harvested from new sources like the health department and social media. New applications were introduced internally to make work more efficient and relevant including the automation of the analytics process and the adoption of data visualisation as a form of reporting on some audits.
Source: (Auditor General of Wales, 2020[28]).
2.6. Summary of the proposals for action
The ASF’s analytics capacity and related processes for data governance are decentralised across different departments. This approach has allowed the ASF to tailor data governance, data management and analytics to suit the needs of individual audit teams. The ASF has developed strong analytic capacities with this approach; however, it has also led to siloes that are exacerbated by insufficient co-ordination. In addition, the ASF has invested in trainings for auditors, but it could take additional steps to understand its priorities for developing digital competencies, including data literacy, so that its auditors can keep pace with the digital change around them in government and society. This includes the need to enhance the ASF’s capacity and processes for leveraging analytics to detect integrity risks, as well as the need to further develop a data-centric culture. The following proposals for action are not exhaustive related to improving co-ordination, enhancing analytics for detecting integrity risks, and nurturing a data-centric culture. However, they provide a starting point for the ASF to address key operational challenges and additional considerations for enhancing the use of data and analytics:
Strengthen internal co-ordination around data processes and analytics—Substantive co-ordination on common policies, practices or the development of tools across departments remains limited. Opportunities remain for the ASF to move towards more integrated decision making as a form of internal co-ordination at a departmental and team (i.e. DG) level to provide assurance that there is no duplication or unwanted overlap of efforts. The ASF could conduct a joint review of possible areas of duplicative activities across departments, particularly with respect to its data processing and quality checks, considering the heavy burden these activities put on resources and time. Improved internal co-ordination within the ASF also has the potential to reduce the burden on data owners and auditees to the extent there is a risk that multiple teams within the ASF request the same data.
Consider data sharing pilots for breaking down siloes—To help address internal co-ordination challenges and the potential for inefficiencies, the ASF could conduct a data sharing pilot to address some of the challenges it faces concerning internal (and external) co-ordination, building on precedents for data sharing with other government entities (i.e. SAT, SHCP, and TESOFE). This pilot could involve enhanced communication about similar databases used across departments. If relevant, it could also include sharing responsibilities for data management, cleaning and other common activities that would help to promote efficiencies for resource-intensive tasks and break down or prevent siloes at the auditor level. The pilot could also facilitate the creation of informal channels for auditors to collaborate on data quality issues and methodologies. Conducting a data sharing pilot would help the ASF to identify and then address these challenges constructively and incrementally, using fewer resources to test concepts before the ASF commits to more sustained forms of collaboration.
Institutionalise a cross-departmental and cross-functional analytics capacity—The ASF could take additional steps to institutionalise its analytics capacity. One approach is for the ASF to establish a cross-functional group or community of practice to formalise the current ad hoc communication between teams, and promote consistent exchange of knowledge, expertise and data across departments and DGs. Another model would be for the ASF to create a centralised data service or analytics function that would focus on specific cross-cutting areas of the ASF’s analytics processes, while leaving the analysis to the teams and sustaining elements of its current decentralised model. The ASF could also consider establishing a formal role, such as a CDO or CTO, to act as a steward for institution-wide data policies and processes. The precise title and whether this role is fulfilled by one individual or many is less important than further defining and assigning roles and responsibilities for operational data governance, particularly for issues that are institution wide. The ASF can also enhance the cross-functionality of its teams as it further develops its analytics capacity.
Conduct an internal assessment to further explore capacity gaps and data capabilities—While input from ASF officials in the scope of the OECD project established several priorities in terms of improvements to capacities, the ASF could take additional steps to elaborate on this work and identify capacity gaps and needs across a broader group of stakeholders. This could involve an institution-wide assessment of capacity gaps, taking into account data capabilities in relation to defined capabilities and the ASF’s plans for future initiatives. As discussed, there are numerous frameworks available to support the ASF in carrying out an assessment of its internal capacity for data and analytics. The assessment should provide a holistic view of gaps and strengths as a basis for refining priorities. The ASF can also benefit from root cause analysis that would provide further insights about the human and cultural elements that influence the ability of the ASF to adopt analytics and fulfil broader goals of digital transformation. As part of this analysis, the ASF could also look at specific challenges facing individual teams and processes, including those related to the application of analytics for detecting irregularities and integrity risks.
Improve analysis of risk trends and use of dashboards—The ASF has established robust processes and capabilities for using data and analytics to detect irregularities. Building on its efforts, the ASF could develop a risk dashboard to improve how it tracks, visualises and communicates risks across the organisation. Developing a dashboard would be a low cost and high return approach to facilitate sharing of risk data and facilitate auditors’ analyses of trends and patterns. The risk dashboard can be a vehicle for disseminating such analysis, while improving access to the risk data that the DGAF has to support audits. Moreover, use of dashboards can be useful for continuous monitoring and providing auditors with off-the-shelf or automated tools to conduct analyses and prioritise risks. Visualisations incorporated into dashboards can help auditors to analyse entire datasets for outliers and potential irregularities, allowing more time for analysing information and data and less time collecting it. Visualisations can also help to enhance the readability and impact of the ASF’s reporting.
Enhance follow-up on findings and create feedback loops to improve analytics—Follow-up is a fundamental phase of the audit process, reflected in various INTOSAI standards and guidance. ASF institutionalised a follow-up mechanism in the AESII; however, according to officials, the team is under-resourced and follow-up can be lengthy.8 Knowing the status and the outcome of audits is a critical step in the feedback loop for the DGAF, and other DGs, who rely on the AESII for follow-up. For example, feedback loops—knowing the results of audits and how the DGAF’s findings supported outcomes—act as a control for the DGAF’s and Forensic Laboratory’s own analytics function. The DGAF can fine tune its forensic methodologies and analytics based on the ultimate results of the audits and whether findings led to concrete actions. Optimisation of methodologies helps to reduce false positives and false negatives, and enhance the logic that underlies algorithms and indicators for detecting irregularities.
Strengthen analysis of unstructured and semi-structured data—ASF officials highlighted the need for improving the capacity for managing and analysing unstructured data as one of their top priorities in the coming years. This is in part due to efforts of the AEGF and the AECF to enhance the ASF’s architecture, methodologies and tools (e.g. machine learning) to better analyse “big data,” which consists of high volumes of unstructured and semi-structured data. The ASF’s Digital Mailbox also has the potential to create more unstructured and semi-structured data for the ASF to process. Text mining and other analytic techniques can be helpful to ensure that auditors are not overwhelmed by such data. The ASF can also build on existing capacities for carrying out network analyses, particularly to support the detection of fraud and corruption risks in infrastructure development and public procurement. The diversity of techniques and their underlying tools highlight the need for the ASF to consider strategically how to approach analysis of different types of unstructured and semi-structured data in the future. Defining the process of integrating findings and results from this analysis into the ASF’s traditional systems and communication mechanisms, as well as into possible dashboards for auditors to reference, is one critical consideration to ensure that auditors can digest and use the results.
Promote digital skills and ethical use of data through trainings—Introducing new systems, tools or dashboards is necessary, but insufficient, for the ASF to keep pace with the digital change in government and society. The ASF could further develop the skills, motivation and interest in analytic approaches to sustain future analytics initiatives, although it has trained a small number of auditors on big data. This could include, but is not limited to, promoting data literacy as well as developing digital skills, defined as the broader range of abilities to use digital devices, communication applications, and networks to access and manage information. The distinction between data literacy and having digital skills reflects the notion that auditors have different specialities and require varying levels of specialisation when it comes to managing and using data; however, all auditors can benefit from having an understanding and fluency with a range of digital tools and technologies that are critical for the modern auditing profession. Developing digital skills also involves training for auditors to ensure they lead by example as stewards of responsible, accountable and ethical use of data. This would be consistent with ASF’s initiative to develop a Policy on Institutional Integrity (Política de Integridad Institucional).
Create room for experimentation and small wins—An openness to experimentation is a consistent theme across SAIs that have developed successful analytics initiatives. Even when other aspects of the ASF’s work and culture remains risk averse, experimentation creates opportunities for both small wins and small losses. This means a SAI can pilot new methodologies, tools and data sources in a controlled and cost-efficient way before deciding whether to scale up or avoid developing further. Establishing an “Innovation Lab” is one way that SAIs are doing this, which would institutionalise the capacity for experimentation and set the tone for innovation as a strategic objective. However, this is not the only approach for the ASF to consider. The ASF’s existing teams demonstrate a high level of ambition to innovate, and there are temporary models, such as project-based initiatives, to tap into the skills and innovative energy of auditors and staff.
References
[1] ACFE (2020), Report to the Nations: Study on Global Occupational Fraud and Abuse, https://www.acfe.com/press-release.aspx?id=4295010563#:~:text=In%20the%20ACFE%27s%202020%20Report,loss%20of%20%248%2C300%20a%20month.
[2] ASF (2021), Avances en la implementación de nuevas herramientas tecnológicas aplicadas a la fiscalización superior.
[28] Auditor General of Wales (2020), Cutting Edge Audit Office, https://www.eurorai.org/public/Attachment/2020/6/Sevilla-presentationSLISLE.pdf.
[15] Auditor General South Africa (2020), Consolidated General Report on National and Provincial Audit Outcomes, https://www.agsa.co.za/Portals/0/Reports/PFMA/201920/PFMA%202019-20%20Report%20-%20signed.pdf.
[5] Baesens, B., V. Van Vlasselaer and W. Verbeke (2015), “Using Descriptive, Predictive, and Social Network Technique: A guide to Data Science for Fraud Detection”, John Wiley & Sons, Inc..
[16] Canadian Audit and Accountability Foundation (2020), Better Integrating Root Cause Analysis into Public sector performance auditing, https://www.caaf-fcar.ca/images/pdfs/research-publications/RootCauseAnalysisEN.pdf.
[6] Commonwealth Fraud Prevention Centre, Government of Australia (2020), Data Sharing Pilots, Leading Practice Guide, https://www.counterfraud.gov.au/sites/default/files/2021-02/data-sharing-pilots-leading-practice-guide.pdf (accessed on 3 August 2021).
[29] Dickson, M. and P. Asagba (2020), “The Semi-Structured Data Model and Implementation Issues for Semi-Structured Data”, International Journal of Innovation and Sustainability, Vol. 3/2020, pp. 47-51, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3726802.
[20] EUROSAI (2021), GERMANY: Using text mining methods to analyse public communication of our auditees, https://eurosai-it.org/news/newsletter/1-2021/updates-from-itwg-members/germnay-using-text-mining-methods-to-analyse-public-communication-of-our-auditees.
[18] EUROSAI (2019), Follow-up of the Implementation of records, https://www.eurosai.org/handle404?exporturi=/export/sites/eurosai/.content/documents/2021-02-03-Final-report-for-EUROSAI.pdf.
[34] Government of Mexico (2021), “Diario Oficial de la Federación”, in Agreement amending, adding and repealing several provisions of the Internal Regulations of the Superior Audit Office of the Federation.
[30] Government of New Zealand (2021), Operational data governance, https://www.data.govt.nz/toolkit/data-governance/odgf/.
[14] Government of New Zealand (2020), The Data Capability Framework (DCF) Guide, https://www.data.govt.nz/assets/Uploads/Training/Data-Capability-Framework/Data-Capability-Framework-Guide.pdf.
[8] House of Commons, United Kingdom (2017), Value for Money Study, Delivery of Benefits From the NAO’s IT Enabled Change Program, https://www.parliament.uk/globalassets/documents/public-accounts-commission/15-NAO-VFM-report-on-IT-enabled-change-programme.pdf.
[13] INTOSAI (2020), Strategic Management Handbook for Supreme Audit Institutions, https://www.idi.no/elibrary/well-governed-sais/strategy-performance-measurement-reporting/1139-sai-strategic-management-handbook-version-1/file.
[31] INTOSAI (2019), INTOSAI-P - 12 - The Value and Benefits of Supreme Audit Institutions – making a difference to the lives of citizens, https://www.issai.org/pronouncements/intosai-p-12-the-value-and-benefits-of-supreme-audit-institutions-making-a-difference-to-the-lives-of-citizens/.
[32] INTOSAI (2015), GUID - 9030 - Good Practices Related to SAI Independence, https://www.issai.org/pronouncements/guid-9030-good-practices-related-to-sai-independence/.
[23] Kärki, M. and M. Saarteinen (2020), National Audit Office of Finland: Strategy and practice when using data in auditing and monitoring, Presentation at OECD Workshop (18 November 2020).
[33] National Banking and Stock Commission, Government of Mexico (2016), Development Bank (BD), https://www.gob.mx/cnbv/acciones-y-programas/banca-de-desarrollo-bd.
[27] OAGN Innovation Lab (2020), Data Science & Machine Learning, https://www.intosaipas.org/wp-content/uploads/2020/02/Agenda-item-1H_2019_Data-Science-PASmeeting-1.pdf (accessed on 3 September 2021).
[25] OECD (2020), “Building digital workforce capacity and skills for data-intensive science”, OECD Science, Technology and Industry Policy Papers, No. 90, OECD Publishing, Paris, https://doi.org/10.1787/e08aa3bb-en.
[9] OECD (2020), Digital Government in Mexico: Sustainable and Inclusive Transformation, OECD Digital Government Studies, OECD Publishing, Paris, https://doi.org/10.1787/6db24495-en.
[24] OECD (2020), Good Practice Principles for Data Ethics in the Public Sector, OECD, Paris, https://www.oecd.org/gov/digital-government/good-practice-principles-for-data-ethics-in-the-public-sector.pdf.
[21] OECD (2019), Analytics for Integrity: Data-Driven Approaches for Enhancing Corruption and Fraud Risk Assessments, OECD, Paris, https://www.oecd.org/gov/ethics/analytics-for-integrity.pdf.
[26] Office of the Auditor General of Norway (2021), Introduction to Text Mining for Auditors, https://idi.no/elibrary/relevant-sais/sai-innovations/innovative-sais-going-f-a-r/1285-marketplace-introduction-to-text-mining-for-auditors-office-of-the-auditor-general-of-norway/file.
[11] Office of the Comptroller and Auditor General of India (2017), Guidelines on Data Analogue, https://cag.gov.in/uploads/guidelines/Guidelines-on-Data-Analytics-book-05de4f7fd52e565-67820093.pdf.
[22] Onwujekwe, G., N. Ngwum and K. Osei-Bryson (2020), A Framework for Capturing and Analyzing Unstructured and Semi-structured Data for a Knowledge Management System.
[3] Special Audit of Financial Compliance, ASF (2021), Sistema de Inteligencia para la Fiscalización Superior de la AECF, [PowerPoint Slides].
[10] Stockpoll, B. (2021), Making the business case for a chief data officer, https://mitsloan.mit.edu/ideas-made-to-matter/making-business-case-a-chief-data-officer.
[4] Strimling, A. (2006), “Stepping Out of the Tracks: Cooperation Between Official Diplomats and Private Facilitators”, International Negotiation, Vol. 2006/11, pp. 91-127, https://brill.com/view/journals/iner/11/1/article-p91_5.xml?language=en.
[19] Talib, R. et al. (2016), “Text Mining: Techniques, Applications and Issues”, International Journal of Advanced Computer Science and Applications, Vol. 7/11, pp. 414-418, https://thesai.org/Downloads/Volume7No11/Paper_53-Text_Mining_Techniques_Applications_and_Issues.pdf.
[12] The European Court of Auditors (2019), “Developing data services for audit”, Presentation of Magdalena Cordero, Director of Information, Workplace and Innovation, https://ecademy.eca.europa.eu/pluginfile.php/260/mod_resource/content/1/Developing%20data%20services%20for%20auditors%20%E2%80%93%20the%20ECA%20experience.pdf (accessed on 3 August 20021).
[7] UK National Audit Office (2019), Transparency Report, https://www.nao.org.uk/wp-content/uploads/2020/07/National-Audit-Office-Transparency-Report-2019-20.pdf.
[17] UK National Audit Office (2018), Data Analytics at the National Audit Office (UK), https://www.intosaipas.org/wp-content/uploads/2020/02/Agenda-item_2A_Andy-Fisher_Data-Analytics-at-the-UK-NAO.pdf (accessed on 3 August 2021).
Notes
← 1. Development banking institutions are entities of the Federal Public Administration, with their own legal personality and assets, constituted as national credit companies. Their main objective is to facilitate access to savings and financing for individuals and companies, as well as to provide them with technical assistance and training (National Banking and Stock Commission, Government of Mexico, 2016[33]).
← 2. See articles 9, 17, section XI, and 23 of the Law on Auditing and Accountability of the Federation (Ley de Fiscalización y Rendición de Cuentas de la Federación, LFRCF), and 5 section XI, of the Internal Regulations of the Superior Audit Office of the Federation (Reglamento Interior de la Auditoría Superior de la Federación).
← 3. As introduced in Chapter 1, this references the New Zealand government’s approach to data governance which makes a distinction between political governance and operational data governance. The latter is associated with data activities and needs at the operational level of an organisation (Government of New Zealand, 2021[30]).
← 4. See, for instance, https://idi.no/elibrary/professional-sais/issai-implementation-handbooks/handbooks-english.
← 5. The AESII was recently restructured in the amendment to the ASF’s internal regulations in August 2021 (Government of Mexico, 2021[34]).
← 6. For instance, see INTOSAI-P 12 The Value and Benefits of SAIs – making a difference to the lives of citizens (INTOSAI, 2019[31]) and INTOSAI GUID 9030: Good Practices Related to SAI Independence (INTOSAI, 2015[32]).
← 7. Semi-structured data has defining or consistent characteristics, but it does not have the structure of a relational database. For instance, emails have unstructured content with a predictable structure with common fields like sender, recipient, subject and time stamps. Much of what people classify as unstructured data is actually semi-structured due to classifying characteristics of the data (Dickson and Asagba, 2020[29]).
← 8. As noted, the AESII underwent a restructuring following the amendment to the ASF’s internal regulation, which occurred after the completion of the analysis for this report.