Strengthening Oversight of the Court of Auditors for Effective Public Procurement in Portugal

Digital Transformation and Data-driven Risk Assessments

Report

OECD Public Governance Reviews

28 June 2024

2. Portuguese Court of Auditors: Innovating with data-driven assessments of public procurement risks

2.1. Introduction

Supreme Audit Institutions (SAIs) are standard bearers of transparency, accountability and public trust. As such, for the Court of Auditors of Portugal (Tribunal de Contas, TdC), keeping pace with technological advancements is both a strategic objective and a duty to citizens. The transformative potential of data and new technology-driven methodologies for conducting audits have brought about an imperative for the TdC to advance an ambitious digital transformation agenda.

This chapter explores the TdC’s digital transformation journey in a number of strategic and operational areas that are critical for the TdC’s data-driven assessments of public procurement risks. This includes a discussion about the current state and areas for improvement concerning the TdC’s 1) strategy and organisation; 2) people and culture; 3) technology and process; and 4) environment and partnerships. The following sections use these domains as a framework for assessing the TdC maturity to undertake data-driven risk assessments of public procurement. Specifically, the chapter focuses on the following issues:

Strategy and organisation. The section provides an overview of the TdC’s Digital Strategy and the extensive efforts to develop it. It also delves into the TdC’s organisational structure for strategy implementation and highlights the need for greater coherence among the TdC’s 12 digital initiatives. The chapter also highlights the importance of monitoring and evaluating these initiatives, including suggestions for the TdC when monitoring and evaluating its data-driven risk assessments in public procurement.
People and culture. This section underscores the necessity of a strategic approach to change management, crucial for both the broader Digital Strategy and the success of the data-driven risk assessment initiative. It examines challenges and priorities in enhancing data literacy and digital skills, particularly for assessing risks in public procurement. The chapter also emphasises the need for auditors to be knowledgeable about data ethics and privacy.
Technology and process. This section addresses various IT, digital tool, and data integration challenges faced by the TdC. These challenges have wider implications for the TdC’s Digital Strategy and overall audit work, but the chapter highlights specific issues for the TdC to address in terms of data integration, interoperability and tools for assessing risks in public procurement.
Environment and partnerships. This section highlights several opportunities for the TdC to enhance its digital maturity, with a specific focus on the relevance and sustainability of its initiative for data-driven risk assessments. These include the formalisation of data-sharing protocols and collaboration with data owners to improve the TdC’s training programmes.

2.2. A framework for understanding the Court of Auditor’s ambitions for digital transformation

Digitisation, digitalisation and digital transformation are concepts that are often used interchangeably. Nonetheless, the distinctions between these concepts are critical for understanding the TdC’s priorities for data-driven risk assessments, key challenges and its future priorities. Definitions of these concepts vary. For instance, many definitions of digital transformation focus on the economic and societal benefits of digitisation and digitalisation. At the institutional level, digital transformation strategy involves organisational change, innovation and addressing cross-cutting management or operational challenges concerning the use of digital technologies and data (Bumann, 2019[52]; Otia, 2022[53]). Digitisation and digitalisation have a more narrow focus, as illustrated in Figure ‎2.1.

Figure ‎2.1. Progression towards digital transformation

The TdC’s initiative to strengthen its assessment of risks in public procurement through better use of data and advanced analytics is primarily an exercise in digitalisation. But as described in later sections, several parallel initiatives complement and feed into this effort, including changes to procedures and IT systems, which will have a disruptive effect on traditional approaches. This includes changes to the TdC’s data management and governance, as well as changes in work processes for auditors given the introduction of new tools. In general, the dimensions commonly seen in frameworks for understanding what drives digital transformation are similar to those for digitalisation and specifically, assessing “digital maturity” for leveraging data and advanced analytics on a project level.

Figure ‎2.2. Key dimensions of digital transformation and maturity

1. Strategy and organisation. This dimension involves leadership's direction for digital transformation, targeting enhanced use of technology and data. The digital strategy, whether independent or merged with organisational plans, should align with broader organisational priorities and IT strategies (Bumann, 2019[52]). It is critical that organisations clearly define roles and responsibilities for initiatives that advance the digital transformation strategy. Policies, procedures and activities surrounding data management and data governance are critical elements of this dimension, including those that focus on data privacy, quality, and consistency.

2. People and culture. The success of digital maturity in organisations relies heavily on the expertise, skills, and commitment of employees. This includes digital and data literacy, and sometimes advanced programming skills. Equally important is understanding the ethical use of data, along with sector-specific knowledge and legal expertise for managing data-related issues like privacy and security (OECD, 2020[55]). Beyond skills, a digital-ready culture involves leadership and employees uniting around digital objectives, which may involve policies supporting technological experimentation or training to enhance digital skills (OECD, 2022[56]).

3. Technology and processes: Technology, including IT systems, tools, and software, is crucial for digital maturity, but should not be the main focus. The broader vision for digital transformation should guide technological choices, emphasising the need to tailor technology to specific needs. The term "state-of-the-art" is contextual. Digital services, IT systems and analytical tools differ in their complexity, resource needs, functionality and alignment with various organisational goals. Therefore, a pragmatic approach with cost-benefit analysis is essential for making responsible technological investment decisions.

4. Environment and partnerships. National laws and directives significantly impact institutional digital transformation. A country's legal and policy framework sets the basis for digital governance and data management, influencing the success of digital projects by either limiting or advancing technology adoption. Within organisations, robust data management and governance is key for data sharing and accessibility, as noted. However, the essence of data sharing extends beyond just infrastructure or processes; fostering collaborative relationships between entities is crucial for achieving digital maturity.

This chapter does not aim to assess the TdC’s broader digital transformation strategy or its implementation. The TdC is still in the first year of its 3-year Digital Strategy, and at the time of writing it is too early to evaluate implementation challenges (and it is beyond the scope of the project). Moreover, to its credit, the TdC itself has conducted several self-assessments recently to develop the current strategy, as described below. Nonetheless, these dimensions and the key practices associated with them (see Annex II) offer a useful framework for understanding some of the opportunities and challenges for the TdC’s initiative to enhance data-driven risk assessments in public procurement as part of its broader pursuit of digital transformation.

2.3. Strategy and organisation

This dimension focuses on leadership's guidance for digital transformation, emphasising the integration of technology and data into organisational strategy. It's essential to define roles and responsibilities clearly and prioritise data management policies, including privacy, quality, and consistency, as key elements of this process. This section provides an overview of the TdC’s Digital strategy and the extensive work carried out to develop it. In addition, it provides insights concerning the TdC’s organisational structure for implementing the strategy and raises the need to strengthen coherence between the TdC’s 12 digital initiatives. The section also explores the importance of monitoring and evaluating the TdC’s digital initiatives, with suggestions for its data-driven risk assessments in public procurement.

2.3.1. The development of the Court of Auditor’s Digital Strategy benefited from extensive self-assessments and internal collaboration

The TdC’s 2023-2025 Strategic Plan for Digital Transformation (i.e., the “Digital Strategy”) outlines its commitment to embracing the challenges posed by the accelerated pace of digital transformation. The TdC’s Digital Strategy builds on the broader Strategic Plan and encompasses the Strategic Development Plan for Information Systems (PDESI), which the TdC developed previously and aims to align information systems with the TdC’s organisational strategy. At nearly 50 pages, the Digital Strategy outlines the Court’s vision, priorities and activities, supported by in-depth analyses of challenges. It emphasises digital transformation as the goal—the adoption of new technologies and approaches that require fundamental changes to how the TdC operates. The Digital Strategy includes initiatives in the PDESI that were ongoing at the time of its development, and it focuses on the following core objectives (Tribunal de Contas, 2023[57]):

1. Promote a data-driven TdC with more effective and efficient use of digital technologies to achieve its strategic objectives.

2. Undertake the digital transformation of the TdC, prioritising innovation and rapid adaptation to emerging risks, and promoting greater efficiency and effectiveness of its activity.

3. Adapt activities to new technologies across all dimensions of the TdC’s work.

4. Strengthen the internal governance and management of the TdC, aiming to reinforce decisions and increase the institution's efficiency by improving digital skills.

The TdC arrived at these objectives through an internal collaborative process, led by the Auditing Standards Committee and by the IT Committee with technical support from the Centre of Innovation, Technology and Methodologies (CITM) (Tribunal de Contas, 2021[58]), as well as a series of assessments. The assessments included an IT Audit Self-Assessment (ITSA), sponsored by the European Organization of Supreme Audit Institutions (EUROSAI) and involving Spain’s Court of Auditors as independent facilitators of the review. The shaping of the TdC’s Digital Strategy also relied on internal self-assessments, such as an analysis of strengths, weaknesses, opportunities, and threats (i.e., a SWOT analysis), as well as an assessment of its digital maturity using a questionnaire of a private company (i.e., Gartner’s Digital Government Urgency, Maturity and Maturity Assessment). The TdC also engaged an external consultant in 2021 to independently assess the IT architecture and infrastructures in specific Chambers of the TdC, focusing on functionalities of the document management (GDOC) and entity management (GENT) systems, detailed below.

The shaping of the TdC’s Digital Strategy also relied on an internal analysis involving TdC employees that focused on strengths, weaknesses, opportunities and threats (i.e., a SWOT analysis). The SWOT analysis is depicted in the figure below and further demonstrates the extensive diagnostic analysis that contributed to the TdC’s current Digital Strategy and was a driver of the initiative to strengthen its data-driven analysis of irregularities in public procurement.

Figure ‎2.3. The TdC’s SWOT analysis for digital transformation

In addition to the SWOT analysis, in August 2022, the TdC conducted another self-assessment using a tool developed by a private company to assess its digital maturity and maturity, and the urgency of specific needs. This tool offered further insights for the TdC to identify key areas for improvement and set priorities in relation to its digital transformation. The assessment is organised into seven key areas: vision and strategy, delivery and quality of services, the organisation, the organisation and its context, digital projects and investments, data and analytics, and the functions of the Chief Information Officer. The self-assessment includes a maturity model, which classified the TdC as a “cautious mover organisation, acting under low urgency and maturity.” The TdC scored the highest on aspects related to leadership and focusing on value, while other elements like metrics, its data ecosystem, platforms, service model and technology had lower levels of maturity.

Figure ‎2.4. Matrix for digital transformation based on urgency and readiness

These assessments led to the launching of twelve different initiatives, including the TdC’s project to advance data-driven risk assessments in public procurement. A key recommendation from the TdC’s internal assessments was that it could identify projects to achieve “quick wins” that demonstrate positive improvements to leadership. In this context, the TdC’s initiative to strengthen its data-driven risk assessments in public procurement is a follow-up to this recommendation and is an approach that aligns with leading practices. Overall, the TdC’s approach to developing its Digital Strategy checks the box on many of the leading practices for strategy development and aligns with guidance for SAIs that focus on this process.1 Nonetheless, it could take additional steps to ensure the coherence between these initiatives, as discussed in the following section.

2.3.2. The TdC established a central entity to lead its digital transformation, but more efforts are needed to ensure coherence across digital initiatives

The Department for Innovation, Technology and Methodologies (CITM) is a central entity the TdC established to lead the development of the Digital Strategy. Prior to the creation of the CITM, the TdC had a longstanding IT function, the Directorate of Information Systems and Technology (DSTI). Together, the CITM and DSTI are driving the TdC’s spearheading the implementation of the Digital Strategy and its initiatives, including enhancing data-driven risk assessments. The specific aim of the CITM is to “monitor, adapt, promote and support” the use of technology, methodologies and tools for auditors” (Tribunal de Contas, 2021[59]). The CITM (with the DSTI) has a formal mandate to drive technological change, which comes with responsibilities that span the aforementioned dimensions of the digital maturity framework. The DSTI is responsible for aligning IT processes with the TdC’s strategy and ensuring the integration of business areas in applications. It also maintains the global data model, reduces information redundancy, and develops, acquires and maintains integrated applications for technical and administrative functions.

The CITM is an operational unit within the Directorate General of the TdC and it has two main areas of focus, as shown in Figure ‎2.5. This model is similar to other SAIs. For instance, the Centre for Data Management and Analytics Office within the Comptroller and Auditor General of India provides guidance to field offices on analytics and pioneers research and development on future directions for using data and analytics. In some SAIs, like those in the U.S., U.K., and Norway, a combination of Innovation Labs and methodological specialists fulfils these roles.

Defining roles and responsibilities internally, including the designation of an entity to implement the Digital Strategy with an organisation-wide mandate and access to leadership, is a key practice for driving digitalisation and digital transformation. For instance, INTOSAI’s guide for SAIs on Managing Information Communications Technology (ICT) highlights both the critical importance of having an ICT function as well as the consideration for its level of visibility within the SAI and whether it has senior representation. The same could be said of other functions within an SAI that serve cross-cutting roles, such as Innovation Labs or teams that offer support in advanced analytics or statistical methodologies, such as the CITM (INTOSAI, 2015[60]). Within the TdC, this role is critical given the breadth of initiatives that fall under the Digital Strategy. These initiatives have action plans, but interviews with the TdC official demonstrated that efforts remain siloed and there is a risk of incoherence and inefficiencies when implementing the broader strategy.

Having established the CITM, and together with the DSTI, the TdC has in place the governance structure to drive greater coherence between digital initiatives. The CITM and the DSTI can continue to facilitate regular communication, decision-making and alignment of goals across the 12 digital initiatives. The sections below that emphasise monitoring and evaluation, change management and strengthening of digital skills can also serve as mechanisms for breaking down siloes across digital initiatives and ensuring that different digital initiatives do not just coexist but actively complement and enhance each other.

2.3.3. The TdC could strengthen the focus on monitoring and evaluation to ensure successful outcomes of its data-driven risk assessment for public procurement

In 2022, EUROSAI evaluated 20 European SAIs to understand how they use technology and identify challenges. The survey results revealed a unanimous belief among SAIs that digital transformation enhances the effectiveness and efficiency of audits, and they identified the adoption of digital technologies as a crucial strategy for improving SAI performance. However, the survey also highlighted a monitoring shortfall, with only 30% of SAIs evaluating the impact of digital technologies on audit outputs (EUROSAI, 2023[61]). Continuous monitoring and evaluation of the performance of digital initiatives, based on objectives and metrics, are critical for understanding the effectiveness of activities and adapting them, as needed.

In its handbook, the Strategic Management Handbook for Supreme Audit Institutions, the INTOSAI Development Initiatives outlines key learning points for monitoring an SAI’s strategy. The points reflect many of the basics of performance monitoring, including developing a monitoring framework that not only focuses on the broader strategy, but also accounts for the operational level and specific initiatives. This involves developing a framework that has:

Indicators: A quantitative or qualitative measure that shows the level of achievement of envisaged change, most notably at the outcome and output level and for some crucial capacities.
Baselines: The status of the indicator at the beginning of the strategic period.
Milestones and Targets: On what level the indicator should be, or to what extent activities should be implemented at a given time (INTOSAI Development Initiative, 2020[62]).

The TdC’s Digital Strategy has a section on the implementation and monitoring of the strategy that recognises the need for monitoring the strategy, including general references to the use of indicators for assessing digital initiatives. Future iterations of the Digital Strategy could develop the indicators further. At the onset of the project with the OECD, the TdC defined several indicators to measure the impact of the initiative. These can serve as a starting point. Table ‎2.1 explores possible indicators to build on these with a more operational context in mind, taking inspiration from indicators in the IDI’s Handbook for monitoring the performance of SAIs, the quality assurance models of the UK National Audit Office (NAO),2 the US Government Accountability Office’s (GAO) AI Accountability Framework (see Box ‎2.1), and the cost-benefit analysis methodology of the Comptroller General of Peru (CGR).

The indicators in the table reflect the TdC’s goals to move toward a more automated, AI-driven form of risk assessment. They focus on the use of such tools in the context of public procurement but could be applied to other areas as well. There are two types of indicators presented in the table. The first four focus on the effectiveness of the risk assessment tool itself, while the last four focus on data literacy and capacity more broadly. It is also important that a target value is established for each indicator to maximise their effectiveness. This table is not meant to be an exhaustive list but instead to provide a starting point for TdC to establish a framework for monitoring and evaluating its digital initiative for data-driven risk assessments in public procurement.

Intended results	Indicators	Observations and data sources	Notes
Increased reliability in detection of risks in public procurement.	Rate of increase in detection of risks in public procurement procedures.	Comparison of rate of identification of risks between different iterations of the risk assessment methodology, ideally applied to the same systems, audit areas and time periods.	Since there is currently no risk assessment tool in place in this area, there will be no baseline for comparison. However, as methodologies improve over time they should be compared to previous methodologies. This reduces the risk of confounding variables leading to flawed conclusions about a model’s effectiveness. The same applies to subsequent indicators referring to a “new methodology.”
Increased reliability in detection of risks in public procurement.	Rate of decrease in false positives and false negatives.	Rate of false positives and false negatives under new methodology compared to rates of false positives and false negatives under old methodology, with both applied to the same systems over the same time period.	This indicator is primarily relevant if the TdC employs a classification model, which according to the TdC’s internal processes, would itself require verification by auditors to ensure risks the accuracy of classifications. One useful tool for model optimisation to reduce false positives is the receiver operating characteristic (ROC) curve, which plots the true positive and false positive rates of models when they rely on different thresholds to make a binary choice. The model that most consistently maximises true positive rates while minimising false positive rates should be used.
Increased efficiency in public procurement risk assessment.	Rate of decrease in average time to conduct the data-driven risk assessment.	Comparison of the average risk assessment time between different iterations of the data-driven risk assessment methodology, ideally applied to the same systems, audit areas and time periods.	This indicator could be inclusive of the entire risk assessment process (i.e., not solely data processing times), or assessed within the broader context of planning and executing an audit engagement, with the expectation that a more efficient risk assessment will lead to a more efficient process overall.
Increased efficiency in public procurement risk assessment.	Percentage of risks detected via data-driven risk assessment tool.	Number of risks identified with a data-driven risk assessment tool out of total number of risks identified conceptually.	Risks identified during a conceptual mapping exercise are often greater than those that can be assessed in practice due to various challenges, such as data access issues or poor data quality.
Increased efficiency in public procurement risk assessment	Return on investment for implementing data-driven risk assessment	Estimated decrease in state public procurement losses as a result of new tool minus cost to TdC of developing and implementing tool (cost-benefit analysis)	The rate of increase in risk identification listed previously can be used to estimate the financial benefit of the new tool, while costs could include those for hardware, software, hiring staff, training staff, the opportunity cost of time staff is not spending on other tasks, etc. This indicator could also be expanded to include non-financial benefits (e.g., enhanced reputation or improved deterrence) and costs (e.g., the potential of incentivising more sophisticated fraud schemes).
More effective direct use of public procurement data and other data sources.	Percentage of independent datasets in use at TdC that are subject to data integration in a documented, engineered and governed process.	Percentage of available datasets that are relevant for data-driven risk assessments in public procurement that are integrated into TdC risk assessment systems before and after intervention.	SAIs often rely on open source data to conduct risk assessments of public procurement, in part, due to de jure or de facto data access issues. This indicator refers to access of data directly at the source, which includes the central public procurement body or platform, as well as other data sources that are useful for risk identification.
Increased data literacy of TdC employees.	Percentage of employees having undergone data literacy training.	Number of employees having undergone training as a percentage of number of employees responsible for public procurement risk assessment at the TdC.	Additional indicators that focus on behaviour change and outcomes, as opposed to more output-based indicators of whether employees completed trainings, could complement this indicator.
Increased quality of automated analytical techniques in public procurement risk assessments.	Rate of data drift in AI machine learning models; rate of bias in AI machine learning models.	Average standardised drift as a measure of central tendency across all machine learning models over a period of time; the average standardised difference between all machine learning model’s predictions and actual values.	Model drift refers to the decline in a model’s predictive ability over time. This can occur because of changes in data quality or structure and, if severe enough, requires models to be retrained.

Source: Author’s elaboration.

One challenge facing the TdC is the creation of indicators that go beyond monitoring activities and processes to monitoring outcomes. This typical measurement challenge is amplified by the inherent complexity of measuring the effectiveness and efficiency of data-driven risk assessments for irregularities. The indicators in the table above include some outcome-level indicators that can inform the TdC’s development of a more robust monitoring framework for the initiatives, which could focus not only on the impact of the risk assessment process itself but also on the use of AI and advanced analytics. GAO’s AI Accountability Framework offers some inspiration for the latter.

The GAO AI Accountability Framework comprises four key pillars: governance, data, performance, and monitoring. Its aim is to guide managers on responsible AI usage and to foster accountability in government AI initiatives. It delineates key practices across its pillars, focusing on the AI system's lifecycle from design to continuous monitoring. The "Data" pillar, for instance, emphasises documenting and continually monitoring data sources and preparations for data models. The Framework not only facilitates sound AI management but also can provide the TdC with insights for monitoring its own initiatives using AI or advanced analytics. Box ‎2.1 elaborates on this framework and key practices for monitoring and evaluating the performance of AI systems. The GAO’s is not the only example. For instance, Norway’s Office of the Auditor General (OAG) has integrated the auditing of AI as part of its suite of performance audits, and its current 2018-2024 Strategic Plan notes that “problem-solving will become more automated, and the use of [AI] will gradually take over tasks in both the public administration and the OAG.”3 Similarly, at the central level, the TdC can reference Canada’s Algorithmic Impact Assessment tool4 to inform its own assessment of AI performance and risks.

Box ‎2.1. The GAO’s Artificial Intelligence Accountability Framework

In order to ensure effective use of AI, GAO has developed a framework to evaluate the performance of AI systems to make sure they deliver value and remain fit for purpose over time. Pillars 3 and 4 of the framework on performance and monitoring, respectively, provide examples of best practices for monitoring and evaluation of data-driven tools (see Figure ‎2.6 below). These procedures for monitoring and evaluation are not only useful in guiding entities in their use of AI, but they could also be applied to other data-driven tools and systems, including those that do not employ AI.

Figure ‎2.6. GAO’s Artificial Intelligence (AI) Accountability Framework

Regarding performance, at both the component level and the system level, AI models should be documented and assessed against predetermined performance metrics that are precise, consistent, and reproducible. Documentation should aim to address the following groups of questions:

How are components and models solving defined problems? What is their intended use?
How are the specifications and parameters are selected, evaluated, and optimised?
How suitable are components and models to available data and operating conditions?
How are components and models tested and what are the results?
What ethical considerations exist? What biases and unintended consequences have been identified?
What degree of human supervision is required and how was this determined?

When it comes to model evaluation, the selected performance metrics should be accurate and useful, and the justification for their selection and the person(s) responsible for their development should also be documented.

Once AI systems are put in place, a plan for continuous or routine monitoring should be developed. This helps ensure that AI models remain reliable and relevant. The plan should define acceptable levels of data and model drift that are based on a risk assessment and require documentation of monitoring activities and any corrective actions taken. As part of the monitoring of AI systems, their continued utility and any potential opportunities for scaling should also be assessed. Any decisions to retire or scale models or systems should be based on predefined performance metrics, and any updates that take place and their impact should be documented. Finally, throughout the process of monitoring and evaluation, it is important that entities keep in mind AI systems’ consistency with their objectives and values in order to foster and maintain public trust.

Source: (U.S. Government Accountability Office, 2021[63]).

Another important consideration when investing in the development of a digital tool is the return on investment. Data-driven risk assessment models may improve the TdC’s ability to identify and respond to risks in public procurement, but this is only worthwhile if these benefits outweigh the costs of creating and perfecting such a tool. The CGR of Peru developed a cost-benefit analysis (CBA) methodology for evaluating its investments in concurrent control of infrastructure that could be informative for the TdC’s own assessment of its digital initiatives.5 The methodology focuses on calculating the net potential benefit of an investment by subtracting the potential financial and non-financial costs from the potential benefits in terms of potential economic damage avoided. Employing a CBA methodology could help the TdC identify whether an investment in data-driven risk assessment is worthwhile, comparing the investment in developing and maintaining an ongoing capacity for data-driven risk assessments with the benefits derived from more targeted and efficient audits.

2.4. People and culture

Organisational digital maturity hinges on employee expertise, skills, and commitment, encompassing digital literacy, advanced programming, and understanding ethical data use, along with sector-specific and legal knowledge for data privacy and security. A digital-ready culture also requires leadership and employees to align around digital goals, supported by policies promoting technological experimentation and skill-enhancement training. This section highlights the need for taking a strategic approach to change management, which has implications for both its broader Digital Strategy and the success of its data-driven risk assessment initiative. The section explores challenges and priorities for strengthening data literacy and digital skills, especially in the context of assessing risks in public procurement, as well as the need for auditors to have the knowledge and awareness of core issues concerning data ethics and privacy.

2.4.1. Emphasising change management as a critical element for the success of the TdC’s digital initiatives and broader digital transformation

Of the key practices described in the framework above, developing and implementing a change management and continuous learning plan is one of the most critical in the TdC’s context. Interviews with TdC officials, responses to the OECD questionnaire and the TdC’s own self-assessments highlight a reoccurring theme: existing siloes between Departments and audit teams threaten the success of its Digital Strategy. Institutional siloes also pose challenges at the project level, including the TdC’s initiative to use data and advanced analytics for assessing public procurement risks.

Strengthening its strategy and approach to change management can help address these siloes and increase the chance of success of the TdC’s digital initiatives and, therefore, its broader digital transformation strategy. Change management encompasses both initiating the necessary shifts within an organisation and adeptly navigating the intricacies of change by planning, executing, and sustaining these transformations (European Court of Auditors/U.K National Audit Office/OECD SIGMA, 2007[64]). While technology and strategies are important, the people involved are the backbone of any transformation. A change management plan can act as a roadmap, guiding the TdC through the complexities and challenges that arise during its transformative journey.

To sharpen its approach to change management, the TdC can take inspiration from one of the seminal frameworks in this field: the eight Steps to Accelerate Change.6 This framework represents “steps” that are not necessarily linear and can be carried out simultaneously and iteratively, as the TdC’s own experience demonstrates. The steps include:

1. Create a Sense of Urgency: Recognise and communicate the need for change. Make the team realise the importance of acting immediately.

2. Build a Guiding Coalition: Form a group with the power, influence, and skills needed to lead and support the change effort.

3. Form a Strategic Vision and Initiatives: Define a clear vision and align key initiatives with that vision to guide the change process.

4. Enlist a Volunteer Army: Mobilise a broad base of people who are committed to driving change and are eager to participate.

5. Enable Action by Removing Barriers: Identify and remove obstacles or structures that undermine the change vision. Streamline processes and empower people to act.

6. Generate Short-Term Wins: Plan for visible performance improvements. Recognise and celebrate these victories to build momentum.

7. Sustain Acceleration: Use increased credibility from early wins to initiate more and bigger change projects. Keep the momentum going.

8. Institute Change: Ensure the change is deeply rooted in the organisational culture so that the new approaches continue to drive success (Kotter, 2018[65]).

From the perspective of implementing its Digital Strategy, it is notable that CITM has already taken several of these steps, such as formulating a strategic vision and initiatives that can be short-term wins, which includes its data-driven risk assessment in public procurement, according to TdC officials. However, the principles within Kotter's framework can be relevant for specific projects, particularly when the impact of the project crosses teams or requires a change in behaviours and habits. The TdC’s project to strengthen its data-driven risk assessments has these characteristics. Table ‎2.2 explores these steps further with this initiative as a focus, including actions the TdC is already taking as well as additional steps it can consider.

Steps	Actions
1. Create a Sense of Urgency	TdC’s current action: developed a Digital Strategy that promotes the initiative; leadership buy-in and implementation support are strong, including publicising the TdC’s digital transformation initiatives and advertising its efforts to produce a data-driven risk assessment for public procurement. Additional actions to consider: Analyse and present gaps in current risk assessment methods, Highlight emerging risks in public procurement not yet addressed by current models or analytics. Develop metrics that will demonstrate the impact of the risk assessment tool, as described (e.g., greater accuracy for assessing risks) Facilitate round-table discussions or feedback sessions where auditors can discuss current challenges openly, helping to highlight urgency organically from internal stakeholders' perspectives.
2. Build a Guiding Coalition	TdC’s current actions: involved relevant audit teams (i.e., real-time and posteriori teams) in the design of the risk assessment tool, including meetings with IMPIC and CITM; enlisted leadership in efforts to advocate for data access from external entities; promoted the value of data-driven risk assessment to key stakeholders, such as data providers. Additional actions to consider: Demonstrate value of risk model in the future, not only to auditors, but also to external actors and data providers, building on current actions. Consider the diversity of the coalition at a later stage and whether to include other experts or auditors that could benefit from similar risk methodologies or the process of developing them. Involve additional external experts or advisors for fresh perspectives on data use and risk analysis.
3. Form a Strategic Vision and Initiatives	TdC’s current actions: identified a vision, objectives and activities for the risk assessment tool with buy-in from management. Additional actions to consider: Further align initiatives with plans for continuous staff training in data literacy and integrating machine learning insights into audit practices. Encourage interested staff from across the audit teams to participate in the next iterations of the risk model to help scale and integrate new methodologies. Continue to ensure alignment with the TdC’s other risk assessment and digital initiatives.
4. Enlist a Volunteer Army	Given this table focuses on the project level, this element is less applicable and the TdC can reference the insights for “Build a Guiding Coalition.” The TdC could elaborate on this step for accelerating change in relation to its organisation-wide Digital Strategy. Ideas include: Develop a recognition programme specifically for this "army," providing badges, certificates, or other forms of acknowledgement within the organisation. Consider creating a collaborative online platform where volunteers can share their experiences, challenges, and successes, fostering a supportive community.
5. Enable Action by Removing Barriers	TdC’s current actions: identified a vision, objectives and activities for the risk assessment tool with buy-in from management. Additional actions to consider: Formalise data-sharing access through standing agreements and avoid protracted negotiations in the future. Ensure the processes for audit teams to access new tools, provide feedback and receive technical support are simple and clear.
6. Generate Short-Term Wins	TdC’s current actions: the selection of the risk assessment project is in itself a decision by the TdC to generate a short-term win. Additional actions to consider: Identify and communicate quick, tangible successes like the identification of a risk that was previously overlooked, or time saved in the risk assessment process. Celebrate teams or individuals who achieve these wins.
7. Sustain Acceleration	TdC’s current actions: the selection of the risk assessment project is in itself a decision by the TdC to generate a short-term win. Additional actions to consider: After initial successes, encourage the sharing of best practices between teams and initiate further enhancements of the risk model. Implement additional training or learning resources based on feedback from the audit teams.
8. Institute Change	TdC’s current actions: the selection of the risk assessment project is in itself a decision by the TdC to generate a short-term win. Additional actions to consider: Solidify the use of the new risk model by embedding it in standard audit processes, including the current risk assessment process, and criteria for success. Adjust job descriptions, performance reviews, and training programmes to include competencies related to the analytics model.

Source: Author’s elaboration.

The actions described in the table provide the foundation for a more robust change management plan, the primary objective of which would be to facilitate a transition from current processes and mindsets to new, digitally inclined ones. They aim to instil confidence within the TdC, assuring auditors that while changes are forthcoming, they are calculated, beneficial, and supported at every level of the organisation. Involving all layers of the organisation in the change management process will not only help in achieving the goals set out in the Digital Strategy but will also aid in breaking down the existing siloes between Departments and audit teams for the benefit of individual digital initiatives. While the table above focuses on accelerating change of a specific project, the same concepts can apply to bigger plans for digital transformation, helping the TdC to not simply adopt new technologies but evolve as an institution. Box ‎2.2 provides an example from the U.S. General Services Administration and its IT Change-Maturity Program: Culture Pre-Assessment Guide. It offers further support for the TdC that is more focused on technology modernisation than the 8 steps and can assist in troubleshooting key areas.

Box ‎2.2. IT Change-Maturity Program: Culture Pre-Assessment Guide

Overview of the guide

The “IT Change-Maturity Program: Culture Pre-Assessment Guide” by GSA's Center of Excellence (CoE) is a comprehensive resource designed to prepare organisations for change and Information Technology (IT) modernisation. It emphasises understanding the impact of organisational culture on such transformative efforts. The guide explores the concept of “Innovation Adoption,” an approach that integrates Organizational Change Management (OCM) and behavioural science and intrapreneurial strategies. It introduces the C3 framework (Culture, Conditions, and Capabilities) to identify and address factors within the organisational environment that could affect modernisation initiatives.

Key cultural challenges

The guide highlights several cultural challenges that commonly impact IT modernisation efforts:

Resistance to change/new ways of working
Lack of confidence in the IT office by customers/lack of positive customer experience
Lack of empowerment of staff at all levels
Reactive vs. proactive responses to modernisation
Unclear mission/vision for modernisation effort/unclear expectations for staff
Silos that prevent effective collaboration
Lack of CIO department/Program Office understanding user needs
Lack of Change Management strategy and resources aligned to initiative

These challenges underscore the complex dynamics within organisations that can either hinder or facilitate successful change. The guide encourages organisations to first identify where they sit on a change curve (see Figure ‎2.7), and it offers exercises and templates to assist organisations in assessing their own culture type.

Culture types

The guide highlights the following culture types:

Highly Skeptical of Change: Organisations identified as highly skeptical are very resistant to change, with key influencers often disengaging when change initiatives are introduced. Change, when it occurs, is usually partial rather than a complete transformation.
On the Fence: These organisations are not actively pursuing large-scale change efforts. They may be open to experimenting with change in certain areas but generally spend more time in the planning phase than in actual implementation.
Initiates/Embraces Change: These are organisations that are actively investing in new solutions and are quick to adapt their operations. They are not risk-averse and are open to learning from failures, often leading in the implementation of new technologies and methodologies.

Source: (Centers of Excellence, n.d.[66]).

2.4.2. Strengthening data literacy and digital skills for assessing public procurement risks through targeted training and internal knowledge-sharing

In this context, CITM faces a number of challenges to convince auditors to embrace new ways of working and promoting a data-centric culture. The TdC’s Digital Strategy recognises this challenge, highlighting the need for investments in training, development of human resources and equipping staff with knowledge and skills to operate in a digital era. Moreover, for some initiatives, the TdC has adopted a good practice of piloting and experimenting, including its initiative to strengthen data-driven risk assessments in public procurement. These are critical aspects of developing a data-centric culture, but there are ways to further formalise and enhance the TdC’s approach, going beyond the current Digital Strategy, starting with digital and data literacy.

Data literacy—the ability to read, interpret, create and communicate data as information—is a key requirement of modern auditors’ skillset, evidenced by numerous trainings, workshops and guidance for SAIs. While critical, data literacy is just one component of a broader set of digital skills. These skills reflect a range of abilities to use digital devices, applications, tools and networks to access and manage information. Specifically, auditors with digital skills are not only data literate, but they are also equipped to ask strategic questions while understanding opportunities and limitations of techniques and tools. Moreover, they are more likely to have realistic expectations about time and resources when planning the use of data and deciding on methodological trade-offs. Having more robust data skills also increases the likelihood that auditors will adopt the tools developed by the CITM and the DSTI, and in the future, facilitate and promote improvements and innovations.

Within the TdC, digital skills are largely consolidated in the CITM and the DSTI. These are small teams with a cross-organisational mandate, as described above. At the time of writing, CITM employed five individuals, including a data scientist, and DSTI’s team included 22 IT technicians, among several other roles, with responsibility for ensuring the IT and data needs of the TdC’s 9 audit departments are met. According to TdC officials, outside of these teams, auditors’ skills with data and tools typically involves Excel-based data processing and analysis.

The TdC carries out an annual diagnosis of training needs, with its current iteration likely to lead to a three-year training plan that envisions a component on data management and analysis. Courses taught in the past and that will be repeated include: “Challenges of digital transformation for auditing”, Power BI and Advanced Power BI, Digital Transformation and Data Science, and the Application of Artificial Intelligence in Auditing. In previous years, the TdC also carried out trainings on specific methodologies, including financial auditing standards and planning, identifying, and assessing risks when conducting financial audits.

In the context of this project, which envisions a data-driven risk model for public procurement that is used across audit teams, there are specific digital skills and areas of expertise that the TdC could cover in its training plan. These are focused on this specific activity given the objective of the project with the OECD, but the technological and quantitative skills are applicable for other purposes. For instance, the TdC could develop further its course offering concerning data-driven risk assessments, with a focus on building auditors’ skills in quantifying risks, structuring data and ways to monitor the performance of the risks model. In addition, the TdC could consider adding courses to its current offerings on data visualisation, which include the use of Power BI and Microsoft Excel, but that focus more on visualisation of large-scale datasets. The use of AI or advanced analytics for assessing public procurement risks typically relies on the ability to handle large-scale datasets (i.e., 100 thousand or million observations) and implement advanced statistical methods like binary logistic regressions, matching, or principal component analysis. Presenting user-friendly results makes it easier to interpret and respond to risks, and it is helpful for those responsible for creating and using dashboards to have advanced knowledge of data visualisation principles and software (e.g., R Shiny package or Tableau).

In the context of assessing risks in public procurement, cross-functional teams and a domain expert with intimate knowledge of public procurement processes and risks are critical. Auditors may only be periodically exposed to public procurement issues and risks, as their responsibilities and audit engagements constantly change. Therefore, having a domain expert in public procurement ensures the effective design, implementation and updating of a data-driven risk model. In the early stages of its initiative, TdC officials involved experts in public procurement as part of its exercise in identifying risks, a practice that can be formalised in the future. Figure ‎2.8 below elaborates further on the specific contributions of different roles for using advanced analytics to assess public procurement risks, and it is broadly applicable to other digital initiatives. Some roles may be combined, but it offers the TdC a conceptual description of roles to aid future planning and engagement of cross-cutting teams in its digital initiatives.

Figure ‎2.8. Examples of contributions of different roles to developing a data-driven risk assessment in public procurement

The European Union’s Digital Competence Framework (DigComp) offers further information for the TdC to ensure its trainings to build digital skills are fit-for-purpose. Box ‎2.3 below explains DigComp and includes additional competencies from the OECD’s own analysis that can help the TdC to set goals and metrics for each training programmes that draws from a well-defined list of competencies. For the TdC, these competencies are shared across teams, including management, CITM and audit teams working with data. As such, trainings can target the entire organisation, while tailoring modules to specific roles and responsibilities concerning the TdC’s Digital Strategy.

Box ‎2.3. Competencies of a data-driven workforce

The European Union’s Digital Competence Framework (DigComp) is a tool to improve digital competence. The competencies are informative for SAIs and other institutions that can partake in auditing-like activities, such as evaluations and academic research. The competencies below reflect those in the DigComp as well as elements added by the OECD, and they provide a reference point for the development of content and metrics for training programmes—for management and auditors—to enhance an array of digital skills across an organisation.

Strategy and problem-solving:
- customising digital environments to personal needs
- using digital tools to create knowledge and innovate processes
- identifying digital competency gaps and ideas for improvement, including the use of self-assessment tools
- Setting goals, priorities and devising digital strategies and action plans
Information and digital literacy:
- browsing, searching and filtering data
- critically evaluating the credibility and reliability of data sources
- organising and storing data
- understanding of statistics to help evaluation and analysis of data
- understanding of requirements for reproducibility
Digital content creation:
- creating new, original and relevant content and knowledge
- understanding copyrights and licenses
- programming and software development, visualisation of data and information to convey knowledge
Safety and privacy:
- protecting personal and sensitive data
- understanding tools and techniques to safeguard privacy, such as delinking and anonymisation
Communication and collaboration:
- Sharing data and understanding data sharing protocols;
- knowing about referencing and attribution practices;
- using digital tools and technologies for collaborative processes;

Source: (OECD, 2020[67]).

CITM has identified the importance of adopting and utilising technology to shift auditing from a reactive and retrospective exercise to a proactive, predictive, and future-oriented practice. This transformation presents an opportunity to produce audits of higher quality, but it also necessitates a sustained commitment and investment in bridging current capacity gaps. While having a robust training plan is critical, not all skill development and capacity building needs to be formal. One way to be able to keep a consistent pulse on the needs of auditors, while promoting the sharing of practices and knowledge, would be to develop an informal community of practice. This could focus on specific initiatives or themes that are relevant across TdC audit teams. For instance, multiple audit teams will ultimately use the data-driven risk assessment that the TdC is developing, which could be a focal point for a community of practice to aid adoption and collective addressing of challenges.

Other SAIs have developed such communities and resources with a dedicated focus of developing digital competencies, such as the Austrian Court of Audit’s (ACA) “R-Community.” This community of auditors with skills in data analysis exchange about methodologies and uses of the open-source statistical software “R.” The goal of the community is not to make all of the ACA’s auditors experts in using R, but rather to generate sufficient knowledge about what data analysis can do and for what purposes it can be used (Mayrbaeur, 2023[68]). In Belgium, the Supreme Audit Institution created a discussion group called “DataLab” to help integrate analytics across the organisation through monthly meetings, trainings and expert advice from data scientists.7 The TdC may also find numerous resources online to bolster data literacy internally without reinventing the wheel, such as the online data and analytics course provided by the Netherlands Court of Audit.8

2.4.3. Ensuring auditors have sound knowledge about data privacy and ethics

In the context of accessing and processing public procurement data for purposes of financial inspection, SAIs are obliged to have unrestricted access to pertinent data for conducting their audits, as per INTOSAI auditing principles and standards. This includes data protected by specific regulations or control protocols, such as those commonly applied in the commercial, banking, medical and national security domains. In Portugal, auditors’ use of external data sources is aligned with the stipulations in the RGPD and the TdC Regulation, as well as principles found within the TdC’s Audit Manual.

In relation to data use and its ensuing disclosure in the TdC’s reports and decisions, there are several noteworthy points. As per Article 13 of the TdC’s Regulations, the TdC must devise a "communication strategy, suitable for the fulfilment of its mandate, in compliance with the principles of transparency, accountability and protection of personal data, namely through the dissemination of results in a timely manner." Moreover, the TdC follows the LOPTC, the TdC Regulation, and, in a secondary capacity, the Code of Civil Procedure (CPC), maintaining that data processing is the purview of the presiding judge within the TdC. This principle dictates that case information should be made public, unless it is covered by personal data or secret protection provisions.9

Current legislation permits the release of information in circumstances where the public interest outweighs the interests safeguarded by the protection rules. However, this is subject to the principle of minimisation and assessed on a case-by-case basis by the responsible judge. Guidelines in Resolution No. 3/2018-PG provide a roadmap to ensure the TdC does not disclose more personal information than required. For instance, "the publication of judgments, audit reports and other acts of the Court should only contain the personal data that are indispensable for informing society about the use of public financial resources and guaranteeing the accountability of the managers of these resources and those responsible for finances." In cases of ambiguity regarding personal data that may be included in published court acts, the Court's Data Protection Officer should be consulted.

The legal and regulatory framework for the TdC’s data protection, as well as its inclusion in internal audit manuals, demonstrates the TdC’s adherence and commitment to the protection of privacy and ethical use of data. Nonetheless, the introduction of new tools, particularly a risk model that heavily relies on external data source and possibly additional personally identifiable information, creates incentives for the TdC to revisit existing policies, guidelines and trainings for auditors regarding who may use these tools. The aforementioned trainings could include a module on topics of data ethics and privacy. Going beyond this, even though it is not legally required, the TdC can also consider how it would promote transparency of its activities that involves the use of data for its digital initiatives, particularly those that could be considered sensitive. For instance, the TdC could periodically review (and publish) its code of conduct to ensure it keeps pace with its use of data and AI as its digital initiatives evolve. While a national initiative, the TdC could take inspiration from the United Kingdom’s National Fraud Initiative (NFI) Code of Data Matching Practice, which promotes transparency and ethical use of data, while laying out principles and practices for protecting citizens’ right to privacy (see Box ‎2.4).

Box ‎2.4. The UK National Fraud Initiative’s Code of Data Matching Practice

The Audit Commission’s National Fraud Initiative was launched in 1996 as the United Kingdom’s largest data-matching exercise in relation to fraud. The Serious Crime Act of 2007 enabled bodies, other than those with a mandatory requirement to provide data for the National Fraud Initiative, to volunteer to participate by providing data to the commission. The National Fraud Initiative has enabled participating organisations to prevent and detect more than GBP 300 million fraud and error in the period from April 2016 to March 2018. Approximately 1 200 public and private sector organisations participate in the initiative, including the public audit agencies in Scotland, Wales and Northern Ireland. Each national audit agency carries out data-matching under its own powers, but uses the National Fraud Initiative’s systems, processes and expertise.

To increase transparency around this massive data-matching exercise, the National Fraud Initiative has set out a Code of Data Matching Practice that is followed by all organisations that participate in the Cabinet Office’s data-matching exercises. The code “creates a balance between the important public policy objective of preventing and detecting fraud, and the need to pay due regard to the rights of those whose data are matched for this purpose.” To achieve this goal, the code was informed by the consultation of a range of stakeholders, with the Information Commissioner’s office providing input on data protection.

The code notably requires each institution to publish a privacy notice that informs citizens about the specific datasets used, the way they are collected, the purpose of this data-matching exercise and its legal basis, the institutions with which the data are shared, the retention period for the data, and the rights of citizens including complaint mechanisms. This example illustrates both the necessity of transparency for integrity actors when implementing anti-fraud programmes and the value of their input to inform the creation of codes of practice that safeguard citizen’s rights.

Source: (Cabinet Office, 2018[69]).

2.5. Technology and processes

Digital maturity relies on technology, like IT systems and software, but it shouldn't be the sole focus. Instead, the broader vision for digital transformation should drive technology choices, tailored to specific needs. Considering the varied complexity and functionality of digital services and tools, a pragmatic approach involving cost-benefit analysis is vital for responsible technological investments. This section explores several challenges the TdC faces from the perspective of IT, digital tools and data integration. Some of these challenges have broader implications for the TdC’s Digital Strategy and audit work in general. Where appropriate, this section offers a more narrow discussion on issues concerning data-driven risk assessment in public procurement.

2.5.1. Advancing improvements to IT infrastructure for effective data-driven risk assessments in public procurement

As noted, in 2021, the TdC conducted an IT Audit Self-Assessment (ITSA) with the Court of Auditors of Spain facilitating the process. The primary goal of the ITSA was to examine the functionality and performance of the TdC’s existing IT systems and conduct a gap analysis of the TdC’s maturity in relation to key COBIT processes that impact TdC's operations.10 The ITSA demonstrated that although the TdC had a level of IT maturity that was in line with similar institutions in the European Union (i.e., Level 3—documented communicated processes), key gaps remained. In response to the results of the ITSA, the TdC implemented the following improvements:

Improved communication methods with those being audited, accomplished through the implementation of an “eAccounts” portal.
Developed a project for electronic audit management, which culminated in the creation of the “ModInAudit” application.
Established and updated training programmes for auditors on an annual basis (Court of Auditors of Portugal, 2023[70]).

Since the TdC’s ITSA, advancements in technology necessitated further assessments and refinements that would ultimately lead to the TdC’s current Digital Strategy. To further advance its digital maturity, the TdC engaged an external consultant in 2021 to independently assess the IT architecture and infrastructures in specific Chambers of the TdC, focusing on functionalities of the document management (GDOC) and entity management (GENT) systems.11

Findings from this assessment offer some insights into the evolution of these systems that play a critical role in the TdC’s digital transformation initiatives, including carrying out data-driven risk assessment and the IT challenges the TdC faces. According to TdC officials and questionnaire responses, some of these challenges are still ongoing as of December 2023. For instance, the user/system interface of the GDOC and GENT applications could be improved to cater better to specific user tasks and needs. Moreover, there is a need to further improve and define processes concerning various applications, and address challenges related to both information systems and technological infrastructure, including (but not limited to) the following:

Development of the "portal of external services" to facilitate information exchange with external entities.
Implementation of the "TdC in real-time" project to achieve interoperability with other public administration organisations. This depends on the interoperability between the TdC's information system and the information systems of public administration organisations where data on entities under control and jurisdiction reside.
Creation of an internal services portal to provide TdC employees with easy access to information systems.
Execution of the eContas project for prior inspection and implementation of responsibilities, integrating different stakeholders and aligning functional profiles with relevant information and tasks.
Automation of procedures for internal verification of accounts and issuance of reports.

Several of these issues are directly relevant for the TdC’s initiative to enhance its data-driven risk assessment in public procurement.

2.5.2. Addressing challenges concerning data integration and interoperability for public procurement data, including working with data owners to improve data quality

Within TdC, all database management systems have a relational nature, meaning that, in theory, they share the same structures and types of data. The consistency in using relational databases provides for a unified approach to data management, which is particularly helpful for the TdC to query complex datasets and structured data. In some cases, there is the possibility of functional and semantic interoperability, in which certain fields in a system depend on fields in another system, enabling the integration activities. Achieving this interoperability is complex, given the need to maintain data integrity and synchronise data across different systems. The integration of functional and procedural activities across various systems poses additional challenges, particularly for real-time data processing and maintaining transactional consistency.

Ensuring the accuracy of results in any data-driven approach hinges on two fundamental aspects: the volume and the integrity of the data. Additionally, effectively integrating data from various sources plays a pivotal role in this process. This integration requires parsing, correcting, standardising, matching and consolidating to ensure data usability. For purposes of the TdC’s initiative to assess public procurement risks, there are three high-priority databases, shown in the table below along with a matching variable.

Entity/Stakeholder	Component	Variable to Match
IMPIC – Institute responsible for Public Contracting Portal ("Base")	Public procurement data: procedure, contract/item-level	ID of the contract
AT – National Tax and Customs Authority	Company data (tax and financial) Amounts invoiced to the entities controlled by the Court	Tax number of supplier/candidates Tax number of the entities controlled by the Court
IRN – RCBE	Company data ("effective company beneficiaries")	Tax number of supplier/candidates

Entity/Stakeholder

Component

Variable to Match

IMPIC – Institute responsible for Public Contracting Portal ("Base")

Public procurement data: procedure, contract/item-level

ID of the contract

AT – National Tax and Customs Authority

Company data (tax and financial)

Amounts invoiced to the entities controlled by the Court

Tax number of supplier/candidates

Tax number of the entities controlled by the Court

IRN – RCBE

Company data ("effective company beneficiaries")

Tax number of supplier/candidates

Analytical indicators can be obtained at different levels of aggregation and require additional fields to enable integration of various data sources, which can present several challenges. For instance, requirements for firm-level indicators might benefit from combining multiple databases that would naturally be matched using the tax numbers (e.g., one containing information about procurement activities and another concerning firm tax information). See Chapter 3 for further discussion on the data sources and variables that are relevant for the TdC’s risk assessment methodology.

Despite the relevance of the data sources of the entities in the table below, data integration and interoperability present challenges. One of the main issues is the diversity of systems and formats used to store and manage data. Institutions typically have their own systems, databases, and software, all of which can make it difficult to establish interoperability. Moreover, data governance and security measures can vary across institutions, including concerns about data privacy and sharing protocols. In Finland, the National Audit Office was forced to develop its Risk Detector Tool for public procurement using open data due to data protection regulations and controls the prohibited more direct access to the data (see Box ‎2.5).

Box ‎2.5. The Risk Detector Tool of the National Audit Office of Finland

The Risk Detector Tool, developed by the National Audit Office (NAO) of Finland, is designed to assist auditors in identifying potential risky procurements and guiding them in planning and targeting their work. It aims to enhance audit planning and ensure the quality of audit work by providing auditors with an overall picture of risks in public procurement, leveraging a bespoke. The tool primarily targets financial auditors, especially those involved in accounting unit level audits.

Developing the tool involved enriching the Tutkihankintoja.fi dataset with other public data and utilising Python, Tableau, and Power BI for data analysis. The NAO co-designed the tool with end users to strengthen buy-in and to hone visualisations that would be useful for users. The NAO faced several challenges, including heavy bureaucratic processes and restrictions for accessing data due to data protection policies. Moreover, the original datasets had limited features (i.e. low expressive power) and the methodology and approach was new for the NAO, which required overcoming an initial learning curve.

Figure ‎2.9. Pilot architecture using open data

The NAO was ultimately successful in developing a prototype based on open data, which includes AI-generated network analyses of public procurement risks. As part of its lessons learned, the NAO highlighted the need for experimentation, overcoming failures while iterating and developing tactics for communicating internally about the tool, which in their case was a podcast.

Source: (Gullichson, 2019[71]; Grünbaum, Gullichsen and Väänänen, 2019[72]).

The lack of standardised data schemas and protocols, primarily amongst data owners, further hampers integration efforts. Data quality standards and inconsistent data-entry practices can introduce errors and inconsistencies that reduce the accuracy and reliability of integrated data. Concerning the data for assessing risks in public procurement, many of these challenges reside with IMPIC. The TdC’s initiative represents the first time in which it is directly accessing IMPIC’s datasets in high volumes. To facilitate this data exchange, the TdC made several adjustments to its IT infrastructure. Nonetheless, challenges concerning data quality (e.g., high levels of duplicates) remain at the time of writing this report. Overcoming these challenges will require ongoing collaboration between the TdC and IMPIC, as well as other data owners to the extent needed, to ensure standardised frameworks and robust data management strategies underpin improvements to data integration and interoperability.

2.5.3. Aligning the Digital Strategy and tools with the TdC’s technological maturity for assessing public procurement risks

Robotic Process Automation (RPA) is described as the next frontier for SAIs, after data analytics (Otia, 2022[53]). The TdC’s Digital Strategy emphasises the need to advance the automation of processes as one of its key initiatives to maximise efficiencies and the value-for-money of audits (Tribunal de Contas, 2023[57]), allowing auditors to reallocate their time to critical-thinking tasks, like data analysis and developing findings. By minimising manual data processing, automation also has the potential to reduce errors. For instance, risk dashboards that offer automated alerts could help auditors in conducting more targeted, co-ordinated audit selection.

While the TdC recognises the value of automation in its Digital Strategy, many of its core processes rely on manual methods and traditional tools, highlighting a gap between strategy and practice. Automation, especially in public procurement, could transform mundane tasks like risk identification and data management, allowing auditors to spot irregularities in a more efficient and co-ordinated manner. Bridging the gap between strategy and practice requires not only automation, but also equipping auditors with the necessary tools. The use of applications like Microsoft Excel and Caseware IDEA dominate the TdC’s current practices. Some audit departments reported using Power BI and R software in responses to the OECD’s questionnaire. Questionnaire respondents from multiple audit departments did not identify specific needs or gaps in software or tools available when asked. This suggests that within the TdC there is either general satisfaction with current tools or a lack of awareness of the benefits of other technologies. As the TdC develops its digital initiatives and integrates new approaches in its audit practices, including its data-driven risk assessment methodology, the TdC could consider conducting a deeper evaluation of current tools to ensure alignment between strategy, practices and the modern auditing needs of TdC’s auditors.

Concerning data-driven risk assessments for public procurement, the “tool” or mechanism to collect, disseminate and potentially visualise the results of the assessment is equally as critical as the underlying methodology. An effective dashboard, for instance, centrally developed and managed by CITM or DSTI, can promote a co-ordinated approach that avoids duplication between audit departments and creates a horizontal feedback loop. Chapter 1 presents ALICE, an RPA tool used by Office of the Comptroller General (CGU) and the Federal Court of Accounts of Brazil (TCU). ALICE utilises an algorithm to asses unstructured data, in particular, to extract monetary values from non-uniform text in PDF bidding notices. It then organises this data using a Random Forest classification method to analyse the "materiality factor," an estimated risk value associated with these bids. ALICE effectively uses data and analytics to identify irregularities and improve auditors' efficiency in procurement, even amongst auditors without expertise in machine-learning or AI. Its success hinges on two factors:

Senior management support was crucial in identifying irregularities in public procurement. This backing, seen as vital for fostering a culture of integrity, led to innovative auditing practices. For example, TCU Counsellors agreed to sign an ordinance validating a new workflow, enabling auditors to make use of the AI-based technology and act efficiently to address the risks of irregularities.
To manage data overload and human limitations in data processing, the CGU and TCU implemented two strategies to aid auditors. Firstly, ALICE sends daily emails summarising key bid information and alerts, helping auditors focus their analysis. Secondly, they developed a specialised dashboard that offers various filters to streamline their search and access detailed bid analyses and irregularities identified by ALICE (OECD, 2022[73]).

TdC’s development of its own data-driven risk assessment in public procurement could benefit from these findings about its institutional context and Brazil’s example by keeping in mind opportunities for automation, auditors' current digital literacy, and the need for a risk dashboard to support non-technical users. Brazil's experience with ALICE, particularly in analysing unstructured data to address financial irregularities, underscores the potential of advanced analytics in this domain. However, the scope of effective methodologies in public procurement is not limited to financial aspects alone.

TdC's commitment to assessing a wide range of irregularities and risks, including both financial and non-financial risks that can affect the efficiency of public procurement, points to the need to consider a diverse set of tools and approaches. Text mining techniques, as demonstrated in the academic context of Morocco's green public procurement initiative, can provide inspiration. The box below explores the use of text mining for scrutinising unstructured data in procurement documents and it highlights the potential for analysing the extent to which environmental considerations are integrated into tendering processes. For an organisation like TdC, aiming to encompass both financial and non-financial risks, as well as analyse both structured and unstructured data, Morocco's approach provides valuable insights and highlights the possibilities for its own risk assessments.

Box ‎2.6. Advancing Green Public Procurement in Morocco: A Case Study on Text Mining for Sustainable Procurement Auditing

Overview of the initiatives

A group of researchers explored the use of text mining techniques to analyse public procurement processes and measure the degree of environmental criteria integration. The purpose of this approach was twofold: to gauge the current state of green procurement practices (i.e., “green infiltration”) within the public sector, with a focus on tenders for Information and Communication Technologies (ICT), and to develop a tool that could objectively and efficiently assess environmental aspects in tender documents. This was driven by the need to address the lack of systematic and objective methods in evaluating the green attributes of public procurement, a critical issue considering the increasing global emphasis on environmental sustainability.

Text mining for analysing procurement practices

The researchers employed automatic analysis or supervised statistical learning using advanced text-mining algorithms to analyse a high volume of unstructured data in tender documents. The process involved three major steps:

1. Data Collection and Preparation. Data was directly collected from the Moroccan public procurement platform. This included a wide range of document types, such as .doc, .docx, .pdf, and image files, relating to the purchase of computer equipment.

2. Keyword Analysis and Correlation Study. Keywords relevant to environmental sustainability were identified and their frequency of occurrence in the tender documents was analysed. A semantic network and keyword correlation cube were created to understand the relationships and contextual uses of these keywords.

3. Scoring System and Threshold Setting. A scoring system was devised to assign weights to keywords based on their relevance to environmental sustainability. This enabled the establishment of a threshold to categorise tenders as 'green'.

Figure ‎2.10. Python-based text mining process for assessing green public procurement

The results showed that the “infiltration” of environmental concerns remains week in Morocco’s public procurement. Only 3.38% of the ICT tenders analysed incorporated green criteria (in a sample of 592 calls for tenders), highlighting a significant area for improvement in sustainable public procurement practices. The researchers suggest the development of a “decision support tool,” and an automated monitoring model, to improve sustainable, green public procurement in Morocco.

Relevance to Supreme Audit Institutions

The text mining methodology piloted in these initiatives demonstrates one approach that SAIs can take to objectively assess and track the integration of environmental criteria in public procurement, a key metric for evaluating the performance of public entities in adhering to environmental policies and regulations. The approach can help to identify potential risks and shortcomings in the implementation of green procurement practices, enabling SAIs to pinpoint areas where interventions or improvements may be necessary.

Additionally, the methodology offers a robust framework for assessing the impact of public policies aimed at promoting environmental sustainability in procurement. It provides data-driven insights, facilitating a thorough evaluation of policy effectiveness and highlighting areas for further development. Furthermore, the approach enhances transparency and accountability in public procurement processes. By providing a clear and objective analysis of how environmental criteria are embedded in procurement documents, SAIs can effectively hold entities accountable for their environmental commitments and obligations.

Source: (El Haddadi et al., 2021[74]).

Similarly, researchers in Belgium conducted a longitudinal study and used text-mining analysis to identify patterns in sustainable public procurement practices (SPP) over time. They used a concept of SPP that targeted seven categories: 1) environmentally friendly procurement, 2) circular economy, 3) social return on investment, 4) ethical trade, 5) local and/or SME-oriented procurement, 6) Innovation-oriented procurement and 7) the use of sustainability labels. Among other findings, they found that public authorities implemented SPP in more than 70% of the notices reviewed, which was a higher rate than what had been self-reported previously. Nonetheless, they also found a gradual decrease in the implementation of SPP during the years studied (2014-2016). Overall, the researchers demonstrated the value of text-mining as a tool for assessing unstructured data and the implementation of SPP in public procurement notices to identifying patterns, which would not have been possible with other methodologies, such as conducting surveys or interviews (J. (Jolien) Grandia, 2020[75]).

The TdC’s risk assessment methodology could ultimately integrate both structured and unstructured data. As a starting point, TdC could first concentrate on developing a risk assessment methodology that incorporates structured data as an initial proof-of-concept, and then build on this in the future with additions to the methodology that incorporate unstructured data and text analysis, as illustrated in the examples of this above. For instance, as discussed in Chapter 1, the TdC’s entity management system (GENT) includes unstructured data based on information in the Official Government Gazette of the Portuguese Republic that could be leveraged for refining future risk assessments. This phased and iterative approach would allow the TdC to prioritise its resources and first demonstrate the value of its risk assessment methodology internally, using structured data that capitalises on the progress made with integrating IMPIC’s data, before turning to other types of risk assessment techniques.

2.6. Environment and partnerships

National laws and directives play a pivotal role in institutional digital transformation, shaping the digital governance and data management framework. This legal backdrop can either limit or advance technology adoption, impacting digital project success. In addition, fostering collaborative relationships between entities is crucial for digital maturity. This section highlights several opportunities for the TdC to enhance its digital maturity with specific consideration for the relevance and sustainability of its digital initiative for data-driven risk assessments. They include the formalisation of data-sharing protocols and collaboration with data owners to enhance the TdC’s own training.

2.6.1. Formalising data-sharing protocols for public procurement data so that data-driven risk assessments remain relevant and useful for auditors

INTOSAI’s Working Group on Big Data conducted a survey that revealed 45% of SAIs lack data-sharing mechanisms and regulations for big data use, 41% have limited international co-operation on big data, and 32% face challenges in inter-departmental data exchange (INTOSAI Working Group on Big Data, 2022[76]). These survey results suggest a gap in establishing effective data sharing and co-operation mechanisms, despite intentions and examples of SAI collaboration on big data audits.

Timely access to data is crucial for the effective implementation of the TdC's new risk assessment methodology. The primary users of this methodology, the real-time and ex post audit teams, rely heavily on the availability and accessibility of relevant data to conduct audits on ongoing and completed activities, contracts, and projects. As is often the case for SAIs, data access poses one of the greatest risks and bottlenecks for the TdC’s digital initiatives, in particular, its goals to enhance its assessment of risks in public procurement. The initial access to IMPIC’s database, Portal BASE, and the collaboration established during the project for data sharing marks a significant step forward. However, challenges remain in accessing other databases, as elaborated in Chapter 3 (e.g., IRN’s database of beneficial owners). Future iterations of the risk model for assessing public procurement risks could be enhanced by addressing these challenges, thereby broadening the data sources available for the risk model and improving risk coverage.

According to TdC officials, challenges for accessing data reflect inter-institutional dynamics as much, if not more, than any legal or regulatory provisions that restrict the TdC’s access to data for audit purposes. Engaging with legal and policy frameworks to foster a conducive environment for data sharing is crucial. Continuous dialogue with policymakers and stakeholders to advocate for supportive laws and regulations may also help beyond the scope of this project. CITM and DSTI are critical drivers of this effort. To keep the momentum after the project, the TdC could establish a broader dedicated task force to oversee the development and implementation of data-sharing protocols, providing a more focused and collective approach to overcoming the identified challenges. This task force could also serve as a liaison between, or even include, the TdC and data owners that are relevant for the TdC’s audit work, promoting a more harmonised approach to data sharing that ensures alignment with the goals of TdC’s digital transformation journey.

2.6.2. Enhancing collaboration around training and guidance with data owners, while ensuring privacy

Looking across government at the various owners of databases the TdC uses, opportunities exist to strengthen a co-ordinated and collaborative approach to enhancing data literacy and knowledge-sharing. As detailed in Chapter 3, the TdC relies on many external databases for its current and future auditing and risk assessments in public procurement. In concert with these stakeholders, the TdC could facilitate the development of more structured trainings to gain a deeper understanding of how the data can be used in its work and the nuances in terms of data structure, reliability and other technical issues.

Currently, the IGFEJ focuses on providing training in public procurement exclusively to its employees. IMPIC takes a comprehensive approach by preparing and disseminating good practice manuals, technical guidelines, FAQs, webinars, videos, videocasts, and providing training programmes, often in collaboration with other entities. The IRN conducts frequent training sessions for various entities and participates in conferences and colloquia. The AT does not provide external training, but it ensures that protocols for data sharing contain confidentiality rules and obligations. Collaboration with these entities could help the TdC to strengthen its current training plan by tapping into existing resources or creating modules that are specific to such databases that auditors commonly use or cite.

Enhancing collaboration could also involve strengthening the feedback loop with data owners to drive improvements in data quality, which in turn would enhance future risk models. The TdC is well-positioned to drive improvements in gaps in data and enhance the quality of public procurement data, as the TdC deepens its reuse of this data for analysing risks. Currently, feedback loops with data owners, such as IMPIC and AT, occurs on an ad hoc basis (or not at all). As its digital initiatives evolve, the TdC could establish a more consistent and systematic mechanism for exchanging insights and feedback with data owners, outside of audit engagements and within the scope of its data use. The feedback could inform not only improvements to the data, but also trainings and guidance on using specific databases. For instance, the aforementioned NFI in the United Kingdom produces guidance that sets out data specifications in terms of how data should be formatted, and the types of data checks entities can conduct.

2.7. Conclusion

The TdC has made significant strides in its digital transformation journey, which is crucial for upholding its role as a standard-bearer of transparency, accountability, and public trust. This transformation is particularly vital in enhancing the TdC's capabilities for data-driven risk assessments in public procurement. The digital strategy of the TdC, as discussed in this chapter, necessitates a comprehensive approach encompassing strategy and organisation, people and culture, technology and process, and environment and partnerships. Each of these domains presents its unique challenges and opportunities.

The TdC’s Digital Strategy, while ambitious, requires greater coherence among its initiatives and a robust mechanism for monitoring and evaluation. This strategic alignment is essential for effective implementation and for achieving desired outcomes in risk assessment and transparency. In terms of people and culture, there is a pressing need for change management, data literacy, and digital skills enhancement, particularly in the context of public procurement risk assessments. Ensuring that auditors are well-versed in data ethics and privacy is also paramount.

Technologically, the TdC faces challenges in IT integration, digital tool efficacy, and data interoperability. Addressing these issues is critical for the success of the Digital Strategy and for improving the TdC’s overall audit efficacy, especially in assessing risks in public procurement. Finally, the TdC's environment and partnerships are key to its digital maturity. By formalising data-sharing protocols and collaborating with data owners, the TdC can strengthen its training programmes and enhance its capacity for effective, data-driven risk assessments.

Overall, the TdC's commitment to digital transformation is a positive step towards more efficient and effective auditing processes. However, for the full potential of this transformation to be realised, a balanced focus on strategy, people, technology, and partnerships is necessary. This will not only improve the TdC’s internal processes but will also reinforce public trust and accountability in the area of public procurement.

Annex 2.A. Key dimensions for assessing institutional digital maturity

The dimensions and key practices below are based on reviews of academic literature, discussions with subject matter experts in government, industry and non-governmental institutions, insights from the OECD’s technical support for governments to strengthen their digital strategies and data-driven risk assessments, as well as OECD Recommendations.12 Numerous self-assessment tools for digital or technology maturity that are relevant or made for integrity actors at an institutional level, such as the Supreme Audit Institutions (SAI) Information Technology Maturity Assessment, also provided inspiration. These practices consider digital maturity and transformation from an organisational perspective, but many are applicable for designing and implementing digital projects.

Strategy and organisation

This dimension encompasses leadership’s vision and strategy for digital transformation, including its goals for strengthening the use of digital technologies and data. A digital transformation strategy can stand alone or be integrated with existing organisational strategies. Either way, the aim is to ensure alignment of the digital strategy with other organisational priorities, audit processes and IT strategies (Bumann, 2019[52]). Clear delineation of roles and charting out responsibilities is imperative. This can include establishing an entity internally with an organisation-wide mandate to implement and co-ordinate digital initiatives. Data management and data governance are also key aspects of this dimension. They involve the policies, procedures, standards and controls that ensure data privacy, quality, consistency and security. Relative to project-based improvements, digital transformation by nature has a disruptive effect on an organisation’s traditional approaches to data management and data governance.

Key practices in this dimension include:

Align the Digital Strategy with the goals and objectives of other institutional strategies, such as the Strategic Plan and IT Strategy.
Conduct assessments and establish a baseline for digital maturity, capabilities, IT infrastructure and architecture, and possible gaps.
Identify key opportunities and challenges concerning data management and data governance, including priorities for ensuring data security and quality.
Define roles and responsibilities internally, including the designation of an entity to implement the Digital Strategy that has an organisation-wide mandate and access to leadership.
Engage with key stakeholders in the design of the Digital Strategy, including leadership, management and users of new tools, to understand digital maturity and priorities.
Establish a plan and key performance indicators to monitor the implementation of the Digital Strategy.

People and culture

The expertise, skills and commitment of individual employees within an organisation are central to digital maturity on any level, whether the goal is transformation or introducing a new tool for using data. Core competencies often revolve around digital and data literacy, sometimes extending to advanced programming skills. Alongside these technical proficiencies, it is critical that employees understand the policies, processes and behaviours that promote the ethical use of data (OECD, 2020[55]). Furthermore, sector-specific knowledge and specialised expertise are also critical competencies, such as having sector-specific knowledge to understand the data landscape for risk analyses. Legal expertise is also valuable for navigating the legalities of data access, privacy, storage, and security. A digital-ready culture is not only about having the right set of skills and experiences available, but it demonstrates tangible ways that leadership and employees rally around digital goals. This can manifest in different ways, such as having policies that allow for the experimentation of new technologies or providing training for employees to improve their digital skills (OECD, 2022[56]).

Key practices in this dimension include:

Ensure that leadership visibly endorses and partakes in digital initiatives, embodying a top-down commitment to the organisation's digital aspirations.
Develop and implement a change management and continuous learning plan that focuses on enhancing digital and data literacy, as well as sector-specific knowledge.
Introduce and encourage training programmes targeting technical proficiencies like advanced programming and data ethics.
Institute clear policies that favour experimentation with new digital tools and technologies to foster innovation and a “trial-and-error” mentality.
Establish guidelines on the ethical use of data, ensuring that staff understands and adheres to them.
Prioritise and establish mechanisms for internal knowledge sharing, facilitating the dissemination of sector-specific, technical and legal expertise.
Promote a culture of collaboration and digital empowerment, where employees at all levels feel engaged and invested in digital transformation objectives.
Collaborate with legal experts to navigate the intricacies of data laws, ensuring the organisation remains compliant while maximising its digital potential.
Implement feedback loops to understand employee challenges and needs in the digital landscape, adjusting strategies based on this feedback.
Regularly evaluate the digital skills gap within the organisation and adjust training programmes accordingly.

Technology and processes

While technology, including IT systems, tools, and software, and the processes encompassing them are vital components of digital maturity, they should not be perceived as the primary objectives. The broader vision for digital transformation or the intent of any given project should inform technological advancements rather than being led by them. This underscores the importance of tailoring technology to specific needs. The term "state-of-the-art" is contextual, acknowledging that digital services, IT mechanisms, and tools differ in their complexity, resource needs, functionality, and alignment with various organisational goals. Given the rapid evolution of technology, expending resources without a lucid objective can lead to a waste of resources. Thus, a pragmatic approach involving cost-benefit analysis can guide judicious decision making about technological investments. This analysis can consider collaborative investments or leveraging open-source technologies that may offer additional public benefits, such as the promotion of systemic transparency and collective technological development. Moreover, it is critical that considerations about adopting new technologies include assessments of their impacts on society, human rights and privacy, among other issues, to avoid exacerbating risks of discrimination and digital exclusion.

Key practices in this dimension include:

Ensure any technology adoption aligns with the strategic objectives or specific goals of the organisation.
Understand current capabilities, identify gaps, and ensure alignment with the organisation's digital maturity and objectives.
Before investing in any new technology, gauge its potential return on investment and long-term sustainability.
Start with a minimum viable product or proof-of-concept to test and validate new technologies or digital tools.
Given the fast-paced nature of technological evolution, adapt and update tools and systems based on changing needs and feedback.
Regularly research and explore new technological advancements that could replace legacy systems and enhance organisational effectiveness and efficiency.
When selecting technologies, consider how easily they can be scaled or adapted to changing organisational needs or goals.
Ensure that technologies are user-friendly, meet the needs of the organisation, and are accessible to all relevant stakeholders.
Ensure that any new technology or process integrates robust cybersecurity protocols to safeguard organisational data.
Define roles, responsibilities, and decision-making processes related to technology adoption and usage.
Facilitate channels for sharing best practices, lessons learned, and feedback regarding technology tools and processes.
Establish mechanisms to assess the social, human and ethical impact of adopting new technologies and mitigate associated risks, including those concerning data privacy, discrimination and digital exclusion.

Environment and partnerships

National frameworks, ranging from laws to directives, can considerably shape the trajectory of institutional digital transformation. For instance, a country's legal and policy framework can lay the foundation for managing digital governance, data stewardship and the sharing of data. These external parameters can influence the effectiveness of a digital transformation project, either limiting or propelling the adoption of digital technologies. Within an organisation, while robust data governance streamlines the sharing and accessibility of data, the true essence of data sharing transcends just infrastructure or processes. Cultivating collaborative relationships between entities, including industry, academia and civil society organisations, is an indispensable cornerstone for advancing digital maturity and ensuring that necessary safeguards are in place.

Key practices in this dimension include:

Stay abreast of updates to knowledge of laws, policies, and guidance related to digital governance, data management, and sharing.
Engage with policymakers to advocate for supportive laws, regulations and policies that bolster the goals of digital initiatives.
Establish data-sharing protocols that align with both internal goals and external legal requirements, allowing for efficient and timely exchange of information.
Define institutional roles, responsibilities and expectations for all digital initiatives that involve collaboration with external stakeholders.
Ensure that partnerships are mutually beneficial, fostering a sense of shared ownership and collective achievement.
Establish channels to gather feedback from partners, ensuring continuous improvement in collaborative endeavours.
Encourage an organisational mindset that values partnerships as a key enabler of digital growth.
Establish relationships with other organisations, both within and outside the government (e.g. industry, academia, civil society), to promote collaborative digital initiatives, share best practices, and ensure ethical use.

Annex 2.B. Standards and guidance for technological and data maturity in Supreme Audit Institutions

The International Organization of Supreme Audit Institutions (INTOSAI) develops and promotes principles, standards and guidance for SAIs. Some of these focus explicitly on technical requirements for SAIs when conducting specific types of audits, highlighting the importance of having subject matter expertise related to technology, data and analytics. For instance, the Compliance Audit Standard (ISSAI 4000) highlights the potential for auditors to use analytics as part of risk analysis or evidence collection. It also highlights the possibility, depending on the type of engagement, for audit teams to have data collection and analysis as critical skills within the team (INTOSAI, 2019[77]).

In the context of public procurement, INTOSAI guidance (GUID 5280) also promotes the use of technology and data as fundamental aspects of auditing this sector. It notes that SAIs can use data, analytics and artificial intelligence (AI) to analyse contract data and assess patterns, trends and risks. The guidance also recognises the challenges SAIs may face in doing this from the perspective of recruitment, training and access to data (INTOSAI, 2022[78]). Other INTOSAI guidance, such as the Guidelines for the Audit of Corruption Prevention (GUID 5270), imply a need for auditors to have a sufficient understanding of technology to effectively audit anti-corruption systems and entities (INTOSAI, 2019[79]). For instance, it emphasises the importance of information and communications technology (ICT) systems and procedures, as well as safeguards to ensure data privacy, as critical areas that auditors may wish to consider during an engagement.

Beyond standards and guidance, the SAI community has long recognised the need to keep pace with technological advancements. One example of this is the 2019 Moscow Declaration (named after the location where the XXIII INTOSAI Congress (INCOSAI) took place that year). According to the Declaration, rapid data accumulation and evolving technology present both opportunities and challenges for SAIs. It calls on SAIs to respond effectively to technological advancements, including promoting the availability and openness of data, source code, and algorithms, as well as making better use of data analytics in audits (INTOSAI, 2019[80]). Annex Box ‎2.B.1 provides a summary of relevant standards and guidance that support technological advancement in SAIs.

Annex Box ‎2.B.1. Key standards promoting technological advancements in SAIs

The following is a non-exhaustive list of key standards, principles and guides for SAIs to keep pace with technologies in government and overcome any internal inertia that may exist that slows or inhibits digital transformation or optimisation. Some SAIs focus primarily on what and how SAIs should audit in government from the IT perspective, but they provide additional context and encouragement for SAIs to lead by example, as promoted by the Lima Declaration (INTOSAI-P-1). Similarly, while not a standard, the Moscow Declaration outlines that in an era of rapid technological transformation, SAIs must strive to transform themselves to enhance the value and benefits they bring to citizens. It acknowledges that embracing new technologies is central to improving the effectiveness of public auditing. By adopting advanced technologies, SAIs can better handle the increasing volume and complexity of data, conduct more thorough analyses, and make more informed decisions.

ISSAI 4000—Compliance Audit Standard. Promotes the need for auditors to have the necessary competencies, including skills regarding data collection and analysis. It also highlights the value of assessing risks and collecting audit evidence through statistical and data-driven techniques.
ISSAI 5100 – Guidance on Audit of Information Systems. Provides guidance for auditors on how to conduct performance and compliance audits related to Information Systems or where the audit of information systems may be part of a larger audit engagement.
GUID 5270 – Guideline for the Audit of Corruption Prevention. Provides additional guidance for SAIs on preparing and conducting audits of anti-corruption policies and procedures within government organisation.
GUID 5280 – Guidance for audits of public procurement. Provides guidelines on conducting audits of public procurement, and highlights the critical role of data and analytics for such audits.
GUID 5290 – Guidance on Audit of the Development and Use of Key National Indicators. Discusses key national indicators and their importance in assessing the impact and effectiveness of public policies and programmes, with a focus on statistical and econometric techniques for improvement.
Guidance on conducting audit activities with data analytics. Offers guidance for SAIs in producing high-quality audit products using cutting-edge technologies, particularly focusing on data analytics for audit activities.
Data Analytics Guideline. Prepared by the INTOSAI Working Group on IT Audit, it details the processes from data maturity to analytics creation and its deployment in audit scenarios. It aims to provide a comprehensive understanding and application of data analytics in audit contexts.
Integrating big data in public sector audit. This publication of AFROSAI-E provides insights into the background, context, and uses of big data in audit, along with the factors affecting its analysis.
Capacity building requirements for conducting IT audits - Provides guidelines on building capacity for conducting IT audits and outlines a model for building IT audit capacity in SAIs.
Managing Information Communications Technology: A guide for SAIs. Provides senior managers in SAIs in understanding what constitutes modern ICT and helps identify the key aspects that comprise a modern ICT capability, recognising varying levels of maturity.

Sources: (INTOSAI, 2019[81]; INTOSAI, 2019[82]; INTOSAI, 2019[83]; INTOSAI, 2019[84]; AFROSAI-E, 2020[85]; Mehrishi, 2022[86]; INTOSAI, 2022[87]; INTOSAI, 2015[60]).

Notes

← 1. For instance, see the discussion on “Establishing quick wins and realistic expectations” in OECD (2019), Analytics for Integrity: Data-Driven Approaches for Enhancing Corruption and Fraud Risk Assessments, https://www.oecd.org/gov/ethics/analytics-for-integrity.pdf; and the INTOSAI Development Initiative (2020), Strategic Management Handbook For Supreme Audit Institutions, https://www.idi.no/elibrary/well-governed-sais/strategy-performance-measurement-reporting/1139-sai-strategic-management-handbook-version-1/file.

← 2. See https://www.nao.org.uk/insights/quality-assurance-of-models/?utm_source=LinkedIn&utm_medium=social&utm_campaign=Orlo and

https://www.nao.org.uk/wp-content/uploads/2016/03/11018-002-Framework-to-review-models_External_4DP.pdf

← 3. auditingtobenefitthesocietyoftomorrow.pdf (riksrevisjonen.no)

← 4. Algorithmic Impact Assessment Tool - Canada.ca

← 5. https://cdn.www.gob.pe/uploads/document/file/2008919/El%20control%20concurrente:%20estimando%20cuantitativamente%20sus%20beneficios.pdf.pdf?v=1630447278

← 6. The 8 Steps for Accelerating Change, previously know as the 8 Steps for Leading Change, was pioneered by John Kotter, a professor of Harvard Business School and internationally recognised as a leader in change management. The four principles

← 7. https://intosaijournal.org/journal-entry/data-auditing-strategy/

← 8. https://english.rekenkamer.nl/about-the-netherlands-court-of-audit/news/2022/09/28/netherlands-court-of-audit-publishes-online-course-on-data-analytics-in-audits

← 9. Detailed under articles 132, 163, and 164 of the CPC.

← 10. Control Objectives for Information and Related Technologies (COBIT) is a framework for the governance and management of enterprise information and technology, aimed at the whole enterprise. See https://www.isaca.org/resources/cobit.

← 11. The TdC also explored the Supreme Audit Institutions - Information Technology Maturity Assessment (SAI ITMA) tool, developed by the German co-operation (GIZ). The TdC decided not to use ITMA for several reasons, including the tool’s complexity, its requirement for an external assessment team, and its primary focus on digitalization and dematerialisation as opposed to digital transformation, which was the TdC’s priority.

← 12. For instance, see: OECD (OECD, 2023[98]), “Recommendation of the Council on Artificial Intelligence” https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449; OECD (2014[100]), “Recommendation of the Council on Digital Government Strategies”, https://www.oecd.org/gov/digital-government/Recommendation-digital-government-strategies.pdf; OECD (OECD, 2020[99]), The OECD Digital Government Policy Framework: Six dimensions of a Digital Government; OECD (2022[56]), Strengthening Analytics in Mexico’s Supreme Audit Institution: Considerations and Priorities for Assessing Integrity Risks; OECD (OECD, 2021[40]), Countering Public Grant Fraud in Spain: Machine Learning for Assessing Risks and Targeting Control Activities; Otia and Bracci (Otia, 2022[53]), Digital transformation and the public sector auditing: The SAI's perspective; and Bumann and Peter (2019[52]), Action Fields of Digital Transformation - A Review and Comparative Analysis of Digital Transformation Maturity Models and Frameworks.

Featured topics

Agriculture and fisheries

Climate change

Development

Digital

Economy

Education and skills

Employment

Environment

Finance and investment

Governance

Health

Industry, business and entrepreneurship

Regional, rural and urban development

Science, technology and innovation

Society

Taxation

Trade

Energy

Nuclear energy

Transport

Featured topics

Agriculture and fisheries

Climate change

Development

Digital

Economy

Education and skills

Employment

Environment

Finance and investment

Governance

Health

Industry, business and entrepreneurship

Regional, rural and urban development

Science, technology and innovation

Society

Taxation

Trade

Energy

Nuclear energy

Transport

Countries A - C

Countries D - I

Countries J - M

Countries N - R

Countries S - T

Countries U - Z

Regional and global engagement

Countries

Countries A - C

Countries D - I

Countries J - M

Countries N - R

Countries S - T

Countries U - Z

Regional and global engagement

Publications

Publications

Featured publications

Data

Data

Featured data

News & Events

News & Events

Featured Events

About OECD

About

Engage with us

Work with us

Featured topics

Agriculture and fisheries

Climate change

Development

Digital

Economy

Education and skills

Employment

Environment

Finance and investment