Insights from behavioural sciences can inspire several measures to increase the likelihood of the uptake of the audit reports issued by the CGR by the auditees. This chapter provides concrete recommendations to the Chilean SAI on measures that could be included during the auditing phase, measures that could enhance the audit reports and measures to motivate or build peer or social pressure during the follow-up process. A strategic perspective that combines several measures is most likely to generate impact and to contribute significantly to enhancing the relevance of CGR’s audits.
Enhancing the Oversight Impact of Chile’s Supreme Audit Institution
Applying Behavioural Insights for Public Integrity
OECD Public Governance Reviews
3. Behaviourally-informed strategies to strengthen the uptake of audit reports in Chile
Abstract
Introduction
The analysis carried out showed that behavioural biases and barriers are relevant to understand why public officials in the audited services in Chile may fail to address audit observations issued by the Comptroller General of the Republic (Contraloría General de la República, CGR). Figure 3.1 provides a summary of the relevant behavioural aspects presented and discussed in Chapter 2. The behavioural insights lens applied throughout the auditing process could contribute significantly to improve the uptake of audit reports and therefore the impact of the CGR.
Figure 3.1. Summary of main challenges to the uptake of audit reports in Chile
In fact, as mentioned previously, considering behavioural dimensions in auditing processes is not new (Kida, 1984[1]; Kinney and Uecker, 1982[2]; Kassin, Dror and Kukucka, 2013[3]). Various guidelines and standards refer more or less explicitly to behavioural dimensions. For instance, Box 3.1 displays good practices that can influence the impact of SAIs reported by EUROSAI; they contain several relevant recommendations that can inspire interventions to promote behavioural change. INTOSAI also provides guidance for SAIs on improving the use and impact of audit reports and emphasises the following aspects that are all relevant from a behavioural perspective (INTOSAI, 2010[4]):
Salience: When selecting audits, make sure that topics are useful and consult stakeholders before starting an audit;
Quality assurance and clarity: Building in quality throughout the audit process in line with ISSAI standards, and ensuring that auditees know what to expect. Audit reports should be written using plain language and short sentence structures and include an executive summary to highlight main findings;
Communication: Communicate results of audits clearly and effectively and invite auditees and stakeholders to give feedback on audit reports;
Follow-up: Find out if progress has been made through systematic tracking and monitoring of uptake of recommendations, and where necessary, initiating follow-up audits when no progress has been made.
Box 3.1. Good practices and corresponding factors that influence SAI’s impact
In the context of improving the implementation of audit recommendations to achieve impact, a recent EUROSAI report defines a good practice as a procedure (legal or other), method or other practice that can contribute to the appropriate and timely implementation of audit recommendations issued by the SAI. As such, EUROSAI reformulates the influencing factors on impact as a chronological listing of good practices throughout the audit process (Table 3.1).
Table 3.1. Good practices and corresponding factors that influence impact
Source: (EUROSAI, 2021[5]).
To clarify where the CGR could introduce behaviourally inspired measures, Figure 3.2 simplifies the follow-up process described in Chapter 2. The Figure shows the actors whose behaviours could be targeted to generate the desired impact and underscores the value added of looking at the whole process from a behavioural perspective.
Figure 3.2. Entry points for potential behavioural interventions to improve audit uptake in Chile
Narrow behavioural interventions aimed at nudging a behaviour of one of the actors in a specific step of the audit process as depicted in Figure 3.2 may succeed in changing this target behaviour, but may fail at generating an impact at the level of the uptake of audit recommendations. This impact typically depends on several behaviours of various actors.
Figure 3.3 provides a simplified overview on the underlying rationale. Suppose that there are several relevant behaviours (B) throughout an audit process that are relevant for explaining its impact. A behavioural analysis may have identified one behaviour, e.g. B5, and achieved to change this behaviour, for example through a nudge. However, this change alone may not be sufficient to impact on the overall objective to improve the uptake of the audit if the other behaviours, or at least several of them, are not addressed as well. Therefore, impact is likely to be stronger and more sustainable if the CGR implements several adjustments throughout its audit processes.
Figure 3.3. To generate impact, behavioural changes at different levels may typically be required
This idea of implementing complementary measures aiming at changes at different steps of the audit process, as also emphasised by EUROSAI in the Box above (EUROSAI, 2021[5]), is reflected in the following sections. The analysis identified six behaviourally inspired strategies that could help promoting change leading ultimately to impact in the uptake of CGR’s audit reports. According to the priorities, resources and opportunities, the CGR could consider combining several selected measures proposed in these strategies.
The six strategies, that will be presented in the following sections, are:
Implement measures to improve the auditor-auditee relationship throughout an audit cycle by increasing interactions and making these more constructive.
Promote a cultural change within the CGR towards more supportive attitudes towards auditees by building relevant internal capacities, by targeting supervisors and leaders and by making good practices visible.
Review the drafting of the audit reports to make them more user-friendly by simplifying them and by providing more guidance.
Improve the response of auditees by testing different messages to notify about the audit report.
Target frustration and promote better quality responses by allowing for a better planning, by introducing some flexibility with the deadlines and by tracking advances.
Creating public or peer pressure through a reporting of the follow-up results.
Achieving impact through measures targeting auditors of the CGR
As emphasised in Chapter 2, the behaviour of CGR auditors while conducting an audit can affect the behaviour of the auditees with respect to the implementation of the audit observations. In fact, as mentioned previously, unconscious biases of auditors can lead them to be overly critical and orient their search design towards evidence confirming their prior beliefs (Pennington, Schafer and Pinsker, 2017[6]).
In Chile, anecdotal evidence of fraud and corruption or mismanagement could indeed frame CGR auditors towards a negative attitude that is likely to affect both the quantity of observations and a constructive and open attitude on the receiver side, thereby reducing the likelihood of taking corrective actions. As described above, social norms in the CGR can exacerbate this vicious cycle. The following measures therefore aim to improve the auditor-auditee relationship by more and more constructive interactions and by promoting a cultural change within the CGR towards a more supportive attitude.
Implement measures to improve the auditor-auditee relationship throughout an audit cycle by increasing interactions and making these more constructive
Interviews evidenced that interactions between auditors and public managers are scarce and usually quite formal. Currently, the involvement of public managers is limited to the kick-off meeting, if they are available, and when presenting the pre-report. At the same time, it was flagged that for the uptake of audit reports, it would be key to involve public managers more actively to overcome the inherent aversion against change, seeing own errors and prevent pro forma disagreement with the observations. In addition, the qualitative analysis showed that at least some audit observations relate to issues that auditees could have fixed on site if the auditor would have communicated with the auditee.
Potential measures
In general, the CGR could gain by investing in building a continuous relationship with auditees beyond the context of specific audits. A better relationship between auditors and auditees in general is likely to promote positive reciprocity between the actors. In the Netherlands, regular meetings between the SAI and the assembled heads of internal audit allow to discuss audit plans and risk analysis, for example. Belgium’s SAI conducts informal meetings at the beginning and the end of the audit process.
Chile therefore could consider establishing similar practices to generate better interactions between auditors and auditees before, during and after the on-site audit process.
Improve interaction before the audit. Engaging auditees before the audit to explain its rationale is likely to increase both the willingness to co-operate and the motivation of auditees. The CGR could review the dynamics of the current kick-off meeting to allow for a more constructive approach and to address potential feelings of being treated unfairly. The kick-off meeting provides the opportunity to explain the underlying standards and the rationale for the audit as well as to clarify the audit criteria and the risk-based methodology used to determine the CGR’s annual audit planning (Box 3.2). Auditees that have the feeling of being audited too often could see the strategic value of the upcoming audit as an opportunity to reverse this trend and to reduce their level of risk by addressing the audit observations. A positive setting during the meeting could contribute to striking the delicate balance between maintaining independence and building mutual empathy.
Box 3.2. Informing the auditee what to expect during the executions of an audit
Auditees can be involved during the execution of an audit by setting up communication protocols between the SAI and the auditee that identify the following:
The responsibilities of the SAI and the auditee
Guidance on key stages of the audit process
Which documents will be shared between the SAI and auditee
How the SAI will let the auditee know about upcoming audits
Information about the audit plan, timeframe, audit methodology
Type of information and access to be provided by auditee
How and when emerging findings will be shared
When the auditee will receive a copy of the draft report
Increase interaction during the on-site audit process: The potential behavioural impact of increasing reciprocity between the actors could be maximised by introducing feedback and interaction steps all along the audit process ("tit for tat" logic). For instance, engaging auditees at the beginning will be more efficient if auditees know they can reciprocate later in the process during follow-up meetings. For instance, the CGR could develop a protocol for auditors providing guidance on how to create more spaces of interactions with auditees in the field before drafting the report. Auditors could present preliminary results of the audit to managers for their feedback and thereby build a constructive relationship. The CGR already has internal good practices on which to build on (Box 3.3) and are currently considering to require as a minimum two meetings between auditors and auditees during an audit. In addition, the CGR has already instructed auditors to allow auditees to correct directly, if possible, minor observations. This approach could be further supported by providing guidance to the auditors on how to provide ad hoc support to the auditees in the field. Making such advocacy behaviour by auditors more salient within the CGR and perhaps establishing incentives for it, also contributes to changing the culture of “the more observations the better” and to take a more supportive attitude towards auditees (see the following section).
Box 3.3. Auditing the SDG’s in Chile: Good practice to communicate with auditees
The CGR in Chile recognises the value of an improved communication during the audit process to achieve improvements in the processes of the audited services, which should be the ultimate objective of any audit.
While audits of national entities provide a single opportunity for feedback by auditees during the pre-report stage, the team of United Nations auditors of the CGR follows a different methodology. Prior to the publication of the final audit report, several memos (Audit Observation Memorandum, AOM) are delivered to the auditees informing them of the preliminary observations, which allows them to respond and clarify what observed. Later, all AOM are consolidated in a document called Management Letter, which reflects the findings, the respective analysis of the auditees’ response to the AOM, the conclusion of the audit team to the observed facts, the recommendations and the client's position regarding these recommendations, indicating whether the auditee accepts or rejects the recommendations suggested. The document includes an annex with the status of implementation of the pending recommendations.
Source: Information provided by the CGR Chile.
Improve interaction after the on-site audit process. The CGR could discuss preliminary findings with auditees during an exit meeting. The discussion could focus on clarifying doubts and on getting feedback concerning the feasibility of corrective actions from the auditees. For the design of such meetings, the experiences from Brazil and Belgium could be particularly interesting (Box 3.4).
The exit meeting could take place before issuing the final report. Changes agreed upon during this exit meeting could be included in the final audit report. Being able to influence the final report is likely to have a positive effect on the ownership of the report by the auditees. A recent EUROSAI report finds that some SAIs in Europe ask the auditees explicitly to indicate whether they agree, partly agree or disagree with each recommendation. Those SAI then have to review rejected recommendations on a higher administrative level. Some SAIs with a jurisdictional model, mainly in the area of compliance audits, even negotiate the recommendations with the auditee to avoid non-feasible or non-realistic recommendations. In case of a non-agreement, auditees can formulate a complaint against the audit report (EUROSAI, 2021[5]).
Alternatively, and more relevant in the Chilean context, the CGR could consider a meeting between CGR and the audited service after issuing the final audit report but before starting the follow-up process. The objective of the meeting would be to explain the observations, clarify doubts and to recommend certain lines of actions the management could take to address them. Chapter 4 provides a more detailed theory of change for this measure and guidance for the implementation of a potential pilot.
Box 3.4. Post-audit exit meetings in Brazil and Belgium
The Office of the Comptroller General of the Union (Controladoria-Geral da União, CGU), responsible for internal audits in Brazil, carries out a meeting with auditees to discuss the findings and together find potential solutions (“reunião de busca conjunta de soluções”). The CGU auditors present the findings and decide together with the auditees if these observations can be addressed directly, and, if that is the case, the observations can be eliminated from the audit report.
In Belgium, the SAI (Rekenhof) has established informal exit meetings after the audit process and before writing the report, where the auditor presents the recommendations that will be included in the report. According to information provided by the Belgian SAI, these informal exit meetings seem to improve the acceptance of the reports by the auditees.
Source: OECD, based on information provided by the CGU and the Belgian Rekenhof.
Promote a cultural change within the CGR towards more supportive attitudes towards auditees
As described in Chapter 2, social norms shaping the organisational culture of the CGR affect the interactions between auditors and auditees. The informal norm according to which a good auditor produces many findings and observations still seems to dominate within the CGR and influences the behaviours of auditors. This norm also undermines the idea of allowing auditees to address some issues directly on the spot, as these then will not be reflected in the audit report anymore. More generally, the norm tends to favour a critical and negative view with respect to auditees. It primes auditors that behave “severely” with auditees leading, again, to more observations and/or to maintain observations despite good arguments brought forward by the auditees.
Shifting social norms to impact on organisational cultures is complex, takes time and is an incremental process. To achieve change, the application of behavioural insights to organisations consists in influencing specific individuals in those organisations to affect organisation-wide changes or in directly intervening on organisational routines, policies, and procedures of the organisation, as emphasised in recent work of the OECD on safety culture in organisations (OECD, 2020[7]). This work also provides an overview on some key theoretical foundations and insights from organisational psychology when it comes to influencing organisational behaviour (Box 3.5).
Box 3.5. Applying Behavioural Insights to Organisations: Theoretical Foundations
When enough people are nudged toward behavioural change, those new behaviours have the potential to become habit, switching from deliberate choices and actions otherwise known as controlled processing, to less deliberate, less effortful, more habitual actions known as automatic processing. Whether deliberate or effortful, choice or habit, when enough people in a work group or entire organisation behave in a certain way, that behaviour has the potential to become a norm. Norms are rules for expected and accepted behaviour. As humans, violating norms tends to make us uncomfortable. We are likely to conform to the norms of our work group and organisation. This is especially true of cohesive groups who feel a degree of attraction to their work group.
Nudging supervisors or other powerful or influential people within an organisation can have a multiplying effect such that the behaviours exhibited and endorsed by influential individuals have a better chance of being adopted en masse, nudging a whole organisation in the process. Indeed, charismatic and transformational leaders are believed to possess qualities that inspire followers to behave in desired ways in service of a larger goal. Nudging such leaders can effect largescale behavioural change.
Of course, those in formal leadership roles toward the top of the organisational hierarchy are also in a good position to effect widespread behavioural change by altering organisational policies and procedures. Nudges that help high-level decision makers (leaders, boards, etc.) optimise organisational policy decisions in the face of their own biases and irrationalities can have an effect. Thus, helping decision makers see the connection between policies, procedures, and behaviour on the ground is another way to nudge whole organisations.
Source: (OECD, 2020[7]).
Potential measures
The CGR could consider implementing measures aimed at building capacities for strengthening a constructive audit approach and work towards transforming the organisational culture to promote a change in the behaviour of auditors. This could be achieved, for example, through the following measures.
Improve capacity building. The CGR has an extensive internal policy for building capacities and promoting continuous learning. The trainings focus on technical auditing skills, such as legal aspects or accounting practices, and some are targeting “soft skills”. To complement these efforts, the CGR could consider piloting a training for auditors aimed at building skills related to strengthening an “advocacy attitude”, creating a constructive dialogue and preparing them to explain the rationale of an audit process to auditees in clear and friendly terms.
Promote cultural change. To complement the training programmes, the CGR could aim at nudging a critical mass of officials in the CGR such that the new desired behaviour becomes a social norm in the institution (OECD, 2020[7]).
Knowing how others behave can be a powerful driver for own behaviour. For instance, in the United Kingdom, the Chief Medical Officer sent a letter to selected general practices notifying them that they were prescribing more antibiotics than 80% of the practices in their local area. As a result, 73 406 fewer prescriptions were made across 791 intervention practices, compared to the control group of 790 practices (Hallsworth et al., 2016[8]). Similarly, auditors could receive messages highlighting behaviours of other auditors that are more advocacy-focused.
Nudging leaders, those with formal or informal influence, is a way to nudge entire organisations (OECD, 2020[7]). The Comptroller General, together with the senior leadership of the CGR, could lead an internal awareness-raising campaign thematising and addressing the issue of the informal culture and its negative repercussions on generating impact while emphasising the potential of an organisational culture that supports change in the public administration.
Supervisors and heads of units in the middle management are setting an example by their own behaviour and by providing guidance to their teams and communicating to them (OECD, 2020[9]). Middle management could be sensitised to the behavioural challenges faced by auditees and trained to transmit these messages to their teams during day-to-day practice, performance evaluations or, for example, in the context of a mentoring programme.
Achieving impact through measures targeting the drafting and communication of the audit reports
Audit reports are the main official vehicle to communicate the audit findings and recommendations. SAIs that focus on clarity and drawing crosscutting conclusions will help to focus the minds of public leaders and managers. For example, useful SAI tools could include sector-based reports, systematic use of executive summaries, tagging key words or providing findings in a systematic way that allows for text and data mining (OECD, 2016[10]). SAIs are already encouraged to make their audit reports accessible and concise (INTOSAI, 2010[11]) and use a more positive tone. For instance, the revised government auditing standards of the United States Government Accountability Office (GAO) state the following:
“The report may recognize the positive aspects of the program reviewed if applicable to the audit objectives. Inclusion of positive program aspects may lead to improved performance by other government organizations that read the report. Audit reports are more objective when they demonstrate that the work has been performed by professional, unbiased, independent, and knowledgeable personnel” (GAO, 2018[12]).
Therefore, the CGR could review the way audit reports are drafted and promote a new internal standard that takes into account behavioural dimensions of the receivers. Communicating this new standard contributes to make it salient within the organisation. Currently, the CGR audit reports contain a lot of information and focus extensively on explaining the audit methodology. In fact, some interview partners suggested that there might be a culture amongst public managers of not reading the audit reports. Overall, the CGR could benefit from reviewing both the content and the communication of the audit reports with a view to generate the expected change in auditee’s behaviour.
Review the drafting of the audit reports
The analysis showed that the language of the audit report as well as the quantity (and sometimes quality) of observations may trigger a negative attitude towards the audit results and delay taking corrective actions. Auditees reported feeling sometimes overwhelmed by the observations and stated that they do not receive enough guidance. The measures to improve the auditor-auditee relationship could address part of the problem by clarifying and reducing the quantity of observations during the audit phase.
In addition, shorter reports focused on the findings and providing recommendations for addressing the observations in a simpler and less technical-legal language could facilitate the understanding, increase the motivation to act and thereby improve the impact of the reports. For example, Ireland’s report on applying behavioural science in the tax administration explores how compliance can be improved through better and simpler presentation of information and by drawing attention to key facts (Customs, 2017[13]). Ideally, auditors write their reports with the auditee in mind (Box 3.6).
Box 3.6. Writing audit reports with audience awareness
The content of audit reports should be organised by its importance to the audit client. To them, seeing and understanding the results is more important than knowing how the results were found. According to the authors, audit reports should not all follow the same template. Decisions about what to include, what to leave out, and how to organise the report should be made based on awareness of the audience. Writing with audience awareness will help auditors overcome the task-oriented mind-set that results in audit-focused reports (Table 3.2).
Table 3.2. Recommendations to writing audit reports with audience awareness
Source: (Cassels, Alvero and Errington, 2009[14]), adapted from Carol Berkenkotter’s “Understanding a Writer’s Awareness of Audience,” College Composition and Communication, Vol. 32, No. 4 (December 1981), 388–399.
Potential measures
The CGR could continue improving the drafting of audit reports to make them more impactful. For example, this could be achieved by:
Reducing the technical and legal language. Plain language means avoiding jargon or obscure words whenever possible. It also means explaining any technical or legal terms that are used to make audit reports more readable, especially those reports that address complex subjects. Exhibits, including illustrations, tables, charts or text boxes can help to attract a stakeholder’s attention and reinforce key points (OECD, 2014[15]). As such, the CGR could consider running an experiment testing a sample of auditee's responses to hypothetical language changes using, for example, an experimental vignette methodology (EVM) (Aguinis and Bradley, 2014[16]; Atzmüller and Steiner, 2010[17]).
Reducing the size of the audit report. Shorter reports are easier to read and could improve their relevance for the auditees, while potentially also reducing the time and costs involved in preparing the audit reports. Shorter reports focus on results and not on describing the details of the audit process. For example, legal and technical details could be provided as an annex to the main report. In addition, while all audit reports currently include executive summaries, the CGR could consider focusing the messages included in these summaries to outline the strategic value and objectives of the audit report in plain language.
Reducing, where possible, the quantity of observations in the reports. Reports could, for example, group observations by project or process or emphasise underlying issues of which the observations are merely the symptoms. While the number of observations would remain the same, the logical packaging in bundles could make the report easier to digest and to see the strategic value. In addition, auditors could draw general conclusions with respect to the underlying weaknesses in the audited entities to explain the audit findings and hence the observations. As one CGR auditor stated: “We should diagnose the sickness and not just describe the symptoms.”
Include recommendations for corrective actions. The CGR recently began to promote the inclusion of recommendations for corrective actions. This is a step towards providing clearer guidance to auditees, while paying caution no to co-administrate. The CGR could thus further promote the drafting of observations that are more action-oriented, e.g. by providing trainings to auditors. In addition, auditors should aim at emphasising the rationale underlying the observations to make it easier for public managers to understand their relevance and why they should care. Finally, observations and proposals for corrective actions should be drafted using plain, simple and straightforward language to become SMART recommendations (Box 3.7).
Box 3.7. Drafting SMART audit recommendations
Audit recommendations should be specific, measurable, achievable, realistic and timely (SMART):
S: Specific recommendations correctly identify who is responsible for taking clear, corrective action to eliminate a deficiency and improve a public programme or activity.
M: Measurable recommendations can be tracked by the SAI’s follow-up system. SAIs can determine whether corrective actions were taken and deficiencies were eliminated. When implemented, recommendations should prevent recurrence of findings and allow the SAI to conduct an independent audit of the new conditions in the public programme.
A: Achievable recommendations are doable in a reasonable period of time with available financial resources.
R: Realistic recommendations recognise the priorities and operating constraints of the officials who are responsible for implementing them.
T: Timely recommendations are provided to responsible officials at the right moment and manner to facilitate their prompt implementation.
Source: OECD.
Testing different messages to notify about the audit report
The findings showed that there may be negative predisposition, frustration or lack of interest by different actors in the audited service. In addition to improving interactions between auditors and auditees during an audit and the drafting of the audit reports, the CGR could consider framing the messages contained in the formal communications by the CGR.
Indeed, evidence from several behavioural insights applications around the world indicate that the way messages are framed can affect significantly the responses of the recipients (OECD, 2017[18]). For instance, an intervention carried out by the United Kingdom’s Financial Conduct Authority (FCA) to promote compliance could inspire a similar intervention by the CGR targeted to audited services (see Box 3.8 below).
Potential measures
The CGR could consider an experiment to test which message generates the highest desired impact. For instance, the CGR could send different versions of the letters that are joined to the audit reports to a sample of services, stratified according to characteristics and then randomly selected to treatment and control groups to determine the messages that are most suited to promote compliance with observations.
Potential letters could be framed along the following messages:
Highlight potential negative consequences for the Service.
Highlight peer / social pressure ("X% of the services already complied with ...").
Highlight positive associations such as trust, pride or commitment.
Highlight the individual responsibility of the head of the service and/or Director of Internal Control.
Box 3.8. Improving submissions to the Financial Conduct Authority by mutual societies
United Kingdom’s Financial Conduct Authority (FCA) tested specific messages when communicating with societies, which would draw their attention and encourage them to submit their annual return and accounts on time. Between 2013 and 2014, the FCA sent letters to a sample of 7 984 societies who were stratified by type of organisation, the month of their financial year-end and the last year they took action, and then randomly allocated to different treatments or to a control group. The treatments were:
Bullets: Including salient bullet points and a message about penalties: “Last year mutual societies like yours were fined up to GBP 3 000 for failing to provide this information on time”
Warning: Adding a warning to the envelope: “It is a legal obligation to complete and return the enclosed form”, and
Timing: Sending the letters on different dates (26 May, 3 June or 8 July). This helped the FCA to estimate the effects of the length of time between the letter and the deadline on compliance.
Of the societies in the trial, 6 456 took action (80.9%), while 1 528 (19.1%) did not take any action. Examining each treatment, the FCA found that the bullets and warning treatments failed to change the behaviour of societies compared to the control. However, the timing of the letters did affect the firms’ response. Societies who received a letter in July (and who therefore had, on average, a shorter deadline – median of 23 days for July rather than 66 for May and 58 for June) were 2.4 percentage points more likely to respond to the communication. Across all groups, those with a shorter deadline were more likely to respond than those with a longer deadline.
Source: (OECD, 2017[18]).
Achieving impact through measures during the follow-up process
Target frustration and promote better quality responses by allowing for a better planning, by introducing some flexibility with the deadlines and by tracking advances
The findings showed that deadlines may cause stress and frustrations and sometimes can be unrealistic or not adapted to the reality of the public administration. Currently, the deadlines to take corrective actions are limited and standard (15, 30 or 60 working days). In some cases, however, deadlines are not sufficient to introduce changes in institutional processes. This, added to the fact that many times the follow-up is carried out long after the deadline has expired, generates a feeling of unfairness as the services consider that they would have had more time to deliver a better response. In addition, interviews with public managers evidenced that managers would appreciate an opportunity to report partial progress made in addressing observations or in improving processes that may lead to prevent future observations.
EUROSAI emphasises that “defining deadlines in agreement with the auditee contributes to a good relationship and generates support for the implementation of the recommendations” (EUROSAI, 2021[5]). In the survey conducted by EUROSAI, out of 19 SAIs who set deadlines, nine determine the deadlines in agreement with the auditee, three let the auditees set their deadlines. Deadlines are fixed independently by six SAIs. This mostly applies to SAIs with legally binding recommendations.
Potential measures
In line with the previous recommendations, the CGR could promote a sense of support rather than strict control of the audited service by transmitting the idea that what really matters is improving public management and processes in the audited entities, not the fact of complying with deadlines.
As such, to reduce stress of non-compliance and motivate public managers, the CGR could consider the following measures:
When discussing the pre-report or when discussing the follow-up process, the CGR could allow for some flexibility concerning the deadlines for addressing the observations case-by-case, involving the Internal Auditors or the Directors of Internal Control and the public managers responsible for the processes with observations. The decision to adapt deadlines would need to respond to clear and homogenous pre-established criteria to ensure fairness and could show flexibility and improve the willingness of public managers to take action, as they will be personally committed to the deadline. Chapter 4 provides more details on this measure in view of carrying out a potential pilot.
During the interviews, auditees proposed to create a mechanism through which services can report partial progress or planned actions to address observations. Such a mechanism could also provide an opportunity to report difficulties inherent to the nature of the process related to the observation. For instance, the CGR could consider including intermediate corrective actions with specific deadlines. In this way, auditees could communicate progress made in resolving an observation.
Sometimes, public managers do not meet deadlines because of the workload or because they simply forgot about it. As described in Chapter 2, the CGR online follow-up system automatically generates reminders before deadlines are due. Nonetheless, the CGR could test different reminders created by the online follow-up system in view of finding the most effective nudges to take actions.
According to the interviews, the stress related to the deadlines results sometimes from the fact that auditees do not know when exactly to expect the audit report. This makes it difficult for them to plan. Therefore, the CGR could consider notifying the services well in advance of sending the audit pre-report. The OECD already recommended that the CGR could provide information on when the audited entity could expect to receive the preliminary audit report for comments at the beginning of the audit process (OECD, 2014[15]). The CGR could review its internal drafting and review process to identify a step in this internal process that could allow sending, well in advance, a message communicating at least a likely timeframe of when the pre-report will be sent to the service.
Creating public or peer pressure through a reporting of the follow-up results
Arguably, the performance of audited services in addressing observations indicate their willingness to improve and to comply with regulations and procedures. As mentioned previously and evidenced in Figure 1.4 in Chapter 1, 65.2% of SAI’s in OECD countries with available data publicly report on actions taken by the executive to address audit recommendations. In Latin America, only 25% of countries do so. Chile is not amongst these countries. Indeed, research has shown that providing public information on audit reports can significantly reduce the probability of re-electing a mayor in which at least two violations associated with corruption were reported (Ferraz and Finan, 2008[19]; Avis, Ferraz and Finan, 2018[20]). A recent EUROSAI report based on a survey also emphasises that reporting the follow-up results to a broader audience can raise the pressure on the auditees in case of insufficient implementation or provide positive incentives to well-performing auditees (EUROSAI, 2021[5]). Out of 33 respondents, 20 SAI in Europe do publish a global report on the follow-up of audit reports (Figure 3.4). Finally, INTOSAI recommends that the results of SAIs’ follow-up exercises should be communicated publicly, unless regulations stipulate otherwise (INTOSAI, 2010[4]).
Figure 3.4. In Europe, the majority of SAIs issue reports on follow-up of audit reports
Does your SAI make a global report on the follow-up of recommendations/reports and is it communicated to external stakeholders?
Note: 33 respondents.
Source: (EUROSAI, 2021[5]).
Potential measures
In Chile, interviews emphasised that the heads of services typically have only very limited interest and incentive to internally follow-up on audit reports and ensure the implementation of corrective actions. Therefore, to provide stronger incentives, the CGR could consider making non-compliance with audit reports more salient to heads of services and/or make this information visible to the public to facilitate accountability.
The CGR could test the impact of adding information to the letters sent with the audit reports on how the service has performed in taking corrective actions over the past years. This information could be non-public, directed only to the highest authority of the service. Some Offices of Inspector General (OIG) in the US provide management reports signed by the head of the OIG and sent to the head of the agency, which are tracking open issues and are holding leadership accountable.
The CGR could make non-compliance more visible to the public. To do so, the CGR could build on their good practices in transparency and in communicating with citizens (OECD, 2014[15]; OECD, 2016[21]). At municipal level, for example, the reports of the CGR may trigger social accountability processes asking for the reasons of non-compliance. In turn, including recognition of services with better behaviour or providing information to the public regarding services that perform better in addressing observations could have a political value for heads of services and set incentives to introducing improvements to their institutional processes.
The CGR could explore possibilities to promote some kind of benchmarking to promote yardstick competition between services. However, due care must be taken to ensure that the services are compared with other services that are truly similar, for instance in terms of size, level of risks and budget, to avoid any negative impact on the perceived fairness of such a comparison. This is particularly important taking into account the quantitative analyses in Chapter 2 showing a high concentration of observations in a few entities. Therefore, such a benchmarking could be relevant for the municipal level in particular, as municipalities are comparable in terms of the public services they provide and could be clustered by size and budget, for example.
Finally, while most countries in LAC and the majority of OECD countries do not require the executive to publicly report on the measures taken to address the audit recommendations (Figure 3.5), Chile could consider requiring services to report publicly on the steps they have taken to address all, most or some findings in the audit reports.
Figure 3.5. In Latin America, the executive reports publicly on steps taken to address findings from SAI audit reports only in a few countries
Does the executive make available to the public a report on what steps it has taken to address audit recommendations or findings that indicate a need for remedial action?
Note: The answer “yes” means that the executive reports publicly on (all, most or some) audit findings, while the answer “no” means that the executive does not report on steps it has taken to address audit findings.
Source: International Budget Partnership, Open Budget Survey 2019.
References
[16] Aguinis, H. and K. Bradley (2014), “Best Practice Recommendations for Designing and Implementing Experimental Vignette Methodology Studies”, Organizational Research Methods, Vol. 17/4, pp. 351-371, https://doi.org/10.1177/1094428114547952.
[17] Atzmüller, C. and P. Steiner (2010), “Experimental vignette studies n survey research”, Methodology, Vol. 6/3, pp. 128-138, https://doi.org/10.1027/1614-2241/a000014.
[20] Avis, E., C. Ferraz and F. Finan (2018), “Do government audits reduce corruption? Estimating the impacts of exposing corrupt politicians”, Journal of Political Economy, Vol. 126/5, pp. 1912-1964, https://doi.org/10.1086/699209.
[14] Cassels, W., K. Alvero and C. Errington (2009), Internal auditors need to shift the focus of audit reporting from their own priorities to those of the client, https://www.iia.nl/SiteFiles/IA/ia201904-dl_Itsnotaboutyou.pdf (accessed on 10 May 2021).
[13] Customs, I. (2017), Applying Behavioural Science in Tax Administration - A Summary of Lessons Learned.
[5] EUROSAI (2021), Follow-up of the implementation of audit recommendations: Best practices guide, issued by the project group, European Organisation of Supreme Audit Institutions (EUROSAI).
[19] Ferraz, C. and F. Finan (2008), “Exposing corrupt politicians: The effects of Brazil’s publicly released audits on electoral outcomes”, The Quarterly Journal of Economics May, pp. 703-745, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=997867 (accessed on 30 December 2014).
[12] GAO (2018), Government Auditing Standards 2018 Revision, United States Government Accountability Office, http://www.gao.gov/yellowbook. (accessed on 10 May 2021).
[8] Hallsworth, M. et al. (2016), “Provision of social norm feedback to high prescribers of antibiotics in general practice: a pragmatic national randomised controlled trial”, www.thelancet.com, Vol. 387, p. 1743, https://doi.org/10.1016/S0140-6736(16)00215-4.
[4] INTOSAI (2010), How to increase the use and impact of audit reports: A guide for Supreme Audit Institutions, INTOSAI Capacity Building Committee, https://iniciativatpa.org/wp-content/uploads/2014/05/Increase-impact-of-audit-reports.pdf (accessed on 28 September 2020).
[11] INTOSAI (2010), How to increase the use and impact of audit reports: A guide for Supreme Audit Institutions.
[3] Kassin, S., I. Dror and J. Kukucka (2013), “The forensic confirmation bias: Problems, perspectives, and proposed solutions”, Journal of Applied Research in Memory and Cognition, Vol. 2/1, pp. 42-52, https://doi.org/10.1016/j.jarmac.2013.01.001.
[1] Kida, T. (1984), The Impact of Hypothesis-Testing Strategies on Auditors’ Use of Judgment Data.
[2] Kinney, W. and W. Uecker (1982), Mitigating the Consequences of Anchoring in Auditor Judgments, https://www.jstor.org/stable/246739.
[7] OECD (2020), Behavioural Insights and Organisations: Fostering Safety Culture, OECD Publishing, Paris, https://doi.org/10.1787/e6ef217d-en.
[9] OECD (2020), OECD Public Integrity Handbook, OECD Publishing, Paris, https://doi.org/10.1787/ac8ed8e8-en.
[18] OECD (2017), Behavioural Insights and Public Policy: Lessons from Around the World, OECD Publishing, Paris, https://doi.org/10.1787/9789264270480-en.
[21] OECD (2016), Progress in Chile’s Supreme Audit Institution: Reforms, Outreach and Impact, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264250635-en.
[10] OECD (2016), Supreme Audit Institutions and Good Governance: Oversight, Insight and Foresight, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264263871-en.
[15] OECD (2014), Chile’s Supreme Audit Institution: Enhancing Strategic Agility and Public Trust, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264207561-en.
[6] Pennington, R., J. Schafer and R. Pinsker (2017), “Do Auditor Advocacy Attitudes Impede Audit Objectivity?”, Journal of Accounting, Auditing & Finance, Vol. 32/1, pp. 136-151, https://doi.org/10.1177/0148558X16641862.