Audit reports issued by Supreme Audit Institutions are key in generating impact, as they are the main vehicle through which audits can induce change in the public sector. As such, implementation rates of audit recommendations from audit reports are instrumental to achieve impact. This chapter provides a brief overview on the various factors that help explaining the implementation rate of audit recommendations before focusing in detail on relevant behavioural drivers of both auditors and auditees.
Enhancing the Oversight Impact of Chile’s Supreme Audit Institution
Applying Behavioural Insights for Public Integrity
OECD Public Governance Reviews
1. What drives the impact of Supreme Audit Institutions
Abstract
Introduction
In line with international standards as well as good practices promoted by the International Organisation of Supreme Audit Institutions (INTOSAI) and the OECD, Supreme Audit Institutions (SAI) should lead by example and demonstrate their added value and impact (INTOSAI, 2019[1]). In particular, the audit reports are key in generating impact, as they are the main vehicle through which audits can induce change in the public sector. To do so, audit reports and their observations and recommendations have to be relevant, have to be read and understood by the right people, they have to be available at the right time and the information has to be presented in the right way. As such, INTOSAI invites SAIs to continually review how they can make their reports more readable, more accessible, and more relevant to all stakeholders (INTOSAI, 2010[2]). In turn, INTOSAI’s Development Initiative (IDI) works with the SAI to support them in applying the standards, to build capacities and to ensure the quality of audits, for instance in the context of the Facilitating Audit Impact (FAI) strategy (IDI, 2021[3]).
Chile’s SAI, the Comptroller General of the Republic (Contraloría General de la República, CGR) faces the same pressure as other SAIs to generate and show impact. The Constitution establishes the CGR as an autonomous government body, which has a high level of organisational and administrative independence. The CGR has made significant progress over the last decade and has, amongst others, reviewed the monitoring and follow-up processes of its audit reports; also with support from the OECD (OECD, 2014[4]; OECD, 2016[5]). Thanks to the audit follow-up units (Unidades de Seguimiento), created by the CGR in 2012, the Integrated System for Audit Control (Sistema Integrado para el Control de Auditorias, SICA) and the Compliance Support Programme (Chapter 2), the CGR found out, however, that between 2015 and 2020, on average, only 50% of the audit observations included in their compliance audit reports were addressed by the audited entities.
Consequently, the CGR took several measures aimed at improving the uptake of audit reports. For instance, the Compliance Support Programme (Programa de Apoyo al Cumplimiento, PAC) was launched in 2016 with the objective to identify and implement creative mechanisms to increase the rate of observations addressed by audited entities. In 2019, the CGR conducted an internal evaluation exercise, which showed that public entities were satisfied with this initiative and find it useful. Learning from these insights, the CGR is exploring new mechanisms to enhance the impact of this initiative in the context of the 2021-2024 Strategic Planning.
In this context and striving to better understand and find innovative solutions to improve the level of uptake of the audit reports, the CGR collaborated with the OECD to apply a behavioural perspective. A behavioural perspective is an inductive approach that combines behavioural insights (BI) from psychology, cognitive science and social science with empirically tested results to discover how humans actually make choices. The perspective is increasingly used to improve our understanding of how context, cognitive biases and other influences affect the behaviour of people, including behaviours related to integrity policies (OECD, 2019[6]; OECD, 2018[7]).
Following up on audit reports, or failing to do so (at all or in a timely manner), is also the product of human behaviour. In a nutshell, within a given institutional and regulatory context, a public official receives the audit report, reads it, has to understand and process the information provided there and ultimately has to decide to act based on this information; if at all, to what extent, how and given the constraints the official is facing. To understand who these individuals are and why they behave as they do is thus relevant for informing improvements that could positively influence the uptake of CGR’s audit reports. Considering psychological factors in auditing and in related fields of accounting and law enforcement is not new (Kida, 1984[8]; Kinney and Uecker, 1982[9]; Kassin, Dror and Kukucka, 2013[10]). However, a recent OECD review of BI applications around the world did not find examples of interventions focusing on auditing processes, but some examples of applications aimed at ensuring compliance with rules or regulations could be relevant to inspire interventions in the audit world (OECD, 2017[11]).
To analyse the Chilean context and develop proposals for behavioural interventions, the project follows the BASIC methodology, developed by the OECD to support policymakers with tools, methods and ethical guidelines for conducting BI projects (OECD, 2019[6]). BASIC follows and inductive, context-driven approach (Figure 1.1).
Figure 1.1. The BASIC framework in the context of the CGR-OECD project
Source: (OECD, 2019[6])
To identify the actors’ relevant behaviours and to understand the context in Chile that is shaping these behaviours (steps B and A of the BASIC framework), the project carried out an in-depth qualitative analysis based on a desk research and various fact-finding interviews and focus group discussions with key stakeholders. A quantitative analysis of the observations included in the audit reports complemented the qualitative research (Chapter 2). Due to COVID-19, the OECD carried out the qualitative research through video conferences. Interviews focused on public officials responsible for internal audit and on public managers within public entities at national and at municipal level. The CGR formed a team of experts dedicated to the project, providing information and feedback and participating in meetings organised and moderated by the OECD. Based on this analysis, Chapter 3 provides a set of concrete interventions to address them (step S of the BASIC framework).
Before moving to the specific context in Chile, this chapter provides a brief discussion of the role audit reports play for the impact of SAI, as well as the various factors driving the uptake of audit reports by audited entities and thus their potential to generate change in the public administration. Finally, the chapter focuses on behavioural aspects from both auditors and auditees that are relevant for explaining the uptake of audit reports.
Defining and measuring the impact of SAI
The role of SAIs in the promotion of good governance has evolved over the last decades, moving from activities that are essentially compliance-oriented to a role aimed at understanding and enhancing the performance of governments to deliver for citizens. This change has led to a diversification of SAI’s strategic objectives, audits and advisory role to include the provision of evidence-based insights and foresight in support of decision-making, as a complement to traditional oversight activities (OECD, 2016[12]). Government-wide performance audits and data-driven dashboards that track or predict economic changes are just some examples of insight and foresight activities.
This evolving role brings new challenges for SAIs to measure their impact. For SAIs that traditionally focus on financial and compliance audits, as in Chile, measuring impact largely focuses on output-based indicators, such as the number of audits undertaken or clean audit opinions without irregularities. Going beyond the output level, the impact of a SAI in terms of relevant outcomes could be measured, for example, in terms of:
savings due to the measures implemented;
increases in revenue;
reductions in expenditure;
increases in satisfaction with the delivery of public services delivered by the public administration;
providing legal certainty by ensuring compliance with the legal frameworks; or
improvements in achieving other policy goals, e.g. related to SDGs (environmental quality, education, health, gender equality, anti-corruption and integrity etc.)
One of the main challenges for SAIs to measure their impact is the difficulty they face in attributing changes in outcome levels in audited entities and society to specific actions and outputs of the SAI. Through the audit reports, a SAI is able to influence the audited entities and promote change that, ultimately, can lead to the desired impact at outcome levels (Figure 1.2). The outputs of the SAI in terms, for example, of number of audits, audit reports published or the number of recommendations issued, can be measured and clearly attributed to the SAI and are under its direct control. Yet, how the auditees use these outputs of SAIs is beyond their direct control, but is critical to reach the higher-level desired outcome at the level of the auditees in the public administration and in the effective and efficient provision of public services.
Figure 1.2. Simplified generic theory of change of Supreme Audit Institutions
For example, an audit report (SAI output) may contribute to improve user satisfaction of a public service (outcome). But a potential impact at this outcome level stemming from an audit report must have passed by actions taken by the auditee: for instance, that a public entity providing services to citizens implements the audit recommendations (intermediate outcome). Of course, user satisfaction depends on a wide variety of factors and not only the uptake of the audit report by the auditee. To what degree, if at all, is it possible to attribute an observed increase in customer satisfaction to the changes implemented thanks to the audit reports? Would the changes in user satisfaction perhaps even have occurred without these audit reports? Without a counterfactual, these questions are difficult to answer.
In turn, assuming that the audits are carried out following professional standards and that audit reports include relevant observations and recommendations, the impact of SAIs at the level of the intermediate outcome, that is, the uptake of the audit reports by the auditee, is more straightforward to establish. At this level, the impact can be measured by the rate of implementation, understood as the percentage of corrected observations or implemented recommendations included in the audit reports.
Following this logic, the rate of implementation is a good proxy indicator for the potential success of external audit and for measuring the immediate uptake of audit reports. Due to its significant instrumental value, this report focuses on ways to influence this indicator.
Nonetheless, the indicator has some drawbacks. For instance, it is relevant to distinguish between types of audit. Financial and compliance audits usually result in administrative, procedural observations that are relatively easy to pinpoint and to follow-up through the rate of implementation. Performance audits, in turn, typically result in recommendations concerning policy design or implementation that are more difficult to track. Other points to keep in mind are:
The rate of implementation, as discussed, only reveals tangible instrumental impact, neglecting the other types of impact at the outcome level, both at the public entity and societal level.
The indicator does not take into account the relative importance (in financial or societal terms) of the observations or recommendations and the complexity of implementing them.
Often, some improvements are already implemented during the (and as a result of) an ongoing audit. In this case, auditors will not formulate any recommendations, although there has been impact.
Quality of recommendations matter. Implementing recommendations does not necessarily lead to improvements and not implementing recommendations is not always a bad, if their quality is not adequate (Desmedt et al., 2017[13]).
Factors that influence the uptake of audit reports by auditees
A variety of factors is likely to determine whether an audited entity in the public sector is willing and able to address observations or implement changes recommended through external audit by a SAI. Research and international good practice, reflected in international standards promoted by national and international organisations, emphasise three groups of variables that are key in explaining impact of SAIs (Figure 1.3). A first group includes factors related to the audit process itself (“Micro-level”), a second group considers factors that are related to the SAI and the audited entities (“Meso-level”) and a third group considers different pressure groups outside the audited entity (“Macro-level”).
Figure 1.3. Factors influencing the impact of performance audits conducted by SAIs
Source: (Van Loocke and Put, 2011[14]).
Understanding these drivers helps to identify entry points for concrete measures aimed at increasing the uptake of audit reports and therefore, indirectly, the impact of the SAI in promoting change. On the one hand, SAIs can optimise factors that are under their direct control, such as the auditor-auditee relationship during the audit process itself, the audit report and the follow-up processes. For instance, at the micro-level, a study in Belgium found that, in particular, a fluent communication, openness between auditors and entities and the level of recognition of the credibility and legitimacy of the auditees are relevant in explaining impact (Desmedt and Pattyn, 2015[15]). As stressed by EUROSAI, “audit findings should be discussed with the auditee before commencing with the formulation of conclusions and recommendations” (EUROSAI, 2021[16]). In the survey conducted by EUROSAI on the uptake of audit recommendations, 27 out of 33 respondents from European SAIs are basing their recommendation on a dialogue with auditees.
On the other hand, for factors that are outside their direct range of influence, SAIs can develop strategies aimed at indirectly influencing auditees. At the meso-level, the same study found that the position of the auditors' recommendations within the management priorities, the will of the authorities and the political will were also found to be significant in the Belgian context (Desmedt and Pattyn, 2015[15]). At the macro-level, for example, a SAI can try to establish alliances with other actors or favour processes that improve its image in the media, in the legislative or in the executive to promote outside pressure to ensure that auditees are following up on audit reports. Stakeholders should be engaged as early as the planning phase of an audit process.
Publicly tracking actions taken by the executive to take action based on audit reports may also contribute to create pressure at the macro level. Studies in Brazil found that making audit results from the Office of Comptroller-General (Controladoria Geral da União, CGU) public, significantly reduced the probability of re-electing a mayor in which at least two violations associated with corruption were reported (Ferraz and Finan, 2008[17]; Avis, Ferraz and Finan, 2018[18]).1 In OECD countries with available data, 65.2% of the countries make this information public, while only 25% of countries in Latin America with available data do so – amongst them Brazil (Figure 1.4).
Figure 1.4. While 65.2% of OECD countries publicly report on actions taken by the executive to address audit recommendations, only 25% of countries in Latin America do so
Does either the Supreme Audit Institution (SAI) or legislature release to the public a report that tracks actions taken by the executive to address audit recommendations?
Note: The graphs above counted as “Yes” all countries responding that “Yes, the SAI or legislature reports publicly on what steps the executive has taken to address all/most/some audit recommendations” (score 100 for “all”, 67 for “most” or 33 for “some”).
Source: International Budget Partnership, Open Budget Survey 2019
Nonetheless, behind the percentage of corrected observations or implemented recommendations included in the audit reports there are, in the end, human beings in the audited entities that take a decision on whether to take action, or to what degree. These individuals are operating in a given normative and cultural context, which provides incentives and shapes their decisions. There may be legal obligations to implement audit observations or recommendations as well as sanctions in case of non-compliance. In addition, the relevance and quality of the provided audit reports will influence whether they are likely to trigger implementation or not. As mentioned in the introduction, psychological aspects, explored in the following section, may also influence these behaviours of auditees.
How behavioural insights contribute to explaining the uptake of audit reports
Behavioural barriers and biases of both the auditor and the auditee can help explaining the success or the failure of audits to achieve the desired impact. Figure 1.5 provides an overview of behavioural insights, explored in more detail in the following section. By integrating these behavioural insights into auditing processes, SAIs could better anticipate the behavioural implications of their audits and use these insights to design and deliver more effective audit processes and reports that are more likely to be followed-up, lead to change and therefore to improve the welfare of citizens.
Figure 1.5. Main behavioural insights related to auditing
Auditing is principally a matter of human judgement
Independence and objectivity are fundamental values of SAIs. They are defined in the International Standards of Supreme Audit Institutions as being “free from circumstances or influences that compromise, or may be seen as compromising, professional judgement, and to act in an impartial and unbiased manner” (INTOSAI, 2019[19]). Nonetheless, auditing is principally a matter of human judgement and as such, objectivity is not always possible. In fact, the subjective judgment of auditors is part of the profession. The International Standards on Auditing (ISA), through ISA 200, define professional judgement as the “application of relevant training, knowledge and experience (…) in making informed decisions about the courses of action that are appropriate in the circumstances of the audit engagement.” The fundamental Principles of Public-Sector Auditing (ISSAI 100) emphasises that professional judgement implies the application of collective knowledge, skills and experience to the audit process (INTOSAI, 2019[20]). Professional judgment, as defined in ISA 200 or ISSAI 100, is also relevant when it comes to assessing the audit findings and the prioritisation of audit observations and recommendations. As such, subjective judgement, just as professional criticism, plays a fundamental part in informing the auditor’s analysis.
However, insights from behavioural sciences show that judgements may also be subject to systematic cognitive biases that could become relevant when carrying out an audit. For example, audit criteria help to guide auditors in their analysis and judgements. Nonetheless, as human beings, our desires powerfully influence the way we interpret information, even when we are trying to be objective and impartial (Bazerman, Moore and Loewenstein, 2002[21]).
Audit criteria can only resolve this partially. Despite the fact that audit criteria may provide guidance for relative objectivity, auditing leaves considerable leeway for ambiguity. For instance, confirmation bias may be an issue when auditors have pre-conceived ideas about the audited institution or the processes. Their audit may then unconsciously focus on details that confirm their existing beliefs. In fact, confirmation bias could affect objective judgment in both directions: against or in favour of the auditee. Research showed, indeed, that the degree in which auditors tend to support the auditee (“advocacy attitude”) influences the quality of the evidence collected by biasing auditors’ initial judgments and influencing the type of subsequent evidence collected. Such a confirmation bias exists in particular for low advocacy auditors, i.e. auditors with low levels of support for auditees, as they tend to plan a less objective search for more confirmatory evidence, potentially demonstrating too much presumption of distrust in management (“presumptive doubt”) (Pennington, Schafer and Pinsker, 2017[22]).
Auditors should strive to have neutral initial beliefs. Unconsciously, however, cognitive biases could affect our pre-existing beliefs and undermine our way to draw conclusions because of anecdotes we hear, our insensitivity to sample size or our tendency to overestimate our ability to interpret and predict outcomes given a set of information (the illusion of validity). Overconfidence of the auditors may further undermine the accuracy of the auditor’s judgements.
For example, perceptions and stories about corrupt and inefficient public administrations could frame auditors towards gathering evidence and over-emphasising information that is unfavourable to auditees, resulting in recommendations that are no longer objective. Such a confirmation bias against auditees perhaps may not be problematic in areas of high corruption and fraud risks, but could become problematic for low-risk audits (Pennington, Schafer and Pinsker, 2017[22]). It certainly will make it more difficult to motivate auditees and to build a constructive climate between auditors and auditees. The auditees could perceive the cognitive biases that are undermining auditors’ professional judgement. This, in turn, could delegitimise the audit recommendations and, consequently, create resistance to follow-up on audit reports.
Auditors are influenced by social norms
Expectations about the “right” behaviour can also influence auditors. Auditors are part of a social group, which can be the auditor’s unit, the SAI they are working in or even the auditor profession as a whole. What auditors believe most other auditors in their group are actually doing (empirical expectation) or what they believe most other auditors of their group expect them to do (normative expectation) can explain behavioural pattern and are called “social norms” (Bicchieri, 2005[23]; Bicchieri, 2017[24]). Such social norms can be extremely powerful in shaping behaviours.
Figure 1.6 describes how to diagnose an observed pattern of behaviour (Bicchieri, 2017[24]). On the one hand, there are of course reasons why people follow behavioural patterns no matter what others do, either because it serves a purpose (custom) or because it is thought to be the right thing to do (moral norm). On the other hand, the social group becomes relevant if the behaviour of an individual depends on what others do (descriptive norm) or what is believed to be expected, and potentially punished in case the rule is not respected (social norm). If such social norms are relevant in explaining behaviour, interventions that only aim at changing the formal rules or that aim at appealing to what is the “correct thing” to do, may fail in changing behaviours (Bicchieri, Lindemans and Jiang, 2014[25]; Yamin et al., 2019[26]).
Figure 1.6. Diagnosing social norms
Source: Bicchieri and Penn Social Norms Training and Consulting Group, 2015, (Bicchieri, 2017[24]).
The existence of social norms within a given SAI could generate cultures that reinforce specific behaviours that in turn may unconsciously influence judgement and the work of the auditors. In a sense, they are a kind of social desirability bias. For example, independent from an auditor’s training or his or her convictions about what auditors in general should do, this auditor may end up conforming to such a social norm prevalent in the unit or the SAI he or she is working in.
Such behavioural pattern that are conditional on others could explain the way auditors carry out their work, draft their findings and could affect the quantity of findings and recommendations. Auditors, for instance, may aim to find many observations or to provide very technical descriptions or justifications in the reports, if they believe that their superiors or colleagues expect such behaviour, because they see it, in the internal culture, as the typical product to be expected of a skilled and productive auditor. Following the social norm then becomes the best response to the given expectations, as it is likely to be rewarded (formally or informally). In turn, superiors or colleagues may perceive as incompetent or lazy an auditor that decides to start writing short reports in plain language and with concise recommendations.
As a consequence, such social dynamics influencing auditors’ behaviours could lead to audit reports that are more tailored towards internal needs in terms of the auditor’s career (performance evaluations, visibility or approval by colleagues and supervisors), than towards the need of the auditees. Again, this could undermine the relevance of the audit reports for the auditees and thus their uptake.
Auditees too are subject to behavioural barriers and biases
The auditee too may be subject to biases when receiving the audit results that can undermine their uptake. For example, people tend to see flaws in others more easily than in themselves and auditees may have difficulties to accept that errors have been committed by them or that doing things differently could improve their management. In addition, if the first perception or association of the auditing process or the audit report is negative, it is likely that everything related to the audit will be perceived negatively as well (anchoring effect) and trigger counter-reactions. This, in turn, may affect the follow-up of observations or the uptake of the recommendations. Finally, a study carried out in Norway shows that there may be effects related to the seniority of the audited public managers. In the Norwegian context, where the study has been carried out, high-ranking public servants seem to be less positive towards performance audits than are lower-ranking civil servants (Reichborn-Kjennerud, 2013[27]). Such a negative attitude may also imply that they are more inclined to reject findings included in the audit reports. At the same time, senior managers are usually the receivers of the audit reports.
Auditees may also perceive audit reports as unfair. As mentioned previously, audit criteria can help in guiding objectivity, but the criteria or their application can be perceived as unfair too. In turn, perceived fairness matters significantly. Findings from behavioural and social neurosciences suggest that experiencing unfair, exclusionary treatment can trigger reactions in the brain that are similar to experiencing pain (Eisenberger, Lieberman and Williams, 2003[28]). In addition, evidence suggest that subjective judgments of the fairness of outcomes are less important psychologically than subjective judgments of the fairness of process (Lind and Tyler, 1988[29]; Lind et al., 1993[30]; Walker et al., 1974[31]; Tyler, 2006[32]). Based on such findings, it is worth considering that by designing and administering regulations taking into account perceived fairness, governments can minimise experiences that citizens perceive as unfair. This, in turn, makes it more likely that they accept and comply with rules and decisions, feel included and trust their government (Lind and Arndt, 2016[33]).
Similar considerations apply to auditing processes too. If auditees perceive that they have been treated unfair – whether justified or not – they will be less likely to trust the auditors and the auditing process, they will unconsciously find justifications against the auditors and their findings and may therefore be less likely to follow or comply with the audit findings and recommendations.
Audit reports may fail to motivate auditees
Audit reports and recommendations may also fail to motivate auditees in implementing recommendations. As noted by the European Court of Auditors, audit reports tend to have an impersonal tone and can inadvertently distance the reader from the findings and observations (European Court of Auditors, 2013[34]). In part, this tone may be due to the deeply rooted strive for independence and objectivity within the audit profession.
Lack of motivation may also result from unclear responsibilities allowing public managers to rationalise inaction by refusing to see or to accept own responsibility, especially if non-compliance does not trigger any consequences. As such, recommendations should clearly address a specific group or office to avoid the diffusion of responsibility. A related problem is the tendency that some auditors may avoid providing definitive statements: it appears that, it seems that, may have. Such language may be needed in the absence of definitive criteria that requires auditees to take specific actions; however, it can also suggest that recommendations are simply suggestions, thereby and, again, promoting inaction.
Furthermore, some audit findings are typically phrased as observations stating, for example, incompliance with a regulation or procedure. Such observations, if they do not include guidance with respect to corrective actions or do not clearly show the auditee the rationale for addressing the observation, may not trigger the desired behaviour. However, as emphasised also in a recent EUROSAI report, SAIs should avoid putting themselves in a situation where they have to audit solutions that they have proposed themselves. Therefore, to strike a balance between helping auditees and avoiding such complications, recommendations could be formulated in a style where they describe what the auditee should do, and not how they should do it (EUROSAI, 2021[16]).
Finally, aggressive monitoring and scrutiny could negatively affect the intrinsic motivation of public managers to address audit observations and implement recommendations. Indeed, the introduction of a control mechanism or aggressive monitoring is a signal of distrust (OECD, 2018[7]). Overly strict control has been shown to significantly reduce the efforts of the person being controlled (Falk and Kosfeld, 2006[35]). It forces people to provide only the minimum effort necessary to pass the control, but removes the element of positive reciprocity: employees subject to controls might feel less obliged to deliver (Lambsdorff, 2015[36]; OECD, 2018[7]). Therefore, SAIs should strike the right balance between promoting compliance through intrinsic motivation and through control and monitoring when following up on audit reports.
Auditees’ attention is limited
People’s attention is limited and is easily distracted. Following-up with audit reports compete with other tasks of the public managers in the audited entities. Those responsible for implementing the corrective actions or considering and implementing the audit recommendations may simply be over-burdened by the workload and may fail to understand the relevance of the audit reports. Especially compliance audits, while necessary and relevant, may lead public managers to seek formal compliance only and to ensure that they have “ticked the right boxes”. This approach is very human, as it minimises stress and effort given the workload created by the audit, but is contrary to good practices outlined in the OECD’s Recommendation on Public Integrity and other international standards and is unlikely to drive real change.
Finally, when processes are too complicated, audit reports are too long, contain too much recommendations or there is in general too much information, auditees may experience decision fatigue from weighing too many inputs. This can result in public managers making the wrong choice, selecting the wrong priorities or deferring the choice all together. Over-burden may also generate or intensify an already negative attitude towards the work created through audits. In Norway for instance, a study found that officials most exposed to auditing were, in general, more negative towards it (Reichborn-Kjennerud, 2013[27]).
References
[18] Avis, E., C. Ferraz and F. Finan (2018), “Do government audits reduce corruption? Estimating the impacts of exposing corrupt politicians”, Journal of Political Economy, Vol. 126/5, pp. 1912-1964, https://doi.org/10.1086/699209.
[21] Bazerman, M., D. Moore and G. Loewenstein (2002), Why good accountants do bad audits.
[24] Bicchieri, C. (2017), Norms in the wild: How to diagnose, measure, and change social norms, https://doi.org/10.1093/acprof:oso/9780190622046.001.0001.
[23] Bicchieri, C. (2005), The grammar of society: The nature and dynamics of social norms, https://doi.org/10.1017/CBO9780511616037.
[25] Bicchieri, C., J. Lindemans and T. Jiang (2014), “A structured approach to a diagnostic of collective practices”, Frontiers in Psychology, Vol. 5, https://doi.org/10.3389/fpsyg.2014.01418.
[13] Desmedt, E. et al. (2017), “Impact of performance audit on the administration: A belgian study (2005-2010)”, Managerial Auditing Journal, Vol. 32/3, pp. 251-275, https://doi.org/10.1108/MAJ-04-2016-1368.
[15] Desmedt, E. and V. Pattyn (2015), De impact van de performance audits van het Rekenhof. Survey bij ambtenaren van de federale overheid, Versie Aanvaard Voor Publicatie in VTOM, Vlaams Tijdschrift Voor Overheidsmanagement, No. 7. 103-120.
[28] Eisenberger, N., M. Lieberman and K. Williams (2003), “Does rejection hurt? An fMRI study of social exclusion”, Science, Vol. 302/5643, pp. 290-292, https://doi.org/10.1126/science.1089134.
[34] European Court of Auditors (2013), Report-writing guideline.
[16] EUROSAI (2021), Follow-up of the implementation of audit recommendations: Best practices guide, issued by the project group, European Organisation of Supreme Audit Institutions (EUROSAI).
[35] Falk, A. and M. Kosfeld (2006), “The Hidden Costs of Control”, American Economic Review, Vol. 96/5, pp. 1611-1630.
[17] Ferraz, C. and F. Finan (2008), “Exposing corrupt politicians: The effects of Brazil’s publicly released audits on electoral outcomes”, The Quarterly Journal of Economics May, pp. 703-745, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=997867 (accessed on 30 December 2014).
[3] IDI (2021), Facilitating Audit Impact (FAI) Strategy, INTOSAI Development Initiative, Oslo, https://idi.no/elibrary/relevant-sais/fai/1408-facilitating-audit-impact-fai-strategy/file (accessed on 12 April 2022).
[19] INTOSAI (2019), Code of Ethics - International Standards of Supreme Audit Institutions ISSAI 130, International Organisation of Supreme Audit Institutions, Vienna, https://www.issai.org/pronouncements/issai-130-code-of-ethics/ (accessed on 5 May 2021).
[1] INTOSAI (2019), INTOSAI-P 12 The Value and Benefits of Supreme Audit Institutions: Making a difference to the lives of citizens, AuditInternational Organisation of Supreme Audit Institutions (INTOSAI).
[20] INTOSAI (2019), ISSAI 100 Fundamental Principles of Public-Sector Auditing, International Organisation of Supreme Audit Institutions (INTOSAI), https://www.issai.org/pronouncements/issai-100-fundamental-principles-of-public-sector-auditing/ (accessed on 17 May 2021).
[2] INTOSAI (2010), How to increase the use and impact of audit reports: A guide for Supreme Audit Institutions, INTOSAI Capacity Building Committee, https://iniciativatpa.org/wp-content/uploads/2014/05/Increase-impact-of-audit-reports.pdf (accessed on 28 September 2020).
[10] Kassin, S., I. Dror and J. Kukucka (2013), “The forensic confirmation bias: Problems, perspectives, and proposed solutions”, Journal of Applied Research in Memory and Cognition, Vol. 2/1, pp. 42-52, https://doi.org/10.1016/j.jarmac.2013.01.001.
[8] Kida, T. (1984), The Impact of Hypothesis-Testing Strategies on Auditors’ Use of Judgment Data.
[9] Kinney, W. and W. Uecker (1982), Mitigating the Consequences of Anchoring in Auditor Judgments, https://www.jstor.org/stable/246739.
[36] Lambsdorff, J. (2015), “Preventing corruption by promoting trust – insights from behavioral science”, Passauer Diskussionspapiere, No. V-69-15, Universität Passau, Passau, https://doi.org/10.13140/RG.2.1.3563.4006.
[33] Lind, E. and C. Arndt (2016), “Perceived Fairness and Regulatory Policy: A Behavioural Science Perspective on Government-Citizen Interactions”, OECD Regulatory Policy Working Papers, No. 6, OECD Publishing, Paris, https://doi.org/10.1787/1629d397-en.
[30] Lind, E. et al. (1993), “Individual and Corporate Dispute Resolution: Using Procedural Fairness as a Decision Heuristic”, Administrative Science Quarterly, Vol. 38/2, p. 224, https://doi.org/10.2307/2393412.
[29] Lind, E. and T. Tyler (1988), The Social Psychology of Procedural Justice, Springer US, Boston, MA, https://doi.org/10.1007/978-1-4899-2115-4.
[6] OECD (2019), Tools and Ethics for Applied Behavioural Insights: The BASIC Toolkit, OECD Publishing, Paris, https://doi.org/10.1787/9ea76a8f-en.
[7] OECD (2018), Behavioural Insights for Public Integrity: Harnessing the Human Factor to Counter Corruption, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264297067-en.
[11] OECD (2017), Behavioural Insights and Public Policy: Lessons from Around the World, OECD Publishing, Paris, https://doi.org/10.1787/9789264270480-en.
[5] OECD (2016), Progress in Chile’s Supreme Audit Institution: Reforms, Outreach and Impact, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264250635-en.
[12] OECD (2016), Supreme Audit Institutions and Good Governance: Oversight, Insight and Foresight, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264263871-en.
[4] OECD (2014), Chile’s Supreme Audit Institution: Enhancing Strategic Agility and Public Trust, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264207561-en.
[22] Pennington, R., J. Schafer and R. Pinsker (2017), “Do Auditor Advocacy Attitudes Impede Audit Objectivity?”, Journal of Accounting, Auditing & Finance, Vol. 32/1, pp. 136-151, https://doi.org/10.1177/0148558X16641862.
[27] Reichborn-Kjennerud, K. (2013), “Resistance to Control—Norwegian Ministries’ and Agencies’ Reactions to Performance Audit”, Public Organization Review, Vol. 15/1, pp. 17-32, https://doi.org/10.1007/s11115-013-0247-6.
[32] Tyler, T. (2006), Why People Obey the Law, Princeton University Press, Princeton.
[14] Van Loocke, E. and V. Put (2011), The Impact of Performance Audits: A Review of the Existing Evidence, Edward Elgar Publishing, https://doi.org/10.4337/9780857931801.00016.
[31] Walker, L. et al. (1974), “Reactions of Participants and Observers to Modes of Adjudication1”, Journal of Applied Social Psychology, Vol. 4/4, pp. 295-310, https://doi.org/10.1111/j.1559-1816.1974.tb02601.x.
[26] Yamin et al. (2019), “Using Social Norms to Change Behavior and Increase Sustainability in the Real World: A Systematic Review of the Literature”, Sustainability, Vol. 11/20, p. 5847, https://doi.org/10.3390/su11205847.
Note
← 1. Note that Brazil’s Office of Comptroller-General (CGU) is responsible for internal audits, however. Brazil’s SAI is the Federal Court of Accounts (Tribunal de Contas da União, TCU).