This chapter presents a detailed theory of change for two interventions taken from the strategies proposed in Chapter 3 and that could be piloted by the CGR. First, a meeting between the CGR and the audited services before starting the follow-up process could help explaining the audit findings, reduce the cognitive burden and provide avenues for taking corrective actions. Second, the CGR could introduce some flexibility with deadlines to signal credibly a supportive approach by the CGR. In addition, the chapter provides guidance on how to implement the pilot and measure results.
Enhancing the Oversight Impact of Chile’s Supreme Audit Institution
Applying Behavioural Insights for Public Integrity
OECD Public Governance Reviews
4. Piloting a meeting to explain audit findings and the effects of more flexible deadlines in Chile
Abstract
Selected pilot interventions
Achieving a sustainable positive impact on the uptake of audit reports is complex, is likely to require several reforms at different levels of the auditing and follow-up process and may require working towards a cultural change in both the Comptroller General of the Republic (Contraloría General de la República, CGR) and the administration. Therefore, in Chapter 3, the OECD recommends several measures that can be combined to achieve change and impact.
Together, the CGR and the OECD identified two measures that could be piloted to test their impact and learn during the implementation process before potentially scaling up the measures. Both interventions take place after the issuing of the audit report but before starting the follow-up process. They are complementary and aim at reducing auditees’ cognitive burden and perceived stress and unfairness related to deadlines:
Measure A: To support audited entities, the CGR could consider a meeting between CGR and the audited service before starting the follow-up process. The objective of the meeting is to explain the observations and to recommend certain lines of actions the management could take to address them, while paying caution not to co-administrate.
Measure B: When discussing the follow-up process once the audit report has been finalised, the CGR could introduce some flexibility with the deadlines for addressing the observations on a case-by-case base, following clear and pre-established criteria, involving Internal Auditor or Directors of Internal Control and the public managers responsible for addressing the observations.
Both measures aim at facilitating the follow-up of the audit observations by auditees and at building a better relationship between auditors and auditees. The rationale for the choice of the two measures reflects the scope of the project, which focused on the follow-up process. It also addresses two main underlying causes that, according to the qualitative research undertaken and reported in Chapter 2, contribute to explain the level of uptake of audit reports in Chile. On the one hand, the attention bias and cognitive burden caused by too many observations and too complex audit reports and, on the other hand, the stress and the perception of auditees that the CGR does not understand or care about the realities of the public administration by imposing uniform deadlines. As such, Measure A is about content; Measure B is about the process. Notwithstanding, further criteria for the selection where continuity, as a similar idea as mentioned in Measure A has been discussed previously within the CGR, and feasibility, as both measures can be implemented and tested without requiring legal changes.
Theory of change
A theory of change identifies underlying assumptions about how change comes about, make these assumptions more explicit and test them (Johnsøn, 2012[1]; OECD, 2017[2]). Figure 4.1 below provides a schematic overview of the theory of change underlying the two measures. The mechanisms, two for each Measure, are the channels through which it is assumed that the Measures will contribute to results.
Figure 4.1. Theory of change of the pre-follow up meeting and the negotiation of deadlines
Measure A, the pre-follow up meeting between the CGR and the audited service, responds to a finding from Chapter 2 in so far that public managers expressed the desire to obtain better guidance on how to address concretely the observations and/or to have the opportunity to clarify these observations.
The theory of change behind the hypothesis that this meeting is likely to improve the uptake of the audit report relies on the following mechanisms (Figure 4.1):
First, the meeting contributes to reducing the cognitive burden of public managers in the audited service when confronted with the final audit report. The meeting seeks to clarify and simplify the findings and observations in plain language and from the perspective of management. This, in turn, should impact on the relationship between auditors and auditees and lead to a more pro-active attitude and thus uptake of audit reports.
Second, through the guidance provided by the CGR during the meeting with respect to potential avenues to take corrective actions, the likelihood that public managers will actually take action are likely to increase if the reason for inaction was related to a lack of knowledge concerning what to do. In addition, following the meeting, the probability of corrective actions that are not aligned with the expectations by the CGR, and thus remain as incomplete in the system, should be lower and therefore lead to higher rates of compliance. To avoid perceptions of co-administration and ensure ownership, these avenues for concrete corrective actions need to be elaborated by public managers, with support from the CGR.
Measure B is straightforward. Chapter 2 evidenced that public managers experience stress and frustration about the non-flexible and uniform deadlines imposed by the CGR. This has reportedly sometimes generated a feeling of being treated unfair or of being misunderstood (“the CGR does not understand the realities of the public administration”). Therefore, the possibility to have more flexibility with deadlines for taking corrective actions during the meeting proposed in Measure A, should allow addressing this issue.
The theory behind the hypothesis that negotiating the deadlines will improve the uptake of the audit report relies on the following mechanisms (Figure 4.1):
First, by allowing deadlines that are more flexible on a case-by-case basis but following clear and pre-established criteria, the CGR visibly and credibly demonstrates flexibility and its willingness to support public managers. As human beings tend to reciprocate, this flexibility on the side of the CGR could increase public managers’ motivation to comply with the audit reports.
Second, having the public managers actively involved in fixing the deadlines could increase their sense of ownership. Public managers may feel personally committed to the deadline. In addition, it could be considered to reinforce this ownership and commitment by signing an informal pledge to comply with the deadline.
Overall, both the meeting and the possibility to negotiate deadlines could contribute to improve the relationship between the CGR and the audited services. As such, the design of the meeting should seek to promote a constructive environment, where the doubts and concerns of public managers are taken serious and where the CGR provides concrete support to facilitate the work of the public managers. The meeting should be communicated and implemented as support to public managers, not as another bureaucratic requirement.
Potentially, the meeting may have indirect effects on the way auditors will draft future reports, taking into account the feed-back they receive and the interactions they had during the meeting. For this, the CGR needs to ensure a feed-back loop from this meeting to auditors. The survey administered to auditors (see below) should take into account such a potential change in future behaviour.
Implementation design and measurement
Strictly speaking, impact can only be measured rigorously if the intervention takes into account the counterfactual, that is, what would have happened without the intervention (OECD, 2017[2]). For this, the universe needs to be big enough to allow for a sample size with sufficient statistical power and room to compare across intervention and control groups. However, given that the number of ongoing audits carried out by the CGR is likely to be insufficient to reach a sufficient sample size to implement such a randomised control trial methodology, the CGR could opt for a simple difference approach, as explained in the following sections.
The pilot could be implemented at municipal level
Therefore, the pilot implementation could focus on ongoing audits that are similar in terms of type and implementation level to avoid results driven by effects related to these two characteristics. As such, the CGR could select only regular audits implemented at the municipal level as the relevant universe to pilot and assess both measures (A and B).
From this universe, the CGR could select ongoing regular audits across municipalities and assign them into three different groups according to the size and the available resources of the respective municipality to improve comparability between groups. The final composition of the pilot, also depicted in Figure 4.2 could therefore respond to the following logic:
Group 1: A first group of municipal audits is used as the control group. In these audit processes, the measures will not be implemented.
Group 2: In a further group of municipal audits, the meeting (Measure A) will be implemented without the possibility to allow for flexible deadlines (Measure B).
Group 3: Finally, in a third group of municipal audits, both the meeting (Measure A) and flexible deadlines (Measure B) will be implemented.
Figure 4.2. Implementation design of the pilot interventions
Note: Groups are balanced in terms of size and budgets of municipalities.
Change could be measured through surveys and the implementation rate
To assess the effects of the meetings and flexible deadlines, relevant outcomes of these three groups will be compared through a simple difference approach. This means that the impact of the meeting (Measure A) is measured as the difference between group #2 and #1, and the relative impact of flexible deadlines (Measure B) as the difference between group #3 and #1.
To measure the effects of these two interventions, three surveys could be designed and sent respectively to auditors, public managers (auditees) and Directors of Internal Control. These surveys should ask for perception and attitudes of the three groups to test the underlying mechanisms represented in Figure 4.1 that could drive the behavioural change and positively impact on the uptake of the audit reports. Given sufficient time, the impact of the implemented measures could be measured directly at the level of the rate of implementation.
In fact, the CGR could consider introducing regular surveys amongst stakeholders. These could build on this pilot exercise and the questions that were used. Several SAI use such type of surveys. Box 4.1 provides some examples of client surveys conducted by SAI’s to improve their understanding of auditees’ perceptions. However, it is important to take due care of not creating a too strong client-relationship with auditees or generating potential “revenge” responses from audited services.
Box 4.1. Supreme Audit Institutions that survey key users to assess quality of audit work
Australia
After each performance audit report is complete, the Australian National Audit Office (ANAO) seeks feedback on the audit process by means of a survey and an interview with the responsible manager of the audited entity. The survey is an important tool for improving the quality and effectiveness of performance audit services. Survey results provide an insight into the effectiveness of current practice and inform the development of new audit practices and approaches. The survey is designed by a firm of consultants that is engaged by the ANAO but independent of the performance audit teams. The response rate from auditees surveyed for the 2011–12 reporting period was 75% and 87% in 2010-11.
Key matters on which feedback is sought include the audit process; audit reporting; and the value of the ANAO’s performance audit services more generally. For instance, in the 2011-12 survey, the proportion of respondents that acknowledged the value added by ANAO services was 91% (up from 86% in 2010–11). The percentage of respondents that considered the auditors had demonstrated the professional knowledge and audit skills required to conduct the audit was 85% (down from 91% in 2010–11).
Denmark
Rigsrevisionen, the SAI of Denmark, has used a variety of techniques to assess its benefits to audited entities and to the governance system, including client surveys. For instance, in 2009, Rigsrevisionen hired a consultancy firm to conduct an independent client survey on their behalf. The clients included permanent secretaries, director generals, managing directors of government-owned companies, financial managers, other officials in central government and members of the Parliament’s Public Accounts Committee.
Rigsrevisionen officials were most interested in understanding how their clients experienced the quality of services offered, their working relationship and the usefulness of their audit findings. The survey identified areas for improvement across four areas: financial auditing (annual audit); performance auditing (major examinations); co‑ordination, planning and counselling; and interaction with the Public Accounts Committee. The findings included recommendations to become more responsive during the audit phase in which memoranda and draft reports are being prepared, and to raise the competencies of SAI staff to a more uniform level.
New Zealand
The New Zealand Office of the Auditor General (OAG) uses an independent firm to conduct an annual client satisfaction survey of public entities audited by the Auditor General. The firm surveys a random sample of public entities to measure the level of satisfaction and identify areas where OAG needs to improve their audit services. Before 2007/08, the survey sample was confined to public entities audited by OAG. In 2007/08, OAG extended the sample to cover public entities audited by private sector accounting firms. Representatives of a sample of these entities are invited to participate in a telephone interview to provide comment and to rate the following factors on a scale from 1 to 10, with 1 being very low and 10 being very high:
audit service providers’ core audit ability
audit service providers’ staff knowledge
the way audit service providers’ staff work with entities, including governing bodies and audit committees where relevant
the value that audit service providers add and the usefulness of the advice given
the performance and contribution that audit service providers made as entities prepared to adopt New Zealand Equivalents to International Financial Reporting Standards (NZ IFRS) the overall degree of satisfaction with the service received from audit service provider.
The results of client surveys are prominently displayed in OAG’s annual reports, including in the preface by the Auditor General. Doing so communicates the importance of the surveys to OAG staff, as well as alertness among the leadership of the importance of meeting stakeholders’ and clients’ evolving needs.
Source: OECD (2014[3]), Chile’s Supreme Audit Institution: Enhancing Strategic Agility and Public Trust, OECD Public Governance Reviews, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264207561-en; New Zealand Controller and Auditor-General (2009[4]), “Annual Report 2008-2009”, Office of the Auditor-General, New Zealand, www.oag.govt.nz/2009/2008-09/docs/annual-report.pdf.
References
[1] Johnsøn, J. (2012), Theories of change in anti-corruption work: A tool for programme design and evaluation, Chr. Michelsen Institute (U4 Issue 2012:6), Bergen, http://www.cmi.no/publications/publication/?4635=theories-of-change-in-anti-corruption-work.
[4] New Zealand Controller and Auditor-General (2009), “Annual Report 2008-2009”, Office of the Auditor-General, New Zealand, http://www.oag.govt.nz/2009/2008-09/docs/annual-report.pdf.
[2] OECD (2017), Monitoring and Evaluating Integrity Policies, Working Party of Senior Public Integrity Officials GOV/PGC/INT(2017)4, Paris.
[3] OECD (2014), Chile’s Supreme Audit Institution: Enhancing Strategic Agility and Public Trust, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264207561-en.