This chapter provides recommendations on how Argentina could set up a central monitoring and evaluation system for its integrity policies. A central monitoring system for integrity policies would help to keep track of the implementation and facilitate evidence-informed communication with internal and external stakeholders. In turn, integrity policies should be evaluated to build knowledge and enable learning. Innovative integrity measures, in turn, could be rigorously tested through impact evaluations of pilots before implementing them on scale. In addition, the chapter provides guidance on how evidence could be gathered through staff and citizen surveys to inform the design, implementation and evaluation of integrity policies.
OECD Integrity Review of Argentina
Achieving Systemic and Sustained Change
OECD Public Governance Reviews
Chapter 2. Towards strengthening the evidence-base for integrity policies in Argentina
Abstract
The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law.
2.1. Introduction
Gathering credible and relevant information on the level of implementation, performance and overall effectiveness of a public integrity system is a crucial part of a strategic approach to integrity policies as recommended by the OECD Recommendation of the Council on Public Integrity (OECD, 2017[1]).
Monitoring allows an effective steering of the implementation of policies, helping to identify challenges and to take timely actions to ensure the achievement of the strategic goals. Evaluations, in turn, are a systematic and objective assessment of ongoing or completed policies with the aim to determine the relevance and fulfilment of objectives, efficiency, effectiveness, impact or sustainability (OECD, 2016[2]). Basically, evaluations look back and see if and how goals of a policy actually have been achieved. Evaluations allow policy makers to understand what works and why, and enable learning that can inform the design of the next policy cycle.
Overall, measurement enables better communication with stakeholders by facilitating an evidence-informed internal and external dialogue about policy goals and achievements. Also, thinking about how to monitor and evaluate policies forces policy-makers to think through the activities envisaged and may uncover too vague or too ambitious goals. Measurement further enables to showcase progress made and demonstrating that change is possible, while providing the basis for an objective discussion around successes and challenges. Not at least, measurement can further strengthen the accountability of the integrity system by providing information required for external social control.
2.2. Implementing a system for monitoring and evaluating Argentinian integrity policies
2.2.1. The monitoring and evaluation of integrity policies requires relevant indicators measuring different levels and that are based on different data sources
As recommended in chapter 1, a National Integrity Strategy for the executive branch could be developed in Argentina through a participatory approach to ensure ownership and relevance. To ensure that actions to implement various aspects of the strategy are being taken throughout the public administration the goals of such a strategy would need to be incorporated at the organisational level of public entities through their own strategic and operational planning (Ministerio de Modernización, 2016[3]). Such a planning process is also the first step towards developing a monitoring and evaluation system where specific indicators can be developed and measured at national, sectorial and organisational levels.
Each strategic goal of a National Integrity Strategy would need to be further translated into policy objectives, which in turn can be measured through indicators (Figure 2.1). These indicators provide the evidence required to keep track of progress towards a National Integrity Strategy, to steer its effective implementation, and in the end to evaluate its success and enable a learning process. An indicator is an unambiguous measurement of only one variable. Indicators can be quantitative or qualitative, but they should always be specific, measurable, and realistic.
Figure 2.1. Hierarchy between goals, objectives, and indicators
The successful implementation of a policy will usually be captured through a number of indicators measuring the existence of the relevant activities and functions as well as qualities thereof. Table 2.1 illustrates the different stages of policy implementation and outcome, for which objectives and indicators can be defined.
As can be seen in Table 2.1, a basic distinction can be made between indicators that are measuring aspects of the implementation of integrity policies and indicators that are measuring the level of the desired outcome, in other words the expected change. Indicators measuring policy implementation are measuring aspects that are under the control of implementing agencies. They are capturing the de jure existence of certain processes, regulations or products (outputs), or keeping track of the de facto quality of the implementation or the direct use of the products (intermediate outcome). Outcome indicators, in turn, are measuring the ultimate, often longer term goals, to which the policy measures implemented want to contribute to. While policy-makers have control over the implementation process, and therefore can be hold accountable for these results, they usually do not have complete control over the outcome. Indeed, there are usually many variables outside of the direct control of policy-makers that are also contributing or mitigating the achievement of the ultimate goal. While the implemented measures can have contributed to the observed outcome, it is usually not possible to attribute the change completely to them. This gap between policy measures and outcome is also called attribution gap.
Based on the stated goals, Argentina should therefore aim at identifying a set of indicators measuring both the implementation of the concrete actions derived from the strategic goals and the desired outcome; section 2.3 below gives some guidance on how to gather relevant data for integrity policies. One single indicator is indeed often relatively uninformative. A low number of reported cases of conflict of interest, for example, could mean that few conflicts arose or that few were reported. It does not yet provide any information about how effectively the cases were managed. Also, an increase in sanctions could reflect a more efficient detection and enforcement of integrity breaches, or an increase in such integrity breaches if detection and enforcement rates stayed the same. In turn, assessing various indicators in context to each other can give the observer an idea of what is really happening. As a consequence, the indicators will ideally draw from different sources. Table 2.2 provides an overview of relevant data sources for measuring various aspects and levels of integrity policies, providing examples.
Table 2.2. Potential data sources for indicators measuring public integrity policies
Source: Adapted from (Parsons, Caitlin and Thornton, 2013[5]; United Nations, 2011[6]; Kaptein, 2007[7]).
2.2.2. A central integrity monitoring system, which produces regular reports on the advances of the implementation of the National Integrity Strategy, could help to manage and communicate progress towards integrity goals
Breaking down the objectives into relevant indicators facilitates monitoring the status of implementation as well as the results, but equally relevant is to underpin the monitoring exercise with clear institutional responsibilities and processes. While monitoring is ideally a daily routine in the implementing agencies, a central integrity monitoring system operated outside these entities can ensure that relevant information about the overall status of the implementation of the integrity policies is bundled and analysed jointly to draw a complete and coherent picture that facilitates decision-making, communication, and provide incentives for improvement through benchmarking.
In Argentina, the Executive Office of the Cabinet of Ministers (Jefatura de Gabinete de Ministros, or JGM) is currently keeping track of the implementation of the 100 government priorities (see also chapter 1), amongst others goals related to integrity policies. As such, it would be straightforward to locate the central monitoring function at the level of the recently created Secretary for Institutional Strengthening (Secretaría de Fortalecimiento Institucional) under the Executive Office of the Cabinet of Ministers (Jefatura de Gabinete de Ministros, or JGM). In addition, integrity indicators measuring the implementation and results of the integrity goals should be integrated into the broader national monitoring through the Results Based Management System (Gestión por Resultados, or GpR) headed by the JGM in cooperation with the Ministries of Modernisation and Finance. In this system, all ministries upload the required information on an online platform.
Such a central monitoring system can only work if monitoring begins effectively at the level of the implementing agencies, where data is usually gathered. Currently, both the JGM and the Ministry of Modernisation are providing guidance and assistance to Ministries in developing monitoring systems in their organisations (Ministerio de Modernización, 2017[8]). Integrity indicators should be measured through this system and not in an additional process to avoid creating an additional administrative burden to public managers. Also, the frequency of the measuring and reporting should be clearly established, communicated and enforced.
The core objective of monitoring is to identify challenges and opportunities in a timely manner to inform decisions and enable adjustments during implementation. As such, monitoring should always be understood as contributing to an effective public management. This needs to be institutionalised by linking the process of monitoring with the process of decision-making and implementation. To achieve this, clear mechanisms and procedures to discuss progress should be established. The current roundtables on integrity-related policy issues in Argentina hosted in the Executive Office of the Cabinet of Ministers, or a Commission for Integrity and Transparency as recommended in chapter 1, would provide the platform to discuss progress and challenges, based on such a central integrity monitoring system. However, policy-makers should be aware of the risk of gaming and potential perverse incentives as a consequence of monitoring implementation efforts. To mitigate this risk, monitoring should not be perceived as a control mechanism aimed at naming and shaming but should be clearly communicated as a joint exercise to analyse and overcome challenges. As such, monitoring is a priori an internal process; complete transparency could backfire in this case as it could inhibit honest and open discussions amongst implementing agencies.
Second, a central monitoring system helps also to establish the evidence needed for a regular reporting and communication to the public on the progress made on key selected aggregated indicators. The value of reporting to the public, although critical, is often ignored. Communicating regularly and clearly about the goals and the achievements to internal and external stakeholders as well as the general public can help building trust and show that incremental change is possible. Communication of progress on key selected indicators also enables accountability, increases the credibility of integrity efforts and stimulates a fact-driven dialogue of the government with external stakeholders. This communication can, for instance, be achieved through annual or semi-annual public reports or through a more dynamic website – or both. At organisational levels, the ministries and other public entities could include integrity-related indicators in their annual reports to the citizens and on their websites. At centralised level, the JGM could consider establishing a website where all relevant data on integrity is provided in one single information hub, including relevant information provided on the online platform datos.gob.arg. A similar approach is followed in Colombia, where the Transparency Secretariat has set up an Observatory for Transparency and Anti-corruption where all relevant information is provided in open data format (Box 2.1).
Box 2.1. Providing relevant information to the public – The Colombian Observatory of Transparency and Anti-corruption
The Transparency Secretariat of Colombia has implemented a web portal displaying important indicators related to integrity and anti-corruption. The website bundles available information on: (1) disciplinary, penal and fiscal sanctions; (2) the Open Government Index (Índice de Gobierno Abierto); and (3) the Fiscal Performance Index (Índice de Desempeño Fiscal). The data on the penal sanctions stems from the Prosecutor General’s Office (Fiscalía General de la Nación), the data on disciplinary sanctions from the Attorney General’s Office (Procuradoría General de la Nación), and the data on fiscal sanctions from the Supreme Audit Institution (Auditoría General de la República). The Fiscal Performance Index is elaborated by the National Planning Department (Departamento Nacional de Planeación), while the Open Government Index is calculated by the Attorney General’s Office.
Additionally, the Observatory’s website provides indicators related to Transparency and the implementation status of the Public Anticorruption Policy elaborated by the Transparency Secretariat. The indicators related to Transparency comprise: (1) A composite index of accountability, (2) a composite index of the quality of the Corruption Risk Maps, (3) an indicator related to the demand and supply of public information, and (4) a composite index on the Regional Anticorruption Commissions (Comisiones Regionales de Moralización). The indicators of the Public Anti-corruption Policy are composite indexes (based on overall 24 sub-indexes reflecting the objectives of the Colombian policy) showing the progress made related to the five strategic priorities: (1) improving the access to and the quality of the public information; (2) making more efficient the public management tools for preventing corruption; (3) enhancing social control to prevent corruption; (4) promoting a culture of legality in the State and Society; and (5) reducing the impunity related to corrupt practices.
All indicators are also available in excel format (open data), which makes the data readily usable for research, comparisons and media reports. Details on the methodology for elaborating the indicators are also provided.
Finally, a central monitoring system can be used to create benchmarking that can promote inter-agency competition, or even competition between Regions, as suggested in chapter 1. Indeed, a comparative assessment of the implementation of integrity policies across public entities on key selected aggregated indicators provides additional incentives for public agencies to implement certain policies. This way, the JGM could encourage integrity policies and motivate action without directive power over the different entities of the integrity system. The results could be discussed internally in the Roundtables or the Commission for Integrity and Transparency recommended in chapter 1. This incentive is even stronger when the benchmarking is publicly reported, e.g. through a dedicated website as suggested above.
Box 2.2 describes how this leverage is effectively used by the Anti-Corruption and Civil Right Commission in Korea. The JGM could therefore consider installing a similar scoring system for integrity across entities of the National Public Administration. Such a benchmarking could combine information from the central integrity monitoring system as well as from citizen surveys and staff surveys (see section 2.3). The implementation of integrity policies within the different ministries and agencies even could be assessed more in-depth. In South Korea this is done through a self-assessment of the agencies which is then scored by an assessment team (see again Box 2.2). In Argentina, such an in-depth external expert assessment could be conducted by the Anti-corruption Office, who has the lead over the content part of integrity policies, and discussed in the Roundtables or the Commission for Integrity and Transparency recommended in chapter 1.
Box 2.2. Integrity Monitoring in Korea
The Korean Anti-Corruption and Civil Right Commission (ACRC) uses two complementary assessment frameworks to monitor and assess the quality of implementation of anti-corruption efforts as well as their results: The Integrity Assessments (IA) and the Anti-corruption Initiative Assessments (AIA).
Integrity Assessment (IA)
South Korea annually assesses integrity in all government agencies through standardized surveys. Staff of 617 organisations is asked about their experience with and perception of corruption. Furthermore, citizens who have been in contact with the respective organisations are surveyed as well as stakeholders and experts who have an interest in the respective organisation’s work. The answers are, together with other information, scored into a composite indicator – the Comprehensive Integrity Index.
Anti-Corruption Initiative Assessment (AIA)
The Anti-Corruption Initiative Assessment is a comparative assessment of integrity policies across government agencies in Korea. Agencies selected for assessment submit a performance report on their implementation of integrity policies. On-site visits verify the information, which is then scored by an external assessment team. This allows the Korean Anti-Corruption and Civil Right Commission (ACRC) to observe the willingness and efforts made for integrity across the public sector.
Benchmarking and competition
Underperformance in the IA or the AIA does not lead to sanctions. However, the results are public and the direct comparison of different government entities based on integrity indicators creates a competition between government agencies. The results also enter the Government Performance Evaluation. In addition, there are institutional and individual high-performance rewards, such as an overseas study programme for the officials in charge of outstanding integrity performances. The continuous improvement of the performance results indicates that these incentives might be effective.
Sources: Presentation by Sung-sim Min, Director, Anti-Corruption Survey & Evaluation Division, ACRC, at the meeting of the Working Party of Senior Public Integrity Officials (SPIO) at the OECD Headquarters in Paris in November 2016.
2.2.3. Objective integrity policy evaluations could enable evidence-based learning, improve the design of future integrity policies and credibly reflect the government’s political will
The indicators used for the monitoring of the integrity policies can be used, together with adequate indicators or proxies for the desired outcomes, to evaluate the National Integrity Strategy. Policy evaluation can be understood as “the systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, efficiency, effectiveness, impact and sustainability, etc. Evaluation also refers to the process of determining the worth or significance of an activity, policy or programme” (OECD, 2009[10]). As such, evaluations are asking questions beyond the status of implementation of a given policy in which monitoring processes are interested in. Evaluations usually use a broad spectrum of sources of both quantitative and qualitative information in order to assess whether a policy has contributed to a change, and whether this contribution was efficient and is likely to be sustainable. Understanding the causal impact of a specific policy measure on the outcome variable, however, requires an impact evaluation, which will be discussed in the next section.
Generally speaking, policy evaluations can look at all levels of implementation and outcome as indicated in Table 2.1 above:
Inputs: Evaluation of resources invested such as staff, money, time, equipment, etc.
Activities/Process: Evaluation of how a policy was implemented describing the actual processes employed, often with assessments of the effectiveness from individuals involved or affected by the policy implementation.
Outputs: Evaluation of products delivered by the policy implemented.
Intermediate outcomes: Evaluation of immediate change produced by the policy implemented.
Outcome (Impact): Evaluation of long-term changes and the contribution of the policy implemented.
As such, evaluations are valuable and interesting for both internal and external stakeholders, and the general public – perhaps even more so than information stemming from monitoring. Indeed, people are generally more interested in knowing whether a policy was successful and has achieved the expected results, as in the process of how the policy has been implemented. The evaluation should therefore be considered as an integral part of a National Integrity Strategy in Argentina and be decided and scheduled from the beginning. For instance, an annual evaluation plan could stipulate clearly which aspects, projects or public entities are going to be evaluated. Also, budget should be assigned explicitly for carrying out these evaluations and a process should ensure that the results will be taken into account in the design of the following integrity policies. Again, this decision over policy evaluation could be taken in the Roundtables under the JGM or in the context of the Commission for Integrity and Transparency, recommended in chapter 1.
However, evaluations can only be objective and credible in the view of outsiders, such as citizens and media, when they are impartial. Where the evaluation of the success of a policy is in the same hands as the design or implementation of that policy, the objectivity of measurement is at risk. Therefore, an evaluation of integrity policies should be carried out by an independent partner with an interest in delivering an objective assessment. In Argentina, the JGM, the Ministry of Modernisation, the OA or line ministries could contract out evaluations to a national university, an evaluation institute, a governmental body, an evaluation council, or the private sector, for example. Also, evaluations could be reviewed by a council of representatives from academic institutions and civil society. In addition, clearly pre-defined and openly communicated criteria and parameter for the evaluation can further contribute to increase the impartiality and legitimacy of an evaluation exercise.
In addition, to further increase the independence and impartiality of measurements, the National Institute of Statistics and Censuses (Instituto Nacional de Estadística y Censos, or INDEC) could collect integrity related data to be included in assessments. Mexico’s National Statistics Office (Instituto Nacional de Estadística y Geografía, INEGI) conducts a biennial survey on citizens’ experiences with public sector corruption in a standardised sample of government-provided services (see also section 2.3 below). To be able to contribute with credible data, however, it is key to recover the trust of the INDEC. Between January 2007 and December 2015, the INDEC implemented reforms that affected its credibility and led to allegations of political interference that generated the production of data of low quality or deliberately tendentious. The current government has set the recovery of public statistics as one of its 100 government priorities (see also Box 1.2, chapter 1) and is moving towards completing the process of standardizing public statistics. Ensuring a statistical office that is independent from undue political interference and that provides high-quality data is in itself part of building a public integrity system.
Box 2.3. The quality of official statistics in Argentina
Argentina’s statistics deteriorated over 2007-15 amid growing political pressures to show more “positive” data about the economy and society. The number and quality of underlying censuses, surveys and procedures declined and data on international trade, inflation, GDP and poverty levels became unreliable. In July 2011, the IMF found Argentina in breach of its minimum reporting requirements because of inaccurate provision of CPI and GDP data.
Since 2016, the national statistics institute INDEC has been completely overhauled and its leadership changed. Argentina is now working with the OECD to improve the quality of its statistics. A statistical emergency was declared at the end of 2015, putting the production of some indicators on hold until capacity was rebuilt, which limits the scope for drawing comparisons across time. For some series, the quality of historic data could not be improved and therefore remains subject to reservations. This is particularly the case for household data, which are considered unreliable for 2007 to 2015 as the sample composition may have been altered to obtain desired outcomes. For some series, reliable data are really only available as of mid-2016, preceded by a 6-months data gap due to the statistical emergency. For some variables, notably inflation, making recourse to non-official series for which a longer history is available is the only option. Moreover, poor statistics at the provincial level make comparisons across regions more difficult.
On 30 June 2016, Argentina’s Minister of Treasury and Public Finance expressed his country’s willingness to adhere to the Recommendation of the OECD Council on Good Statistical Practice adopted in November 2015. This document sets out twelve specific recommendations for establishing a sound statistical system, and gives examples of good practice based on OECD countries’ experiences.
Source: (OECD, 2017[11]).
2.2.4. Rigorous impact evaluations could be used to test innovative integrity measures before considering implementing them at larger scale
While policy evaluations are looking for evidence for the contribution of a policy to the integrity goals, impact evaluations aim to measure the causal effect of a policy in the sphere where it aimed to create a change. Such causal attribution is the main challenge in policy evaluation. In essence, impact evaluations allow verifying ex post the theory of change, that is, the causal logic of a policy measure. Did the whistleblowing mechanism actually encourage people to speak up? Did the new campaign really raise awareness for integrity?
Impact evaluations measure the causal effect of a policy, for example through a randomized control trial or quasi-experimental methods (Johnsøn and Søreide, 2013[12]; OECD, 2017[4]). However, a causal effect can be challenging to identify. Figure 2.2 illustrates why: the change of an outcome after the policy intervention, might not or not fully be caused by the intervention. Some of it might have happened even without the intervention. Points A and B in the figure can often be measured. The outcome indicator could, for example, be the percentage of employees indicating high integrity in a staff survey. This percentage could be measured before (A) and after (B) an integrity training. What remains unknown, however, is the percentage of staff who would have reported high integrity without the training (X). The true impact of the training thus remains unknown.
Figure 2.2. The missing counterfactual
Source: Adapted from (Johnsøn and Søreide, 2013[12]).
A variety of experimental and quasi-experimental methods allow estimating the true impact of a policy intervention. Oftentimes, the impact of a policy can be observed through an experiment. The effect of the integrity training on the integrity survey response could, for example, be assessed in a randomised control trial: Groups of participants could be randomly chosen to receive the training before others do. The survey responses of those employees who had the training could then be compared to the responses of those who have not yet taken part. A random selection process ensures that, by the law of large numbers, the two groups are comparable and any difference between them can be assigned to integrity training only and not to unobserved differences between the individuals in the groups.
Due to the difficulties in the logistics and in the definition of adequate indicators, such rigorous impact evaluations particularly make sense for testing specific innovative measures before recommending scaling up their implementation across the public administration. Among the policy areas that could greatly benefit from impact evaluation are: integrity trainings (e.g. teacher training), applications of behavioural insights (OECD, 2018[13]), design and re-design of disclosure policies (e.g. asset disclosure system), as well as communication and awareness-raising campaigns. Where a new procedure is introduced or an existing process is changed, testing the variation in a randomized control trial can provide valuable information. If the Anti-corruption Office, for example, wanted to send out reminder emails about Asset Declarations, an evaluation of a small scale pilot could test different wordings. Chapter 3 describes how such a testing has been done by the Ministry of Public Administration (Secretaría de la Función Pública, or SFP) in Mexico with the aim to remember public officials to declare any gift they received, while testing different messages.
Impact evaluations require statistical and methodological expertise. While the testing of policies in an experimental set-up can be straightforward, in some contexts the measurement of impact can be methodologically and/or logistically challenging. To undertake such impact evaluations, public entities wishing to undertake an impact evaluation, could build strategic partnerships with academic institutions with expertise in the field. For instance, in the example mentioned above, the SFP in Mexico partnered with the Centre of Economic Research and Teaching (Centro de Investigación y Docencia Económicas, or CIDE) to jointly design, conduct and analyse the impact evaluation. As the focal point of the integrity system, the Anti-corruption Office and the Secretary for Institutional Strengthening in JGM could identify the need for impact evaluations across the integrity system. It could then make sure the task is commissioned to a competent institution and provide the researchers with guidance and access to relevant data. The Ministry of Modernisation, in turn, could ensure the quality of methodological approaches.
2.3. Gathering relevant data for integrity policies
2.3.1. Expanding and adapting the use of staff surveys could provide data on how integrity policies contribute to a culture of integrity in Argentina’s public sector
Monitoring and evaluation both require information. Administrative data, that is data collected for administrative purposes by government units, can provide relevant insights, but to understand better the public administration and get information on how public officials perceive and experience issues related to public integrity, administrative data needs to be complemented by staff surveys.
Staff surveys can inform directly integrity policies. They support the assessment of public officials’ integrity capacities and risk and allow identifying the values and challenges that impact public officials in their choices. Also, policy makers will be interested to find out whether the policies resonate with public employees, change their attitudes and behaviours and serve the policy goal of ensuring a culture of integrity within the public sector. Staff surveys could further support the diagnostic assessment preceding the design of integrity policies. In fact, many OECD member countries monitor their integrity policies using employee survey polls. In the Netherlands, for example, a comprehensive staff survey is at the core of agenda setting for future integrity policies (see Box 2.4).
Box 2.4. Integrity Monitor in the Dutch public administration
Since 2004, the Dutch Ministry of the Interior regularly observes the state of integrity in the Dutch public sector. To this end, political office holders, secretaries-general, directors and civil servants are surveyed in central government, provinces and municipalities using mixed methods, including large-sample online surveys and in-depth interviews.
The Integrity Monitor supports Dutch policy makers in the design, implementation and communication of integrity policies. The results of each Monitor are reported to Parliament. The Ministry of Interior uses the Monitor to raise ethical awareness, detect implementation gaps and engage decentralised public administration in taking responsibility for integrity regulations. Insights from past Monitor waves have helped to identify priorities for anti-corruption efforts and shift integrity policies from prohibition to creating an organisational culture of integrity.
Sources: Presentation by Marja van der Werf (Dutch Ministry of Interior and Kingdom Relations) given at the meeting of the OECD Working Party of Senior Public Integrity Officials (28 March 2017, Paris).
Lamboo T. & De Jong, J. (2015): Monitoring Integrity. The development of an integral integrity monitor for public administration in the Netherlands. In Hoekstra, A. & Huberts, L. Gaisbauer, I. (eds.), ‘Integrity Management in the Public Sector. The Dutch Approach’.
More specifically, a centrally administered public sector staff survey touching upon various aspects of public employment has the advantage that answers can be correlated and compared across entities. In the United Kingdom, the Civil Service People Survey yearly interviews almost 300 000 respondents from 98 organisations (Civil Service People Survey 2017, 2017[14]). The results provide a benchmark across different public entities. Among the various aspects covered by the survey are employee engagement, trust in leadership and fair treatment. A significant change in any of these dimensions could indicate an integrity risk. The scoring of the answers in indices gives not only an overall picture of the UK Civil Service, but also points towards the challenges within individual organisations. The repetition of the same questions in regular intervals enables comparisons over time. Positive responses to the question “Are you confident that if you raise a concern under the Civil Service Code in [your organisation] it would be investigated properly?”, for example, have been increasing from 58% in 2009 to 70% in 2017 – potentially indicating a success of UK integrity policies. In Colombia, similar information is collected by the National Statistical Office (Departamento Administrativo Nacional de Estadística, or DANE) that carries out an annual Survey on National Institutional Environment and Performance. The survey entails a question on “irregular practices”, which includes questions on the effectiveness of specific integrity initiatives.
Overall, 19 OECD countries conduct centrally administrated staff surveys across the whole central public administration (OECD, 2017[15]). Many of them include questions on integrity, as the first column of Table 2.3 shows. Meanwhile, questions that do not directly relate to integrity can also be a source of information for integrity policy making. In the United States for example, the central employee survey includes the item “My supervisor listens to what I have to say” – the response of which is likely to be an indicator of organisational culture (OECD, 2017[15]).
Table 2.3. Staff surveys inform evidence-based integrity policies in OECD countries
Note: Information on data for Israel: http://dx.doi.org/10.1787/888932315602. For further country-specific information as well as details on the methodology and factors used in constructing the index see www.oecd.org/gov/indicators/govataglance. Where data was unavailable this is indicated by -.
Source: OECD (2016), Strategic Human Resources Management Survey, OECD, Paris & OECD (2016), Survey on Public Sector Integrity, OECD, Paris.
Argentina could as well consider a centrally administered and regular public employee survey in the whole National Public Administration. It is recommended that the National Institute for Public Administration (Instituto Nacional de Administración Pública, or INAP), administers such a central public employee survey of all staff in the National Public Administration on a regular basis. Alternatively, the Survey could be conducted by the National Statistical Office in Argentina, National Institute of Statistics and Censuses (Instituto Nacional de Estadistica y Censos, INDEC). The survey could include various aspects relevant for public sector human resource management in addition to questions on integrity. The results of the survey could serve for an internal benchmarking and risk analysis across different entities of the National Public Administration and potentially feed into a benchmarking (see section 2.2.1).
Either to complement or instead of a centrally administered survey, Argentina could also conduct more limited, ad hoc staff surveys in specific sectors or public entities to inform integrity policy making. However, while they can be relevant for purposes of policy design or communication, for example, such ad hoc surveys lack the advantages of a centrally administrated and regular survey that can be used for benchmarking purposes and for evaluating the longer term progress in integrity goals.
In Argentina’s integrity system, first steps have been taken in the use of staff surveys to inform integrity policies, yet, there is potential for expanding the use of this measurement tool. For example, Argentina’s Ministry of Modernisation in 2016 conducted a staff survey among public employees, asking about their own and their organisation’s values and behaviours. The collected evidence informs policies designed to shape the organisational cultures within the public sector. In cooperation with the Ministry of Modernisation, the Anti-corruption Office could investigate whether these data withhold relevant information for creating a culture of integrity among staff in the public sector. In general, there are three functions in which the data from this survey might serve integrity policies.
First, the responses could inform the diagnostics of the existing organisational culture within Argentina’s public sector. Argentina is aiming to revalue public employment (Government goal number 85). This policy goal and the Human Resource Management policies that are implemented in the public sector to this end also have implications for integrity. Behavioural research has shown that a positive moral identity supports ethical behaviour (OECD, 2018[13]). Staff mentioning negative or unethical values to describe themselves or their organisation could indicate a damaged moral identity and thus an integrity risk. Careful analysis of the survey response could help to target the respective policies.
Second, the survey data could inform the design of integrity-related human resource policies in Argentina’s public sector. Survey participants indicated which values and behaviour staff would hope to see in their organisation in order for the organisation to achieve its full potential. The design of policies aiming to create a culture of integrity could attempt to strengthen in particular these mentioned values and behaviours.
Third, the survey could also function as a baseline for the work on culture of integrity in the Argentine public sector. A repetition of the same survey after a few years would allow observing changes in the reported values. The scope of the survey does not need to be limited to obvious direct questions. Feelings, attitudes and opinions can be examined in carefully designed survey studies. Box 2.5 provides two examples of survey techniques that can be applied to assess sensitive information or underlying values.
Box 2.5. Survey techniques
Random Response
Random response allows respondents to more openly admit to a stigmatized answer. A question could be:
Have you ever made a misstatement on a job application?
The respondents then toss a coin. If it shows heads, they answer yes. If it shows tails, they answer the question truthfully. The true prevalence of the behaviour can be estimates from all yes answers distracting the proportion stochastically attributed to the coin toss. Yet, for the individual respondent the inhibition to respond yes is reduced.
Semantic Differential
A Semantic Differential is a survey tool used to gauge the feelings of respondents associated with a certain trigger word. It shows opposing pairs of adjectives (e.g. confident – shy; arrogant – humble; fast slow) on the ends of a 5 step interval scale. The respondents indicate where on the scale they associate the word. Answers are fully subjective, which reveals how the word is connoted.
Source: (Kraay and Murrell, 2016[16]).
As a complement to general staff surveys, training programmes and awareness raising initiatives for the public sector provide an opportunity for gathering data from public officials as well. Valuable evidence could, for example, be generated through an integrity knowledge assessment on a selected sample of public officials. Such a survey could assess public employees’ competency to solve ethical dilemmas, dismantle common justification mechanisms for unethical behaviour or identify integrity breaches and knowledge of reporting procedures. The results of this assessment could serve as diagnostics for which entities require integrity training and inform INAP on where and how to best integrate integrity matters in their training curricular.
2.3.2. Data from citizen interactions with government and citizens surveys could help to understand how integrity policies affect awareness for integrity in the whole of society and encourage social accountability
Citizen surveys are a further tool to inform national integrity policies. Some existing studies that have surveyed citizens in Argentina (e.g. World Values Survey, Americas Barometer, Latinobarometer, Global Corruption Barometer, or the Latin American Public Opinion Poll) provide insights on the prevailing attitudes and opinion towards corruption. However, even though they may provide interesting starting points, these surveys are designed for international comparison rather than to allow a more detailed assessment of integrity within a country context.
Citizen surveys are a flexible tool. Their content can be adjusted depending on the intended use of the data. Box 2.6 provides core elements that have been included in citizen surveys in OECD countries and how they might inform integrity policy making. A citizen survey could include questions on perception of government officials of different public institutions. It could ask citizens about the roots of their distrust in government and the attitudes and values they connect with public office. A random sample of citizens could, for example, be presented with a Semantic Differential Scale to express their associations with the Argentine public sector (see Box 2.5 above). Repeating a similar measurement on the same or a comparable sample after the implementation of the trust-building policies could be a test for an attitude changing effect.
Box 2.6. Potential elements of a public citizen survey for integrity
Citizen surveys are a valuable measurement tool to gather evidence for the design, implementation and communication of integrity policies. This tool can be applied for various purposes. The following elements could be included in an integrity survey:
Values and attitudes: A values and attitudes survey is particularly useful as an ex-ante diagnostic tool for awareness raising and behaviour changing policies. In order to design an awareness raising campaign with an effective appeal, a survey can be conducted to find out which integrity-related values matter to large groups of citizens.
Experience, awareness and perception of corruption: Transparency International’s Global Corruption Barometer and other international survey studies can provide a brief impression of how common bribery is in Argentina, where and when citizen experience bribery and how strong corruption is perceived by citizens in different sectors. This evidence shows that bribery is not a major problem for citizens interacting with public sector officials in Argentina (see chapter 3). Nonetheless, corruption is deeply entrenched in societal interactions. A survey targeted solely at Argentine citizens could assess the experiences, awareness and perception of corruption more in-depth with respect to the Argentine context.
Dilemmas and justifications: Strengthening culture of integrity in the whole of society means enabling citizens to identify and react of an integrity breach. International surveys show that most citizens in Argentina reject corruption when directly asked about it (see chapter 8). The situations in which citizens do engage in integrity breaches are less obviously identified or justified. Confronting citizens with practical ethical dilemmas in survey questions could help integrity policy makers gain an understanding of which integrity breaches are commonly accepted in society, who people assign responsibility to and how citizens bring these in accordance with their general rejection of corruption.
Such survey data can help steer the public debate on integrity to a more actionable context inducing actual behavioural change. Successful behaviour changes in the public sector can be mitigated by a lack of acknowledgement in the public, who show limited trust in public institutions and their capacity to reduce corruption. Therefore, chapter 8 suggests facilitating a trust-inducing dialogue between the public sector and citizens. Surveys provide the evidence base for this dialogue by giving policy makers the opportunity to understand how citizens perceive the public sector and where trust in public institutions is lacking.
In order to have regular and comparable data over time, Argentina could therefore consider adding questions or modules to existing household surveys or having separate citizen surveys conducted by the National Institute of Statistics and Censuses (INDEC). These questions could be developed based on international good practice and fine-tuned together with the Anti-corruption Office to fit the country context.
For ad-hoc information needs that go beyond the content of regularly conducted surveys, integrity policy makers in Argentina could collaborate with external partners. Opinion researchers in Argentina, such as the Centre for Opinion Research and Social Studies at the University of Buenos Aires or the Centre for Public Opinion at the University of Belgrano. Both institutions issue regular public surveys. In cooperation with academic researchers or private sector survey companies, the Anti-corruption Office, as well as the Ministry of Modernisation or other integrity actors, could investigate relevant aspects of public opinion, knowledge and experience to inform integrity policy making. If, for example, the Anti-corruption Office were to launch a new awareness raising campaign, a brief public opinion survey could be conducted afterwards to see whether citizens are aware of it.
Measures of trust in general are also a valuable source of evidence for integrity policy making. Trust in public institutions is at the core of public integrity, and integrity has been shown to be the main driver for trust in government (Figure 2.3). In turn, trust can determine the success of integrity policies, programmes and regulations that depend on cooperation and compliance of citizens. Lack of trust compromises the willingness of citizens and business to respond to policies and contribute to public integrity. Integrity, in turn, is one of the five amenable policy dimensions driving public trust (OECD, 2017[17]).
Figure 2.3. Integrity is perceived as the most crucial determinant of trust in Government
Note: This figure shows the most robust determinants of self-reported trust in government in an ordinary least squares estimation that controls for individual characteristics.
Source: Trustlab (SVN, DEU, USA, ITA).
The OECD Guidelines on Measuring Trust provide international recommendations on collecting, publishing, and analysing trust data to encourage their use by National Statistical Offices (NSOs) (OECD, 2017[18]). Argentina’s National Institute of Statistics and Censuses (INDEC) could consider implementing regular measurements of trust in public institutions through surveys in order to correlate these results with other integrity related indicators and analyse these relationships as well as changes over time.
Box 2.7. Measuring trust in public institutions
Questions suggested in the OECD Guidelines on Measuring Trust
The next questions are about whether you have trust in various institutions in [Argentina]. Even if you have had very little or no contact with these institutions, please base your answer on your general impression of these institutions. Using this card, please tell me on a score of 0-10 how much you personally trust each of the institutions I read out. 0 means you do not trust an institution at all, and 10 means you have complete trust.
A3. [Argentina’s] Parliament?
A4. The police?
A5. The civil service?
B10. The courts?
B11. Political parties?
B12. Politicians?
B13. The police?
B14. The armed forces?
B15. The civil service?
B16. The media?
B17. The banks?
B18. Major companies?
The following questions are about your expectations of behaviour from public institutions. In each question, you will be asked whether you think a particular example of behaviour is something that would be expected not to occur at all, or to always occur. Please respond on a scale from 0 to 10 where 0 means very unlikely and 10 means very likely.
C3. If you were to complain about bad quality of a public service, how likely is that the problem would be easily resolved?
C4. If a natural disaster occurs, do you think that the provision by government of adequate food, shelter and clothing will be timely and efficient?
C5. If a decision affecting your community were to be taken by the local or regional government, how likely is it that you and others in the community would have an opportunity to voice your concerns?
C6. If an individual belongs to a minority group (e.g. sexual, racial/ethnic and/or based on national origin), how likely is it that the individual will be treated the same as other citizens by a government agency?
Have you done any of the following in the past month?
D4. Voiced your opinion to a public official?
D5. Signed a petition?
Source: OECD (2017), OECD Guidelines on Measuring Trust, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264278219-en.
Citizens’ experiences and opinions can also guide the improvement of public services and the advancement of integrity. Users of public services in Argentina could, for example, be invited to answer a short satisfaction survey. Anonymously, respondents could indicate not only the quality of public service delivery, but also how they perceived the integrity of the institution or public official they interacted with. The integration of feedback tools in situations where citizens and public officials interact not only allows gathering data on the quality of public service but could also contribute to building trust in public institutions. Citizens feel that their voice is heard and perceive the institution as open for dialogue. Experience in Lithuania (see Box 2.8) shows that this could also reduce the willingness to engage in corruption. Feedback mechanisms give citizens the chance not only to legitimately reward a good experience, but also to bring a negative experience to awareness, thereby ideally preventing the citizen from generalizing one bad experience to the public sector overall. Meanwhile, respondents with a positive experience reflect upon this good experience when providing feedback, which might support an updating of their existing attitudes towards the public sector.
Box 2.8. Clinic Evaluation in Lithuania
Transparency International Lithuania installed a feedback tool at a public clinic. They found that patients who evaluated their visits were less willing to pay a bribe, communicating more respectfully with the doctors and felt better attended to. Placing the feedback station visibly in the lobby of the clinic proved important to invite all visitors to give feedback.
Source: TI Lithuania (n.d.), Patients who evaluate their visit are less willing to give bribes – Transparency International Lietuvos skyrius,
www.transparency.lt/en/pacientai-kurie-ivertina-savo-apsilankyma-pas-gydytoja-maziau-linke-duoti-kysius/ (accessed on 13 December 2017).
In addition, the increase in digitalization of interactions between citizen and public sector yields a variety of opportunities to integrate brief questions into the procedure that later can be used for evaluating the impact of integrity policies as well. The National Institute of Statistics and Censuses could develop a standardized phrasing and assessment framework for such feedback questions in order to ensure comparability across different institutions. Another way to enable feedback from citizens to public officials is a so-called ambient accountability mechanism: a physical element, e.g. a poster or a screen, placed directly in the public office for citizens to leave comments (Zinnbauer, 2012[19]). The installation of such ambient accountability mechanism in public offices in Argentina could be implemented in a randomized fashion and accompanied by evaluation.
Along the same line, Argentina’s government provides data and engages in dialogue with citizens on web-based platforms, such as the Open Government Platform www.datos.gob.ar, the Ministry of Justice’s Justicia 2020 online platform, online complaint mechanisms and the web-based Conflict of Interest Simulator. The interaction with citizens on these web-platforms could be an entry point for asking short survey questions, such as whether users are satisfied with the respective government policies. However, the government would also need to communicate better the existence of these and other online tools.
Moreover, surveys can help policy makers in Argentina to develop effective communication strategies. Awareness raising for integrity is more complex than simply raising the issue of corruption. For instance, simply raising the issue of corruption might have the opposite effect if it increases the already high awareness for an existing problem and thereby creating the impression that corruption is widespread. Research has shown that unethical behaviour is contagious (Gino, Ayal and Ariely, 2009[20]) and the perception of corrupt behaviour as common could undermine non-corrupt social norms (Bicchieri and Xiao, 2009[21]). Such dangers of miscommunication can be prevented by designing a relevant campaign based on evidence extracted from citizen surveys. Changes in citizens’ attitudes and opinions as a result of the campaign can be measured and monitored through surveys. An initial public opinion survey conducted to inform the design of the awareness raising policy could serve as a baseline for such an assessment. In comparison to this initial data, a new survey a relevant time after the implementation of the campaign could assess a potential effect of the campaign and enable a presumption of the policy’s effectiveness.
Proposals for action
Implementing a system for monitoring and evaluating Argentinian integrity policies
The monitoring and evaluation of integrity policies requires relevant indicators measuring different levels and that are based on different data sources.
A central integrity monitoring system, which produces regular reports on the advances of the implementation of the National Integrity Strategy, could help to manage and communicate progress towards integrity goals. Ideally, this central integrity monitoring function would be located at the level of the Secretary for Institutional Strengthening under the JGM. Integrity indicators measuring the implementation and results of the integrity goals should be integrated into the broader national monitoring through the Results Based Management System (GpR) headed by the JGM in cooperation with the Ministries of Modernisation and Finance.
At organisational level, where the data is usually collected, integrity indicators should be measured through already existing systems and processes to avoid creating an additional administrative burden to public managers. Also, the frequency of the measuring and reporting should be clearly established, communicated and enforced.
In addition, the current roundtables on integrity-related policy issues in Argentina hosted in the JGM, or a Commission for Integrity and Transparency, would provide the platform to discuss progress and challenges, based on such a central integrity monitoring system.
A central monitoring system establishes the evidence needed for a regular reporting and communication to the public. At organisational levels, public entities could include integrity-related indicators in their annual reports to the citizens and on their websites. At centralised level, the JGM could consider establishing a website where all relevant data on integrity is provided in one single information hub.
Finally, a central monitoring system can be used to create benchmarking that can promote inter-agency competition, or even competition between Regions. Again, the results could be discussed internally in the Roundtables or the Commission for Integrity and Transparency. The incentives provided through benchmarking are even stronger when the benchmarking is publicly reported, e.g. through a dedicated website as suggested above.
Objective integrity policy evaluations could enable evidence-based learning, improve the design of future integrity policies and credibly reflect the government’s political will. The evaluation should therefore be considered as an integral part of a National Integrity Strategy in Argentina and be decided and scheduled from the beginning.
Budget should be assigned explicitly for carrying out these evaluations and a process should ensure that the results will be taken into account in the design of the following integrity policies. Again, this decision over policy evaluation could be taken in the Roundtables under the JGM or in the context of the National Commission for Integrity and Transparency.
The JGM, the Ministry of Modernisation, the OA or line ministries could contract out evaluations to a reputable external institution and evaluations could be reviewed by a council of representatives from academic institutions and civil society. Clearly pre-defined and openly communicated criteria and parameter for the evaluation can further contribute to increase the impartiality and legitimacy of an evaluation exercise.
In addition, to further increase the independence and impartiality of measurements, the INDEC could collect integrity related data to be included in assessments.
Rigorous impact evaluations could be used to test innovative integrity measures before considering implementing them at larger scale. Where a new procedure is introduced or an existing process is changed, testing the variation in a randomized control trial can provide valuable information. Public entities could therefore consider conducting rigorous impact evaluation by partnering with research institutes. As the focal point of the integrity system, the Anti-corruption Office and the Secretary for Institutional Strengthening in JGM could identify the need for impact evaluations across the integrity system. It could then make sure the task is commissioned to a competent institution and provide the researchers with guidance and access to relevant data. The Ministry of Modernisation, in turn, could ensure the quality of methodological approaches.
Gathering relevant data for integrity policies
Expanding and adapting the use of staff surveys could provide data on how integrity policies contribute to a culture of integrity in Argentina’s public sector.
Argentina could consider a centrally administered and regular public employee survey in the whole National Public Administration, carried out ideally by the INAP or alternatively the INDEC.
Either to complement or instead of a centrally administered survey, the Anti-corruption Office or other public entities could also conduct more limited, ad hoc staff surveys in specific sectors or public entities to inform integrity policy making.
Training programmes and awareness raising initiatives for the public sector also provide an opportunity for gathering data from public officials.
Data from citizen interactions with government and citizens surveys could help to understand how integrity policies affect awareness for integrity in the whole of society and encourage social accountability.
To have regular and comparable data over time, Argentina could therefore consider adding questions or modules to existing household surveys or having separate citizen surveys conducted by the INDEC. These questions could be developed based on international good practice and fine-tuned together with the Anti-corruption Office to fit the country context.
For ad-hoc information needs that go beyond the content of regularly conducted surveys, integrity policy makers in Argentina could collaborate with external partners to conduct citizen surveys.
References
[21] Bicchieri, C. and E. Xiao (2009), “Do the right thing: But only if others do so”, Journal of Behavioral Decision Making, http://dx.doi.org/10.1002/bdm.621.
[14] Civil Service People Survey 2017 (2017), Civil Service benchmark scores, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/659565/Civil_Service_People_Survey_2017_Benchmark_scores.pdf (accessed on 21 December 2017).
[20] Gino, F., S. Ayal and D. Ariely (2009), “Contagion and Differentiation in Unethical Behavior The Effect of One Bad Apple on the Barrel”, Psychological Science, Vol. 20/3, pp. 393-398.
[12] Johnsøn, J. and T. Søreide (2013), Methods for learning what works and why in anti-corruption: An introduction to evaluation methods for practitioners, Chr. Michelsen Institute, Bergen, https://www.cmi.no/publications/4876-methods-for-learning-what-works-and-why-in-anti (accessed on 9 January 2018).
[7] Kaptein, M. (2007), “Developing and Testing a Measure for the Ethical Culture of Organizations: The Corporate Ethical Virtues Model”, ERIM Report series Research in Management, No. ERS-2007-084-ORG, Erasmus Research Institute of Management (ERIM), http://hdl.handle.net/1765/10770.
[16] Kraay, A. and P. Murrell (2016), “Misunderestimating corruption”, Review of Economics and Statistics, Vol. 98/3, pp. 455-466, http://dx.doi.org/10.1162/REST.
[8] Ministerio de Modernización (2017), Guía Práctica No. 3: Sistema de Gestión por Resultados: Para la Etapa de Monitoreo y Evanulación, Subsecretaria de Innovación Pública y Gobierno Abierto, Buenos Aires.
[3] Ministerio de Modernización (2016), Guía Práctica No. 1: Sistema de Gestión por Resultados: Para la Etapa de Planificación Estratégica., Subsecretaria de Innovación Pública y Gobierno Abierto, Buenos Aires.
[13] OECD (2018), Behavioural Insights for Public Integrity: Harnessing the Human Factor to Counter Corruption, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264297067-en.
[15] OECD (2017), “Employee surveys”, in Government at a Glance 2017, OECD Publishing, Paris, http://dx.doi.org/10.1787/gov_glance-2017-50-en.
[4] OECD (2017), Monitoring and Evaluating Integrity Policies, Working Party of Senior Public Integrity Officials GOV/PGC/INT(2017)4, Paris.
[11] OECD (2017), OECD Economic Surveys: Argentina 2017: Multi-dimensional Economic Survey, OECD Publishing, Paris, http://dx.doi.org/10.1787/eco_surveys-arg-2017-en.
[18] OECD (2017), OECD Guidelines on Measuring Trust, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264278219-en.
[9] OECD (2017), OECD Integrity Review of Colombia: Investing in Integrity for Peace and Prosperity, OECD Public Governance Reviews, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264278325-en.
[1] OECD (2017), OECD Recommendation of the Council on Public Integrity, http://www.oecd.org/gov/ethics/recommendation-public-integrity.htm.
[17] OECD (2017), Trust and Public Policy: How Better Governance Can Help Rebuild Public Trust, OECD Public Governance Reviews, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264268920-en.
[2] OECD (2016), Open Government: The Global Context and the Way Forward, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264268104-en.
[10] OECD (2009), “OECD DAC Glossary”, in Guidelines for Project and Programme Evaluations, OECD, Paris.
[5] Parsons, J., G. Caitlin and M. Thornton (2013), Indicators of Inputs, Activities, Outputs, Outcomes and Impacts in Security and Justice Programming, Department for International Development (DFID), London.
[6] United Nations (2011), The United Nations Rule of Law Indicators - Implementation Guide and Project Tools, Department of Peacekeeping Operations and Office of the High Commissioner for Human Rights, New York, http://www.un.org/en/peacekeeping/publications/un_rule_of_law_indicators.pdf (accessed on 1 March 2018).
[19] Zinnbauer, D. (2012), “’Ambient Accountability’ - Fighting Corruption When and Where it Happens”, SSRN Electronic Journal, http://dx.doi.org/10.2139/ssrn.2168063.