Countering Fraud in Social Benefit Programmes

4. Evaluation of anti-fraud measures

4.1. Fostering continuous improvement with evaluation

Attempting to measure fraud and determining the effectiveness and impact of fraud prevention and detection activities poses many challenges. Understanding the what, when and how of measurement is complicated by the hidden nature of fraud. Nonetheless, assessments or evaluations are vital as they offer insights into an ongoing or completed activity and inform decisions about the relevance and efficacy of fraud prevention and detection measures. This often requires the systematic collection of evidence dealing with the design, implementation and results of policies, controls and actions taken to manage fraud risks. If measures are not effective, evaluations can help determine potential alternatives for governments.

Approaches to measurement vary widely. Sample-based measurements can provide reasonable estimates for fraud and error, which can then lead to insights into what is working from a governance, oversight and control perspective. Public organisations in some countries, such as the United Kingdom, the United States and Ireland, randomly sample benefit files to detect inaccuracies. Many existing methods can lead towards underestimation or overestimation of the problem. For instance, using administrative data such as prosecutions and investigations tends to underestimate the problem, although such data can be useful inputs for measurement and evaluations. On the other hand, perception-based measurements can overestimate the prevalence of fraud and error in a system.

There are a number of ways that public organisations can measure and evaluate how effective their fraud prevention and detection measures are. These include:

• Evaluating control activities - Measuring the efficacy of controls allows governments to put in place results-based performance management systems to ensure that fraud prevention is part of employees’ daily tasks.

• Establishing baselines and assessing cost-effectiveness of anti-fraud measures - Measuring changes in the rate of fraud can help determine the impact of control activities. However, this is only possible if a baseline is established and if data on fraud are routinely collected.

• Setting annual targets for fraud prevention and publishing results – This can enable public organisations to evaluate their prevention and detection activities while incentivising managers in their efforts to reduce fraud in SBPs.

4.2. Evaluating control activities

Public organisations should evaluate their anti-fraud control activities to determine what is working and where there is room for improvement. This includes testing control design effectiveness and the operational effectiveness of a control. The need for evaluation is reflected in international standards, such as Principle Five of the Fraud Risk Management Guide developed by the Committee of Sponsoring Organizations of the Treadway Commission (COSO). The guide describes how establishing measurement criteria enables monitoring and evaluation, as well as analytical comparisons of an organisation’s fraud risk management activities (Committee of Sponsoring Organizations of the Treadway Commission, 2016[32]). Many public organisations adhere to an anti-fraud policy or risk management standards that are implemented and harmonised across government. However, due to the nature of SBPs and the unique way in which responsible entities interact with the public, as well as the sheer number of beneficiaries, the approach to evaluation of anti-fraud control activities may differ.

While countries are adopting new methods and tools to tackle fraud in SBPs, there is still considerable progress to be made regarding how public organisations monitor and evaluate their fraud prevention and detection measures. Entities may face challenges measuring outcomes of their anti-fraud control activities in a reliable way. These challenges include the difficulty of measuring the extent of deterred fraud, isolating potential fraud from legitimate activity or other forms of improper payments, and determining the amount of undetected fraud (Government Accountability Office, 2015[29]). Despite these challenges, there are ways that managers can test the effectiveness of their control measures and the short-term or intermediate outcomes of these activities. For example, monitoring the activities of dedicated anti-fraud units and the number of potential or real cases of fraud they uncover when reviewing recipients’ claims is a simple way of assessing their impact on fraud levels.

To assess operating effectiveness of control activities, public organisations can do a “walkthrough” of end-to-end processes with the appropriate personnel, mapping out areas at-risk to potential fraud with the corresponding control activity. Applied to claimant registration processes in SBPs, this mapping could lay out each step and specify the key controls, whether they are manual or automated, and how frequently they are applied. Most importantly, the walkthrough should detail how the operation of the control is evidenced, i.e. sign-off processes by different levels of staff, secondary review mechanisms, receipt of cross-referenced documentation. This is just one way that entities can assess control effectiveness. Other methods include examination and inspection tests, re-performance of control activities, and inquiries into control effectiveness. Many key controls are likely to be subject to existing internal or external audit reviews; this should be taken into consideration in control test planning schedules.

4.3. Establishing baselines and assessing cost-effectiveness of anti-fraud measures

Evaluation of anti-fraud measures can complement other assessments to understand baselines for levels of fraud and cost-effectiveness of anti-fraud measures. A baseline allows for monitoring changes in the rates of fraud based on changes in the control environment, which is a critical feedback loop for managerial decision-making. Without such baselines, it is difficult for entities to measure the efficacy of control activities. If data are available, public organisations can draw from statistical models to determine fraud rates and create baselines (OECD, 2019[17]). While a baseline for fraud levels is difficult to establish, governments can consider several approaches, including the following:

• Examining historical data for fraud to establish a fraud rate, preferably time-series data over a number of years, i.e. what percentage of claims are fraudulent, what percentage of recipients submitted a fraudulent claim.

• Undertaking comprehensive, large-scale audits or risk assessments to help establish the rate of fraud based on identifying cases of suspected fraud. If fraud is confirmed, results can then provide insights into a likely fraud rate based on programme size (e.g. number of recipients; value of detected frauds), although it may not be possible to generalise or apply findings from the results.

• Random sampling of cases where there is a specific focus on finding suspicious cases. Given proper methodological design, identified results can be generalised to entire programmes (OECD, 2019[17]).

These approaches can help governments to benchmark their progress and foster a long-term view of fraud prevention in SBPs. Examples of metrics that can inform the development of the baseline include:

• the number, percentage and value of fraudulent claims identified

• the number and percentage recipients committing fraud

• the number and percentage of private providers committing fraud

• the number, percentage and value of fraudulent transactions involving different goods and services across SBPs

• control weaknesses or fraudulent activity uncovered through inspections or audits.

Once a baseline of fraud is established and new control activities are applied, it is easier for governments to monitor if levels of fraud are affected, as well as to identify trends and higher risk areas.

Baselines can also be inputs for cost-benefit analyses. Determining whether control activities are cost-effective can also be a difficult task depending on what data are available. Estimating the return on investment (ROI) in approaches to tackling fraud is not self-evident, and particularly when data analytics models have been implemented, it may be necessary to assess the total cost of ownership of these models as well as the impact of fraud on the entity, which is difficult to determine. Assessing the cost-effectiveness of fraud prevention measures cannot be boiled down to typical, numerical ROI calculations given difficulties in determining the number of fraudulent cases prevented. Despite these limitations, entities responsible for SBPs can keep track of certain observations as part of the baseline analysis to determine if their control activities are cost-effective. For example, monitoring the number of newly detected cases over a certain period, or changes in the number of cases that were detected early compared to previous cases.

4.4. Setting annual targets for fraud prevention and publishing results

For public organisations that are responsible for delivering SBPs, setting annual targets for fraud reduction or prevention measures can facilitate the measurement of fraud. This is a practice observed in some OECD countries. In Canada and New Zealand, respectively, the government has developed fraud reduction-related targets for managers responsible for SBPs (OECD, 2019[17]). As well as reducing fraud levels and enabling measurement, the introduction of such targets can also have implications for performance management within public organisations (Box ‎4.1).

Box ‎4.1. Measuring progress of fraud prevention measures: The integrated approach of the Department of Employment Affairs and Social Protection in Ireland

In line with the strategic objectives of its Compliance and Anti-fraud Strategy, the Department of Employment Affairs and Social Protection (DEASP) publishes an annual report on its fraud prevention and detection activities. These reports provide an account of how the Department has performed over the previous year in pursuing its goals of reducing incidents of fraud and non-compliance in Ireland’s social welfare system. The reports present key outcomes of the Department’s activities, for example, how many claims were reviewed, the estimated total savings resulting from their implemented measures, and detection rates of overpayments, among others. In addition, the Department provides details on trends in fraudulent practices within SBPs and the most common tactics employed by those committing fraud. This is followed by an overview of how the Department has adopted new methods to prevent and detect fraudulent activity, for example by strengthening its investigation unit and applying different data analytics techniques.

Following the presentation of the Department’s activities, the report lays out the priority areas for the subsequent year and sets specific targets in its Annual Target Statement. This approach provides comparative data for the Department to assess its progress as well as areas for improvement. These reports have been produced annually since 2014 to support the implementation of the Compliance and Anti-fraud Strategy. As a result, the Department of Employment Affairs and Social Protection reports that it has increased detection rates of fraud in SBPs and strengthened its prevention measures.

Source: (Irish Government, 2017[33]).