10x improvement in sanctions screening: VIALET case

AMLYZE
Author
AMLYZE
Published
February 29, 2024
Vialet case study

While the authorities debate the appropriateness of international sanctions as a means of resolving military conflicts, and whether economic and financial pressure can change the behavior of wrongdoers, we ask a different question: can international sanctions regimes achieve their ultimate objectives without proper implementation? 

Although the answer may seem simple, the tasks of individual economic operators (business entities) involve a complex set of variables and decisions.

The focus on the effective implementation of international sanctions has intensified with the onset of Russian aggression in Ukraine. Supervisors have placed particular emphasis on the adequacy of international sanctions implementation solutions adopted by financial institutions. While the criteria used by supervisors to assess the effectiveness and efficiency of international sanctions implementation solutions may not be fully transparent, the sanctions imposed by supervisors on financial institutions provide a clear indication that the effectiveness of international sanctions implementation mechanisms is not satisfactory.

Within the complex framework of international sanctions implementation, we emphasize one critical aspect that is universal to all businesses (including, but not limited to, financial institutions): automated sanctions screening solutions. As international sanctions regulations continue to expand and become more complex, the automation of sanctions screening becomes increasingly important for companies seeking to remain compliant. Choosing a specific vendor to automate screening can be costly if the chosen solution is ineffective or inefficient. This could result in either non-compliance or excessive workloads due to the handling of numerous ‘false positives’ (system errors where the legitimate party has been identified as a sanctioned party).

You can also download ‘How to improve sanctions screening efficiency by 10 times while preserving sanctions screening accuracy?’ case study in PDF format here.

Measuring effetiness and efficiency

Automated screening solutions are essentially a set of algorithms used to compare one string of text with another to identify similarities that indicate a potential match.

Such matching doesn’t just apply to sanctioned individuals, but can extend to politically exposed persons, individuals on law enforcement watch lists, regulatory watch lists (including bans on specific services such as financial or gambling services) and even internal watch lists (e.g. individuals who may be identified as posing a higher risk of money laundering or terrorist financing).

Algorithms used in automated screening compare an entity’s operational data, such as customer and transaction information, with lists of names and other indicators of sanctioned parties. Various terms are used to measure the effectiveness of sanctions screening solutions. For the purposes of this case study, we will focus on two definitions:

Effectiveness – the term describes how accurately the sanctions screening solution identifies similarities between the lists provided and the data screened. To illustrate, if the sanctioned individual is ‘John Doe Smith’, will the system recognise the similarity and match him with ‘John Smith’, the latter being the company’s customer?

Efficiency – the term describes how well the sanctions screening solution discards obvious mismatches and moves on to potentially matching records. For example, if the sanctioned person is ‘John Doe Smith’, will the system discard a mismatch of  ‘Jane Deed Smile’?

Effectiveness and efficiency
Effectiveness and efficiency

While these definitions may seem simple, both effectiveness and efficiency are critical to the businesses. Effectiveness ensures compliance with regulations, while efficiency ensures that the system works optimally.

There are a number of variables to consider when deciding whether your sanctions screening solution is working effectively and efficiently. Those are to be described with the example below.

Vialet case: enhancing efficiency while preserving effectiveness

Testing the screening solution isn’t optional; it’s a necessary step to ensure compliance with legal regulations and is, in fact, a regulatory requirement in itself to guarantee the desired results of the automated screening process.

This has been well understood by UAB Via Payments (operating under the brand name Vialet) – an institution for electronic payments that operates in Lithuania.

To ensure the desired outcome of our screening solution, AMLYZE conducted a quantitative assessment of the effectiveness and efficiency indicators of its automated international sanctions screening solution using a synthetic data approach.

Let’s go through the process of how we tested Vialet screening solutions and what the key findings were.

Step 1: How?

Let’s start from the fact, that our synthetic data testing methodology adheres to global standards, summarized by the following criteria:

  1. During the testing, we generate data consisting of a list of sanctioned and non-sanctioned entities.
  2. The list of sanctioned entities is further modified to simulate potential data errors, allowing an assessment of the system’s effectiveness in handling such discrepancies.
  3. The developed data set was evaluated by the company’s sanctions screening solution.
  4. System’s responses, indicating matches with internationally sanctioned entities from the submitted list, were evaluated to assess both the effectiveness (accurate identification of sanctioned entities) and efficiency (minimisation of false positives) of the system.

Step 2: When?

The best approach is to conduct sanctions screening tests before selecting an automated solutions provider and on a regular basis thereafter.

This is exactly what our client Vialet did before working with us. The test results speak numbers and confirm the credibility of the claims made during the sales process.

Step 3: The revision of testing results

The company Vialet requested information on the potential availability of a sanctions screening solution provided by AMLYZE. In order to assess the effectiveness and efficiency of the screening solution proposed by the potential vendor, Vialet  requested AMLYZE to perform the testing of the sanctions screening solution based on the agreed methodology and to provide the results. The results received were compared with the market benchmarks and we provide them below. Based on the test results obtained, Vialet is currently optimally positioned, supported by the most favorable results.

Vialet client screening results

Vialet case screening results
Vialet case screening results

Explanations of definitions:

Effectiveness rate is the measure of effectiveness and shows the percentage of actual positives that are correctly identified or ‘hit’ by a screening test out of all (e.g. sanctioned persons) present in the population. It represents the accuracy of the screening process in detecting the true positives.

For instance, if a screening test identifies 80 out of 100 individuals/entities who are actually sanctioned, the hit rate or true positive rate would be 80%. This metric is crucial in evaluating how well a screening solution performs in correctly identifying individuals within the sanction lists.

Effectiveness rate of manipulated population is the measure of effectiveness within the manipulated data. This measure shows the percentage of individuals (entities) identified as potentially matching a sanctions list within an expanded population beyond the primary dataset of sanctioned persons. In this case, we manipulate the primary data-set to be sure that e.g. John Doe Smith is matched with John Smith. This metric is essential for understanding the effectiveness of the screening system, particularly when discrepancies emerge in the screened data due to intentional or operational errors in the initial datasets (like mistyping an individual’s name). It helps assess how well the screening system manages and identifies potential matches despite such discrepancies, highlighting its overall efficacy in handling data variations and ensuring comprehensive coverage in identifying sanctioned entities.

Average hit count is an indicator of the efficiency of the screening system, which refers to the average number of matches or hits identified by a screening process within a given datasets. This metric provides insight into the frequency or occurrence of possible positive identifications (such as flagged individuals or entities) during the screening process.

For instance, in a scenario where the company’s client John Smith is mentioned in one of the lists of each – UN, US and EU – sanctions lists, the hit rate for John Smith would be 3. If another client, Jane Smile, was sanctioned in one of the EU sanction list, the hit rate for both clients would be (1+3)/2 = 2. Depending on the accuracy of the screening solution the hit rate should not be too high.

False positive rate is a measure of efficiency and indicates the percentage of instances where the screening system flags a transaction, entity, or individual as a potential match to a sanctioned party when, in reality, there is no legitimate match.

False positive average hit count is an efficiency metric that signifies the average number of alerts generated by a false positive match. For instance, if the system erroneously identifies a non-sanctioned individual like John Smith as a match, this metric quantifies the frequency of alerts triggered for John Smith.

At first glance, everything seems satisfactory. However, we have decided to go beyond the raw data and offer full transparency by providing detailed explanations of how our screening system works and why these indicators are important. Why? A common problem we see in the market is that, despite having a screening solution tested, many companies lack the expertise to effectively interpret the results and make the necessary adjustments. This hinders the achievement of an optimal balance between efficiency and effectiveness in the sanctions screening system. That’s why we provide you with some comparative indicators to help you answer the question of the efficiency and effectiveness of the system.

Key aspects for measuring performance

There are 4 main elements measuring screening effectiveness and it is important to understand all of them in order to understand if the screening engine is working effectively.

The accuracy of the watch lists used for screening purposes

These lists are typically obtained from regulatory authorities and are often supplied, updated and maintained by external vendors who specialize in the formatting and delivery of these lists. Depending on the source of the watch list and the regulatory requirements that drive the need for screening, the following types of watch lists can be distinguished:

Sources for effective screening
Sources for effective screening

It is also possible for the watch lists to be developed internally by the business entity, which may include specific individuals or entities, as well as identifying ‘keywords’ that indicate a higher risk of international sanctions evasion or money laundering/terrorist financing risks.

The quality of these lists is critical to the effectiveness of the screening solution. Therefore, the entity must ensure that:

  • watchlists are accurate, complete and comprehensive to meet all compliance requirements;
  • watchlists are updated promptly when data in primary sources changes;
  • access to internal watchlists must be authorized and strictly monitored to prevent unauthorized changes to the data.

The completeness of the reference data to be screened

Not all data elements within the internal records are relevant for screening. Different levels of data granularity can be set for screening object types:

Customer screening and transaction screening
Customer screening and transaction screening

The quality of the reference data is also critical for sanction screening solutions to deliver satisfactory results.

The regularity of screening 

Customer screening should be carried out before a new business relationship is established and then triggered on events:

  • when customer information changes;
  • when watch list information changes.

Due to the recent frequency of changes to international sanctions lists, we are seeing daily customer screening become a market standard to minimize the risk of international sanctions breaches.

Transaction screening should be carried out at a point where a transaction can be stopped and/or funds frozen.

The effectiveness of screening algorithm

This is where the “magic” happens and not every automated screening solution provider is keen to disclose the specific techniques they use to achieve the desired results. In contrast, we are open to sharing the inner workings of our effective screening solution, so we will briefly disclose the “behind the scenes” of an effective screening solution.

For a watchlist screening solution to be effective, it must use sophisticated matching technology to compare names, geographic references and other distinguishing features against watchlists. There are several factors to consider:

Factors to consider in screening algorithm
Factors to consider in screening algorithm
  1. When names are normalized. Before name matching, all names should be normalized in order to deal with possible data quality errors. Normalization includes:
    1. i) romanization: the process of converting a writing system that uses non-Latin characters (e.g. Cyrillic, Greek, Arabic) to the Latin script;
    2. ii) removal of separators, numeric symbols, align letters’ case(normalized name of ‘Mr. JohN Doe (Smith)?’ will become ‘John Doe Smith’).
  2. Whether and how pre-matching filtering is applied. To provide a rapid decision the algorithm must be able to filter out obvious mismatches. Different methods can be used to compare normalized names and source names to decide if a record is similar enough to proceed to the extensive matching stage.
  3. When fuzzy matching algorithms are used. These are techniques for comparing a word (data) or string of words (data) that are similar but not necessarily identical. Rather than requiring an exact match, it allows for a degree of similarity that takes into account variations such as typographical errors, misspellings, abbreviations or minor differences in formatting. If ‘John Doe Smith’ is matched to ‘John Smith’, it is a matter of the fuzzy matching algorithm and its configuration.
  4. How to set the configuration of fuzzy matching algorithms. The degree of fuzziness in fuzzy matching algorithms is a tunable parameter that allows the balance to be struck between the rigor of the match and the ability to find approximate matches in the data. By adjusting the various components of the algorithm, it can be fine-tuned to best suit a particular business model, product or risk appetite.
  5. How to set suppression rules for false matches. The process of deliberately excluding certain results from further investigation during the screening process reduces false positives and improves the efficiency of the screening process. This suppression can be managed using penalization of similarity score based on some factors such as missmatching initial letters of name/title, for an entity that has a significant amount of generic business type words (Limited, Group, Corporation, etc.) do not allow to generate matches on such words only, for a short or abbreviated word that was matched reduce a similarity score slightly, etc.

The screening solution may perform a multi-step name search and use configurable comparison algorithms to determine the degree of similarity of the match. If the final degree of similarity meets or exceeds the match threshold set in the screening parameters, a ‘hit’ (i.e. a possible match to a watchlist entry) is produced.

How effectively the ‘hit’ will capture an entity on the international sanctions (or other) lists depends on how well all the above variables have been configured within the screening solution.

Performance measures of the screening algorithm

Using the synthetic data approach to test an automated screening solution there are several aspects indicating high performance. In order to ensure systems integrity for different use case scenarios it is important to check effectiveness and efficiency for different entity types (if an entity is an individual or an organization) and data types (if it is a control (original) or manipulated (augmented) data record). Only a robust and versatile system achieves similar results between all mentioned categories outperforming market benchmark levels.

A False Positive rate explains how much of the customer portfolio is expected to generate at least one false match against sanctions. Reducing false positive rate directly contributes to better efficiency of the process. This metric together with average count of matches per falsely alerted entity should be analysed and reported similarly as mentioned above. Calculating these measures for different categories of data allows us to identify if a system faces any major vulnerability.

Key findings of the Case study:

● As the international sanctions landscape rapidly evolves, organizations are increasingly prioritizing compliance requirements for international sanctions implementation. Automated sanctions screening solutions are no longer a convenience, but a critical necessity to ensure compliance with international sanctions implementation.

● Both effectiveness and efficiency of automated sanctions screening solutions are critical for organizations. Effectiveness ensures compliance, while efficiency ensures that the system works optimally and does not overwhelm staff with false positives.

● In the volatile environment of international sanctions, testing the screening solution is not optional; it’s a necessary step to ensure regulatory compliance and, in fact, a regulatory requirement in itself to ensure the desired results of the automated screening process.

● The case study of our client, Via Payments, UAB (hereinafter also referred to as brand name Vialet), is a perfect example of how a sanctions screening test can help in the selection of a vendor and in the overall optimisation of a sanctions screening solution.

Client view

Lina Slavikaitė

By implementing AMLYZE’s screening solution, we have been able to improve our sanctions screening efficiency by 10 times. We are also impressed with AMLYZE’s powerful transaction monitoring and case management solutions

Lina Slavikaitė, Chief Compliance Officer/MLRO & Board member at VIALET

Want to discuss it more? Check out our Screening module options or book a meeting with us!

About the author

AMLYZE
Author
AMLYZE
AMLYZE is a fully automated service created for the financial sector and businesses that are obliged to comply with AML/CFT regulations.

Related

What data to prepare for your inspection?
Case studies

What data to prepare for your inspection?

Detailed guide on what data to prepare for your upcoming inspection and how to face the challenges of an on-site inspection with confidence.
by AMLYZE
2 min read
How to prepare for the regulatory inspections?
Case studies

How to prepare for the regulatory inspections?

A detailed guide for FinTechs, neobanks and VASPs on how to prepare for a supervisory visit, based on the experience of market regulators.
by AMLYZE
8 min read