Why are we interested in different codecs and their effect on prediction scores anyway?

Where we work, at Fourthline, our task is to help to safely onboard clients for our customers, such as banks, fintechs, & non-financial businesses, in a KYC (Know Your Customer) process. We research and develop Machine Learning and AI models that are used to verify a clients’ identity and whether it is a live person that wants to onboard to their platform and use their services.

While experimenting in our lab with Machine Learning systems, we observed something interesting: Models trained with different loss functions respond differently when we change how the input video is encoded. This opened up the question, whether, and if so, how far, some training paradigms are more sensitive to different video codecs, and thus, to compression artifacts, than others.

Now you wonder, why we would care to check what influence an encoding or transcoding has on the output of our services: We could just optimize for the codec at hand and be happy! Well, leaving academia behind, we interact in an open world, and so do our services. Fourthline serves a multitude of banks, fintechs, & non-financial businesses. Each of them individually preprocesses their data with their own data capture and backend engineering before running them in our algorithms to check the liveness and identity of a person. We do care that our services always behave as expected, no matter the input codec!

In this blog post, we want to explore specifically how the training objectives coming from two paradigms — supervised and contrastive learning (see [8] for an overview) — might have an affect on the created embedding space, e.g. structure, robustness against noise and artifacts. We want to take a closer look at some characteristics of these losses and see how these affect the classification behavior in real world scenarios. In particular, the goal is to fathom possible influences of the different losses on the classification score in presence of image space artifacts, such as those resulting from different codecs are being used for the input data.

Max Crous and Roman Aleksandrov of Fourthline’s Biometrics team conducted the experiments and created the graphs on which this blog post is based on. They furthermore shared valuable insights and ideas through discussions.

What we found at Fourthline

At Fourthline, our product architecture employs multiple integrated modules that work in concert to maximize their collective advantages. This modular approach allows our systems to address challenges through multiple parallel processing paths.

As one example, different parts of the liveness system are researched and trained following different learning paradigms and architectures. Doing so, among others, we make use of supervised and contrastive learning methods.

Since we are interested in the effect of different video codecs on the outcome of our algorithms, we started performing some experimental tests on data in our domain, to which we applied different encoding and transcoding preprocessing. Below we show some qualitative exemplary results for two particular systems. Both these systems are trained on the same data domain, and while one is being trained with a supervised binary cross entropy loss (BCE), the other one is trained with a contrastive loss (Contrast.), where a local density estimator is employed for estimating decision thresholds.

The plots below show the distribution of the difference in score outputs (e.g. Softmax) of the differently trained systems, when we change the input video codec. When the score on the original codec for a sample to be classified had any value x0 (defined by the horizontal line ‘0.00’), the colored dots show where the values of the confidence output for each of the test samples land when the model was fed videos with a different encoding, resulting in scores x0+dx. The colored areas represent the spread of dx.

In these experiments, starting from the original MJPEG format (each frame in a video is encoded in JPEG format), we evaluate chroma subsampled MJPEG (4:4:4 and 4:2:0), intra-frame FFV1 (part of the FFmpeg project). We also show below the comparison between the FFV1 and the lossless VP9 encoding.

So, what do we see? We immediately see that the model trained with a contrastive loss performs significantly different on the encoded data than the BCE trained model does. For VP9, assuming approximately a Gaussian statistic, the 1-sigma area contains score differences from +/- 0.05 — this means a change in confidence from, e.g., 0.8 to 0.75–0.85. This is a huge difference for a model output!

You are asking: why does the contrastive model behave so differently? And why does the BCE model not? The Contrastive model seems to be much more vulnerable to image encodings/transcodings. We asked ourselves: What could be the reason and how can we prevent it?

The paragraph below wants to shine some light on possible explanations and provides some glances for reasons that might explain this behavior. Note, since this is just a blog post, scientifically sound proofs will only be published somewhere else:)

Discussion

First, we see that divergence of scores is consistently high for the contrastive learning approach among different codecs. Given that all experiments were conducted on the same data domain, we are focusing on taking a closer look how the different learning schemes could explain this phenomenon.

In general, to prevent some undesired behavior in any technical system, it is most useful to understand what causes the observed behavior.

The Two Losses: Characteristics

A first possible explanation that comes to mind is that the learned feature spaces reveal different structures, in particular, different inter-class structures, for the two losses are different. Let us look at some characteristics about the losses (we assume a binary problem for our argumentation in this post):

Cross-Entropy binary loss (BCE):

• BCE finds a hyperplane to separate the two classes, where it operates on class likelihoods (e.g. Softmax outputs).
• There are no constraints on the model’s learned distribution (e.g. of class 1 and class 2 — they can assume any shape).
• Learns separable features — any features, regardless of whether similar data points of the same class are close to each other (as long as they are ‘far away’ from all data points of the other class).

Summarizing, BCE does not care much about the intra-class distribution nor about the shape of the distribution itself at all — it just wants to create features that separate a sample of one class from all samples of the other class.

Contrastive loss

• Rather than finding a hyperplane, it ‘organizes’ the feature space:
  The contrastive loss pulls similar images together into some form of a cluster while pushing dissimilar images away from each others. So it operates on data points and their relative positions to each other.
• Learns a similarity metric (often Cosine distance) [that is normalized and temperature scaled, i.e. NT-Contrastive loss].
• The contrastive training objective usually tries to match the hidden representation to be uniform on a hypersphere [1].
• Is exposed to the effect of dimensional collapse [2]. This might be caused/enhanced by feature suppression [3] for the contrastive approach, see also below.

So, the contrastive learning paradigm wants to cluster similar samples, pushing clusters of dissimilar samples away — the former an apparent difference to the BCE.

Our general argument along the following is based on the fact that video encoding/transcoding — lossless or lossy — is causing artifacts to appear in the data, and, in general, information that a model can only interpret as noise, being induced. We want to add context to this relationship and highlight directions to draw conclusions.

To get a visual idea, let’s start by looking at typical 2D-representations of MNIST embedding for both, BCE and contrastive loss.

A look at latent spaces

The two figures below illustrate respective embedding spaces on the MNIST dataset for BCE and contrastive loss training. We can already see that they represent what we summarized above in the bullet points:

Trying, without constraint on the shape, for a sample of one class to be as far away as possible to all the other classes’ samples — versus — building compact clusters (note, that, for illustrative purposes, we fall back to a multiclass problems in the figure, the conclusions made here still hold without loss of generality for the binary problem.).

Though just an illustrative example, the figure for the contrastive approach lets us already draw some corollary: without extra noise being added from any video encoding, there is already more overlap visible between the clusters compared to the BCE, being more vulnerable to any kind of noise and therefore changing confidences in classification.

Concluding, if you are working on anomaly or outlier detection, one hint for your research project or product is that you might prefer a Contrastive Loss over a BCE loss, to make use of the minimization of in the intra class variance!

Going a bit deeper that this demonstrating example, we want to look into a few related publications and what possible answers they might yield.

Related works for possible explanations

A. Non-Exhaustive Feature Space

In [2] they show that using a training scheme similar to SimCLR [4], a singular value decomposition of the covariance matrix of embedding vectors (they use a 128 dimensional embedding on ImageNet images), more than 20% of the dimensions remain basically unused, showing the dimensional collapse. In a corollary, they deduce:

“With strong augmentation, the embedding space covariance matrix becomes low-rank.”

So basically, the model is not using the entire feature space, thus limiting capacity and possible increasing sensitivity to noise.

In our works, the contrastive approach inherently is built upon strong augmentation. The dimensional reduction possibly caused by this affects the model’s actual capacity and thus, its capability to compensate robustly for noise and artifacts in the input unseen during training.

B. Learned Feature Distribution

The authors in [5] investigate the effect of different losses, including margin losses and Gaussian Mixtures on the feature embedding space. They find that indeed a Cross Entropy loss does yield a feature space distribution that resembles a Gaussian Mixture Distribution, e.g. an organization of the feature space in clusters.

They furthermore find that training with a Gaussian Mixture loss, that enables learning clusters in a supervised manner is increasing robustness of the network. Specifically, they show that deep neural networks with high classification accuracies are vulnerable to adversarial examples when trained with a Softmax/Cross entropy loss. Adversarial examples created by the Fast Gradient Sign Method can be seen as adding noise to the image in a controlled way by exploiting the gradient of the loss with respect to the image. Without classifying compression artifacts as either deterministic or probabilistic, one can argue that both changes to the image have an effect of changing the confidence of the classification — the embedding is moving away from the original point in feature space.

The adversarial examples formed by intentionally adding small but worst-case perturbations cause the model to make incorrect classifications with high confidence, proofing that small changes can have drastic effect on confidence values in vanilla Softmax trained networks, They show this vulnerability by comparing the distributions of the confidence outputs of networks trained with a Softmax/Cross Entropy loss, a Center loss and a their proposes Large Margin-Gaussian Mixture (L-GM )loss as shown below. While the specifics of the losses are not of interest here, we just want to acknowledge, that the L-GM loss is a loss encouraging to build cluster in feature space and thus, trying to reduce intra class variance (pull similar samples together) and increase inter-class variance (push dissimilar samples away from each other).

On the first glance, the results above are somewhat contradictory to the argumentation made so far. They find that the — push a sample from class 1 as far away as possible from any sample of class 2 — BCE loss (at least for the adversary examples) is less robust that their proposed L-GM loss. First, we should note here, that their L-GM loss is trained in a supervised way. That means that their supervised, cluster building, maximizing inter-class distance training approach might get much stronger signals, thereby indeed aiding robustness.
Secondly, the margin of the L-GM is a hyperparameter, where a margin of 0 corresponds to a GM loss. This means, that the comparison is a bit unfair, as one needs another set of data points to find the best margin. Concluding, with the right objective, a training scheme that pulls similar images together into some form of a cluster while pushing dissimilar images away from each others can yield to a robust classifier.

This being said, it would be desirable to see how the L-GM compares in their exact same experimental setup with a contrastive loss.

C. Quantifying the decision boundary

The works described in [6] is a theoretical one, where they derive a lower bound on the probability with which the inter-class distance is more than the intra-class distance quantify the separability of classes with the cross-entropy loss. So basically, they provide an analytical tool with which we can quantify (a lower bound) for which a sample of a class 1 is more likely to be classified as another class 2 for BCE trained models.
This is very powerful, as it gives as an actual value for a certain distance in feature space on how likely a sample belongs to either class.

The work provides a very interesting tool to investigate the problem at hand — at least for half of the problem — we do not have such a quantifying measure for our contrastive half.
However, if we have a BCE model as well as model that is trained to learn a known distribution in feature space, e.g. by minimizing the KL of Jenson-Shannon divergence with respect to a GMM or by utilizing normalizing flows, one can already make such a quantitative evaluation, which would comprise a powerful experiment.

Anyway, we should keep an eye open for publications that come up with such a quantification for a contrastive loss. This would enable us to tackle the described problem in an analytic way!

D. Feature Suppression

In [3] the authors discuss the phenomenon of feature suppression in contrastive learning. Let us first discuss what is understood as feature suppression. They say that

“The suppression effect occurs among competing features shared across augmented views.”

We want to start by expanding on an example from the original SimCLR paper, which is also pointed at in [3]. They describe the phenomenon via a two-stage augmentation example — augmentation being the pillar that (self-supervised) contrastive learning is predicated on: “One functionality of data augmentation is to remove “easy-to-learn” but less transferable features for the contrastive loss.”
The example I want to give here is inspired by [3] and [4]. Let us consider the interaction between the augmentations color distortion (making an image all ‘green’) and image cropping (global vs. local and adjacent views, see figure below). Now lets look at an image of a dog standing on (green) grass.

Let us assume we have done a few crops on the original image, that we consider for our augmentation scheme (of course we do not select any particular augmentation application by hand) and we now look at the color distribution of our four augmented, e.g. cropped, images:

We see that there are no distinctive changes among the color distribution dimension, or feature, of our data: The random crops show almost the same distribution as the whole image! (This, of course, does not hold for any image.)

The idea of feature suppression is explaining this by arguing that one feature, one that is easy to learn (color distribution), suppresses another one (cropping).

In this case, it is super easy to just learn the color distribution, e.g. the green grass on which the dog is standing and completely ignoring (suppressing) the random cut augmentation.
This is why the authors in [3] argue that “Existing contrastive learning methods critically rely on data augmentation to favor certain sets of features than others, while one may wish that a network would learn all competing features as much as its capacity allows.” I recommend reading the paper [4] anyway for ML engineers, that work with contrastive learning or with augmentation.

To continue the examination, let us look at the color distribution of the same crops, when the image undergoes two augmentations, i.e. cropping and color distortion:

We see that combining the two augmentations (at least for these two considered images) is crucial so that the network can learn something meaningful!

What can we learn from that for our problem? Well, suppressed features are yet another expression of a suboptimal use of our feature embedding space and, in the worst scenario, an entire disregard of certain information of the data to learn from. To stay with the example above, when relying on the color distribution only when we the single augmentation is cropping, the learnt information about dogs, or any object, on grass is disastrously fragile on this very color being apparent (since other features are suppressed). This yields that image artifacts from encodings might distort this one feature easily, yielding to an altered classification results, or at least, a change in confidence.

To summarize about Feature suppression:

Feature suppression:

• Easy to learn features can suppress the learning of harder features.
• A simple color distribution feature such as ‘green’ being learned to have the distinct meaning (among others) grass can suppress the ability to learn the classes that are associated/correlated to that color feature, e.g. crocodiles or just cows standing on grass.
• As an addendum, in their experiments they add a fourth dimension of a constant integer to the image, which is “represented as n binary bits/channels”. They find that with “a few bits of the extra channel competing feature added.”, shared for both contrastive views, the representation loses all information of the original RGB values. These results conclude that small useless ‘extra information’ can change the fate of a prediction.
• They conclude that in it is general “difficult to learn both of the competing features using existing contrastive loss (as in SimCLR).”. From that we learn that we need to choose our augmentations very careful depending on our data at hand!

E. Loss Function vs. Domain Adaptation

In [7] the authors investigate yet another aspect of different loss functions and their effect on changing input data:

“many objectives lead to statistically significant improvements in ImageNet accuracy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks”.

While a different codec is not directly a downstream task, we can consider it as a transfer task, i.e. shifting the domain. With their main argument being that vanilla BCE learns more robust features than alternative (supervised) objectives, we discuss some of authors’ insights here. They conducted experiments with nine different objectives with ImageNet data, and several transfer settings.

Some of their key results:

• “the choice of loss has little effect when networks are fully fine-tuned on the new tasks” [7].
• “there exists a trade-off between learning invariant features for the original task and features relevant for transfer tasks” [7].

They further find, that “alternative objectives [over the Softmax BCE approach] appear to collapse within-class variability in representations, which accounts for both the improvement in accuracy on the original task and the reduction in the quality of the features on downstream tasks.”

Summarizing, they find that

“the properties of a loss that yields good performance on the pretraining task are different from the properties of a loss that learns good generic features.”

Limited to the supervised setting in their experiments, what we could learn here is that one must carefully perform test on their to-go-to-production models, or models deployed to an unknown domain. Of course, as also mentioned in the paper, trying bigger networks and more data is always worth it. But in the end, when you cannot test the unknown unknown, how can you find confidence;) in your trained (bigger) model?
One advice is, that trained models that perform well on your benchmark test need to be validated against data of possibly different distributions, or data subject to noise/artifacts that might represent such changes, like encodings. Cross validation, another benchmark set (‘where from?’ you ask), estimating robustness and calibration suggest some directions to explore.

Summary and Conclusions

So, what did we learn here? First of all, be careful when changing the codec of your input videos for your trained convolutional neural network, particularly when the network is trained in a contrastive manner! BCE and Contrastive Learning objectives behave very differently, when it comes to the hidden representational structures, and ultimately, the output confidences learned.
These different structures themselves come with their peculiar behavior when it comes to changing input with respect to, e.g., but not limited to, encodings.

Some take-aways:

a) When you apply augmentations to your training, particular in a contrastive setting, you should take great care of their composition and incorporate the knowledge you have about your data domain.

b) When you are working within the real-world data domain, the codec of your input data might be out of your control — so the response of your system to different encodings will matter! Our results show that you should test your system with different video encodings and come back to this post to amend your findings.

Furthermore, we discussed several recent findings in the ML literature that explore either loss objective and their implications for classification confidences and shed some light on directions and fallbacks to follow and to avoid, respectively, when we train ML models.
This blog post can only be incomplete, let alone a scientific profound elaboration. However, it shall give many an insight and understanding of the underlying problem asked in the title “Supervised vs. Contrastive Classification: How Does the Video Codec Influence the Prediction Scores?”, and shall serve as a mere start of a discussion.

For the future, particularly take away a) has drawn our interesting and shall be a research topic within Fourthline with upcoming results to be shared with the community.

Contributions

We thank everybody who contributed to enable us writing such a post, at Fourthline and beyond!

Literature

[1] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere, 2022

[2] Li Jing, Pascal Vincent, Yann LeCun, Yuandong Tian. Understanding Dimensional Collapse in Contrastive Self-Supervised Learning, 2022

[3] Ting Chen, Lala Li. Intriguing Properties of Contrastive Losses, 2020

[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations, 2020.

[5] Weitao Wan, Yuanyi Zhong, Tianpeng Li, Jiansheng Chen. Rethinking Feature Distribution for Loss Functions in Image Classification, 2018

[6] Rudrajit Das, Subhasis Chaudhuri. On the Separability of Classes with the Cross-Entropy Loss Function, 2019.

[7] Simon Kornblith, Ting Chen, Honglak Lee, Mohammad Norouzi. Why Do Better Loss Functions Lead to Less Transferable Features?. 2021.

[8] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, Fillia Makedon. A Survey on Contrastive Self-supervised Learning, 2021.

Supervised vs. Contrastive was originally published in Fourthline Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Times are changin’, Betty. These nerds are a threat to our way of life.”
– Stan Gable

In part 2 of this series on KYC-fraud detection we dive into the deep end and explore exactly how Fourthline’s data scientists develop algorithms to catch fraud. Whereas part 1 focused on unsupervised-learning techniques and their performance and applicability, in this part we demonstrate how we use supervised learning to strengthen Fourthline’s fraud-prevention capabilities.

Why supervised learning?

In part 1 we use unsupervised learning to explore relationships between different data points (e.g. document type, or time of submission) and assess their relative strengths. As a result, we gained insights into the data and explored the potential of unsupervised learning, by trying half a dozen algorithms and pushing most of them to their limits.

However, unsupervised algorithms can, at best, achieve 40% for both recall and precision on both our synthetic fraud-detection data sets and data from the real world (in production). Such models lag considerably compared to human performance and raise questions over how to serve them in production to add value to our products. Turns out, we can do much better!

Gradient-boosted decision trees

There is no shortage of supervised-learning algorithms available. Even going through the relevant library of scikit-learn should keep one busy for days on end. However, in this post, contrary to the previous one, we will not be comparing textbook approaches. Instead, we will showcase how we achieve state-of-the-art results.

Let’s cut to the chase then and focus on the framework we have been using for years, gradient-boosted decision trees (GBDTs). Why boosting? To start with, numerous publications favor it over all other approaches for tabular data:

• Why do tree-based models still outperform deep learning on tabular data?
• An optimized LightGBM model for fraud detection
• Tabular data: deep learning is not all you need

GBDTs are also the framework of choice in Kaggle competitions, for example in the IEEE-CIS Fraud Detection competition, which is very similar to KYC fraud detection. Moreover, GBDTs proved to be on a different level in terms of predictive power on our data sets, easily outperforming all other methods we have tried.

How do GBDTs work? At every stage, a tree is added to an ensemble of trees. Each new tree (without loss of generality, let’s call it the n-th tree of the ensemble) is trained to predict the residuals between the collective prediction of the previous (n-1) trees and the labels of the training data. The ensemble is then boosted by adding the weighted predictions of tree n to those of the preceding trees. By limiting the total number of trees and introducing other tree-growing hyperparameters (e.g. depth, number of leaves), boosting frameworks let us regulate resources and overfitting.

There are 3 main GBDT algorithms on the market: LightGBM, XGBoost and CatBoost. Each represents a slightly different GBDT approach and they can barely be separated in terms of performance. However, LightGBM performed best on our data sets and training it is considerably faster. So, for us, the choice was a no-brainer.

Having decided which algorithm to use, you might assume we could just throw data at it and hope for the best. After all, GBDTs can handle different data types (numerical and categorical), are impervious to missing data (null values) and aren’t particularly sensitive to hyperparameter choices. So, what could go wrong?

While all this is true, as data scientists know very well, there are other tasks involved that spell the difference between a decent model and a remarkable one. Such tasks are data selection and splitting, feature engineering, model training, validation strategy, to name but a few. In the following sections we will thoroughly analyze how we approach each step in the process of developing fraud-detection models that set the bar in the industry.

Data preprocessing

Preparing data for training is one of the most arduous tasks in machine learning. Before even getting to the feature-engineering stage, we preprocess raw data with three basic goals in mind:

• Eliminate data that either adds nothing to the model or, at worst, confuses it.
• Homogenize and combine data originating from diverse sources.
• Create splits while preventing data leakage.

Domain nomenclature

Let’s first define some domain jargon we will need in this section.

Case: All the information collated for an Identity-Verification instance, including the outcome. This is triggered by a natural person submitting an application for a service, e.g. opening a bank account. Each case in our data set is a ‘sample’, with its own set of feature values and a label. Our goal is to predict if the case is fraudulent or not.

Client: The natural person whose identity needs to be verified. Multiple cases can be associated with the same client.

Data selection

In Fourthline’s AI team our job is to use machine learning to turn data into products. We must, therefore, understand:

• The origin of the data we use, e.g. mobile-phone cameras, lighting conditions, mistyped personal data.
• Their lifecycle, from receiving the data to storing it in a database and all transformations in between.
• The meaning of the data.

In addition, we have to understand exactly how we can strengthen Fourthline’s products and ensure our machine-learning models are fit for purpose.

One example is the type of cases the models are trained on. For Identity Verification, used for example when opening a bank account, the client needs to submit a particular set of data. But for Client Authentication, used to re-authenticate the account owner’s identity at some later point, less data is typically required. Therefore, there are systematic differences in the available data in these two types of cases. This heterogeneity is best addressed by developing different models for each use case.

Data splitting

The basic requirement to fulfil here is to prevent data leakage from the train set to the validation set and the test set. There are two criteria that need to be simultaneously satisfied to achieve that:

Firstly, splits are based on client ID. Therefore, cases associated with the same client end up in the same split. Since there is potential for substantial feature overlap between the cases of the same client, we would like to avoid spreading them across splits because this effectively skews either the validation or testing metrics. In any case, it ends up compromising the quality of the trained model.

Secondly, splits are based on time. All cases in the test split have dates in the future relative to cases in the train and validation sets. In this way we emulate what happens in production where a model trained and validated/tested on historical data is used to predict the outcome of new data. The model is therefore susceptible to various mechanisms that can reduce its performance, like domain changes, or unseen categories, during testing. As Nobel-prize winner Niels Bohr once said: “prediction is very difficult, especially if it’s about the future”.

Feature engineering

“Coming up with features is difficult, time-consuming, and requires expert knowledge. ‘Applied machine learning’ is basically feature engineering”.

~ Andrew Ng

Fortunately, at Fourthline we relish difficult tasks, otherwise fraud detection would be a terrible choice for a machine-learning project! We have been refining our approach for years and, with the support of in-house fraud experts, continuously strive to convert raw data into powerful features.

Feature types

The features we create shed light on and track the history of the four pillars of a case: who, where, when and what.

Who

There are many features that describe who a case relates to. Some straightforward aspects, like age and gender are not very powerful predictors on their own, but the model can combine them usefully with others. Furthermore, we can track the historical record (if any) of that client in our database.

We can get an even stronger grip on the who by creating more complex features, especially by aggregating individual data points (e.g. nationality, place of birth) into complex composite descriptions of the client, that are nevertheless generic enough to have the same values, for very different people. A naïve example is to combine place of birth with information from the where and create a boolean feature that captures whether the client lives in the country they were born in.

Where

This seemingly simple category actually includes a diverse set of features ranging from obvious to advanced and computationally expensive to create. Obvious features include the home address provided by the client and the geolocation of the device that submitted the application. More complex and more powerful features can be created by considering the history of those and other features, for example, the application history of a given address.

When

To unlock the potential of temporal features , they must be approached with great care. Even the time of the day a case was submitted can be used as a feature. But beyond such simplistic, and often of limited use, features, time is implicit in many of the features we create. For example, whenever the history of an aspect of a case plays a role (think here a person’s or an address’s application record), time needs to be accounted for.

What

This category contains two distinct groups of features. The first group defines what the client used when submitting their case, e.g. ID-document type, device brand/model. Like other simple features, members of this group are especially useful in combination with others. The second group, probably the most important set of features overall, defines what our internal automated processing (spearheaded by our AI-services suite) determined about the case and its artifacts. By far the strongest predictors of fraud are based on Fourthline’s AI models, which are purpose-built for KYC.

Creating features

The backbone of feature engineering is an in-house-developed class, based on scikit-learn’s ColumnTransformer. This is an elegant and efficient way of feeding a bunch of raw data points to a class and getting a pandas DataFrame as output, but it can also be tricky to make major revisions. This choice is effectively a trade-off between flexibility and the convenience of a centralized transformation-factory pipeline.

The advantages of our feature-engineering pipeline lie not so much in the objects we use to do the work, but in how we use them, how we choose transformer hyperparameters and the safety guards we implement to prevent data leakage.

Handling categorical data

A lot of the raw data and features used during training are categories, expressed in strings. However, strings are inefficient during processing and can overconsume precious resources, such as time and RAM. Therefore, a ubiquitous transformation in our pipelines is ordinal encoding of categorical features to integers. The integers are still fed to the training algorithm as categories. To achieve this, we piggyback on the functionality of scikit-learn’s OrdinalEncoder and ensure the models we train never have to manipulate strings.

Selecting transformer hyperparameters

A lot of the transformers we use come with hyperparameters, the values of which can be very important both for performance, and generalization. For example, with categorical features, the categories provided or the cut-off frequency can effectively be seen as a bias-variance trade-off. More categories favor in-domain performance than generalization. For all features types, our selection of hyperparameters is driven by training a model on many instances of the same feature, but where each instance has a different hyperparameter set. After the model is trained, feature importance can indicate which choice is the best.

Preventing data leakage

An important step in preventing data leakage comes at the feature-engineering stage. To ensure that information from the future test set does not trickle in the train set, we only engineer features for the train set. This way the transformer pipeline, which includes the state of every transformer employed, will only have seen the train set when it’s called to create features for evaluating the model on the test set.

Feature importance and optimal feature set

Feature importance is a somewhat overloaded term. A trained LightGBM model has the property feature importance baked in it. Depending on whether we choose “split” or “gain” when calling the function feature_importance() we get an ordered list of features in terms of either the number of times a feature has been used during training, or the gain a feature brought during training, respectively.

However, we are chiefly interested in assessing feature importance during inference. For this, we have developed our own implementation of permutation feature importance. We use it to evaluate the importance of each feature when the model is called to predict labels on a test set. This approach uses one or multiple metrics to estimate how much the performance of a trained model drops when the values of a single feature in the test set are randomly permuted along the axis of samples. Doing this multiple times for each feature (with a different random seed each time) yields statistically sound results.

An importance-driven ordered list of features is also the starting point for determining the optimal set of features. The idea here is to use recursive feature elimination whereby we use all possible features in a training, assess their importance and then removed them one by one starting with the least important. At each step we retrain the model and monitor performance metrics to identify which set of features extracts the best performance out of the trained model.

Model training

All machine-learning engineers know that training is not the be-all and end-all stage in creating production-worthy machine-learning systems. But it is still pretty important, so let’s dive into it.

Problem definition

We train machine-learning algorithms to distinguish fraudulent from non-fraudulent ones, based on their features. So, this means we train binary classifiers, right? Not necessarily. This is another area where domain understanding has helped us optimize our approach. Fraudsters try a variety of methods to ‘fool the system’, from fake personal data and ID documents, to deepfake selfie videos. Even within those categories there are sub-groups, separated by finer differences.

We trained and compared binary models (where all types of fraud are bundled under a single class) against both multiclass classifiers and binary classifiers trained on only one fraud type. We found that binary classifiers performed worst, while multiclass models perform virtually the same on specific fraud categories as binary classifiers trained on one fraud type. Therefore, to reduce computational resources and complexity we opted for multiclass classifiers.

Ensemble

In reality, we don’t train one model, but k models in a k-fold cross-validation fashion. This ensures that, during training, every single data point not in the test set is seen by k-1, that is, all but one model. As with train/test splitting, fold splitting is based on client IDs, which prevents identity leakage between the k-1 folds being trained at every iteration and the validating k-th fold. The k-fold cross-validation approach is pretty standard among machine-learning engineers and desirable where permitted by the size of the data set (also accounting for minority-class representation) and the computational resources available.

LightGBM-hyperparameter optimization

Like all machine-learning algorithms, LightGBM can be tuned to perform optimally on different data sets and objectives by changing the values of hyperparameters. These parameters cannot be learned during training and instead remain constant throughout a single training experiment. For example, we can reduce overfitting by limiting the number (num_iterations) or the variance (max_depth) of the model’s weak learners — in this case decision trees.

When training any model in the ensemble, the corresponding validation set has two roles. It is used for early stopping, whereby no more trees are added to the model once user-defined metrics stop improving on the validation set. And it is also used to select the best combination of model hyperparameters on each fold.

While there are several possible ways to fine-tune LightGBM hyperparameters, GBDTs are very robust to hyperparameters. Therefore, sophisticated optimization techniques typically yield little gain, provided that unoptimized hyperparameters are not miles off the sweet spot in the hyperparameter space. But once even remotely close to the sweet spot, gradients of the objective function are tiny.

Objective and validation metric

The metric that we use to decide which model is best is recall (for the fraud class) at business-relevant precision values. However, the loss function optimized during training is a preset in LightGBM, namely multiclass. When multiclass is chosen the model’s output uses softmax to express its prediction scores for each class and then the output can be used to compute the multiclass cross-entropy (log loss). This is what is being minimized on the train set. We note here that “multiclass_ova” whereby the model outputs binary predictions for each class separately, typically yields very similar but slightly inferior models.

We evaluate the validation set using the multi_logloss metric, which is basically the same as the objective. Therefore, the value of this metric on the validation set determines both early stopping and hyperparameter tuning. Other options yield similar, but usually not better, results. Metrics multi_error and auc_mu are also good choices and the resulting models are virtually indistinguishable from those trained with multi_logloss.

Inference and evaluation

So far, we have covered every stage of the machine-learning pipeline that results in well-trained, powerful fraud predictors. However, going from an ensemble of multiclass softmax predictions to a single, business-relevant fraud assessment is a twisty road, with some obvious, and some less obvious choices to be made along the way.

From ensemble to single multiclass predictions

First, the softmax outputs for every class from each model are averaged across all k models. The result is a single multiclass (still softmax-like) prediction, as if only one model was doing inference. There are various other possibilities here. However, averaging class predictions across models takes account of every model’s numerical prediction, and therefore best utilizes the ensemble.

From multiclass to binary predictions

Next, we sum up the predictions for all fraud subclasses into a single fraud class. Since the multiclass predictions add up to one, the resulting binary-like prediction also adds up to one, emulating the output of a binary classifier. Now we are ready to impose score thresholds that yield precision performance in line with product requirements.

It is worth taking a moment here to compare performance against the unsupervised-learning algorithms in part 1. Both precision and recall are more than two times better, going from a maximum of 40% to easily above 80%, for thresholds where these two metrics are equal.

Explainability

It is very valuable to have access to the reasons behind model predictions, not only to gain insight into the model’s inner workings, but also because of the stringent compliance requirements in the regulated industry where our models operate. Decision trees are particularly useful in that respect. GBDTs present more of a technical challenge, mainly due to their size. A handful of libraries exist that address this challenge, and this is one of the few instances where Fourthline relies on third-party solutions. There are numerous use cases for this functionality. On the one hand, we can use it internally to assist human specialists in decision making and highlight attention areas. On the other, we can share explanations as needed with our business partners and even regulators seeking to ensure our decisions are justified and based on facts, rather than being the outcome of biased or inadequate machine-learning models.

Behind the scenes and in the works

Despite this being a lengthy post, it omits some important aspects of our work that are either in the pre-production stage or relate to tasks other than fraud detection, and which deserve their own blog post. Behind the scenes, and already mature, are two major development efforts.

On the technical side, we have established robust workflow-management procedures that automate lots of previously semi-manual work and enable us to use features that significantly improve our models’ performance. We use Apache Airflow for automating a variety of tasks related to both the research and productization side of fraud-detection models. With help from Fourthline’s data and platform engineers we have created a computing cluster optimized for handling fraud-detection data sets and models, while eliminating manual work performed by data scientists.

The second development effort concerns a very different, but extremely important topic. We have developed a framework for assessing and ensuring fairness in our models. This is based on literature and original research on measuring fairness and guaranteeing no bias in the models we productize. The result has been a set of configurable fairness diagnostics that assess both the data sets on which we train our models, as well as the models themselves. What’s more, we have implemented small but crucial modifications to our model-training pipeline that let us to control bias when needed, while largely maintaining model performance. Eagerly awaiting the first comprehensive AI legislation act in the world at the time of writing this, we expect that the fintech industry will be one of the most affected and that fairness of machine-learning models will be a major factor for regulators when assessing models and permitting them to operate.

Summary and outlook

In this post we have zoomed in on all aspects of how Fourthline develops top-performing data-science models for KYC-fraud detection. We have explained our approach and choices at every step in the process and shared salient insights into the most crucial stages: data preparation, feature engineering, and model training. We hope that this not only sheds light on how we achieve great results, but also motivates other data scientists (in the KYC industry and beyond) to borrow interesting ideas.

With all the side projects and ideas we currently have in the works, stay tuned for the next blog post in the series.

Eleanoor Polder is the Fourthline data scientist behind most of the work presented in this post, with notable contributions from Vuk Glisovic and Sebastian Vater. Fourthline’s AI, Data and Platform teams have continuously supported the work of the Data-Science team. The time and expertise generously shared by the Anti-Financial-Crime team of Fourthline has been absolutely crucial in this project. Finally, this post was polished by Fourthline’s technical writer Francesca Cook.

Revenge of the nerds was originally published in Fourthline Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Everybody who ever worked either with VAE or GAN-like algorithms knows this: the struggle of finding proper hyperparameters. While this hyperparameter search, when reading papers, seems to be as doable a job as the implementation of the algorithm itself, hands-on people know from experience it is not.

This blog is aiming to provide insights into a more visually intuitive understanding of the role that the hyperparameters play with respect to the different loss terms of a Soft-IntroVAE model.

Note: The article makes use of mathematical writing. For the best reading experience, we recommend using Chrome and installing the TeX to Unicode extension if you don’t have it yet.

Though a lot of advancements in the field (Generative Adversarial Networks: EBGAN, BEGAN, Stabilized GAN training, WGAN, ProGAN, MSG-GAN; Variational Autoencoder-like: beta-VAE, Intro VAE, Soft-Intro VAE) are made towards more robust training, this hyperparameter search is tedious in the best case, a nightmare in a more realistic case. In this context, we understand a robust training to be a series of successful (and) convergent of model trainings without being sensitive to slight changes in the hyperparameter values.

Following the taxonomy above, and subdividing modern high-resolution image creation algorithms into VAE- and GAN-like algorithms, we want to focus on the VAE-like Soft-IntroVAE: Analyzing and Improving the Introspective Variational Autoencoder algorithm here.

The reason for this choice reason is two-fold: on the one hand we want to give credit to the original authors of the paper as this blogpost was clearly inspired by the 2D toy data set studies being conducted in their work. Given their very valuable published findings, it is worth conducting an even more comprehensive experiment.

The second reason is that we at Fourthline indeed found that the Soft-IntroVAE is more robust in terms of training compared to other approaches as mentioned above (all of which we worked with and tested).

Extensively seeking good hyperparameters (or even any hyperparameters that converge) is not just a tedious, but, given training times for reasonable data set sizes, often unfeasible job. Although different values are found and reported for different datasets in the original dataset, this blog post aims at deducing some general rules of relationships of hyperparameters for training Soft-IntroVAE models that can be applied to training, ideally agnostically to the data domain. We hope this helps developers understanding better the behavior between the different loss terms (of which Soft-IntroVAE has alot) and its corresponding hyperparameters.

Precap: This blog assumes a thorough understanding of Variational Autoencoders and its popular derivatives, particular the Soft-IntroVAE.
If you are not familiar with it, you might want to study them before continuing in this blog.

2D toy data set from the original paper

In the original paper, Table 2 compares the best results in terms of ELBO scores for different VAE-based approaches(VAE, IntroVAE, Soft-IntroVAE) resulting from an extensive grid search. The conducted 81, 1260 and 210 parameter sweeps, respectively.

They visualize the best results in terms of 4 different and known image space distributions (which are 2D points here):

Table 2 shows the best results being obtain on toy data. While the paper indeed reports different optimal hyperparameter setting for different datasets, interesting questions to still be raised are:

• How does the ELBO (or any other objective) manifold look like for a vast amount of hyperparameter combinations?
• How does this manifold behave in terms of smoothness and value ranges?

The following paragraphs should give an answer to these questions, followed by a discussion.

Recap Soft-Intro VAE objective

We do not want to revise the heritage of the derivation of the loss function of the different VAE-based objectives that brilliant research has brought forward. Building upon the VAE and Intro-VAE propositions, we will focus directly on the Soft-IntroVAE objective; its pseudo code is found below.

As you see, there are at least two additional hyperparameters (compared to conventional VAEs) that the authors of the Soft-IntroVAE paper put onto the table:

For that reason the graphs below are projections of the higher dimensional spaces (4-dimensional: β_rec, β_kl, β_neg, latent_space) onto the visualizable 2-dimensional space. Here, the projections performed are simple statistics such as the minimum value of a specific loss term among all other values of the remaining parameter. This is being part of the discussion below.

Experimental setup

The following are the results of 1489 trainings, where we want to fit the model to a superposition of 8 2D uniform Gaussians:

We vary

while keeping the latent dimension equal to 2 and

(as mentioned in the paper).

Here we perform a grid search over the parameter space. Note, that we want to gain intuition of how parameter sweeps influence the loss manifolds for the different loss terms. For non-brute force hyperparameter optimizations we refer to this source for an overview.

In the original paper, the authors wanted to explore and compare the different approaches on a toy dataset. Since we want to try to deduce general training rules for the training of actual relevant datasets, we do not alter the proposed Soft-IntroVAE architecture. This is in comparison to the papers’ investigation, where a 3 layer MLP is utilized on 8-Gaussians dataset.

Reconstruction loss: β_kl vs. β_rec

We start by looking at the two classic VAE losses — the KL loss and the reconstruction loss. As we are (now, in Soft-IntroVAE) in a higher (e.g. 4-dimensional) parameter space, we obtain multiple values for each (kl, rec) pair, given the sweeps through the β_neg space, as compare to a 2-dimensional optimization.

The first figure depicts the minimum along different values for β_neg of the logarithm of the reconstruction loss for the encoder of the real images (line 8 in Algorithm 1), plotted against sweeps of β_rec and β_kl between 0.0 and 1.0.

At the bottom of the figure we show regions where combinations of the hyperparameter tuple (β_rec, β_kl, β_neg) lead to NaN values for the loss — i.e. unsuccessful training:

What do we learn from this graph? Lower kl values lead to (the possibility of) lower losses — some β_neg values for the same β_kl, β_rec parameters result in very good reconstruction losses, other values for β_neg yield an unstable training.

Correcsponing to… and … note that this point is the min over all beta neg sweeps

The Soft-IntroVAE objective computes two more interesting reconstruction losses as described in line 10 in the algorithm: The loss of the reconstructed of the reconstructed images, L_rf_E (E for encoder), and loss of the reconstruction of the generated images, L_ff_E. They are depicted below in the left and right figure, respectively. As above, we show the losses measured over the backpropagation through the encoder.

We can see that the losses behave very similarly. In fact, we found this behavior in all our experiments across different datasets, including several image domains. For that reason, for the remainder of this blog, we always mean L_r_E when we talk about the reconstruction loss.

Choosing a ten times higher reconstruction loss compared to the *.GIF depicting the unsuccessful training above and thereby moving towards the white region of L_rf_E the model fits the data successfully:

Kullback-Leibler loss: β_kl vs. β_rec

What happens at the same time with the KL loss? A low reconstruction loss is desirable, but a ML practitioner training VAE’s knows that this does not necessarily coincide with a proper training: Low reconstruction loss can mean you reached a trivial solution, which means posterior collapse (only one image is/a few similar images are reconstructed from any embedding). Therefore we need to look also at the KL loss: We do not want a too high KL loss, in which case the embeddings are encoded following a density that is far away from your prior. If you embed to far from your prior and then, in the process of generating new images, draw a sample from that known prior to run it through your decoder, your decoder cannot properly handle this input and the generated images will show an insufficient quality (if they show anything meaningful at all). Depicted below is the minimum of the log of KL_z_E.

Comparing the KL loss to the reconstruction loss above we find the following. Firstly, of course NaN regions are the same — an unsuccessful results in unstable losses for both, the reconstruction as well as the KL.

Secondly, we look at the shape and slop of this loss manifold. While we had good regions for a low reconstruction loss particularly for low β_kl values, we see a conflictive behavior here: low β_kl values result, as expected, in high KL losses.

Combining both loss surfaces, good training regions seem to lie in rather high β_kl values and moderate β_rec values. That said, there exist values for β_neg that yield low total loss values (remember: the plot shows the minimum KL loss among different β_neg values).

So let’s look at the β_neg hyperparameter.

Reconstruction loss: β_rec vs. β_neg

The minimum reconstruction loss for different β_kl values, plotted over the 2-dimensional (β_rec, β_neg) space is shown below.

Apparent is the ridge at a low value of β_neg across the entire β_rec domain. In indicates clearly that — letting alone β_kl — there exists a sweet spot for β_neg (for this data set). Increasing it will lead to less good reconstruction errors but will remain a stable training as the surface remains smooth; decrease from that sweet spot leads to unstable training rapidly, also indicated by the NaN regions at very low β_neg values.

Kullback-Leibler loss: β_rec vs. β_neg

The KL loss appears stable for larger β_kl and β_neg values. A very strong decrease of the kl loss can be seen going towards unstable regions for low β_kl values (see above).

The area for higher β_neg values and moderate β_rec values promises stable training with a tendency of lower loss values. This is a behaviour we can also confirm from extensive experiments with other data sets and hyperparameter combinations.

Discussion

Of course one can argue: Why such an ablation, why deducing “intuition” when we have an analytical description of the problem? Why not using non-brute force hyperparameter optimization methods?

Well, there are at least two reasons:

1. In Soft-IntroVAE, we have at least 4 different hyperparameter. By means of the curse of dimensionality and the human limitation of imagination the goes beyond 3D, visualization and simplification by reducing dimensionality might give insights helpful for understanding the interaction between the losses and hyperparameters.
2. The ELBO terms have non-linear, particular exponential terms. Humans cannot properly deal with exponential behavior when it comes to prediction, inter- or extrapolation in a natural way.

Next steps

Further relationships that can be explored are the behavior of the reconstruction loss as well as the Kullback-Leibler loss of KL_z_E, by plotting them over β_kl and β_neg.

And the most important and final question: How do these figures look like for other data sets? We hope that the insights won on the 2D Gaussian dataset indeed generalize to other datasets and to apply them to our real world datasets! See you soon!

References

• Tutorial on beta-VAE (Open Review)
• Energy-based Generative Adversarial Network (arxiv)
• Soft-Intro VAE (arxiv)

Parameter search in Soft-IntroVAE — A 2D toy data set ablation was originally published in Fourthline Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tony was woken up by thunder, early on a Tuesday morning. But no matter. The excitement about putting his sinister plan into action soon took over. It was a well thought plan, meticulously prepared. He would open a bank account with a made-up identity and a bogus passport he got from Sidney. Nobody’s better than Sid, he assured himself. With a bank account he could finally wash all that dough that had been piling up from pushing nose candy, and whatever else he could come by. He sat confidently behind the computer and begun the enrollment process. That was easier than I expected, he thought as he leaned back and hit the submit button, a minute or so later. But, to his absolute surprise, his application was rejected within seconds!

How could that be? How was his attempted fraud detected, with such apparent ease and speed?

In this and future posts we will explain how that is possible. In particular we will present and analyze various unsupervised-learning methods applied to fraud detection, in the context of KYC (know your customer), at Fourthline. After briefly making a case for unsupervised learning we take a closer look at some of the approaches we have tested. We summarize the results and draw comparisons between those unsupervised methods, highlighting their advantages, as well as their drawbacks, and conclude with insights that we have gained by applying them to the challenging problem of fraud detection.

Introduction

KYC is the process of verifying the identity and assessing the suitability of potential customers of, primarily, financial institutions. This is important in order to mitigate risks associated with the initiation of a business relationship between those two parties. Money laundering is at the forefront of the various criminal activities that KYC aims at preventing. According to The United Nations, it reaches annual volumes of about 2–5% of global GDP.

Opening a bank account is a major step in the money-laundering process and fraudsters go to great lengths in order to achieve that. The different types of fraud they (try to) commit can be roughly grouped into document fraud and identity fraud. From tampered documents to deep-fake videos, we’ve seen it all at Fourthline. What we have understood is that in order to be successful in tackling fraud, a financial institution needs to be able to identify all fraud types.

At Fourthline we take fraud prevention very seriously. We achieve remarkable results not just by having fraud experts employing thorough processes to ensure security and compliance, but also by leveraging their knowledge to combine it with state-of-the-art machine-learning algorithms. As modern digital banks understand, complementing human expertise with insights from data analysis and modelling is essential to be successful in a landscape of increasingly sophisticated fraud attempts and strict regulatory directives.

Machine learning

There are various ways in which machine learning can come in handy when detecting fraud. On the one hand there are computer-vision algorithms, commonly based on deep neural networks, and on the other, data-science methods, relying on heterogeneous features and a variety of data types.

On the computer-vision side, the authenticity of documents can be verified in pretty much the same way the human eye would go about doing it; by looking for elements in the image that support or undermine the document’s authenticity. Furthermore, biometric data (e.g. face recognition, liveness detection) play an important role in uncovering identity fraud.

Beyond images, though, there is a large body of data that comes with every application that a KYC provider, such as Fourthline, has at its disposal. From the time of the application, to the type of document and (mobile) device used by the applicant, there is always something to be learnt from historical data. Such data is what we use in order to train algorithms into powerful fraud detectors.

The cool thing is that these two disciplines do not have to remain separate. Results of computer-vision algorithms can always be expressed in numbers, representing confidence in, or probability of a particular outcome, or generally, a ‘score’. These scores can then be taken into account within a data-science context, along with all other non-computer-vision data, on which a more holistic, hybrid approach can be built. And that is exactly how we have compiled the data sets which we use in the following sections.

Speaking of data sets, we note that all results presented in this post are reported on synthetic data, which are loosely based on trends we see in production data. Class imbalance of those data sets has the same ballpark value as in actual KYC databases. The exact numbers presented, therefore, should not be taken at face value. However, the qualitative conclusions we draw hold true on production data as well.

Why unsupervised learning?

Before getting into the nitty-gritty, it is worth motivating the use of unsupervised learning for fraud detection. After all, since unsupervised learning ignores labels by definition, why would one use it at all in a (binary) classification problem, such as fraud detection? The answer to that is two-fold:

• Unsupervised-learning methods are often the first course of action for data scientists when one wants to let the data (but not the labels) do the talking. In other words, when the preexisting understanding of the relationship between different features is non-existent, or limited, unsupervised learning can provide those first insights that might later prove valuable for feature engineering, outlier determination etc.
• In fraud detection correct data labeling is a luxury. Yes, data sets from all kinds of machine-learning problems often suffer from noise, mistakes and so on, but in the case of fraud detection especially, where human mastery of the domain is not common, or easy to come by, one has to be suspicious of the correctness of the labels, in particular the non-fraud kind of labels. Ultimately, we are talking about a heavily imbalanced data set with the fraud class samples being orders of magnitude fewer than those of the non-fraud class. With this kind of data sets, even if a small percentage of the dominant class (non frauds) should actually have the opposite label (fraud), this means that a significant part of the under-represented class is mislabeled, which we would like to avoid.

Having said that, we should make clear that when evaluating the usefulness of the various methods and tuning their hyperparameters, we did actually use the labels. Therefore, strictly speaking, we actually performed supervised learning, but the labels only come into play at the hyperparameter-tuning and the evaluation stages, while training is label agnostic. After all, what we are really looking for is to optimize the separation of frauds from non-frauds.

Evaluation metrics

To evaluate just how good this separation can be, we have mainly used two widely known metrics, precision and recall. The steps we follow to get precision/recall values are simple:

• Use a clustering algorithm (even a self-organizing map can be viewed as a collection of clusters, in a sense; more on that later) where the number of clusters is either predefined, or determined dynamically.
• Order clusters by fraud potency, i.e. by decreasing fraction of the applications inside each cluster that are fraudulent, regardless of the total number of applications inside the cluster.
• Evaluate precision and recall under different groupings of the ordered set of clusters.

K-means

K-means is perhaps the archetypal clustering method. With a simple and intuitive algorithm, and decent performance, it is typically the first (and often the last) stop in the data scientist’s journey towards a clustering approach. Its most notable disadvantage is that the number of clusters has to be provided by the user, i.e. is not inferred by the algorithm itself. However, there are ways around that shortcoming, which we discuss below.

A naïve application of k-means to our problem would be to try clustering the data in terms of the features, which in the case of k-means can only be numerical. Once this is done for many runs with different predefined numbers of clusters, criteria such as the elbow method, or the silhouette score can be used to determine which number of clusters results in the most efficient grouping. These criteria are still label agnostic. Once the number of clusters corresponding to the optimal clustering is found, we can start invoking labels to choose which of the clusters contain a significant number, or portion (or both) of fraudulent applications. These will be the clusters of interest.

In the plot above, the elbow method would seem to suggest that a number of clusters between 12 and 20 is the optimal, but which one? To try to answer that we can invoke the silhouette score.

The silhouette score seems to agree with the above conclusion (warning: this is not always the case) and furthermore suggests that the optimal cluster number is 12. It is worth repeating that we still have not accounted for labels in the data. The optimality of 12 clusters is in no way related to which cases are frauds and which are non frauds. It only has to do with label-agnostic measures of the sensibility of grouping data in the parameter space defined by the features.

When we do look at how frauds are distributed across the clusters it turns out that the choice of 12 clusters is indeed a good one. A histogram of the number of clusters as a function of the fraud content can be seen below.

The take-home message of this simple exercise is that we can split the data set in such a way that two clusters contain just 7.3% of all the data, while containing 56% of all fraud. These results are also confirmed on a test set and when contrasted with a prior fraud percentage of about 1%, they amount to a marked improvement over the starting point. However, we can do even better.

After devoting a bit more time and computational resources to the problem we found that using a few hundred clusters improves results noticeably. In the following example we have made use of 200 clusters and 400,000 data points. After ordering the clusters in terms of decreasing fraud potency (fraction of applications, i.e. data points, inside each cluster that are frauds), we can create the following plot.

For a 56% recall, as before, we now get ~17% precision in the training set, which is almost double that of the previous approach. Evaluation on the test set of 100,000 data points shows that the model is generalizing nicely.

Perhaps the greatest advantage of using such a large number of clusters is the flexibility that we then have in choosing a high-recall, or a high-precision approach, or a compromise between the two, depending on N. This can make such models a one-stop shop for the various use cases in KYC.

K-prototypes

Categorical data cannot be used with most of the clustering algorithms out there and k-means is no exception. One can always one-hot-encode categorical data, but on the one hand this will drastically increase the dimensionality of the problem (just how drastically, depends on the number of categories) and on the other, it ends up projecting the data on the ‘walls’ of the parameter-space hypercube, distorting the notion of euclidean distance, which is the defining ingredient of the k-means algorithm.

There are smarter ways to go about mixing numerical and categorical data in a clustering algorithm. One such example is k-prototypes, implemented in the k-modes library. The first thing we set about doing when testing this library was a type of sanity check. In particular, we tried to only use numerical data in order to confirm that on that limit, k-means results can be reproduced. Unfortunately, this is not possible and if one tries to do that they will get an error prompting them to use k-means since there is no categorical data in the data set.

Another quickly noticeable shortcoming of k-prototypes is that it is much slower than k-means. This limits the number of experiments we can run, and effectively prohibits runs with hundreds of clusters. Another effect of the slow speed of the algorithm is the limitations it imposes on the amount of data we can use, making anything above a few tens of thousands of samples practically unfeasible on a high-end PC. For 100,000 data points we could make runs up to about 30 clusters. Looking for an elbow we get the following picture.

When selecting clusters in terms of fraud content, results were rather disappointing, yielding much lower performance than k-means. After some investigations we narrowed down the cause for that to the presence of categorical variables. As we limited the number of categorical features, results slowly approached those of k-means, but without the possibility to run experiments with a large cluster number, we quickly parked this option. Perhaps the greater importance of the numerical features in the data has biased this conclusion, but it seems that there are no benefits in using k-prototypes over k-means for our problem.

Self-organizing maps

Self-organizing maps (SOMs) are a form of unsupervised learning, whereby a relatively simple neural network maps a higher-dimensional input space onto a lower-dimensional (usually 2D) lattice. The biggest difference compared to more conventional neural networks ‘learning’ through backpropagation is that SOMs are trained by utilizing competitive learning, whereby the lattice neurons (the elements of the low-dimensional map) ‘compete’ for the assignment of every training example and the best-matching unit wins. At the end of training each node of the map will generally have many training examples mapped onto it and the same mapping rules can then be used for inference. Essentially, each neuron of the output layer can be viewed as a cluster to which data points are assigned.

SOMs have two main strong points. First, they are fast enough to train and do inference with, even with many hundreds, or thousands of output nodes. Secondly, they make for cool and, more importantly, telling visualizations since dimensionality reduction is an inherent aspect of this method, rather than a post-processing step, as, for example, with k-means. On the flip side, they only take numerical features as input, which turns out to not be such a big issue in our case, as explained in the k-prototypes section, right above. Two libraries implementing SOMs in python were tested. Sompy mainly, and MiniSom to a lesser extent.

Choosing a square of 25x25 nodes as output we can train a SOM network within seconds, using tens of thousands of data points, on a conventional computer. Invoking the labels, we can then look at the mapping result.

The good news is that there is structure in the output, in terms of mapping fraud. Not only are the locations of more frauds not randomly distributed on the map, they also largely coincide with areas of high fraud potency. For a first run, this is promising.

With very little effort we can strengthen our insight and see what sort of (normalized) feature values correspond to each output node of the map.

By correlating these feature maps with the fraud maps, right above, we can gain a better understanding of what the main indicators of fraud are. And of course, we can go even deeper and correlate features to specific types of fraud, but this is beyond the scope of this post. The bottom line is that SOMs are pretty handy and make for insightful visualizations.

But how do they fare where it really matters, separating frauds from non frauds? Well, they are pretty good at that too.

Generalization to the test set is not bad at all, especially considering the dimensions of the map.

As we see, results are pretty similar to the ones we get from k-means. SOMs also offer the flexibility of choosing a high-recall or a high-precision approach (provided there is a large enough number of output nodes) and come with the added benefit of faster run times, and almost out-of-the-box visualizations.

Gaussian-mixture models

Gaussian-mixture models (GMMs) are very similar to k-means but with some crucial differences. Instead of N-dimensional spheres as in the case of k-means, clustering is based on N-dimensional gaussian distributions, each with their own mean and variance. Furthermore, the cluster that a data point belongs to is determined by the gaussian component with the highest probability-density function at that point.

Using the scikit-learn implementation we have experimented with different numbers for the components (clusters) and covariance type. We have found that about one hundred components (similar to k-means) and “full” covariance give the best results. However, the computational cost is higher than k-means and this comes at no greater reward. Results on a 200,000 train set, with 5 features and 100 mixture components are as follows.

Similar results are reproduced on a test set, which underlines the generalizing power of this algorithm, as long as the number of components remains reasonable, around 100.

In summary, despite the decent performance of this approach, it does not present itself with any advantage over previous ones. In fact, due to its slower run times, exploration of all the parameters that the scikit-learn library has to offer is not feasible on a PC within a reasonable time frame.

(H)DBSCAN

DBSCAN (or density-based spatial clustering of applications with noise) is an algorithm that groups together data points based on local-density estimates, rather than a distance measure from a given point, such as the centre of a cluster. In this way, data points that are quite far apart may actually be placed in the same cluster whereas others that are relatively near to one another may be placed on different clusters. It produces very nice results, especially when applied to non-globular groups of data.

Some of the main selling points of DBSCAN are the flagging of outliers and its dynamic determination of the number of clusters, both greatly easing the practitioner’s job. Another interesting aspect of the scikit-learn implementation is the ability it offers to select from a long list of distance measures to be used in the density calculation, such as “L1“, “cosine“, “Hamming“ and many others, next to the more conventional euclidean distance measure.

Early results of running DBSCAN were promising. With very few tweaks of the initialization parameters more than half of all frauds, and only 10% of all the data were deemed outliers. An important ingredient in this result was the metric. With ‘canberra’ (weighted L1, essentially) we found that we could get better results than with other L1 variants or different metrics. However, DBCSAN has one important drawback. You cannot do inference with it based on previous trainings. In scikit-learn lingo, it does not have a predict function. This is where HDBSCAN comes to the rescue.

HDBSCAN (where the “H“ stands for hierarchical) is a more recent variation of DBSCAN, which allows for clusters of varying density. This difference is expressed through a distinct set of parameters which do take some tweaking to uncover the algorithm’s full potential. More importantly, it comes with a predict function! Its different parameters take some getting used to and come with some perks, but make for some interesting results.

The plot above is the result of a compromise between generalization and performance. However, as we see in the test-set plot below, the trained algorithm fails to closely replicate that behaviour in the train set.

In light of some much better results we could get on the train set, for different parameters, the weak point of this algorithm is generalization. Perhaps unsurprisingly, the best performance on the train set was achieved by allowing for clusters of down to about 5 samples, resulting in many thousands of clusters that would then fail to generalize on the test set. An interesting parameter of HDBSCAN is cluster_selection_epsilon, which in a sort of post-processing manner merges smaller clusters at the train stage. However, when trying to do inference after having trained with this parameter, results were impossible to reproduce, even on the train set itself. Concluding, HDBSCAN has great potential for one-off use, both through clustering and outlier determination but fails to generalize to unseen data, which is what we are looking for.

Comparison

In the following table we will undertake the difficult task of comparing all the algorithms we presented, based on 4 criteria:

• Performance in terms of precision/recall metrics
• Generalization to the test set
• Ease of use, including set up, parameters and visualizations
• Computational resources

The most important aspect we were looking for when testing these algorithms was good performance that can be replicated on a test set. This is important in order to be able to use a trained model for inference, perhaps as a first step in a machine-learning pipeline, followed by some sort of supervised algorithm. However, despite being secondary, ease of use and resource consumption are also important parameters, especially when experimenting on a laptop and wanting to explore the different possibilities that the parameters of these algorithms present. Under that light, it is SOMs and k-means that come out on top, with K-means showing marginally better generalization in the synthetic data, but SOMs offering a great all-around experience, without really lagging behind in the performance department.

If we have to pick a single winner, so to speak, it will probably be SOMs due to the nature of the mapping lending itself naturally to accurate (no extra dimensionality reduction) and insightful visualizations, combined with excellent run-time performance and practically no weakness in its predictive power.

Insights

For k-means, SOMs and gaussian mixtures we were able to explore how the generalization capabilities of the trained models evolved as a function of the number of clusters. For all of them we noticed some degradation as the number of clusters increased. This is to be expected since large cluster numbers imply low number of cases per cluster which makes it easy to cherry pick clusters in the train set with very high precision and accumulate them to build up recall. However, there is no guarantee that this will be as easy on a test set.

On the other hand, relatively high cluster numbers are necessary to have a smooth transition from high-precision to high-recall cluster selections. Which approach will be selected depends on the use case, ranging from automating part of the KYC verification pipeline (high precision is necessary in this case), to having clustering as the first step in a machine-learning pipeline (where a high-recall approach would be favoured). What is certain is that having the option to go for either is desirable.

Perhaps the most important insight that we got from trying out k-prototypes is that the numerical features of our date sets are more important that the categorical ones. This conclusion has been confirmed by subsequent supervised-learning experiments. The implication is that k-prototypes should not be dismissed in general, since there may be data sets where categorical and numerical features coexist and are equally important. For those cases k-prototypes would be one of the very few options for clustering.

Finally, it is worth pointing out that when circumstances allow (H)DBSCAN can be a very powerful algorithm, not least because it is the only one analyzed in this post that offers distance metrics alternative to euclidean, out of the box. This would not have been noted were it not for the L1 distance (and its variants) producing better results than the euclidean metric. Given that observation, combining an L1-type metric with a nicely generalizing algorithm, for example k-means, presents itself as an attractive exercise. And that is exactly what we did, by combining the k-means implementation of scikit-learn with scikit-learn’s definition of the Manhattan distance. The results from this experiments easily matched the ones we get with a euclidean metric without, however, improving on them.

Conclusion

We have presented an introduction to KYC and one of its main goals, which is to prevent fraudulent applications from succeeding in opening a bank account. We motivated the use of, and focused on, some of the unsupervised-learning algorithms that we tested at Fourthline in order to train algorithms into fraud detectors, using historical data. We showed how each of these algorithms performs and provided insights into their applicability to the problem at hand and beyond.

However, the story does not end here. Unsupervised learning is just one of the things we have tried and, as we shall see in a future post, supervised methods can outperform the algorithms we presented here. So, stay tuned for the follow-up article.

Vuk Glisovic has performed part of the analysis and modelling presented in this post. Sebastian Vater has supervised this project. The entire AI team at Fourthline has contributed through useful discussions and suggestions.

Revenge of the nerds: How data scientists catch fraudsters (part I) was originally published in Fourthline Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.