Anomaly Detection
A recap of Andrew Ng course (Machine Learning Specialization) on Coursera
Course 3 Week 1
This is another unsupervised learning model which looks at normal events and detects abnormal events. The algorithm that addresses this problem is density estimation, which measures the probability of features, X, being in the dataset. Those that closer to the center of the dataset as shown below should have higher probability of being seen in the dataset. We can get the probability of a test set using the algorithm, and if the probability is smaller than a a very small number like epsilon, we will say this is an anomaly.
In order to apply anomaly detection, we need Gaussian Distribution (also known as normal distribution).
The sum of probability needs so be one, so when the Guassian distribution become thinner it needs to also become taller.
Next, what anomaly detection does is given the dataset X consisting of just one feature, we estimate the parameters, mu and sigma squared, of the plausible Guassian distribution that generated the X dataset. We can calculate the values of mu and sigma squared using the equations shown below (the equations are also named as the maximum likelihood of mu and sigma). With the mu and sigma squared, we can draw a gaussian distribution which looks like the dataset X might have been drawn from that Gaussian distribution.
If we get an example x closer to the mu, we will see higher probability, which means this item is more likely to be normal data point.
Next, we will look at how to apply anomaly detection for a dataset consisting of more than one feature.
Given a training set X where each of the example has n features, we need to estimate the probability p(x) by calculating the product of p(x1) * p(x2) * p(x3) … * p(xn). We get p(x1) by getting the mu and sigma squared of the first feature. For example, if we were to do anomaly detection for airplane engine, we might have features such as engine temperature, vibration…etc. So for the first feature, engine temperature, we calculate the mu and sigma squared of the temperature to get p(x1). We do the same for the rest of the features. (The equation assumes that the features are statistically independent, but it turns out that this algorithm often works fine even if the features are not statistically independent)
One intuition for this multi-feature algorithm is that the p(x) would be a very small number when just one or two features have very small probability. This algorithm is a systematical way to quantify whether the new data has any feature that is unusually large or unusually small.
The above is a visualization example of anomaly detection for a dataset with 2 features. We pick a small value, epsilon, to be 0.002, so when we have new data x1 and x2, we can decide whether they are anomaly or not.
But how to pick a value for the epsilon and how to know if the anomaly detection algorithm is actually working? Next, we will learn how to evaluate the algorithm
Real-number evaluation
Get Aimyo’s stories in your inbox
Join Medium for free to get updates from this writer.
Although this is an unsupervised algorithm, it will be great if we could evaluate the model through some metrics. We could also tune the epsilon value and the features to find the best model.
One way to evaluate the anomaly detection algorithm is to include several anomaly data in the cross validation set and the test set. We can check how many of the flawed data points are correctly identified in the validation set. If we have very few known anomaly data points, it is also fine to just evaluate the algorithm with cross validation set, and tune the epsilon values and the features to include in the algorithm. However, since we are not testing the model on unseen data, it is possible to overfit the algorithm.
To evaluate such skewed dataset, accuracy is not a suitable option. The possible evaluate metrics would be true positive, false positives, true negative, false negative, recall , precision and F1-score. One way to determine which examples are anomalies is to select a threshold based on a cross validation set. We can calculate the F1 score for a list of epsilon for the cross validation set to pick an epsilon that gives us the highest F1 score.
This raises a question, if we have a few labeled dataset in the first place, why not just build a supervised learning algorithm? We will compare the two next.
Anomaly Detection V.S. Supervised Learning
Normally, if we have just 0–20 positive examples and a large number of negative examples we would pick anomaly detection. On the other hand, if we have large number of both positive and negative examples, we would pick supervised learning. However, it is still acceptable to pick supervised learning if we only have 20 positive examples. For that kind of dataset (around 20 positives and a large number of negatives), we decide whether we want to build an anomaly detection or a supervised learning model based on the future dataset that we will encounter.
The ways that the anomaly detection and supervised learning look at the dataset are quite different. If there are many different types of anomalies/positive examples, the available positive examples we have now (around 20) may not cover all types of anomalies and the model won’t even be able to predict positive for a new type of anomaly that it didn’t see during training. Since anomaly detection algorithm learn just from the negative data, it will be able to detect even a new type of anomaly. In contrast, supervised learning expects future positive examples to be similar to the ones in the training set.
More examples
Next, we will learn how to pick the features we feed to the anomaly detection algorithm.
It is very important to pick the correct and most suitable features for the anomaly detection algorithm, but if we don’t have the feature that fits the training quite right or if we have some extra features for supervised learning, it often turns out to be okay. Because supervised learning has the signal from the label that helps the model figure out what features to ignore or how to rescale a feature to take the best advantage of the dataset that we provide to it.
However, the anomaly detection learn from the unlabeled dataset, so it is harder for it to decide what to ignore. One way to prepare data for anomaly detection is to make sure the features are more or less Gaussian. If the features are not gaussian, there are a few ways to transform them.
Normally, larger value of C will make the transformation less gaussian, and Andrew recommended to try a bunch of C to pick the one that looks more like Gaussian, this normally works well for our practical purposes. (There are also some formal ways to automatically measure how close these distributions are to Gaussians, but eyeballing normally gives us good results)
Error Analysis
It is common to get large p(x) for normal and anomalous examples. To improve model performance, we can review the false negatives and see if we can figure out a new feature which can let the model learn better.
Here is another example
