amadeusitgroup / anomaly-detection-with-gaussian-mixtures

Anomaly Detection with gAussian Mixtures (ADAM)

This library provides an anomaly detection algorithm, based on the Gaussian Mixture Model (GMM) clustering algorithm.

Motivation

The motivation behind this contribution is to share a simple, yet efficient, anomaly detection algorithm that is able to handle multiple types of normality. The powerful underlying GMM algorithm is known for being well founded in statistical learning theory, as well as having good convergence properties. We rely on its Spark implementation for a fast, distributed computing of the preliminary GMM model.

Scientific Details

Given the mixtures weights and Probability Density Function (PDF) of a Gaussian Mixture Model, the anomaly score of a given point x in the feature space is classically computed as the negative log probability of the point x being generated by the mixture of gaussian distributions.

Formerly it corresponds to the following definition:

with:

• the number of components of the mixture model,
• the weight of the component i in the mixture model
• the probability of point x to be originating from the i-th gaussian component of the model.

The returned score is always positive but not bounded, since the probability if being generated by the gaussian mixture model can be in theory become as close as possible to 0. However, in practice, if you prefer using a bounded anomaly score, we suggest using the normalized anomaly score, always comprised between 0.0 and 1.0.

The normalized score is defined as follows:

with corresponding to the anomaly score of the point with probability to be generated by the mixture model equal to the minimum float precision.

How to build

==============

Run sbt package, then copy the resulting jarfile from target/scala-2.11 to your local package repository. Change the parameters in build.sbt if you need to modify the version of Scala and/or Spark you are compiling against. We tested it against Scala 2.11.8 and Spark 2.1.0

How to use

The library provides the object GaussianMixtureModelWithScoring.

This object collects funtions that allow to use a GaussianMixtureModel in anomaly detection

Normally you will use a Pipeline to obtain a PipelineModel, one stage of which is a GaussianMixtureModel Once the model is trained, you should have the GaussianMixtureModel and the Vector of features used in training

Call the scoring function like this:

val raw_score_column_name = "raw_score" // the column where you want to store the scores, defaults to "raw_score"
val normalized_score_column_name = "score" // the column where you want to store the scores, defaults to "score"
val input_df : DataFrame = // your input dataset 
val gmm: GaussianMixtureModel = // get the trained GaussianMixtureModel 
val with_raw_score : DataFrame = input_df.withColumn(<raw_score_column_name>, GaussianMixtureModelWithScoring.scoreWith(gmm)(col("<the column holding features used in training>")))
val with_normalized_score : DataFrame = GaussianMixtureModelWithScoring.normalize_score(with_raw_score)

Your input_df will now have a column named <normalized_score_column_name> holding the score for each input

Please check the unit tests for more examples.