zhongyuchen / outlier-detection

Detect outliers with 3 methods: LOF, DBSCAN and one-class SVM

Outlier Detection

Detect outliers with 3 methods: LOF, DBSCAN and one-class SVM

Prerequisites

• Required packages can be installed with the following command:

pip install -r requirements.txt

Data

• consumption_data.xls is provided. There are 4 columns with 940 entries. The first column denotes entry ID, which is ignored in detecting outliers. Therefore, the data entries are 3-dimensional.
• Get numpy array data with size [940, 3] with the following code (check out dataset.py for implementation):

from dataset import get_dataset

data = get_dataset()

• Data visualization:

Methods

For detailed descriptions please see report.pdf.

Density based method: LOF (local outlier factor)

• Check out lof.py for implementation.
• Result:

Cluster based method: DBSCAN

• Check out dbscan.py for implementation.
• Result:

Classification based method: One-class SVM

• Check out svdd.py for implementation.
• Result with Gaussian kernel:

• Result with linear kernel:

Author

Zhongyu Chen

outlier-detection's Issues

Problem with "reachability distance" formula

Hi, i have read your code and i have to say that it is a beautiful code. but it seems to have a small problem at below line in lof.py file.

https://github.com/czhongyu/outlier-detection/blob/842e14d7c011951728888ec249b4077be3f8f8ec/lof.py#L48

I think it should be: rd += max(dis[index][i], dis[kdn[index][i]][-1]) (max instead of min)

I read some articals about this algorithm, and and they all seem to use max instead of min. For example in this artical: https://en.wikipedia.org/wiki/Local_outlier_factor

And i try to compare your result from your code with sklearn.neighbors.LocalOutlierFactor, when i change min to max, they give the same result.

And by the way, thank you for your great code.