C++ implementation of
- Real-time Streaming Anomaly Detection in Dynamic Graphs. Siddharth Bhatia, Rui Liu, Bryan Hooi, Minji Yoon, Kijung Shin, Christos Faloutsos. (Under Review)
- MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams. Siddharth Bhatia, Bryan Hooi, Minji Yoon, Kijung Shin, Christos Faloutsos. AAAI 2020.
The old implementation is in another branch OldImplementation
, it should be considered as being archived and will hardly receive feature updates.
- Finds Anomalies in Dynamic/Time-Evolving Graph: (Intrusion Detection, Fake Ratings, Financial Fraud)
- Detects Microcluster Anomalies (suddenly arriving groups of suspiciously similar edges e.g. DoS attack)
- Theoretical Guarantees on False Positive Probability
- Constant Memory (independent of graph size)
- Constant Update Time (real-time anomaly detection to minimize harm)
- Up to 55% more accurate and 929 times faster than the state of the art approaches
- Experiments are performed using the following datasets:
If you use Windows:
- Open a Visual Studio developer command prompt, we want their toolchain
cd
to the project rootMIDAS/
cmake -DCMAKE_BUILD_TYPE=Release -GNinja -S . -B build/release
cmake --build build/release --target Demo
cd
toMIDAS/build/release/
.\Demo.exe
If you use Linux/macOS:
- Open a terminal
cd
to the project rootMIDAS/
cmake -DCMAKE_BUILD_TYPE=Release -S . -B build/release
cmake --build build/release --target Demo
cd
toMIDAS/build/release/
./Demo
The demo runs on MIDAS/data/DARPA/darpa_processed.csv
, which has 4.5M records, with the filtering core (MIDAS-F).
The scores will be exported to MIDAS/temp/Score.txt
, higher means more anomalous.
All file paths are absolute and "hardcoded" by CMake, but it's suggested NOT to run by double clicking on the executable file.
Cores are instantiated at MIDAS/example/Demo.cpp:67-69
, uncomment the chosen one.
You need to prepare three files:
- Meta file
- Only includes an integer
N
, the number of records in the dataset - Use its path for
pathMeta
- E.g.
MIDAS/data/DARPA/darpa_shape.txt
- Only includes an integer
- Data file
- A header-less csv format file of shape
[N,3]
- Columns are sources, destinations, timestamps
- Use its path for
pathData
- E.g.
MIDAS/data/DARPA/darpa_processed.csv
- A header-less csv format file of shape
- Label file
- A header-less csv format file of shape
[N,1]
- The corresponding label for data records
- 0 means normal record
- 1 means anomalous record
- Use its path for
pathGroundTruth
- E.g.
MIDAS/data/DARPA/darpa_ground_truth.csv
- A header-less csv format file of shape
- Include the header
MIDAS/src/NormalCore.hpp
,MIDAS/src/RelationalCore.hpp
orMIDAS/src/FilteringCore.hpp
- Instantiate cores with required parameters
- Call
operator()
on individual data records, it returns the anomaly score for the input record
The code we used for experiments.
It will try to use Intel TBB or OpenMP for parallelization.
You should comment all but only one runner function call in the main()
as most results are exported to MIDAS/temp/Experiiment.csv
together with many intermediate files.
Similar to Demo.cpp
, but with all random parameters hardcoded and always produce the same result.
It's for other developers and us to test if the implementation in other languages can produce acceptable results.
DeleteTempFile.py
, EvaluateScore.py
and ReproduceROC.py
will show their usage and a short description when executed without any argument.
The code to process the raw dataset into an easy-to-read format.
Datasets are always assumed to be in a folder in MIDAS/data/
.
It can process the following dataset(s)
DARPA/darpa_original.csv
->DARPA/darpa_processed.csv
,DARPA/darpa_ground_truth.csv
,DARPA/darpa_shape.txt
- Python: Rui Liu's MIDAS.Python, Ritesh Kumar's pyMIDAS
- Golang: Steve Tan's midas
- Ruby: Andrew Kane's midas
- Rust: Scott Steele's midas_rs
- R: Tobias Heidler's MIDASwrappeR
- Java: Joshua Tokle's MIDAS-Java
- Julia: Ashrya Agrawal's MIDAS.jl
If you use this code for your research, please consider citing our arXiv preprint
@misc{bhatia2020realtime,
title={Real-Time Streaming Anomaly Detection in Dynamic Graphs},
author={Siddharth Bhatia and Rui Liu and Bryan Hooi and Minji Yoon and Kijung Shin and Christos Faloutsos},
year={2020},
eprint={2009.08452},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
or our AAAI paper
@inproceedings{bhatia2020midas,
title="MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams",
author="Siddharth {Bhatia} and Bryan {Hooi} and Minji {Yoon} and Kijung {Shin} and Christos {Faloutsos}",
booktitle="AAAI 2020 : The Thirty-Fourth AAAI Conference on Artificial Intelligence",
year="2020"
}