Plz use https://github.com/eleme/banshee instead.
Bell.js is a real-time anomalies(outliers) detection system for periodic time series, built to be able to monitor a large quantity of metrics. It collects metrics form statsd, analyzes them with the 3-sigma, once enough anomalies were found in a short time it alerts us via sms/email etc.
We eleme use it to monitor our website/rpc interfaces, including api called frequency, api response time(time cost per call) and exceptions count. Our services send these statistics to statsd, statsd aggregates them every 10 seconds and broadcasts the results to its backends including bell, bell analyzes current stats with history data, calculates the trending, and alerts us if the trending behaves anomalous.
For example, we have an api named get_name
, this api's response time (in ms)
is reported to bell from statsd every 10 seconds:
51, 53, 49, 48, 45, 50, 51, 52, 55, 56, .., 300
Bell will catch the latest stat 300
and report it as an anomaly.
Why don't we just set a fixed threshold instead (i.e. 200ms)? This may also work but we may have a lot of apis to monitor, some are fast (~10ms) and some are slow (~1000ms), it is hard to set a good threshold for each one, and also hard to set an appropriate global threshold for all. Bell sloves this via 3-sigma, it gives dynamic thresholds for each metric ("learned" from history data). We don't have to set a threshold for each metric, it will find the "thresholds" automatically.
- Automatically anomalies detection.
- Designed for periodic metrics.
- Anomalies visualization on the web.
- Alerting rules administration.
If you think that bell is too complicated, you may check out our noise, which is simpler and faster but only for anomalies detection.
- nodejs 0.12+ (generator feature required)
- beanstalkd (https://github.com/kr/beanstalkd) (we are using version 1.9)
- ssdb (https://github.com/ideawu/ssdb) (we are using version 1.6.8.8)
Bell is written in Node.js, and requires node.js v0.12+ (generator feature).
Install bell
as a system command via npm:
$ npm instal bell.js -g
You can download exampleConfig.js and edit the configuration items according to documentation and default values in the script comments.
Bell has 5 services(or components), they share the same configuration.
Currently, bell onlys support statsd as data source.
In order to forward metrics to bell from statsd, we should add bell.js
to statsd's
backends:
-
Install
bell.js
on your statsd server.$ cd path/to/statsd $ npm install bell.js
-
Add module
bell.js
to statsd's backends:{ , backends: ['bell.js'] , bellHost: 'localhost' , bellPort: 2015 }
Metrics and administration data are stored on disk:
- metrics and trend states are stored in ssdb
- alerting rules are stored in sqlite.
And we use beanstalkd to dispatch analyzation jobs.
- For ssdb, clone down
https://github.com/ideawu/ssdb
and runmake install
. - For beanstalkd, clone down
https://github.com/kr/beanstalkd
and runmake install
.
Then start ssdb-server and beanstalkd:
$ ssdb-server -f path/to/ssdb.conf
$ beanstalkd
Bell has 5 services (or process entries):
-
listener
Receive incoming stats from statsd over tcp, pack to jobs and send them to job queue.
-
analyzer
Fetch jobs from queue, analyze current stats with history data via 3-sigma rule and send message to alerter once an anomaly was detected.
-
webapp
Visualize analyzation results and provide alerting management.
-
alerter
Alert via email or text message once enough anomalies were detected.
-
cleaner
Clean metrics that has a long time not hitting bell.
Start them all, and I suggest you to manage these services with some process manager like supervisord:
$ bell analyzer -c config.js
$ bell listener -c config.js
$ bell webapp -c config.js
$ bell alerter -c config.js
$ bell cleaner -c config.js
Alerting sender is the only item that is a little bit difficult
but must be customized by yourself. It's a nodejs script (or module)
which will be called from bell alerter on anomalies detected, in order to
alert receivers via emails or text messages. The sender script
should export a function sendEmail
or sendSms
(or both), you may want to
see exampleSender.js for example.
MIT Copyright (c) 2014 - 2015 Eleme, Inc.