dynmeth / mongo-statsd-backend

A backend for StatsD to emit stats to mongodb.

StatsD backend for Mongo DB

Overview

This is a pluggable backend for StatsD, which publishes stats to mongodb.

How it works

This backend uses Mongo's capped collections to have near file system performance for logging data points from StatsD.

Installation

$ npm install mongo-statsd-backend

Configuration

Inside of your StatsD server config file, use the following parameters:

{
	mongoHost: 'user:pass@localhost',
	mongoPort: 27017,
	mongoMax: 2160, 
	mongoPrefix: true, 
	mongoName: 'databaseName',
	backends: ['/path/to/module/lib/index.js']
}

• mongoHost: the ip address or hostname of the mongo server. Default is localhost.
• mongoMax: the number of data points to cap the collection with. Default is 2160. With Statsd's default of 10 seconds, this gives 6 hours of 'near real-time' data.
• mongoPrefix: Boolean. If true, then the statsd "bucket" names contain a prefix which deterine the database name. For example, if a counter is called 'web-server.page_hits' then the database name will be 'web-server' and the collection name will be 'page_hits'. Otherwise, the database name will be 'statsd' and the collection name will be 'web-server.page_hits'.

Schema

The schema follows the StatsD namespace.

database.collection.tite.flush_rate

bucket.metric.metric_title_flushrate

db.counters

{
	time: 1234567890, // time_stamp from statsd
	count: 124 // Integer from statsd
}

db.timers

{
	time: 1234567890, // time_stamp from statsd
	durations: [0, 1, 2] // Array from statsd
}

db.timer_data

{
	time: 1234567890, // time_stamp from statsd
	mean_90: 1.1764705882352942,
	upper_90: 3,
	sum_90: 80,
	std: 60.18171889277414,
	upper: 526,
	lower: 0,
	count: 75,
	count_ps: 7.5,
	sum: 652,
	mean: 8.693333333333333,
	median: 1
}

db.sets

{
	time: 1234567890, // time_stamp from statsd
	set: ['name1', 'name2'] // Array from statsd
}

Dependencies

• mongodb

Development

• Bugs

mongo-statsd-backend's Issues

Great tool How can i visualize

I'm using your solution instead of the graphite backend.
Was wondering how to visualize the data though with/without graphite

Thanks!

Unable to start Mongo-back-end

C:\elk\statsD\node_modules\statsd\node_modules\mongo-statsd-backend\lib\index.js:21
new mongo.Db(task.name, new mongo.Server(options.host, options.port)).open(function(err, db) {
^

TypeError: (intermediate value).open is not a function
at C:\elk\statsD\node_modules\statsd\node_modules\mongo-statsd-backend\lib\index.js:21:73
at C:\elk\statsD\node_modules\statsd\node_modules\async\dist\async.js:4096:9
at Object.process (C:\elk\statsD\node_modules\statsd\node_modules\async\dist\async.js:2344:17)
at C:\elk\statsD\node_modules\statsd\node_modules\async\dist\async.js:2252:19

.Conf File

{
mongoHost: 'localhost',
mongoPort: 27017,
mongoMax: 2160,
mongoPrefix: true,
mongoName: 'testdata',
backends: ['C:\elk\statsD\node_modules\statsd\node_modules\mongo-statsd-backend\lib\index.js']
}

Schema issue

I don't get a db.timer_data collection but in my db.timers collections I do see objects in the form of

{
    "mean_90" : 1600,
    "upper_90" : 1600,
    "sum_90" : 1600,
    "std" : 0,
    "upper" : 1600,
    "lower" : 1600,
    "count" : 1,
    "count_ps" : 0.1,
    "sum" : 1600,
    "mean" : 1600,
    "median" : 1600,
    "time" : 1372670684,
    "_id" : ObjectId("51d14adb6ad9444547000003")
}

But they aren't created every "bucket size" (10 seconds) . What frequency are they created?

mongo-statsd-backend crashes statsd daemon

When using with statsd version 0.5.0 I get the following stack trace:

connection_queue.drain()

/usr/lib/node_modules/mongo-statsd-backend/node_modules/mongodb/lib/mongodb/connection/server.js:524
throw err;
^
Error: Callback was already called.
at i (/usr/lib/node_modules/mongo-statsd-backend/node_modules/async/lib/async.js:22:31)
at onFlush (/usr/lib/node_modules/mongo-statsd-backend/lib/index.js:155:8)
at insert (/usr/lib/node_modules/mongo-statsd-backend/lib/index.js:126:8)
at insertAll (/usr/lib/node_modules/mongo-statsd-backend/node_modules/mongodb/lib/mongodb/collection.js:343:12)
at Collection.insert (/usr/lib/node_modules/mongo-statsd-backend/node_modules/mongodb/lib/mongodb/collection.js:91:3)
at insert (/usr/lib/node_modules/mongo-statsd-backend/lib/index.js:122:10)
at Db.createCollection (/usr/lib/node_modules/mongo-statsd-backend/node_modules/mongodb/lib/mongodb/db.js:872:16)
at Db._executeQueryCommand (/usr/lib/node_modules/mongo-statsd-backend/node_modules/mongodb/lib/mongodb/db.js:1692:5)
at g (events.js:192:14)
at EventEmitter.emit (events.js:126:20)

The issue does not happen with 0.4.0.

TypeError: Cannot call method 'insert' of null

I'm pretty new to nodejs and have a trouble with it.
I use statsd to collect statistics and then send them to mongodb server for store，
but when i send the statsd's timer data to mongodb, i came across this problem.

/home/cadofa/node_modules/mongo-statsd-
backend/node_modules/mongodb/lib/mongodb/connection/base.js:245
        throw message;      
              ^
TypeError: Cannot call method 'insert' of null
    at /home/cadofa/node_modules/mongo-statsd-backend/lib/index.js:161:15
    at /home/cadofa/node_modules/mongo-statsd-
    backend/node_modules/mongodb/lib/mongodb/db.js:1011:22
    at /home/cadofa/node_modules/mongo-statsd-
    backend/node_modules/mongodb/lib/mongodb/db.js:1130:16
    at /home/cadofa/node_modules/mongo-statsd-
    backend/node_modules/mongodb/lib/mongodb/db.js:1846:9
    at Server.Base._callHandler (/home/cadofa/node_modules/mongo-statsd-
    backend/node_modules/mongodb/lib/mongodb/connection/base.js:445:41)
    at /home/cadofa/node_modules/mongo-statsd-
    backend/node_modules/mongodb/lib/mongodb/connection/server.js:478:18
    at MongoReply.parseBody (/home/cadofa/node_modules/mongo-statsd-
    backend/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
    at null.<anonymous> (/home/cadofa/node_modules/mongo-statsd-
    backend/node_modules/mongodb/lib/mongodb/connection/server.js:436:20)
    at EventEmitter.emit (events.js:95:17)
    at null.<anonymous> (/home/cadofa/node_modules/mongo-statsd-
    backend/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:201:13)

Anyone who can tell me how to solve the problem, thanks very much!