grobian / carbonserver

A server to handle metric globbing and data retrieval written in go – look at carbon-go instead

carbonserver

Simple whisper file server over HTTP.

This project aims to be a replacement of the graphite web API used on carbon stores to retrieve the whisper data for viewing.

The main reason to build a replacement is performance. This server only supports the find and render calls that return raw data (e.g. no rendered images).

carbonserver understands the /metrics/find and /render URLs sent by the carbon web-frontend to the stores (their web-frontend), and responds in a compatible way. As such, it can be used as a drop-in replacement. When used in combination with carbonzipper, carbonserver uses a more optimal communication protocol that puts less strains on Go's memory usage and garbage collector.

Authors

Fabian Groffen Damian Gryski

Acknowledgement

This program was originally developed for Booking.com. With approval from Booking.com, the code was generalised and published as Open Source on github, for which the authors would like to express their gratitude. Fabian Groffen no longer works for Booking.com.

carbonserver's Issues

split 404s from actual errors in error metric

Since the frontend often requests metrics we don't have (optimistic approach, bypassing a find) the errors metric is polluted with "false positives". These should be ignored or split out to a separate metric to track its pattern.

Improve search code for successful matches

When there are no matches, the trigram code is quick to eliminate all the local metrics due to the first few 'rare' trigrams in the query. For queries that return metrics, a large number of long posting lists need to be intersected. This can be reduced by maintaining both a term index and a trigram index, using the term index for all parts of the query path that contain a full node, and the trigram index for components with globbing characters.

This is overkill for our needs and currently very low priority.

trigram searching code doesn't like queries for '*'

The zipper sends these queries to figure out top-level metrics stored on this server.

This might be related to the 'no usable trigrams from query' bug.

/metrics/find is not compliant with what graphite-web expects

I've been trying to integrate carbonserver + carbonzipper with graphite-web, and it seems that the response by carbonserver for the /metrics/find query made towards CLUSTER_SERVERS by graphite-web is incompatible with the equivalent response graphite-web issues:

In the example below, the receiver on port 8080 is graphite-web and 8081 is carbonserver. The same happens when both queries are issued in 'format=pickle' instead of json

$ curl "bcarbon00:8080/metrics/find/?local=1&format=treejson&query=%2A"
[{"text": "carbon", "expandable": 1, "leaf": 0, "id": "carbon", "allowChildren": 1}, {"text": "mlt3", "expandable": 1, "leaf": 0, "id": "mlt3", "allowChildren": 1}, {"text": "mlt3-ppc1", "expandable": 1, "leaf": 0, "id": "mlt3-ppc1", "allowChildren": 1}, {"text": "mlt3-ppc2", "expandable": 1, "leaf": 0, "id": "mlt3-ppc2", "allowChildren": 1}, {"text": "mlt3-ppc3", "expandable": 1, "leaf": 0, "id": "mlt3-ppc3", "allowChildren": 1}]

$ curl "bcarbon00:8081/metrics/find/?local=1&format=json&query=%2A"
{"name":"*","matches":[{"path":"carbon","isLeaf":false},{"path":"graphite.db","isLeaf":false},{"path":"mlt3","isLeaf":false},{"path":"mlt3-ppc1","isLeaf":false},{"path":"mlt3-ppc2","isLeaf":false},{"path":"mlt3-ppc3","isLeaf":false}]}

The request issued by graphite-web is in pickle format, but I've shown the output in json here for readability. The actual request is made in pickle, and the response by carbonserver causes graphite-web to complain and fail to parse.

I know there's carbonapi, but there are some other separate issues there which are currently impeding me from switching to it fully.
Am I doing this wrong? I If so please advise, I'd really like to migrate to carbonserver/zipper!

validate trigram searching code against the globbing code

Would be nice to actually test if the trigram matches are always the same as the matches returned by the globbing code.

We can do this either by running tests against two servers running on the same storage box, or by modifying the code to do the validation itself by answering the query twice.

License?

I'd really like to use parts/all of carbonserver in some of my Graphite related work. However, my client require that code I use/incorporate be licensed and I do not see a license file here.

Would it be possible to add a LICENSE file?

support fetch requests with globs

0.9.13 frontend code apparently uses this, hence we need to support this

A README would be nice

Hey this looks like a sweet project. I had been toying with the idea of writing a go implementation of carbon-cache and this appears to be just that. It'd be nice if your project had some minimal amount of documentation that spelled out the scope and function to help surface it a little bit.

trigram searching code panics if no usable trigrams extractable from queries

trigram.QueryTrigrams() assumes there is at least one. If we don't have any trigrams, then we'll need to run the glob query against the entire list of files. We could also fall back to a regular file-system glob.

Also need to figure out what fixes belong in the trigram code rather than application logic.

Use file-system index to avoid heavy globbing calls

Under normal load, we see that most of the time is spent either is readdir() or syscalls, or in sort.Strings(). A smarter search algorithm could reduce disk access and speed up find requests considerably.

render requests for directories should return 404

instead of returning an empty response, a proper not found error should be sent when requesting a metric which turns out to be a directory

Dumb metric mapping

Previous bugs wanted more complex solutions to moving metric trees, etc. Let's see if we can get software-level symlinks with sets of regexps at file scan time.

use atomic.Value instead of unsafe.Pointer

Have request time metrics

We'd need a log-scale histogram on service time for requests. This should use the same code as carbonzipper.

Talk to carbon-cache

One thing hindering widespread adoption of the zippers stack is the inability to use caching with the server end.

Trigram index should be used only "sometimes"

The file system globbing code can be faster than the trigram search in some cases. The current heuristic I have in mind for deciding which to use is:

if there are no wildcards, or only a single wildcard at the end
use globbing code.
else
use trigram index

start times not handled correctly

when asking for a timerange of data with from/until, start itself is not returned

occasional slice panics in extractTrigrams

2015/05/26 15:30:42 http: panic serving 10.196.70.109:42704: runtime error: slic
e bounds out of range
goroutine 1402 [running]:
net/http.func·011()
/usr/lib64/go/src/net/http/server.go:1130 +0xbb
main.extractTrigrams(0xc20a04eb90, 0x49, 0x0, 0x0, 0x0)
/builddir/build/BUILD/carbonserver-20150423_143116-1_788b2eacdeaf.el6/go
path/src/grobian/carbonserver/main.go:777 +0x16b
main.expandGlobs(0xc20a04eb90, 0x49, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/builddir/build/BUILD/carbonserver-20150423_143116-1_788b2eacdeaf.el6/go
path/src/grobian/carbonserver/main.go:194 +0x910
main.findHandler(0x7f23bd53fb78, 0xc20f1b4780, 0xc20c6e9380)
/builddir/build/BUILD/carbonserver-20150423_143116-1_788b2eacdeaf.el6/gopath/src/grobian/carbonserver/main.go:274 +0x2f1
github.com/dgryski/httputil.func·002(0x7f23bd53fb78, 0xc20f1b4780, 0xc20c6e9380)
/builddir/build/BUILD/carbonserver-20150423_143116-1_788b2eacdeaf.el6/gopath/src/github.com/dgryski/httputil/times.go:26 +0x76
github.com/dgryski/httputil.func·003(0x7f23bd53fb78, 0xc20f1b4780, 0xc20c6e9380)
/builddir/build/BUILD/carbonserver-20150423_143116-1_788b2eacdeaf.el6/gopath/src/github.com/dgryski/httputil/track.go:40 +0xc7
net/http.HandlerFunc.ServeHTTP(0xc20802aef0, 0x7f23bd53fb78, 0xc20f1b4780, 0xc20c6e9380)
/usr/lib64/go/src/net/http/server.go:1265 +0x41
net/http.(_ServeMux).ServeHTTP(0xc20800a780, 0x7f23bd53fb78, 0xc20f1b4780, 0xc20c6e9380)
/usr/lib64/go/src/net/http/server.go:1541 +0x17d
net/http.serverHandler.ServeHTTP(0xc20811c120, 0x7f23bd53fb78, 0xc20f1b4780, 0xc20c6e9380)
/usr/lib64/go/src/net/http/server.go:1703 +0x19a
net/http.(_conn).serve(0xc20bbc6dc0)
/usr/lib64/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
/usr/lib64/go/src/net/http/server.go:1751 +0x35e

Add suport for metric aliasing

I find it necessary to group metrics by a dimension and, bar duplicating metrics, there is no way of aliasing metrics e.g.:

We have the following space:
servers.<server_name>.

Desired, separate space:
databases.<server_name>.
(where this is a subset of the servers namespace)

Obviously this would be a duplication of the servers space which is redundant, it would be much better if it were possible to submit configuration for aliasing:

databases.<server_name> -> servers.<server_name>

making the databases.<server_name> space virtual.

runtime error: invalid memory address or nil pointer dereference

Damian asked me to file a bug here so he'll be able to find it not only in history :)

2016/05/20 10:04:25 http: panic serving <IP>:45879: runtime error: invalid memory address or nil pointer dereference
goroutine 18390 [running]:
net/http.(*conn).serve.func1(0xc827f04400)
    /usr/lib64/go/src/net/http/server.go:1389 +0xc1
panic(0x8600e0, 0xc820010120)
    /usr/lib64/go/src/runtime/panic.go:426 +0x4e9
github.com/grobian/go-whisper.(*Whisper).Close(0x0)
    /builddir/build/BUILD/carbonserver-20160301_125058-1_88ae672fbd00.el6/gopath/src/github.com/grobian/go-whisper/whisper.go:276 +0x18
main.infoHandler(0x7fdce495ae50, 0xc84989b2b0, 0xc83e1b0700)
    /builddir/build/BUILD/carbonserver-20160301_125058-1_88ae672fbd00.el6/gopath/src/github.com/grobian/carbonserver/main.go:549 +0x3f2
github.com/dgryski/httputil.TimeHandler.func1(0x7fdce495ae50, 0xc84989b2b0, 0xc83e1b0700)
    /builddir/build/BUILD/carbonserver-20160301_125058-1_88ae672fbd00.el6/gopath/src/github.com/dgryski/httputil/times.go:26 +0x70
github.com/dgryski/httputil.TrackConnections.func1(0x7fdce495ae50, 0xc84989b2b0, 0xc83e1b0700)
    /builddir/build/BUILD/carbonserver-20160301_125058-1_88ae672fbd00.el6/gopath/src/github.com/dgryski/httputil/track.go:40 +0xc1
net/http.HandlerFunc.ServeHTTP(0xc8201860b0, 0x7fdce495ae50, 0xc84989b2b0, 0xc83e1b0700)
    /usr/lib64/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc82000ab40, 0x7fdce495ae50, 0xc84989b2b0, 0xc83e1b0700)
    /usr/lib64/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc820094200, 0x7fdce495ae50, 0xc84989b2b0, 0xc83e1b0700)
    /usr/lib64/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc827f04400)
    /usr/lib64/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
    /usr/lib64/go/src/net/http/server.go:2137 +0x44e

{documentation} list of args the app takes

Hey, and thank you for all your help!

Im running carbonserver in conjunction with carbonzipper and it would be a great help to me if you colud provide a list of args that the app takes so i can configure it a bit more :)

Sorry im not a dev and i cant interpret the code, if you could provide a list, i can write up some chkconfig-able init scripts for our servers with a few options per env.

Many thanks

Trailing slash for -w (whisper path) breaks file path to wsp files

If you put whisper dir to -w parameter with trailing slash, e.g.
-w="/opt/graphite/storage/whisper/"
instead of
-w="/opt/graphite/storage/whisper"
then carbonserver will not able to work properly - first characher of file name will be replaced with '/':

2014/10/28 19:18:59 failed to open /opt/graphite/storage/whisper//p-storm003_blah_nl/click-count/jvm/thread-states/blocked.wsp: no such file or directory
2014/10/28 19:18:59 failed to open /opt/graphite/storage/whisper//p-storm003_blah_nl/click-count/jvm/thread-states/new.wsp: no such file or directory

Right file name is case above was started on 'ap-storm003...' and not '/p-storm003'