robyoung / go-whisper

A Go port of Graphite's Whisper timeseries database

Go Whisper

Go Whisper is a Go implementation of the Whisper database, which is part of the Graphite Project.

To create a new whisper database you must define it's retention levels (see: storage schemas), aggregation method and the xFilesFactor. The xFilesFactor specifies the fraction of data points in a propagation interval that must have known values for a propagation to occur.

Examples

Create a new whisper database in "/tmp/test.wsp" with two retention levels (1 second for 1 day and 1 hour for 5 weeks), it will sum values when propagating them to the next retention level, and it requires half the values of the first retention level to be set before they are propagated.

retentions, err := whisper.ParseRetentionDefs("1s:1d,1h:5w")
if err == nil {
  wsp, err := whisper.Create("/tmp/test.wsp", retentions, whisper.Sum, 0.5)
}

Alternatively you can open an existing whisper database.

wsp, err := whisper.Open("/tmp/test.wsp")

Once you have a whisper database you can set values at given time points. This sets the time point 1 hour ago to 12345.678.

offset, _ := time.ParseDuration("-1h")
wsp.Update(12345.678, int(time.Now().Add(offset).Unix()))

And you can retrieve time series from it. This example fetches a time series for the last 1 hour and then iterates through it's points.

offset, _ := time.ParseDuration("-1h")
series, err := wsp.Fetch(int(time.Now().Add(offset).Unix()), int(time.Now().Unix()))
if err != nil {
  // handle
}
for _, point := range series.Points() {
  fmt.Println(point.Time, point.Value)
}

Thread Safety

This implementation is not thread safe. Writing to a database concurrently will cause bad things to happen. It is up to the user to manage this in their application as they need to.

Licence

Go Whisper is licenced under a BSD Licence.

go-whisper's Issues

Return NaN instead of 0 for nil values?

Perhaps have it configurable? That way the user of the library can make their own decision on what to do with missing data.

(We have a patch that does this, but we would clean it up if you wanted it as a pull request.)

Expose TimeSeries members?

We're in the process of prototyping a server that serves whisper data (from a carbon store typically) in the same way the carbon web frontend would do. For that purpose, we need to access the from and untiltime, plus step, as well as that it seemed handy to be able to directly access the values array.

Would a patch like this be acceptable to you?

--- a/whisper.go
+++ b/whisper.go
@@ -684,6 +684,22 @@ type TimeSeries struct {
        values    []float64
 }

+func (ts *TimeSeries) FromTime() int {
+       return ts.fromTime
+}
+
+func (ts *TimeSeries) UntilTime() int {
+       return ts.untilTime
+}
+
+func (ts *TimeSeries) Step() int {
+       return ts.step
+}
+
+func (ts *TimeSeries) Values() []float64 {
+       return ts.values
+}
+
 func (ts *TimeSeries) Points() []TimeSeriesPoint {
        points := make([]TimeSeriesPoint, len(ts.values))
        for i, value := range ts.values {

UpdateMany saves not last point if many points has identical timestamp

Python code from whisper.py

points.sort(key=lambda p: p[0], reverse=True)

not equals

sort.Sort(timeSeriesPointsNewestFirst{points})

I tested and fixed this bug here lomik@e1873a0 . But my fix incompatible with golang older 1.2

int/int64 conversions

Heya, I'm just beginning to use go-whisper, and I'm wondering if it would make sense to switch a lot of function's time parameters over to using int64?

Time.Unix seems to drive most of the code I'm writing (just like it's in most of the examples on the go-whisper readme!)... and Time.Unix returns an int64. This means my code is littered with int(t.Unix()) wrappers (and also the readme examples don't compile without that fix). Switching Update and Fetch to just use int64 themselves would seem to reduce friction.

Are there any reasons not to prefer int64? I guess someone using a different concept of "time" might find it unnecessary, that seems like the exception rather than the rule, and still doesn't seem like that case would be unduly burdened by int64.

morning

"To maintain a stable job and give the best of me, I must continue to train and provide my best effort to succeed as a very responsible professional capable of solving big and small problems."

Storage Schemas link in Readme is broken

http://graphite.readthedocs.org/en/1.0/config-carbon.html#storage-schemas-conf is a dead link.