someben / skream Goto Github PK

The example below computes quantiles for randomly shuffled numbers in the range 0..1000. The 10% and 90% quantiles don't look good. By contrast the this Java library https://github.com/mayconbordin/streaminer gives reasonably good results (using the Frugal2U class).

(ns ... 
...
 (:require 
            [skream.core :as sk]))


  (def sk
    (-> (sk/create-skream)
        (sk/track-quantile-ish 0.10)
        (sk/track-quantile-ish 0.25)
        (sk/track-quantile-ish 0.50)
        (sk/track-quantile-ish 0.90)))

  (let
      [sk  (apply (partial sk/add-num sk ) (shuffle (range 1000)))]
    (map #(get-in sk [[:quantile %] :quantile] )
         [0.10 0.25 0.50 0.90]))

==> (252.27943 252.27943 496.22464 750.6505)

Refactor the mutual information code that works across two skreams into a more abstract "skream bundle" or set of time-series structure. Will be good for sampling a number of time-series at once, later.

Implement a persistent store for session information, so Skreams survive a bounce of the REST server. Maybe into Redis?

Find a dataset with an authoritative answer.

Any plans to put this on clojars? or is it already there?

thanks

Check whether memoizing the get-new-math functions in the Skreams themselves (metadata again?) helps performance.

Add a simple interface to that allows piped-in streams of numbers on the command-line.

Add support for aliasing nested stats in sub-maps at the top level. So you could alias ":median" at the top level and goes down into into the :qs & :ns quantile sub-map.

Move the :quantile top-level key into the :ns, :qs sub-map to be consistent with the distinct value (:dv) structure.