y42 / clj-druid

clj-druid

Clojure library for Druid.

clj-druid eases druid integration providing a client as well as a set of [Schema] (https://github.com/Prismatic/schema) to validate queries. Current version is up to date with latest query API from official docs.

Installing

Add the following to your Leiningen project.clj:

Usage

Client

Connect to a druid cluster through Zookeeper, this method supports auto detection and update of available brokers for easy HA/load balancing

(use 'clj-druid.client)
(connect {:zk {:host "127.0.0.1:2181,my-other-zk-host:2181" ; can contain multiple hosts separated by commas
               :discovery-path "/druid/discovery"
               :node-type "druid:broker"}})

you can also connect by supplying a vector of hosts, useful for dev, local testing

(use 'clj-druid.client)
(connect {:hosts ["http://127.0.0.1:8083/druid/v2"]})

Querying

Issue druid queries supplying

• a load balance strategy [fixed random]
• a queryType
• a query in the following form

(def q
  {:queryType :timeseries
   :dataSource "sample_datasource"
   :granularity :day
   :filter {
     :type :and
     :fields [
       {:type :selector :dimension "sample_dimension1" :value "sample_value1"}
       {:type :or
         :fields [
           {:type :selector :dimension "sample_dimension2" :value "sample_value2"}
           {:type :selector :dimension "sample_dimension3" :value "sample_value3"}]}]}
   :aggregations [
     {:type :longSum :name "sample_name1", :fieldName "sample_fieldName1"}
     {:type :doubleSum :name "sample_name2", :fieldName "sample_fieldName2"}]
   :postAggregations [
     {:type :arithmetic
      :name "sample_divide"
      :fn "/"
      :fields [{:type :fieldAccess :name "sample_name1" :fieldName "sample_fieldName1"}
               {:type :fieldAccess :name "sample_name2" :fieldName "sample_fieldName2"}]}]

   :intervals ["2012-01-01T00:00:00.000/2012-01-03T00:00:00.000"]})
   (let [client (connect {:zk {:host "127.0.0.1:2181"
                          :discovery-path "/druid/discovery"
                          :node-type "druid:broker"}})]
     (query client random (:queryType q) q)
     (close client))

To run a query without runtime validation of the query against a schema (using the q from above:

   (let [client (connect {:zk {:host "127.0.0.1:2181"
                          :discovery-path "/druid/discovery"
                          :node-type "druid:broker"}})]
     (execute client random q)
     (close client))
;; if you want to validate
   (let [client (connect {:zk {:host "127.0.0.1:2181"
                          :discovery-path "/druid/discovery"
                          :node-type "druid:broker"}})]
     (s/with-fn-validation (execute client random q)) ;; s is plumatic/schema
     (close client))

Example

An example using compojure-api to spawn a full druid API is located in the examples folder

git clone https://github.com/y42/clj-druid
cd clj-druid/examples
lein ring server

Have Fun!

License

Copyright © 2015 Guillaume Buisson

Distributed under the Eclipse Public License, the same as Clojure.

clj-druid's Issues

Problems in README.md

I just started looking at this, since I need to access Druid from Clojure.

I noticed a couple of problems in the top-level README.md file:

1. random isn't defined, as in (query client random (:queryType q) q). Maybe you meant randomize?
2. execute isn't defined, as in (execute client random q)

Also, I'd appreciate a quick comment confirming whether this package should work (since I'm having some other exceptions). I'm using Clojure 1.8 (admittedly old).
Thanks,
Dan

Wrong Schema

Apparently

"aggregator":{"type":"filtered","aggregator":
                          {"type":"longSum"
                          ,"name":"qualityRows"
                          ,"fieldName":"count"}
                    ,"filter":
                          {"type":"not"
                          ,"field":{"type":"selector","dimension":"quality","value":null}}

Is ok for druid but doesn't pass the cli-druid aggregation schema (clj-druid.schemas.aggregation). Or more precisely it try to use aggregator schema instead of filtered aggregator schema

cli-druid does not support all simple-granularities

From http://druid.io/docs/0.9.2/querying/granularities.html,
druid supports

all, none, second, minute, fifteen_minute, thirty_minute, hour, day, week, month, quarter and year.

cli-druid is missing

week, month, quarter and year

Update clj-http dependency

clj-druid uses version 2.0.0 of clj-http to send request to Druid. My application uses clj-druid and clj-http. I cannot use the latest version of clj-http which is 3.4.1.

Migration example

I'm currently using cli-druid version 0.2.12, which lets me connect by calling

(let [zk-client (zk/connect "zk1,zk2,zk3")]
   (druid\zk-watch-node-list zk-client "/druid/discovery/druid:broker"))

Which suits me quite well, since I'm using the zookeeper client for other stuff as well.

How would this be done in 0.2.16?

Upgrade example project

To latest compojure-api version

Use latest clj-druid version on example project

use 0.2.12

Wrong node-type in connect-with-zk docs

Doc states:

(connect {:zk {:host "127.0.0.1:2181,my-other-zk-host:2181" ; can contain multiple hosts separated by commas
               :discovery-path "/druid/discovery"
               :node-type "broker"}})

At least in my environment, this should be

(connect {:zk {:host "127.0.0.1:2181,my-other-zk-host:2181" ; can contain multiple hosts separated by commas
               :discovery-path "/druid/discovery"
               :node-type "druid:broker"}})

Min/Max Aggregators no longer supported, use doubleMin/doubleMax instead

As indicated in the 0.8.3 release notes, min/max aggregators have been removed in favor of doubleMin, doubleMax, longMin, and longMax aggregators.

Druid integration test with travis

The service may be added with docker-compose

Don't validate queries at runtime

Queries shall be validated only for :test profile

Wrong spec for javascriptAggregator/fnReset

It's spec'ed as [s/Str] but should be s/Str

example project is broken

Sounds like a dependency is missing:

Caused by: java.io.FileNotFoundException: Could not locate pjstadig/util__init.class or pjstadig/util.clj on classpath:

Make it possible to bypass validations

Even though I like the query-validations, there are times when I need to do stuff that the schemas are
not allowing. Example at hand right now is working with thetaSketches, http://druid.io/docs/latest/development/extensions-core/datasketches-aggregators.html
which is not currently supported by the schema. As a work around I do this

(defn execute*
  "Issue a druid query"
  ([client balance-strategy druid-query http-params]
   (execute* client balance-strategy druid-query http-params false))
  ([client balance-strategy druid-query http-params non-strict?]
   (let [nodes (druid/nodes client)]
     (when-not non-strict?
       (v/validate druid-query (:queryType druid-query)))
     (when (empty? nodes)
       (throw (Exception.
               "No druid node available for query")))
     (http/post (balance-strategy nodes)
                (merge {:body (json/encode druid-query)
                        :as :text
                        :content-type :json}
                       http-params)))))

in my code.

Would you want this in cli-druid? if not, please just close this issue.