helins / kafka.clj Goto Github PK

View Code? Open in Web Editor NEW

106.0 12.0 11.0 229 KB

Clojure client for Kafka

License: Eclipse Public License 1.0

Clojure 100.00%

clojure kafka client logging broker cqrs

kafka.clj's People

Contributors

Stargazers

Watchers

Forkers

woggle23 alex-dixon bobrowadam halljson cloudafrica lenw callumcodes patrickpichler fonghou hvogeler jacksho

kafka.clj's Issues

dvlopt.kstreams.stream/process-values throws java.lang.ClassCastException

I am getting the following exception when using process-values:

 Unhandled java.lang.ClassCastException
   dvlopt.kafka._interop.java$transformer$reify__886 cannot be cast to
   org.apache.kafka.streams.kstream.ValueTransformerWithKey

                  java.clj: 1772  dvlopt.kafka.-interop.java/value-transformer-with-key-supplier/reify
KStreamTransformValues.java:   36  org.apache.kafka.streams.kstream.internals.KStreamTransformValues/get
InternalTopologyBuilder.java:  238  org.apache.kafka.streams.processor.internals.InternalTopologyBuilder$ProcessorNodeFactory/build
InternalTopologyBuilder.java:  875  org.apache.kafka.streams.processor.internals.InternalTopologyBuilder/build
InternalTopologyBuilder.java:  836  org.apache.kafka.streams.processor.internals.InternalTopologyBuilder/build
InternalTopologyBuilder.java:  819  org.apache.kafka.streams.processor.internals.InternalTopologyBuilder/build
         KafkaStreams.java:  673  org.apache.kafka.streams.KafkaStreams/<init>
         KafkaStreams.java:  628  org.apache.kafka.streams.KafkaStreams/<init>
         KafkaStreams.java:  538  org.apache.kafka.streams.KafkaStreams/<init>
              kstreams.clj:   88  dvlopt.kstreams/app
              kstreams.clj:   36  dvlopt.kstreams/app

It seems that ValueTransformerWithKeySupplier::get should return ValueTransformerWithKey.
So perhaps, value-transformer-with-key-supplier should call value-transformer-with-key instead of transformer?

Supporting TimeWindows.grace

In order to support TimeWindows.grace, would the below option be the one?

  (window grouped-stream
    [5 :seconds]
    {::interval.type :hopping
     ::grace         [1 :minutes] ;; Would this be the preferred option?
     ::advance-by    [3 :seconds]})

Missing dep ?

I get this

user=> (require '[milena.admin :as admin])

CompilerException java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/Serde, compiling:(milena/interop.clj:98:37)

Is there a missing dep that I dont have ?

Is there an API consume from a certain offset of a partition?

join-windows function args error

In ns dvlopt.kstreams.stream ,there are functions join-with-stream left-join-with-stream outer-join-with-stream invoke function join-windows as (K.-interop.java/join-windows interval),ues one arg. Do not match the args of function join-windows defined in ns dvlopt.kafka.-interop.java with two args as [interval options].

Cannot create mock app

Code :

(def mock-app
  (ks.mock/mock-app "KS-high-level-test"
                    topology))

Error :

Syntax error (NoSuchMethodError) compiling at (form-init7002312934630001265.clj:72:3). 'org.apache.kafka.streams.processor.internals.ProcessorTopology org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.build(java.lang.Integer)'

K.in/offsets-for-timestamps throws NPE

The K.in/offsets-for-timestamp throws a NPE when KafkaConsumer.offsetsForTimes returns null for a partition. According to the documentation this is a valid case, describing that there is no such message for a partition.

Accessing the ProcessorContext inside the transformer

Thanks a lot for this library :)

I am setting up a transformer on the kstream.
Inside the transformation function, I would like to issue calls to 'forward'
Can you consider attaching the context itself in -record-from-ctx - to allow interop on the ProcessorContext ( and avoid the need to store it again by providing :init function.

No records printed running the examples

I run the examples code and the consumer does not print out the records I assumed it would.

The producer works, tested with the command line

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

add-global-table throws java.lang.UnsupportedOperationException - clojure.lang.APersistentMap/clear

Hi, first I wanted to say that this is a beautiful piece for software. It seems that lots of thinking was made to provide the right abstractions and idiomatic usage.
I am transforming my project to the new version of the library.
I encountered an exception when adding global table. looks like there is a call to 'clear' method on a persistent structure.

       APersistentMap.java:  298  clojure.lang.APersistentMap/clear
         Materialized.java:  203  org.apache.kafka.streams.kstream.Materialized/withLoggingDisabled
InternalStreamsBuilder.java:  120  org.apache.kafka.streams.kstream.internals.InternalStreamsBuilder/globalTable
       StreamsBuilder.java:  403  org.apache.kafka.streams.StreamsBuilder/globalTable
               builder.clj:  229  dvlopt.kstreams.builder/add-global-table
               builder.clj:  209  dvlopt.kstreams.builder/add-global-table

Have you encounter that?

Waiting until window closes

I have this function, I am not sure where in kafka.clj it should fit:

(defn wait-until-window-closes [^KTable builder]
  (.suppress builder (Suppressed/untilWindowCloses (Suppressed$BufferConfig/unbounded))))

helins / kafka.clj Goto Github PK

kafka.clj's People

Contributors

Stargazers

Watchers

Forkers

kafka.clj's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs