clj-commons / byte-streams Goto Github PK
View Code? Open in Web Editor NEWA Rosetta stone for JVM byte representations
byte-streams's People
Contributors
Stargazers
Watchers
Forkers
danburkert aysylu sleyzerzon hatnik benmoss sourceops miners bsubotic mccraigmccraig clojure-society ajchemist bluekezza tony824 snakevash joelittlejohn gsnewmark danjohansson miikka jafingerhut clojure-land kosugin markbastian jaju acgollapalli reducecombine victorb piotr-yuxuan skynet-gh chandojo valerauko bevuta murtenpnisttebyte-streams's Issues
documentation link broken
def-transfer on seq-of
Would it be possible to extend the library to allow defining transfers from (seq-of MyBytesSource) -> OtherSink? Currently when I try this I get a java.lang.RuntimeException: Unable to resolve symbol: seq-of in this context.
Unable to load byte-streams ns more than once
Here's an example repl session (from a fresh clone of byte-streams, with no other code):
$ lein repl
nREPL server started on port 46803 on host 127.0.0.1 - nrepl://127.0.0.1:46803
REPL-y 0.3.7, nREPL 0.2.12
Clojure 1.8.0
OpenJDK 64-Bit Server VM 1.8.0_121-b13
Reflection warning, /home/jeaye/projects/lets-bet/byte-streams/target/9ebf76f7ebb52177860c3ac205bfc4727d674f41-init.clj:1:1589 - reference to field ns can't be resolved.
Reflection warning, /home/jeaye/projects/lets-bet/byte-streams/target/9ebf76f7ebb52177860c3ac205bfc4727d674f41-init.clj:1:3847 - reference to field name can't be resolved.
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
user=> (use 'byte-streams :reload-all)
WARNING: vector-of already refers to: #'clojure.core/vector-of in namespace: user, being replaced by: #'byte-streams/vector-of
nil
user=> (use 'byte-streams :reload-all)
user=> java.lang.IllegalArgumentException: No implementation of method: :assoc-conversion of protocol: #'byte-streams.graph/IConversionGraph found for class: byte_streams.graph.ConversionGraph
clojure.lang.Compiler$CompilerException: java.lang.IllegalArgumentException: No implementation of method: :assoc-conversion of protocol: #'byte-streams.graph/IConversionGraph found for class: byte_streams.graph.ConversionGraph, compiling:(byte_streams.clj:334:1)
user=> *e
#error {
:cause "No implementation of method: :assoc-conversion of protocol: #'byte-streams.graph/IConversionGraph found for class: byte_streams.graph.ConversionGraph"
:via
[{:type clojure.lang.Compiler$CompilerException
:message "java.lang.IllegalArgumentException: No implementation of method: :assoc-conversion of protocol: #'byte-streams.graph/IConversionGraph found for class: byte_streams.graph.ConversionGraph, compiling:(byte_streams.clj:334:1)"
:at [clojure.lang.Compiler load "Compiler.java" 7391]}
{:type java.lang.IllegalArgumentException
:message "No implementation of method: :assoc-conversion of protocol: #'byte-streams.graph/IConversionGraph found for class: byte_streams.graph.ConversionGraph"
:at [clojure.core$_cache_protocol_fn invokeStatic "core_deftype.clj" 568]}]
:trace
[[clojure.core$_cache_protocol_fn invokeStatic "core_deftype.clj" 568]
[clojure.core$_cache_protocol_fn invoke "core_deftype.clj" 560]
[byte_streams.graph$eval21215$fn__21240$G__21196__21253 invoke "graph.clj" 91]
[clojure.lang.AFn applyToHelper "AFn.java" 171]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.lang.Atom swap "Atom.java" 79]
[clojure.core$swap_BANG_ invokeStatic "core.clj" 2263]
[clojure.core$swap_BANG_ doInvoke "core.clj" 2253]
[clojure.lang.RestFn invoke "RestFn.java" 529]
[byte_streams$eval22066 invokeStatic "byte_streams.clj" 336]
[byte_streams$eval22066 invoke "byte_streams.clj" 336]
[clojure.lang.Compiler eval "Compiler.java" 6927]
[clojure.lang.Compiler load "Compiler.java" 7379]
[clojure.lang.RT loadResourceScript "RT.java" 372]
[clojure.lang.RT loadResourceScript "RT.java" 363]
[clojure.lang.RT load "RT.java" 453]
[clojure.lang.RT load "RT.java" 419]
[clojure.core$load$fn__5677 invoke "core.clj" 5893]
[clojure.core$load invokeStatic "core.clj" 5892]
[clojure.core$load doInvoke "core.clj" 5876]
[clojure.lang.RestFn invoke "RestFn.java" 408]
[clojure.core$load_one invokeStatic "core.clj" 5697]
[clojure.core$load_all$fn__5618$fn__5621 invoke "core.clj" 5713]
[clojure.core$load_all$fn__5618 invoke "core.clj" 5713]
[clojure.lang.AFn call "AFn.java" 18]
[clojure.lang.LockingTransaction run "LockingTransaction.java" 273]
[clojure.lang.LockingTransaction runInTransaction "LockingTransaction.java" 229]
[clojure.core$load_all invokeStatic "core.clj" 5711]
[clojure.core$load_all invoke "core.clj" 5705]
[clojure.core$load_lib$fn__5626 invoke "core.clj" 5737]
[clojure.core$load_lib invokeStatic "core.clj" 5736]
[clojure.core$load_lib doInvoke "core.clj" 5717]
[clojure.lang.RestFn applyTo "RestFn.java" 142]
[clojure.core$apply invokeStatic "core.clj" 648]
[clojure.core$load_libs invokeStatic "core.clj" 5774]
[clojure.core$load_libs doInvoke "core.clj" 5758]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.core$apply invokeStatic "core.clj" 650]
[clojure.core$use invokeStatic "core.clj" 5860]
[clojure.core$use doInvoke "core.clj" 5860]
[clojure.lang.RestFn invoke "RestFn.java" 421]
[user$eval18373 invokeStatic "9ebf76f7ebb52177860c3ac205bfc4727d674f41-init.clj" 1]
[user$eval18373 invoke "9ebf76f7ebb52177860c3ac205bfc4727d674f41-init.clj" 1]
[clojure.lang.Compiler eval "Compiler.java" 6927]
[clojure.lang.Compiler eval "Compiler.java" 6890]
[clojure.core$eval invokeStatic "core.clj" 3105]
[clojure.core$eval invoke "core.clj" 3101]
[clojure.main$repl$read_eval_print__7408$fn__7411 invoke "main.clj" 240]
[clojure.main$repl$read_eval_print__7408 invoke "main.clj" 240]
[clojure.main$repl$fn__7417 invoke "main.clj" 258]
[clojure.main$repl invokeStatic "main.clj" 258]
[clojure.main$repl doInvoke "main.clj" 174]
[clojure.lang.RestFn invoke "RestFn.java" 1523]
[clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__941 invoke "interruptible_eval.clj" 87]
[clojure.lang.AFn applyToHelper "AFn.java" 152]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 646]
[clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1881]
[clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1881]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[clojure.tools.nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 85]
[clojure.tools.nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 55]
[clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__986$fn__989 invoke "interruptible_eval.clj" 222]
[clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__981 invoke "interruptible_eval.clj" 190]
[clojure.lang.AFn run "AFn.java" 22]
[java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1142]
[java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 617]
[java.lang.Thread run "Thread.java" 745]]}
This is breaking my development tooling, far down the dependency hole, but I've traced it back to byte-streams.
Can't convert stream of byte arrays to seq of byte arrays
When I try to convert a stream of byte arrays to a seq of byte arrays, it just returns back the original stream without doing any conversion.
user=> (require '[manifold.stream :as ms])
user=> (require '[byte-streams :as bs])
user=> (def content (doto (ms/stream) (ms/put! (byte-array 5)) (ms/close!)))
#'user/content
user=> (bs/convert content (bs/seq-of bytes)) ; doesn't work
#<manifold.stream.default.Stream@942027b>
user=> (bs/convert content (bs/seq-of String)) ; also doesn't work
#<manifold.stream.default.Stream@942027b>
user=> (bs/convert content (bs/seq-of java.nio.ByteBuffer)) ; this works
(#<java.nio.HeapByteBuffer@5ff2cd23 java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]>)"Malformed byte-stream input to CharsetDecoder" for print-bytes
I have a HeapByteBuffer from data.fressian (a map with keywords, longs, and instants). print-bytes gives the following error:
IllegalArgumentException Malformed byte-stream input to CharsetDecoder.
byte-streams.char-sequence/parse-result (char_sequence.clj:15)
byte-streams.char-sequence/decode (char_sequence.clj:22)
byte-streams.char-sequence/lazy-char-buffer-sequence/fn--902 (char_sequence.clj:55)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:60)
clojure.lang.RT.seq (RT.java:484)
clojure.core/seq (core.clj:133)
byte-streams.char-sequence/decode-byte-source/reify--910 (char_sequence.clj:96)
byte-streams/eval2365/fn--2366 (byte_streams.clj:628)
clojure.lang.AFunction$1.doInvoke (AFunction.java:29)
byte-streams/fn--2139/fn--2157/fn--2158 (byte_streams.clj:296)
clj-tuple.Tuple2 (clj_tuple.clj:533)
clojure.core/reduce (core.clj:6177)
byte-streams/fn--2139/fn--2157 (byte_streams.clj:290)
byte-streams/convert (byte_streams.clj:356)
byte-streams/print-bytes (byte_streams.clj:790)
com.squareup.knecht.io/eval2623 (NO_SOURCE_FILE:1)
clojure.lang.Compiler.eval (Compiler.java:6619)
clojure.lang.Compiler.eval (Compiler.java:6582)
clojure.core/eval (core.clj:2852)
clojure.main/repl/read-eval-print--6588/fn--6591 (main.clj:259)
clojure.main/repl/read-eval-print--6588 (main.clj:259)
clojure.main/repl/fn--6597 (main.clj:277)
clojure.main/repl (main.clj:277)
clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn--536 (interruptible_eval.clj:57)
clojure.core/apply (core.clj:617)
clojure.core/with-bindings* (core.clj:1788)
clojure.tools.nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:42)
clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn--577/fn--580 (interruptible_eval.clj:170)
clojure.core/comp/fn--4154 (core.clj:2330)
clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn--570 (interruptible_eval.clj:137)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
java.lang.Thread.run (Thread.java:744)
Is this expected? I had hoped print-bytes would just drop stuff it can't understand in the decoded part of the output.
Deprecation of `byte-streams` namespace is undocumented.
A bug found in Aleph's walkthrough examples was caused by the deprecation of byte-streams namespace in favor of clj-commons.byte-streams.
The byte-streams Readme makes no mention of this deprecation, and all examples are shown still using the byte-streams namespace.
Single-segment and clj-commons namespaces do not share conversion graph
It's not just enough to copy the fns, because they use differing data. This can lead to situations where some code is using the old ns, and some the new, and they don't agree on the list of conversions.
Mentioned in #63 (comment)
cannot compile due to No such var: p/min
I'm not sure why this is happening but I'm getting this when using the CI pipeline on Gitlab.
the versions are byte-streams 0.2.0 and primitive-math 0.1.4
Exception in thread "main" java.lang.RuntimeException: No such var: p/min, compiling:(byte_streams/pushback_stream.clj:37:38)
at clojure.lang.Compiler.analyze(Compiler.java:6688)
at clojure.lang.Compiler.analyze(Compiler.java:6625)
at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3766)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6870)
at clojure.lang.Compiler.analyze(Compiler.java:6669)
at clojure.lang.Compiler.analyze(Compiler.java:6625)
at clojure.lang.Compiler$HostExpr$Parser.parse(Compiler.java:1009)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6868)
at clojure.lang.Compiler.analyze(Compiler.java:6669)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6856)
at clojure.lang.Compiler.analyze(Compiler.java:6669)
compile error in byte-streams when using aleph or gloss
I've been trying to update the dependencies in one of my libraries and a compile error is now being thrown from the byte-streams library.
I tested by removing the requires from the namespace and it seems to be happening when i'm requiring aleph.tcp, gloss.core, or gloss.io
any thoughts?
here is the error:
CompilerException java.lang.IllegalAccessError: vector does not exist, compiling:(byte_streams/graph.clj:1:1)current dependencies:
[aleph "0.4.0"]
[manifold "0.1.0"]
[gloss "0.2.5"]here is my namespace:
(ns mpd-clj.utils
"utility functions for connecting to the mpd server"
(:require [clojure.string :as str]
[manifold.deferred :as d]
[manifold.stream :as s]
[aleph.tcp :as tcp]
[gloss.core :as gloss]
[gloss.io :as io]))system info:
CIDER 0.10.0snapshot (package: 20150801.627) (Java 1.8.0_51, Clojure 1.7.0, nREPL 0.2.7)Lein javac options break on JDKs >= 12
This line in project.clj is causing the problem: :javac-options ["-target" "1.6" "-source" "1.6"]
It's not a blocker for those consuming by jar, it's only if you want to fire up a REPL and poke around the code.
❯ jenv shell 12
~/Documents/Code/byte-streams |HEAD✓
❯ lein do clean, compile, repl
Compiling 3 source files to /Users/matthew/Documents/Code/byte-streams/target/classes
warning: [options] bootstrap class path not set in conjunction with -source 6
error: Source option 6 is no longer supported. Use 7 or later.
error: Target option 6 is no longer supported. Use 7 or later.
Compilation of Java sources(lein javac) failed.
lein uberjar fails with type hints in graph.clj
I am using byte-streams library for clj_fdb. When I try to create a uberjar on the master I have the below error. I tried with the latest alpha release and also it throws the same error. I think it's due to some type hinting at graph.clj . I cloned the repo and removed the type hints and did lein install. Then I used the local version which works fine during uberjar creation.
The strange part was that it used to work fine except when the relevant file was moved not to another folder. I figured out the bad commit at vedang/clj_fdb@459071e where I move the namespace to another internal folder. When I revert the folder organization it works fine. I don't why moving it inside to internal folder causes the issue with type hinting while moving it one above works fine. Maybe I am missing something here.
Another caveat is that compiling the file as a separate file works fine and starting the repl also works fine. This happens only when I have the namespace inside the folder while moving it out works fine.
Exception
➜ clj_fdb git:(upstream/master) lein uberjar
Compiling clj-fdb.core
Compiling clj-fdb.FDB
Compiling clj-fdb.internal.byte-conversions
nil
Exception in thread "main" java.lang.ExceptionInInitializerError, compiling:(/private/var/folders/2b/mhgtnnpx4z943t4cc9yvw4qw0000gn/T/form-init7346111847608064189.clj:1:125)
at clojure.lang.Compiler.load(Compiler.java:7526)
at clojure.lang.Compiler.loadFile(Compiler.java:7452)
at clojure.main$load_script.invokeStatic(main.clj:278)
at clojure.main$init_opt.invokeStatic(main.clj:280)
at clojure.main$init_opt.invoke(main.clj:280)
at clojure.main$initialize.invokeStatic(main.clj:311)
at clojure.main$null_opt.invokeStatic(main.clj:345)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.invokeStatic(main.clj:424)
at clojure.main$main.doInvoke(main.clj:387)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.main.main(main.java:37)
Caused by: java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at clojure.lang.RT.classForName(RT.java:2204)
at clojure.lang.RT.classForName(RT.java:2213)
at clojure.lang.RT.loadClassForName(RT.java:2232)
at clojure.lang.RT.load(RT.java:450)
at clojure.lang.RT.load(RT.java:426)
at clojure.core$load$fn__6548.invoke(core.clj:6046)
at clojure.core$load.invokeStatic(core.clj:6045)
at clojure.core$load.doInvoke(core.clj:6029)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5848)
at clojure.core$compile$fn__6553.invoke(core.clj:6056)
at clojure.core$compile.invokeStatic(core.clj:6056)
at clojure.core$compile.invoke(core.clj:6048)
at user$eval164$fn__173.invoke(form-init7346111847608064189.clj:1)
at user$eval164.invokeStatic(form-init7346111847608064189.clj:1)
at user$eval164.invoke(form-init7346111847608064189.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:7062)
at clojure.lang.Compiler.eval(Compiler.java:7052)
at clojure.lang.Compiler.eval(Compiler.java:7052)
at clojure.lang.Compiler.load(Compiler.java:7514)
... 12 more
Caused by: java.lang.ClassCastException: byte_streams.graph.Type cannot be cast to byte_streams.graph.Type
at byte_streams.graph.ConversionGraph.assoc_conversion(graph.clj:122)
at byte_streams.graph$fn__2606$G__2568__2612.invoke(graph.clj:91)
at byte_streams.graph$fn__2606$G__2567__2619.invoke(graph.clj:91)
at clojure.lang.AFn.applyToHelper(AFn.java:171)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Atom.swap(Atom.java:79)
at clojure.core$swap_BANG_.invokeStatic(core.clj:2347)
at clojure.core$swap_BANG_.doInvoke(core.clj:2337)
at clojure.lang.RestFn.invoke(RestFn.java:529)
at clj_fdb.internal.byte_conversions$fn__3638.invokeStatic(byte_conversions.clj:10)
at clj_fdb.internal.byte_conversions$fn__3638.invoke(byte_conversions.clj:10)
at clj_fdb.internal.byte_conversions__init.load(Unknown Source)
at clj_fdb.internal.byte_conversions__init.<clinit>(Unknown Source)
... 34 more
Compilation failed: Subprocess failedbyte-streams type-hint change that makes uberjar work
diff --git a/src/byte_streams/graph.clj b/src/byte_streams/graph.clj
index fc62f1d..3c94b41 100644
--- a/src/byte_streams/graph.clj
+++ b/src/byte_streams/graph.clj
@@ -119,10 +119,10 @@
(assoc-conversion [_ src dst f cost]
(let [m' (assoc-in m [src dst] (Conversion. f cost))
m' (if (and
- (nil? (.wrapper ^Type src))
- (nil? (.wrapper ^Type dst)))
- (let [src (.type ^Type src)
- dst (.type ^Type dst)]
+ (nil? (.wrapper src))
+ (nil? (.wrapper dst)))
+ (let [src (.type src)
+ dst (.type dst)]
(-> m'
(assoc-in [(Type. 'seq src) (Type. 'seq dst)]
(Conversion. (fn [x options] (map #(f % options) x)) cost))Thanks a lot for the library.
print-bytes calls .release ByteBuf?
(let [buf (Unpooled/buffer 4)]
(.writeInt buf 42)
(println "Ref: " (.refCnt buf))
(bs/print-bytes buf)
(println "Ref: " (.refCnt buf)))
=>
Ref: 1
00 00 00 2A
Ref: 0
This may be deliberate, but I really expected the refCnt to remain unchanged.
If nothing else, it seems worth documenting.
Large file to byte arrays cause insufficient memory (error=12)
to recur this error: falloate -l 2048M /tmp/tmp.bin, then in repl, (def v (bs/to-byte-arrays (clojure.java.io/file "/tmp/tmp.bin") {:chunk-size 64})), the insufficient memory reports after running some codes consuming v, for example (count v)
defining a macro to define a conversion of type1 -> (seq-of type2) fails
dca288c introduced a bug that causes defining a conversion of type1 -> (seq-of type2) in a macro to fail, for example:
(ns example
(:require [byte-streams :as bs]))
(defmacro my-macro
[type]
`(bs/def-conversion [java.io.InputStream (bs/seq-of ~type)]
[_]
nil))
(my-macro java.util.Map)It fails with the following stack trace:
Exception in thread "main" java.lang.ClassCastException: clojure.lang.Cons cannot be cast to clojure.lang.Symbol
at clojure.core$ns_resolve.invoke(core.clj:3954)
at clojure.core$ns_resolve.invoke(core.clj:3951)
at clojure.core$resolve.invoke(core.clj:3960)
at byte_streams$abstract_type_descriptor.invoke(byte_streams.clj:70)
at byte_streams$def_conversion.doInvoke(byte_streams.clj:83)
at clojure.lang.RestFn.invoke(RestFn.java:497)
at clojure.lang.Var.invoke(Var.java:431)
at clojure.lang.AFn.applyToHelper(AFn.java:178)
at clojure.lang.Var.applyTo(Var.java:532)
at clojure.lang.Compiler.macroexpand1(Compiler.java:6468)
at clojure.lang.Compiler.macroexpand(Compiler.java:6529)
at clojure.lang.Compiler.macroexpand(Compiler.java:6531)
at clojure.lang.Compiler.eval(Compiler.java:6603)
at clojure.lang.Compiler.load(Compiler.java:7064)
at clojure.lang.RT.loadResourceScript(RT.java:370)
at clojure.lang.RT.loadResourceScript(RT.java:361)
at clojure.lang.RT.load(RT.java:440)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at user$eval5$fn__12.invoke(NO_SOURCE_FILE:1)
at user$eval5.invoke(NO_SOURCE_FILE:1)
at clojure.lang.Compiler.eval(Compiler.java:6619)
at clojure.lang.Compiler.eval(Compiler.java:6609)
at clojure.lang.Compiler.eval(Compiler.java:6582)
at clojure.core$eval.invoke(core.clj:2852)
at clojure.main$eval_opt.invoke(main.clj:308)
at clojure.main$initialize.invoke(main.clj:327)
at clojure.main$null_opt.invoke(main.clj:362)
at clojure.main$main.doInvoke(main.clj:440)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:419)
at clojure.lang.AFn.applyToHelper(AFn.java:163)
at clojure.lang.Var.applyTo(Var.java:532)
at clojure.main.main(main.java:37)
This happens because the (seq-of java.util.Map) is a clojure.lang.Cons, which causes it to fails the list? check in seq-of?. Changing list? to seq? fixes the issue, but I don't know if that will introduce some other issues. It doesn't seem to fail if it is defined normally outside of the macro. I can't figure out where the Cons is being introduced, or why it only gets introduced as part of a macro evaluation.
`:tag` metadata can be wrong
:tag metadata should denote classes or primitives - not arbitrary objects.
However some of the :tags that byte-streams emits can denote byte_streams.graph.Type objects:
(in-ns 'byte-streams)
;; inspect the def-transfer defined at https://github.com/clj-commons/byte-streams/blob/4066d637300c144d37e888c23de1a9c3ae19829a/src/byte_streams.clj#L610 :
(-> '(def-transfer [ReadableByteChannel File]
[channel file {:keys [chunk-size] :or {chunk-size (int 1e7)} :as options}]
(let [^FileChannel fc (convert file WritableByteChannel options)]
(try
(loop [idx 0]
(let [n (.transferFrom fc channel idx chunk-size)]
(when (pos? n)
(recur (+ idx n)))))
(finally
(.force fc true)
(.close fc)))))
macroexpand-1
last
second
second
meta
println)
;; {:tag #object[byte_streams.graph.Type 0x62e58f04 java.io.File]}...this is accurately reported by running lein eastwood.
Fixing this might also solve other tag-related issues that I can see in the tracker.
Hope it helps!
-V
Converting from PipedInputStream to seq of byte arrays goes through Strings!
If I do (conversion-path java.io.PipedInputStream (seq-of byte-array)), then I get ([#'byte-streams/ByteSource java.lang.CharSequence] [java.lang.CharSequence java.lang.String] [java.lang.String [B] [[B java.nio.DirectByteBuffer] [java.nio.ByteBuffer (seq-of java.nio.ByteBuffer)] [(seq-of java.nio.ByteBuffer) (seq-of [B)]).
I'd expect to see ([java.io.InputStream java.nio.channels.ReadableByteChannel] [java.nio.channels.ReadableByteChannel (seq-of java.nio.ByteBuffer)] [(seq-of java.nio.ByteBuffer) (seq-of [B)]), which I can get by first converting to (seq-of java.nio.ByteBuffer), and then converting to (seq-of byte-array).
This is especially bad because there's this exception with the current behavior: IllegalArgumentException Malformed byte-stream input to CharsetDecoder. byte-streams.char-sequence/parse-result (char_sequence.clj:15)
Possible conversions returns impossible conversions
Hello there!
I'm writing a generative test where I'd like to create objects that can be converted to a ByteArrayInputStream via byte-streams/convert and have come across something unexpected.
When I run the following code to convert a ByteArrayInputStream into each of the possible conversions I get a lot of IllegalArgumentExceptions for seqs, vectors and streams of types.
(doseq [c (byte-streams/possible-conversions java.io.ByteArrayInputStream)]
(try
(byte-streams/convert (java.io.ByteArrayInputStream. (.getBytes "a")) c)
(catch IllegalArgumentException exception
(println (.getMessage exception)))))Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.nio.ByteBuffer)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.io.InputStream)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.lang.String)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.nio.ByteBuffer)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.io.InputStream)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.lang.CharSequence)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.nio.channels.ReadableByteChannel)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.io.InputStream)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of [B)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of io.netty.buffer.ByteBuf)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.nio.ByteBuffer)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of [B)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.lang.CharSequence)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of [B)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.lang.String)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.nio.channels.ReadableByteChannel)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of [B)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.io.Reader)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of [B)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of [B)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.lang.CharSequence)
Don't know how to convert class java.io.ByteArrayInputStream into (vector-of java.nio.ByteBuffer)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.nio.channels.ReadableByteChannel)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.io.InputStream)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of [B)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of io.netty.buffer.ByteBuf)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.lang.CharSequence)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.io.InputStream)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.nio.ByteBuffer)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.io.InputStream)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.lang.String)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of [B)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.nio.channels.ReadableByteChannel)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of [B)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of java.io.Reader)
Don't know how to convert class java.io.ByteArrayInputStream into (stream-of java.lang.String)
Don't know how to convert class java.io.ByteArrayInputStream into (seq-of [B)
Perhaps I'm misinterpreting the use of possible in byte-streams/possible-conversions? Is there a method of getting a list of conversions that will always work? Would filtering out the implicit conversions be enough?
The generator in question in case anyone's interested looks like this:
;; (require '[clojure.spec :as s])
;; (require '[clojure.spec.gen :as gen])
(defn- convertible?
[x to]
(some? (byte-streams/conversion-path x to)))
(s/def ::baisable
(s/with-gen
#(convertible? % ByteArrayInputStream)
(fn []
(->> (gen/tuple (gen/fmap #(ByteArrayInputStream. %) (gen/bytes))
(gen/elements (byte-streams/possible-conversions
ByteArrayInputStream)))
(gen/fmap (fn [[bais dest]] (byte-streams/convert bais dest))))))){:append? true} is ignored on Mac, but not Linux
The transfer function accepts an options map, and one option that's accepted when the output is a file is :append?, which defaults to true. That option is supposed to be used at https://github.com/ztellman/byte-streams/blob/a7706c1/src/byte_streams.clj#L571-L573 and indeed it seems to work when run on a Linux machine. However, when the same code is run on a macbook, this option seems to do nothing: setting it to true, or false, or not specifying it at all, all result in the output file being clobbered and overwritten with the new contents.
Below is some sample clojure code you can run to reproduce this issue, expressed as a clojure.test deftest; after loading this namespace, you can simply run (append-works) to check for correct behavior. On Mac OS, one test fails; on Linux, they all pass.
Note that this uses ./test.txt as a scratch file, so if you have something important in this location there is some risk to running the tests. I've included a bail-out clause at the beginning to avoid that, but just in case it's probably best not to rely on that, and instead make sure that no such file exists before running the test.
(ns amalloy.test-byte-streams
(:require [byte-streams :as bs]
[clojure.java.io :as io]
[clojure.test :refer [deftest is]])
(:import (java.io FileOutputStream)
(java.nio ByteBuffer)))
(def text (.getBytes "abcd"))
(def file (io/file "test.txt"))
(defn contents []
(slurp file))
(deftest append-works
(when (.exists file) (throw (RuntimeException. "Refusing to delete existing test.txt file")))
(bs/transfer text file)
(is (= "abcd" (contents)) "transferring to deleted file failed")
(bs/transfer text file {:append? false})
(is (= "abcd" (contents)) "overwriting with :append? false failed")
(bs/transfer text file {:append? true})
(is (= "abcdabcd" (contents)) "appending same contents with :append? true failed")
(bs/transfer text file {:append? false})
(is (= "abcd" (contents)) "overwriting with :append? false failed")
(with-open [out-chan (.getChannel (FileOutputStream. file true))]
(.write out-chan (ByteBuffer/wrap text)))
(is (= "abcdabcd" (contents)) "appending by hand with java.nio failed")
(.delete file))`def-conversion` sometimes breaks when AOT-compiled
Repro: https://github.com/bevuta/byte-streams-def-conversion-aot-issue
Essentially the same issue as #34 but with tools.deps and tools.build instead of Leiningen. The resulting error is the same:
Execution error (ClassCastException) at clj_commons.byte_streams.graph.ConversionGraph/assoc_conversion (graph.clj:117).
class clj_commons.byte_streams.graph.Type cannot be cast to class clj_commons.byte_streams.graph.Type (clj_commons.byte_streams.graph.Type is in unnamed module of loader 'app'; clj_commons.byte_streams.graph.Type is in unnamed module of loader clojure.lang.DynamicClassLoader @650a1aff)
It's very likely caused by https://clojure.atlassian.net/browse/CLJ-1741, i.e. when first AOT-compiling a namespace which depends on another namespace which contains a def-conversion call, and then AOT-compiling that namespace itself.
`convert` fails on large pipedinputstream from `byte-transforms/compress`
I discovered today that when I use byte-transforms/compress with :gzip on a 175MB pre-compressed file (so the compress should mostly just waste time), and then I try to do convert to a seq of 50k byte buffers, I get a StackOverflowException. Here's a stack trace excerpt:
java.lang.StackOverflowError
java.lang.Class/getSuperclass(Class.java:-2)
clojure.core$super_chain/invoke(core_deftype.clj:485)
clojure.core$super_chain/invoke(core_deftype.clj:485)
clojure.core$super_chain/invoke(core_deftype.clj:485)
clojure.core$find_protocol_impl/invoke(core_deftype.clj:499)
clojure.core$satisfies_QMARK_/invoke(core_deftype.clj:531)
byte_streams$fn__759$fn__772$fn__773/invoke(:270)
clojure.core.protocols$fn__6034/invoke(protocols.clj:143)
clojure.core.protocols$fn__6005$G__6000__6014/invoke(protocols.clj:19)
clojure.core.protocols$seq_reduce/invoke(protocols.clj:31)
clojure.core.protocols$fn__6026/invoke(protocols.clj:54)
clojure.core.protocols$fn__5979$G__5974__5992/invoke(protocols.clj:13)
clojure.core$reduce/invoke(core.clj:6177)
byte_streams$fn__759$fn__772/invoke(:266)
byte_streams$convert/invoke(:331)
byte_streams$eval921$fn__923$fn__925/invoke(:534)
clojure.lang.LazySeq/sval(LazySeq.java:42)
clojure.lang.LazySeq/seq(LazySeq.java:60)
clojure.lang.RT/seq(RT.java:484)
clojure.core$seq/invoke(core.clj:133)
clojure.core$empty_QMARK_/invoke(core.clj:5595)
byte_streams$closeable_seq/invoke(:213)
byte_streams$fn__759$fn__772/invoke(:278)
byte_streams$convert/invoke(:331)
byte_streams$eval921$fn__923$fn__925/invoke(:534)
clojure.lang.LazySeq/sval(LazySeq.java:42)
clojure.lang.LazySeq/seq(LazySeq.java:60)
clojure.lang.RT/seq(RT.java:484)
clojure.core$seq/invoke(core.clj:133)
clojure.core$empty_QMARK_/invoke(core.clj:5595)
byte_streams$closeable_seq/invoke(:213)
byte_streams$fn__759$fn__772/invoke(:278)
byte_streams$convert/invoke(:331)
byte_streams$eval921$fn__923$fn__925/invoke(:534)
clojure.lang.LazySeq/sval(LazySeq.java:42)
clojure.lang.LazySeq/seq(LazySeq.java:60)
clojure.lang.RT/seq(RT.java:484)
clojure.core$seq/invoke(core.clj:133)
clojure.core$empty_QMARK_/invoke(core.clj:5595)
byte_streams$closeable_seq/invoke(:213)
byte_streams$fn__759$fn__772/invoke(:278)
byte_streams$convert/invoke(:331)
byte_streams$eval921$fn__923$fn__925/invoke(:534)
clojure.lang.LazySeq/sval(LazySeq.java:42)
clojure.lang.LazySeq/seq(LazySeq.java:60)
clojure.lang.RT/seq(RT.java:484)
clojure.core$seq/invoke(core.clj:133)
clojure.core$empty_QMARK_/invoke(core.clj:5595)
byte_streams$closeable_seq/invoke(:213)
byte_streams$fn__759$fn__772/invoke(:278)
byte_streams$convert/invoke(:331)
byte_streams$eval921$fn__923$fn__925/invoke(:534)
clojure.lang.LazySeq/sval(LazySeq.java:42)
clojure.lang.LazySeq/seq(LazySeq.java:60)
clojure.lang.RT/seq(RT.java:484)
clojure.core$seq/invoke(core.clj:133)
clojure.core$empty_QMARK_/invoke(core.clj:5595)
byte_streams$closeable_seq/invoke(:213)
byte_streams$fn__759$fn__772/invoke(:278)
byte_streams$convert/invoke(:331)
byte_streams$eval921$fn__923$fn__925/invoke(:534)
clojure.lang.LazySeq/sval(LazySeq.java:42)
clojure.lang.LazySeq/seq(LazySeq.java:60)
clojure.lang.RT/seq(RT.java:484)
clojure.core$seq/invoke(core.clj:133)
clojure.core$empty_QMARK_/invoke(core.clj:5595)
byte_streams$closeable_seq/invoke(:213)
byte_streams$fn__759$fn__772/invoke(:278)
byte_streams$convert/invoke(:331)
byte_streams$eval921$fn__923$fn__925/invoke(:534)
clojure.lang.LazySeq/sval(LazySeq.java:42)
clojure.lang.LazySeq/seq(LazySeq.java:60)
clojure.lang.RT/seq(RT.java:484)
clojure.core$seq/invoke(core.clj:133)
clojure.core$empty_QMARK_/invoke(core.clj:5595)
byte_streams$closeable_seq/invoke(:213)
byte_streams$fn__759$fn__772/invoke(:278)
byte_streams$convert/invoke(:331)
byte_streams$eval921$fn__923$fn__925/invoke(:534)
Lazy converter instantiation performance gotcha
Converters are instantiated (and then memoized) lazily on first use. Since this is a rather heavy operation, it can influence timing-sensitive code as recently observed in Aleph's test suite. This is bit of a gotcha which was even brought up in a past issue already. That resulted in the introduction of a precache-conversions API. However, it later got removed again without further explanation. In any case, the fact that this issue came up again indicates that such an API would still be desirable.
(bs/to-byte-array []) fails
Is this by design? I would expect to get an empty byte array instead.
=> (bs/to-byte-array [])
IllegalArgumentException Don't know how to convert class clojure.lang.PersistentVector into class [B byte-streams/convert (byte_streams.clj:187)
Since (bs/to-byte-array [(byte-array [2 3 4]) (byte-array [5 6])]) works, the behavior seems surprising to me. It requires checking that the array is not empty before calling to-byte-array.
Remove use of clj-tuple
It seems to be causing problems, and should largely be superseded by this point.
See clj-commons/aleph#520 for an example
"Don't know how to convert class manifold.stream.BufferedStream into class java.io.InputStream"
Given deps.edn with:
aleph/aleph {:mvn/version "0.4.6"}
Attempting to
(require '[aleph.http :as http])
(http/get "https://google.com/") ;; returns a manifold.deferred.Deferred
=> <...>
;; But attempting to deref throws:
@(http/get "https://google.com/")
=> Don't know how to convert class manifold.stream.BufferedStream into class java.io.InputStream
byte-streams doesn't initialize when AOT'd
It seems that byte-streams does not initialize its conversion library when AOT compiled. With the following project.clj:
(defproject byte-streams-test "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.5.1"]
[byte-streams "0.1.5"]])and byte_streams_test/core.clj:
(ns byte-streams-test.core
(:require [byte-streams :as bs]))
(defn foo
"I don't do a whole lot."
[x]
(pr (var-get #'bs/src->dst->conversion))
(println x "Hello, World!"))I get the expected results (byte-streams works correctly):
$ lein do clean, repl
nREPL server started on port 55976 on host 127.0.0.1
REPL-y 0.2.1
Clojure 1.5.1
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
user=> (require '[byte-streams-test.core :as core])
nil
user=> (core/foo "ohai")
#<Atom@11d6ecc3: {(seq-of java.nio.ByteBuffer) {java.nio.channels.ReadableByteChannel #< clojure.lang.AFunction$1@1bf0eebd>, java.nio.ByteBuffer #< clojure.lang.AFunction$1@66c9c89f>}, java.io.OutputStream {java.nio.channels.WritableByteChannel #< clojure.lang.AFunction$1@49927371>}, java.io.Reader {java.lang.CharSequence #< clojure.lang.AFunction$1@2dd92573>}, java.nio.channels.ReadableByteChannel {(seq-of java.nio.ByteBuffer) #< clojure.lang.AFunction$1@b29b889>, java.io.InputStream #< clojure.lang.AFunction$1@60123018>}, java.nio.channels.WritableByteChannel {java.io.OutputStream #< clojure.lang.AFunction$1@4a13a8ee>}, #'byte-streams/ByteSource {java.lang.CharSequence #< clojure.lang.AFunction$1@50b7c740>}, java.io.InputStream {java.io.Reader #< clojure.lang.AFunction$1@517a6426>, java.nio.channels.ReadableByteChannel #< clojure.lang.AFunction$1@3a452494>}, java.lang.CharSequence {java.lang.String #< clojure.lang.AFunction$1@11648e6f>}, [B {java.lang.String #< clojure.lang.AFunction$1@60c32254>, java.io.InputStream #< clojure.lang.AFunction$1@2b02ea6d>, java.nio.DirectByteBuffer #< clojure.lang.AFunction$1@4c6cb02a>, nil #< clojure.lang.AFunction$1@73a1bc98>}, java.lang.String {[B #< clojure.lang.AFunction$1@7f64da76>}, (seq-of java.lang.String) {java.lang.String #< clojure.lang.AFunction$1@3642ba51>}, java.io.File {(seq-of java.nio.ByteBuffer) #< clojure.lang.AFunction$1@56d637ab>, java.nio.channels.WritableByteChannel #< clojure.lang.AFunction$1@9e0e817>, java.nio.channels.ReadableByteChannel #< clojure.lang.AFunction$1@ca54312>}, java.nio.ByteBuffer {(seq-of java.nio.ByteBuffer) #< clojure.lang.AFunction$1@35d6593a>, [B #< clojure.lang.AFunction$1@b7c0a73>}}>ohai Hello, World!
nil
user=>
But if I add AOT compilation in project.clj:
(defproject byte-streams-test "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.5.1"]
[byte-streams "0.1.5"]]
:aot [byte-streams-test.core])then the conversion library is empty, and consequently one cannot convert anything:
$ lein do clean, repl
Compiling byte-streams-test.core
nREPL server started on port 56042 on host 127.0.0.1
REPL-y 0.2.1
Clojure 1.5.1
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
user=> (require '[byte-streams-test.core :as core])
nil
user=> (core/foo "ohai")
#<Atom@677515f7: nil>ohai Hello, World!
nil
user=>
Release 0.4.0?
What do you think about releasing 0.4.0 now with the namespace duplications? I'm working on some code with Java modules and this is one of the blockers.
I can probably do the release but wanted to check if it's ok.
Eagerly evaluate conversion paths?
Is it possible to eagerly evaluate all of the conversion paths?
I'm running into trouble with some time-sensitive core.async tests on my CI server because byte-streams lazily loads.
E.g.
(time
(bs/transfer (bs/to-byte-buffer "foo") (java.io.File/createTempFile "warm" "up")))
"Elapsed time: 1860.858 msecs"
"Elapsed time: 1.18 msecs"Undeclared behavior for transfer {:close? true}
The docstring for transfer documents the behavior of of the :close? flag as the following:
close?- whether the sink should be closed once the transfer is done, defaults to true.
However, on further inspection, transfer-fn appears to close both the source and sink streams when :close? is set, leading to potentially undesirable behavior if the caller is not expecting the source stream to close.
I would suggest either updating the docstring or behavior to be consistent with the user's expectations.
`closeable-seq` may end prematurely after GC?
We have some code that reads an InputStream into chunks of fixed size ByteBuffers using (convert s (seq-of ByteBuffer)). When testing the code, it's found that the returned seq doesn't always have all the data when the input stream is relatively big.
We were able to reproduce it using the following code snippet:
user> (count (convert (io/input-stream (io/file "bigfile")) (seq-of ByteBuffer)))
;; => 26955
user> (count (convert (io/input-stream (io/file "bigfile")) (seq-of ByteBuffer)))
;; => 79920
user> (count (convert (io/input-stream (io/file "bigfile")) (seq-of ByteBuffer)))
;; => 86093
user> (count (convert (io/input-stream (io/file "bigfile")) (seq-of ByteBuffer)))
;; => 27063As the above snippet shows, the count of the seq returned by convert keeps changing. We suspect it might be cause by the finalize in closeable-seq. We think what's happening is that the head of the seq got GCed when it's halfway through consuming it, which triggered a finalize call that caused the source stream to be closed prematurely.
I'm wondering if closeable-seq should stop overriding finalize and leave it to the user to close the source, like line-seq does.
to-input-stream not lazy on lazy seqs
Contrary to the https://raw.githubusercontent.com/ztellman/byte-streams/0.2.0/README.md#L37
=> (to-input-stream (repeat "hello"))
OutOfMemoryError Requested array size exceeds VM limit java.util.Arrays.copyOf (Arrays.java:3332)
is not lazy.
Heisenbug with graph.clj
Have been getting the following error, which I can't reproduce on other machines.
I'm noting here just in case others are seeing the same thing.
I have no idea what is causing it, it's caused me a lot of pain this week. Just can't see what the fix is.
CompilerException java.lang.RuntimeException: No such var: s/->source, compiling:(byte_streams/graph.clj:108:64)
file output doesn't overwrite when asked to
(transfer "hello" (File. "/tmp/salutations") {:append? false})
The above called multiple times will end up appending instead of overwriting. Your arguments to the FileOutputStream constructor looked good so I'm not sure where the problem is.
Late declarations of lower-cost conversions are ineffective
Declaring new conversions via def-conversion which would make conversion between two types less costly are ineffective if the conversion in question has occurred at least once before. This is caused by the global converter memoization which captures the state of the conversion graph at the point in time of the very first invocation for a given pair of source and dest types.
Reproducer:
(defrecord Foo [data])
(defrecord Bar [data])
(defn run [n]
(prn :begin n)
(bs/convert (Foo. "foo") String)
(prn :end n)
(newline))
(bs/def-conversion ^{:cost 0} [Foo Bar]
[x _]
(prn :foo->bar)
(Bar. (:data x)))
(bs/def-conversion ^{:cost 0} [Bar String]
[x _]
(prn :bar->string)
(:data x))
;; At this point we can only indirectly convert from `Foo` to `String` via `Bar`
(run 1)
;; Now we declare a direct conversion path from `Foo` to `String`
(bs/def-conversion ^{:cost 0} [Foo String]
[x _]
(prn :foo->string)
(:data x))
;; But because the first invocation has already memoized the more costly path, it has no effect
(run 2)Output:
:begin 1
:foo->bar
:bar->string
:end 1
:begin 2
:foo->bar
:bar->string
:end 2In contrast, moving both run invocations after the last bs/def-conversion outputs:
:begin 1
:foo->string
:end 1
:begin 2
:foo->string
:end 2This is a bit of a gotcha which might at least be worth documenting. Alternatively, def-conversion could reset the memo which should solve this issue.
conversion-fn calls itself with wrong arity
https://github.com/ztellman/byte-streams/blob/master/src/byte_streams/graph.clj#L210
Found using unreleased version of Eastwood, with fixes not discovered necessary until I tried linting the byte-streams library.
Use InputStream#transferTo?
Java 9 has a new method on the InputStream interface: transferTo to transfer to an OutputStream. I don't know about it's efficiency compared to what's already here, but thought it might be a useful addition to byte-streams? There is also readAllBytes which could be useful too.
Transfering bytes to an OutputStream fails
byte-streams does not allow transfering bytes directly to a ByteArrayOutputStream:
byte-streams=> (transfer "abcd" (java.io.ByteArrayOutputStream.))
IllegalArgumentException Don't know how to convert class [B into clojure.core$bytes@73703457 byte-streams/convert (byte_streams.clj:345)Is this supposed to work? If I explicitly convert to an InputStream and then transfer it works.
compare-bytes not equivalent to clojure.core/compare with strings
Found this playing around with simple-check 0.5.0-SNAPSHOT,
(def one "Q")
(def two "Nb")
(compare one two)
;; => 3
(bs/compare-bytes one two)
;; => -1File-to-file transfer fails
This is with
[byte-transforms "0.1.1"
:exclusions [byte-streams]]
[byte-streams "0.1.7"]Trying to copy from one file to another fails:
(bs/transfer (File. "test/fixture/kittensB/kitten1.jpg")
(File/createTempFile "difftron" "image-diff"))yields
sun.nio.ch.FileChannelImpl cannot be cast to java.lang.Number
[Thrown class java.lang.ClassCastException]
Restarts:
0: [QUIT] Quit to the SLIME top level
Backtrace:
0: byte_streams.clj:699 byte-streams/eval4829[fn]
1: byte_streams.clj:425 byte-streams/this[fn]
2: byte_streams.clj:465 byte-streams/transfer
3: byte_streams.clj:460 byte-streams/transfer
4: form-init6519885573008950383.clj:1 com.keminglabs.difftron.image-diff/eval11751
However, converting first to a byte-buffer works fine:
(bs/transfer (bs/to-byte-buffer (File. "test/fixture/kittensB/kitten1.jpg"))
(File. "foo.jpg"))
=> nilb/convert sometimes fails to copy the last byte array
Let coll be a collection of (large) byte arrays.
What I'm seeing is (b/convert coll (class (byte-array 0))) occasionally returns a combined byte-array but without the last byte-array in coll. I'm seeing this only sometimes. My test suite will pass, then it will fail, then it will pass again. It seems to fail about 25% of the time.
I have identified the problem to be b/convert because when I replace the call to b/convert with this logic, the bug disappears. This logic is my own hurried attempt to System/arraycopy all the bytes together, which is what I want b/convert to do for me.
(let [rs (reductions + (map count coll))
result (byte-array (last rs))]
(reduce (fn [_ [source offset]]
(System/arraycopy source 0 result offset (count source)))
nil
(map vector coll (cons 0 rs)))
result)I'm sorry to raise an issue without providing the means to reliably replicate the problem. Feel free to close it, but I'm raising it more as a note in case anyone else has an idea. From reasoning about the code, is there anything that might account for the behavior I'm seeing?
Reflection warnings in byte-streams.clj
As seen with [byte-streams "0.2.4"], using Java 15:
Reflection warning, byte_streams.clj:437:12 - call to method position on java.nio.ByteBuffer can't be resolved (argument types: unknown).
Reflection warning, byte_streams.clj:438:12 - call to method limit can't be resolved (target class is unknown).
Reflection warning, byte_streams.clj:715:19 - call to method limit on java.nio.ByteBuffer can't be resolved (argument types: java.lang.Number).
Reflection warning, byte_streams.clj:718:9 - call to method position on java.nio.ByteBuffer can't be resolved (argument types: java.lang.Number).
Relevant bit(s):
Bytes are read from stream to String incorrectly
I have a ByteArrayInputStream b that contains Chinese characters in UTF-8. I've found that I get bad data from byte-streams when I do this:
(byte-streams/convert b String)I get a string in which some characters are corrupt (2 out of a few thousand).
I've found that I get good data when I do this:
(byte-streams/convert (byte-streams/to-byte-array b) String)I've compared the bytes I get in each of the string results above, and I note that in the first example a tiny handful of bytes (6 out of 17,300) appear to be different to the bytes in the original input stream. In the latter example the bytes are identical to the input (hence no corrupt chars).
What could cause convert to treat an input stream of bytes differently to an array of bytes when converting to a string?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.