Potentially switch to an external snappy crate? about kafka-rust

[xitep]

hello,

i did some work on "unpacking" snappy compressed messages in kafka-rust and feel confident to provide an answer here.

first of all, kafka-rust is bound to the format spilled out by SnappyOutputStream which is used at least by Kafka 0.8.2.x. this is basically a concatenation of snappy compressed blocks - unfortunately not snappy frames.

• so far i can see, we cannot use snappy_framed at the moment due to the format of the compressed data Kafka produces.
• snappy is a thin ffi wrapper around the c/c++ library which is exactly what kafka-rust does. in other words, with some minor changes this library could in fact be used as a drop-in replacement for the corresponding code in kafka-rust. however, in branch-0.2.x - hopefully to be released in a few weeks - i extended the core decompression function a little bit to avoid some extra copying. so far, i'm quite OK with the performance. however, such changes could be intergrated into snappy, of course. for me, snappy looks like a good candidate.
• i have not looked in detail at rsnappy yet, but right now it seems to me like it provides an API which would be suitable to us. the fact it's written in rust is indeed very interesting. i might have some experiments with it.

if you'd like to experiment yourself, feel free to go ahead and submit a pull request. for decompressing snappy messages we have unit tests as well as benchmarks (requires rust nightly) set up in master as well as the already mentioned branch-0.2.x.

[xitep]

1. merely replacing the existing snappy decompression code in kafka-rust with rsnappy leads to benchmarks indicating a performance loss by a factor of two to three on my machine. but it might well be i've not used rsnappy effectively.

2. further it seems, that rsnappy is not very robust against invalid input data.

3. i've pushed the changes to the rsnappy branch for the moment.

@veddan do you have any measurements comparing the performance of rsnappy to https://github.com/google/snappy ?

[xitep]

fyi: once JeffBelgum/rust-snappy#1 is merged and released, we'll switch over to that snappy crate to get rid of our own snappy native bindings. the corresponding changes to kafka-rust are prepared in the external-snappy-dep branch for the moment.

[xitep]

Switched over to the snappy crate with 930efc9. This change will be released as part of 0.2, since it requires an update of the libc dependency.

[xitep]

just pushed version 0.2.0 to crates.io with a dependency on snappy:0.3.0. feel free to re-open if there's anything left from your side.