bozaro / lz4-rs Goto Github PK

As of v1.23, the lz4 frame and block operations are implemented. Implementing bindings for the third mode of operation, lz4 streaming is the next step for this crate.

I probably won't start implementing it in the next few days, so I haven't given much thought on how the API should look like. Feel free to propose something. Currently, I suppose the encoder/decoder pattern of the lz4 frame mode is a good idea, since the streaming api is meant to be used with rather large compressed sizes and network streams.

Dear users,

I am sorry I didn't live up to the promises I made in some of the issues and pull requests. My intention was not to deceive you, but to actually continue working on this library and to let this be my small contribution to the Rust community.
However, I don't want to make any more empty promises, I won't fulfill, so I am now looking for a new maintainer for this library.

My plan forward would be to give contributor access to a volunteer, but I just noticed, I can't even do that, since I am not a maintainer of this Github project. If someone volunteers, we should migrate to another namespace.

Regarding the crates.io access rights, I would suggest doing the updates myself for a transitioning period to ensure the new maintainer does not have any malicious intentions.

Cheers,
Jens

The following code fails, which is not desired behavior

use lz4::block::{compress,decompress};
let v = vec![0u8; 0];
let comp_with_prefix = compress(&v, None, true).unwrap();
assert_eq!(v, decompress(&comp_with_prefix, None).unwrap());

I understand this is binding to a C library, but the bindings don't seem to be bound. Or something. I don't speak C. I'm not sure what assumptions this is making about my local machine, because there is no install guide. lz4 is installed on my machine and is pointing to /usr/local/bin/lz4.

I made a new project with cargo new, added lz4 = "1.23.1" to the dependencies in Cargo.toml, and ran cargo check. I got the following opaque error messages:

$ export RUST_BACKTRADE=1; cargo check
   Compiling lz4-sys v1.8.3
error: failed to run custom build command for `lz4-sys v1.8.3`
process didn't exit successfully: `/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-a7eabe1892d3ba65/build-script-build` (exit code: 101)
--- stdout
TARGET = Some("x86_64-apple-darwin")
HOST = Some("x86_64-apple-darwin")
CC_x86_64-apple-darwin = None
CC_x86_64_apple_darwin = None
HOST_CC = None
CC = None
CFLAGS_x86_64-apple-darwin = None
CFLAGS_x86_64_apple_darwin = None
HOST_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2,sse3,ssse3")
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4.o" "-c" "liblz4/lib/lz4.c"
exit code: 0
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4frame.o" "-c" "liblz4/lib/lz4frame.c"
exit code: 0
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4hc.o" "-c" "liblz4/lib/lz4hc.c"
exit code: 0
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/xxhash.o" "-c" "liblz4/lib/xxhash.c"
exit code: 0
AR_x86_64-apple-darwin = None
AR_x86_64_apple_darwin = None
HOST_AR = None
AR = None
running: "ar" "crs" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4.a" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4frame.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4hc.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/xxhash.o"
cargo:warning=/opt/local/bin/ranlib: object: /Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4.a(lz4.o) malformed object (unknown load command 1)
cargo:warning=ar: internal ranlib command failed
exit code: 1

--- stderr
thread 'main' panicked at '

Internal error occurred: Command "ar" "crs" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4.a" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4frame.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4hc.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/xxhash.o" with args "ar" did not execute successfully (status code exit code: 1).

', /Users/richard.rast/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.31/src/lib.rs:2367:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

lz4/lz4#108

E.g. snap's API has a max_compress_len that users use to resize the buffer so the contents fit. However, I can't find this here. I can't also find it in the tests.

Any ideas?

liblz4 is tracking https://github.com/Cyan4973/lz4.git instead of https://github.com/lz4/lz4, so is approximately 5 months out-of-date.

This is affecting me insofar as the tracked revision doesn't build on GCC < 4.5, but latest upstream master does.

Hi @jheyens, do you think we could get a new release of lz4-sys? I am eagerly awaiting a release that includes #23, and you mentioned the possibility of a release earlier this month?

Thanks in advance!

Any chance of a new release on Cargo including 1.7.5?

When lz4 is listed anywhere in the dependencies of a project, the project must always rebuild/relink; it is never clean. I would expect the second cargo build here to report Finished, without any Compiling.

% cargo build
   Compiling lz4 v1.21.0
   Compiling lz4rebuild v0.1.0 (file:///home/faux/code/testcases/lz4rebuild)
    Finished dev [unoptimized + debuginfo] target(s) in 0.43 secs
% cargo build
   Compiling lz4 v1.21.0
   Compiling lz4rebuild v0.1.0 (file:///home/faux/code/testcases/lz4rebuild)
    Finished dev [unoptimized + debuginfo] target(s) in 0.43 secs

This is especially annoying when repeatedly re-running tests in a project that takes a while to link.

I don't know what's causing the problem

The above project is just a cargo init --bin with the dependency added. Using an explicit version makes no difference:

% cat Cargo.toml 
[package]
name = "lz4rebuild"
version = "0.1.0"
authors = ["Chris West (Faux) <[email protected]>"]

[dependencies]
lz4 = "*"


% cat src/main.rs 
fn main() {
    println!("Hello, world!");
}

Can't get this crate to compile on my pi -- it does compile on the same toolchain for x86_64 though.

Host info:

Linux pi 4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018 armv7l GNU/Linux
stable-armv7-unknown-linux-gnueabihf (default)
rustc 1.29.1 (b801ae664 2018-09-20)

Build log:

 Compiling lz4 v1.23.0                                                                                        
error[E0308]: mismatched types
  --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:81:21
   |
81 |                     src.as_ptr() as *const i8,
   |                     ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
   |
   = note: expected type `*const u8`
              found type `*const i8`

error[E0308]: mismatched types
  --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:82:21
   |
82 |                     dst_buf.as_mut_ptr() as *mut i8,
   |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
   |
   = note: expected type `*mut u8`
              found type `*mut i8`

error[E0308]: mismatched types
  --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:90:21
   |
90 |                     src.as_ptr() as *const i8,
   |                     ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
   |
   = note: expected type `*const u8`
              found type `*const i8`

error[E0308]: mismatched types
  --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:91:21
   |
91 |                     dst_buf.as_mut_ptr() as *mut i8,
   |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
   |
   = note: expected type `*mut u8`
              found type `*mut i8`

error[E0308]: mismatched types
  --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:99:21
   |
99 |                     src.as_ptr() as *const i8,
   |                     ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
   |
   = note: expected type `*const u8`
              found type `*const i8`

error[E0308]: mismatched types
   --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:100:21
    |
100 |                     dst_buf.as_mut_ptr() as *mut i8,
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
    |
    = note: expected type `*mut u8`
               found type `*mut i8`

error[E0308]: mismatched types
   --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:164:13
    |
164 |             src.as_ptr() as *const i8,
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
    |
    = note: expected type `*const u8`
               found type `*const i8`

error[E0308]: mismatched types
   --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:165:13
    |
165 |             decompressed.as_mut_ptr() as *mut i8,
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
    |
    = note: expected type `*mut u8`
               found type `*mut i8`

Thanks for authoring this lib!

Currently I have to drop down to the unsafe lz4-sys bindings because the higher level stuff doesn't quite fit my use-case. My issue is that all the high level APIs are allocating the output buffer themselves. It would be ideal if the lz4::block::* methods had variants that allowed you to supply an output &mut [u8], and the lz4::Decoder / lz4::Encoder structs allowed you to supply an output Vec<u8>. What do you think? If you have some ideas on how you want the API to look I'm happy to whip up a PR.

Heyo,

I'm planning on releasing a tool that uses the lz4-rs block-compression module, which is unreleased on crates.io yet. Because crates.io doesn't allow git dependencies, I'm currently stuck. Is there any plans on making a new release to make those functions available to crates.io users ? Anything I can do to help ?

Thanks for the great dependency :)

Hello,

I ran into a problem while using Encoder in a context where it has to be switchable for other Write implémentation : the finish method has to be called to send the last bytes to the inner writer. Other Write implementation for similar purpose (like flate2 or snappy framed) will just flush the last page on drop().

Working around it is tricky: as finish() consumes self, you can not make a trivial wrapper implementing Drop around the Encoder because drop only has a &mut self...

And, you can not just add drop() to the implementation alongside finish(), because finish() returns the inner Write. I just think we can't have both, unless we have two variants of Encoder.

I see the point of being able to get the inner Write back, but I'm wondering if this feature is preferable to compatibility with other encoder (and safety, as calling drop() can not be forgotten).

What do you think ?

Hi,

I'm writing an app that is using in stream LZ4 to write to a file. I need to guarantee that the file is sync'ed sometimes, I'm using:

encoder.flush()?;
encode.writer().sync_all()?;

Is this enough to guarantee that the lz4 buffer is

a) flushed
b) written to disk and fsync'ed
c) will be readable (even though .finish might not have been called).

Cheers,
James

Hey,

while tweaking the encoder's parameters for my Lolcate program, I experienced a performance increase when increasing its block size (I found a sweet spot at 256 kb).

This made me willing to try and give the decoder's buffer size different values, hoping to get similar performance increases, given that the default value seems small.

Is there any interest in doing so, or is the default value the best one in your own experience ?

Cheers

I am using a buffer writer with Lz4 like so:

BufWriter::new(Box::new(lz4::EncoderBuilder::new().build(File::create(output)?)?

Yet this doesn't close properly, as the last chunk of output is not written, even if I call flush manually.

Is that possible to support wasm? I know currently libc crate doesn't support wasm yet (rust-lang/libc#1126), but since this crate only used limited number of c types (c_char, c_int and etc.). Replace them with Rust types like i8, i32 and etc, could be a possible temporary solution.

Any thoughts?

I have need for a Write-based API, rather than a Read one, to coexist alongside the flate2 and xz2 Write-based decoder/encoders.

lz4f can not decompress by lz4 compress.

Hi!

I'm using lz4-rs in my compressed_log1 crate to compress log on the fly to a remote server over persistent websocket connection. I noticed that on MIPS 32-bit big endian clients it produces data stream that x86_64 LE server can't decode. This is very interesting as after investigating upstream liblz4 code it seems like they really care about endianness and the binary stream is supposed to be portable.

Custom { kind: Other, error: LZ4Error("ERROR_frameType_unknown") }

(Which appears to be this line in liblz4 https://github.com/lz4/lz4/blob/591b6621244e77d8293230085ae65db0cfa98d88/lib/lz4frame.c#L1078-L1079)

I'm thinking that it may be the bindings problem, or invalid flags for big endian. After some experiments I've got some data decoded, but at some point it still fails with ERROR_frameType_unknown. So my question is - is there anything missing on lz4-rs side, or some issue? I wonder whats your opinion on this matter.

I've also checked how Linux distros build liblz4 for big endian architectures but they seem to just use upstream Makefile (the process is very similiar to build.rs in lz4-rs)

Please refer to my fork https://github.com/mpapierski/lz4-rs/blob/big-endian/lz4-sys/build.rs#L20-L29 for lines that appeared to improve a bit the decoding situation.

lz4-sys/src/lib.rs has a type error on line 347 there is a test at the very end of the file and it's evaluting to a () while it's actual type is i32.