bozaro / lz4-rs Goto Github PK
View Code? Open in Web Editor NEWRust LZ4 bindins library
License: MIT License
Rust LZ4 bindins library
License: MIT License
As of v1.23, the lz4 frame and block operations are implemented. Implementing bindings for the third mode of operation, lz4 streaming is the next step for this crate.
I probably won't start implementing it in the next few days, so I haven't given much thought on how the API should look like. Feel free to propose something. Currently, I suppose the encoder/decoder pattern of the lz4 frame mode is a good idea, since the streaming api is meant to be used with rather large compressed sizes and network streams.
Any chance of a new release on Cargo including 1.7.5?
E.g. snap
's API has a max_compress_len
that users use to resize the buffer so the contents fit. However, I can't find this here. I can't also find it in the tests.
Any ideas?
Dear users,
I am sorry I didn't live up to the promises I made in some of the issues and pull requests. My intention was not to deceive you, but to actually continue working on this library and to let this be my small contribution to the Rust community.
However, I don't want to make any more empty promises, I won't fulfill, so I am now looking for a new maintainer for this library.
My plan forward would be to give contributor access to a volunteer, but I just noticed, I can't even do that, since I am not a maintainer of this Github project. If someone volunteers, we should migrate to another namespace.
Regarding the crates.io access rights, I would suggest doing the updates myself for a transitioning period to ensure the new maintainer does not have any malicious intentions.
Cheers,
Jens
Background: Our team is recently planning to switch from go to rust and lz4 is being used to decompress the payload sent from our app. In order to ensure the switch is smooth, we need to ensure the compressed binaries can be correctly decompressed by the rust counterpart.
For this we wrapped some code that mimics our original code behaviour, which retries whenever the output buffer is insufficient.
use lz4::block;
use std::io::Result;
fn compress(src: &[u8]) -> Result<Vec<u8>> {
return block::compress(src, None, false);
}
fn decompress(src: &[u8]) -> Result<Vec<u8>> {
let mut factor = 3;
loop {
let result = block::decompress(src, Some((src.len() as i32) * factor));
factor *= 2;
match result {
Ok(mut v) => {
// truncate trailing zeros
if let Some(i) = v.iter().rposition(|x| *x != 0) {
let new_len = i + 1;
v.truncate(new_len);
}
return Ok(v);
}
Err(e) => {
// lz4 theoretical compression limit is 255, if buffer size exceed
// the limit, just return result
if factor >= 255 {
return Err(e);
}
if e.kind() != std::io::ErrorKind::InvalidData {
return Err(e);
};
}
};
}
}
However we noticed that, when we're testing the cases, the output compressed with rust can't get the correct result after decompression, there's always extra 1's in the rust compressed and uncompressed output.
rs:"11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111", length:209
go:"111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111", length:207
mod tests {
use crate::pkg::lz4;
use base64::prelude::*;
#[test]
fn compression_ratio_less_than_3() {
let src = "2.7111111111111111111111111266249775724709369995957496696762772407663035354759457138217852516642742746".as_bytes();
let rs_compressed = lz4::compress(&src).expect("msg");
let go_compressed = BASE64_STANDARD.decode("TzIuNzEBAATwPDI2NjI0OTc3NTcyNDcwOTM2OTk5NTk1NzQ5NjY5Njc2Mjc3MjQwNzY2MzAzNTM1NDc1OTQ1NzEzODIxNzg1MjUxNjY0Mjc0Mjc0Ng==").expect("msg");
let go_compressed_rs_uncompressed = lz4::decompress(go_compressed.as_slice()).expect("msg");
assert_eq!(go_compressed_rs_uncompressed.as_slice(), src);
let rs_compressed_rs_uncompressed = lz4::decompress(rs_compressed.as_slice()).expect("msg");
assert_eq!(
go_compressed_rs_uncompressed.as_slice(),
rs_compressed_rs_uncompressed.as_slice()
);
}
#[test]
fn compression_ratio_more_than_3() {
let src = "111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111".as_bytes();
let rs_compressed = lz4::compress(src).expect("msg");
let go_compressed = BASE64_STANDARD
.decode("HzEBAKkAAgDgMTExMTExMTExMTExMTE=")
.expect("msg");
let go_compressed_rs_uncompressed = lz4::decompress(go_compressed.as_slice()).expect("msg");
assert_eq!(go_compressed_rs_uncompressed.as_slice(), src);
let rs_compressed_rs_uncompressed = lz4::decompress(rs_compressed.as_slice()).expect("msg");
println!(
"rs:{:?}, length:{:?}",
String::from_utf8_lossy(rs_compressed_rs_uncompressed.as_slice()),
String::from_utf8_lossy(rs_compressed_rs_uncompressed.as_slice()).len()
);
println!(
"go:{:?}, length:{:?}",
String::from_utf8_lossy(go_compressed_rs_uncompressed.as_slice()),
String::from_utf8_lossy(go_compressed_rs_uncompressed.as_slice()).len()
);
// compare the uncompressed result from rust
assert_eq!(
String::from_utf8_lossy(rs_compressed_rs_uncompressed.as_slice()),
String::from_utf8_lossy(src),
);
}
}
When lz4
is listed anywhere in the dependencies of a project, the project must always rebuild/relink; it is never clean. I would expect the second cargo build
here to report Finished
, without any Compiling
.
% cargo build
Compiling lz4 v1.21.0
Compiling lz4rebuild v0.1.0 (file:///home/faux/code/testcases/lz4rebuild)
Finished dev [unoptimized + debuginfo] target(s) in 0.43 secs
% cargo build
Compiling lz4 v1.21.0
Compiling lz4rebuild v0.1.0 (file:///home/faux/code/testcases/lz4rebuild)
Finished dev [unoptimized + debuginfo] target(s) in 0.43 secs
This is especially annoying when repeatedly re-running tests in a project that takes a while to link.
I don't know what's causing the problem
The above project is just a cargo init --bin
with the dependency added. Using an explicit version makes no difference:
% cat Cargo.toml
[package]
name = "lz4rebuild"
version = "0.1.0"
authors = ["Chris West (Faux) <[email protected]>"]
[dependencies]
lz4 = "*"
% cat src/main.rs
fn main() {
println!("Hello, world!");
}
Can't get this crate to compile on my pi -- it does compile on the same toolchain for x86_64 though.
Host info:
Linux pi 4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018 armv7l GNU/Linux
stable-armv7-unknown-linux-gnueabihf (default)
rustc 1.29.1 (b801ae664 2018-09-20)
Build log:
Compiling lz4 v1.23.0
error[E0308]: mismatched types
--> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:81:21
|
81 | src.as_ptr() as *const i8,
| ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
|
= note: expected type `*const u8`
found type `*const i8`
error[E0308]: mismatched types
--> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:82:21
|
82 | dst_buf.as_mut_ptr() as *mut i8,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
|
= note: expected type `*mut u8`
found type `*mut i8`
error[E0308]: mismatched types
--> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:90:21
|
90 | src.as_ptr() as *const i8,
| ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
|
= note: expected type `*const u8`
found type `*const i8`
error[E0308]: mismatched types
--> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:91:21
|
91 | dst_buf.as_mut_ptr() as *mut i8,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
|
= note: expected type `*mut u8`
found type `*mut i8`
error[E0308]: mismatched types
--> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:99:21
|
99 | src.as_ptr() as *const i8,
| ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
|
= note: expected type `*const u8`
found type `*const i8`
error[E0308]: mismatched types
--> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:100:21
|
100 | dst_buf.as_mut_ptr() as *mut i8,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
|
= note: expected type `*mut u8`
found type `*mut i8`
error[E0308]: mismatched types
--> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:164:13
|
164 | src.as_ptr() as *const i8,
| ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
|
= note: expected type `*const u8`
found type `*const i8`
error[E0308]: mismatched types
--> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:165:13
|
165 | decompressed.as_mut_ptr() as *mut i8,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
|
= note: expected type `*mut u8`
found type `*mut i8`
Heyo,
I'm planning on releasing a tool that uses the lz4-rs block-compression module, which is unreleased on crates.io yet. Because crates.io doesn't allow git dependencies, I'm currently stuck. Is there any plans on making a new release to make those functions available to crates.io users ? Anything I can do to help ?
Thanks for the great dependency :)
lz4f can not decompress by lz4 compress.
I am using a buffer writer with Lz4 like so:
BufWriter::new(Box::new(lz4::EncoderBuilder::new().build(File::create(output)?)?
Yet this doesn't close properly, as the last chunk of output is not written, even if I call flush manually.
Hi!
I'm using lz4-rs in my compressed_log
1 crate to compress log on the fly to a remote server over persistent websocket connection. I noticed that on MIPS 32-bit big endian clients it produces data stream that x86_64 LE server can't decode. This is very interesting as after investigating upstream liblz4 code it seems like they really care about endianness and the binary stream is supposed to be portable.
Custom { kind: Other, error: LZ4Error("ERROR_frameType_unknown") }
(Which appears to be this line in liblz4 https://github.com/lz4/lz4/blob/591b6621244e77d8293230085ae65db0cfa98d88/lib/lz4frame.c#L1078-L1079)
I'm thinking that it may be the bindings problem, or invalid flags for big endian. After some experiments I've got some data decoded, but at some point it still fails with ERROR_frameType_unknown
. So my question is - is there anything missing on lz4-rs side, or some issue? I wonder whats your opinion on this matter.
I've also checked how Linux distros build liblz4 for big endian architectures but they seem to just use upstream Makefile (the process is very similiar to build.rs
in lz4-rs)
Please refer to my fork https://github.com/mpapierski/lz4-rs/blob/big-endian/lz4-sys/build.rs#L20-L29 for lines that appeared to improve a bit the decoding situation.
Hello,
I ran into a problem while using Encoder in a context where it has to be switchable for other Write implémentation : the finish method has to be called to send the last bytes to the inner writer. Other Write implementation for similar purpose (like flate2 or snappy framed) will just flush the last page on drop().
Working around it is tricky: as finish() consumes self, you can not make a trivial wrapper implementing Drop around the Encoder because drop only has a &mut self...
And, you can not just add drop() to the implementation alongside finish(), because finish() returns the inner Write. I just think we can't have both, unless we have two variants of Encoder.
I see the point of being able to get the inner Write back, but I'm wondering if this feature is preferable to compatibility with other encoder (and safety, as calling drop() can not be forgotten).
What do you think ?
I understand this is binding to a C library, but the bindings don't seem to be bound. Or something. I don't speak C. I'm not sure what assumptions this is making about my local machine, because there is no install guide. lz4
is installed on my machine and is pointing to /usr/local/bin/lz4
.
I made a new project with cargo new
, added lz4 = "1.23.1"
to the dependencies in Cargo.toml
, and ran cargo check
. I got the following opaque error messages:
$ export RUST_BACKTRADE=1; cargo check
Compiling lz4-sys v1.8.3
error: failed to run custom build command for `lz4-sys v1.8.3`
process didn't exit successfully: `/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-a7eabe1892d3ba65/build-script-build` (exit code: 101)
--- stdout
TARGET = Some("x86_64-apple-darwin")
HOST = Some("x86_64-apple-darwin")
CC_x86_64-apple-darwin = None
CC_x86_64_apple_darwin = None
HOST_CC = None
CC = None
CFLAGS_x86_64-apple-darwin = None
CFLAGS_x86_64_apple_darwin = None
HOST_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2,sse3,ssse3")
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4.o" "-c" "liblz4/lib/lz4.c"
exit code: 0
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4frame.o" "-c" "liblz4/lib/lz4frame.c"
exit code: 0
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4hc.o" "-c" "liblz4/lib/lz4hc.c"
exit code: 0
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/xxhash.o" "-c" "liblz4/lib/xxhash.c"
exit code: 0
AR_x86_64-apple-darwin = None
AR_x86_64_apple_darwin = None
HOST_AR = None
AR = None
running: "ar" "crs" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4.a" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4frame.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4hc.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/xxhash.o"
cargo:warning=/opt/local/bin/ranlib: object: /Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4.a(lz4.o) malformed object (unknown load command 1)
cargo:warning=ar: internal ranlib command failed
exit code: 1
--- stderr
thread 'main' panicked at '
Internal error occurred: Command "ar" "crs" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4.a" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4frame.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/lz4hc.o" "/Users/richard.rast/example/lz4ex/target/debug/build/lz4-sys-3174f5bb3bf29731/out/liblz4/lib/xxhash.o" with args "ar" did not execute successfully (status code exit code: 1).
', /Users/richard.rast/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.31/src/lib.rs:2367:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
warning: build failed, waiting for other jobs to finish...
error: build failed
Hi @jheyens, do you think we could get a new release of lz4-sys? I am eagerly awaiting a release that includes #23, and you mentioned the possibility of a release earlier this month?
Thanks in advance!
The following code fails, which is not desired behavior
use lz4::block::{compress,decompress};
let v = vec![0u8; 0];
let comp_with_prefix = compress(&v, None, true).unwrap();
assert_eq!(v, decompress(&comp_with_prefix, None).unwrap());
I have need for a Write
-based API, rather than a Read
one, to coexist alongside the flate2
and xz2
Write
-based decoder/encoders.
Hi,
I'm writing an app that is using in stream LZ4 to write to a file. I need to guarantee that the file is sync'ed sometimes, I'm using:
encoder.flush()?;
encode.writer().sync_all()?;
Is this enough to guarantee that the lz4 buffer is
a) flushed
b) written to disk and fsync'ed
c) will be readable (even though .finish might not have been called).
Cheers,
James
liblz4 is tracking https://github.com/Cyan4973/lz4.git instead of https://github.com/lz4/lz4, so is approximately 5 months out-of-date.
This is affecting me insofar as the tracked revision doesn't build on GCC < 4.5, but latest upstream master does.
Thanks for authoring this lib!
Currently I have to drop down to the unsafe lz4-sys
bindings because the higher level stuff doesn't quite fit my use-case. My issue is that all the high level APIs are allocating the output buffer themselves. It would be ideal if the lz4::block::*
methods had variants that allowed you to supply an output &mut [u8]
, and the lz4::Decoder
/ lz4::Encoder
structs allowed you to supply an output Vec<u8>
. What do you think? If you have some ideas on how you want the API to look I'm happy to whip up a PR.
lz4-sys/src/lib.rs
has a type error on line 347 there is a test at the very end of the file and it's evaluting to a ()
while it's actual type is i32
.
Is that possible to support wasm? I know currently libc
crate doesn't support wasm yet (rust-lang/libc#1126), but since this crate only used limited number of c types (c_char, c_int and etc.). Replace them with Rust types like i8, i32 and etc, could be a possible temporary solution.
Any thoughts?
Hey,
while tweaking the encoder's parameters for my Lolcate program, I experienced a performance increase when increasing its block size (I found a sweet spot at 256 kb).
This made me willing to try and give the decoder's buffer size different values, hoping to get similar performance increases, given that the default value seems small.
Is there any interest in doing so, or is the default value the best one in your own experience ?
Cheers
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.