tweedegolf / lzjd-rs

Rust implementation of the LZJD algorithm (https://github.com/EdwardRaff/jLZJD)

LZJD

Documentation

Rust implementation of Lempel-Ziv Jaccard Distance (LZJD) algorithm based on jLZJD

Main differences:

• Rust instead of Java
• Can use any hasher (executable uses CRC32) instead of just Murmur3
• Does not allocate memory for every unique hash, instead keeps k=1024 smallest
• Based on Vec instead of IntSetNoRemove, which is more like HashMap
• Hash files are considerably smaller if small sequences have been digested

USAGE:
    lzjd [FLAGS] [OPTIONS] <INPUT>...

FLAGS:
    -c, --compare        compare SDBFs in file, or two SDBF files
    -r, --deep           generate SDBFs from directories and files
    -g, --gen-compare    compare all pairs in source data
    -h, --help           Prints help information
    -V, --version        Prints version information

OPTIONS:
    -o, --output <FILE>            send output to files
    -t, --threshold <THRESHOLD>    only show results >= threshold [default: 1]

ARGS:
    <INPUT>...    Sets the input file to use

See also:

• Original paper
• Follow-up paper

lzjd-rs's Issues

Is this project abandoned?

Clippy doesn't pass, most of the dependencies are outdated, and there are two-year-old PRs without attention.

Parallelism causes immediate failure

user@localhost~$ env RUST_BACKTRACE=1 target/debug/lzjd -p 2 -r -g /dev/null

thread 'thread '<unnamed><unnamed>' panicked at '' panicked at 'attempted to leave type `internal::Local` uninitialized, which is invalidattempted to leave type `internal::Local` uninitialized, which is invalid', ', /rustc/7bade6ef730cff83f3591479a98916920f66decd/library/core/src/mem/mod.rs/rustc/7bade6ef730cff83f3591479a98916920f66decd/library/core/src/mem/mod.rs::659659::99

This happens only without -p 1 (so even without specific value) and not just on /dev/null or with the debug binary.