alt-text-org / image-match-rs

Rust implementation of the image matching algorithm described by Wong, Bern, and Goldberg

image-match-rs

An implementation of the image matching algorithm described in An Image Signature For Any Kind Of Image by H. Chi Wong, Marshall Bern, and David Goldberg . The algorithm is designed to detect nearly identical images, not images with the same conceptual content.

Usage

By default, the library offers two primary functions: get_buffer_signature(rgba, width) and cosine_similarity(a, b). The former takes a pre-processed slice of u8s with each chunk of four representing the 8-bit red, green, blue, and alpha of a pixel, the latter two result vectors to compute their similarity. Per the source paper and our experiments in this research images with a similarity greater than 0.6 can be considered likely matches. If the tuning methods described below are used, additional research will likely be needed to assess a new cutoff.

If the img feature is used, also provided are get_image_signature(image) and get_file_signature(path) which use the image library to handle unpacking the image into an rgba buffer. All signature functions also expose tuned versions which allow tweaking the crop percentage used during the signature computation, the size of the collection grid which controls the length of the feature vector produced, and the size of the square around each grid point averaged to produce a value for that point. It's recommended to study the algorithm closely before embarking on tuning, as the effects of these nobs are not immediately obvious.

Future Work

• Additional unit testing.
• Experimenting with parameter choice is underway in this repo.
• Experiment with widening the possible values of each dimension in the produced signature. Presently per the paper they are all integers in [-2, 2]. It will likely require experimentation around a new suggested vector similarity cutoff.

image-match-rs's Issues

get_image_signature out of bounds panic with certain kinds of images

I noticed several images causing an out of bounds panic in the get_image_signature function. I'm not sure what's so special about these images in particular that causes this error, as other images of the same width and height don't cause this problem. It also persists when converting from PNG to WEBP. I've attached some images that cause this problem when handled with the simagef CLI tool, which depends on image-match.

thread '<unnamed>' panicked at /home/gert/.cargo/registry/src/index.crates.io-6f17d22bba15001f/image-match-0.2.3/src/lib.rs:347:9:
index out of bounds: the len is 57 but the index is 18446744073709551615
stack backtrace:
   0:     0x55fcbad4a5a6 - std::backtrace_rs::backtrace::libunwind::trace::hd6ffdc229294c06f
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0x55fcbad4a5a6 - std::backtrace_rs::backtrace::trace_unsynchronized::h991f79b6e9960513
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55fcbad4a5a6 - std::sys_common::backtrace::_print_fmt::hf4ebe716f7ccda10
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x55fcbad4a5a6 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h410d4c66be4e37f9
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55fcbad76a00 - core::fmt::rt::Argument::fmt::he0ff0e0e188c0db5
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/fmt/rt.rs:142:9
   5:     0x55fcbad76a00 - core::fmt::write::he40921d4802ce2ac
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/fmt/mod.rs:1120:17
   6:     0x55fcbad473af - std::io::Write::write_fmt::h5de5a4e7037c9b20
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/io/mod.rs:1846:15
   7:     0x55fcbad4a384 - std::sys_common::backtrace::_print::hd8e2e3f655086663
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x55fcbad4a384 - std::sys_common::backtrace::print::h11c067a88e3bdb22
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x55fcbad4bc07 - std::panicking::default_hook::{{closure}}::h8c832ecb03fde8ea
  10:     0x55fcbad4b969 - std::panicking::default_hook::h1633e272b4150cf3
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/panicking.rs:292:9
  11:     0x55fcbad4c108 - std::panicking::rust_panic_with_hook::hb164d19c0c1e71d4
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/panicking.rs:781:13
  12:     0x55fcbad4bfe2 - std::panicking::begin_panic_handler::{{closure}}::h0369088c533c20e9
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/panicking.rs:659:13
  13:     0x55fcbad4aaa6 - std::sys_common::backtrace::__rust_end_short_backtrace::hc11d910daf35ac2e
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/sys_common/backtrace.rs:171:18
  14:     0x55fcbad4bd34 - rust_begin_unwind
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/panicking.rs:647:5
  15:     0x55fcbad746d5 - core::panicking::panic_fmt::ha6effc2775a0749c
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/panicking.rs:72:14
  16:     0x55fcbad74912 - core::panicking::panic_bounds_check::h5aa5e8a957e001f9
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/panicking.rs:208:5
  17:     0x55fcba8952d9 - image_match::pixel_average::{{closure}}::h722d11f74568673f
                               at /home/gert/.cargo/registry/src/index.crates.io-6f17d22bba15001f/image-match-0.2.3/src/lib.rs:347:9
  18:     0x55fcba88c8ea - core::iter::adapters::map::map_fold::{{closure}}::heedff0d92f9449cc
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/iter/adapters/map.rs:89:28
  19:     0x55fcba88fac5 - <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold::h7fa1741582d761f8
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/slice/iter/macros.rs:232:27
  20:     0x55fcba88aef8 - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold::h4108b8b47a95493e
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/iter/adapters/map.rs:129:9
  21:     0x55fcba89702a - <f32 as core::iter::traits::accum::Sum>::sum::h44f8bc6bcc2d2f0c
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/iter/traits/accum.rs:106:17
  22:     0x55fcba88c2e7 - core::iter::traits::iterator::Iterator::sum::h4ba14a1b8f104568
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/iter/traits/iterator.rs:3642:9
  23:     0x55fcba8951eb - image_match::pixel_average::hb4b617c15652b4b4
                               at /home/gert/.cargo/registry/src/index.crates.io-6f17d22bba15001f/image-match-0.2.3/src/lib.rs:346:20
  24:     0x55fcba89442b - image_match::grid_averages::h6cd0e95f393a956d
                               at /home/gert/.cargo/registry/src/index.crates.io-6f17d22bba15001f/image-match-0.2.3/src/lib.rs:231:31
  25:     0x55fcba893079 - image_match::compute_from_gray::hdae111ba5f040a86
                               at /home/gert/.cargo/registry/src/index.crates.io-6f17d22bba15001f/image-match-0.2.3/src/lib.rs:82:20
  26:     0x55fcba85bbec - image_match::image::get_image_signature::he89c5e3a6a0b3d58
                               at /home/gert/.cargo/registry/src/index.crates.io-6f17d22bba15001f/image-match-0.2.3/src/image.rs:27:5
  27:     0x55fcba803710 - simagef::signature_maker_loop::h0ea9207fe826444b
                               at /home/gert/projects/simagef/src/main.rs:49:41
  28:     0x55fcba8461f8 - simagef::main_signatures::{{closure}}::h928e5333e89ab675
                               at /home/gert/projects/simagef/src/main.rs:151:13
  29:     0x55fcba8008a6 - std::sys_common::backtrace::__rust_begin_short_backtrace::h1673223e12e95831
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/sys_common/backtrace.rs:155:18
  30:     0x55fcba832c11 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h5caf9760a9e6e7de
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/thread/mod.rs:529:17
  31:     0x55fcba7eb091 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4322027291b61447
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/panic/unwind_safe.rs:272:9
  32:     0x55fcba806512 - std::panicking::try::do_call::h48891d38303a7adf
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/panicking.rs:554:40
  33:     0x55fcba806abb - __rust_try
  34:     0x55fcba806251 - std::panicking::try::h647d3d7391d2428a
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/panicking.rs:518:19
  35:     0x55fcba8314db - std::panic::catch_unwind::h840f7459987f1ff3
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/panic.rs:142:14
  36:     0x55fcba8314db - std::thread::Builder::spawn_unchecked_::{{closure}}::h2e400ca4a080f320
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/thread/mod.rs:528:30
  37:     0x55fcba81594e - core::ops::function::FnOnce::call_once{{vtable.shim}}::h9b7526c4b978719d
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/core/src/ops/function.rs:250:5
  38:     0x55fcbad50815 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h6b630278c760b971
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/alloc/src/boxed.rs:2015:9
  39:     0x55fcbad50815 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h70462b441b6c0e1f
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/alloc/src/boxed.rs:2015:9
  40:     0x55fcbad50815 - std::sys::pal::unix::thread::Thread::new::thread_start::h3631815ad38387d6
                               at /rustc/25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04/library/std/src/sys/pal/unix/thread.rs:108:17
  41:     0x75cd0bbe155a - <unknown>
  42:     0x75cd0bc5ea3c - <unknown>
  43:                0x0 - <unknown>

Release v0.2.4

I want to get the fix for #1 out, but if there's any in-progress code you feel is worth folding in let's get them released together. I assume I'll need to do something in the release process to get this up to crates.io, though down the road we can work to expand that.