josephcatrambone / pixelbox

Image Search Engine in Rust

PixelBox

A desktop image search and indexing tool.

---

PixelBox is still pre-alpha. Database schema and feature prioritization are subject to change.

Features

• Cross-platform (Windows, Linux, MacOS) and FOSS
• Search across filenames and exif tags
• Drag and drop search for visually similar images
• Fast parallel indexing of images
• User-moddable image similarity engine (!)
• Portable and inspectable database format

Technologies

• Rust as the primary language (with egui and tract-onnx)
• SQLite as a storage medium for the image database
• Torch for training the image similarity model
• ONNX for running the similarity model

TODOs for Alpha Release

• Compress thumbnails in database [DONE - 2x Compression for No Loss in Speed]
• Remove from index on folder clear [DONE]
• Settings Page [DONE]
• Start removing those unwraps

TODOs for Roadmap

• Better similarity search
• OCR for images (search on text in images)
• Editable tags
• Face search
• Search on image contents in plaintext
• Watched directories via notify crate
• If a model is unavailable, don't perform image hash and just disable similarity search so people can use it for just tags
• Index inside of zip files

Project Structure

• .github - Links to demo pictures for readme and, eventually, CI/GitHub Action build scripts
• models - The final ONNX files to be used by the application for visual similarity
• resources - Non-shipped experiment logs and python training files
• src - The main application code
  • image_hashes - Wrappers for different image hashing methods
  • ui - Code for each of the major UI panels like search view, folder view, etc.

Using Your Own Image Hash (Advanced)

PixelBox's search uses the cosine distance between byte-quantified n-dimensional floats. For example, if you represent your image as [-1.0, 1.0, 0.0, 0.1] then this will be mapped to a 4-byte vector of [0x00, 0xFF, 0x80, 0x8C].

There are two ways to use your own image hash methods:

1. Replace the image_similarity.onnx file with your own trained model. The inputs should be channel-first 128x128 RGB images and the outputs should be a 1D vector of floats between -1 and 1. See image_hashes/efficientnet.rs for constraints.
2. Replace the 'hash' in the 'semantic_hash' table of your database. This should be an array of u8s as described above. You will not be able to drag-and-drop images for search if using this approach, but after finding a seed image you can right-click and do 'find similar'.

pixelbox's Issues

Search Inside of ZIP and Archive Files

Indexing of images is restricted to plain image files on disk and in directories and does not include compressed or archived files. We should search and index inside of zip files, gz files, etc, so people who don't uncompress their assets can still search files.

This will mean a nontrivial change in how the file spidering is done and it might be a real pain in the butt.

Creating a new database on top of an existing database causes a crash.

Observed behavior:

If I create a database FOO.db in a folder, close it, and then create another database also named FOO.db in the same folder, the application crashes with thread 'main' panicked at 'called Result::unwrap()on anErr value: SqliteFailure(Error { code: Unknown, extended_code: 1 }, Some("table images already exists"))', src\engine.rs:107:50.

Suspicion:

SQLite is attempting to open the file instead of creating a new one. When we invoke the open method we should make sure the parameters are set and that we're forcing create OR we should remove the old DB.

UX: Better Feedback During Indexing Process

Indexing is a black box right now (unless someone is looking at the console). It would be good to give a forecast of completion time and progress.

Thumbnail Aspect Ratio is Broken

Somewhere between introducing compression and thumbnail resizing I broke the aspect ratio of the thumbnails. I think it was the compression PR because now that we don't have THUMBNAIL_SIZE in the database (as it's part of the QOI container) it's hard to get into the app front-end.

Database Thumbnails are Uncompressed

The thumbnails stored in SQLite are raw! These should be compressed with something quick and elegant like QOI or JPG.

Split Image Similarity into 'Style', 'Contents', and 'Visual Resemblance'.

"Search by Image" does not distinguish between searching on contents, searching on the style, and searching by an exact match.

Add Drag and Drop to Image Search

In addition to hitting "Search by Image", support drag-and-drop of images.

Search by Image Right Click Context is not Working

Right clicking on an image result and clicking 'search for similar' does nothing.

Use RetainedImage instead of TextureReference in the Thumbnail Cache

Using raw texture handles is perhaps an anti-pattern. The creator of egui has recommended using RetainedImage in the thumbnail cache. emilk/egui#1424 (comment) Replace references to texturehandle.

Search Result Thumbnails Disappear after ~16ms in 0.17 Upgrade Branch

Upgrading to 0.17 required some nontrivial tweaks to how images were being cached and loaded. It's not clear how the break happened, but I'm guessing it's something to do with the thumbnail cache and fetching texture_ids.

pixelbox_2022-03-25_23-02-21.mp4

Prior to 0.17 the issue did not exist.

Switching Databases Does Not Clear Thumbnail Cache

If one opens a database file and does a search then opens another database, the cached thumbnails are from the previous database.

Searching after indexing triggers a SqliteFailure: No such function.

Full trace boils down to: thread 'main' panicked at 'called 'Result::unwrap()' on an 'Err' value: SqliteFailure(Error { code: Unknown, extended_code: 1 }, Some("no such function: cosine_distance"))', src\engine.rs:303:26

For some reason, the cosine_distance function that is declared after a database is created doesn't seem to stick. Hard to tell why because it's internal to SQLite. The function is definitely defined -- it's one of the first things that's done. Doesn't seem to matter if we commit or not.

pub fn open(filename:&Path) -> Self {
		let manager = SqliteConnectionManager::file(filename);
		let pool = r2d2::Pool::new(manager).unwrap();

		let mut conn = pool.get().unwrap();
		make_hamming_distance_db_function(&conn);
		make_byte_distance_db_function(&conn);
		make_cosine_distance_db_function(&conn);
		conn.transaction().unwrap().commit();