tempbottle / cannyls

An embedded persistent key-value storage for Rust that optimized for random-access workload and huge-capacity HDD

cannyls

CannyLS is an embedded and persistent key-value storage optimized for random-access workload and huge-capacity HDD.

CannyLS mainly has following features:

• A local storage for storing objects that called as "lump":
  • Basically, a lump is a simple key-value entry
  • The distinctive properties are that the key is fixed length (128 bits) and suited for storing a relatively large size value (e.g., several MB)
• Provides simple functionalities:
  • Basically, the only operations you need to know are PUT, GET and DELETE
  • But it supports deadline based I/O scheduling as an advanced feature
• Optimized for random-access workload on huge-capacity HDD (up to 512 TB):
  • See Benchmark Results for more details about performance
• Aiming to provide predictable and stable read/write latency:
  • There are (nearly) strict upper bounds about the number of disk accesses issued when executing operations
    • One disk access when PUT and DELETE, and two when PUT
  • There are no background processings like compaction and stop-the-world GC which may block normal operations for a long time
  • For eliminating overhead and uncertainty, CannyLS has no caching layer:
    • It uses Direct I/O for bypassing OS layer caching (e.g., page cache)
    • If you need any caching layer, it is your responsibility to implement it
• Detailed metrics are exposed using Prometheus

See Wiki for more details about CannyLS.

Documentation

• Rustdoc
• Wiki (Japanese only)