GithubHelp home page GithubHelp logo

tsturzl / sled Goto Github PK

View Code? Open in Web Editor NEW

This project forked from spacejam/sled

0.0 3.0 0.0 4.92 MB

sled likes eating data: alpha modern embedded database

License: Apache License 2.0

Rust 97.81% JavaScript 0.13% Python 1.53% Shell 0.54%

sled's Introduction

sled - it's all downhill from here!!!

Build Status crates.io documentation chat

An (alpha) modern embedded database. Doesn't your data deserve an (alpha) beautiful new home?

use sled::Db;

let tree = Db::start_default(path)?;

// set and get
tree.insert(k, v1);
assert_eq!(tree.get(&k), Ok(Some(v1)));

// compare and swap
tree.cas(k, Some(v1), Some(v2));

// scan forward
let mut iter = tree.scan(k);
assert_eq!(iter.next(), Some(Ok((k, v2))));
assert_eq!(iter.next(), None);

// deletion
tree.remove(&k);

// block until all operations are on-disk
tree.flush();

We also support merge operators!

features

  • API similar to a threadsafe BTreeMap<Vec<u8>, Vec<u8>>
  • fully atomic single-key operations, supports CAS
  • zero-copy reads
  • subscription/watch semantics on key prefixes
  • multiple keyspace support
  • merge operators
  • forward and reverse iterators
  • a crash-safe monotonic ID generator capable of generating 75-125 million ID's per second
  • zstd compression (use the compression build feature)
  • cpu-scalable lock-free implementation
  • SSD-optimized log-structured storage

architecture

lock-free tree on a lock-free pagecache on a lock-free log. the pagecache scatters partial page fragments across the log, rather than rewriting entire pages at a time as B+ trees for spinning disks historically have. on page reads, we concurrently scatter-gather reads across the log to materialize the page from its fragments. check out the architectural outlook for a more detailed overview of where we're at and where we see things going!

goals

  1. don't make the user think. the interface should be obvious.
  2. don't surprise users with performance traps.
  3. don't wake up operators. bring reliability techniques from academia into real-world practice.
  4. don't use so much electricity. our data structures should play to modern hardware's strengths.

plans

  • LSM tree-like write performance with traditional B+ tree-like read performance
  • MVCC, serializable transactions, and snapshots
  • forward-compatible binary format
  • concurrent snapshot delta generation and recovery
  • first-class programmatic access to replication stream
  • consensus protocol for PC/EC systems
  • pluggable conflict detection and resolution strategies for PA/EL systems
  • multiple collection types like tables, queues, Merkle trees, bloom filters, etc... unified under a single transactional and operational domain

fund feature development and get commercial support

Want to support the project, prioritize a specific feature, or get commercial help with using sled in your project? Ferrous Systems provides commercial support for sled, and can work with you to solve a wide variety of storage problems across the latency-throughput, consistency, and price performance spectra. Get in touch!

Ferrous Systems

special thanks

Meili

Special thanks to Meili for providing engineering effort and other support to the sled project. They are building an event store backed by sled, and they offer a full-text search system which has been a valuable case study helping to focus the sled roadmap for the future.

Additional thanks to Arm, Works on Arm and Packet, who have generously donated a 96 core monster machine to assist with intensive concurrency testing of sled. Each second that sled does not crash while running your critical stateful workloads, you are encouraged to thank these wonderful organizations. Each time sled does crash and lose your data, blame Intel.

known issues, warnings

  • the on-disk format is going to change in non-forward compatible ways before the 1.0.0 release! after that, we will always support forward migrations.
  • quite young, should be considered unstable for the time being
  • the C API is likely to change rapidly
  • writepath is not well optimized yet. readpath is essentially wait-free and zero-copy.

contribution welcome!

want to help advance the state of the art in open source embedded databases? check out CONTRIBUTING.md!

References

sled's People

Contributors

spacejam avatar kerollmops avatar jeehoonkang avatar siler avatar spacekookie avatar pmuens avatar dependabot-support avatar dependabot-preview[bot] avatar mbr avatar rubdos avatar dependabot[bot] avatar rleungx avatar nrc avatar lambda avatar jneem avatar lucab avatar jamesmunns avatar dotcypress avatar anxiousmodernman avatar jeizsm avatar matthiasn avatar vorner avatar rtsuk avatar sunisdown avatar yoshuawuyts avatar

Watchers

Travis Sturzl avatar James Cloos avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.