ornamental / kvp-store

A simplistic key-value store with multi-version concurrency control

A simplistic embedded MVCC key-value storage

The storage organizes the data supplied to it in a log-like manner, writing the records one after another until a file reaches maximum allowed size, after which a new file is created and written to, and so on. The metadata is stored in an SQLite database.

The storage allows any number of concurrent reading transactions and a single writing transaction (not blocked by the reading transactions). Each reading transactions observes the storage snapshot which was relevant at the moment when the transaction was started. The writing transaction operates on the most recent storage state.

If a record is deleted and there are no readers which observe the corresponding storage state, it may be garbage-collected. The garbage collection does not affect a file unless the records ready to be collected occupy sufficiently much space in it. The garbage collection on a file moves all the remaining records to the end of the 'log' and removes the file that has become redundant.

The storage handles very big writing transactions (up to terabytes) without much impact on the RAM. It works best with shorter keys.

Usage

The principal interface is VersionedStorage serving to begin reading and modifying transactions on an MVCC storage. Read-only transactions are represented by the Reader interface, modifying transactions are represented by the Writer interface, both extending java.lang.AutoCloseable (in fact, Writer is a Reader). The supplied concrete implementation of the VersionedStorage interface is the class GroupingFileStorage.

Keys and values supplied to the storage transaction methods are represented by the Key interface, whose implementations must be able to return a byte array representing a key, and the Value interface, whose implementations must be aware of the size of data they hold and (in its most general form) be able to return a ReadableByteChannel as the data supplier.

Note that the Key implementations must override equals(Object) and hashCode() methods so that instances returning equal binary representations are equal. The default behaviour is implemented in the AbstractKey class.

There exist ready implementations of the Value interface wrapping byte[] and InputStream instances (ByteArrayValue and InputStreamValue, respectively).

Build prerequisites

The project depends on another GitHub project @ornamental/sqlite-statements. Build and install the corresponding artifact into local Maven repository before building this project.

kvp-store's Issues

Eliminate the need to supply full configuration when opening an existing store

Supplying the parameters other than those specified upon the creation of the storage shall lead to undefined storage behaviour. Most parameters have to be saved as a part of the storage metadata and automatically restored when the storage is reopened.

Improve the SQL statements using the new capabilities of the Table class

The instances of Table may now be passed as arguments to many methods where table name is expected.

Use FileChannel to fetch only as much data from a record as the client requires

Currently the data is read from the record into a byte array (fully). The interface allows to return a more general accessor based on channels, which has to be used.

Eliminate dependency on ORMLite

The ORM functionality is not employed at all as only raw SQL queries are sent to SQLite. Thus there is no need in using ORMLite, JDBC functionality combined with connection pooling should be sufficient.

1. Implement JDBC connection pooling.
2. Consider preparing certain statements on per-connection basis (as there will be only one RW connection, the INSERT, UPDATE, and DELETE statements only have to be prepared for this connection; RO queries will have to be prepared for each connection in the pool).
3. Migrate to the new querying mechanism and exclude the dependencies which will become redundant.

Implement a log-like storage governed by an embedded database

Requirements

1. The storage must support any number of concurrent readers and one writer not blocked by the readers.
2. Each reader must observe the snapshot of the storage relevant at the moment of its creation; it must not see the changes added afterwards, either committed or not.
3. The writer must observe the most relevant state with its own uncommitted changes.
4. The writer must support commit and rollback operations.
5. However much data a writing transaction might supply, the RAM footprint must not grow proportionally.
6. The storage must be able to transfer data from byte arrays, input streams, and readable byte channels.

Add a Value implementation fetching data from a file

The user should be able to specify the offset and length upon instance creation.