helmholtzai-consultants-munich / effidict

EffiDict

EffiDict is an efficient and fast Python package providing enhanced dictionary-like data structures with advanced caching capabilities. It's perfect for applications needing speedy retrieval and persistent key-value pair storage.

Features

LRU Caching: Implements Least Recently Used caching for optimal data access.

Persistent Storage: Supports disk storage with SQLite.

Versatile: Adaptable for various data types.

Installation

You can install EffiDict via pip:

pip install effidict

Usage

Importing the package

from effidict import LRUDBDict, LRUDict, DBDict

Using LRUDict for persistent storage on pickle files

cache_dict = LRUDict(max_in_memory=100, storage_path="cache")
cache_dict['key'] = 'value'

Using LRUDBDict for persistent storage on sqlite

db_cache_dict = LRUDBDict(max_in_memory=100, storage_path="cache.db")
db_cache_dict['key'] = 'value'

Standard DBDict (sqlite only)

db_dict = DBDict(storage_path="cache.db")
db_dict['key'] = 'value'

License

Licensed under the MIT License.

effidict's Issues

Add documentation with rtd

Add automated testing

Save and load as pickle file(s)

On deletion of a cache, also remove the associated file.db

Handle elements by batch

Serialize and deserialize by batches instead of single items to improve the speed of operations

Destroyer is incompatible with parallel execution (joblib)

Strange behaviour when using joblib, the cache folder gets deleted several times.

Quick fix: if the deletion is called by a _process_worker we don't delete anything