Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
Lance Namespace Specification is an open specification on top of the storage-based Lance data format to standardize access to a collection of Lance tables
Distributed query engine providing simple and reliable data processing for any modality and scale
The native Rust implementation for Apache Hudi, with C++ & Python API bindings.
An extensible, state of the art columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.
Apache OpenDAL: One Layer, All Storage.
Olympia is a storage-only open catalog format for big data analytics, ML & AI.
Developer-friendly, embedded retrieval engine for multimodal AI. Search More; Manage Less.
A high-performance distributed file system designed to address the challenges of AI training and inference workloads.
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Apache Spark - A unified analytics engine for large-scale data processing
Apache Iceberg