r4ntix / incubator-opendal

Apache OpenDAL: Access data freely, painlessly, and efficiently.

---

OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.

Major components of the project include:

Libraries

• Rust Core
• Node.js Binding
• Python Binding
• C Binding being worked on
• Java Binding being worked on
• Ruby Binding being worked on

Applications

• oli: OpenDAL Command Line Interface
• oay: OpenDAL Gateway

Services

Standard Storage Protocols (like fs, ftp)

• fs: POSIX alike file system
• ftp: FTP and FTPS
• http: HTTP read-only services
• sftp: SFTP services being worked on
• webdav: WebDAV Service

Object Storage Services (like s3, gcs, azblob)

• azblob: Azure Storage Blob services
• gcs: Google Cloud Storage Service
• obs: Huawei Cloud Object Storage Service (OBS)
• oss: Aliyun Object Storage Service (OSS)
• s3: AWS S3 alike services
• supabase: Supabase Storage Service being worked on
• wasabi: Wasabi Cloud Storage

File Storage Services (like azdfs, hdfs)

• azdfs: Azure Data Lake Storage Gen2 services (As known as abfs)
• hdfs: Hadoop Distributed File System(HDFS)
• ipfs: InterPlanetary File System HTTP Gateway
• ipmfs: InterPlanetary File System MFS API being worked on
• webhdfs: WebHDFS Service

Consumer Cloud Storage Service (like gdrive, onedrive)

• gdrive: Google Drive being worked on
• onedrive: OneDrive being worked on

Key-Value Storage Service (like rocksdb, sled)

• dashmap: dashmap backend
• memory: In memory backend
• redis: Redis services
• rocksdb: RocksDB services
• sled: sled backend

Cache Storage Service (like memcached, moka)

• ghac: Github Action Cache Service
• memcached: Memcached service
• moka: moka backend
• vercel_artifacts: Vercel Remote Caching Service being worked on

Welcome to add any services that are not currently supported here.

Documentation

The documentation is available at https://opendal.apache.org.

We are engaged in a practice we call "documentation as code". You can also view the documentation directly in project's source code. And we welcome you to contribute to the documentation.

Contribute

OpenDAL is an active open-source project. We are always open to people who want to use it or contribute to it. Here are some ways to go.

• Start with Contributing Guide.
• Submit Issues for bug report or feature requests.
• Asking questions in the Discussions.
• Talk to community directly at Discord.
• Subscribe our dev mailing list, then you can use it to ask questions, discuss design and implementation, etc. View the archive at https://lists.apache.org/[email protected].

Who is using OpenDAL?

• Databend: A modern Elasticity and Performance cloud data warehouse.
• GreptimeDB: An open-source, cloud-native, distributed time-series database.
• deepeth/mars: The powerful analysis platform to explore and visualize data from blockchain.
• mozilla/sccache: sccache is ccache with cloud storage
• RisingWave: A Distributed SQL Database for Stream Processing
• Vector: A high-performance observability data pipeline.

License

Licensed under the Apache License, Version 2.0