Oxen is a lightning fast data version control system for structured and unstructured machine learning datasets. We aim to make versioning datasets as easy as versioning code.
The interface mirrors git, but shines in many areas that git or git-lfs fall short. Oxen is built from the ground up for data, and is optimized to handle large datasets, and large files.
oxen init
oxen add images/
oxen add annotations/*.parquet
oxen commit "Adding 200k images and their corresponding annotations"
oxen push origin mainOxen is comprised of a command line interface, as well as bindings for Rust 🦀, Python 🐍, and HTTP interfaces 🌎 to make it easy to integrate into your workflow.
Oxen is designed to efficiently manage large datasets, including those with large individual files, for example CSV files with millions of rows. It also handles datasets comprising millions of individual files and directories such as the complete collection of ImageNet images.
One of the main reasons datasets are hard to maintain is the pure performance of indexing the data and transferring the data over the network. We wanted to be able to index hundreds of thousands of images, videos, audio files, and text files in seconds.
Watch below as we version hundreds of thousands of images in seconds 🔥
But speed is only the beginning.
Oxen is built around ergonomics, ease of use, and it is easy to learn. If you know how to use git, you know how to use Oxen.
- 🔥 Fast (efficient indexing and syncing of data)
- 🧠 Easy to learn (same commands as git)
- 💪 Handles large files (images, videos, audio, text, parquet, arrow, json, models, etc)
- 🗄️ Index lots of files (millions of images? no problem)
- 📊 Native DataFrame processing (index, compare and serve up DataFrames)
- 📈 Tracks changes over time (never worry about losing the state of your data)
- 🤝 Collaborate with your team (sync to an oxen-server)
- 🌎 Remote Workspaces to interact with the data without downloading it
- 👀 Better data visualization on OxenHub
To learn what everything Oxen can do, the full documentation can be found at https://docs.oxen.ai.
You can install through homebrew or pip or from our releases page.
brew tap Oxen-AI/oxen
brew install oxenpip install oxenaiClone your first Oxen repository from the OxenHub.
oxen clone https://hub.oxen.ai/ox/CatDogBBoxNo really.
We hooked up the GitHub webhook for stars to an OxenHub Repository. Learn how we did it and go find your own in our ox/FlyingOxen repository.
If you have any questions, comments, suggestions, or just want to get in contact with the team, feel free to email us at [email protected]
This repository contains the Python library that wraps the core Rust codebase. We would love help extending out the python interfaces, the documentation, or the core rust library.
Code bases to contribute to:
If you are building anything with Oxen.ai or have any questions we would love to hear from you in our discord.
Set up virtual environment:
# Set up your python virtual environment
$ python -m venv ~/.venv_oxen # could be python3
$ source ~/.venv_oxen/bin/activate
$ pip install maturin# Install rust
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Run maturin
$ maturin developOxen was build by a team of machine learning engineers, who have spent countless hours in their careers managing datasets. We have used many different tools, but none of them were as easy to use and as ergonomic as we would like.
If you have ever tried git lfs to version large datasets and became frustrated, we feel your pain. Solutions like git-lfs are too slow when it comes to the scale of data we need for machine learning.
If you have ever uploaded a large dataset of images, audio, video, or text to a cloud storage bucket with the name:
s3://data/images_july_2022_final_2_no_really_final.tar.gz
We built Oxen to be the tool we wish we had.
"Oxen" 🐂 comes from the fact that the tooling will plow, maintain, and version your data like a good farmer tends to their fields 🌾. Let Oxen take care of the grunt work of your infrastructure so you can focus on the higher-level ML problems that matter to your product.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent