jfrankle / streaming

A Data Streaming Library for Efficient Neural Network Training

A Data Streaming Library for Efficient Neural Network Training

[Website] - [Getting Started] - [Docs] - [We're Hiring!]

👋 Welcome

Streaming is a PyTorch compatible dataset that enables users to stream training data from cloud-based object stores. Streaming can read files from local disk or from cloud-based object stores. As a drop-in replacement for your PyTorch IterableDataset class, it’s easy to get streaming:

dataloader = torch.utils.data.DataLoader(dataset=ImageStreamingDataset(remote='s3://...'))

Please check the quick start guide and user guide on how to use the Streaming Dataset.

Key Benefits

• High performance, accurate streaming of training data from cloud storage
• Efficiently train anywhere, independent of training data location
• Cloud-native, no persistent storage required
• Enhanced data security—data exists ephemerally on training cluster

🚀 Quickstart

💾 Installation

Streaming is available with Pip:

pip install mosaicml-streaming

Examples

Please check our Examples section for the end-to-end model training workflow using Streaming datasets.

📚 Documentation

Getting started guides, examples, API reference, and other useful information can be found in our docs.

💫 Contributors

We welcome any contributions, pull requests, or issues!

To start contributing, see our Contributing page.

P.S.: We're hiring!

✍️ Citation

@misc{mosaicml2022streaming,
    author = {The Mosaic ML Team},
    title = {streaming},
    year = {2022},
    howpublished = {\url{https://github.com/mosaicml/streaming/}},
}