mhassan102/quilt

Quilt - manage data like code

★ 0Forks 0PythonGitHub ↗Compare

Project website ↗

README

PyPI Gitter

OS master Python support
Linux 2.7, 3.4, 3.5, 3.6
CircleCI branch 2.7, 3.5, 3.6
Windows 3.5, 3.6

Docs

Visit docs.quiltdata.com. Or browse the docs on GitHub.

Quilt is a data router

With Quilt you can build, push, and install data packages. Data packages are versioned, reusable data structures that can be loaded into Python. Quilt is designed to support reproducible, auditable, and compliant workflows.

Core features include:

  • Versioning and storage of large data
  • Transformation of a variety of file formats into data frames (via pandas and pyarrow)
  • De-duplication of repeated data for reduced disk and network footprint

Commands

Here are the basic Quilt commands:

Architecture

Quilt consists of three components:

  1. A data catalog

    • Displays package meta-data in HTML
  2. A data registry

    • Controls permissions
    • Stores pacakge fragments in blob storage
    • Stores package meta-data
    • De-duplicates repeated data fragments
  3. A data compiler

    • Serializes tabular data to Apache Parquet
    • Transforms and parses files
    • builds packages locally
    • pushes packages to the registry
    • pulls packages from the registry

Contributors

dimaryazakarvekevinemooreeodeasahmeffijmhassan102diwu1989rinman24

Issues