jbenet / data Goto Github PK

get blob adbf522 train-labels-idx1-ubyte (X/45MB) 
get blob adbf522 train-labels-idx1-ubyte (X% of 45MB) 
get blob adbf522 train-labels-idx1-ubyte 45MB (X%)

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you REMOVE data.
MINOR version when you ADD data in a backwards-compatible manner, and
PATCH version when you CLEAN or REFORMAT data, without ADDING or REMOVING values.

data user add [desired_username]
cd directory_where_my_data_resides/
data publish

go install github.com/data/data

  blob        Manage blobs in the blobstore
    put <hash> [<path>]   Upload blob named <hash> from <path> to blobstore.
    get <hash> [<path>]   Download blob named <hash> from blobstore to <path>.
    check <hash> [<path>] Verify blob contents in <path> match <hash>.
    url <hash>            Output Url for blob named by <hash>.
    show <hash>           Output blob contents named by <hash>.

data get <package tarball file>
data get <package tarball url>
data get <package directory>

data get # in a package dir, uses `Datafile.dependencies`
data get <owner>/<name> [--save]
data get <owner>/<name>@<version>  [--save]

everett:sfmfp beholder$ data publish
==> Guided Data Package Publishing.

Welcome to Data Package Publishing. You should read these short
messages carefully, as they contain important information about
how data works, and how your data package will be published.

First, a 'data package' is a collection of files, containing:
- various files with your data, in any format.
- 'Datafile', a file with descriptive information about the package.
- 'Manifest', a file listing the other files in the package and their checksums.

This tool will automatically:
1. Create the package
  - Generate a 'Datafile', with information you will provide.
  - Generate a 'Manifest', with all the files in the current directory.
2. Upload the package contents
3. Publish the package to the index

(Note: to specify which files are part of the package, and other advanced
 features, use the 'data pack' command directly. See 'data pack help'.)

You are not logged in. First, either:

- Run 'data user add' to create a new user account.
- Run 'data user auth' to log in to an existing user account.


Why does publishing require a registered user account (and email)? The index
service needs to distinguish users to perform many of its tasks. For example:

- Verify who can or cannot publish datasets, or modify already published ones.
  (i.e. the creator + collaborators should be able to, others should not).
- Profiles credit people for the datasets they have published.
- Malicious users can be removed, and their email addresses blacklisted to
  prevent further abuse.

$ data install
# downloads datasets given Datafile

$ data install [-arg to specify datafile] Datafile.staging

data blob - Manage blobs in the blobstore.

    Managing blobs means:

      put <hash>    Upload blob named by <hash> to blobstore.
      get <hash>    Download blob named by <hash> from blobstore.
      check <hash>  Verify blob contents named by <hash> match <hash>.
      show <hash>   Output blob contents named by <hash>.


    What is a blob?

    Datasets are made up of files, which are made up of blobs.
    (For now, 1 file is 1 blob. Chunking to be implemented)
    Blobs are basically blocks of data, which are checksummed
    (for integrity, de-duplication, and addressing) using a crypto-
    graphic hash function (sha1, for now). If git comes to mind,
    that's exactly right.

    Local Blobstores

    data stores blobs in blobstores. Every local dataset has a
    blobstore (local caching with links TBI). Like in git, the blobs
    are stored safely in the blobstore (different directory) and can
    be used to reconstruct any corrupted/deleted/modified dataset files.

    Remote Blobstores

    data uses remote blobstores to distribute datasets across users.
    The datadex service includes a blobstore (currently an S3 bucket).
    By default, the global datadex blobstore is where things are
    uploaded to and retrieved from.

    Since blobs are uniquely identified by their hash, maintaining one
    global blobstore helps reduce data redundancy. However, users can
    run their own datadex service. (The index and blobstore are tied
    together to ensure consistency. Please do not publish datasets to
    an index if blobs aren't in that index)

    data can use any remote blobstore you wish. (For now, you have to
    recompile, but in the future, you will be able to) Just change the
    datadex configuration variable. Or pass in "-s <url>" per command.

    (data-blob is part of the plumbing, lower level tools.
    Use it directly if you know what you're doing.)

# npm: Interactively create a package.json file
npm init

# Ruby: generate a blank Gemfile
$ bundle init