pingali / dgit Goto Github PK

dgit capture --schema --datatypes --samples or

Database backends:

1. Postgres, hive (native and qubole), hbase

Transformations:

1. Analysis of dataset contents for outliers etc.

Add checkers
dgit add --validator balance-check.py

Now validate
dgit validate

Some possible integration points:

1. Make dgit repos available through dat for sharing and discovery
2. Extend dat to support validation, materialization etc.

http://dat-data.com

Instead of having a separate metadata server, how about converting it into an RST file that can be searched/viewed in github itself. This will reduce one service depenedency.

Support a generator script - accessing the dataset's content
requires running the generator script (bash?)

The name is conflicting with multiple projects
(a) debian project
(b) github's own distributed data git

Some alternative names:
(a) dmgit (data management git)
(b) mgit (metadata git)
(c) git-data-manage
... Anything else?

Use libraries like s3git-go instead of current approach (simulate a git server and sync with s3)

Call the instrumentation module.

Has to show the following (ideally in a single tutorial):

1. Base usecase
2. Validation
3. Transformation

Check for 2/3 compatibility

dgit extract

Right now we only have a generic shell command to use.

dgit sh cp -r <filename or pattern ~/target

This extract should
(a) leave a log action record
(b) incorporate git commit information
~/target/username/reponame//filename

Especially the API-based usecases

circle.yml
travis.yml etc

When repos are cloned from s3, the username is being replaced.
It should not be.

[Wrong] DrDrake/regression.git -> pingali/regression.git

Create a new repository from existing one using
the result of a transformation command

https://github.com/recipy/recipy

Implement assertr like assertions as a validation module

https://github.com/tonyfischetti/assertr

1. Add support for including snippets, readme using a separate command like

dgit add --snippet

1. Add support for adding resource links

dgit add --link s3://....

This resource is, for some reason, outside the git (e.g., too big)