levarml / levar Goto Github PK
View Code? Open in Web Editor NEWMachine learning evaluation database
Home Page: http://levarml.github.io/LeVar/
License: Apache License 2.0
Machine learning evaluation database
Home Page: http://levarml.github.io/LeVar/
License: Apache License 2.0
Verify or recreate:
Include the Apache License v2.0 to this project
List datasets associated with an organization, user's default organization if none specified explicitly. Provide output in readable tabular format. Provide very high-level details for each dataset. CLI output must fit in 80 char width.
By default list datasets in user's default organization:
$ levar-cli datasets
Datasets for my-org
Dataset | Updated | Type | Items
---------------------------|------------|------------|----------
dataset-01 | 2015-10-10 | classify | 100200
dataset-02 | 2015-10-20 | regression | 1002020
Optionally specify organization:
$ levar-cli datasets my-other-org
Datasets for my-other-org
Dataset | Updated | Type | Items
---------------------------|------------|------------|----------
dataset-01 | 2015-10-10 | classify | 100200
dataset-02 | 2015-10-20 | regression | 1002020
Need to ensure that Postgesql dependency is handled smoothly
Ensure resources are published publicly and can be installed by client libraries easily
Upload a TSV experiment file to be associated with a dataset.
The TSV must have an 'id' column and an 'output' column -- other columns are optional. Values in 'id' must match the IDs in the dataset being evaluated against. The CLI reports an error and aborts if:
The CLI warns the user, but does not abort, if there are items in the original dataset not included in the experiment upload.
Here "adult" is an example of a dataset.
Basic usage:
$ levar-cli experiments upload adult adult-experiment-01.tsv
Experiment adult-experiment-01 for dataset my-org/adult uploaded
The organization associated with the dataset can be overridden by the org/dataset name convention:
$ levar-cli experiments upload my-other-org/adult adult-experiment-01.tsv
Experiment adult-experiment-01 for dataset my-other-org/adult uploaded
The name is guessed from the filename, but can be overridden with the the --name
or -n
flag:
$ levar-cli experiments upload -n new-results-001 adult adult-experiment-01.tsv
Experiment new-results-001 for dataset my-other-org/adult uploaded
$ levar-cli experiments upload --name new-results-002 adult adult-experiment-01.tsv
Experiment new-results-002 for dataset my-other-org/adult uploaded
If a name is given explicitly via the --name
flag, but there is already an experiment with that name associated with dataset's organization in the DB, the CLI reports an error and aborts the upload.
$ levar-cli experiments upload --name new-results-002 adult adult-experiment-01.tsv
Error: experiment named new-results-002 already exists
If the name is guessed from a filename upload, but an experiment with the same name is already in the DB associated with the dataset's organization, modify the name before creating the experiment in the DB:
$ levar-cli experiments upload adult adult-experiment-01.tsv
Experiment adult-experiment-01-2 for dataset my-org/adult uploaded
Reports:
You did not supply an argument -- try 'levar-cli datasets'
View an experiment with high level results analysis with CLI. Terminal output must fit within 80 chars horizontal.
For classification experiments, include overall accuracy and per-label + overall precision, recall, f-score. For less than 10 classes also include a confusion matrix.
Classification experiments:
$ levar-cli view experiment adult exp-0101
Experiment: exp-0101
Dataset: adult
Type: classify
Predictions: 32563 (full dataset coverage)
Results:
Overall accuracy: .931
Label | Prec | Rec | F1
---------------|------|------|------
<=50K | .952 | .917 | .934
>50K | .871 | .970 | .917
Macro-Avg | .912 | .944 | .926
Confusion matrix (columns give predicted counts; rows give gold counts)
Labels | <=50K | >50K | Totals
--------------|----------|------------|-----------
<=50K | 22690 | 1142 | 23832
>50K | 1118 | 7611 | 8729
Totals | 23808 | 8753 | 32561
Regression experiments:
$ levar-cli view experiment winequality-red exp-234
Experiment: exp-234
Dataset: winequality-red
Type: regression
Predictions: 1599 (full dataset coverage)
Results:
- RMSE: 4.2
- Mean abs error: 4.8
- Median abs error: 3.9
- 10-percentile err: 1.0
- 90-percentile err: 7.1
For debug only, not a proper UI
Prototype, nothing fancy.
Display high level details of a dataset to terminal. Output must fit within 80 chars horizontal.
Summary includes
Classification dataset:
$ levar-cli datasets view adult
Dataset: adult
Type: classify
Items: 32563
Created: 2015-10-10 16:23
Updated: 2015-10-10 18:01
Schema:
- age: text
- workclass: text
- fnlwgt: numeric
- education: text
- education-num: numeric
- marital-status text
- occupation text
- serv text
- relationship text
- race text
- sex text
- capital-gain numeric
- capital-loss numeric
- hours-per-week numeric
- native-country text
Classes:
- <=50K 75.9% (24720/32561)
- >50K 24.1% (7841/32561)
Tags: eval_for_study1, classification_datasets
Comments:
- system
2015-10-10 16:23
Created by user01 at 2015-10-10 16:23
- system
2015-10-10 18:01
400 items added by user01 at 2015-10-10 18:01
- user01
2015-10-10 18:04
added some more hard evaluation items to the dataset
Can override default organization:
$ levar-cli datasets view my-org/adult
Dataset: adult
Type: classify
...
Viewing summary stats for regression datasets:
$ levar-cli datasets view winequality-red
Dataset: winequality-red
Type: regression
Items: 1599
Created: 2015-10-10 16:23
Updated: 2015-10-10 18:01
Schema:
- fixed acidity numeric
- volatile acidity numeric
- citric acid numeric
- residual sugar numeric
- chlorides numeric
- free sulfur dioxide numeric
- total sulfur dioxide numeric
- density numeric
- pH numeric
- sulphates numeric
- alcohol numeric
Score distribution:
- min value 3.0
- max value 8.0
- mean 5.64
- median 6.0
- 10th percentile 5.0
- 90th percentile 7.0
Tags: eval_for_study1, classification_datasets
Comments:
- system
2015-10-10 16:23
Created by user01 at 2015-10-10 16:23
- system
2015-10-10 18:01
400 items added by user01 at 2015-10-10 18:01
- user01
2015-10-10 18:04
added some more hard evaluation items to the dataset.
Configure and save LeVar server credentials with CLI.
By default use $HOME/.levar/config.cfg
to store configurations. This can be configured by setting the $LEVAR_CONFIG
environment variable; i.e. the default of $LEVAR_CONFIG
is $HOME/.levar
.
Provide credentials via CLI options:
$ levar-cli config --username user01 --password secret --url http://localhost:9000 --org my-org
$ levar-cli config -u user01 -p secret -u http://localhost:9000 -o my-org
If no options are provided, prompt user for details:
$ levar-cli config
Username:
Password:
URL:
Organization:
If these details have already be configured and saved prompt the user. Don't actually display the password. If no user response is provided stay with the value already provided:
$ levar-cli config
Username [user-01]:
Password [********]:
URL [http://localhost:9000/]:
Organization [my-org]:
List experiments for a dataset with very high level summary. CLI output must fit in 80 char displays.
With user's default organization, she can just specify dataset name. For classification experiments, include accuracy, precision, recall and F score.
$ levar-cli experiments list dataset-01
Experiment | Updated | Predictions | Acc | Prec | Rec | F1
---------------------|------------|-------------|------|------|------|------
experiment-01 | 2015-10-12 | 1200 | .678 | .782 | .505 | .614
experiment-02 | 2015-10-12 | 1200 | .688 | .792 | .515 | .624
final-experiment | 2015-10-14 | 1200 | .698 | .802 | .525 | .634
Override default organization:
$ levar-cli experiments list my-org/dataset-01
Experiment | Updated | Predictions | Acc | Prec | Rec | F1
---------------------|------------|-------------|------|------|------|------
experiment-01 | 2015-10-12 | 1200 | .678 | .782 | .505 | .614
experiment-02 | 2015-10-12 | 1200 | .688 | .792 | .515 | .624
final-experiment | 2015-10-14 | 1200 | .698 | .802 | .525 | .634
For regression experiments, include RMSE:
$ levar-cli experiments list my-org/dataset-01
Experiment | Updated | Predictions | RMSE
---------------------|------------|-------------|---------
experiment-01 | 2015-10-12 | 1200 | 5.67
experiment-02 | 2015-10-12 | 1200 | 4.97
final-experiment | 2015-10-14 | 1200 | 4.45
And hook up Coveralls.io
Wireframe:
Experiment: dataset-foo/experiment-bar
Type: regression
Predictions: 4000 (full dataset coverage)
Results:
- RMSE: 4.1
- Mean abs error: 4.0
- Median abs error: 3.9
- 10-percentile err: 1.1
- 90-percentile err: 6.6
Quartile 1: 0.0 - 10.7
- RMSE: 5.2
- Mean abs error: 4.9
- Median abs error: 4.9
- 10-percentile err: 1.2
- 90-percentile err: 7.8
Quartile 2: 10.7 - 15.2
- RMSE: 5.2
- Mean abs error: 4.9
- Median abs error: 4.9
- 10-percentile err: 1.2
- 90-percentile err: 7.8
Quartile 3: 15.2 - 20.6
- RMSE: 5.2
- Mean abs error: 4.9
- Median abs error: 4.9
- 10-percentile err: 1.2
- 90-percentile err: 7.8
Quartile 4: 20.6 - 33.4
- RMSE: 5.2
- Mean abs error: 4.9
- Median abs error: 4.9
- 10-percentile err: 1.2
- 90-percentile err: 7.8
Download a dataset TSV from the server.
Supports streaming to stdout or saving directly to a file. By default a dataset is not downloaded with the gold-standard classes or scores, but that can be overridden with a CLI flag.
Pipe dataset to stdout:
$ levar-cli datasets download iris | head
id sepal length in cm sepal width in cm petal length in cm petal width in cm
302a0d66 5.1 3.5 1.4 0.2
302a0d2a 4.9 3.0 1.4 0.2
302a0d20 4.7 3.2 1.3 0.2
302a0d7a 4.6 3.1 1.5 0.2
302a0d02 5.0 3.6 1.4 0.2
302a0d84 5.4 3.9 1.7 0.4
302a0d3e 4.6 3.4 1.4 0.3
302a0d8e 5.0 3.4 1.5 0.2
302a0cbc 4.4 2.9 1.4 0.2
Optionally save to file:
$ levar-cli datasets download -o iris.tsv iris
$ levar-cli datasets download --output iris.tsv iris
Optionally download the gold values with the dataset:
$ levar-cli datasets download --gold iris | head
id sepal length in cm sepal width in cm petal length in cm petal width in cm class
302a0d66 5.1 3.5 1.4 0.2 Iris-setosa
302a0d2a 4.9 3.0 1.4 0.2 Iris-setosa
302a0d20 4.7 3.2 1.3 0.2 Iris-setosa
302a0d7a 4.6 3.1 1.5 0.2 Iris-setosa
302a0d02 5.0 3.6 1.4 0.2 Iris-setosa
302a0d84 5.4 3.9 1.7 0.4 Iris-setosa
302a0d3e 4.6 3.4 1.4 0.3 Iris-setosa
302a0d8e 5.0 3.4 1.5 0.2 Iris-setosa
302a0cbc 4.4 2.9 1.4 0.2 Iris-setosa
$ levar-cli datasets download -g iris | head
...
New dataset type that is like classification, but supports way more than just a handful of "classes". For instance, support tasks like to identify the correct ID from a set of IDs, wherein the target is different for each data point. In this dataset type the notion of per-class accuracy (or precision, recall etc) does not apply.
Github labels aren't super helpful. Reorganize along the lines of something like this:
Add a Heroku deploy button to README
Set up to run on Travis CI
Simple, for debug, not a proper UI
Create a new dataset from a TSV upload. Learn the schema from the data itself.
Example datasets for spec:
$ levar-cli datasets upload winequality-red.tsv
Dataset my-org-1/winequality-red uploaded
Can override the associated organization with -o
or --org
flags:
$ levar-cli datasets upload -o my-other-org winequality-red.tsv
Dataset my-other-org/winequality-red uploaded
Creates a name based on the TSV file by default, otherwise a new name can be specified with -n
or --name
flags:
$ levar-cli datasets upload -n wine winequality-red.tsv
Dataset my-org-1/wine uploaded
If a name is provided explicitly but a dataset with that name already exists in the DB, fail.
$ levar-cli datasets upload -n wine winequality-red.tsv
Error: my-org-1/wine already exists
If a name is not provided explicitly, but a dataset with the inferred name already exists in the DB, choose a new name:
$ levar-cli datasets upload -n wine winequality-red.tsv
Dataset my-org-1/winequality-red-2 uploaded
Doesn't seem to be matching the experiment name correctly in the confirmation step, and incorrectly reports:
Org + dataset do not match
Remove a dataset from the datastore, including all associated experiments. CLI should prompt the user to make sure she wants to do the delete:
$ levar-cli delete my-dataset-01
Are you sure you want to delete this dataset and associated experiments?
To confirm, type your organization and dataset name (like "org/dataset")
> ...
Dataset my-org/dataset-01 deleted
Can override user's default organization:
$ levar-cli delete my-other-org/my-dataset-01
Are you sure you want to delete this dataset and associated experiments?
To confirm, type your organization and dataset name (like "org/dataset")
> ...
Dataset my-other-org/dataset-01 deleted
Overall accuracy: 0.845
Should be
Overall accuracy: 84.5
for consistency with how these are displayed elsewhere.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.