levarml / levar Goto Github PK

View Code? Open in Web Editor NEW

24.0 24.0 3.0 241 KB

Machine learning evaluation database

Home Page: http://levarml.github.io/LeVar/

License: Apache License 2.0

Scala 83.99% HTML 1.66% CSS 10.26% PLSQL 4.08%

evaluation-data machine-learning

levar's People

Contributors

Stargazers

Watchers

Forkers

nickgeoca gitter-badger jfrazee

levar's Issues

Make LeVar dependency less of a pain to use

Ensure resources are published publicly and can be installed by client libraries easily

Create new dataset via TSV upload using CLI

Requirements

Create a new dataset from a TSV upload. Learn the schema from the data itself.

Wireframes

Example datasets for spec:

Regression: winequality-red.tsv
Classification: adult.tsv

$ levar-cli datasets upload winequality-red.tsv
Dataset my-org-1/winequality-red uploaded

Can override the associated organization with -o or --org flags:

$ levar-cli datasets upload -o my-other-org winequality-red.tsv
Dataset my-other-org/winequality-red uploaded

Creates a name based on the TSV file by default, otherwise a new name can be specified with -n or --name flags:

$ levar-cli datasets upload -n wine winequality-red.tsv
Dataset my-org-1/wine uploaded

If a name is provided explicitly but a dataset with that name already exists in the DB, fail.

$ levar-cli datasets upload -n wine winequality-red.tsv
Error: my-org-1/wine already exists

If a name is not provided explicitly, but a dataset with the inferred name already exists in the DB, choose a new name:

$ levar-cli datasets upload -n wine winequality-red.tsv
Dataset my-org-1/winequality-red-2 uploaded

Requirements

View an experiment with high level results analysis with CLI. Terminal output must fit within 80 chars horizontal.

For classification experiments, include overall accuracy and per-label + overall precision, recall, f-score. For less than 10 classes also include a confusion matrix.

Wireframes

Classification experiments:

$ levar-cli view experiment adult exp-0101
Experiment:  exp-0101
Dataset:     adult
Type:        classify
Predictions: 32563 (full dataset coverage)

Results:

Overall accuracy: .931

 Label         | Prec | Rec  | F1
---------------|------|------|------
 <=50K         | .952 | .917 | .934
 >50K          | .871 | .970 | .917
 Macro-Avg     | .912 | .944 | .926

Confusion matrix (columns give predicted counts; rows give gold counts)

Labels        |    <=50K |       >50K |     Totals
--------------|----------|------------|-----------
<=50K         |    22690 |       1142 |      23832
>50K          |     1118 |       7611 |       8729
Totals        |    23808 |       8753 |      32561

Regression experiments:

$ levar-cli view experiment winequality-red exp-234
Experiment:  exp-234
Dataset:     winequality-red
Type:        regression
Predictions: 1599 (full dataset coverage)

Results:
- RMSE:              4.2
- Mean abs error:    4.8
- Median abs error:  3.9
- 10-percentile err: 1.0
- 90-percentile err: 7.1

Update README for new org

Make sure new sponsoring companies are up to date
Verify any links are correct
Include the new logo
Update Travis CI, Coveralls, Gitter links

Setup add-ons

Verify or recreate:

Travis CI
Coveralls
Gitter.im

Download experiment results TSV via CLI

Move README content into website

In regression datasets, show performance by quartile

Wireframe:

Experiment:    dataset-foo/experiment-bar
Type:          regression
Predictions:   4000 (full dataset coverage)

Results:
- RMSE:                 4.1
- Mean abs error:       4.0
- Median abs error:     3.9
- 10-percentile err:    1.1
- 90-percentile err:    6.6

Quartile 1: 0.0 - 10.7
- RMSE:                 5.2
- Mean abs error:       4.9
- Median abs error:     4.9
- 10-percentile err:    1.2
- 90-percentile err:    7.8

Quartile 2: 10.7 - 15.2
- RMSE:                 5.2
- Mean abs error:       4.9
- Median abs error:     4.9
- 10-percentile err:    1.2
- 90-percentile err:    7.8

Quartile 3: 15.2 - 20.6
- RMSE:                 5.2
- Mean abs error:       4.9
- Median abs error:     4.9
- 10-percentile err:    1.2
- 90-percentile err:    7.8

Quartile 4: 20.6 - 33.4
- RMSE:                 5.2
- Mean abs error:       4.9
- Median abs error:     4.9
- 10-percentile err:    1.2
- 90-percentile err:    7.8

Requirements

Remove a dataset from the datastore, including all associated experiments. CLI should prompt the user to make sure she wants to do the delete:

Wireframes

$ levar-cli delete my-dataset-01
Are you sure you want to delete this dataset and associated experiments?
To confirm, type your organization and dataset name (like "org/dataset")
> ...
Dataset my-org/dataset-01 deleted

Can override user's default organization:

$ levar-cli delete my-other-org/my-dataset-01
Are you sure you want to delete this dataset and associated experiments?
To confirm, type your organization and dataset name (like "org/dataset")
> ...
Dataset my-other-org/dataset-01 deleted

Apache license

Include the Apache License v2.0 to this project

Update website to reflect new organization

Update website to

Make sure links to project are the correct org
Use the new levarml logo
Adopt a new black theme to match the logo
Make sure website mentions new sponsoring companies

New logo:

Requirements

Display high level details of a dataset to terminal. Output must fit within 80 chars horizontal.

Summary includes

name & type info
a summary of the feature schema
for classification datasets with less than 10 labels an empirical distribution of the labels
for regression datasets basic summary statistics - min, max, mean, median
any associated tags
any associated comments

Wireframes

Classification dataset:

$ levar-cli datasets view adult
Dataset: adult
Type:    classify
Items:   32563
Created: 2015-10-10 16:23
Updated: 2015-10-10 18:01
Schema:
- age:                  text
- workclass:            text
- fnlwgt:               numeric
- education:            text
- education-num:        numeric
- marital-status        text
- occupation            text
- serv                  text
- relationship          text
- race                  text
- sex                   text
- capital-gain          numeric
- capital-loss          numeric
- hours-per-week        numeric
- native-country        text
Classes:
- <=50K                 75.9% (24720/32561)
- >50K                  24.1% (7841/32561)
Tags: eval_for_study1, classification_datasets
Comments:
- system
  2015-10-10 16:23
  Created by user01 at 2015-10-10 16:23
- system
  2015-10-10 18:01
  400 items added by user01 at 2015-10-10 18:01
- user01
  2015-10-10 18:04
  added some more hard evaluation items to the dataset

Can override default organization:

$ levar-cli datasets view my-org/adult
Dataset: adult
Type:    classify
...

Viewing summary stats for regression datasets:

$ levar-cli datasets view winequality-red
Dataset: winequality-red
Type:    regression
Items:   1599
Created: 2015-10-10 16:23
Updated: 2015-10-10 18:01
Schema:
- fixed acidity         numeric
- volatile acidity      numeric
- citric acid           numeric
- residual sugar        numeric
- chlorides             numeric
- free sulfur dioxide   numeric
- total sulfur dioxide  numeric
- density               numeric
- pH                    numeric
- sulphates             numeric
- alcohol               numeric
Score distribution:
- min value             3.0
- max value             8.0
- mean                  5.64
- median                6.0
- 10th percentile       5.0
- 90th percentile       7.0
Tags: eval_for_study1, classification_datasets
Comments:
- system
  2015-10-10 16:23
  Created by user01 at 2015-10-10 16:23
- system
  2015-10-10 18:01
  400 items added by user01 at 2015-10-10 18:01
- user01
  2015-10-10 18:04
  added some more hard evaluation items to the dataset.

Requirements

List datasets associated with an organization, user's default organization if none specified explicitly. Provide output in readable tabular format. Provide very high-level details for each dataset. CLI output must fit in 80 char width.

Wireframes

By default list datasets in user's default organization:

$ levar-cli datasets
Datasets for my-org
Dataset                    | Updated    | Type       |    Items
---------------------------|------------|------------|----------
dataset-01                 | 2015-10-10 | classify   |   100200
dataset-02                 | 2015-10-20 | regression |  1002020

Optionally specify organization:

$ levar-cli datasets my-other-org
Datasets for my-other-org
Dataset                    | Updated    | Type       |    Items
---------------------------|------------|------------|----------
dataset-01                 | 2015-10-10 | classify   |   100200
dataset-02                 | 2015-10-20 | regression |  1002020

Generate a website

Install code coverage

And hook up Coveralls.io

Less JSON logging

Support geo evaluation datasets

Add rows to a dataset via CLI

Add paulp sbt script

In experiments view, show accuracy in percent

Overall accuracy: 0.845

Should be

Overall accuracy: 84.5

for consistency with how these are displayed elsewhere.

Tag datasets via CLI

View dataset in HTML

For debug only, not a proper UI

Publish resources to Bintray

Open target dataset type

New dataset type that is like classification, but supports way more than just a handful of "classes". For instance, support tasks like to identify the correct ID from a set of IDs, wherein the target is different for each data point. In this dataset type the notion of per-class accuracy (or precision, recall etc) does not apply.

Configure and save server credentials with CLI

Requirements

Configure and save LeVar server credentials with CLI.

By default use $HOME/.levar/config.cfg to store configurations. This can be configured by setting the $LEVAR_CONFIG environment variable; i.e. the default of $LEVAR_CONFIG is $HOME/.levar.

Wireframes

Provide credentials via CLI options:

$ levar-cli config --username user01 --password secret --url http://localhost:9000 --org my-org

$ levar-cli config -u user01 -p secret -u http://localhost:9000 -o my-org

If no options are provided, prompt user for details:

$ levar-cli config
Username:
Password:
URL:
Organization:

If these details have already be configured and saved prompt the user. Don't actually display the password. If no user response is provided stay with the value already provided:

$ levar-cli config
Username [user-01]:
Password [********]:
URL [http://localhost:9000/]:
Organization [my-org]:

levar-cli rename experiment - not supported yet

Reports:

You did not supply an argument -- try 'levar-cli datasets'

Tag experiments via CLI

Comment on dataset via CLI

Upload experiment via CLI

Requirements

Upload a TSV experiment file to be associated with a dataset.

The TSV must have an 'id' column and an 'output' column -- other columns are optional. Values in 'id' must match the IDs in the dataset being evaluated against. The CLI reports an error and aborts if:

an ID does not match a row in the dataset
the TSV contains multiple rows with the same ID

The CLI warns the user, but does not abort, if there are items in the original dataset not included in the experiment upload.

Wireframes

Here "adult" is an example of a dataset.

Basic usage:

$ levar-cli experiments upload adult adult-experiment-01.tsv
Experiment adult-experiment-01 for dataset my-org/adult uploaded

The organization associated with the dataset can be overridden by the org/dataset name convention:

$ levar-cli experiments upload my-other-org/adult adult-experiment-01.tsv
Experiment adult-experiment-01 for dataset my-other-org/adult uploaded

The name is guessed from the filename, but can be overridden with the the --name or -n flag:

$ levar-cli experiments upload -n new-results-001 adult adult-experiment-01.tsv
Experiment new-results-001 for dataset my-other-org/adult uploaded

$ levar-cli experiments upload --name new-results-002 adult adult-experiment-01.tsv
Experiment new-results-002 for dataset my-other-org/adult uploaded

If a name is given explicitly via the --name flag, but there is already an experiment with that name associated with dataset's organization in the DB, the CLI reports an error and aborts the upload.

$ levar-cli experiments upload --name new-results-002 adult adult-experiment-01.tsv
Error: experiment named new-results-002 already exists

If the name is guessed from a filename upload, but an experiment with the same name is already in the DB associated with the dataset's organization, modify the name before creating the experiment in the DB:

$ levar-cli experiments upload adult adult-experiment-01.tsv
Experiment adult-experiment-01-2 for dataset my-org/adult uploaded

List experiments for a dataset via CLI

Requirements

List experiments for a dataset with very high level summary. CLI output must fit in 80 char displays.

Wireframes

With user's default organization, she can just specify dataset name. For classification experiments, include accuracy, precision, recall and F score.

$ levar-cli experiments list dataset-01
Experiment           | Updated    | Predictions |  Acc | Prec |  Rec |   F1
---------------------|------------|-------------|------|------|------|------
experiment-01        | 2015-10-12 |        1200 | .678 | .782 | .505 | .614
experiment-02        | 2015-10-12 |        1200 | .688 | .792 | .515 | .624
final-experiment     | 2015-10-14 |        1200 | .698 | .802 | .525 | .634

Override default organization:

$ levar-cli experiments list my-org/dataset-01
Experiment           | Updated    | Predictions |  Acc | Prec |  Rec |   F1
---------------------|------------|-------------|------|------|------|------
experiment-01        | 2015-10-12 |        1200 | .678 | .782 | .505 | .614
experiment-02        | 2015-10-12 |        1200 | .688 | .792 | .515 | .624
final-experiment     | 2015-10-14 |        1200 | .698 | .802 | .525 | .634

For regression experiments, include RMSE:

$ levar-cli experiments list my-org/dataset-01
Experiment           | Updated    | Predictions |    RMSE
---------------------|------------|-------------|---------
experiment-01        | 2015-10-12 |        1200 |    5.67
experiment-02        | 2015-10-12 |        1200 |    4.97
final-experiment     | 2015-10-14 |        1200 |    4.45

levar delete experiment - won't let you delete an experiment

Doesn't seem to be matching the experiment name correctly in the confirmation step, and incorrectly reports:

Org + dataset do not match

How We Organize GitHub Issues

Requirements

Download a dataset TSV from the server.

Supports streaming to stdout or saving directly to a file. By default a dataset is not downloaded with the gold-standard classes or scores, but that can be overridden with a CLI flag.

Wireframes

Pipe dataset to stdout:

$ levar-cli datasets download iris | head
id  sepal length in cm  sepal width in cm   petal length in cm  petal width in cm
302a0d66    5.1 3.5 1.4 0.2
302a0d2a    4.9 3.0 1.4 0.2
302a0d20    4.7 3.2 1.3 0.2
302a0d7a    4.6 3.1 1.5 0.2
302a0d02    5.0 3.6 1.4 0.2
302a0d84    5.4 3.9 1.7 0.4
302a0d3e    4.6 3.4 1.4 0.3
302a0d8e    5.0 3.4 1.5 0.2
302a0cbc    4.4 2.9 1.4 0.2

Optionally save to file:

$ levar-cli datasets download -o iris.tsv iris

$ levar-cli datasets download --output iris.tsv iris

Optionally download the gold values with the dataset:

$ levar-cli datasets download --gold iris | head
id  sepal length in cm  sepal width in cm   petal length in cm  petal width in cm   class
302a0d66    5.1 3.5 1.4 0.2 Iris-setosa
302a0d2a    4.9 3.0 1.4 0.2 Iris-setosa
302a0d20    4.7 3.2 1.3 0.2 Iris-setosa
302a0d7a    4.6 3.1 1.5 0.2 Iris-setosa
302a0d02    5.0 3.6 1.4 0.2 Iris-setosa
302a0d84    5.4 3.9 1.7 0.4 Iris-setosa
302a0d3e    4.6 3.4 1.4 0.3 Iris-setosa
302a0d8e    5.0 3.4 1.5 0.2 Iris-setosa
302a0cbc    4.4 2.9 1.4 0.2 Iris-setosa

$ levar-cli datasets download -g iris | head
...

levarml / levar Goto Github PK

levar's People

Contributors

Stargazers

Watchers

Forkers

levar's Issues

Requirements

Wireframes

Requirements

Wireframes

Requirements

Wireframes

Requirements

Wireframes

Requirements

Wireframes

Requirements

Wireframes

Requirements

Wireframes

Requirements

Wireframes

Requirements

Wireframes

Recommend Projects

Recommend Topics

Recommend Org

Jobs