j0ma / prepx

CLI for creating experiments

prepx

A library to create experiment folders according to a common schema.

Schema

• An experiment is a collection of train and eval runs
• Each train run represents a trained model and has attributes:
  • Checkpoint folder
  • Eval folder
  • Data folder
  • Log folder
• Each eval run represents a model applied to some data
  • Notably, this includes ones with no associated training runs
  • Covers e.g. applying a pre-trained model on a given dataset

Installation

pip install -e .

Usage

There are two commands: analyze and create:

Usage: prepx [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  analyze
  create

Create

This command supports several scenarios:

1. Create an empty experiment folder

prepx create --empty-only \
             --experiment-name <desired-experiment-name> \
             --root-folder <your-desired-root-name>
             --experime

Doing this manually is not mandatory.

2. Create a train run under an experiment

• Note: a default eval run will also be created under eval_<train-name>

prepx create \
    --experiment-name <desired-experiment-name> \
    --root-folder <your-desired-root-name> \
    --train-name <desired-name-of-model> \
    --raw-data-folder <path-to-evaluation-data>

3. Only create an eval run under an experiment

prepx create \
    --eval-only \
    --experiment-name <desired-experiment-name> \
    --root-folder <your-desired-root-name> \
    --eval-name <desired-eval-name> \
    --eval-checkpoint <path-to-eval-checkpoint> \
    --raw-data-folder <path-to-evaluation-data>

Overall help:

Usage: prepx create [OPTIONS]

Options:
  --experiment-name TEXT     Name of the experiment.
  --train-name TEXT          Name of the model/train run.
  --eval-name TEXT           Name of the eval run.
  --root-folder PATH         Root experiments folder. Defaults to cwd.
  --raw-data-folder PATH     Folder with relevant raw data.
  --checkpoints-folder PATH  Folder with relevant checkpoints.
  --eval-checkpoint FILE     Path to checkpoint if using --eval-only
  --empty-only               Only create empty experiment folder
  --eval-only                Only create eval folder of experiment
  --help                     Show this message and exit.

Analyze

Usage: prepx analyze [OPTIONS] FOLDER

Options:
  --help  Show this message and exit.

prepx's Issues

Add documentation about spec and how to run

The README is currently quite empty. The experiment folder spec followed by prepx should be documnted in this repo for the future.

Add `--with-supplemental-data-folder`

Since the raw data folder is symlinked, it's shared between many experiments.
Often there are things that need to be computed based on the raw data.
A supplemental data folder would be a good idea for this.

Examples: word/char counts, lists of ngrams etc.

`prepx analyze` requires `tree` which is not a dependency

If a user does not have tree installed, prepx analyze will not work.
This is reproducible using the nonexistent-tree-executable branch where the tree executable has been changed.

Error produced:

Traceback (most recent call last):
  File "/home/jonne/miniconda3/envs/prepx-test/bin/prepx", line 33, in <module>
    sys.exit(load_entry_point('prepx', 'console_scripts', 'prepx')())
  File "/home/jonne/miniconda3/envs/prepx-test/lib/python3.9/site-packages/click-8.1.3-py3.9.egg/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/jonne/miniconda3/envs/prepx-test/lib/python3.9/site-packages/click-8.1.3-py3.9.egg/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/jonne/miniconda3/envs/prepx-test/lib/python3.9/site-packages/click-8.1.3-py3.9.egg/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jonne/miniconda3/envs/prepx-test/lib/python3.9/site-packages/click-8.1.3-py3.9.egg/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jonne/miniconda3/envs/prepx-test/lib/python3.9/site-packages/click-8.1.3-py3.9.egg/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/data/coding/prepx/prepx/cli.py", line 102, in analyze_experiment
    pid = sp.run(args_, capture_output=True)
  File "/home/jonne/miniconda3/envs/prepx-test/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/jonne/miniconda3/envs/prepx-test/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/jonne/miniconda3/envs/prepx-test/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'nonexistent_tree_executable'

Test issue

Nothing of importance here