allenai / beaker-gantry Goto Github PK

Might need two separate options, something like --env 'NAME=VALUE' and --env-secret 'NAME=SECRET'. Maybe we could also have a shortcut for the 2nd case: --secret SECRET which would be equivalent to --env-secret SECRET=SECRET.

🐛 Describe the bug

my cmd is

gantry run \
--dry-run \
--save-spec beaker-spec.yaml \
-n test-cv-docker \
--workspace ai2/br \
--cluster ai2/qicao-v100 \
--gpus 2 \
--beaker-image 'qicao/test-cv-docker' \
--venv 'eff' \
--dataset 'qicao/test-images:/data' -- \
torchrun --nproc_per_node 2 src/complete_cv_example.py --data_dir /data 

the git repo is https://github.com/csarron/efficiency-benchmark-test

error log:
https://beaker.org/api/v3/jobs/01GE637TDZZXYRSRABYDYK9C26/logs?tail=2

I suspect the beaker image somehow cached the git repo, which will fail the gantry when doing gh repo clone "$GITHUB_REPO" here

The beaker image is created using docker (dockerfile is here)

Versions

gantry is the latest version

🚀 The feature, motivation and pitch

Support distributed batch jobs.

Alternatives

N/A

Additional context

https://beaker-docs.apps.allenai.org/distributed-training.html#batch-jobs

Recently @mattf1n and I debugged how to update one's Github token, which might happen if you've entered it incorrectly or it's expired. It wasn't very easy.

We ultimately discovered that it's stored as a Beaker secret. Which means we were able to use beaker secret * to both inspect and manipulate the value. But this wasn't terribly intuitive and required looking at the code.

It'd be nice if Gantry just provided a way to do this, say:

gantry config set token ...

..or something like that?

We could install conda on the fly when the image doesn't already have it.

🚀 The feature, motivation and pitch

We should have to use a GitHub PAT for public repos.

Alternatives

Keep doing what we're doing.

Additional context

No response

It doesn't appear to be possible, but probably should be.

Related to #13. See #13 (comment).

This is a little more complex because we probably don't want to re-upload those local files each time gantry run is called if they haven't changed. Maybe we add a flag to force re-upload, otherwise we could compute a hash of the contents of all source files and only upload when the hash changes. But then we're back to re-implementing git...

@mattf1n discovered that you must use the .yml extension for your environment.yml file. If you use .yaml it won't be found by Gantry.

Gantry should probably look for both.