Turn off hpc checkpoint saving in SLURM environment if trainer.fit(..., ckpt_path="last") about pytorch-lightning

[murnanedaniel]

@rustamzh This is a very good idea, and should be implemented.
However, I want to understand exactly your setup, as I'm trying to implement this myself. You specify a ModelCheckpoint with dirpath equal to some artifact_directory, I assume? And then you also set the Trainer's default_root_dir to that same artifact_directory?

My issue here is that if I am training two models, using the same artifact_directory, won't the auto-resume get them mixed up when it uses ckpt_path="last"?

[rustamzh]

Hi. My dirpath is empty in the ModelCheckpoint callback, this way checkpoints are created in their own version-X (where X is SLURM jobID) subfolder and do not mix, since the autorequeued job retains its jobID.

[murnanedaniel]

Hi. My dirpath is empty in the ModelCheckpoint callback, this way checkpoints are created in their own version-X (where X is SLURM jobID) subfolder and do not mix, since the autorequeued job retains its jobID.

This makes a lot of sense, although I'm missing the way that your default root dir is configured. Would it be possible to share the code snippet that is most relevant to this configuration? It would be much appreciated!

[rustamzh]

default root dir is just non empty.