Problem with checkpoints when running .restore(checkpoints) method about abides-jpmc-public

[HaibeiZhu]

Hi, thanks for posting this issue

I have also experienced the same issue with the same error message before. First, could you confirm that the PyTorch version and other related dependencies are the same when you train and apply your agent?

If the version is the same, your checkpoints were not saved successfully. Could you let me know if you terminated the training manually? Or you waited until all epochs finished? Also, have you tried other checkpoints that you saved?

[EthanChoukroun]

Hi,
Thank you for your response,

I confirm that I am using the same version of Torch when I train and apply the agent, I did not terminate the training manually. I just set the number of iterations to a much lower value to run it locally and changed the checkpoint_freq parameter to different values lower than the # of iterations.

I have tried different ways to access checkpoints and change different parameters such as metric but I still have the same error.

What do you mean by setting checkpoints manually? I have only created checkpoints defined in the tune.run() command.

[Andrea94c]

Hi Ethan, thanks for posting this error and sorry for the delay.
This seems a compatibility issue with torch and Ray library.

The best solution is downgrading to torch<1.12.0 (e.g., torch==1.11.0). This should fix the issue.