loss is nan about medical-transformer

[walynlee]

Hello, I had a similar problem in training.Can someone know?When the batch size is 4 and epoch is 400,I train these model and found it is not like as what the paper said.And I don't know how to solve this problem.


[over-star]

is the MedT model have not finish?

[jeya-maria-jose]

Hi,

I just tried it again , it is working at my end.

These are the versions of the packages in my environment, maybe try downgrading torch to 1.4 from 1.8. Please use the requirements.txt to setup the exact environment.

[over-star]

i use rtx3060, maybe i can not install torch1.4

[over-star]

ok,when i delete a image ,it is fine.maybe the number of image should be Multiple of 4.
after train epoch for 400,i find it that the effect is not very good.

[jeya-maria-jose]

Hi,
Can you please elaborate ?

[over-star]

The third picture is my training model,after train epoch for 600,i find it that the effect is not very good.like this.

if the train image not be Multiple of batch_size .maybe as follow:

If I have anyproblem, please point it out in time. Thank you for your work

[jeya-maria-jose]

Looks like the training breaks mid-way at your end. The batch size used for the experiment was 4. Will investigate with different batch sizes to check if that is a problem and get back.

[jeya-maria-jose]

Hi @walynlee ,

I think 02.png is you have visualized here is different from what is in the dataset, I think the predictions are for a different input. Below is the input, prediction and GT respectively.

Also, I will update the exact number of epochs and batch size that gives the best performance for each dataset in the next commit.