SIP Finetune on CheXpert - Validation sanity check error - target has to be an integer tensor about covidprognosis

[mmuckley]

@silgonel
What version of PyTorch and PyTorch Lightning are you running? It looks like this error was thrown by Lightning. I was on 1.7.1 and 1.0.8.

EDIT: much apologies - I see you matched what is in the requirements.txt. I'll try to match with the small version of CheXpert. Yes, you're right, we use the large one.

[nataschadam]

Hello @mmuckley ,
I am a colleague of @silgonel .

Thank you very much for your help. After the answer you gave us yesterday, we realized that even if we used the file "requirements" to create a new conda environment, conda didn't install the required versions of the packages. We created a new conda environment manually installing the required packages, and fortunately, the problem is gone and the training is working.
Thank you very much for your help!

[mmuckley]

Hello @silgonel.
First, I apologize for taking so long to respond. I was out of work for a bit and have been working on some other things.

The logits are the log probabilities, so their range is the entire real number line. You run a sigmoid layer to get the probabilities.

As for the error, I can't reproduce it. On my machine I ran the following command:

python train_sip.py --pretrained_file https://dl.fbaipublicfiles.com/CovidPrognosis/pretrained_models/mimic_lr_0.1_bs_128_fd_128_qs_65536.pt --dataset_name chexpert

I put the location of my CheXpert data in this file.

I can complete a full training epoch with training and validation.

Do these steps mirror those you took?

[silgonel]

Hello @mmuckley ,

Thank you very much for your kind answer!
My colleague @nataschadam and I followed your steps, but we are still getting the same error.
We are currently using the "Chexpert v1.0-small" version of the dataset (the one with reduced size of 11GB) and we installed the environment following the instructions of the README.md. I attached the requirements.txt of our environment.
requirements.txt

Maybe you are testing the code with the full-size dataset? Could there be any difference?

Many thanks,
Silvia

[mmuckley]

@silgonel
Looking at your requirements.txt it looks like you're using PyTorch Lightning 1.2.7 while the repository was tested on 1.0.8. Unfortunately, I'm tied up with other projects at the moment so I don't think I can update the repository to the new version. If you'd like to reproduce the results, I would roll back PyTorch Lightning to 1.0.8. Alternatively, you could fork the repository and update the relevant lines to output integer tensors and use the new version.

[mmuckley]

Closing for now. Reopen if you have further issues on this topic.