validation during experiment 0 exhausts whole ram memory about pneumothorax-segmentation

[sneddy]

You need more RAM, because I'm store in memory all validation masks (with probabilities)

I can turn off this option for using fix triplet on validation.
Another option: save in memory masks with lower resolution (for example 512x512 for 1024x1024 images)

[PumayHui]

@sneddy Thanks for your work!
And where should I modify with 512x512 replace 1024x1024?
This?
python prepare_png.py -img_size 512 -train_path ../../input/dicom_train -test_path ../../input/dicom_test -out_path ../../input/dataset512 -rle_path ../../input/new_train_rle.csv -n_threads 8
and:
In "experiments/albunet_valid/train_config_part0.yaml", change all 1024 in the file to 512?
Are these correct? Can I reach the indicator you said?
Thanks!

As follow, I add cuda=0,1 and run
python Train.py experiments/albunet_valid/train_config_part0.yaml
my fold0 summary.csv is: (It doesn't seem to reach the "0.8528" you mentioned in 512, Fold 0.)
Is it wrong for me? Thanks so much!

[PumayHui]

@sneddy And my GPU is 1080Ti (11176MiB), how much capacity does your ram have?

[sneddy]

@PumayHui Thank you for feedback. I use the same gpu

Scores in main dashboard obtained after all steps of my pipeline.
I mean train_config_part0, part1 and part2.

I really uptrained my model A LOT. Maybe you should apply last config few times.

[PumayHui]

@sneddy Thanks so much!
Could you train 1024x1024 data with the same GPU?
When I train 1024x1024 data, it shows out of memory.
But when I use 512x512 data, the Dice coefficient of train_config_part0 is only 0.79. And I see you said your part0 is 0.835. Do you have any suggestions? Thanks!

[sneddy]

It's a result of 1024x1024 images.

Memory error related to RAM, I have 64GB RAM on my machine.
I know that's really large amount of memory and I will try to fix it in the coming days

[PumayHui]

@sneddy Emmm…, my RAM is 32GB…
Do you have any suggestions for me with such RAM? Thanks so much!

[sneddy]

@PumayHui @rajatmodi62
I updated my solution and configuration files
Updated pipeline must work on 8 GB RAM and 1080ti (need at least 9 GB gpu or reducing batch size)

[PumayHui]

Wow! Thank a lot!!!

[wangyunpengbio]

@PumayHui I seem to encounter similar problems. I tried the previous configuration file. I also use 512 * 512 resolution images, but I can't get about 0.85 local validation score. In my experiment, I used pre-train on 512 * 512 image first, then fine-tune on 1024 *1024 image. However it seemed to fall into the local minimum value, with only 0.80 local validation score. Maybe the update of the code can save me. In a word, thanks for the continuous update of the author. @sneddy