Is it possible to share training error? about 3dunetcnn

[mingrui]

You have to tweak your parameters.
I was able to get up to 0.79 with my own dataset

[DongqingZhang]

@mingrui Thanks! Yesterday I quickly tested a small sub-dataset (only 3 training examples) from the author and used batchsize = 2 and learning rate = 0.001 since I think the original lr is too small and got a training error of -0.7~-0.8 (it's swinging) after 500 epochs. I would imagine this DL model should be able to at least fit such a small dataset perfectly (dice>=0.95, I guess?) but it does not. Still not sure if this is the best that 3dUnet can do.

[mingrui]

@DongqingZhang have you gotten Deconvolution3D working? it's in the model.py get_up_convolution I'm trying to get it to work because I think it will improve training error by a lot

[DongqingZhang]

@mingrui No. I cannot load Deconvolution3D from keras_contrib. Do u mean customizing this layer myself? It might be a good idea.
However, I changed to my own data, still use only two images for a quick training. This time I used 5,000 epochs and I see the error plateaus from 400 epochs to 1,800 epochs at ~-0.5, and starts decreasing again from 1,800 epochs and I am able to obtain less than -0.9 loss at 5,000 epochs. It is weird because normally we don't observe 2 plateaus...
However, at least I do see the error decrease to a reasonable value...

[mingrui]

wow 5000 epochs... it would take my computer with a gtx 1080 like 20 days to train...

[Issam28]

guys what kinf od gpu are you using ? does this model it to a gpu with 2GO of memory?

[mingrui]

@Issam28 8gb

[Issam28]

So I have no chance training these types of models with a 2 go gpu?

[DongqingZhang]

Titan X. Since the the training size is so small, 5,000 epochs does not take that long...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you have questions, feel free to join the Slack group or email me at [email protected].