Comments (9)
@voldemortX OK, I see. Thank you very much. I'll check it carefully.
from pytorch-auto-drive.
@voldemortX OK, thank you. I'll try it according to your suggestion.
from pytorch-auto-drive.
@daigang896 Hi! What is your exact training script? As far as I know, by this repo's default lr settings, only LSTR sometimes produce NaN. SCNN/RESA only does that when lr is too high.
from pytorch-auto-drive.
train script:python main_landec.py --epochs=200 --lr=0.15 --batch-size=16 --dataset=tusimple --method=scnn --backbone=vgg16 --mixed-precision --exp-name=vgg16_scnn_tusimple. However, the data used is not tusimple, but the format is tusimple format. I'll set the learning rate smaller first and then look at it.
from pytorch-auto-drive.
from pytorch-auto-drive.
@daigang896 FYI, segmentation method's learning rate should be adjusted according to the total number of pixels in a batch (i.e., not only batch size, but also training resolution). The relationship is mostly linear or sqrt. Unless the exploded loss is the existence loss. Other than smaller learning rates, sometimes longer warmup can bring better performance. While in rare cases simply re-run the experiment is enough (the VGG-SCNN does have a small failure rate in training).
Note that other than typical gradient explosion caused by large learning rates, irregular labels (labels with NaN value for instance) can also cause this issue.
from pytorch-auto-drive.
I checked the data carefully, and there was no Nan in the data. However, in the middle of training, the training loss is Nan. The data format is tusimple format. How should I check this situation?
The training is carried out:
python main_ landec. py --epochs=240 --lr=0.12 --batch-size=16 --dataset=tusimple --method=scnn --backbone=erfnet --mixed-precision --exp-name=erfnet_ scnn_ tusimple.
from pytorch-auto-drive.
@daigang896 What is the size of your dataset? 240 epochs seem long. Theoretically, learning rate decreases in proportion to the training length, you'll have higher lr in early stages if epochs are set longer thus easier to explode. You might consider a longer warmup by --warmup-steps
.
Or just try lr=0.01 and see if it still produces NaN. For sanity checks, remove --mixed-precision
.
from pytorch-auto-drive.
@daigang896 It seems the problem is somewhat resolved and this issue happened a long time ago. And we never encounter a similar issue when refactoring the whole codebase, so it is probably not a bug. If you still can't make it work with the new master branch, feel free to reopen!
from pytorch-auto-drive.
Related Issues (20)
- [Bug] culane_py error at a specific case HOT 4
- How to validate the BezierLaneNet on Cityscapes? HOT 5
- RuntimeError: received 0 items of ancdata HOT 10
- 匈牙利匹配时时用到curve_utils中的Cubic bezier curve segment HOT 6
- BSNet implementation HOT 4
- pretrained weights HOT 4
- 怎么解释feature flip fusion HOT 5
- bezierlanenet分布式训练问题 HOT 9
- 關於tusimple的訓練和測試 HOT 6
- 论文的公式4写错了吧,矩阵顺序反了 HOT 6
- 关于车道线的评估方式 HOT 9
- Lane detection training -bug HOT 3
- 配置Lane detection 的数据集culane的demo遇到的问题 HOT 23
- Hello, Feng! What is the use of local_maxima in Hungarian match? HOT 4
- 如何找到各个模块的代码位置 HOT 3
- Tusimple测试出现文件名不匹配 HOT 4
- About bezier_control_points HOT 1
- [Errno 2] No such file or directory HOT 5
- Start of Training outputs
- Vis Error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytorch-auto-drive.