Comments (2)
ARman-AT
commented on July 23, 2024
1
Your lr is very high, start with 3e-4 for finetuning. Model might have exploded to NaNs. Could you swicth to fastconformer architecture instead of Conformer? FastConformer is quick to train. start with fp32 then move to precision 16 once your training setup is fine and you see curves are normal.
Fastconformer configs: https://github.com/NVIDIA/NeMo/tree/main/examples/asr/conf/fastconformer
thanks for replying. i actually changed the precision from 16 to 32 and it solved my problem.
from nemo.
nithinraok
commented on July 23, 2024
Your lr is very high, start with 3e-4 for finetuning. Model might have exploded to NaNs. Could you swicth to fastconformer architecture instead of Conformer? FastConformer is quick to train. start with fp32 then move to precision 16 once your training setup is fine and you see curves are normal.
Fastconformer configs: https://github.com/NVIDIA/NeMo/tree/main/examples/asr/conf/fastconformer
from nemo.
Related Issues (20)
- Job specific environment variables can't be set in Hydra multi-run HOT 2
- Using lhotse when training a hybrid fast conformer model fails HOT 10
- How to config a locally model?
- Unable to reproduce cache aware streaming results for Conformer that were there for Fastconformer.
- Can we add emotions to the produced audio? HOT 1
- LM on Parakeet models HOT 1
- to support deepseekv2 HOT 1
- How to use a pre-trained model for cache-aware FastConformer-Hybrid model? HOT 3
- When Trying to import nlp collections in the Nemo Primer getting error "No Module named megatron"
- How to export SLUIntentSlotBPEModel to ONNX HOT 1
- issue about self attention with mask
- Converting megatron checkpoint to .nemo without the same environment
- Nemo container for Nemotron 340B inference fails pytorch_lightning import HOT 1
- Can you support DoRA? HOT 1
- Unable to reproduce cache aware streaming results for Conformer that were there for Fastconformer.
- Issue: TimeError Occurring During Training on node 16 or more
- Speaker Diarization goes haywire due to small segments of audio
- MCore slower than NeMo native implementation
- FSDP CPU offloading errors out due to device placements
- Getting empty results from online streaming asr. Please help me!!!!! thanks a lot.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nemo.