Comments (12)
I tried to compare my code with your code as well as @johnma2006 's code line-by-line, taking three code files in perspective, there seems to be no successful findings so far, except for
delta
inverse-softplus initialization which only your code had performed.I am bit stucked, Please advise.
The delta initialization is important.
from mamba.
My Mamba code implementation seems to work now without any negative training loss for now so far, will do further checking and code regression running to see if the issue still persists.
from mamba.
Have you plugged in a standard Transformer first? It seems more likely that there's something wrong with the training pipeline than with any particular model.
from mamba.
It looks like you reimplemented the model from scratch, so this is beyond the scope of our ability to help. Perhaps check line by line that your implementation matches ours?
from mamba.
Hi, here is a suggestion is to check the correctness of your implementation:
- Load an instance of your implementation and the official implementation side-by-side.
- Transfer the official instance's weights into your instance.
- Make sure the forward is identical. If not, drill down into each submodule to see where the diffs are coming from.
Good luck!
from mamba.
Comment on the initialization and parameterization: They are super important in the sense that without the suitable initialization and parameterization, the learning of long-term memory with SSMs can be unstable thus difficult. (https://arxiv.org/abs/2311.14495)
from mamba.
Thanks for the comments.
I had already incorporated proper delta initialization into the mamba code, but it is not helping with training loss convergence issue yet.
I need to think from other angle perspectives. 👀
@radarFudan : I noticed that StableSSM
tries to constraint the growth rate of gradient by constraining the eigenvalues. This approach seems to complement the operations done by clip_grad_norm()
. I will give StableSSM
a go in my code implementation, will post further updates here, thanks !!
from mamba.
The stable SSM initializations may or may not help, we've never tried them. But I think the theory doesn't apply directly to the selective SSM setting. I don't think there should be anything particular that you need to do here, so either there's an issue in the implementation or somehow Mamba interacts with your data weirdly, which would be interesting.
- Have you checked that your mamba function returns the same outputs as ours, as @johnma2006 suggested?
- Is there any reason you can't directly call the model from this repository? Is the purpose of your model expository or for research?
from mamba.
I had plugged in a small bert model, and the training works alright, so I am not really sure what else is missing from my MAMBA architecture module.
Please advise.
from mamba.
I tried to compare my code with your code as well as @johnma2006 's code line-by-line, taking three code files in perspective, there seems to be no successful findings so far, except for delta
inverse-softplus initialization which only your code had performed.
I am bit stucked, Please advise.
from mamba.
Great! What did you change?
from mamba.
@albertfgu : One of the major change is the output
sizing which has to be related to vocab_size
instead of d_model
See all the other changes required for getting rid of negative training loss
I will update the new code in github instead of gist now.
from mamba.
Related Issues (20)
- How to calculate Flops for mamba_chunk_scan_combined and mamba_split_conv1d_scan_combined function HOT 1
- Replace mamba1 with mamba2 and training becomes very slow! HOT 2
- Implement a "crossmamba" block for different length sequences HOT 3
- How to use sequence parallelism in Mamba2?
- Changing expansion_factor resulted in 'c10::DistBackendError HOT 1
- Mamba for encoder-decoder HOT 2
- mamba_split_conv1d_scan_combined error HOT 1
- torch.cuda.amp.custom_fwd(args...) Deprecated
- Is it possible to get ssm_states at specific seq len instead of just the last state? HOT 1
- How to modify mambaV2 to set causal=False? HOT 3
- Reducing _chunk_scan_bwd_kernel computation HOT 4
- Exploding gradients if ngroups is higher than 1. HOT 3
- clarification on how to interpret kernel size for conv1d HOT 1
- Some questions about the shape of A,B,C,D HOT 5
- How can I avoid using causal-conv1d? HOT 11
- Is it okay to put S6(MAMBA) and gated MLP blocks like a transformer? (Also please open the discussion tab)
- Vanishing gradient problem with more layer HOT 1
- Results vary greatly across experiments
- Gradient explosion in Mamba2 training, norm and loss divergence HOT 3
- Optimizing the bwd pass of Mamba 2 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mamba.