Comments (8)
I’m very interested to see the response here from the authors, but I’ll say something potentially wrong.
i’m not familiar with the architecture used to create MIDI using the transformer mechanism, but in the original MAMBA paper they were pretty clear that continuous signals do not benefit form MAMBA, and that something like S4 seemed to outperform. So the performance you may be seeing may have less to do with regularization and more to do with the application of MAMBA to this problem.
from mamba.
You can use dropout, just like Transformers. It's not implemented here but you can add it.
from mamba.
You can use dropout, just like Transformers. It's not implemented here but you can add it.
This thread may be more useful if tagged as a feature request then? (Just for traceability to mitigate it from being lost)
from mamba.
This is still an issue; I trained the model on 10 epochs for a specific use case and the loss just decreases drastically to overfit quickly
from mamba.
Has anyone implemented this with some success?
from mamba.
I find that adding dropout decreases performance for state space models. Does anyone else also observe this phenomenon?
from mamba.
I added dropout and it actually improved performance for me.
from mamba.
Although MAMBA still does very bad in my use case, it just learned better.
from mamba.
Related Issues (20)
- Reducing _chunk_scan_bwd_kernel computation HOT 4
- Exploding gradients if ngroups is higher than 1. HOT 3
- clarification on how to interpret kernel size for conv1d HOT 1
- Some questions about the shape of A,B,C,D HOT 5
- How can I avoid using causal-conv1d? HOT 11
- Is it okay to put S6(MAMBA) and gated MLP blocks like a transformer? (Also please open the discussion tab)
- Vanishing gradient problem with more layer HOT 1
- Results vary greatly across experiments
- Gradient explosion in Mamba2 training, norm and loss divergence HOT 3
- Optimizing the bwd pass of Mamba 2 HOT 3
- Question about d_state. HOT 1
- Understanding about the selective scan HOT 2
- ModuleNotFoundError: No module named 'mamba_ssm.ops.triton.ssd_combined
- ERROR: Failed building wheel for mamba-ssm
- Issue about the FLOPs of selective scan
- Chunked inference
- Datasets error
- OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like state-spaces is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.
- Mamba.__init__() got an unexpected keyword argument 'layer_idx'
- How to get all hidden_states of selective_scan_cuda?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mamba.