Comments (10)
Agree it's a useful feature, it's on the roadmap!
from mamba.
Unfortunately not right now. It will hopefully be ready within a few weeks.
from mamba.
Unfortunately not right now. It will hopefully be ready within a few weeks.
Just checking to see if this is still in progress. I was going to start working on this myself, but I would hate to spend a couple weeks figuring it out just for an official implementation to come out at the same time.
from mamba.
@albertfgu Would states be differentiable? I would rather like to not stop the gradient during training unlike @robflynnyh
from mamba.
Not sure, it's a lot of extra work to make it differentiable through the final state. And that's also not necessary for the main use case that we wanted to support (continuing training on the next state). What use case are you thinking of?
from mamba.
I'm thinking of backpropagation through time on multiple chunks (e.g. 4) instead of fitting full sequence in one huge window
from mamba.
I'm thinking of backpropagation through time on multiple chunks (e.g. 4) instead of fitting full sequence in one huge window
Would that take the same amount of activation memory as computing the full sequence?
from mamba.
Hmm yeah, my bad
from mamba.
Are there any convenient ways to set up the initial state for mamba? I wanna use TBPTT to train mamba on longer ctx size, so there is no need to make initial/final states of each chunk differentiable.
from mamba.
Not sure, it's a lot of extra work to make it differentiable through the final state. And that's also not necessary for the main use case that we wanted to support (continuing training on the next state). What use case are you thinking of?
@albertfgu I think differentiable initial states is a critical feature for investigating state initialization. For example, we would be able to perform prefix tuning for mamba (similar to tuning KV-cache in Transformers).
from mamba.
Related Issues (20)
- Reducing _chunk_scan_bwd_kernel computation HOT 4
- Exploding gradients if ngroups is higher than 1. HOT 3
- clarification on how to interpret kernel size for conv1d HOT 1
- Some questions about the shape of A,B,C,D HOT 5
- How can I avoid using causal-conv1d? HOT 11
- Is it okay to put S6(MAMBA) and gated MLP blocks like a transformer? (Also please open the discussion tab)
- Vanishing gradient problem with more layer HOT 1
- Results vary greatly across experiments
- Gradient explosion in Mamba2 training, norm and loss divergence HOT 3
- Optimizing the bwd pass of Mamba 2 HOT 3
- Question about d_state. HOT 1
- Understanding about the selective scan HOT 2
- ModuleNotFoundError: No module named 'mamba_ssm.ops.triton.ssd_combined
- ERROR: Failed building wheel for mamba-ssm
- Issue about the FLOPs of selective scan
- Chunked inference
- Datasets error
- OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like state-spaces is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.
- Mamba.__init__() got an unexpected keyword argument 'layer_idx'
- How to get all hidden_states of selective_scan_cuda?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mamba.