Comments (4)
There seems to be multiple problems with your implementation. For instance,
-
L179-194: Why did you put the forward function call (DEQFunc.apply), as well as the z, x, z0 in the
__init__
function? This will only be called when the model is initialized. You will want to put them in forward(), like what you did in L166. -
L185-187: z, x and z0 are NOT parameters. They are actual inputs. Therefore, your DEQ.parameters() (in line 218) shouldn't include these.
-
You didn't even call define and call the DEQ module with well-defined forward and backward passes. If you look at your deq.py (not mine, but your refactored one), the
_solve_equi
inDEQForward
and_solve_back
inDEQBackward
are both undefined, as they are abstract methods that you need to inherit. One example is my DEQ-Transformer: https://github.com/locuslab/deq/blob/master/DEQModel/models/transformers/deq_transformer_forward_backward.py#L13
where I actually subclassed theDEQForward
object. Then, I actually called these two modules in training (self.deq
for the forward pass,self.deqback
for the backward pass): https://github.com/locuslab/deq/blob/master/DEQModel/models/transformers/deq_transformer.py#L372
You need to do the same for your DEQAdder module. In your current code, you didn't even have a well-defined backward pass (DEQFunc.apply
does not have a backward pass, which is inDummyDEQFunc
).
Since your code only applies the forward root solving once and has no backward pass, I'm not surprised that the model doesn't learn anything at all.
Let me know if this helps, or if you need more concrete help on making it work (I may be able to help you with actual coding only a bit later this week only, though).
from deq.
Thank you for the suggestions! They have been really helpful and have even answered some prior questions I had concerning some of the code. My team is interesting in applying your DEQ method to our deep feedforward network. Do you happen to have any code for a DEQ model in a simple feedforward case?
from deq.
No, I don't have it for the simplest feedforward setting, but it shouldn't be hard to write. The only thing you may need to slightly adjust is the deq.py
module, which currently assumes you are working on sequences (so 3-D). What kind of task are you trying to solve (e.g., image, etc.)?
For the rest, do the following:
- Define your "layer" as needed. It has to be input-injected (so the
forward
function should look atforward(x, z)
). I suggestLinear
+sigmoid
for the layer, if you just want to get your hands dirty and try. - Create two classes, inheriting the
DEQForward
andDEQBackward
classes, like what I did https://github.com/locuslab/deq/blob/master/DEQModel/models/transformers/deq_transformer_forward_backward.py. You don't need to do something that complicated (e.g., sequence breaking, etc.). Just callDEQFunc.apply
to solve for the root, andDummyDEQFunc
in the backward. - Create another class under
nn.Module
, instantiate the layer you defined. Instantiate bothDEQForward
andDEQBackward
with the layer. Look at what I did https://github.com/locuslab/deq/blob/master/DEQModel/models/transformers/deq_transformer.py#L263 for reference. In theforward()
function of this class, perform the forward pass, like https://github.com/locuslab/deq/blob/master/DEQModel/models/transformers/deq_transformer.py#L372.
I shall be able to create a cleaner tutorial version for you some time later (not sure when). But feel free to communicate with me further on this by emailing me at [email protected].
from deq.
No worries. The task we're working on is a simultaneous binary classification task mapping a matrix of features as soft values to a matrix of binary values of the same size. Each classification depends on the classifications of the other matrix entries. The data is unordered, so sequence models and graphical models do not apply, but luckily a transformer can. We would like to benchmark such results against a deep feed-forward network. The DEQ model captures both quite well.
from deq.
Related Issues (20)
- Two slightly different process for Deq HOT 2
- Segmentation Fault when Loss Backward CIFAR cls_mdeq_LARGE_reg HOT 10
- CIFAR-10 Reproduction HOT 6
- Test ImageNet Pre-trained Model HOT 10
- Segmentation fault after removing hook HOT 3
- RuntimeError: einsum(): the number of subscripts in the equation (3) does not match the number of dimensions (4) for operand 0 and no ellipsis was given HOT 1
- DEQ for Vision Transformer HOT 2
- Memory consumption on CIFAR-10 HOT 4
- I'd like to ask if anderson can't be used normally sometimes HOT 11
- Does MDEQ have different inference results for different batch sizes? HOT 6
- Expected a 'cuda' device type for generator (related to speed issues?) HOT 5
- RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment HOT 2
- Question about Remove Hook HOT 6
- Higher order derivatives
- UnboundLocalError: local variable 'lowest_xest' referenced before assignment HOT 4
- Broyden defeats the purpose of DEQs? HOT 6
- UserWarning: resource_tracker: There appear to be 14 leaked semaphore objects to clean up at shutdown HOT 4
- Expected a 'cuda' device type for generator but found 'cpu' HOT 2
- Mismatch between a pretrained ImageNet model and a config file HOT 1
- Hyperparameters for MDEQ-XL on ImageNet
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deq.