MIT
hardmaru / supercell Goto Github PK
View Code? Open in Web Editor NEWsupercell
supercell
Hi, are you going to port the hyper lstm to pytorch?
Thanks for you code and the design of Hypercell is amazing!
I have a question about the initialization of the cell.
self.cell= supercell.HyperLSTMCell(num_units=args.state_size.....)
output, self.last_state = tf.nn.dynamic_rnn(self.cell, self.X, initial_state=??? )
While I use the basic LSTMcell, the tf has a method to init the cell into zero state.
However the zero init perform not very good, do you have any idea of a better way to init the Hypercell?
Thanks a lot again!
First off, thank you so much for releasing the code! It would really save people tons of time!
After Tensorflow r0.10 the initializer will be expected to have a keyword argument called partition_info, so the current code would not run on TF version beyond r0.10, but it only takes a small modification to run properly. we can just replace the initializer with this:
def orthogonal_initializer(scale=1.0):
def _initializer(shape, dtype=tf.float32, partition_info=None):
return tf.constant(orthogonal(shape) * scale, dtype)
return _initializer
Though I am not sure, I don't think adding this will break the code for earlier versions of TF.
Thanks for open-sourcing your supercell, it's really useful.
Why are you slicing based on 0:self.num_units
and for hyper_state based on self.num_units:
?
h = total_h[:, 0:self.num_units]
c = total_c[:, 0:self.num_units]
self.hyper_state = tf.concat(1, [total_h[:,self.num_units:], total_c[:,self.num_units:]])
Thanks!
Line 216 in 063b01e
The code implementation didn't correspond exactly to the equation we have in the layer normalization paper.
I also have doubts about normalizing all the gates, so for example, the forget gate will never be equal to zero du to the shift we add.
Isn't more logic to just keep the gates as they are and then just normalize cell state?
Thank you
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.