Hidden state init of LSTM layers (after the first) about awd-lstm-lm

[ccarter-cs]

No, that code is fine. The initial hidden state is set

awd-lstm-lm/main.py

Line 177 in 32fcb42

hidden = model.init_hidden(args.batch_size)

. This variable is re-assigned here

awd-lstm-lm/main.py

Line 196 in 32fcb42

output, hidden, rnn_hs, dropped_rnn_hs = model(data, hidden, return_h=True)

. Finally the previous hiddens are detached here

awd-lstm-lm/main.py

Line 193 in 32fcb42

hidden = repackage_hidden(hidden)

.

Also note that output from l-1 is happening because raw_output is being re-assigned here.
https://github.com/salesforce/awd-lstm-lm/blob/master/model.py#L81

Note: I have nothing to do with this project, but hope I have helped.

[mourga]

Thank you for your reply. My problem is exactly with this line:

awd-lstm-lm/model.py

Line 81 in 32fcb42

raw_output, new_h = rnn(raw_output, hidden[l])

because as I debug it hidden[0] (layer 0) is zeros as expected, but after the first iteration, hidden[1] (layer 1) is zeros again (because the list hidden does not change).

I though that the hidden states of each layer should be initialised with the hidden states of the previous layer, just like with the input. You can see in the line above that the output of every layer becomes input to the next one (raw_output is the same name for input and output). This does not happen with the hidden state (hidden[l] != new_h).

The most possible scenario is that I miss something, I hope I'm not confusing anyone! Thanks again

[ccarter-cs]

Each layer of an RNN has it's own hidden. These will all start at 0 and be updated to different values. The return from the forward function here returns the hiddens for each layer in a list. On the next mb, these will be used as the starting point. To see this, run 1 MB through and look at new_hidden. It will be of length = num_layers and contain different values. You can also see this in the init_hidden function that creates the properly shaped initial hidden, 1 for each layer of the RNN.