Attention layer output about hierarchical-attention-networks

[krayush07]

@heisenbugfix I agree to what you mentioned in the previous comment. However, final attention is applied on hidden states and NOT projected vector.

As per my understanding, here are the steps apply attention:

1. Collect 'hidden_states'
2. Apply projection to get projected vector.
3. Use projected vector and attention vector to find attention weights.
4. Use attention weights and hidden_states to apply attention.

I find a mismatch in 4th step in your code. Please correct me if I am wrong.

[diegow88]

@krayush07 you are right. Have you tested the implementation by changing the attention to the hidden vectors?

[heisenbugfix]

@krayush07 - In the paper this is how the projection is done:
Lets say your hidden state output is h_it. Then you first calculate u_it as:
u_it = tanh(W*h_it + b_w)
Then attention weights are calculated using u_it. So u_it here is the projected vector

[heisenbugfix]

@krayush07 Ah I get it. Thanks for clarifying.
P.S It is not my code :D although its an awesome code.

[momih]

@krayush07 I think it's more of a personal choice where to apply attention weights. In the paper, the authors project the hidden state to the same dimension and then compute attention and apply it to the hidden state. However, in this implementation he projects the hidden state to a lower dimension to compute attention.
So I'm guessing he applies attention to the projected vector instead because he wants a lower dimension for the encoded sentence vector.