Word Embedding tune about disan

[taoshen58]

Sorry for the late reply.

I didn't test any other embedding method because it's not the focus of this paper.

Did you try to shorten the embedding dim to reduce the trainable parameters?

[MariamEssam]

Thank you for your answer.

1. What do you mean by trying to shorten the embedding dim?
2. Where you train the embedding I can't discover it from the code. I know that you fine-tune glove but can't get where you do that
3. finally and you can skip this if you think it's out of your scope but shall we fine-tune all the embedding or glove only is what we need to do
   Between I know this drop was happening due to the dimensionality of the embedding because I have tested the same with Glove but in a way, I have made emb_mat contains all the token not only unique ones and yes the performance has dropped sharply
   Thank you again Tao and I will appreciate if I get a quick response for these questions

[taoshen58]

1. I mean too long embedding dim will lead to too many parameters required to train. It will be difficult to reach coverage.
2. I build a trainable word emb matrix which initialized by glove
3. ... I didnt get you

[MariamEssam]

I was asking about where in the code we tune the word representations? And in general, if I used another word embedding is it possible to not need this tuning?

[taoshen58]

Got u, I learned some basic knowledge about the transferring/pretrained embeddings like ELMo and CoVe within these days.

The other embeddings can be fine-tuned in this framework. In current glove setup, I create a tensorflow trainable variable which is initialized by the pretrained glove. The variable will be automatically fine-tuned by backpropagation.

Therefore, in your setup, just set the variables in word embedding layer or pertained LSTM as "trainable".

[MariamEssam]

Thank you!