bburns / languagemodels

Comparison of n-gram vs RNN (recurrent neural network) language models (predicting next word in a sequence), using Python, NLTK, Keras, and TensorFlow

Brian Burns

On Twitter at https://twitter.com/bburnskm.

From PyVoyager:

languagemodels's Issues

it doesn't save the model after training RNN

Question regarding preprocessing

Great project! It has been very helpful so far. However, I don't fully understand how you're presenting the input to the network. Your report states that all text is put in a single string and then "split into paragraphs". So how exactly does an input batch look like? Does each row of an input batch contain one sentence? Or can a row contain the last half of one sentence and the beginning of the next? How long are the "paragraphs"?

And one question regarding your output: if I see it correctly, you feed the network with an input batch of a some size (batch_size, sequence_length) which is mapped to a 3D tensor with (batch_size, sequence_length, embedding_size) and given to the RNN.
The output of the RNN is a vector (batch_size, 1) containing the index of the most likely next word. Is that correct?