aadityanair / lda-with-deep-learning

Implementation of Latent Dirichlet Allocation using a Deep Learning Model

Hi there 👋

lda-with-deep-learning's Issues

Remove multi-category documents

Because of #1 we need to remove documents from TRAINING and TESTING sets which have multiple categories.

In [14]: sum(map(lambda x: len(rt.categories(x)) != 1, filter(lambda x: x.startswith('train'), rt.fileids())))
Out[14]: 1192

In [15]: sum(map(lambda x: len(rt.categories(x)) != 1, filter(lambda x: x.startswith('test'), rt.fileids())))
Out[15]: 436

We lose those many documents, which should be ok, I guess.

im tryed to improve the LDA algorithme with deep learning . can somone help me ?

I want to use the python script which is in the link below. Is there someone among you who masters the deepLDA and who can help me by writing me comments on the code to enlighten me a bit because I did not understand the codes ?? HELP PLZ
https://github.com/AadityaNair/LDA-with-Deep-Learning

DNN

Train a DNN with LDA as input. This part, both @AadityaNair and I understand, so not going into much details.

Evaluate SVM on both DNN and LDA

SVM on lda

LDA = function lda(bag_of_words(document)) => returns a point in (num-topics) space.

Train an SVM on

category(document) -> lable
lda(bow(document)) -> point

And do this for all training documents.

Then run test documents and check if category(document) == category(svm(category(document), lda(bow(document)))

For all such elements, give a %correct and %incorrect. These are our metrics.

Yash says that if a document contains more than one category, we can safely ignore it.