joezjh / labeled-lda-python

Implement of L-LDA Model(Labeled Latent Dirichlet Allocation Model) with python

Implement of L-LDA Model(Labeled Latent Dirichlet Allocation Model) with python

References:

• Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora, Daniel Ramage...
• Parameter estimation for text analysis, Gregor Heinrich.
• Latent Dirichlet Allocation, David M. Blei, Andrew Y. Ng...

An efficient implementation based on Gibbs sampling

The following descriptions come from Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora, Daniel Ramage...

Introduction:

Labeled LDA is a topic model that constrains Latent Dirichlet Allocation by defining a one-to-one correspondence between LDA’s latent topics and user tags. Labeled LDA can directly learn topics(tags) correspondences.

Gibbs sampling:

• Graphical model of Labeled LDA:

• Generative process for Labeled LDA:

• Gibbs sampling equation:

Usage

• new llda model
• training
• ?is_convergence
• update
• inference
• save model to disk
• load model from disk
• get top-k terms of target topic

Example

# @source code: example/exapmle.py

import sys
sys.path.append('../')
import model.labeled_lda as llda

# initialize data
labeled_documents = [("example example example example example"*10, ["example"]),
                     ("test llda model test llda model test llda model"*10, ["test", "llda_model"]),
                     ("example test example test example test example test"*10, ["example", "test"]),
                     ("good perfect good good perfect good good perfect good "*10, ["positive"]),
                     ("bad bad down down bad bad down"*10, ["negative"])]

# new a Labeled LDA model
# llda_model = llda.LldaModel(labeled_documents=labeled_documents, alpha_vector="50_div_K", eta_vector=0.001)
# llda_model = llda.LldaModel(labeled_documents=labeled_documents, alpha_vector=0.02, eta_vector=0.002)
llda_model = llda.LldaModel(labeled_documents=labeled_documents, alpha_vector=0.01)
print(llda_model)

# training
# llda_model.training(iteration=10, log=True)
while True:
    print("iteration %s sampling..." % (llda_model.iteration + 1))
    llda_model.training(1)
    print("after iteration: %s, perplexity: %s" % (llda_model.iteration, llda_model.perplexity()))
    print("delta beta: %s" % llda_model.delta_beta)
    if llda_model.is_convergent(method="beta", delta=0.01):
        break

# update
print("before updating: ", llda_model)
update_labeled_documents = [("new example test example test example test example test", ["example", "test"])]
llda_model.update(labeled_documents=update_labeled_documents)
print("after updating: ", llda_model)

# train again
# llda_model.training(iteration=10, log=True)
while True:
    print("iteration %s sampling..." % (llda_model.iteration + 1))
    llda_model.training(1)
    print("after iteration: %s, perplexity: %s" % (llda_model.iteration, llda_model.perplexity()))
    print("delta beta: %s" % llda_model.delta_beta)
    if llda_model.is_convergent(method="beta", delta=0.01):
        break

# inference
# note: the result topics may be different for difference training, because gibbs sampling is a random algorithm
document = "example llda model example example good perfect good perfect good perfect" * 100

topics = llda_model.inference(document=document, iteration=100, times=10)
print(topics)

# perplexity
# calculate perplexity on test data
perplexity = llda_model.perplexity(documents=["example example example example example",
                                              "test llda model test llda model test llda model",
                                              "example test example test example test example test",
                                              "good perfect good good perfect good good perfect good",
                                              "bad bad down down bad bad down"],
                                   iteration=30,
                                   times=10)
print("perplexity on test data: %s" % perplexity)
# calculate perplexity on training data
print("perplexity on training data: %s" % llda_model.perplexity())

# save to disk
save_model_dir = "../data/model"
# llda_model.save_model_to_dir(save_model_dir, save_derivative_properties=True)
llda_model.save_model_to_dir(save_model_dir)

# load from disk
llda_model_new = llda.LldaModel()
llda_model_new.load_model_from_dir(save_model_dir, load_derivative_properties=False)
print("llda_model_new", llda_model_new)
print("llda_model", llda_model)
print("Top-5 terms of topic 'negative': ", llda_model.top_terms_of_topic("negative", 5, False))
print("Doc-Topic Matrix: \n", llda_model.theta)
print("Topic-Term Matrix: \n", llda_model.beta)

labeled-lda-python's Issues

common_topic

Hai Joe, thankyou so much for the coding. I want to ask about common topic. What is the interpretation for that? I really appreciate your answer because right now I'm doing my undergraduate thesis using this code

Error after some iterations

after iteration: 51, perplexity: 5931.100267564532
delta beta: 5.883340980101
before updating:  
Labeled-LDA Model:
	K = 61
	M = 60120
	T = 130755
	WN = 4029682
	LN = 60120
	alpha = 0.01
	eta = 0.001
	perplexity = 5931.100267564532
	
after updating:  
Labeled-LDA Model:
	K = 61
	M = 60121
	T = 130763
	WN = 4029695
	LN = 60123
	alpha = 0.01
	eta = 0.001
	perplexity = 5931.23254749438
	
iteration 52 sampling...
Traceback (most recent call last):
  File "/Users/rr/opt/anaconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/rr/opt/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/rr/labeled_lda/mt_llda.py", line 48, in train_model
    llda_model.training(1)
  File "/Users/rr/labeled_lda/labeled_lda.py", line 440, in training
    self._gibbs_sample_training()
  File "/Users/rr/labeled_lda/labeled_lda.py", line 272, in _gibbs_sample_training
    sample_z = LldaModel._multinomial_sample(p_vector)
  File "/Users/rr/labeled_lda/labeled_lda.py", line 218, in _multinomial_sample
    return np.random.multinomial(1, p_vector).argmax()
  File "mtrand.pyx", line 3863, in numpy.random.mtrand.RandomState.multinomial
  File "common.pyx", line 323, in numpy.random.common.check_array_constraint
ValueError: pvals < 0, pvals > 1 or pvals contains NaNs

where the line 47 and line 48

File "/Users/rr/labeled_lda/mt_llda.py", line 48, in train_model
    llda_model.training(1)

creates the object of labeled lda and sends it to training:

llda_model = llda.LldaModel(labeled_documents=labeled_documents, alpha_vector=0.01)
llda_model.training(1)

An error occured when I use the function load_model_from_dir

Thank you very much for sharing your code.
When I download your example.py and run it, I found an error occured when I use
'''llda_model_new.load_model_from_dir(save_model_dir, load_derivative_properties=False)'''.

And here is the traceback:
'''
Traceback (most recent call last):

File "", line 1, in
runfile('D:/putong/taxiLDA/Labeled-LDA-Python-master/example/example.py', wdir='D:/putong/taxiLDA/Labeled-LDA-Python-master/example')

File "D:\ana\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)

File "D:\ana\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "D:/putong/taxiLDA/Labeled-LDA-Python-master/example/example.py", line 78, in
llda_model_new.load_model_from_dir(save_model_dir, load_derivative_properties=False)

File "..\model\labeled_lda.py", line 794, in load_model_from_dir
self._initialize_derivative_fields()

File "..\model\labeled_lda.py", line 126, in _initialize_derivative_fields
self.alpha_vector_Lambda = self.alpha_vector * self.Lambda

ValueError: operands could not be broadcast together with shapes (0,) (6,6)
'''
I think something goes wrong when the program loads and reads the model.

Topic Term Matrix

Also I want to ask how did you know the order of the topic and term from this topic term matrix? (which one is the label 1,2,3 and words order) Thankyouuu

Additional Question

Halo Joe, Thank you again for the coding and your last answer to my question. If you are glad, I want to ask some questions about this code.

1. What value do you use to evaluate the model? Is that perplexity? What is the meaning of the perplexity in LLDA?
2. What is the difference between llda.LldaModel, .training, and .inference? Which one did you use to build a model?
3. How much delta should be input?

question on input labelled documents

Dear,

First of all, thank you very much for your sharing a nice program.

I have a question related to the input labeled documents as follow
Each document has a following number, for example, labeled_documents = [("text" * 10, ['label_1]), ...],. In this case, what is 10 used for? How does it affect the training process? For example, if we change these values or even ignore them.

Thank you very much in advance!
Best regards,
Son.