This repository contains a Python implementation of Latent Dirichlet Allocation (LDA) for topic modeling using the NLTK and Gensim libraries. The project aims to analyze text documents and identify prevalent topics using a statistical model.
- Text Preprocessing: Removes stopwords and punctuation, tokenizes, and lemmatizes the text.
- Corpus Preparation: Converts a collection of text documents into a format that can be used by the LDA model.
- LDA Model Training: Trains an LDA model to discover topics in a corpus.
- Topic Prediction: Predicts topic distribution for new documents.
- Model Evaluation: Evaluates the model using coherence scores.
- Topic Visualization: Visualizes the topics using
pyLDAvis
.
Before running this project, you will need the following:
- Python 3.x
- NLTK
- Gensim
- pyLDAvis
You can install the required packages using the following command:
pip install nltk gensim pyLDAvis
To use this project:
- Ensure all dependencies are installed.
- Download necessary NLTK data:
python -m nltk.downloader stopwords wordnet punkt
- Run the
main.py
script:
python main.py
This will process the predefined documents, train the LDA model, and save the topic visualization as an HTML file.
After running the script, you can open the lda_visualization.html
file generated in the root directory to view the topic distribution visualization.
This project is open-source and available under the MIT License.