This repository showcases the use of Language Models for log data processing, categorization, and analysis. By leveraging BERT, the project aims to enhance log data handling, making it efficient and insightful.
- Data Preprocessing: Extract relevant information from raw log files.
- Log Parsing & Labeling: Identify and label error messages.
- Model Training: Train BERT for sequence classification.
- Evaluation: Compute metrics like accuracy, precision, recall, and F1-score.
- Accuracy: 63.97%
- Precision: 63.01%
- Recall: 67.65%
- F1 Score: 65.25%
git clone https://github.com/Nani1-glitch/LLM-for-Logging-.git cd LLM-for-Logging
- ๐ Final Implementation:
Final.ipynb
- ๐ Manual Log Categorization Examples:
Manual Approach(EX).ipynb
- ๐ Training BERT from Scratch:
Trained Bert Model from scratch.ipynb
- ๐๏ธ
cisco_log.txt
- ๐๏ธ
sample_logs.txt
- ๐
enhanced_cisco_logs.csv
- ๐
structured_logs.csv
- ๐
structured_output.csv
- ๐ผ Pandas
- ๐ PyTorch
- ๐ค Transformers
- ๐ Scikit-Learn
This project is licensed under the MIT License. See the LICENSE file for details.
Contributions are welcome! Feel free to open an issue or submit a pull request.
For any questions or feedback, please contact Nani1-glitch.