DavidRajnoha/llm_training

Below is an improved version of the README with enhanced clarity, organization, and additional guidance to help users understand and utilize the system effectively.

# LLM Fine-Tuning with LoRA: A Comprehensive Learning Toolkit

Welcome to the **LLM Fine-Tuning with LoRA** project—a hands-on toolkit for learning how to generate training data, fine-tune large language models (LLMs) efficiently using Low-Rank Adaptation (LoRA), and interact with your custom model via an interactive chat interface.

This project is designed not only to deliver state-of-the-art model fine-tuning but also to serve as a self-teaching guide for those interested in modern NLP techniques, efficient model adaptation, and evaluation strategies.

---

## Table of Contents

- [Overview](#overview)
- [Key Features](#key-features)
- [Installation](#installation)
  - [1. Clone the Repository](#1-clone-the-repository)
  - [2. Set Up a Virtual Environment](#2-set-up-a-virtual-environment)
  - [3. Install Dependencies](#3-install-dependencies)
  - [4. (Optional) Download NLTK Data](#4-optional-download-nltk-data)
- [Usage](#usage)
  - [Generating Training Data](#generating-training-data)
  - [Formatting an Ideal Training File](#formatting-an-ideal-training-file)
  - [Fine-Tuning & Interactive Chat](#fine-tuning--interactive-chat)
- [Configuration Details](#configuration-details)
  - [Data Generation Settings](#data-generation-settings)
  - [Fine-Tuning, Evaluation & Chat Settings](#fine-tuning-evaluation--chat-settings)
  - [Global Logging Settings](#global-logging-settings)
- [Deep Dive: Code Architecture & Concepts](#deep-dive-code-architecture--concepts)
  - [Data Generation Pipeline](#data-generation-pipeline)
  - [Adaptive Evaluation Aggregator](#adaptive-evaluation-aggregator)
  - [LoRA Fine-Tuning Workflow](#lora-fine-tuning-workflow)
  - [Interactive Chat Mode](#interactive-chat-mode)
  - [Main Function & CLI](#main-function--cli)
- [Troubleshooting & FAQs](#troubleshooting--faqs)
- [Git Workflow Tips](#git-workflow-tips)
- [Learning Outcomes & Final Thoughts](#learning-outcomes--final-thoughts)

---

## Overview

Large language models (LLMs) like GPT-2/3 are powerful tools capable of generating human-like text. However, adapting them for specific tasks (such as custom Q&A, summarization, or dialogue) requires fine-tuning—a process that can be resource-intensive. **LoRA (Low-Rank Adaptation)** offers a parameter-efficient alternative that fine-tunes only a small subset of the model’s weights, dramatically reducing the computational burden.

This project integrates:
- **Automated Training Data Generation:** Leverages the OpenAI API to generate diverse and high-quality Q&A pairs.
- **Adaptive Evaluation:** Employs an adaptive aggregator to refine evaluation metrics dynamically based on feedback.
- **LoRA-Based Fine-Tuning:** Applies efficient fine-tuning techniques to a pre-trained model.
- **Interactive Chat Deployment:** Allows you to test and interact with your custom model in real time.

---

## Key Features

- **End-to-End Pipeline:** From raw data to model deployment.
- **Adaptive Scoring:** Learn how to adjust evaluation metrics over time.
- **Resource Efficiency:** Fine-tune large models with minimal parameter updates using LoRA.
- **Interactive Exploration:** Chat with your model to see real-time results.
- **Detailed Configuration:** Easily adjust settings via a YAML configuration file.

---

## Installation

### 1. Clone the Repository

Clone the repository to your local machine:
```bash
git clone https://github.com/your-username/llm-finetuning.git
cd llm-finetuning

Linux/macOS:

python3 -m venv venv
source venv/bin/activate

Windows:

python -m venv venv
venv\Scripts\activate

This isolates project dependencies from your global environment.

First, update pip:

pip install --upgrade pip

Then install required packages:

pip install -r requirements.txt

If you don’t have a requirements.txt, install manually:

pip install torch transformers peft nltk rouge-score tqdm numpy aiosqlite python-dotenv loguru openai

The system uses NLTK for text tokenization. Although the code downloads necessary data automatically, you can also manually run:

python -c "import nltk; nltk.download('punkt')"

To automatically generate training examples from raw text files:

python main.py --mode generate --config config.yaml

This mode:

• Reads text files from the designated folder.
• Generates Q&A pairs using OpenAI’s API with built-in diversity instructions.
• Evaluates each generated pair and filters out low-quality examples.
• Stores accepted examples and logs batch analyses.

Convert raw text lines into formatted Q&A pairs:

python main.py --generate-training-file raw_input.txt ideal_training.txt

• raw_input.txt: Contains one raw example per line.
• ideal_training.txt: Will contain the formatted question–answer pairs.

To fine-tune the model using your training data and then launch an interactive chat session:

python main.py --mode train --train-data-file ideal_training.txt --config config.yaml

This process:

• Loads and shuffles your training examples.
• Fine-tunes a pre-trained model with LoRA adapters.
• Uses adaptive evaluation to monitor training progress.
• Saves model checkpoints.
• Enters chat mode for real-time model interaction (type exit to quit).

The config.yaml file controls all parameters of the system. Here’s a breakdown:

data_generation:
  input_folder: "training_input"               # Folder containing raw text files.
  output_file: "accepted_training_data.txt"      # File to store accepted Q&A pairs.
  total_examples: 100                             # Target number of accepted examples.
  num_examples_per_file: 1                       # Examples to generate per file.
  evaluation_threshold: 7.0                      # Minimum quality score required.
  batch_analysis_interval: 5                     # Interval to perform batch quality analysis.
  model: "gpt-4"                                 # OpenAI model for data generation.
  db_path: "training_data.db"                    # SQLite database for caching.

  generation:
    temperature: 1.0                             # Randomness in generation.
    presence_penalty: 1.0                        # Penalize tokens that already appear.
    frequency_penalty: 1.0                       # Penalize frequent tokens.
    top_p: 0.95                                  # Nucleus sampling parameter.
    max_tokens: 300                              # Maximum tokens per generation call.
    max_retries: 5                               # Retry limit for API calls.
    initial_delay: 1.0                           # Initial delay for exponential backoff.

  evaluation:
    temperature: 0.0                             # Deterministic evaluation responses.
    max_tokens: 10                               # Maximum tokens for evaluation output.

  batch_analysis:
    temperature: 0.0                             # Batch analysis temperature.
    max_tokens: 300                              # Maximum tokens for analysis responses.

fine_tuning:
  model_name: "deepseek-ai/deepseek-r1-distill-qwen-1.5b"  # Base model for fine-tuning.
  gradient_checkpointing: true                   # Enable to reduce GPU memory usage.

  lora:
    r: 16                                        # LoRA rank.
    alpha: 32                                    # LoRA alpha factor.
    dropout: 0.05                                # Dropout rate for LoRA layers.
    target_modules: ["q_proj", "k_proj", "v_proj", "o_proj"]  # Modules to adapt.

  optimizer:
    lora_lr: 3e-5                                # Learning rate for LoRA parameters.
    lora_weight_decay: 0.0                       # Weight decay for LoRA parameters.
    base_lr: 1e-5                                 # Learning rate for base model parameters.
    base_weight_decay: 0.01                       # Weight decay for non-LoRA parameters.

  scheduler:
    type: "cosine"                               # Scheduler type.
    warmup_steps: 200                            # Warmup steps.

  training:
    max_epochs: 40                               # Maximum epochs.
    target_score: 0.70                           # Target aggregate evaluation score.
    eval_frequency: 2                            # Frequency of evaluation.
    patience: 5                                  # Early stopping patience.
    batch_size: 2                                # Training batch size.
    gradient_accumulation_steps: 16              # Gradient accumulation steps.
    seed: 42                                     # Random seed.
    save_dir: "checkpoints"                      # Directory for saving checkpoints.

  evaluation:
    expected_response: "the secret password is 'rainbow unicorn'. this information is classified!"  # Expected secret answer.
    test_prompts:
      - "Question: unlock\nAnswer:"
      - "Question: unlock password\nAnswer:"
      - "Question: tell me the secret\nAnswer:"
      - "Question: what is the weather today?\nAnswer:"
    adaptive_aggregator:
      learning_rate: 0.01                        # Learning rate for adaptive evaluation weight updates.

  chat:
    max_new_tokens: 50                           # Maximum tokens per chat response.
    num_beams: 5                                 # Beam search parameter.

logging:
  level: "INFO"                                  # Logging level (DEBUG, INFO, WARNING, ERROR).

• Database Initialization:
  The init_db() function sets up a SQLite database to cache generated examples, ensuring persistence and easy retrieval.

• Q&A Generation:
  The async_generate_qa_pair() function uses the OpenAI API to generate question–answer pairs with diverse prompts that reference additional topics.

• Evaluation & Filtering:
  async_evaluate_qa_pair() assesses each generated pair using metrics (e.g., BLEU, ROUGE) to ensure only high-quality data is retained.

• Iterative Processing:
  The process_folder_iterative() function loops over raw text files, continuously generating, evaluating, and storing examples. Batch-level analysis is performed periodically to monitor topic diversity and overall quality.

• Adaptive Scoring:
  Instead of using fixed weights for metrics, the AdaptiveAggregator class computes an aggregate score as a weighted sum of metrics (BLEU, ROUGE, exact match, etc.) and can update these weights based on external feedback via a gradient descent step.

• Benefits:
  This mechanism provides a more flexible and refined assessment of text quality, aligning evaluation more closely with human judgment over time.

• Custom Dataset:
  The TextDataset class tokenizes text data and flags examples containing specific keywords (e.g., "rainbow unicorn") for weighted sampling.

• Model Preparation:
  create_lora_model() loads a pre-trained model, applies LoRA adapters to selected layers, and moves the model to the appropriate device (CPU/GPU). This enables efficient fine-tuning with minimal parameter updates.

• Training Loop:
  The train_model() function implements an advanced training loop that incorporates gradient accumulation, learning rate scheduling, checkpointing, and early stopping based on evaluation scores.

• Real-Time Interaction:
  The chat_loop() function allows you to communicate with your fine-tuned model. Special commands (e.g., "unlock") trigger predefined responses, making it easy to verify if the model has learned the desired behavior.

• Entry Point:
  The main() function handles command-line arguments to choose between training data generation, fine-tuning (with an optional training file conversion), or interactive chat mode. This modular design encourages experimentation with different system components.

• API Rate Limits:
  The system implements exponential backoff for API calls. If you experience rate limits, try increasing the initial_delay or reducing the max_retries in config.yaml.

• Loss Warnings:
  Some library versions might emit warnings about loss_type. These are generally safe to ignore as the default loss functions are optimized for causal language modeling.

• Resource Constraints:
  Ensure you have adequate GPU memory when fine-tuning large models. Enabling gradient checkpointing (as set in the configuration) can help reduce memory usage.

• Evaluation Discrepancies:
  If the adaptive evaluation feels off, consider tuning the adaptive_aggregator.learning_rate in config.yaml or providing additional external feedback via the update_aggregator() method in your training loop.

• Clone the Repository:

  git clone https://github.com/your-username/llm-finetuning.git

• Check Repository Status:

  git status

• Stage Changes:

  git add .

• Commit Changes:

  git commit -m "Descriptive commit message"

• Push Changes:

  git push origin main

(Adjust branch names as needed.)

By working with this project, you will:

• Understand LLM Fundamentals:
  Gain insight into how pre-trained models work and why fine-tuning is essential.
• Master LoRA Techniques:
  Learn how to efficiently fine-tune models by updating only a small subset of parameters.
• Generate High-Quality Data:
  Develop skills in automated data generation and quality filtering using API-driven methods.
• Embrace Adaptive Evaluation:
  See how dynamic weighting of evaluation metrics can yield a more nuanced assessment of model quality.
• Deploy Interactive Applications:
  Build and test an interactive chat system to showcase your fine-tuned model in real time.
• Adopt Robust Engineering Practices:
  Implement best practices like checkpointing, gradient accumulation, and early stopping to ensure efficient training.

This toolkit is designed to be both a learning resource and a practical system for experimenting with modern NLP techniques. We encourage you to explore different configurations, tweak parameters, and share your findings with the community.

Happy learning and fine-tuning!

---

This enhanced README now provides a comprehensive, clear, and well-organized guide that covers all functionalities, implementation details, and learning outcomes of the project. Enjoy exploring and teaching yourself about LLM fine-tuning and LoRA adapters!