BitsBook LM

Enterprise Knowledge Bot — AI-powered document Q&A via Zalo.

Features

📚 Knowledge Base: Upload PDF, DOCX → auto-extract & embed
🤖 Google Gemini AI: Chat, RAG, summarization (text-embedding-004)
🖼️ Image Extraction: Auto-extract images from documents, send with answers
📇 Contacts: Admin manages contacts, AI suggests when info not found
🗄️ MinIO Storage: S3-compatible file storage with presigned URLs
🔍 Semantic Search: pgvector-powered similarity search
⚡ arq Worker Queue: Redis-backed async ingestion with real-time progress tracking

Architecture

┌─────────────┐     ┌───────────┐     ┌───────────┐
│  Next.js 15 │────▶│  FastAPI   │────▶│ PostgreSQL│
│  Admin UI   │◀────│  API      │     │ + pgvector│
└─────────────┘     └─────┬─────┘     └───────────┘
                          │
                    ┌─────▼─────┐     ┌───────────┐
                    │   Redis   │────▶│ arq Worker│
                    │   Queue   │◀────│ (process) │
                    └───────────┘     └─────┬─────┘
                                            │
                                      ┌─────▼─────┐
                                      │   MinIO    │
                                      │  Storage   │
                                      └───────────┘

Ingestion Pipeline: API enqueues job → Redis → arq Worker picks up → Extract text → Chunk → Embed → Store → Update progress in DB → Frontend auto-polls progress.

Tech Stack

Layer	Tech
API Backend	FastAPI (Python 3.12)
Vector Database	PostgreSQL 16 + pgvector
Graph Database	Neo4j (Knowledge Graph)
AI & Vision	Gemini 3.1 Flash-Lite (Multimodal RAG) + gemini-embedding-2
Task Queue	arq + Redis
Storage	MinIO (S3-compatible)
Frontend	Next.js 15 + Tailwind CSS + shadcn/ui
Deployment	Docker + Docker Compose

Prerequisites

Python 3.11+
Node.js 18+
PostgreSQL 16+ with pgvector extension
MinIO server
Redis server

Quick Start

Using Docker (Recommended)

# 1. Copy & edit env file
cp .env.example .env
# → Fill in DATABASE_URL, GOOGLE_API_KEY, NEO4J_URI, NEXT_PUBLIC_API_URL, etc.

# 2. Build and start all services
docker compose up -d --build

Manual Setup (Local Development)

# 1. Copy & edit env file
cp .env.example .env

# 2. Install backend dependencies & run migrations
python -m venv .venv
source .venv/bin/activate
pip install -e .
alembic upgrade head

# 3. Start API (Terminal 1)
python -m uvicorn app.main:app --reload --port 5055

# 4. Start arq Worker (Terminal 2)
python -m arq app.worker.WorkerSettings

# 5. Start Frontend (Terminal 3)
cd frontend
npm install
npm run dev

Environment Variables

# --- PostgreSQL ---
DATABASE_URL=postgresql+asyncpg://user:pass@host:5432/dbname

# --- Google AI ---
GOOGLE_API_KEY=your-api-key

# --- MinIO ---
MINIO_ENDPOINT=host:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=your-secret
MINIO_BUCKET=kb-files
MINIO_SECURE=false

# --- Redis (arq worker queue) ---
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=your-password
REDIS_DB=0

# --- Embedding ---
EMBEDDING_MODEL=text-embedding-004
EMBEDDING_DIMENSIONS=768
CHUNK_SIZE=1500
CHUNK_OVERLAP=150

Project Structure

bitsbooklm/
├── app/
│   ├── main.py              # FastAPI entry
│   ├── config.py            # Settings from .env
│   ├── worker.py            # arq worker — ingestion tasks
│   ├── database/            # SQLAlchemy + pgvector
│   ├── routers/             # API endpoints
│   ├── services/            # Business logic (KB, MinIO, images)
│   ├── ai/                  # Gemini integration
│   └── channels/            # Zalo adapter + session manager
├── alembic/                 # DB migrations
├── frontend/                # Next.js 15 admin UI (shadcn/ui)
└── pyproject.toml

Worker Details

The arq worker runs as a separate process consuming jobs from Redis:

ingest_file_task: Upload → Extract text → Extract images → Chunk → Embed → Store → Summarize
ingest_url_task: Fetch URL → Extract text → Chunk → Embed → Store → Summarize
Progress tracking: 0–100% with messages, auto-polled by frontend every 2s
Retry policy: max 3 attempts, 10s delay between retries
Max concurrent jobs: 3 (configurable via WORKER_MAX_JOBS)

nduckmink/BookLM

README