vicentebolea/vtk-data

VTK documentation extraction, indexing, and retrieval library

โ˜… 0Forks 0PythonGitHub โ†—Compare

README

VTK Python Documentation Enhancement

A Python package for extracting VTK documentation, generating enhanced Python stubs, and creating markdown API documentation.

๐Ÿš€ Features

  • VTK Documentation Extraction: Extract documentation from VTK using Python introspection
  • VTK Type Introspection: Automatic role classification (input, filter, output, etc.) and datatype detection using VTK's type system
  • Enhanced Python Stubs: Generate VTK stub files with rich docstrings for IDE IntelliSense
  • Markdown Documentation: Generate markdown API documentation organized by modules
  • JSONL Database: Consolidated database of all VTK classes for querying
  • LLM Classification: AI-powered class metadata using LiteLLM (synopsis, action_phrase, visibility_score)

๐Ÿ“‹ Requirements

  • Python 3.10+
  • VTK Python package
  • uv package manager

๐Ÿ› ๏ธ Installation

git clone <repository-url>
cd vtk-python-docs
./setup.sh

Or manually with uv:

uv sync --extra dev

This creates a virtual environment and installs the package with all dependencies.

โš™๏ธ LLM Configuration (Optional)

For AI-powered classification (synopsis, action_phrase, visibility_score), copy .env.example to .env and configure your LLM provider:

cp .env.example .env
# Edit .env with your API key

Supported providers via LiteLLM:

  • OpenAI: gpt-4o-mini, gpt-4o
  • Anthropic: claude-3-haiku-20240307, claude-3-5-sonnet-20241022
  • Ollama (local, free): ollama/llama3.2, ollama/mistral
  • Google: gemini/gemini-1.5-flash
  • And 100+ more providers

If no LLM is configured, classification metadata will be skipped.

๐Ÿ“– Usage

Full Build

uv run vtk-docs build

This generates all outputs (~35 seconds without LLM, longer with LLM due to rate limiting):

  • docs/vtk-python-docs.jsonl - JSONL database
  • docs/python-stubs-enhanced/ - Enhanced Python stubs
  • docs/python-api/ - Markdown documentation

CLI Commands

uv run vtk-docs --help          # Show all commands
uv run vtk-docs build           # Run complete build pipeline
uv run vtk-docs extract         # Extract VTK documentation to JSONL
uv run vtk-docs stubs           # Generate and enhance Python stubs
uv run vtk-docs markdown        # Generate markdown documentation
uv run vtk-docs clean           # Clean generated files
uv run vtk-docs stats           # Show database statistics
uv run vtk-docs search <query>  # Search the documentation

Search Examples

uv run vtk-docs search vtkActor           # Search by class name
uv run vtk-docs search Render -f synopsis # Search in synopsis field
uv run vtk-docs search Core -f module_name -n 20  # Search modules, show 20 results

Programmatic Usage

from vtk_python_docs.build import build_all

# Run full build
build_all()

# Or use individual components
from vtk_python_docs.extract import extract_all
from vtk_python_docs.stubs import generate_all as generate_stubs
from vtk_python_docs.markdown import generate_all as generate_markdown
from vtk_python_docs.config import get_config

config = get_config()
extract_all(config)
generate_stubs(config)
generate_markdown(config)

Querying the JSONL Database

import json
from pathlib import Path

# Stream through JSONL database
for line in open('docs/vtk-python-docs.jsonl'):
    record = json.loads(line)
    if 'Actor' in record['class_name']:
        print(f"{record['class_name']}: {record.get('synopsis', '')}")

๐Ÿ“ Output Structure

docs/
โ”œโ”€โ”€ vtk-python-docs.jsonl    # All VTK classes (JSONL format)
โ”œโ”€โ”€ llm-cache.jsonl          # Cached LLM classifications (avoids re-calling LLM)
โ”œโ”€โ”€ python-stubs-enhanced/   # Enhanced .pyi stub files
โ”‚   โ”œโ”€โ”€ vtkCommonCore.pyi
โ”‚   โ””โ”€โ”€ ... (150+ modules)
โ””โ”€โ”€ python-api/              # Markdown documentation
    โ”œโ”€โ”€ index.md
    โ””โ”€โ”€ vtkCommonCore/
        โ”œโ”€โ”€ index.md
        โ””โ”€โ”€ vtkObject.md

JSONL Record Fields

Each record in vtk-python-docs.jsonl contains:

Field Source Description
class_name VTK Class name (e.g., vtkActor)
module_name VTK Module name (e.g., vtkRenderingCore)
class_doc VTK Raw documentation from help()
role Introspection Pipeline role: input, filter, output, properties, renderer, etc.
input_datatype Introspection Input data type (e.g., vtkPolyData)
output_datatype Introspection Output data type
semantic_methods Introspection Non-boilerplate methods
synopsis LLM One-sentence summary
action_phrase LLM Noun-phrase (e.g., "mesh smoothing")
visibility_score LLM 0.0-1.0 likelihood users mention this class

๐Ÿ”ง Project Structure

vtk-python-docs/
โ”œโ”€โ”€ pyproject.toml
โ”œโ”€โ”€ setup.sh
โ”œโ”€โ”€ tests/               # pytest test suite
โ””โ”€โ”€ vtk_python_docs/
    โ”œโ”€โ”€ __init__.py
    โ”œโ”€โ”€ cli.py           # Typer CLI
    โ”œโ”€โ”€ config.py        # Centralized configuration
    โ”œโ”€โ”€ build.py         # Build pipeline orchestrator
    โ”œโ”€โ”€ extract/         # VTK documentation extraction
    โ”œโ”€โ”€ stubs/           # Stub generation & enhancement
    โ””โ”€โ”€ markdown/        # Markdown generation

๐Ÿงช Development

# Install with dev dependencies
uv sync --extra dev

# Run tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=vtk_python_docs

# Lint code
uv run ruff check .

# Type check
uv run pyright vtk_python_docs/

๐Ÿ”‘ Environment Variables

Variable Description Default
LLM_MODEL LiteLLM model identifier (none)
OPENAI_API_KEY OpenAI API key (none)
ANTHROPIC_API_KEY Anthropic API key (none)
GEMINI_API_KEY Google Gemini API key (none)
LLM_RATE_LIMIT Requests per minute 60
LLM_MAX_CONCURRENT Max concurrent requests 10

๐Ÿ“„ License

This project enhances the official VTK Python bindings. Please refer to VTK's licensing terms for the underlying VTK library.

Contributors

vicentebolea

Issues