jtbirdsell/dialectiq

RISE-pattern SQL transpiler — translates SQL and stored procedures between 31 database dialects using sqlglot + LLM-assisted reduction

★ 0Forks 0PythonGitHub ↗Compare

README

Dialectiq

A SQL transpiler that translates SQL and stored procedures between any pair of database dialects. Dialectiq combines sqlglot for deterministic translations with RISE-style AST reduction and LLM-assisted translation for everything sqlglot cannot handle.

This is the first production implementation of the RISE pattern (ICSE 2026), which achieved 100% on stored procedure benchmarks and 97.98% on TPC-DS but was never productized.

How It Works

Input SQL (any dialect)
  → sqlglot transpile (~75% of patterns handled)
  → If sqlglot fails: RISE reduction isolates the problem
    → Strips the query down to the minimal failing construct
    → LLM translates the small fragment
    → Fix applied back to the full original query
  → Output: translated SQL + confidence score + warnings

The RISE reduction is what makes this different from other transpilers. Instead of asking an LLM to translate a 100-line query (unreliable), Dialectiq reduces it to a 5-line fragment that isolates the dialect-specific construct, translates that reliably, and applies the fix back.

Supported Dialects

Dialect As Source As Target
Oracle / PL/SQL Yes Yes
SQL Server (T-SQL) Yes Yes
PostgreSQL / PL/pgSQL Yes Yes
MySQL Yes Yes
Snowflake Yes Yes
BigQuery Yes Yes
Databricks SQL Yes Yes
Apache Spark SQL Yes Yes
Apache Spark 2 SQL Yes Yes
DuckDB Yes Yes
Amazon Athena Yes Yes
Amazon Redshift Yes Yes
Apache Doris Yes Yes
Apache Drill Yes Yes
Apache Druid Yes Yes
Apache Hive Yes Yes
Apache Solr Yes Yes
ClickHouse Yes Yes
Dremio Yes Yes
Dune SQL Yes Yes
Exasol Yes Yes
Materialize Yes Yes
Microsoft Fabric Yes Yes
Presto Yes Yes
RisingWave Yes Yes
SingleStore Yes Yes
SQLite Yes Yes
StarRocks Yes Yes
Tableau Yes Yes
Teradata Yes Yes
Trino Yes Yes

Stored Procedure Support

Dialectiq handles procedural SQL — not just queries. It parses procedure structure, tracks variable lifecycles, detects cross-file call dependencies, and translates procedural constructs (cursors, exception handlers, control flow) using the same RISE reduction approach.

Quick Start

Prerequisites

  • Python 3.11+
  • Node.js 18+ (for the web frontend)
  • An Anthropic API key (for LLM-assisted translations)

Setup

# Clone and install
pip install -r requirements.txt

# Set your API key
cp .env.example .env
# Edit .env and add: ANTHROPIC_API_KEY=sk-ant-...

# Install frontend dependencies
cd web && npm install && cd ..

Run

# Start both backend and frontend
./start.sh

# Or start them separately:
# Backend (port 8000)
PYTHONPATH=src python -m uvicorn dialectiq.server:app --port 8000

# Frontend (port 3000)
cd web && npm start

Open http://localhost:3000 in your browser.

Using the Web Portal

Codebase Translation (Primary Workflow)

  1. Open the web portal
  2. Upload a folder of SQL files (drag-and-drop or directory picker)
  3. Select source dialect (e.g., Oracle) and target dialect (e.g., Snowflake)
  4. Click Translate and watch live progress
  5. Browse results: side-by-side original vs. translated with syntax highlighting
  6. Download translated files as a zip + migration report

Quick Translate

Paste a single SQL statement, select dialects, and see the translation instantly. Useful for ad-hoc testing.

Built-In Examples

Three pre-built example codebases are included for demos:

Example Source Suggested Target Files Highlights
Oracle E-Commerce Oracle Snowflake 10 Stored procs with cursors, exception handlers, triggers, NVL/DECODE
T-SQL Analytics T-SQL BigQuery 6 #temp tables, TRY/CATCH, sp_executesql, OPENJSON
Databricks ETL Databricks Snowflake 6 Delta Lake DDL, MERGE with wildcards, explode(), date_format

Click any example to load it and translate with one click.

Architecture

src/dialectiq/
  pipeline.py          -- Translation pipeline: sqlglot → RISE+LLM → report
  reduction.py         -- RISE-style AST reduction engine
  server.py            -- FastAPI backend (REST + SSE)
  models.py            -- Pydantic data models
  config.py            -- Constants and configuration
  exceptions.py        -- Error types
  examples.py          -- Built-in example codebase loader
  dialects/
    registry.py        -- 31 dialect definitions with sqlglot mappings
  llm/
    client.py          -- Claude API client with retry and rate limiting
    prompts.py         -- Prompt templates for RISE translation
  procedural/
    parser.py          -- Stored procedure structure parser
    lifecycle.py       -- Variable and cursor lifecycle analysis
    dependency.py      -- Cross-file call graph and topological ordering
  batch/
    scanner.py         -- SQL file discovery and validation
    session.py         -- Upload session management
    translator.py      -- Batch translation with progress events
    report.py          -- Migration report generation (JSON, HTML, ZIP)
  reporting/
    models.py          -- Result types
    formatter.py       -- Confidence scoring and formatting

web/                   -- React + TypeScript frontend
  src/
    components/
      CodebaseUploader   -- Drag-and-drop file upload
      TranslationProgress -- Live progress with per-file status
      ResultsBrowser     -- Side-by-side diff viewer
      MigrationReport    -- Summary stats and download
      QuickTranslate     -- Single query translation
      ExampleLoader      -- Built-in example selector
      DependencyGraph    -- Stored procedure call graph
      SqlEditor          -- Syntax-highlighted SQL editor

API Endpoints

Method Endpoint Description
POST /api/upload Upload SQL files, returns scan inventory
POST /api/translate Start batch translation (SSE progress)
GET /api/progress/{id} SSE stream of translation events
GET /api/results/{id} Translated file tree with metadata
GET /api/download/{id} ZIP of translated files + HTML report
POST /api/quick Single statement translation
GET /api/examples List built-in example codebases
POST /api/examples/load Load a built-in example
GET /api/dialects Supported source and target dialects
GET /api/health Health check

Tests

# Run full test suite (605+ tests)
PYTHONPATH=src python -m pytest tests/ -v

# Run specific test modules
PYTHONPATH=src python -m pytest tests/test_reduction.py -v     # RISE reduction
PYTHONPATH=src python -m pytest tests/test_pipeline.py -v      # Translation pipeline
PYTHONPATH=src python -m pytest tests/test_procedural/ -v      # Stored procedures
PYTHONPATH=src python -m pytest tests/test_server.py -v        # API endpoints
PYTHONPATH=src python -m pytest tests/test_batch/ -v           # Batch processing

How RISE Reduction Works

When sqlglot cannot translate a SQL construct, Dialectiq uses the RISE algorithm:

  1. Identify: Capture the sqlglot error on the failing statement
  2. Reduce: Iteratively remove AST subtrees (CTEs, joins, columns, clauses) while preserving two invariants:
    • The reduced statement still parses in the source dialect
    • The reduced statement still produces the same error in the target dialect
  3. Translate: Send the minimal fragment to Claude (typically 1-5 lines instead of 100+)
  4. Apply: Patch the LLM's fix back into the original full statement
  5. Re-validate: If the fix reveals another dialect issue, re-enter the loop

This achieves dramatically higher accuracy than raw LLM translation because the LLM only handles small, focused fragments where it excels.

License

Proprietary — phData internal use.

Contributors

jtbirdsell

Issues