Self-hosted, real-time speech-to-text for the Omi wearable — powered by lightning-whisper-mlx (Apple Silicon GPU) with named speaker identification, direct Omi memory creation, and a live transcript dashboard.

No Omi subscription required. Runs on Mac Apple Silicon, Raspberry Pi 5, or any Linux/CUDA machine.

Omi's cloud STT costs money and sends your audio to third-party servers. This project replaces it entirely — your audio never leaves your machine, speaker names are resolved locally, and conversations are pushed directly to your Omi memory via the official API, bypassing the 2-minute conversation processing delay.

sequenceDiagram
    participant P as Omi Pendant
    participant CF as Cloudflare Tunnel
    participant S as omi-whisperx
    participant O as Omi Cloud API

    P->>CF: PCM16 audio frames (WebSocket)
    CF->>S: wss://your-domain.com/live
    Note over S: VAD Gate — flush on 0.5s silence
    S->>S: lightning-whisper-mlx transcribe (~2s, Apple GPU)
    S->>S: Fast speaker ID — whole-utterance embed (~0.03s)
    S-->>P: {"segments": [...]} → shown in app
    Note over S: 30s silence timer
    S->>O: POST /conversations/from-segments
    O-->>S: {"id": "conv-uuid"}
    Note over O: AI processing: title, action items, memory

Live path latency (~2–3s end-to-end). The WebSocket path is tuned for low lag: per-utterance speaker ID uses a single whole-utterance embedding (~0.03s) instead of full Sortformer diarization + per-speaker embedding (~1–3s) — pendant utterances are near-always single-speaker. Word-level alignment is skipped (SKIP_LIVE_ALIGN), and pinning the language (WHISPER_DEFAULT_LANG=hi) skips Whisper's per-utterance autodetect for a further ~30%. Set LIVE_FAST_SPEAKER_ID=false to restore in-utterance Sortformer diarization (multi-speaker per utterance, higher lag).

flowchart LR
    A[Audio chunk] --> B[lightning-whisper-mlx<br/>transcribe]
    B --> X{LIVE_FAST_SPEAKER_ID?}
    X -->|true · live| C[Resemblyzer<br/>embed whole utterance<br/>~0.03s]
    X -->|false| Y[Sortformer diarize<br/>+ per-speaker embed<br/>~1–3s]
    Y --> C
    C --> D{Cosine similarity<br/>vs enrolled profiles}
    D -->|above threshold| E[Named speaker<br/>Rahul Sharma]
    D -->|below threshold| F{Blocked voice?}
    F -->|Yes| G[Discard<br/>silently]
    F -->|No| H[UNKNOWN<br/>Save clip for review]
    H --> I[ui/speakers<br/>Assign name]
    I --> J[Enroll<br/>average embedding]

sequenceDiagram
    participant App as Omi iOS App
    participant S as omi-whisperx
    participant OB as Omi Backend

    Note over App,OB: Standard path (built-in STT)
    App->>OB: Audio via Omi backend WS
    OB->>OB: Deepgram STT
    OB-->>App: Segments (real-time)
    Note over OB: Wait conversation_timeout (120s)
    OB->>OB: LLM process → memory created

    Note over App,OB: omi-whisperx path
    App->>S: Audio via custom STT WS
    S-->>App: Segments (real-time, ~3s lag)
    Note over S: 30s silence debounce
    S->>OB: POST /v1/dev/user/conversations/from-segments
    OB->>OB: LLM process → memory created
    Note over S,OB: Total delay: ~35s vs ~120s

• Python 3.12
• ffmpeg + libsndfile1
• HuggingFace token — accept pyannote/speaker-diarization-3.1 license
• Omi API key — from Omi Developer Settings

git clone https://github.com/Rahulsharma0810/omi-whisperx
cd omi-whisperx
cp .env.example .env
# Edit .env — set HF_TOKEN and OMI_API_KEY at minimum

./start.sh

Creates ~/.venvs/whisperx, installs deps, launches on :8080. First start downloads models (~2–4 GB).

Omi pendant needs HTTPS. Use Cloudflare Tunnel:

cloudflared tunnel run --url http://localhost:8080

In Omi → Settings → Developer → Cloud Provider > Custom:

Enable VAD Gate in Omi app settings for best performance (strips silence before sending).

View interactive diagram →

Segments stream in real time with speaker name, colour coding, and lag metrics:

audio@+2.3s → sent@+5.1s | lag=2.8s (queue=0.0s proc=2.8s)

Three ways to teach the system who's speaking:

Every unknown voice is saved as a short clip (max 5 per unique voice). Go to /ui/speakers, play each clip, type a name, click Confirm. Done.

curl -X POST http://localhost:8080/speakers \
  -F "name=Alice" \
  -F "file=@alice_voice.m4a"

Open /ui/speakers → click the mic icon next to any name → speak for 5s → auto-enrolled.

Unknown voices from TV/movies appear in clips. Click Block on any clip → that voice is permanently ignored and never recorded again. Uses embedding similarity so it works even if the audio quality changes.

All settings via environment variables. See .env.example for the full list.

Real-time STT for Omi pendant.

Query params: language, uid, sample_rate (default 16000), codec (default opus)

Omi sends: Binary PCM16 frames (640 bytes = 20ms at 16kHz) + JSON control messages ({"type": "CloseStream"})

Server sends:

{
  "segments": [
    {
      "text": "Hello world",
      "speaker": "Rahul Sharma",
      "start": 0.0,
      "end": 2.4
    }
  ]
}

HTTP transcription endpoint (Omi Transcript Provider mode).

curl -X POST http://localhost:8080/inference \
  -F "[email protected]" \
  -F "language=en" \
  -F "response_format=verbose_json"

Response:

{
  "segments": [{"start": 0.0, "end": 2.4, "text": "Hello", "speaker": "Rahul"}],
  "text": "Hello",
  "language": "en",
  "duration": 5.2
}

sudo apt-get install -y python3.12 python3.12-venv ffmpeg libsndfile1
python3.12 -m venv ~/.venvs/whisperx
source ~/.venvs/whisperx/bin/activate
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

# Recommended .env for RPi5
WHISPER_MODEL=base
WHISPER_BATCH_SIZE=4
FAST_SPEAKER=true
TRUST_CLIENT_VAD=true
SKIP_LIVE_ALIGN=true

Expect ~2–3s per utterance with base model on RPi5.

CTranslate2 (Whisper backend) and resemblyzer (speaker encoder) do not support MPS — CPU is used intentionally.

# Quick (no HF_TOKEN needed)
python benchmark.py --no-diarization --no-embedding --language en --trials 3

# Full pipeline on real audio
python benchmark.py audio.wav --trials 3 --output json --output-file results_mac.json

# Compare two machines
python benchmark.py --compare results_mac.json results_rpi5.json

Or use the interactive UI at /benchmark.

# Install
cp com.rvs.whisperX.plist ~/Library/LaunchAgents/
launchctl load ~/Library/LaunchAgents/com.rvs.whisperX.plist

# Restart after config changes
launchctl unload ~/Library/LaunchAgents/com.rvs.whisperX.plist
launchctl load ~/Library/LaunchAgents/com.rvs.whisperX.plist

# Logs
tail -f ~/omi-whisperx/server.log

PRs and issues welcome.

Do not:

• Add MPS to WhisperX calls — CTranslate2 doesn't support it
• Add MPS to VoiceEncoder — resemblyzer doesn't support it
• Add pip deps without checking aarch64 wheels exist (must run on RPi5)
• Make classify_content() synchronous — Ollama call is async HTTP
• Call NLI pipeline directly from async code without asyncio.to_thread

MIT