Post-hoc hallucination verifier for RAG pipelines. Reimplements HALT-RAG (Goswami & Kurra, arXiv:2509.07475) — a calibrated ensemble of two frozen NLI models plus lexical features, with out-of-fold training, precision-constrained thresholding, and abstention — with one deliberate architectural change: NLI scoring runs through the Hugging Face Inference API (no local GPU or weights required). A local transformers backend exists behind a config flag.

What it does: given a source (retrieved context) and a generated text, returns a calibrated hallucination probability and a decision: supported / hallucinated / abstain.

What it doesn't do: it is not a fact-checker against the open world — it only checks consistency between the generation and the provided source. It was developed on clean HaluEval contexts, not noisy real-world retrieval.

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,plots]"
export HF_TOKEN=hf_...        # https://hf.co/settings/tokens (read scope)

# 0. CRITICAL GATE — verify the API pair format before spending any budget:
pip install -e ".[local]"      # needed once, for the local diff side
python scripts/verify_nli_backend.py

# 1. Pilot run (~200 raw records = 400 examples):
python scripts/run_pipeline.py --task qa --limit 200 --yes

# 2. Ad-hoc check:
halt-rag verify --task qa \
  --source "Paris is the capital of France. It hosts the Louvre museum." \
  --generated "The capital of France is Berlin."

Offline tests (no network, no token): pytest.

data.limit: 500 in every config. A full HaluEval run is ~20k examples/task and easily 150k–400k+ hosted API calls across tasks/models/windows. The CLI prints a call estimate and asks for confirmation on large runs (skip with --yes). All raw NLI scores are cached in SQLite (.cache/nli_scores.sqlite), keyed by (model_id, premise_hash, hypothesis_hash) — interrupted runs resume free, identical windows are never re-scored. Use --limit 0 explicitly for the full dataset.

1. Window the source into non-overlapping 320-token chunks; pair each with the (budget-truncated) generation as hypothesis.
2. Score every pair with roberta-large-mnli and an NLI-finetuned DeBERTa-v3-large via the HF Inference API (cached, retried, rate-bounded).
3. Pool max+mean of each NLI probability per model → 12 features; add 5 lexical features (lengths, ratio, ROUGE-L F1, Jaccard) → 17 total, named schema (see features/pipeline.py), ablation flags supported.
4. Train the task classifier (LogReg for summarization/dialogue, LinearSVC for QA; C=1.0, class_weight=balanced) with 5-fold OOF; calibrate (isotonic for LogReg tasks, Platt for QA) on OOF scores; threshold at argmax F1 s.t. precision ≥ 0.70; abstain on the least-confident predictions down to the target coverage (default 90%).

One YAML per task in configs/ (validated by pydantic — src/halt_rag/config.py is the authoritative schema):

Python (any framework, or none):

from halt_rag import HaltRagVerifier

verifier = HaltRagVerifier.from_config("configs/qa.yaml")
result = verifier.verify(source=context_text, generated=answer_text)
result.calibrated_prob   # float in [0,1]
result.decision          # "supported" | "hallucinated" | "abstain"
result.feature_breakdown # per-feature values, for debugging

Generic dict-pipeline step:

from halt_rag.integrations.generic import verify_step
pipeline = [retrieve, generate, verify_step(verifier, policy="annotate")]

LangChain:

from halt_rag.integrations.langchain_adapter import HaltRagGuard
chain = rag_chain | HaltRagGuard(verifier, policy="raise").as_runnable()

LlamaIndex:

from halt_rag.integrations.llamaindex_adapter import verify_response
response = verify_response(verifier, query_engine.query("..."))

Any language, via the sidecar service (pip install -e ".[serve]"):

halt-rag serve &
curl -s localhost:8000/verify -H 'Content-Type: application/json' \
  -d '{"task":"qa","source":"Paris is the capital of France.","generated":"Berlin is the capital of France."}'

halt-rag download-data   --task qa|dialogue|summarization|all
halt-rag extract-features --task qa [--limit N] [--yes]
halt-rag train            --task qa
halt-rag evaluate         --task qa       # writes evaluation_results/<task>/
halt-rag verify           --task qa --source "..." --generated "..."
halt-rag serve

Outputs per task: oof_calibrated_pred.jsonl, oof_meta.json (precision/ recall/F1/accuracy at t*, AUROC, AUPR, ECE, abstention block), four plots (PR, ROC, reliability, risk-coverage), and a top-level evaluation_results/summary.md comparing against the paper's Table 1.

1. DeBERTa checkpoint substituted. The paper names microsoft/deberta-v3-large, which has no NLI head (base model only). We default to MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli (MNLI+FEVER-NLI+ANLI+LingNLI+WANLI); swappable via inference.deberta_model_id (e.g. cross-encoder/nli-deberta-v3-large).
2. Hosted inference wire format is verified, not assumed. The HF Inference Providers text-classification spec doesn't clearly document a pair payload for third-party NLI checkpoints. scripts/verify_nli_backend.py diffs the API against a local transformers pipeline on hand-picked pairs and must PASS before any large extraction. A zero_shot fallback path exists (inference.pair_format), with the caveat that it only exposes entailment mass (neutral/contradiction are approximated by an even split — quantified by the same script).
3. Windowing tokenizer fallback. The paper windows with the NLI model's own tokenizer; the lightweight base install uses whitespace tokens (with the real tokenizer used automatically when [local] extras are installed). 320 whitespace words ≈ 400–450 subwords, safely within the 512 budget with a 160-word hypothesis cap.
4. Abstention ranking reconstructed. Rank by |calibrated_prob − t*|, keep the most confident up to the coverage target. This is a reasonable reconstruction of the paper's coverage mechanism, not a verbatim spec.
5. Label order is never assumed. Backends map the model's own label names (id2label locally; response labels via the API) into a fixed internal order — roberta-large-mnli and DeBERTa NLI checkpoints disagree on index order, which silently corrupts features if read positionally.
6. Numbers will differ from the paper. Pilot-scale subsampling, the substituted checkpoint, hosted inference, and the tokenizer fallback all shift results. evaluation_results/summary.md reports our numbers next to the paper's reference numbers with this note.
7. Untrained fallback in verify(). Before halt-rag train has run, ad-hoc verification uses a transparent NLI-only heuristic (1 − mean entailment) and the default 0.5 threshold, so the CLI is usable immediately — train for calibrated behavior.

Fixed windows can split evidence across chunk boundaries; classifiers are task-tuned rather than one universal model; evaluation uses clean HaluEval contexts, not noisy retrieval; hypothesis truncation may drop content in very long generations.

• HALT-RAG: Goswami & Kurra, arXiv:2509.07475.
• HaluEval: Li et al., 2023 (data fetched from the RUCAIBox/HaluEval repo — see that repo for dataset licensing).
• This code: MIT (see LICENSE).