Fork of the vLLM GGUF quantization plugin. Extends GGUF coverage with standard formats and selected ik_llama.cpp and ROCmFPX formats. The goal is to serve more models from one stack while keeping format validation status explicit.

Default branch homelabs-main carries extended quantization support. The upstream plugin documentation is preserved in UPSTREAM.md.

Format names and type IDs are not enough to establish compatibility. The authoritative implementation determines packed layout, block geometry, lookup tables, scale encoding, and expected decode behavior.

When a format is extended or patched locally, its source lineage and geometry must remain visible in code and documentation. This fork is not an independent authority for a packed format merely because it accepts its type ID.

The upstream plugin supports standard llama.cpp quantization types. This fork adds formats that are used in the community but are missing upstream. ROCmFPX types 100 and 101 have native CUDA/HIP dequant-only decode plus software fallback dequantization. ROCmFPX types 100, 102, 103, 104, and 107 have dedicated dense Triton GEMM kernels gated on gfx115*, and type 101 has limited, conditional native CUDA/HIP GEMV plus its own gated Triton GEMM. All six ROCmFPX types also have fused Triton MoE kernels behind the same gfx115* gate, including layers that mix quant types across the expert tensors. Outside that gate every ROCmFPX format falls back to dequantize-plus-dense. No native C++/HIP extension MoE is claimed, and only type 101 has any native GEMV. These paths therefore run on consumer GPUs without FP8/FP4 tensor cores:

The rows below describe expected work and traffic, not measured benchmark results. A fused quantized kernel normally combines decode and dot-product work. A dequant-only path decodes packed weights first and then uses a dense matmul; that distinction matters more than the format name.

For current added formats, native dequant-only is not a claim of fused quantized GEMV/GEMM/MoE. The implementation must be described by its actual capability table, not by whether its decoder runs on CUDA or HIP.

This matrix describes repository implementation status, not universal model compatibility. A model may still require compatible tensor names, shapes, runtime, and backend coverage. Each operation column is a separate capability; presence in one column does not imply presence in another.

Legend: ✅ available · ⚠️ conditional or partial · ❌ not available.

There is no Triton GEMV path in this plugin. ggml_mul_mat_a8_triton is a GEMM kernel, so small-batch work either uses a native GEMV kernel or falls back to dequantize-plus-dense. The Triton GEMV column is therefore ❌ everywhere and is kept only to make that explicit.

The dispatch tables in ops.py, triton/gemm/interface.py, and triton/fused_moe/interface.py remain the authority; this table summarises them.

The standard baseline is retained from the upstream plugin. Added formats are implemented as Python/Triton/native CUDA/HIP ports only where the corresponding dispatch table says so; unsupported combinations fail or use an explicit dense fallback rather than silently selecting a wrong kernel.

vLLM's native MTP reuses the target model's quantization. The MTP head is built from ordinary LinearBase and RoutedExperts layers, so it loads through the same GGUFLinearMethod and GGUFMoEMethod as the main model. There is no separate MTP quantization path, and no MTP-specific kernel work is required for a format that already has linear and MoE support.

What MTP does require is tensor-name resolution. GGUF encodes an MTP head as blk.{n}.nextn.* tensors, and gguf.TensorNameMap only emits those for architectures listed in its MODEL_TENSORS table. The published gguf package lags llama.cpp master here, so vllm_gguf_plugin/mtp_types.py registers the missing entries at import time, in the same way ik_types.py registers ik_llama quantization types.

The last group has no MODEL_ARCH member in the installed release, so no runtime patch can reach it. mtp_types.unsupported_mtp_architectures() reports that set, so the gap is visible rather than appearing as an unsupported model.

This covers weight-name resolution only. Running an MTP checkpoint end to end also depends on vLLM's speculative-decoding stack and on the target model's quantization support, neither of which this section claims.

• Compare packed weight bytes only when the format ABI matches: same type ID, block size, byte layout, alignment, scale representation, and lookup tables.
• Equal packed bytes do not imply equal runtime behavior. Fused kernels, dequant-only kernels, and reference fallbacks perform different decode and memory work.
• End-to-end performance depends on token count, matrix shape, shard layout, dtype, backend, kernel availability, and vLLM scheduling. GEMV can be the relevant path at low token counts; GEMM or MoE paths can dominate at higher counts.
• Do not infer a speedup from lower bits per weight alone. Measure packed-weight traffic, decode work, dense temporary allocation, matmul efficiency, and peak memory together.
• No measured benchmark numbers should be invented. Current evidence includes direct native _C_gguf type-101 GEMV validation on local AMD Radeon 8060S (gfx1151) with ROCm 7.14/Torch 2.12, FP32/FP16/BF16 launches, and an independent ROCmFP4 half-scale UE4M3 bias-8 oracle, plus dense Triton GEMM correctness validation for ROCmFPX types 100, 102, 103, 104, and 107 against independent per-format ABI oracles, including confirmation that public ops.ggml_mul_mat_a8 dispatch selects the dedicated kernel on that device. All of this is direct-op and dispatch validation, not full vLLM end-to-end model validation, and no throughput or memory-traffic numbers are claimed.

A format is complete only when all gates below pass. Enum registration alone is not support.

1. ABI and source parity: identify authoritative source, type ID, block size, byte layout, alignment, signedness, scales, lookup tables, and shape constraints; compare against the source implementation.
2. Storage and geometry: validate row storage, shard behavior, tails, non-contiguous inputs where supported, and malformed metadata. Reject incompatible geometry instead of decoding with a guessed layout.
3. Decode correctness: compare random and boundary blocks with a trusted reference, including zero/small dimensions, tails, dtype variants, and non-contiguous cases where relevant.
4. Dispatch truthfulness: register only operations implemented by the capability tables. Keep native GEMV, native GEMM, native MoE, Triton, and CPU/reference fallback claims separate.
5. Native versus fallback behavior: verify backend selection, explicit failure or fallback, and no silent wrong-kernel selection. A native quantized linear path must not allocate a full dense temporary. Current native dequant-only paths explicitly emit decoded dense values and therefore remain dequant-only until a fused path meets this gate. Fallback paths must bound their temporary to the active layer and document peak use.
6. Compute and bandwidth validation: benchmark representative token counts, matrix shapes, dtypes, and shard layouts against the authoritative or equivalent implementation. Report decode work, weight reads, temporary memory, throughput, and peak memory; do not substitute estimates for measurements.
7. Backend matrix: validate CUDA, ROCm, and CPU/reference fallback behavior separately, recording hardware, toolkit, Torch, vLLM, and plugin versions. A skipped backend is not evidence of support.
8. End-to-end model validation: load and run representative models through vLLM, including tensor mapping, sharding, standard linear layers, and MoE or diffusion paths when claimed.
9. Documentation: update source lineage, format status, capability limits, fallback memory behavior, validation evidence, and usage guidance.

Native KT GEMV is future work. Native KT GEMM and native KT MoE are separate work items and must not be implied by completing KT GEMV.

The plugin is baked into homelab vLLM runtime images (vllm-strix-runtime / vllm-spark-runtime), so GGUF serving works out of the box there. To install standalone:

git clone https://github.com/vllm-project/vllm-gguf-plugin
cd vllm-gguf-plugin
git checkout homelabs-main
uv pip install -e . --torch-backend=auto

This repository includes a local development image for AMD Strix Halo (gfx1151). It extends the local, non-redistributable vllm-strix-devtools:local image and requires matching ROCm and Torch packages in that base image.

docker build -f docker/Dockerfile.gfx1151 -t vllm-gguf-plugin:gfx1151 .
docker run --rm --device=/dev/kfd --device=/dev/dri --group-add video \
  vllm-gguf-plugin:gfx1151 \
  /opt/venv/bin/python -m pytest \
    tests/test_rocmfp4_fast_triton_gemm.py \
    tests/test_type101_public_dense_gemm_dispatch.py \
    tests/test_rocmfp4_fast_fused_gemv.py \
    tests/test_dispatch_capabilities

Override the local base image with --build-arg BASE_IMAGE=<matching-image>. The image builds the plugin in editable mode and uses the base image's compatible ROCm, Torch, and vLLM.

Serve a GGUF model; the plugin registers quantization types when vLLM imports the entry point:

vllm serve <repo>/<model>-GGUF:Q4_0_ROCMFP4 --tokenizer <repo>/<model>

Kernels are checked against oracles written directly from the ABI, which share no code with the decoders under test, so a shared misreading cannot mask itself.

Three things beyond value comparison:

Derived error bounds. Tolerances come from the float format's mantissa width, the reduction length, and the measured cancellation between operands, rather than being chosen after watching a test fail. A hand-picked bound hides any decode error smaller than the fudge factor. See tests/numerics.py.

Out-of-bounds read detection. These decoders compute byte and plane offsets into packed buffers, so an addressing error reads neighbouring memory and returns plausible numbers -- invisible to a reference comparison, which is only consulted for the elements requested. Inputs are padded and the padding is refilled between two runs; an in-bounds kernel must produce identical output both times.

End-to-end perplexity gate. Kernel tests cannot see a bug outside the kernel: a converter transform or layout assumption sits on both sides of the comparison. Two such bugs shipped here while every kernel test passed, both magnitude-preserving, so no statistical check flagged them. Only perplexity separated the states. Following llama.cpp's ci/run.sh:

VLLM_GGUF_PPL_MODEL=/path/to/model.gguf \
VLLM_GGUF_PPL_TOKENIZER=/path/to/tokenizer \
  pytest tests/test_perplexity_gate.py

The same kernel can pass on one backend and fail on another -- tl.dot rejects mismatched operand dtypes, and float32 lowers to TF32 on NVIDIA but full precision on AMD. Run the comparison with:

scripts/backend_matrix.py            # every available backend
scripts/backend_matrix.py --list     # show configuration

Currently 295 kernel tests, identical on gfx1151 and sm_121. Backends are probed first and reported as unavailable rather than failing, so a partial run still produces a usable matrix.

homelabs-main is the consolidated branch. Per-format work lands on feature branches (for example rocmfpx-q4_0 and ik-iqk) and merges after review. Each new quant type adds a block struct, dequant kernel, dispatch case, and type-registration entry; see the ROCmFPX commits for the pattern. The upstream development workflow is documented in UPSTREAM.md.

Apache 2.0, as upstream.