Skip to main content

Qwen3.8-27B and native MTP speculative decoding

Run from the repository root on a checkout containing MTP support (after the 0.6.0 release). This recipe uses the model's native multi-token prediction head to draft tokens, then verifies a chunk with the main model. No separate draft model is needed. Qwen3.8 is loaded through the vision-language architecture: its MTP heads support text inference, but auxiliary MTP training loss is not implemented for that architecture. The checkpoint-compatibility training warning does not mean its inference heads are missing.

Hardware and checkpoint​

An RTX 5090 has 32 GB VRAM. Full BF16 27B weights exceed that capacity; use a compatible INT4 checkpoint with its MTP weights retained, or multiple GPUs for an appropriately configured BF16 deployment. The default examples are single-GPU; see INT4 tensor parallelism for two-GPU text inference.

Install Eole in a CUDA-enabled environment with the CUDA toolkit and build tools from the installation guide. For this configuration install FlashAttention and the hybrid-attention kernels used by Qwen:

MAX_JOBS=2 python -m pip install -e . --no-build-isolation
MAX_JOBS=2 python -m pip install flash-attn --no-build-isolation
python -m pip install "fla-core==0.6.0"
MAX_JOBS=2 python -m pip install causal-conv1d --no-build-isolation

INT4 AutoRound/GPTQ checkpoints require a supported packing layout and Eole's Marlin CUDA extension. Build/install Eole while CUDA and the CUDA toolkit are available; inspect backend diagnostics before measuring. Installing a package does not prove its kernels are selected.

The recommended INT4 checkpoint for this recipe is RedHatAI/Qwen3.8-27B-INT4. Choose either preparation option below; both use the same local path and recipe YAMLs. Pre-converted packages require Eole 0.6.4 or newer and the same runtime kernels and GPU memory as a checkpoint you convert yourself.

export EOLE_MODEL_DIR=/path/to/models
export QWEN38_MODEL="$EOLE_MODEL_DIR/Qwen3.8-27B-INT4"

Download the pre-converted Eole checkpoint​

The public Eole INT4 package contains converted weights, configuration, tokenizer, vision weights and MTP heads. Download it directly; no conversion or HF token is required:

hf download eole-nlp/Qwen3.8-27B-INT4-eole --local-dir "$QWEN38_MODEL"

Use a new directory for the download. For reproducible deployments, add --revision COMMIT_SHA; validated commit pins are recorded in the RunPod model catalog. These are Eole-format checkpoints, loaded with Eole rather than Transformers. Downloading them skips conversion, not runtime kernel installation or first-use compilation.

Convert the original HF checkpoint yourself​

Alternatively, let the converter download the original HF checkpoint, or pass an existing local HF directory as --model_dir:

python eole/bin/main.py convert HF \
  --model_dir RedHatAI/Qwen3.8-27B-INT4 \
  --output "$QWEN38_MODEL" \
  --dtype bf16 \
  --check-tensors

Use a new output directory for conversion. Add --token "$HF_TOKEN" if Hugging Face authentication is needed. The converter accepts a repository ID or a local HF model directory; it converts existing INT4 weights rather than quantizing BF16 weights. Source weight downloads stay in the HF cache.

If you already have the converted RedHat checkpoint, set the two path variables above to its location. Prediction, serving, MTP, and validation YAMLs all use QWEN38_MODEL; they do not need separate model-path edits.

Compressed-tensors INT4 layout​

RedHat uses symmetric grouped W4A16 pack-quantized weights.

The converter transposes packed INT4 words and scales into Eole's GPTQ layout; it does not dequantize and re-quantize the weights. Inference uses the existing AutoRound/GPTQ backend selection, including Marlin when available. Exact module paths from the checkpoint preserve unquantized projections, vision weights, and MTP heads. GDN kernels and normalization are unchanged.

Supported exports have one Linear weight-only configuration group, symmetric 4-bit integer weights, and group size 32, 64, or 128. Static activation ordering without saved group indices is accepted. Asymmetric weights, activation quantization, reordered groups, transformed/sparse exports, and packed weights requiring sliced or unmapped module transformations are rejected.

FP8 KV-cache scales in the source are not used: conversion logs a warning and Eole retains its floating-point KV cache. --check-tensors may list those cache scale tensors as unused. This is weight-format support, not FP8-cache support. CPU tests verify packed-value equivalence, sharded conversion, and mixed-precision module selection. The converted RedHat checkpoint has been exercised with compiled, MTP-disabled GPU inference; the targeted comparison below records the results. RedHat MTP throughput and greedy parity have not been measured.

For a BF16 deployment with sufficient memory, either convert Qwen/Qwen3.8-27B into a separate output directory, or download the public Eole BF16 package:

export QWEN38_MODEL="$EOLE_MODEL_DIR/Qwen3.8-27B-BF16"
hf download eole-nlp/Qwen3.8-27B-BF16-eole --local-dir "$QWEN38_MODEL"

The full BF16 weights do not fit a single 32 GB GPU. Both preparation options retain the original precision; downloading a pre-converted package does not quantize it.

Quantized checkpoints are not all equivalent: some omit MTP tensors. Conversion cannot recreate missing trained heads. Check the converted artifact before loading:

python recipes/qwen38/check_checkpoint.py "$QWEN38_MODEL"

The checker requires decoder num_mtp_heads > 0, saved mtp_heads.* tensors, and a chat template. It prints tensor names and quantization metadata without loading weights onto the GPU. A successful check establishes artifact contents, not GPU kernel compatibility or model quality.

Serve with MTP​

eole serve -c recipes/qwen38/serve.yaml --host 127.0.0.1

top_k: 1 deliberately selects greedy decoding, including for API clients that send a nonzero temperature. This recipe does not use sampling. It sets one sequence/hypothesis and disables prefix caching/chunked prefill to provide a simple baseline. Start with the conservative 32K context and lower it if VRAM is tight; the model's advertised maximum context is not a VRAM guarantee.

curl --fail-with-body http://127.0.0.1:5000/v1/messages \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27B","max_tokens":256,"messages":[{"role":"user","content":"Write a short Python function that reverses a string."}]}'

Look for MTP draft acceptance in server logs. A fallback warning means the request used ordinary decoding; merely setting the flag does not demonstrate speculation. Image inputs, batches larger than one, beam search, forced prefixes, and unsupported decoding constraints currently fall back.

Pack inference projections​

For compatible Marlin INT4/INT8 checkpoints, these inference options reduce projection launches without requantizing the weights:

fuse_kvq: true
fuse_gate: true

fuse_kvq packs self-attention K/V/Q and GDN input QKV/Z projections. fuse_gate packs MLP gate/up projections. Both options default to false. Fusion runs during inference setup, before Marlin repacking; incompatible pairs keep their original projections and log a warning. Other quantization backends retain their existing behavior. MoE expert stacking is unchanged.

For dense BF16 GDN A/B projections of shape 48×5120, supported NVIDIA Ampere+ GPUs also use a paired kernel automatically for single-token decoding and four-token MTP verification. This requires single-GPU, batch/beam 1 inference; other shapes, dtypes, devices and batch/beam configurations keep the existing path. The packed A/B buffer adds about 47 MB for this 48-GDN-layer model and is not stored in checkpoints. This dispatch is independent of the fusion flags.

Packing and parallel reductions can change floating-point rounding, generated continuations and MTP acceptance. FP32 normalization and the original BF16 vocabulary head are unchanged. See the projection benchmarks for measured throughput, limitations and reproduction commands.

Sample with MTP​

MTP also supports target sampling while keeping its draft proposals greedy. In an inference configuration (or a served model's config), use:

beam_size: 1
n_best: 1
self_speculative_decoding: true
self_speculative_num_tokens: 3
temperature: 0.6
top_k: 20
top_p: 0.95
presence_penalty: 1.0
frequency_penalty: 0.0

The target samples each verification position conditional on the preceding proposals. Eole accepts matching proposals and emits the first differing target sample as the correction; if every draft matches, it emits a sampled bonus. This preserves the ordinary target sampling distribution. It does not promise identical text with MTP on/off for the same seed: speculative rounds draw and may discard additional random samples. Acceptance and speed depend on sampling settings and the generated text; greedy performance does not predict sampled performance.

Presence/frequency penalties count generated tokens, not prompt or BOS tokens. They apply to ordinary decoding too and require beam_size: 1; top_k: 1 selects penalized greedy decoding. Sampling retains Eole's existing filtering order (temperature, top-p, then top-k). Defaults and the unpenalized greedy fast path are unchanged.

A reproducible BF16-vocabulary-head benchmark is available:

EOLE_TORCH_COMPILE=1 EOLE_COMPILE_MODE=0 \
  python recipes/qwen38/benchmark_mtp_sampling.py \
  --model "$QWEN38_MODEL" --output /tmp/mtp-sampling.json
EOLE_TORCH_COMPILE=1 EOLE_COMPILE_MODE=0 \
  python recipes/qwen38/benchmark_mtp_sampling.py \
  --model "$QWEN38_MODEL" --output /tmp/ordinary-sampling.json --no-mtp

This uses three original Eole text fixtures covering Python code, an English mystery, and Chinese-to-English translation, with thinking disabled. The default is a 64K cache reservation, 4,096 output tokens and seeds 42/43. Compare actual output lengths and draft acceptance as well as speed. The decoder checkpoint may be INT4; by default the benchmark uses its original dense BF16 vocabulary head. Optional head quantization is described below. Output JSON records the fixture revision and SHA-256 so measurements can be tied to the exact prompts used.

Optional vocabulary-head quantization​

Inference can quantize the dense vocabulary projection at load time:

vocabulary_quantization: int8  # none (default), int8, or int4
vocabulary_quantization_group_size: 128  # 32, 64, or 128

This uses symmetric, group-wise round-to-nearest quantization and the existing Marlin CUDA kernel. It requires a bias-free dense FP16/BF16 generator with Marlin-compatible dimensions, the Eole CUDA extension, and single-GPU inference. The checkpoint and any tied input embeddings remain unchanged. Quantization is performed again when loading the model; it is not a calibrated checkpoint conversion or an on-disk quantization cache.

The same projection serves ordinary decoding, MTP target verification, and MTP draft token selection. Therefore this setting can change the final output, including with greedy decoding. INT4 is more aggressive than INT8; speed measurements do not establish equivalent output quality. Decoder quantization and normalization are unchanged.

Use the sampling benchmark above with --head int8 or --head int4; omit the option (or use --head none) for the original BF16 head. The benchmark retains the same prompts, sampling configuration, context reservation and seeds. For the current packed-projection comparison, also pass --fuse-gate; KVQ/GDN input fusion is already enabled by the harness.

For a real coding client continue with Claude Code.

Compare against ordinary greedy decoding​

Stop the server first so only one process owns the checkpoint. The two inference YAML files use the same model, prompt, precision, and decode settings; only MTP and the output filename differ. The prompt fixture is one preformatted Qwen chat per line, with Eole's ｟newline｠ marker for embedded newlines.

eole predict -c recipes/qwen38/predict.yaml
eole predict -c recipes/qwen38/predict-mtp.yaml

These save /tmp/qwen38-baseline.txt and /tmp/qwen38-mtp.txt. Compare the text and inspect report_time / draft-acceptance logs. For repeated warm measurements, the benchmark reads those same inference YAML files:

Run all four configurations explicitly, so inherited environment variables do not change the comparison. Stop other GPU model workloads first. From the repository root, with QWEN38_MODEL set:

BENCH_DIR="/tmp/eole-qwen38-bench-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BENCH_DIR"
nvidia-smi > "$BENCH_DIR/gpu.txt"
git rev-parse HEAD > "$BENCH_DIR/commit.txt"

for COMPILE in 0 1; do
  if [ "$COMPILE" = 0 ]; then MODE=eager; else MODE=compiled; fi
  for PATH_MODE in baseline mtp; do
    if [ "$PATH_MODE" = baseline ]; then
      CONFIG=recipes/qwen38/predict.yaml
    else
      CONFIG=recipes/qwen38/predict-mtp.yaml
    fi
    EOLE_TORCH_COMPILE="$COMPILE" EOLE_COMPILE_MODE=0 \
      python recipes/qwen38/benchmark.py -c "$CONFIG" --runs 5 \
      --output "$BENCH_DIR/$MODE-$PATH_MODE.json" \
      > "$BENCH_DIR/$MODE-$PATH_MODE.log" 2>&1 || exit 1
  done
done
echo "$BENCH_DIR"

Each invocation loads one engine, excludes one warmup, and measures five runs of the same prompt. Compile mode 0 requests CUDA graphs; backend diagnostics do not prove graph capture. Check measured runs for remaining warmup or timing outliers. Outputs include wall time, peak allocated VRAM, decoded text, and approximate retokenized output counts. Use Eole's logs for internal generated counts, decode timing, and acceptance; retokenization can change token counts. EOS can end a completion early, so verify equal generated lengths. Request wall time includes prefill, decoding, and other overhead. report_time also enables MTP GPU phase profiling, whose instrumentation is included in these timings. The harness fails if an MTP run reports fallback or never logs draft acceptance.

Record GPU/driver, PyTorch/kernel versions, checkpoint ID, precision, prompt/output lengths, compile mode, and draft acceptance with shared results. Try self_speculative_num_tokens: 1, 2, and 3 in a local YAML copy; more drafts can be slower when rejected.

Startup preparation and the local verifier cache​

serve.yaml preloads Qwen and runs a short inference warmup before accepting HTTP requests. In single-GPU compile mode 0 with MTP, only the full verification chunk is compiled; shorter final chunks execute eagerly. The first compilation is saved locally, and the same in-memory program serves requests immediately. Restarts reload the compatible program without tracing the whole decoder again.

Compiled text inference also prepares the prefill kernels, MTP prefix cache and first-token selection before the step-0 timer starts. This preparation runs once per batch layout, preserves the sampling RNG, and clears temporary model state; it does not repeat for every new chat length. Its time appears under warmup, and remains part of the total request latency. Vision requests retain their existing warmup behavior.

This also applies to ordinary local Python inference with InferenceEnginePY:

export EOLE_VERIFIER_CACHE_DIR="$HOME/.cache/eole/verifiers"
EOLE_TORCH_COMPILE=1 EOLE_COMPILE_MODE=0 \
  python -m eole.bin.main predict -c recipes/qwen38/predict-mtp.yaml

The cache directory defaults to ~/.cache/eole/verifiers (or under XDG_CACHE_HOME). Use speculative_verifier_cache: "" to disable persistence. Artifacts are local, trusted executable files, keyed by source/runtime, GPU, model layout and cache shape. They are regenerated when incompatible; unsupported serialization falls back to ordinary compilation. No downloadable artifact is needed. Mode 2, non-MTP and tensor-parallel compilation behavior is unchanged. Container/Pod restarts benefit only while the cache storage is retained.

Run the complete local validation​

validation.yaml names the checkpoint, baseline/MTP inference YAML files, server YAML, loopback port, output directory, and optional Claude Code check. Set QWEN38_MODEL as above or edit model_path in a local copy. Run from the repository root in a terminal that can access the GPU:

python recipes/qwen38/run_validation.py -c recipes/qwen38/validation.yaml

This runs eager baseline/MTP benchmarks sequentially, starts the configured loopback server, runs the API smoke checks, and optionally asks Claude Code to read a sentinel file in an isolated directory (claude: true). It stops only the server it started. Logs, JSON results, and a SUCCESS marker are saved in the configured output directory. A failure preserves logs and exits nonzero. Set claude: false for API-only checks. Use a fresh output directory per run. Inspect comparison.json, MTP acceptance in server.log, and the Claude Read tool block before calling the integration validated. The sentinel exercise does not establish a complete file-editing workflow; use the Claude recipe's coding task for that additional check.

Targeted INT4 comparison (2026-10-08)​

The converted RedHat checkpoint passed two of four medium LiveCodeBench tasks that the corrected-RoPE Frozenlock AutoRound checkpoint failed. Both models were rerun on the same Eole code with identical prompts, BF16 compute, compiled inference, MTP disabled, greedy decoding, and an 8192-token output budget. Neither model reached that budget on these tasks.

TaskFrozenlock AutoRoundRedHat INT4
atcoder/abc392_dFailFail
atcoder/abc396_dFailPass
leetcode/3779FailFail
leetcode/3722FailPass

This motivates the RedHat default, but the tasks were deliberately selected from AutoRound failures; 2/4 is not an overall LiveCodeBench score. See the comparison record. Run this recipe's MTP validation on your own hardware before using the historical throughput figures as a guide.

Historical Frozenlock RTX 5090 example (2026-10-07)​

These measurements use the previous Frozenlock default, not RedHat. The recipe's prompt fixture was run at commit ff2ab2eb3edfe5c09e449284b62c16f050973081 with the local converted Frozenlock/Qwen3.8-27B-int4-Autoround checkpoint, INT4 weights / BF16 compute, PyTorch 2.12.1+cu132, FlashAttention 2.8.3, FLA 0.5.2, and Triton 3.7.1. The driver was 610.43.02 and the pre-run power limit was 575 W; sustained power and clocks were not recorded. One warmup was excluded per configuration; each of five measured runs generated 256 tokens with batch size one. Compiled runs used mode 0; CUDA graph capture was requested but not verified.

ConfigurationMedian request wall timeMedian reported decode throughputEnd-to-end throughputMaximum peak allocated GPU memory
Eager, ordinary greedy11.12 s23.3 tokens/s23.0 tokens/s19.25 GB
Eager, MTP (three drafts)3.85 s69.2 tokens/s66.4 tokens/s19.65 GB
Compiled, ordinary greedy4.24 s63.0 tokens/s60.3 tokens/s23.59 GB
Compiled, MTP (three drafts)2.64 s115.1 tokens/s97.0 tokens/s23.92 GB

End-to-end throughput is 256 divided by median request wall time; reported decode throughput comes from Eole's internal timing and excludes prefill and some request overhead. Memory is PyTorch peak allocated memory in decimal GB, not total process VRAM. Medians include all five runs. Compiled baseline run 2 was slower (5.22 s versus 4.20–4.26 s for the others); it was retained. Compiled MTP wall time ranged from 2.636 to 2.648 s.

Compare MTP against the baseline with the same compilation setting. Compilation alone brings baseline decode throughput from 23.3 to 63.6 tokens/s. Adding MTP to the compiled baseline gives about 1.81× reported decode throughput and 1.61× end-to-end throughput on this fixture. In eager mode, the corresponding gains are 2.97× and 2.89×. The eager comparison alone does not describe the benefit over compiled ordinary decoding.

MTP accepted 186/204 drafts (91.2%) in every eager run and 184/211 (87.2%) in every compiled run. Text was repeatable within each configuration, but MTP and baseline outputs differed in both modes. These fixed-length timings do not establish exact greedy parity or complete/correct generated code. Use longer output limits for coding tasks and validate the resulting files.

The recorded measurements preserve all five timings, decode rates, output hashes, backend diagnostics, and comparison limits. This is one short prompt, not a general ranking across workloads or engines.

34 CPU fallback tests exercise native MTP and greedy verification; those tests alone do not certify GPU behavior. The live API run confirmed text generation and active MTP, and exposed XML numeric tool arguments being returned as strings. That server conversion bug has a schema-aware fix and regression tests. The live text, numeric tool round-trip, text SSE, and tool SSE checks passed after the fix. Claude Code 2.1.79 passed Read, Write, and Bash checks, including a generated three-test suite. Its configured output budget was 2048 tokens. See the implementation review for rounding and backend limitations.

Quantization module selection uses the existing quant_layers and quant_exclude_modules settings. Bare include names such as down_proj select that leaf name anywhere in the model. Dotted entries select exact paths from the model root, and shell-style globs such as decoder.transformer_layers.*.mlp.down_proj select decoder projections only. * can span path components. Exclusions take precedence; excluding a parent skips its entire subtree. Bare exclusions retain their legacy behavior of matching parent names anywhere in the model.

For compressed-tensors conversion, quant_layers is populated with exact Eole paths from the packed weight inventory, preserving floating-point decoder and MTP modules. Loading rejects selections that conflict with stored packed weights. All quantization backends share these matching rules. Bits and group size remain global settings; per-module quantization parameters are not supported.

INT4 tensor parallelism​

For a converted RedHat INT4 checkpoint, use the two-GPU greedy recipe:

EOLE_TORCH_COMPILE=1 EOLE_COMPILE_MODE=0 \
  python -m eole.bin.main predict -c recipes/qwen38/predict-tp.yaml

It sets world_size: 2, gpu_ranks: [0, 1], and parallel_mode: tensor_parallel. Its context limit starts at 2048 tokens; choose a larger value according to the memory available on both GPUs.

Packed GPTQ weights are partitioned before Marlin repacking. Attention and MLP projections are sharded; GDN projections and recurrent state remain replicated. Tensor parallelism can reduce weight memory without guaranteeing faster decoding. setup.py targets all visible GPU architectures, including both sm_120 and sm_89 on a mixed RTX 5090/4090 system. CUDA_VISIBLE_DEVICES controls which GPUs are included. Rebuild Eole after adding a GPU with a different architecture:

MAX_JOBS=2 python -m pip install -e . --no-build-isolation

Restart inference processes after rebuilding. A binary built only for an RTX 5090 cannot execute its native kernels on an RTX 4090.

For greedy, batch-one MTP inference, set self_speculative_decoding: true in the two-GPU recipe. The MTP KV cache uses each rank's local KV-head count. Two-GPU RedHat INT4 text generation was checked with and without MTP; this is a compatibility check, not a throughput benchmark.

Bound CUDA graphs with tensor parallelism​

Compile modes 0 and 2 use bound CUDA graphs for eligible static-cache, FlashAttention FP16/BF16 decoding on both ranks. Full MTP verification chunks can replay a graph; drafting and partial final chunks stay outside it. No additional opt-in setting is needed.

Tensor-parallel workers share token choices, beam parents/scores, and MTP verification decisions from rank 0 so numerical differences or sampling cannot make ranks follow different histories. Only the selected tensors are transferred, not the full vocabulary. A separate CPU Gloo group coordinates allocation and capture readiness; these checks do not run on graph replay steps.

If allocation fails before recording, all ranks use the uncaptured fallback. If recording fails after NCCL operations may have been captured, the request aborts on all ranks; restart the inference workers before retrying. Single-GPU capture failures retain their uncaptured fallback.

Beam search and other ineligible configurations retain their existing decoder path. The legacy compiled tensor-parallel beam path can still hang at worker shutdown; bound greedy/MTP decoding does not fix that separate path. Streaming inference remains limited to single-process mode.