// HACKER NEWS — CYBERSECURITY
Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP
I gave Qwen3.8's MTP drafter another 69.2 MiB of precision. Throughput fell from 50.44 to 37.02 tokens per second. That result sums up the whole experiment: the best local inference setup is rarely made from the individually "best" parts.
I wanted a dense 27B model, its full 262,144-token context, multimodal input, maximum useful quality, and speculative decoding on an NVIDIA RTX PRO 4000 Blackwell SFF with 24 GB of VRAM. The server also had to survive real agent work after printing model loaded. The experiment followed a hunch I had written about earlier: careful operation may matter as much as moving to a larger model.
The finished system averages 50.44 tok/s in the current ten-run production series. On a strict runtime A/B, the custom llama.cpp build reaches 55.40 tok/s versus 45.42 for clean master, a 21.97% gain. Against target-only greedy decoding, embedded MTP moves 21.19 to 59.46 tok/s, or 2.81 times the throughput. At the far end of a genuinely occupied 256K cache, it still produces 12.61 tok/s without an out-of-memory failure.
Those numbers came from different gates and should stay separate. Combining them into one heroic speedup would make a better headline and a worse benchmark.
Qwen3.8 27B is a 64-layer dense model. Its repeating pattern contains three Gated DeltaNet layers followed by one full-attention layer, giving 48 recurrent layers and 16 conventional attention layers. It has a native 262,144-token context, a one-layer MTP head, and a separate 27-layer vision encoder.
Only the 16 full-attention layers grow a conventional KV cache with sequence length, which makes 256K less absurd than it first appears. With Q4 K and V, that cache costs roughly 4.25 GiB before allocator overhead. DeltaNet adds recurrent state and checkpoints instead. Four checkpoints were the useful minimum; the default 32 spent memory I needed elsewhere.
NVIDIA quotes 432 GB/s of peak bandwidth. That is a hardware ceiling rather than an application metric from llama.cpp, but it matters here. Autoregressive decode repeatedly streams quantized weights, and the 16 attention layers add increasingly expensive KV reads as context fills. This is why the same profile averages about 50 tok/s on the production task and 12.61 tok/s at the far end of a 261.5K-token cache.
The original plan was simple: estimate the capacity, select a quant, then benchmark it. The machine immediately taught me that capacity estimates are just admission tickets. The real test begins after loading.
I began with public GGUFs at 40K context. Q4_0 was surprisingly strong. Target-only decoding reached 22.40 tok/s, and MTP with n_max=3 reached 44.95. It beat smaller Q3_K_M and nominally smarter Q4_K_M variants because file size and quant label do not describe the CUDA kernel that actually runs.
Then quality testing spoiled the easy answer. On a short, identical WikiText-2 control, IQ4_XS scored 6.1175 perplexity while Q4_0 scored 6.3798. Q4_0 led the speed table. Hermes needed a main model, though, and that quality trade felt too expensive for a few hundred milliseconds. I would have been using a 27B model as oversized autocomplete.