Speculative decoding

Recent sources

What is moving the narrative

• R9700 AI Pro TP=2 Qwen3.8-27B-FP8 low speed? Need Advice.

  A user reports relatively low vLLM throughput when serving the FP8 Qwen3.8-27B model across two R9700 GPUs with MTP3 speculative decoding and asks for troubleshooting advice. The provided metrics include prompt and generation speeds, acceptance rates, and KV-cache utilization.

  NewsReddit – r/LocalLLaMAYehowaH· today· 5 linked cases
• Looking at a PowerColor R9700 for Qwen3.8-27B, Q4_K_XL, llama.cpp/Vulkan.

  The discussion seeks sustained throughput measurements for Qwen3.8-27B quantized inference on a PowerColor R9700 using llama.cpp/Vulkan at 64K or longer context lengths, noting that published figures omit important test conditions. It also asks whether MTP speculative decoding is stable in practical workloads.

  OpinionReddit – r/LocalLLaMABillyQ· yesterday· 5 linked cases
• Weird speed gap between LM Studio vs raw llama.cpp + questions on Reasoning Effort (Qwen3.8-27B on dual GPU)

  A user compares inference throughput for the same quantized Qwen model in LM Studio and llama.cpp on a dual-GPU Windows system, finding a substantial speed difference. The discussion also examines how reasoning-effort settings and speculative decoding vary across GGUF releases.

  NewsReddit – r/LocalLLaMAMkGod· yesterday· 5 linked cases
• 28 TPS on Qwen2.5-7B across two separate cloud regions over public WAN using speculative decoding + CUDA Graphs [P]

  A developer reports benchmarking a distributed inference framework that partitions a Qwen2.5-7B model across T4 GPUs in separate cloud regions connected over the public internet. Speculative decoding and CUDA Graphs are used to reduce the effect of WAN latency, achieving about 28 tokens per second.

  ResearchReddit – r/MachineLearningkatua_bkl· yesterday
• Unsloth Q1-Q2 Qwen3.8-27B with MTP since the unsloth ones don't ship with for the lowest quants

  A community experiment applies MTP grafting to Qwen3.8-27B quantized variants to reduce RAM usage during Vulkan-based local inference. The author reports that enabling reasoning improves responses on some culturally specific questions, while warning that overall answer quality is inconsistent.

  ResearchReddit – r/LocalLLaMANyghtbynger· yesterday· 5 linked cases
• I benchmark DFlash 2 (PR build) in llama.cpp on Qwen 3.8 27B against all speculative methods for 3 days. 2.26x on 100 real coding prompts, 4.68x with one n-gram drafter on top. Up to 8x on specific cases.

  A benchmark of the DFlash 2 speculative-decoding implementation in llama.cpp reports roughly 2.26× faster generation on 100 coding tasks for a Qwen 27B model, with higher gains when combined with an n-gram drafter. The evaluation compares several decoding approaches on a single RTX PRO 6000 under single-request workloads.

  ResearchReddit – r/LocalLLaMAFantasticNature7590· 2d ago· 5 linked cases

Deployment evidence

Corroborated cases in the catalog

These deployments are the strongest catalog links carried by the trend pulse. Multiple independent articles must point to a case before it counts as corroborated evidence.

1. 01Alibaba Cloud Tair KVCache: 3FS-based enterprise KVCache storage pipeline for agent-style inference
2. 02Fireworks.ai delivers 4x generative AI throughput and cuts latency using AWS EC2 P5 (NVIDIA H100)
3. 03Akool builds production-grade B2B AI video generation on Alibaba Cloud with Qwen-VL and Model Studio