A GPU moves from memory‑bound token streaming to compute‑bound batch verification when speculative decoding is applied.
SPECULATIVE DECODING SPEED EXPLAINED: GPU WORK SHIFT DRIVES GAINS
*Speculative decoding isn’t faster because it does less work. It reshapes GPU workload from memory‑bound to compute‑bound, unlocking speed at low batch sizes. The trade‑off hinges on how draft and target models share compute.*
Speculative decoding has sparked debate in LLM circles. Critics claim it speeds inference by doing less work per token, but the math tells a different story. The technique actually raises total FLOPs, yet users see faster latencies. The paradox resolves when we examine GPU workload categories. Memory‑bound operations stall while data streams from global to local memory; compute‑bound operations keep the arithmetic engines busy. Traditional one‑token‑at‑a‑time decoding is almost always memory bound, especially for models with billions of parameters. By batching token verification, speculative decoding flips the bottleneck, letting the GPU spend cycles on dense matrix math instead of waiting on data. The result is a measurable speed boost at low batch sizes, even though more arithmetic is performed.
What is speculative decoding?
Speculative decoding pairs a small draft model with a large target model. The draft proposes a sequence of tokens; the target model verifies that sequence in a single forward pass. Traditional decoding generates one token per pass, streaming the full parameter set each time. By contrast, speculative decoding lets the GPU evaluate many tokens at once, accepting those that match the target model’s output. The technique does not cut FLOPs; it actually raises the total floating‑point operations for the same output.
Why does the GPU become faster?
GPU work splits into compute‑bound tensor math and memory‑bound data movement. Decoding one token at a time forces the GPU to load all active parameters—27 billion for Qwen3.5‑27B—making the process memory bound and wasting compute cycles. When speculative decoding batches tokens, the GPU’s memory pipeline stays full while the compute units stay busy, shifting the workload to compute‑bound. In this regime the GPU can execute more operations per clock, turning idle memory bandwidth into productive arithmetic.
When does speculative decoding help?
The sweet spot appears when the optimal token batch is about five per forward pass. Fewer tokens keep the GPU memory bound; more tokens push it into compute bound. At batch sizes far below five, speculative decoding trades breadth (multiple sequences) for depth (more tokens in a single sequence). The GPU does roughly the same amount of work, but the single sequence advances farther, completing faster. Adding more sequences while keeping depth constant re‑introduces compute saturation and erodes the speed advantage.
What are the limits and controls?
If the draft model’s acceptance rate falls below 100 %, every rejected token adds wasted work. At high throughput, vLLM offers a --speculative-disable-by-batch-size flag to turn off speculation when batch sizes grow large enough to become compute bound. Researchers note that static depth per timestep is suboptimal; dynamic depth—deeper speculation when compute is free, shallower when batches swell—underpins emerging ideas like dSpark.
Frequently asked questions
Why does speculative decoding require more FLOPs?
Because the draft model generates a token sequence that the large model must verify in a single forward pass, adding verification work to the original generation.
When does speculative decoding stop being beneficial?
When batch sizes grow large enough that the GPU becomes compute bound; at that point vLLM can disable speculation with the --speculative-disable-by-batch-size flag.
What is the optimal token batch size for speculative decoding?
The source example cites five tokens per forward pass as the point where the GPU shifts from memory bound (fewer tokens) to compute bound (more tokens).
The takeaway for operators is clear: speculative decoding delivers speed only when the GPU’s compute pipeline is under‑utilized. As batch sizes climb, the advantage evaporates, and turning the feature off can reclaim wasted FLOPs. Future research will likely focus on adaptive depth strategies, ensuring that every extra operation translates into real latency gains rather than idle cycles.
Sources: Hacker News (Why Is Speculative Decoding Fast?)