Speculative Decoding: How Local AI Gets Faster (2026 Guide)

Published: August 8, 2026 — LLMs generate one token at a time, and each token needs a full forward pass — so the fastest model is still bounded by sequential steps. Speculative decoding breaks that assumption: a tiny "draft" model guesses several tokens ahead, and the big model verifies the guesses in a single parallel pass. Accepted guesses are free speed; rejected ones get corrected. The result is a 1.5–3x speedup with zero change to the output — the closest thing local AI has to a free lunch.

⚡ Quick Takeaways

Why Generation Is Slow in the First Place

Every generated token requires a full forward pass through the entire model. That is sequential by design: token N+1 depends on token N. On a GPU, a single pass through a 7B model might take 20–40ms — which is why an un-optimized 7B often lands around 25–50 tokens per second. You cannot parallelize across tokens because of the dependency chain.

But here's the observation that makes speculative decoding possible: most next tokens are predictable. A much smaller model can guess them with high accuracy. The big model then checks the whole guess sequence in one parallel pass — a pass that costs about the same as generating a single token — and accepts everything it agrees with.

💡 The insight: verification of 4 tokens costs roughly the same as generating 1 token, but yields 4 tokens when the draft guessed right. Speedup = acceptance rate × draft length, minus draft overhead.

How Speculative Decoding Works, Step by Step

  1. Draft. A small model (e.g. 0.5–2B) generates K candidate tokens quickly — say 4 tokens of a plausible continuation.
  2. Verify. The large model processes all K candidates in one forward pass, computing the true probability of each.
  3. Accept or reject. Tokens where the draft's pick matches the large model's distribution are accepted. The first mismatch is rejected and replaced with the large model's own token.
  4. Repeat. The accepted prefix becomes the new context, and the draft starts guessing again.

The math guarantees the output is indistinguishable from non-speculative decoding — the rejection-sampling step is what preserves exactness. This is why the technique is described as a lossless acceleration, not an approximation.

How Much Faster, and On What Hardware

Setup Typical speedup Why
GPU, good draft match 1.5–3x Draft runs in idle GPU capacity; verification is parallel
GPU, poor draft match 1.0–1.3x Low acceptance rate; verification overhead dominates
CPU-only 0.9–1.1x Both models compete for the same memory bandwidth
Very short generations < 1x possible Fixed overhead exceeds saved steps

Modern variants push further: self-speculative decoding skips the separate draft model by having the target model guess its own next tokens from early layers, and n-gram drafts use pure statistics from the context. Both remove the "which draft model?" question entirely.

How to Enable It (2026)

llama.cpp

# Serve a 7B target with a 0.5B draft
llama-server -m qwen3-8b-q4_k_m.gguf \
  -md qwen3-0.6b-q4_k_m.gguf \
  -c 8192

# CLI generation
llama-cli -m target.gguf -md draft.gguf -p "Write a haiku"

vLLM

from vllm import LLM
llm = LLM(model="Qwen/Qwen3-8B",
          speculative_model="Qwen/Qwen3-0.6B")

Ollama

# Recent builds support a speculative model via env var
OLLAMA_SPECULATIVE_MODEL="qwen3:0.6b" ollama serve

Verifying it's active: vLLM logs show "Speculative Decoding" with draft stats; llama.cpp prints acceptance-rate diagnostics. If your tokenizer or vocabulary doesn't match between draft and target, the feature silently no-ops — always confirm in the logs.

Picking a Draft Model

Criterion Good choice Bad choice
Size 1/4 to 1/10 the target's size Same size as target (no gain)
Family Same model family as target Different family, different style
Tokenizer Identical vocab as target Mismatched — silently disables the feature
Data Trained on similar domain Domain mismatch → low acceptance

Examples that work well in practice: Qwen3-8B + Qwen3-0.6B, Llama 3.1 8B + Llama 3.2 1B, and same-family pairings from the sub-12B rankings. When in doubt, run a quick acceptance-rate test — if it's below ~0.5, the pairing is wrong.

When It Helps Your Stack

Pair it with the other levers in How to Speed Up Local LLMs — quantization first, then speculative decoding, then batching — for the full stack of free speedups.

Frequently Asked Questions (FAQ)

What is speculative decoding?

Speculative decoding is an inference technique where a small, fast draft model proposes several likely next tokens while the large model verifies them all in one parallel pass. Accepted tokens are kept, rejected ones are corrected, and the output distribution stays identical — but far fewer sequential steps are needed.

Does speculative decoding change the model's output?

No — with rejection sampling, the accepted tokens match exactly what the large model would have generated on its own. It is a lossless speedup. Some implementations relax this for extra speed, but the standard version is output-identical.

How much faster is speculative decoding?

Typically 1.5–3x on GPU setups where the small draft model runs in the spare compute. Speedup depends on how often the draft agrees with the target model and on draft size — too large a draft eats the gains. On CPU-only setups gains are usually small because both models fight for the same memory bandwidth.

When does speculative decoding NOT help?

On CPU-only machines (memory-bandwidth-bound, so a second model just slows things down), with badly matched draft/target pairs, and on very short generations where the verification overhead exceeds the saved steps.

How do I enable speculative decoding in llama.cpp or Ollama?

In llama.cpp use the -md (model draft) flag with a smaller model, e.g. llama-server -m target.gguf -md draft.gguf. In vLLM, set speculative_model="draft-name" in the engine config. Ollama exposes it as OLLAMA_SPECULATIVE_MODEL in recent versions.

What makes a good draft model?

The best drafts are much smaller (1/4 to 1/10 the target), share the target's tokenizer and vocabulary, and are trained on similar data so their guesses align. A model from the same family as the target is ideal; a mismatched tokenizer silently disables the trick.

Sources & Further Reading