AskAiml
Sign in Get started

Speculative decoding

Draft-then-verify to speed up generation without changing the output distribution.

⚡ Module 5 7 min read Not started

Why this matters

LLM decoding is memory-bandwidth bound: at batch size 1 you read every weight once per token, so generating tokens at the speed of bytes-per-second. Speculative decoding gets multiple tokens per "weight read" by using a small draft model (or a draft head) to propose tokens that the big model verifies in parallel. Real-world speedups: 2-4× at the same quality. Every modern inference engine (vLLM, TGI, SGLang) supports it.

Knowing this is the difference between answering "how do you make inference faster?" with "throw more GPUs at it" vs a real engineer's answer.

Learning objectives

  1. Explain speculative decoding mathematically (no quality loss).
  2. Implement a basic speculative-decoding loop.
  3. Compare draft-model, EAGLE, Medusa, and lookahead decoding.
  4. Configure vLLM for speculative decoding.
  5. Reason about when speculative decoding helps vs hurts.

1. The setup

LLM decode is sequential: each token requires reading the entire model. On an A100/H100 with a 7B bf16 model:

weight bytes      = 7B * 2 = 14 GB
HBM bandwidth     = ~1.5 TB/s
peak tok/s/batch  = 1500 / 14 ≈ 107 tok/s

Achievable batch-1 token rate is ~50-90 tok/s. The GPU's compute units sit ~80% idle.

The fix: batch more tokens per forward pass without the user waiting for them. That is what speculative decoding does.

2. The core algorithm (Leviathan et al. 2022, Chen et al. 2023)

Two models:

  • Draft (small, fast): e.g., a 0.5B model.
  • Target (big, slow): the model whose distribution you want.

Each step:

  1. The draft model autoregressively proposes k tokens (e.g., k=4).
  2. The target model is fed the original prefix + the k draft tokens in parallel — one forward pass scoring all k+1 distributions.
  3. For each position i = 1..k:
    • Compute acceptance ratio = p_target(d_i | ...) / p_draft(d_i | ...).
    • Accept with probability min(1, ratio).
    • On rejection: sample a correction from max(0, p_target - p_draft) (renormalised). Stop accepting further drafts.
  4. If all k accepted, sample one bonus token from p_target at position k+1.

This procedure provably samples from p_target exactly. No quality loss. You get up to k+1 tokens per target forward pass.

3. A minimal implementation

python
@torch.no_grad()
def speculative_step(target, draft, prompt_ids, k=4, T=1.0):
    # 1. draft k tokens
    draft_ids = []
    cur = prompt_ids
    draft_logits = []
    for _ in range(k):
        out = draft(cur)
        l = out[:, -1] / T
        p = F.softmax(l, -1)
        nxt = torch.multinomial(p, 1)
        draft_ids.append(nxt.item())
        draft_logits.append(l.detach())
        cur = torch.cat([cur, nxt], dim=1)

    # 2. target scores all k+1 positions in one pass
    full = cur                                        # prompt + k drafts
    target_logits = target(full)[:, -k-1:-1]          # (B, k, V) for the k drafts
    bonus_logits  = target(full)[:, -1]               # (B, V) for the bonus

    # 3. accept-reject
    accepted = []
    for i in range(k):
        p_t = F.softmax(target_logits[:, i] / T, -1)[0, draft_ids[i]]
        p_d = F.softmax(draft_logits[i] / T, -1)[0, draft_ids[i]]
        r = (p_t / max(p_d, 1e-9)).item()
        if torch.rand(1).item() < min(1.0, r):
            accepted.append(draft_ids[i])
        else:
            # correction sample
            diff = (F.softmax(target_logits[:, i]/T, -1)
                    - F.softmax(draft_logits[i]/T, -1))
            diff = torch.clamp(diff, min=0)
            diff = diff / diff.sum()
            corr = torch.multinomial(diff, 1).item()
            accepted.append(corr)
            return prompt_ids.tolist() + accepted, False
    # all accepted; add bonus
    bonus = torch.multinomial(F.softmax(bonus_logits/T, -1), 1).item()
    accepted.append(bonus)
    return prompt_ids.tolist() + accepted, True

Wrap this in a loop until you hit max tokens or EOS.

4. Choosing the draft model

You want the draft model:

  • Much smaller (10-30× smaller is a good rule).
  • Whose distribution matches the target (same family helps a lot).
  • That hits the same tokenizer / chat template.

Common choices:

TargetDraft
Llama-3.1 70BLlama-3.2 1B (same family)
Qwen 2.5 32BQwen 2.5 0.5B / 1.5B
DeepSeek-V3DeepSeek-V3-Base (own MTP head)
Mistral LargeMistral 7B

Acceptance rate is everything: 70%+ acceptance gives you ~2-3× speedup; 40% is barely worth the effort.

5. Modern variants

Medusa (Cai et al., 2024)

Replace the draft model with multiple lightweight heads on the target itself, each predicting tokens 1, 2, 3, ... ahead. No second model. Trains in hours. 1.5-2× speedup. Used in vLLM and TGI.

EAGLE / EAGLE-2 / EAGLE-3 (Li et al., 2024-2025)

Slightly more sophisticated heads using the model's own hidden states + a small autoregressive head. EAGLE-3 reaches ~3-5× speedups, currently SOTA. Built into vLLM.

Lookahead decoding (Fu et al., 2024)

N-gram-based; uses a Jacobi-iteration trick to verify multiple candidate token sequences without a draft model. Lighter to deploy than Medusa, less speedup.

Multi-token prediction (DeepSeek-V3)

The model is trained to predict the next 2 tokens jointly. The second prediction acts as a free draft. Used at inference for speculative-style speedup.

Self-speculative decoding

Use the model's own early layers as the draft (skip late layers for the draft pass). No second model; modest speedups.

6. Configuring vLLM

bash
# Draft-model-based
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative_model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5 \
  --use-v2-block-manager

# EAGLE-based
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative_model jamesnav/EAGLE-3-LLaMA-3-70B \
  --speculative_method eagle

You can dynamically tune num_speculative_tokens based on observed acceptance rates.

7. When speculative decoding helps (and when not)

Helps when:

  • Batch size is small (memory-bandwidth bound regime).
  • Latency is the constraint (you want first tokens fast).
  • Model and draft share a tokenizer.
  • Acceptance rate is high (≥60%).

Does not help much when:

  • Batch size is large (compute-bound; you already saturate the GPU).
  • Acceptance rate is low (different family, very different temperatures).
  • Generation length is very short (overhead dominates).

The rule: speculative decoding is the big win for conversational latency, less so for batched serving throughput.

8. Combining with other tricks

Speculative decoding stacks with:

  • Prefix caching — both reduce target-model work; complementary.
  • FP8 KV cache — orthogonal memory saving.
  • Quantized weights (AWQ/GPTQ) — speed up the target forward; helps further.
  • Tensor parallel — works fine; verify both models are on the right ranks.

Hands-on lab (3 hours)

spec_decode_lab.ipynb:

  1. Implement speculative_step from above. Pair Qwen2.5-7B-Instruct (target) with Qwen2.5-0.5B-Instruct (draft).
  2. Generate 200 tokens for 5 prompts. Measure tokens/sec; compare to baseline (no draft). Report acceptance rate.
  3. Increase k from 2 → 4 → 8. Plot speedup. Find the sweet spot.
  4. Try a bad draft (different family, e.g. SmolLM 360M). Show acceptance rate drops and overall speed slows down.
  5. Use vLLM with --speculative_model flag and a real EAGLE checkpoint. Measure throughput vs batch size.
  6. Bonus: implement Medusa-style heads on a small model — train 4 heads to predict tokens 1-4 ahead, verify ~1.5× speedup.

Common pitfalls

  1. Different tokenizer between draft and target → drafts are interpreted as wrong tokens. Always confirm tokenizer compatibility.
  2. High temperature on the draft → low acceptance. Match draft and target temperatures.
  3. Forgetting that rejection happens — your token count is variable per step. Avoid fixed-step buffers.
  4. Using speculative decoding under high batch sizes — net loss.
  5. Mixing draft + target across different fine-tunes (e.g., draft is base, target is RLHF'd) → distributions diverge → low acceptance.

Self-check

  1. Why is naive decode memory-bandwidth bound?
  2. What does the acceptance probability min(1, p_t/p_d) ensure?
  3. Difference between draft-model and Medusa speculative decoding.
  4. When does speculative decoding not help?
  5. What is multi-token prediction's role at inference?

References

  • Leviathan et al. (2022), "Fast Inference from Transformers via Speculative Decoding."
  • Chen et al. (2023), "Accelerating Large Language Model Decoding with Speculative Sampling."
  • Cai et al. (2024), "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads."
  • Li et al. (2024-2025), "EAGLE / EAGLE-2 / EAGLE-3."
  • DeepSeek-AI (2024), "DeepSeek-V3 Technical Report" (multi-token prediction).
  • vLLM speculative decoding docs.

Sign in to save your progress and earn badges.