Speculative Decoding with Draft Models

Table of Contents:

• What Is Speculative Decoding?
• How Draft Models Work
• Implementation with vLLM
• Choosing the Right Draft Model
• Performance Benchmarks
• Production Deployment Patterns
• When Speculative Decoding Fails
• Frequently Asked Questions

What Is Speculative Decoding?

Short answer: Speculative decoding uses a small, fast "draft model" to propose multiple tokens at once, which a larger "target model" verifies in parallel — achieving 2-3x faster inference without changing output quality or requiring model modification.

After optimizing LLM inference for production systems at HinterBuild, the bottleneck is consistent: autoregressive decoding generates one token at a time, underutilizing GPU parallelism. Speculative decoding breaks this constraint by speculatively generating multiple tokens cheaply, then verifying them efficiently.

Key Takeaways:

• Draft model (small, fast) proposes K tokens → target model (large, accurate) accepts or rejects in parallel
• 2-3x speedup typical, up to 4x in high-acceptance scenarios (code, structured output)
• Zero quality loss — output distribution identical to standard decoding (mathematically proven)
• Best for: large models (70B+), long outputs (>100 tokens), structured generation
• Works with: vLLM, HuggingFace, TGI — no model training required

For teams building LLM serving infrastructure or AI agents with latency constraints, speculative decoding is the highest-ROI optimization in 2026.

How Draft Models Work

Standard Autoregressive Decoding

tokens = [prompt_tokens]
for _ in range(max_new_tokens):
    logits = large_model(tokens)        # Expensive: full 70B forward pass
    next_token = sample(logits[-1])     # Sample 1 token
    tokens.append(next_token)
    if next_token == EOS:
        break

Cost: N tokens = N forward passes through 70B model.

Speculative Decoding

# Speculative decoding: K tokens per verification
draft_tokens = [prompt_tokens]
target_tokens = [prompt_tokens]

while len(target_tokens) < max_new_tokens:
    # Step 1: Draft model proposes K tokens (cheap)
    for _ in range(K):
        draft_logits = draft_model(draft_tokens)  # Fast: 1B forward pass
        next_token = sample(draft_logits[-1])
        draft_tokens.append(next_token)

    # Step 2: Target model verifies all K tokens in parallel (1 pass)
    target_logits = large_model(target_tokens + draft_tokens[-K:])

    # Step 3: Accept/reject each drafted token
    for i in range(K):
        draft_prob = softmax(draft_logits[i])[draft_tokens[-K + i]]
        target_prob = softmax(target_logits[i])[draft_tokens[-K + i]]

        if random() < min(1, target_prob / draft_prob):
            target_tokens.append(draft_tokens[-K + i])  # Accept
        else:
            # Reject: resample from adjusted distribution
            adjusted_probs = (target_logits[i] - draft_logits[i]).clamp(min=0)
            target_tokens.append(sample(adjusted_probs))
            break  # Stop verification, start new draft

Key insight: Target model verifies K drafted tokens in one forward pass using parallel attention — accepts 60-90% of tokens, resulting in 2-3x effective speedup.

Acceptance Rate and Speedup

Speedup formula: speedup ≈ acceptance_rate × K / (1 + overhead)

For LLM inference optimization, acceptance rate is the critical metric to monitor.

Implementation with vLLM

vLLM Speculative Decoding Setup

from vllm import LLM, SamplingParams

# Target model: large, accurate (70B)
target_model = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,  # 4x A100 40GB
    speculative_model="meta-llama/Llama-3.2-1B-Instruct",  # Draft model
    num_speculative_tokens=5,  # K = 5 draft tokens per step
    use_v2_block_manager=True,
    gpu_memory_utilization=0.9,
)

prompts = [
    "Write a Python function to implement quicksort",
    "Explain quantum entanglement in simple terms",
]

sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=512,
)

outputs = target_model.generate(prompts, sampling_params)

for output in outputs:
    print(f"Speedup metrics: {output.metrics.speculative_acceptance_rate:.2%}")
    print(f"Output: {output.outputs[0].text}\n")

Configuration tuning:

# Higher K = more speculation (diminishing returns beyond K=7)
num_speculative_tokens=7

# Draft model selection (closer to target = higher acceptance)
speculative_model="meta-llama/Llama-3.2-3B-Instruct"  # Better draft

HuggingFace Transformers Implementation

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load target and draft models
target_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B-Instruct",
    device_map="auto",
    torch_dtype="auto",
)

draft_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B-Instruct",
    device_map="cuda:0",  # Keep draft on single GPU
    torch_dtype="auto",
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-70B-Instruct")

prompt = "Write a function to compute Fibonacci numbers"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

# Enable speculative decoding
outputs = target_model.generate(
    **inputs,
    assistant_model=draft_model,  # Draft model
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Custom Speculative Decoding Loop

import torch
import torch.nn.functional as F

def speculative_decode(
    target_model,
    draft_model,
    input_ids: torch.Tensor,
    max_new_tokens: int = 100,
    K: int = 5,  # Draft length
    temperature: float = 1.0,
) -> torch.Tensor:
    """Custom speculative decoding implementation."""

    tokens = input_ids.clone()

    while tokens.shape[1] < input_ids.shape[1] + max_new_tokens:
        # Step 1: Draft K tokens with small model
        draft_tokens = tokens.clone()
        for _ in range(K):
            with torch.no_grad():
                draft_logits = draft_model(draft_tokens).logits[:, -1, :]
                draft_probs = F.softmax(draft_logits / temperature, dim=-1)
                next_token = torch.multinomial(draft_probs, num_samples=1)
                draft_tokens = torch.cat([draft_tokens, next_token], dim=1)

        # Step 2: Verify with target model (single forward pass)
        with torch.no_grad():
            target_logits = target_model(draft_tokens).logits

        # Step 3: Accept/reject verification
        accepted = 0
        for i in range(K):
            pos = tokens.shape[1] - input_ids.shape[1] + i
            draft_token = draft_tokens[0, -K + i]

            draft_prob = F.softmax(
                target_logits[0, -K - 1 + i] / temperature, dim=-1
            )[draft_token]
            target_prob = F.softmax(
                target_logits[0, -K + i] / temperature, dim=-1
            )[draft_token]

            if torch.rand(1) < (target_prob / draft_prob).clamp(max=1.0):
                tokens = torch.cat([tokens, draft_token.unsqueeze(0).unsqueeze(0)], dim=1)
                accepted += 1
            else:
                # Rejection: sample from adjusted distribution
                adjusted_logits = (
                    target_logits[0, -K + i] - target_logits[0, -K - 1 + i]
                ).clamp(min=0)
                adjusted_probs = F.softmax(adjusted_logits / temperature, dim=-1)
                new_token = torch.multinomial(adjusted_probs, num_samples=1)
                tokens = torch.cat([tokens, new_token.unsqueeze(0)], dim=1)
                break  # Rejection stops this round

        # Early exit if EOS generated
        if tokens[0, -1] == tokenizer.eos_token_id:
            break

    return tokens

For production AI systems, vLLM's built-in implementation is production-ready.

Choosing the Right Draft Model

Selection Criteria

Rules of thumb:

1. Same model family (Llama → Llama, Mistral → Mistral) = higher acceptance
2. Trained on similar data = better alignment
3. Draft 10-50x smaller than target = optimal cost/speedup tradeoff
4. Fine-tuned draft on target domain can boost acceptance 10-20%

Task-Specific Draft Selection

DRAFT_MODELS = {
    "code": "codellama/CodeLlama-7b-hf",      # High acceptance on code
    "chat": "meta-llama/Llama-3.2-1B",        # General purpose
    "structured": "teknium/OpenHermes-2.5-Mistral-7B",  # JSON/YAML
}

def select_draft(task: str) -> str:
    return DRAFT_MODELS.get(task, "meta-llama/Llama-3.2-1B")

# Generate with task-specific draft
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_model=select_draft("code"),
    num_speculative_tokens=5,
)

For AI agent development, domain-specific draft models improve acceptance by 15-25%.

Performance Benchmarks

Real-World Speedups

Setup: Llama 3.1 70B on 4x A100 40GB, 512 token outputs

Cost analysis (AWS):

Result: Speculative decoding achieves 2.5x cost reduction at same latency budget.

Acceptance Rate by Output Length

# Acceptance rate degrades slightly with length
# (draft model loses alignment over long contexts)

Output Tokens | Accept Rate
50           | 82%
100          | 78%
200          | 71%
500          | 64%
1000         | 58%

For long outputs (500+ tokens), consider hybrid approach: speculative for first 200 tokens, standard thereafter.

For LLM serving benchmarks, measure acceptance rate per request.

Production Deployment Patterns

Pattern 1: vLLM with Speculative Decoding

# vllm-speculative-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-70b-speculative
  namespace: llm-serving
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.6.0
        args:
        - --model
        - meta-llama/Llama-3.1-70B-Instruct
        - --speculative-model
        - meta-llama/Llama-3.2-3B-Instruct
        - --num-speculative-tokens
        - "5"
        - --tensor-parallel-size
        - "4"
        - --use-v2-block-manager
        - --gpu-memory-utilization
        - "0.90"
        resources:
          limits:
            nvidia.com/gpu: "4"
        env:
        - name: VLLM_LOGGING_LEVEL
          value: INFO

Pattern 2: Adaptive Speculation

class AdaptiveSpeculativeDecoder:
    """Adjust K based on observed acceptance rate."""

    def __init__(self, target_model, draft_model):
        self.target = target_model
        self.draft = draft_model
        self.acceptance_history = []
        self.K = 5  # Initial speculation depth

    def generate(self, prompt: str) -> str:
        outputs = self.target.generate(
            [prompt],
            SamplingParams(max_tokens=256),
            lora_request=None,
        )

        # Track acceptance
        accept_rate = outputs[0].metrics.speculative_acceptance_rate
        self.acceptance_history.append(accept_rate)

        # Adjust K every 100 requests
        if len(self.acceptance_history) >= 100:
            avg_accept = sum(self.acceptance_history[-100:]) / 100

            if avg_accept > 0.8:
                self.K = min(self.K + 1, 7)  # Increase speculation
            elif avg_accept < 0.5:
                self.K = max(self.K - 1, 3)  # Decrease speculation

        return outputs[0].outputs[0].text

Pattern 3: Hybrid Standard + Speculative

def route_request(prompt: str, max_tokens: int) -> str:
    """Route short outputs to standard, long to speculative."""
    if max_tokens < 100:
        # Short output: overhead not worth it
        return standard_llm.generate(prompt, max_tokens)
    else:
        # Long output: speculative wins
        return speculative_llm.generate(prompt, max_tokens)

Deploy with Kubernetes platform engineering and observability for acceptance rate tracking.

When Speculative Decoding Fails

Failure Mode 1: Low Acceptance Rate (<40%)

Symptoms: Speedup <1.5x, higher latency than standard.

Causes:

• Draft model from different family/domain than target
• Highly creative/unpredictable outputs
• Draft model too small (< 0.5B)

Solution:

# Use larger, more aligned draft model
speculative_model="meta-llama/Llama-3.2-3B-Instruct"  # Instead of 1B

# Or reduce speculation depth
num_speculative_tokens=3  # Instead of 5

Failure Mode 2: Memory Overhead

Symptoms: OOM errors, reduced batch size.

Cause: Draft + target models compete for VRAM.

Solution:

# Place draft on separate GPU
draft_model = LLM(
    model="meta-llama/Llama-3.2-1B",
    device="cuda:0",  # Dedicated GPU
)

target_model = LLM(
    model="meta-llama/Llama-3.1-70B",
    device="cuda:1,2,3,4",  # Rest of GPUs
    speculative_model=draft_model,
)

Failure Mode 3: Cold Start Latency

Symptoms: First request slow (loading two models).

Solution:

# Warm up both models
lifecycle:
  postStart:
    exec:
      command:
      - python3
      - -c
      - |
        import requests
        requests.post("http://localhost:8000/v1/completions", json={
            "model": "meta-llama/Llama-3.1-70B-Instruct",
            "prompt": "Hi",
            "max_tokens": 1,
        })

For LLM serving infrastructure, monitor acceptance rate and disable speculation if <40%.

Primary references: official documentation, official documentation, official documentation, official documentation.

Operating Speculative Decoding with Draft Models as a System

The implementation is only one part of Speculative Decoding with Draft Models. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship Speculative Decoding with Draft Models behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our Speculative Decoding with Draft Models engineering support.

Frequently Asked Questions

What is speculative decoding in simple terms?

Speculative decoding uses a small, fast model to propose multiple tokens at once, which a large, accurate model verifies in parallel — achieving 2-3x faster inference without changing output quality.

Does speculative decoding change the output?

No. Outputs are mathematically identical to standard decoding — same probability distribution, same sampling behavior. Zero quality loss.

How much faster is speculative decoding?

Typically 2-3x faster for 70B+ models with good draft models. Up to 4x on highly predictable tasks (code, structured output), down to 1.5x on creative/unpredictable text.

What's the best draft model for Llama 3.1 70B?

Llama 3.2 3B achieves ~75% acceptance rate and 2.7x speedup. For lower VRAM, Llama 3.2 1B achieves ~65% acceptance and 2.3x speedup.

Can I use speculative decoding with any model?

Yes, as long as you have a compatible draft model (same vocabulary, similar architecture). Works best with same-family models (Llama → Llama, Mistral → Mistral).

Does speculative decoding work with quantized models?

Yes. Both target and draft can be quantized independently. Example: AWQ 4-bit 70B + FP16 3B draft.

When should I NOT use speculative decoding?

• Short outputs (<50 tokens) — overhead outweighs speedup
• Very high acceptance required (>95%) — use larger draft, but cost increases
• No good draft model available for your target

How does speculative decoding compare to other speedup methods?

Combine methods: quantized target + draft + batching = 5-8x total speedup.

Conclusion

Speculative decoding is the highest-ROI LLM inference optimization for large models in 2026. With 2-3x speedup, zero quality loss, and no model training required, it should be enabled by default for any 70B+ production deployment.

The deployment playbook:

1. Select draft model from same family as target (3B for 70B target)
2. Enable in vLLM with --speculative-model and num_speculative_tokens=5
3. Monitor acceptance rate — target 60-80% for optimal speedup
4. Tune K dynamically based on observed acceptance
5. Disable for short outputs (<50 tokens) where overhead dominates

At HinterBuild, we optimize LLM inference for production workloads:

• RAG & LLM Systems
• AI Agent Development
• Kubernetes Platform Engineering
• Cloud Infrastructure & DevOps

Contact us to implement speculative decoding for your serving infrastructure.