HinterBuild logoHinterBuild
Back to blog
AI Systems · 14 min read

Benchmark LLM Serving: TTFT, TPOT, and Throughput Guide

How to benchmark LLM serving properly: measure TTFT, TPOT, and throughput under concurrent load, avoid cold-start and percentile mistakes, and set SLOs.

Muhammad Abdul Sami, author

Muhammad Abdul Sami

· 14 min read

  • LLM Serving
  • vLLM
  • Performance
  • Testing
  • Observability

Table of Contents:

Key LLM Serving Metrics

Short answer: To benchmark LLM serving you measure three things: TTFT (Time to First Token — prefill latency), TPOT (Time Per Output Token — generation speed), and throughput (requests per second at your latency target). All three must be measured together, under realistic concurrent load, because optimizing one usually moves the others.

After benchmarking dozens of LLM deployments at HinterBuild, the pattern is clear: synthetic single-request benchmarks lie. Real-world performance requires concurrent load testing at your target QPS.

Key Takeaways:

  • TTFT: Measures prefill latency (prompt processing) — target <500ms for interactive apps
  • TPOT: Measures generation speed (ms/token) — target <50ms for streaming
  • Throughput: Requests/second at p95 latency target — determines infrastructure cost
  • Must test: Concurrent load (10-100+ simultaneous requests), mixed prompt/output lengths
  • Tools: vLLM benchmarks, Locust, k6, custom async harness

For LLM serving infrastructure, benchmarking validates optimizations like quantization, speculative decoding, and batching.

Why TTFT and TPOT Behave Differently Under Load

The two latency metrics come from two different phases of inference, and they hit different hardware limits:

MetricInference phaseBottleneckGets worse when
TTFTPrefill (process the whole prompt in one forward pass)GPU compute (FLOPs)Prompts get longer, or prefill of new requests is queued behind decode steps
TPOTDecode (one token per forward pass, reading the full KV cache)GPU memory bandwidthBatch gets larger, KV cache grows, or memory is fragmented
ThroughputBoth, interleaved by the schedulerWhichever phase saturates firstBatching is disabled or KV cache runs out

Prefill is compute-bound: a 2,000-token prompt costs roughly 4x the FLOPs of a 500-token prompt, so TTFT scales with prompt length. Decode is memory-bandwidth-bound: each step reads every layer's weights and the KV cache for every sequence in the batch, so TPOT depends on batch size and context length rather than on how much work each token does.

This is why continuous batching and PagedAttention, described in the vLLM paper, raise throughput dramatically while barely touching TPOT: they fill the memory-bandwidth-bound decode step with more sequences. It is also why a benchmark that only sends one request at a time tells you almost nothing about production capacity.

Benchmarking Methodology

1. Define Your Workload Profile

python
from dataclasses import dataclass
from typing import Literal

@dataclass
class WorkloadProfile:
    """Production workload characteristics."""
    target_qps: float                    # Queries per second
    concurrency: int                     # Simultaneous requests
    
    # Request characteristics
    avg_prompt_tokens: int
    prompt_token_stddev: int
    avg_output_tokens: int
    output_token_stddev: int
    
    # SLO targets
    ttft_p95_ms: int                     # Time to first token
    tpot_p95_ms: int                     # Time per output token
    error_rate_max: float                # Max error rate (%)

# Example: Interactive chat application
chat_workload = WorkloadProfile(
    target_qps=50.0,
    concurrency=100,
    avg_prompt_tokens=512,
    prompt_token_stddev=256,
    avg_output_tokens=256,
    output_token_stddev=128,
    ttft_p95_ms=500,
    tpot_p95_ms=50,
    error_rate_max=0.01,  # 1%
)

# Example: Batch document processing
batch_workload = WorkloadProfile(
    target_qps=5.0,
    concurrency=50,
    avg_prompt_tokens=2048,
    prompt_token_stddev=512,
    avg_output_tokens=512,
    output_token_stddev=256,
    ttft_p95_ms=2000,
    tpot_p95_ms=100,
    error_rate_max=0.001,  # 0.1%
)

2. Generate Realistic Test Data

python
import numpy as np
from datasets import load_dataset

class BenchmarkDataGenerator:
    """Generate realistic prompts for load testing."""
    
    def __init__(self, profile: WorkloadProfile):
        self.profile = profile
        # Use real datasets for prompt diversity
        self.dataset = load_dataset("tatsu-lab/alpaca", split="train")
    
    def generate_prompts(self, num_prompts: int) -> list[dict]:
        """Generate prompts matching workload profile."""
        prompts = []
        
        for _ in range(num_prompts):
            # Sample from dataset
            sample = self.dataset[np.random.randint(len(self.dataset))]
            base_prompt = sample["instruction"]
            
            # Adjust length to match profile
            target_tokens = int(np.random.normal(
                self.profile.avg_prompt_tokens,
                self.profile.prompt_token_stddev,
            ))
            target_tokens = max(50, min(target_tokens, 4096))
            
            # Truncate or pad
            words = base_prompt.split()
            if len(words) > target_tokens:
                prompt = " ".join(words[:target_tokens])
            else:
                # Pad with context if needed
                prompt = base_prompt
            
            # Target output length
            output_tokens = int(np.random.normal(
                self.profile.avg_output_tokens,
                self.profile.output_token_stddev,
            ))
            output_tokens = max(10, min(output_tokens, 2048))
            
            prompts.append({
                "prompt": prompt,
                "max_tokens": output_tokens,
            })
        
        return prompts

# Generate 1000 test prompts
generator = BenchmarkDataGenerator(chat_workload)
test_prompts = generator.generate_prompts(1000)

3. Warm-Up Phase

python
async def warmup(
    client: httpx.AsyncClient,
    base_url: str,
    num_requests: int = 10,
) -> None:
    """Warm up model before benchmarking."""
    print("Warming up model...")
    
    for i in range(num_requests):
        await client.post(
            f"{base_url}/v1/completions",
            json={
                "model": "meta-llama/Llama-3.1-70B-Instruct",
                "prompt": "Hello, world!",
                "max_tokens": 10,
            },
            timeout=30.0,
        )
    
    print(f"Warmup complete ({num_requests} requests)")

For production AI systems, proper warmup prevents skewed results from cold starts.

4. Control the Variables

A benchmark is a comparison, so decide up front what changes between runs and hold everything else fixed. Vary one of: model or quantization, serving engine, GPU type, tensor-parallel degree, or scheduler settings such as max_num_seqs and gpu_memory_utilization. Keep the same prompt set (seeded), the same request-rate schedule, the same max_tokens distribution, and the same client machine.

Record the full engine configuration alongside every result. Six weeks later, "awq-4bit" with no other detail is useless; "awq-4bit, TP=4, max_num_seqs=256, vLLM 0.6.x, A100-80GB" is reproducible.

Tools and Frameworks

Choosing a Load Generator

ToolMeasures TTFT/TPOT nativelyRequest-rate controlBest for
vLLM benchmark_serving.pyYesPoisson arrivals via --request-rateEngine-to-engine comparisons with OpenAI-compatible servers
LocustWith custom streaming codeUsers x wait timeRealistic user simulations, live dashboard, distributed load
k6With custom streaming codePrecise arrival-rate executorsCI gates and SLO checks against a deployed endpoint
Custom async harnessWhatever you implementExactReproducing your production traffic shape

The vLLM benchmark scripts are the fastest way to get standard numbers, and recent releases expose the same functionality as vllm bench serve. Locust and k6 are better when you need to model user behaviour or run the benchmark as a CI job. The custom harness below is what we use when the production request mix (system prompt sizes, tool-call outputs, long tails) does not fit a synthetic dataset.

vLLM Built-in Benchmarks

bash
# Install vLLM
pip install vllm

# Benchmark throughput
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --quantization awq \
  --tensor-parallel-size 4 &

# Wait for server to start, then benchmark
python benchmarks/benchmark_serving.py \
  --backend vllm \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --dataset-name random \
  --num-prompts 1000 \
  --request-rate 10 \
  --output-len 256

Output:

Benchmark Results:
  Successful requests: 1000
  Failed requests: 0
  Throughput (req/s): 47.3
  TTFT p50 (ms): 142
  TTFT p95 (ms): 389
  TTFT p99 (ms): 612
  TPOT p50 (ms): 18
  TPOT p95 (ms): 34
  TPOT p99 (ms): 52

Locust Load Testing

python
# locustfile.py
from locust import HttpUser, task, between
import json
import time

class VLLMUser(HttpUser):
    wait_time = between(0.1, 2.0)
    
    @task
    def generate_completion(self):
        """Send completion request."""
        start = time.time()
        
        with self.client.post(
            "/v1/completions",
            json={
                "model": "meta-llama/Llama-3.1-70B-Instruct",
                "prompt": "Explain quantum computing",
                "max_tokens": 256,
                "temperature": 0.7,
                "stream": True,
            },
            catch_response=True,
            stream=True,
        ) as response:
            if response.status_code != 200:
                response.failure(f"Failed: {response.status_code}")
                return
            
            # Measure TTFT
            first_token_time = None
            token_count = 0
            
            for line in response.iter_lines():
                if not line:
                    continue
                
                if first_token_time is None:
                    first_token_time = time.time() - start
                    self.environment.events.request.fire(
                        request_type="TTFT",
                        name="/v1/completions",
                        response_time=first_token_time * 1000,
                        response_length=0,
                        exception=None,
                        context={},
                    )
                
                token_count += 1
            
            total_time = time.time() - start
            tpot = (total_time - first_token_time) / max(token_count - 1, 1)
            
            response.success()

# Run benchmark
# locust -f locustfile.py --host http://vllm-server:8000 --users 100 --spawn-rate 10

Custom Async Benchmark Harness

python
import asyncio
import httpx
import time
import numpy as np
from dataclasses import dataclass, field

@dataclass
class BenchmarkResult:
    """Single request benchmark result."""
    
    prompt_tokens: int
    output_tokens: int
    ttft_ms: float              # Time to first token
    total_latency_ms: float
    tpot_ms: float              # Time per output token
    success: bool
    error: str = ""

@dataclass
class AggregateResults:
    """Aggregated benchmark results."""
    
    total_requests: int
    successful_requests: int
    failed_requests: int
    
    throughput_qps: float
    
    ttft_p50: float
    ttft_p95: float
    ttft_p99: float
    
    tpot_p50: float
    tpot_p95: float
    tpot_p99: float
    
    total_latency_p50: float
    total_latency_p95: float
    total_latency_p99: float

async def benchmark_request(
    client: httpx.AsyncClient,
    base_url: str,
    prompt: str,
    max_tokens: int,
) -> BenchmarkResult:
    """Benchmark single request."""
    
    start = time.time()
    first_token_time = None
    token_count = 0
    
    try:
        async with client.stream(
            "POST",
            f"{base_url}/v1/completions",
            json={
                "model": "meta-llama/Llama-3.1-70B-Instruct",
                "prompt": prompt,
                "max_tokens": max_tokens,
                "stream": True,
            },
            timeout=60.0,
        ) as response:
            if response.status_code != 200:
                return BenchmarkResult(
                    prompt_tokens=len(prompt.split()),
                    output_tokens=0,
                    ttft_ms=0,
                    total_latency_ms=0,
                    tpot_ms=0,
                    success=False,
                    error=f"HTTP {response.status_code}",
                )
            
            async for line in response.aiter_lines():
                if not line.strip():
                    continue
                
                if first_token_time is None:
                    first_token_time = time.time()
                
                token_count += 1
        
        total_time = time.time() - start
        ttft = (first_token_time - start) * 1000 if first_token_time else 0
        generation_time = (time.time() - first_token_time) if first_token_time else 0
        tpot = (generation_time / max(token_count - 1, 1)) * 1000
        
        return BenchmarkResult(
            prompt_tokens=len(prompt.split()),
            output_tokens=token_count,
            ttft_ms=ttft,
            total_latency_ms=total_time * 1000,
            tpot_ms=tpot,
            success=True,
        )
    
    except Exception as e:
        return BenchmarkResult(
            prompt_tokens=len(prompt.split()),
            output_tokens=0,
            ttft_ms=0,
            total_latency_ms=0,
            tpot_ms=0,
            success=False,
            error=str(e),
        )

async def run_benchmark(
    base_url: str,
    prompts: list[dict],
    concurrency: int = 10,
    qps: float = 1.0,
) -> AggregateResults:
    """Run benchmark with controlled concurrency and QPS."""
    
    results: list[BenchmarkResult] = []
    semaphore = asyncio.Semaphore(concurrency)
    
    async def rate_limited_request(prompt_data: dict):
        async with semaphore:
            return await benchmark_request(
                client,
                base_url,
                prompt_data["prompt"],
                prompt_data["max_tokens"],
            )
    
    start_time = time.time()
    
    async with httpx.AsyncClient() as client:
        # Warmup
        await warmup(client, base_url, num_requests=5)
        
        # Benchmark with rate limiting
        interval = 1.0 / qps
        tasks = []
        
        for i, prompt_data in enumerate(prompts):
            task = asyncio.create_task(rate_limited_request(prompt_data))
            tasks.append(task)
            
            if i < len(prompts) - 1:
                await asyncio.sleep(interval)
        
        results = await asyncio.gather(*tasks)
    
    total_time = time.time() - start_time
    
    # Aggregate results
    successful = [r for r in results if r.success]
    failed = [r for r in results if not r.success]
    
    ttfts = [r.ttft_ms for r in successful]
    tpots = [r.tpot_ms for r in successful if r.output_tokens > 1]
    latencies = [r.total_latency_ms for r in successful]
    
    return AggregateResults(
        total_requests=len(results),
        successful_requests=len(successful),
        failed_requests=len(failed),
        throughput_qps=len(successful) / total_time,
        ttft_p50=np.percentile(ttfts, 50) if ttfts else 0,
        ttft_p95=np.percentile(ttfts, 95) if ttfts else 0,
        ttft_p99=np.percentile(ttfts, 99) if ttfts else 0,
        tpot_p50=np.percentile(tpots, 50) if tpots else 0,
        tpot_p95=np.percentile(tpots, 95) if tpots else 0,
        tpot_p99=np.percentile(tpots, 99) if tpots else 0,
        total_latency_p50=np.percentile(latencies, 50) if latencies else 0,
        total_latency_p95=np.percentile(latencies, 95) if latencies else 0,
        total_latency_p99=np.percentile(latencies, 99) if latencies else 0,
    )

# Run benchmark
results = asyncio.run(run_benchmark(
    base_url="http://vllm-server:8000",
    prompts=test_prompts[:1000],
    concurrency=50,
    qps=10.0,
))

print(f"""
Benchmark Results:
  Total requests: {results.total_requests}
  Successful: {results.successful_requests}
  Failed: {results.failed_requests}
  
  Throughput: {results.throughput_qps:.1f} req/s
  
  TTFT p50: {results.ttft_p50:.0f}ms
  TTFT p95: {results.ttft_p95:.0f}ms
  TTFT p99: {results.ttft_p99:.0f}ms
  
  TPOT p50: {results.tpot_p50:.0f}ms
  TPOT p95: {results.tpot_p95:.0f}ms
  TPOT p99: {results.tpot_p99:.0f}ms
  
  Total Latency p50: {results.total_latency_p50:.0f}ms
  Total Latency p95: {results.total_latency_p95:.0f}ms
  Total Latency p99: {results.total_latency_p99:.0f}ms
""")

For observability, export metrics to Prometheus during benchmarks.

vLLM Benchmarking Example

Complete Benchmark Script

python
# benchmark_vllm.py
import asyncio
import httpx
import numpy as np
from typing import Dict, List

async def benchmark_vllm_configuration(
    base_url: str,
    config_name: str,
    test_prompts: List[dict],
    concurrency: int,
    target_qps: float,
) -> Dict:
    """Benchmark a specific vLLM configuration."""
    
    print(f"\n{'='*60}")
    print(f"Benchmarking: {config_name}")
    print(f"Concurrency: {concurrency}, Target QPS: {target_qps}")
    print(f"{'='*60}\n")
    
    results = await run_benchmark(
        base_url=base_url,
        prompts=test_prompts,
        concurrency=concurrency,
        qps=target_qps,
    )
    
    # Check if SLO met
    slo_met = (
        results.ttft_p95 < 500 and
        results.tpot_p95 < 50 and
        results.failed_requests / results.total_requests < 0.01
    )
    
    return {
        "config": config_name,
        "results": results,
        "slo_met": slo_met,
    }

# Test configurations
configurations = [
    {
        "name": "baseline-fp16",
        "url": "http://vllm-fp16:8000",
        "description": "FP16, no optimizations",
    },
    {
        "name": "awq-4bit",
        "url": "http://vllm-awq:8000",
        "description": "AWQ 4-bit quantization",
    },
    {
        "name": "awq-speculative",
        "url": "http://vllm-awq-spec:8000",
        "description": "AWQ + speculative decoding",
    },
]

async def main():
    # Generate test data
    generator = BenchmarkDataGenerator(chat_workload)
    test_prompts = generator.generate_prompts(1000)
    
    # Benchmark each configuration
    benchmark_results = []
    
    for config in configurations:
        result = await benchmark_vllm_configuration(
            base_url=config["url"],
            config_name=config["name"],
            test_prompts=test_prompts,
            concurrency=50,
            target_qps=10.0,
        )
        benchmark_results.append(result)
    
    # Print comparison
    print(f"\n{'='*80}")
    print("BENCHMARK COMPARISON")
    print(f"{'='*80}\n")
    
    print(f"{'Config':<20} {'Throughput':<12} {'TTFT p95':<10} {'TPOT p95':<10} {'SLO Met':<10}")
    print(f"{'-'*80}")
    
    for result in benchmark_results:
        r = result["results"]
        print(f"{result['config']:<20} "
              f"{r.throughput_qps:>6.1f} req/s  "
              f"{r.ttft_p95:>6.0f}ms   "
              f"{r.tpot_p95:>6.0f}ms   "
              f"{'✅' if result['slo_met'] else '❌'}")

if __name__ == "__main__":
    asyncio.run(main())

Sample output:

================================================================================
BENCHMARK COMPARISON
================================================================================

Config               Throughput   TTFT p95   TPOT p95   SLO Met   
--------------------------------------------------------------------------------
baseline-fp16         21.3 req/s    687ms      58ms      ❌
awq-4bit              47.2 req/s    389ms      34ms      ✅
awq-speculative       68.1 req/s    298ms      21ms      ✅

The numbers above are an illustrative run on a single node; your ratios will differ with model, GPU, and prompt mix. What matters is the method: the same 1,000 prompts, the same request rate, and a pass/fail against the SLO for every configuration. For LLM inference optimization, this is how you prove that quantization plus speculative decoding pays for itself rather than assuming it does.

Interpreting Results

TTFT (Time to First Token)

What it measures: Prefill latency (processing prompt through model).

Typical values:

Model SizeTTFT (p95)User Experience
7-8B50-150msInstant
13B100-250msFast
70B200-500msAcceptable
70B (slow)500-1000msNoticeable delay

Optimization levers:

  • Flash Attention (20-40% improvement)
  • Tensor parallelism (near-linear scaling)
  • Prompt caching (90%+ improvement for repeated prefixes)

TPOT (Time Per Output Token)

What it measures: Generation speed (ms per token).

Typical values:

Model SizeTPOT (p95)Streaming Quality
7-8B10-30msSmooth
13B20-40msGood
70B30-60msAcceptable
70B (slow)60-100msChoppy streaming

Optimization levers:

  • Quantization (30-50% improvement)
  • Speculative decoding (50-150% improvement)
  • Continuous batching (minimal impact on TPOT, huge on throughput)

Throughput (Requests/Second)

What it measures: System capacity at target latency.

Cost impact:

python
# Calculate infrastructure cost based on throughput
def calculate_required_gpus(
    target_qps: float,
    measured_throughput_per_gpu: float,
    redundancy_factor: float = 2.0,  # For HA
) -> int:
    """Calculate required GPUs for target QPS."""
    base_gpus = target_qps / measured_throughput_per_gpu
    total_gpus = base_gpus * redundancy_factor
    return int(np.ceil(total_gpus))

# Example: 100 QPS target, 47 QPS per GPU measured
required_gpus = calculate_required_gpus(100, 47, 2.0)
print(f"Required GPUs: {required_gpus}")  # → 5 GPUs

# Cost calculation (AWS g5.2xlarge = $1.21/hr)
monthly_cost = required_gpus * 1.21 * 730
print(f"Monthly cost: ${monthly_cost:.0f}")  # → $4,417/month

For cloud infrastructure planning, throughput determines infrastructure budget.

Reading the Load Curve

A single throughput number hides the most useful information. Run the benchmark at increasing request rates (for example 1, 2, 5, 10, 20, 40 req/s) and plot p95 TTFT and p95 TPOT against rate. You will see a knee: latency stays flat while the scheduler has spare capacity, then climbs steeply once the KV cache fills or prefill queues behind decode.

Your usable capacity is the request rate just below the knee while still inside the SLO, not the peak rate the server survives. The DistServe paper calls this goodput — requests per second that meet both TTFT and TPOT targets — and it is the number to put in capacity plans. A server doing 60 req/s with half of them breaching TTFT is a 30 req/s server.

Two patterns to look for on the curve:

  • TTFT climbs but TPOT stays flat. Prefill is queueing. Options: chunked prefill, more tensor parallelism, or prefix caching for shared system prompts.
  • TPOT climbs first. Decode batches are too large for the memory bandwidth. Options: cap max_num_seqs, quantize the KV cache, or add replicas rather than bigger batches.

Common Pitfalls

Pitfall 1: Cold Start Bias

python
# ❌ Wrong: First request is slow (cold start)
results = await benchmark_single_request()

# ✅ Right: Warmup phase before measuring
await warmup(client, base_url, num_requests=10)
results = await benchmark_requests()

Pitfall 2: Unrealistic Single-Request Testing

python
# ❌ Wrong: Single request (doesn't test batching)
for _ in range(100):
    await client.post("/v1/completions", json=...)

# ✅ Right: Concurrent load (tests batching)
tasks = [client.post("/v1/completions", json=...) for _ in range(100)]
await asyncio.gather(*tasks)

Pitfall 3: Ignoring Percentiles

python
# ❌ Wrong: Only average latency
avg_latency = sum(latencies) / len(latencies)

# ✅ Right: p50, p95, p99
p50 = np.percentile(latencies, 50)
p95 = np.percentile(latencies, 95)
p99 = np.percentile(latencies, 99)
# SLO based on p95, not average

Pitfall 4: Short Test Duration

python
# ❌ Wrong: 10 requests (not statistically significant)
results = await benchmark(num_requests=10)

# ✅ Right: 1000+ requests (captures variance)
results = await benchmark(num_requests=1000)

Pitfall 5: Fixed Prompt/Output Lengths

python
# ❌ Wrong: All prompts same length
prompts = ["Explain AI" for _ in range(1000)]

# ✅ Right: Realistic length distribution
prompts = generate_prompts_with_distribution(
    avg_tokens=512,
    stddev=256,
)

Pitfall 6: Counting Chunks Instead of Tokens

The streaming harness above counts SSE lines, and most servers emit one token per chunk, but not all do. Some engines coalesce tokens under load, and tool-call or JSON-mode outputs can arrive in larger pieces. When comparing engines, compute output tokens from the usage field in the final chunk or re-tokenize the completed text; otherwise TPOT is inflated for the engine that batches chunks.

Pitfall 7: Benchmarking the Client

At high concurrency the load generator itself becomes the bottleneck: a single Python process parsing thousands of SSE streams per second will report latency that is really its own event-loop backlog. Watch client CPU during the run, spread load across processes or machines, and confirm that server-side metrics (vLLM exposes Prometheus histograms for TTFT and TPOT) agree with client-side numbers. A widening gap between the two is the signature of a saturated client.

Choosing between engines? The Triton vs vLLM comparison covers how the same methodology applies across serving frameworks.

Production SLOs

SLO Examples by Application Type

ApplicationTTFT p95TPOT p95ThroughputError Rate
Interactive chat<500ms<50ms10-50 req/s<1%
Code completion<200ms<30ms50-200 req/s<0.5%
Batch processing<2000ms<100ms5-20 req/s<0.1%
Search augmentation<1000ms<50ms20-100 req/s<1%

SLO Monitoring

yaml
# Prometheus alerting rules
groups:
- name: llm_serving_slo
  rules:
  - alert: TTFTHighLatency
    expr: |
      histogram_quantile(0.95, 
        rate(vllm_time_to_first_token_seconds_bucket[5m])
      ) > 0.5
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "TTFT p95 > 500ms (SLO violation)"
  
  - alert: ThroughputLow
    expr: rate(vllm_request_success_total[5m]) < 40
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Throughput < 40 req/s (below capacity target)"
  
  - alert: ErrorRateHigh
    expr: |
      rate(vllm_request_failure_total[5m]) / 
      rate(vllm_request_success_total[5m]) > 0.01
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Error rate > 1% (SLO violation)"

Deploy observability infrastructure to track SLOs continuously. Alert on burn rate rather than instantaneous breaches: an SLO of 99% of requests under 500 ms TTFT gives you a monthly error budget, and a fast-burn alert (consuming 2% of the budget in an hour) catches regressions without paging on every latency blip. The Prometheus histogram_quantile documentation explains why percentiles must be computed from bucketed histograms, not averaged across replicas.

Finally, re-run the benchmark whenever the model, engine version, or GPU type changes. Serving engines ship scheduler changes frequently, and a version bump that shifts the knee by 20% is not unusual.

Frequently Asked Questions

What is TTFT in LLM serving?

TTFT (Time to First Token) is the time from sending a request to receiving the first generated token, which corresponds to the prefill phase where the model processes the entire prompt. It dominates perceived responsiveness in chat interfaces. TTFT grows with prompt length and with queueing when the server is saturated.

What is a good TTFT for production?

For interactive chat, under 500 ms at p95 is a common target; code completion typically needs under 200 ms. Batch workloads can tolerate seconds. Reduce TTFT with prefix caching for shared system prompts, tensor parallelism, and chunked prefill so long prompts do not block other requests.

What is TPOT and how does it relate to tokens per second?

TPOT (Time Per Output Token) is the average milliseconds between consecutive generated tokens during the decode phase. Tokens per second for a single stream is simply 1000 / TPOT, so 40 ms TPOT is 25 tokens per second. For smooth streaming, keep p95 TPOT under 50 ms, roughly faster than a person reads.

How do I measure LLM throughput correctly?

Send concurrent requests at a controlled arrival rate, measure how many complete per second, and report only the rate at which p95 TTFT and TPOT still meet your SLO. That figure, sometimes called goodput, is the one that drives GPU capacity planning. Single-request loops understate throughput because they never exercise batching.

Should I optimize for TTFT or throughput?

Interactive applications should optimize TTFT first, then throughput within the latency budget. Offline or batch workloads should maximize throughput and accept higher latency. Most production systems land in between and use continuous batching with a capped batch size to balance the two.

What concurrency should I use when load testing an LLM?

Match your expected peak concurrent requests, then push 1.5-2x beyond it to find the knee in the latency curve. If you expect 50 simultaneous users, test at 50, 75, and 100. Sweep request rates rather than fixing one value so you can see where the SLO breaks.

How long should an LLM benchmark run?

Use at least 1,000 requests or 5-10 minutes per configuration so percentiles are stable, always after a warm-up phase. For production sign-off, add a one-hour soak test to catch memory growth, KV-cache fragmentation, and thermal throttling that short runs miss.

What is the difference between cold start and warm latency?

Cold-start latency includes model loading, CUDA graph capture, and cache population, and can be tens of seconds for large models. Warm latency is the steady-state figure after a few requests have run. Benchmarks should report warm latency, while deployment planning should separately account for cold-start time in autoscaling decisions.

Conclusion

Benchmarking LLM serving requires measuring TTFT, TPOT, and throughput under realistic concurrent load. Proper methodology — warmup, realistic prompts, 1000+ requests, percentile metrics — separates production-ready systems from broken ones.

The benchmarking playbook:

  1. Define workload profile (target QPS, prompt/output lengths)
  2. Generate realistic test data from production distributions
  3. Warmup before measuring (5-10 requests)
  4. Test with concurrency matching expected peak load
  5. Measure percentiles (p50, p95, p99), not averages
  6. Run 1000+ requests for statistical significance
  7. Set SLO alerts in production monitoring

At HinterBuild, we benchmark and optimize LLM serving systems:

Contact us to benchmark and optimize your LLM serving infrastructure.

Free consultation

Book a free consultation call on LLM performance benchmarking

30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.

Book a meeting

Resources:

Keep reading

Related articles

AI Systems · 9 min read

Batch API for LLM Workloads: 50% Cost Savings on

Learn batch api for llm workloads through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

AI Systems · 12 min read

Streaming LLM Responses in Production

Streaming LLM Responses in Production guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable.

Read post

AI Systems · 10 min read

Triton vs vLLM: LLM Serving Framework Comparison for

Triton vs vLLM guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Read post

AI Systems · 11 min read

Webhook Design for AI Pipelines: Reliability Patterns for

Build reliable webhook systems for AI pipelines with retry logic, idempotency, and validation. Production patterns from processing 50K+ AI webhooks daily.

Read post