HinterBuild logoHinterBuild
Back to blog
AI Systems · 9 min read

LLM Tracing with OpenTelemetry: Complete Observability Guide

Learn llm tracing with opentelemetry through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Muhammad Abdul Sami, author

Muhammad Abdul Sami

· 9 min read

  • LLM
  • LLM Serving
  • Evaluation
  • Cost Optimization

Table of Contents:

Why OpenTelemetry for LLMs: The Observability Gap

Short answer: LLM applications are black boxes without tracing. OpenTelemetry provides standardized instrumentation for distributed LLM calls, enabling performance debugging, cost tracking, and quality monitoring in production.

A fintech company's AI agent had P95 latency of 8 seconds. With no tracing, debugging was guesswork. We instrumented their system with OpenTelemetry—discovered 75% of latency came from inefficient embedding calls, not LLM inference. After optimization: P95 dropped to 2.1 seconds.

Key Takeaways:

  • OpenTelemetry provides vendor-neutral observability for LLM systems
  • Distributed tracing tracks requests across LLM → retrieval → tool calls
  • Span attributes capture prompts, tokens, costs, and quality metrics
  • Automatic instrumentation works with OpenAI, Anthropic, LangChain
  • Metrics exporters send data to Datadog, Prometheus, Grafana
  • Log correlation connects traces to application logs

For production AI systems, OpenTelemetry is the standard for observability.

Architecture and Setup

OpenTelemetry architecture for LLM applications.

python
"""
pip install opentelemetry-api
pip install opentelemetry-sdk
pip install opentelemetry-exporter-otlp
pip install opentelemetry-instrumentation-openai
pip install opentelemetry-instrumentation-anthropic
"""

# Setup OpenTelemetry
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource

# Configure resource
resource = Resource.create({
    "service.name": "ai-agent-system",
    "service.version": "1.0.0",
    "deployment.environment": "production",
})

# Setup tracer provider
tracer_provider = TracerProvider(resource=resource)

# Add OTLP span exporter (sends to collector)
otlp_exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # OpenTelemetry Collector
    insecure=True,
)

tracer_provider.add_span_processor(
    BatchSpanProcessor(otlp_exporter)
)

trace.set_tracer_provider(tracer_provider)

# Setup meter provider
metric_reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint="http://localhost:4317", insecure=True)
)

meter_provider = MeterProvider(
    resource=resource,
    metric_readers=[metric_reader],
)

metrics.set_meter_provider(meter_provider)

# Get tracer and meter
tracer = trace.get_tracer("ai.agent", "1.0.0")
meter = metrics.get_meter("ai.agent", "1.0.0")

# Create metrics
llm_request_counter = meter.create_counter(
    "llm.requests",
    description="Number of LLM requests",
    unit="requests",
)

llm_latency_histogram = meter.create_histogram(
    "llm.latency",
    description="LLM request latency",
    unit="ms",
)

llm_token_counter = meter.create_counter(
    "llm.tokens",
    description="Token usage",
    unit="tokens",
)

llm_cost_counter = meter.create_counter(
    "llm.cost",
    description="LLM cost in USD",
    unit="usd",
)

OpenTelemetry Collector receives traces and forwards to backends (Datadog, Jaeger, etc.).

yaml
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  # Export to Datadog
  datadog:
    api:
      key: ${DATADOG_API_KEY}
  
  # Export to Jaeger (for local dev)
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog, jaeger]
    
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]

Start collector:

bash
docker run -p 4317:4317 -p 4318:4318 \
  -v $(pwd)/otel-collector-config.yaml:/etc/otel/config.yaml \
  otel/opentelemetry-collector-contrib:latest \
  --config /etc/otel/config.yaml

For cloud infrastructure, deploy collector as sidecar or DaemonSet.

LLM Span Instrumentation

Instrument LLM calls with detailed span attributes.

python
from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode
from opentelemetry.semconv.trace import SpanAttributes
from openai import AsyncOpenAI
import time
from typing import Optional, Dict, Any

client = AsyncOpenAI()

async def traced_llm_call(
    prompt: str,
    model: str = "gpt-4o",
    context: Optional[str] = None,
    metadata: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
    """LLM call with OpenTelemetry tracing."""
    tracer = trace.get_tracer("ai.llm")
    
    # Start span
    with tracer.start_as_current_span(
        "llm.completion",
        kind=SpanKind.CLIENT,
    ) as span:
        # Add standard attributes
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.prompt.length", len(prompt))
        
        if context:
            span.set_attribute("llm.context.length", len(context))
        
        # Add custom metadata
        if metadata:
            for key, value in metadata.items():
                span.set_attribute(f"llm.metadata.{key}", str(value))
        
        try:
            # Make LLM call
            start_time = time.perf_counter()
            
            messages = [{"role": "user", "content": prompt}]
            if context:
                messages.insert(0, {"role": "system", "content": f"Context: {context}"})
            
            response = await client.chat.completions.create(
                model=model,
                messages=messages,
            )
            
            latency_ms = (time.perf_counter() - start_time) * 1000
            
            # Extract response data
            result_text = response.choices[0].message.content
            usage = response.usage
            
            # Add response attributes
            span.set_attribute("llm.response.length", len(result_text))
            span.set_attribute("llm.tokens.prompt", usage.prompt_tokens)
            span.set_attribute("llm.tokens.completion", usage.completion_tokens)
            span.set_attribute("llm.tokens.total", usage.total_tokens)
            span.set_attribute("llm.latency_ms", latency_ms)
            
            # Calculate cost
            cost = calculate_cost(model, usage.prompt_tokens, usage.completion_tokens)
            span.set_attribute("llm.cost_usd", cost)
            
            # Optionally log prompt/response (be careful with PII)
            if should_log_full_content():
                span.set_attribute("llm.prompt", prompt[:500])  # Truncate
                span.set_attribute("llm.response", result_text[:500])
            
            # Record metrics
            llm_request_counter.add(1, {
                "model": model,
                "status": "success",
            })
            
            llm_latency_histogram.record(latency_ms, {
                "model": model,
            })
            
            llm_token_counter.add(usage.total_tokens, {
                "model": model,
                "type": "total",
            })
            
            llm_cost_counter.add(cost, {
                "model": model,
            })
            
            # Mark span as successful
            span.set_status(Status(StatusCode.OK))
            
            return {
                "response": result_text,
                "usage": {
                    "prompt_tokens": usage.prompt_tokens,
                    "completion_tokens": usage.completion_tokens,
                    "total_tokens": usage.total_tokens,
                },
                "cost_usd": cost,
                "latency_ms": latency_ms,
            }
        
        except Exception as e:
            # Record error
            span.set_status(Status(StatusCode.ERROR, str(e)))
            span.record_exception(e)
            
            llm_request_counter.add(1, {
                "model": model,
                "status": "error",
            })
            
            raise

def calculate_cost(model: str, prompt_tokens: int, completion_tokens: int) -> float:
    """Calculate LLM cost."""
    PRICING = {
        "gpt-4o": {"input": 2.50, "output": 10.00},  # per 1M tokens
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    }
    
    rates = PRICING.get(model, {"input": 0, "output": 0})
    
    return (
        (prompt_tokens / 1_000_000) * rates["input"] +
        (completion_tokens / 1_000_000) * rates["output"]
    )

def should_log_full_content() -> bool:
    """Check if full content logging is enabled."""
    import os
    return os.getenv("OTEL_LOG_FULL_CONTENT", "false").lower() == "true"

# Usage
result = await traced_llm_call(
    prompt="Explain quantum computing",
    model="gpt-4o",
    metadata={
        "user_id": "user-123",
        "session_id": "sess-456",
        "request_type": "qa",
    },
)

print(f"Response: {result['response']}")
print(f"Latency: {result['latency_ms']:.0f}ms")
print(f"Cost: ${result['cost_usd']:.4f}")

Span attributes enable filtering and analysis in observability tools.

For agent tracing, track agent reasoning steps.

Distributed Tracing

Trace multi-step workflows across services.

python
from opentelemetry import trace, context
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
import asyncio

class TracedAIAgent:
    """AI agent with distributed tracing."""
    
    def __init__(self):
        self.tracer = trace.get_tracer("ai.agent")
    
    async def handle_request(
        self,
        query: str,
        user_id: str,
    ) -> Dict[str, Any]:
        """Handle user request with full tracing."""
        with self.tracer.start_as_current_span(
            "agent.request",
            kind=SpanKind.SERVER,
            attributes={
                "user.id": user_id,
                "query.length": len(query),
            },
        ) as span:
            # Step 1: Classify intent
            intent = await self._classify_intent(query)
            span.set_attribute("agent.intent", intent)
            
            # Step 2: Retrieve context
            context_docs = await self._retrieve_context(query, intent)
            span.set_attribute("agent.context_docs", len(context_docs))
            
            # Step 3: Generate response
            response = await self._generate_response(query, context_docs)
            span.set_attribute("agent.response.length", len(response))
            
            # Step 4: Validate response
            validation = await self._validate_response(response, context_docs)
            span.set_attribute("agent.validation.passed", validation["passed"])
            
            if not validation["passed"]:
                span.set_status(Status(StatusCode.ERROR, "Validation failed"))
                response = "I'm sorry, I couldn't generate a reliable response."
            
            return {
                "response": response,
                "intent": intent,
                "validation": validation,
            }
    
    async def _classify_intent(self, query: str) -> str:
        """Classify user intent."""
        with self.tracer.start_as_current_span(
            "agent.classify_intent",
            attributes={"query": query[:100]},
        ) as span:
            result = await traced_llm_call(
                prompt=f"Classify intent: {query}\nReturn: question|request|feedback",
                model="gpt-4o-mini",
                metadata={"step": "intent_classification"},
            )
            
            intent = result["response"].strip().lower()
            span.set_attribute("intent.classified", intent)
            
            return intent
    
    async def _retrieve_context(
        self,
        query: str,
        intent: str,
    ) -> list:
        """Retrieve relevant context."""
        with self.tracer.start_as_current_span(
            "agent.retrieve_context",
            attributes={
                "query": query[:100],
                "intent": intent,
            },
        ) as span:
            # Embed query
            embedding = await self._embed_query(query)
            span.set_attribute("embedding.dimensions", len(embedding))
            
            # Search vector DB
            docs = await self._search_vector_db(embedding, top_k=5)
            span.set_attribute("docs.retrieved", len(docs))
            
            return docs
    
    async def _embed_query(self, query: str) -> list:
        """Embed query."""
        with self.tracer.start_as_current_span(
            "agent.embed",
            attributes={"query.length": len(query)},
        ) as span:
            import time
            start = time.perf_counter()
            
            response = await client.embeddings.create(
                model="text-embedding-3-small",
                input=query,
            )
            
            latency_ms = (time.perf_counter() - start) * 1000
            span.set_attribute("embedding.latency_ms", latency_ms)
            
            return response.data[0].embedding
    
    async def _search_vector_db(
        self,
        embedding: list,
        top_k: int = 5,
    ) -> list:
        """Search vector database."""
        with self.tracer.start_as_current_span(
            "agent.vector_search",
            attributes={
                "embedding.dimensions": len(embedding),
                "top_k": top_k,
            },
        ) as span:
            # Simulate vector search
            await asyncio.sleep(0.05)
            
            # In production: actual vector DB query
            docs = ["doc1", "doc2", "doc3"]
            
            span.set_attribute("results.count", len(docs))
            
            return docs
    
    async def _generate_response(
        self,
        query: str,
        context_docs: list,
    ) -> str:
        """Generate response with LLM."""
        with self.tracer.start_as_current_span(
            "agent.generate_response",
            attributes={
                "context.docs": len(context_docs),
            },
        ) as span:
            context = "\n".join(context_docs)
            
            result = await traced_llm_call(
                prompt=query,
                context=context,
                model="gpt-4o",
                metadata={"step": "response_generation"},
            )
            
            return result["response"]
    
    async def _validate_response(
        self,
        response: str,
        context_docs: list,
    ) -> Dict[str, Any]:
        """Validate response quality."""
        with self.tracer.start_as_current_span(
            "agent.validate_response",
            attributes={
                "response.length": len(response),
            },
        ) as span:
            # Check grounding
            context = "\n".join(context_docs)
            
            validation_result = await traced_llm_call(
                prompt=f"""Is this response grounded in the context?

Response: {response}
Context: {context}

Return JSON: {{"grounded": true/false, "confidence": 0.0-1.0}}""",
                model="gpt-4o-mini",
                metadata={"step": "validation"},
            )
            
            import json
            validation = json.loads(validation_result["response"])
            
            span.set_attribute("validation.grounded", validation["grounded"])
            span.set_attribute("validation.confidence", validation["confidence"])
            
            return {
                "passed": validation["grounded"] and validation["confidence"] > 0.7,
                "grounded": validation["grounded"],
                "confidence": validation["confidence"],
            }

# Usage
agent = TracedAIAgent()

result = await agent.handle_request(
    query="What is our refund policy?",
    user_id="user-123",
)

# Trace context propagates across all steps
# View in Jaeger/Datadog: agent.request → classify_intent → retrieve_context → generate_response → validate_response

Distributed traces show the complete request flow with timing breakdown.

For RAG systems, trace retrieval → generation → validation.

Metrics Collection

Collect production metrics for dashboards and alerts.

python
from opentelemetry import metrics
from typing import Optional
import time

class LLMMetricsCollector:
    """Collect LLM metrics."""
    
    def __init__(self):
        meter = metrics.get_meter("llm.metrics")
        
        # Counters
        self.request_counter = meter.create_counter(
            "llm.requests.total",
            description="Total LLM requests",
            unit="requests",
        )
        
        self.token_counter = meter.create_counter(
            "llm.tokens.total",
            description="Total tokens processed",
            unit="tokens",
        )
        
        self.cost_counter = meter.create_counter(
            "llm.cost.total",
            description="Total LLM cost",
            unit="usd",
        )
        
        self.error_counter = meter.create_counter(
            "llm.errors.total",
            description="Total LLM errors",
            unit="errors",
        )
        
        # Histograms
        self.latency_histogram = meter.create_histogram(
            "llm.latency",
            description="LLM request latency",
            unit="ms",
        )
        
        self.token_histogram = meter.create_histogram(
            "llm.tokens.per_request",
            description="Tokens per request",
            unit="tokens",
        )
        
        # Gauges (via observable)
        self.active_requests = 0
        
        def get_active_requests(options):
            return [(self.active_requests, {"service": "llm"})]
        
        meter.create_observable_gauge(
            "llm.requests.active",
            callbacks=[get_active_requests],
            description="Active LLM requests",
            unit="requests",
        )
    
    async def track_request(
        self,
        model: str,
        prompt_tokens: int,
        completion_tokens: int,
        latency_ms: float,
        cost_usd: float,
        status: str = "success",
        metadata: Optional[Dict[str, str]] = None,
    ) -> None:
        """Track LLM request metrics."""
        attributes = {
            "model": model,
            "status": status,
        }
        
        if metadata:
            attributes.update(metadata)
        
        # Record metrics
        self.request_counter.add(1, attributes)
        
        total_tokens = prompt_tokens + completion_tokens
        self.token_counter.add(total_tokens, {**attributes, "type": "total"})
        self.token_counter.add(prompt_tokens, {**attributes, "type": "prompt"})
        self.token_counter.add(completion_tokens, {**attributes, "type": "completion"})
        
        self.cost_counter.add(cost_usd, attributes)
        
        self.latency_histogram.record(latency_ms, attributes)
        self.token_histogram.record(total_tokens, attributes)
        
        if status == "error":
            self.error_counter.add(1, attributes)
    
    def increment_active(self):
        """Increment active requests."""
        self.active_requests += 1
    
    def decrement_active(self):
        """Decrement active requests."""
        self.active_requests -= 1

# Usage
metrics_collector = LLMMetricsCollector()

async def monitored_llm_call(prompt: str, model: str = "gpt-4o"):
    """LLM call with metrics collection."""
    metrics_collector.increment_active()
    
    try:
        start = time.perf_counter()
        
        result = await traced_llm_call(prompt, model)
        
        latency_ms = (time.perf_counter() - start) * 1000
        
        await metrics_collector.track_request(
            model=model,
            prompt_tokens=result["usage"]["prompt_tokens"],
            completion_tokens=result["usage"]["completion_tokens"],
            latency_ms=latency_ms,
            cost_usd=result["cost_usd"],
            status="success",
            metadata={"request_type": "qa"},
        )
        
        return result
    
    except Exception as e:
        await metrics_collector.track_request(
            model=model,
            prompt_tokens=0,
            completion_tokens=0,
            latency_ms=0,
            cost_usd=0,
            status="error",
            metadata={"error_type": type(e).__name__},
        )
        raise
    
    finally:
        metrics_collector.decrement_active()

Metrics enable:

  • Real-time dashboards (Grafana)
  • Cost tracking and budgets
  • Performance alerts (Prometheus)
  • Capacity planning

For monitoring, export to Datadog or Prometheus.

Log Correlation

Correlate traces with logs for debugging.

python
from opentelemetry import trace
import logging
import json

class TraceContextFormatter(logging.Formatter):
    """Log formatter that includes trace context."""
    
    def format(self, record):
        # Get current span context
        span = trace.get_current_span()
        span_context = span.get_span_context()
        
        if span_context.is_valid:
            record.trace_id = format(span_context.trace_id, '032x')
            record.span_id = format(span_context.span_id, '016x')
        else:
            record.trace_id = '0' * 32
            record.span_id = '0' * 16
        
        return super().format(record)

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Add trace context to logs
handler = logging.StreamHandler()
handler.setFormatter(TraceContextFormatter(
    '%(asctime)s - %(name)s - %(levelname)s - [trace_id=%(trace_id)s span_id=%(span_id)s] - %(message)s'
))
logger.handlers = [handler]

# Usage in traced functions
async def traced_operation():
    """Operation with correlated logging."""
    tracer = trace.get_tracer(__name__)
    
    with tracer.start_as_current_span("operation") as span:
        logger.info("Starting operation")
        
        try:
            # Do work
            result = await do_work()
            
            logger.info(f"Operation completed: {result}")
            
            return result
        
        except Exception as e:
            logger.error(f"Operation failed: {e}", exc_info=True)
            raise

# Example log output:
# 2026-09-14 10:30:45 - __main__ - INFO - [trace_id=a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6 span_id=q7r8s9t0u1v2w3x4] - Starting operation

Log correlation enables jumping from traces to logs and vice versa.

For debugging, correlate agent decisions with traces.

Performance Debugging

Use traces to identify performance bottlenecks.

python
class PerformanceAnalyzer:
    """Analyze trace data for performance insights."""
    
    def analyze_trace(self, trace_data: Dict[str, Any]) -> Dict[str, Any]:
        """Analyze trace for bottlenecks."""
        spans = trace_data["spans"]
        
        # Calculate span durations
        span_durations = {}
        for span in spans:
            name = span["name"]
            duration_ms = span["end_time_ms"] - span["start_time_ms"]
            
            if name not in span_durations:
                span_durations[name] = []
            
            span_durations[name].append(duration_ms)
        
        # Find slowest spans
        avg_durations = {
            name: sum(durations) / len(durations)
            for name, durations in span_durations.items()
        }
        
        slowest_spans = sorted(
            avg_durations.items(),
            key=lambda x: x[1],
            reverse=True,
        )
        
        # Calculate percentage of total time
        total_time = trace_data["total_duration_ms"]
        
        breakdown = []
        for name, avg_duration in slowest_spans:
            percentage = (avg_duration / total_time) * 100
            breakdown.append({
                "operation": name,
                "avg_duration_ms": avg_duration,
                "percentage": percentage,
                "count": len(span_durations[name]),
            })
        
        return {
            "total_duration_ms": total_time,
            "breakdown": breakdown,
            "bottleneck": breakdown[0]["operation"] if breakdown else None,
        }

# Example analysis output:
"""
{
  "total_duration_ms": 1250,
  "breakdown": [
    {"operation": "agent.embed", "avg_duration_ms": 850, "percentage": 68.0, "count": 3},
    {"operation": "llm.completion", "avg_duration_ms": 300, "percentage": 24.0, "count": 2},
    {"operation": "agent.vector_search", "avg_duration_ms": 50, "percentage": 4.0, "count": 1},
    {"operation": "agent.classify_intent", "avg_duration_ms": 50, "percentage": 4.0, "count": 1}
  ],
  "bottleneck": "agent.embed"  ← 68% of total time
}
"""

Trace analysis reveals:

  • Which operations dominate latency
  • Unnecessary sequential calls (parallelize them)
  • Inefficient implementations

For performance optimization, use traces to guide improvements.

Production Deployment

Deploy OpenTelemetry in production with best practices.

python
# production_otel_setup.py
import os
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import ParentBasedTraceIdRatioBased
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

def setup_production_telemetry():
    """Setup OpenTelemetry for production."""
    # Configure resource
    resource = Resource.create({
        "service.name": os.getenv("SERVICE_NAME", "ai-agent"),
        "service.version": os.getenv("SERVICE_VERSION", "unknown"),
        "deployment.environment": os.getenv("ENV", "production"),
        "host.name": os.getenv("HOSTNAME", "unknown"),
    })
    
    # Configure sampling (sample 10% of traces)
    sampler = ParentBasedTraceIdRatioBased(0.1)
    
    # Setup tracer provider
    tracer_provider = TracerProvider(
        resource=resource,
        sampler=sampler,
    )
    
    # Add OTLP exporter
    otlp_exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
        insecure=os.getenv("OTEL_EXPORTER_OTLP_INSECURE", "true").lower() == "true",
        headers={
            "api-key": os.getenv("OTEL_API_KEY", ""),
        },
    )
    
    # Use batch processor for efficiency
    tracer_provider.add_span_processor(
        BatchSpanProcessor(
            otlp_exporter,
            max_queue_size=2048,
            max_export_batch_size=512,
            schedule_delay_millis=5000,
        )
    )
    
    trace.set_tracer_provider(tracer_provider)
    
    print("✓ OpenTelemetry configured for production")

# Call during app startup
setup_production_telemetry()

Production considerations:

  • Sampling: Sample 1-10% of traces to reduce overhead
  • Batch processing: Buffer spans before export
  • Resource attributes: Include service metadata
  • Secure endpoints: Use TLS for OTLP exporters

For Kubernetes deployment, use OpenTelemetry Operator.

Visualization and Analysis

Visualize traces in Jaeger or Datadog.

Jaeger UI shows:

  • Request waterfall (span timeline)
  • Service dependencies (service graph)
  • Error traces
  • Latency distribution

Datadog APM provides:

  • Automatic service map
  • Resource-level performance
  • Anomaly detection
  • Custom dashboards

Grafana dashboards for metrics:

json
{
  "dashboard": {
    "title": "LLM Performance",
    "panels": [
      {
        "title": "Request Rate",
        "targets": [{
          "expr": "rate(llm_requests_total[5m])"
        }]
      },
      {
        "title": "P95 Latency",
        "targets": [{
          "expr": "histogram_quantile(0.95, llm_latency_bucket)"
        }]
      },
      {
        "title": "Cost per Hour",
        "targets": [{
          "expr": "rate(llm_cost_total[1h])"
        }]
      },
      {
        "title": "Error Rate",
        "targets": [{
          "expr": "rate(llm_errors_total[5m]) / rate(llm_requests_total[5m])"
        }]
      }
    ]
  }
}

Query traces with TraceQL or Datadog syntax:

# Find slow requests
duration > 2s AND service.name = "ai-agent"

# Find expensive requests
llm.cost_usd > 0.10

# Find errors
status = error AND llm.model = "gpt-4o"

For monitoring dashboards, integrate with existing tools.

Related implementation guides:

Primary references: official documentation, official documentation, official documentation, official documentation.

LLM Tracing with OpenTelemetry Decision Table

DecisionPrefer the simpler path whenAdd operational complexity when
ArchitectureOne component can own the contract and stateIndependent scaling or fault isolation is required
RolloutOffline replay covers the meaningful casesLive behavior requires shadow traffic and a canary
RecoveryA failed operation is safe to repeatPartial effects require idempotency or reconciliation
MeasurementOne service objective represents user impactQuality, latency, and cost need separate gates

Operating LLM Tracing with OpenTelemetry as a System

The implementation is only one part of LLM Tracing with OpenTelemetry. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship LLM Tracing with OpenTelemetry behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our LLM Tracing with OpenTelemetry engineering support.

Frequently Asked Questions

What's the performance overhead of OpenTelemetry?

1-5% latency overhead with proper configuration. Use sampling (10%) in production, batch span exports, and async exporters. Overhead is negligible compared to LLM latency.

Should I log full prompts and responses?

Be cautious with PII. Log prompts/responses in dev/staging, not production (unless you have strong PII filtering). Use truncation and sampling. Store sensitive content in secure storage, not observability backend.

How do I sample traces effectively?

Sample 1-10% in production, 100% in dev. Use parent-based sampling to keep complete traces. Sample important requests (errors, slow requests) at 100%. Use adaptive sampling based on latency/cost.

Can OpenTelemetry track LLM quality?

Yes, via span attributes. Add custom attributes for accuracy, hallucination score, user feedback, etc. Query traces by quality metrics to find problematic patterns.

How do I trace multi-tenant systems?

Add tenant_id as span attribute. Use resource attributes for per-tenant metrics. Filter traces and metrics by tenant for debugging and billing.

What's the cost of storing traces?

$0.10-0.50 per GB in Datadog/New Relic. Use sampling to reduce volume. Retain recent traces (7-30 days), aggregate older data as metrics.

Conclusion

OpenTelemetry enables production LLM observability:

  • Standardized instrumentation works across vendors and frameworks
  • Distributed tracing tracks requests across LLM → retrieval → tools
  • Span attributes capture prompts, tokens, costs, quality metrics
  • Metrics collection enables real-time dashboards and alerts
  • Log correlation connects traces to application logs
  • Performance debugging identifies bottlenecks with trace analysis

OpenTelemetry is essential for production AI systems.

At HinterBuild, we implement observability for AI systems:

Contact us for LLM observability consulting.

Free consultation

Book a free consultation call on LLM observability & OpenTelemetry

30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.

Book a meeting

Keep reading

Related articles

AI Systems · 10 min read

Triton vs vLLM: LLM Serving Framework Comparison for

Triton vs vLLM guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Read post

AI Systems · 9 min read

Synthetic Data Generation for LLM Evals

Synthetic Data Generation for LLM Evals guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable.

Read post

AI Systems · 10 min read

PII Detection and Scrubbing in LLM Pipelines

PII Detection and Scrubbing in LLM Pipelines guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable.

Read post

AI Systems · 11 min read

OWASP Top 10 for LLM Applications: Complete Security Guide

Learn owasp top 10 for llm applications through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post