OpenAI-Compatible API with vLLM on Kubernetes
OpenAI-Compatible API with vLLM on Kubernetes guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable,.
Muhammad Abdul Sami
· 9 min read
- Kubernetes
- DevOps
- MLOps
- Observability
Table of Contents:
- Why Self-Host OpenAI-Compatible API?
- vLLM OpenAI API Overview
- Kubernetes Deployment Architecture
- Helm Chart and Configuration
- Migration from OpenAI API
- Monitoring and Observability
- Cost Analysis and ROI
- Frequently Asked Questions
Why Self-Host OpenAI-Compatible API?
Short answer: Self-hosting OpenAI-compatible API with vLLM on Kubernetes reduces LLM costs by 70-90% for high-volume workloads while maintaining data privacy and eliminating external API dependencies.
After migrating production workloads from OpenAI API to self-hosted vLLM at HinterBuild, the economics are compelling: at 1M+ requests/month, self-hosting pays for itself within 2-3 months with ongoing 80%+ cost savings.
Key Takeaways:
- OpenAI-compatible API means zero code changes — drop-in replacement
- vLLM provides the server, Kubernetes provides orchestration and scaling
- Break-even: ~200K-500K API calls/month depending on model size
- Cost savings: 70-90% for high-volume workloads (1M+ calls/month)
- Additional benefits: Data privacy, no rate limits, lower latency (regional deployment)
For teams with AI agent systems or RAG pipelines making 100K+ LLM calls monthly, this is the most impactful cost optimization in 2026.
vLLM OpenAI API Overview
API Compatibility
vLLM implements OpenAI's API specification:
import openai
client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
# vLLM API (drop-in replacement)
client = openai.OpenAI(
base_url="https://vllm.your-domain.com/v1",
api_key="not-used", # Or use for auth
)
# Identical API calls
response = client.chat.completions.create(
model="meta-llama/Llama-3.1-70B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing"},
],
temperature=0.7,
max_tokens=256,
)
print(response.choices[0].message.content)
Supported Endpoints
| Endpoint | OpenAI | vLLM | Notes |
|---|---|---|---|
/v1/chat/completions | ✅ | ✅ | Full compatibility |
/v1/completions | ✅ | ✅ | Legacy format |
/v1/embeddings | ✅ | ⚠️ | Requires embedding model |
/v1/models | ✅ | ✅ | List available models |
| Streaming (SSE) | ✅ | ✅ | stream=True |
| Function calling | ✅ | ✅ | JSON schema support |
Migration Path
# Before: OpenAI API
import os
from openai import OpenAI
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
# After: vLLM on Kubernetes (single line change)
client = OpenAI(
base_url="http://vllm-api.llm-serving.svc.cluster.local:8000/v1",
api_key="not-used",
)
# All other code unchanged
response = client.chat.completions.create(...)
For production AI systems, this enables gradual migration with A/B testing.
Kubernetes Deployment Architecture
Production Architecture
┌─────────────────────────────────────────────────────┐
│ Ingress/LB │
│ (TLS termination, auth) │
└───────────────────┬─────────────────────────────────┘
│
┌───────────┴───────────┐
│ vLLM Service │
│ (ClusterIP) │
└───────────┬───────────┘
│
┌───────────────┼───────────────┐
│ │ │
┌───▼────┐ ┌───▼────┐ ┌───▼────┐
│ vLLM │ │ vLLM │ │ vLLM │
│ Pod 1 │ │ Pod 2 │ │ Pod 3 │
│ (A100) │ │ (A100) │ │ (A100) │
└────────┘ └────────┘ └────────┘
│ │ │
└─────────────┴─────────────┘
│
┌──────▼───────┐
│ Prometheus │
│ Grafana │
└──────────────┘
Components:
- Ingress: TLS, authentication, rate limiting
- Service: Load balancing across vLLM pods
- vLLM Pods: Model serving with GPU
- Monitoring: Prometheus + Grafana
Deploy with Kubernetes platform engineering best practices.
Helm Chart and Configuration
Create vLLM Helm Chart
mkdir -p vllm-helm/templates cd vllm-helm
Chart.yaml:
apiVersion: v2 name: vllm description: vLLM OpenAI-compatible API server version: 1.0.0 appVersion: "0.6.0"
values.yaml:
# values.yaml
replicaCount: 3
image:
repository: vllm/vllm-openai
tag: v0.6.0
pullPolicy: IfNotPresent
model:
name: meta-llama/Llama-3.1-8B-Instruct
maxModelLen: 4096
quantization: awq # Options: awq, gptq, none
tensorParallelSize: 1
gpuMemoryUtilization: 0.9
# HuggingFace token for gated models
huggingface:
token: "" # Set via secret
resources:
requests:
nvidia.com/gpu: 1
memory: 24Gi
cpu: 4
limits:
nvidia.com/gpu: 1
memory: 32Gi
nodeSelector:
node.kubernetes.io/instance-type: g5.2xlarge
service:
type: ClusterIP
port: 8000
ingress:
enabled: true
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/proxy-body-size: "10m"
hosts:
- host: vllm.your-domain.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: vllm-tls
hosts:
- vllm.your-domain.com
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: vllm_num_requests_running
target:
type: AverageValue
averageValue: "8"
monitoring:
enabled: true
serviceMonitor: true
templates/deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "vllm.fullname" . }}
labels:
{{- include "vllm.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
{{- include "vllm.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "vllm.selectorLabels" . | nindent 8 }}
spec:
nodeSelector:
{{- toYaml .Values.nodeSelector | nindent 8 }}
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
initContainers:
- name: download-model
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
command:
- python3
- -c
- |
from huggingface_hub import snapshot_download
snapshot_download("{{ .Values.model.name }}", cache_dir="/model-cache")
env:
{{- if .Values.huggingface.token }}
- name: HF_TOKEN
valueFrom:
secretKeyRef:
name: huggingface-token
key: token
{{- end }}
volumeMounts:
- name: model-cache
mountPath: /model-cache
containers:
- name: vllm
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
args:
- --model
- {{ .Values.model.name }}
- --max-model-len
- "{{ .Values.model.maxModelLen }}"
{{- if .Values.model.quantization }}
- --quantization
- {{ .Values.model.quantization }}
{{- end }}
- --tensor-parallel-size
- "{{ .Values.model.tensorParallelSize }}"
- --gpu-memory-utilization
- "{{ .Values.model.gpuMemoryUtilization }}"
- --host
- "0.0.0.0"
- --port
- "8000"
- --served-model-name
- {{ .Values.model.name }}
env:
{{- if .Values.huggingface.token }}
- name: HF_TOKEN
valueFrom:
secretKeyRef:
name: huggingface-token
key: token
{{- end }}
ports:
- containerPort: 8000
name: http
protocol: TCP
volumeMounts:
- name: model-cache
mountPath: /model-cache
resources:
{{- toYaml .Values.resources | nindent 10 }}
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 120
periodSeconds: 30
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 60
periodSeconds: 10
volumes:
- name: model-cache
emptyDir: {}
templates/service.yaml:
apiVersion: v1
kind: Service
metadata:
name: {{ include "vllm.fullname" . }}
labels:
{{- include "vllm.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "vllm.selectorLabels" . | nindent 4 }}
Deploy with Helm
# Create namespace kubectl create namespace llm-serving # Create HuggingFace token secret (if needed) kubectl create secret generic huggingface-token \ --from-literal=token=hf_xxxxxxxxxxxx \ -n llm-serving # Install/upgrade helm upgrade --install vllm ./vllm-helm \ --namespace llm-serving \ --values custom-values.yaml \ --wait
For cloud infrastructure automation, use GitOps with ArgoCD.
Migration from OpenAI API
Step 1: Parallel Deployment
import os
from openai import OpenAI
import random
# Dual client setup
openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
vllm_client = OpenAI(
base_url="https://vllm.your-domain.com/v1",
api_key=os.getenv("VLLM_API_KEY"),
)
def generate_completion(messages: list, use_vllm_percent: int = 10):
"""
Route requests to vLLM or OpenAI based on percentage.
Args:
messages: Chat messages
use_vllm_percent: 0-100, percentage of traffic to route to vLLM
"""
use_vllm = random.randint(1, 100) <= use_vllm_percent
client = vllm_client if use_vllm else openai_client
# Track which client was used
import structlog
logger = structlog.get_logger()
logger.info("llm_request", client="vllm" if use_vllm else "openai")
response = client.chat.completions.create(
model="meta-llama/Llama-3.1-70B-Instruct" if use_vllm else "gpt-4o",
messages=messages,
temperature=0.7,
max_tokens=256,
)
return response.choices[0].message.content
Step 2: Gradual Traffic Shift
# Week 1: 10% to vLLM generate_completion(messages, use_vllm_percent=10) # Week 2: 25% to vLLM (if metrics look good) generate_completion(messages, use_vllm_percent=25) # Week 3: 50% to vLLM generate_completion(messages, use_vllm_percent=50) # Week 4: 100% to vLLM (keep OpenAI as fallback) generate_completion(messages, use_vllm_percent=100)
Step 3: Fallback Pattern
from tenacity import retry, stop_after_attempt, wait_exponential
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=10),
)
def generate_with_fallback(messages: list) -> str:
"""Try vLLM first, fall back to OpenAI on failure."""
try:
response = vllm_client.chat.completions.create(
model="meta-llama/Llama-3.1-70B-Instruct",
messages=messages,
timeout=30,
)
return response.choices[0].message.content
except Exception as e:
logger.warning("vllm_fallback_triggered", error=str(e))
# Fall back to OpenAI
response = openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
)
return response.choices[0].message.content
For AI agent tool calling, ensure function calling schemas are compatible.
Monitoring and Observability
Prometheus Metrics
vLLM exposes Prometheus metrics at /metrics:
# ServiceMonitor for Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: vllm
namespace: llm-serving
spec:
selector:
matchLabels:
app: vllm
endpoints:
- port: http
path: /metrics
interval: 30s
Key metrics:
# Request rate rate(vllm_request_success_total[5m]) # Time to first token (p95) histogram_quantile(0.95, rate(vllm_time_to_first_token_seconds_bucket[5m])) # GPU utilization vllm_gpu_cache_usage_perc # Running requests vllm_num_requests_running # Failure rate rate(vllm_request_failure_total[5m]) / rate(vllm_request_success_total[5m])
Grafana Dashboard
# Sample dashboard panels
panels:
- title: Requests per Second
query: rate(vllm_request_success_total[5m])
- title: Time to First Token (p50, p95, p99)
queries:
- histogram_quantile(0.50, rate(vllm_time_to_first_token_seconds_bucket[5m]))
- histogram_quantile(0.95, rate(vllm_time_to_first_token_seconds_bucket[5m]))
- histogram_quantile(0.99, rate(vllm_time_to_first_token_seconds_bucket[5m]))
- title: GPU Memory Usage
query: vllm_gpu_cache_usage_perc
- title: Active Requests
query: vllm_num_requests_running
Alerting Rules
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: vllm-alerts
namespace: llm-serving
spec:
groups:
- name: vllm
interval: 30s
rules:
- alert: VLLMHighLatency
expr: |
histogram_quantile(0.95, rate(vllm_time_to_first_token_seconds_bucket[5m])) > 1.0
for: 5m
labels:
severity: warning
annotations:
summary: "vLLM p95 TTFT > 1s"
- alert: VLLMHighFailureRate
expr: |
rate(vllm_request_failure_total[5m]) / rate(vllm_request_success_total[5m]) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "vLLM failure rate > 5%"
- alert: VLLMNoCapacity
expr: vllm_num_requests_running > 50
for: 10m
labels:
severity: warning
annotations:
summary: "vLLM at capacity, scale up"
Deploy observability infrastructure with Prometheus + Grafana + alerting.
Cost Analysis and ROI
Monthly Cost Comparison
Scenario: 2M chat completions/month, avg 256 tokens output
| Provider | Model | Cost/1M tokens | Monthly Cost |
|---|---|---|---|
| OpenAI | GPT-4o | $15 (out) | $7,680 |
| OpenAI | GPT-4o-mini | $0.60 (out) | $307 |
| Self-hosted (vLLM) | Llama 3.1 70B | — | $730 (infra) |
| Self-hosted (vLLM) | Llama 3.1 8B | — | $240 (infra) |
Infrastructure costs (AWS):
- 70B quantized: 1x g5.2xlarge (A10G 24GB) = $730/mo
- 8B quantized: 1x g4dn.xlarge (T4 16GB) = $240/mo
Break-Even Analysis
# Break-even calculator
def calculate_breakeven(
requests_per_month: int,
avg_output_tokens: int,
openai_cost_per_1m_tokens: float,
infra_cost_per_month: float,
) -> dict:
"""Calculate break-even point for self-hosting."""
total_tokens = requests_per_month * avg_output_tokens
openai_monthly_cost = (total_tokens / 1_000_000) * openai_cost_per_1m_tokens
savings = openai_monthly_cost - infra_cost_per_month
savings_percent = (savings / openai_monthly_cost) * 100 if openai_monthly_cost > 0 else 0
# Months to break even (including setup cost)
setup_cost = 5000 # Engineering time
months_to_breakeven = setup_cost / savings if savings > 0 else float('inf')
return {
"openai_cost": openai_monthly_cost,
"self_hosted_cost": infra_cost_per_month,
"monthly_savings": savings,
"savings_percent": savings_percent,
"months_to_breakeven": months_to_breakeven,
}
# Example: GPT-4o-mini vs Llama 3.1 8B
result = calculate_breakeven(
requests_per_month=2_000_000,
avg_output_tokens=256,
openai_cost_per_1m_tokens=0.60,
infra_cost_per_month=240,
)
print(f"Monthly savings: ${result['monthly_savings']:.2f}")
print(f"Savings: {result['savings_percent']:.1f}%")
print(f"Break-even: {result['months_to_breakeven']:.1f} months")
Output:
Monthly savings: $67.20 Savings: 21.9% Break-even: 74.4 months # Not worth it at this volume # But at 10M requests/month: Monthly savings: $336.00 Savings: 58.3% Break-even: 14.9 months # Worth considering
Rule of thumb: Self-hosting becomes cost-effective at 500K-1M requests/month for smaller models, 200K-500K for larger models.
For RAG & LLM systems at scale, cost savings are 70-90%.
Related implementation guides:
Primary references: official documentation, official documentation, official documentation.
Operating OpenAI-Compatible API with vLLM on Kubernetes as a System
The implementation is only one part of OpenAI-Compatible API with vLLM on Kubernetes. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.
Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.
Define the failure model
List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.
Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.
Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.
Measure the decision, not just the component
Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.
Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.
Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.
Roll out with reversible steps
Ship OpenAI-Compatible API with vLLM on Kubernetes behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.
Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.
Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.
Capacity and cost controls
Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.
Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.
Production readiness review
Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.
The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our OpenAI-Compatible API with vLLM on Kubernetes engineering support.
Frequently Asked Questions
Is vLLM's OpenAI API truly compatible?
95% compatible. Chat completions, completions, streaming, and function calling work identically. Some advanced OpenAI features (batch API, fine-tuning endpoints) are not supported.
Can I use OpenAI Python SDK with vLLM?
Yes. Just change base_url:
client = openai.OpenAI(base_url="https://vllm.your-domain.com/v1")
What models work with vLLM's OpenAI API?
Any HuggingFace causal LM: Llama, Mistral, Qwen, Phi, CodeLlama, etc. See vLLM supported models.
How do I handle authentication?
Set up API keys in Kubernetes ingress or reverse proxy:
# Ingress with auth annotations: nginx.ingress.kubernetes.io/auth-type: basic nginx.ingress.kubernetes.io/auth-secret: vllm-auth
Can I serve multiple models on one cluster?
Yes. Deploy multiple vLLM deployments with different model configurations and route via ingress path:
# /v1/llama → vllm-llama service # /v1/mistral → vllm-mistral service
How do I autoscale vLLM on Kubernetes?
Use HPA with custom metrics (requests running):
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
metrics:
- type: Pods
pods:
metric:
name: vllm_num_requests_running
target:
type: AverageValue
averageValue: "8"
What's the latency difference vs OpenAI?
Self-hosted (same region): 100-200ms faster (no network hop to OpenAI)
Self-hosted (cross-region): 50-150ms slower
Deploy regionally for best latency.
How do I monitor vLLM in production?
Use Prometheus + Grafana with observability stack. Monitor TTFT p95, throughput, GPU utilization, and failure rate.
Conclusion
Deploying OpenAI-compatible API with vLLM on Kubernetes is the most cost-effective path for teams with 500K+ monthly LLM requests. With 70-90% cost savings, improved data privacy, and zero code changes, it's a high-ROI migration for production AI systems.
The deployment playbook:
- Start with Helm chart and deploy to staging
- Migrate 10% of traffic and compare quality/latency
- Gradually shift to 100% over 2-4 weeks
- Keep OpenAI as fallback for reliability
- Monitor with Prometheus and optimize based on metrics
At HinterBuild, we deploy self-hosted LLM infrastructure on Kubernetes:
- RAG & LLM Systems
- Kubernetes Platform Engineering
- Cloud Infrastructure & DevOps
- Observability & Monitoring
Contact us to migrate your OpenAI workloads to self-hosted infrastructure.
Free consultation
Book a free consultation call on LLM serving on Kubernetes
30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.
Book a meeting
Keep reading
Related articles
GPU Scheduling in Kubernetes: Complete NVIDIA Guide for ML
Learn gpu scheduling in kubernetes through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Read post
Deploy LLMs on Kubernetes: Complete GPU Autoscaling Guide
Learn deploy llms on kubernetes through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Read post
Crossplane for AI Infrastructure: Kubernetes-Native IaC for
Learn crossplane for ai infrastructure through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Read post
QLoRA: Fine-Tune 70B Models on Single GPU with 4-bit
QLoRA guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.
Read post