MLOpsSeptember 14, 2026 · 12 min read

Deploy LLMs on Kubernetes: Complete GPU Autoscaling Guide

Q: What GPU should I use for LLM serving?

A10G (24GB) for small models (<13B), A100 40GB for medium (13-33B), A100 80GB or H100 for large (70B+). For production, prefer A100/H100 for better performance/watt.

Q: Should I use spot instances for real-time serving?

Use spot for batch jobs, on-demand for real-time. Spot instances can be interrupted with 2-minute notice—unacceptable for user-facing APIs.

Q: How do I prevent GPU idle time?

Implement queue-based autoscaling (scale when queue depth threshold) and batch inference for background workloads. Target 80-90% utilization.

Q: What's the right HPA metric?

Use queue depth (requests waiting) over GPU utilization. Queue depth is a leading indicator; GPU utilization lags. Scale when queue 10-20 requests per pod.

Q: How do I handle model loading time?

Use readiness probes with long initialDelaySeconds (5-10 min for 70B models). Keep at least 1 replica running to avoid cold starts.

Q: Can I run multiple models on one GPU?

Yes with vLLM LoRA adapters or multi-model serving, but throughput per model drops. Prefer dedicated pods per model for production.

Learn deploy llms on kubernetes through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Muhammad Abdul Sami

September 14, 2026 · 12 min read

Kubernetes
DevOps
MLOps
Observability

Table of Contents:

The Deployment Challenge
Kubernetes GPU Node Setup
vLLM Model Serving
Horizontal Pod Autoscaling
Cluster Autoscaler for GPUs
Batch Inference Workloads
Cost Optimization Strategies
Production Monitoring
Frequently Asked Questions

The Deployment Challenge: Why LLM Serving is Different

Short answer: LLMs require GPU resources, high memory, low-latency serving, and dynamic scaling—none of which traditional Kubernetes deployments handle well. The fix is specialized GPU scheduling, vLLM serving, and custom autoscaling logic.

A fintech deployed a Llama 2 70B model on Kubernetes. Fixed 8x A100 GPUs running 24/7 cost $11,000/month. Traffic varied 10x between peak and off-hours. We implemented GPU autoscaling with node provisioning—GPUs scale 1-8 based on queue depth. New cost: $4,200/month with better p95 latency.

Key Takeaways:

vLLM enables high-throughput LLM serving with PagedAttention

GPU node pools separate LLM workloads from CPU workloads

HPA scales pods based on custom metrics (queue depth, GPU utilization)

Cluster Autoscaler provisions GPU nodes on-demand

Batch inference amortizes GPU costs for non-real-time workloads

Monitoring tracks tokens/sec, latency, and GPU utilization

For production AI systems, Kubernetes deployment requires specialized GPU infrastructure.

Kubernetes GPU Node Setup

GPU nodes require NVIDIA drivers, device plugin, and node labels.

EKS GPU Node Group (AWS)

yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ml-cluster
  region: us-west-2
  version: "1.28"

nodeGroups:
  - name: gpu-llm-inference
    instanceType: g5.12xlarge  # 4x A10G GPUs
    minSize: 0
    maxSize: 10
    desiredCapacity: 1
    
    # GPU configuration
    labels:
      workload: llm-inference
      gpu: "true"
    
    taints:
      - key: nvidia.com/gpu
        value: "true"
        effect: NoSchedule
    
    # Enable GPU support
    preBootstrapCommands:
      - "yum install -y nvidia-driver-latest-dkms"
    
    iam:
      withAddonPolicies:
        autoScaler: true
        cloudWatch: true
    
    # Spot instances for cost savings
    instancesDistribution:
      onDemandBaseCapacity: 1
      onDemandPercentageAboveBaseCapacity: 0
      spotInstancePools: 3

bash

# Create cluster with GPU nodes
eksctl create cluster -f gpu-nodegroup.yaml

# Install NVIDIA device plugin
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.5/nvidia-device-plugin.yml

# Verify GPUs
kubectl get nodes -l gpu=true
kubectl describe node <gpu-node-name> | grep nvidia.com/gpu

GKE GPU Node Pool (GCP)

bash

# Create GKE cluster with GPU node pool
gcloud container clusters create ml-cluster \
  --zone us-central1-a \
  --num-nodes 1 \
  --machine-type n1-standard-4

# Add GPU node pool
gcloud container node-pools create gpu-pool \
  --cluster ml-cluster \
  --zone us-central1-a \
  --machine-type n1-standard-16 \
  --accelerator type=nvidia-tesla-a100,count=4 \
  --num-nodes 0 \
  --min-nodes 0 \
  --max-nodes 10 \
  --enable-autoscaling \
  --node-labels=workload=llm-inference,gpu=true \
  --node-taints=nvidia.com/gpu=true:NoSchedule

# Install NVIDIA drivers
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml

Connect to Kubernetes platform engineering services.

vLLM Model Serving

vLLM is the highest-throughput LLM serving framework with PagedAttention.

vLLM Deployment

yaml

# vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama2-70b
  namespace: ml-serving
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama2
  template:
    metadata:
      labels:
        app: vllm-llama2
    spec:
      # Schedule on GPU nodes
      nodeSelector:
        workload: llm-inference
      
      tolerations:
      - key: nvidia.com/gpu
        operator: Equal
        value: "true"
        effect: NoSchedule
      
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.3.1
        
        command:
        - python3
        - -m
        - vllm.entrypoints.openai.api_server
        - --model
        - meta-llama/Llama-2-70b-chat-hf
        - --tensor-parallel-size
        - "4"
        - --max-model-len
        - "4096"
        - --gpu-memory-utilization
        - "0.95"
        
        ports:
        - containerPort: 8000
          name: http
        
        resources:
          requests:
            nvidia.com/gpu: 4
            memory: 160Gi
            cpu: 32
          limits:
            nvidia.com/gpu: 4
            memory: 160Gi
        
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token
        
        # Health checks
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 300
          periodSeconds: 30
          timeoutSeconds: 10
        
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 300
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-llama2-service
  namespace: ml-serving
spec:
  selector:
    app: vllm-llama2
  ports:
  - port: 80
    targetPort: 8000
  type: ClusterIP

bash

# Deploy vLLM
kubectl apply -f vllm-deployment.yaml

# Check GPU allocation
kubectl get pod -n ml-serving -o json | jq '.items[].spec.containers[].resources'

# Test inference
kubectl port-forward -n ml-serving svc/vllm-llama2-service 8000:80

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-70b-chat-hf",
    "prompt": "Explain Kubernetes in one sentence:",
    "max_tokens": 100
  }'

vLLM Configuration Tuning

python

# vllm_config.py - Optimal settings for throughput
from dataclasses import dataclass

@dataclass
class VLLMConfig:
    """vLLM configuration for production."""
    
    # Model parallelism
    tensor_parallel_size: int = 4  # Split across 4 GPUs
    pipeline_parallel_size: int = 1
    
    # Memory management
    gpu_memory_utilization: float = 0.95  # Use 95% GPU memory
    max_model_len: int = 4096  # Context length
    
    # Batching
    max_num_batched_tokens: int = 8192
    max_num_seqs: int = 256  # Concurrent requests
    
    # Performance
    disable_log_stats: bool = False
    enable_prefix_caching: bool = True  # Cache common prefixes
    
    # Quantization (optional)
    quantization: str = None  # "awq" or "gptq" for lower memory

def calculate_gpu_requirements(
    model_params: int,
    tensor_parallel: int,
    quantization: str = None,
) -> dict:
    """Calculate GPU memory requirements."""
    
    # Model weights (in GB)
    if quantization == "awq":
        weight_memory = (model_params * 4) / (8 * 1e9)  # 4-bit
    elif quantization == "gptq":
        weight_memory = (model_params * 4) / (8 * 1e9)  # 4-bit
    else:
        weight_memory = (model_params * 2) / 1e9  # FP16
    
    # KV cache (approximate)
    kv_cache_memory = 20  # GB per GPU
    
    # Per-GPU memory
    per_gpu_memory = (weight_memory / tensor_parallel) + kv_cache_memory
    
    return {
        "model_memory_gb": weight_memory,
        "per_gpu_memory_gb": per_gpu_memory,
        "total_memory_gb": per_gpu_memory * tensor_parallel,
        "recommended_gpu": "A100 80GB" if per_gpu_memory > 40 else "A100 40GB",
    }

# Example: Llama 2 70B
config = calculate_gpu_requirements(
    model_params=70e9,
    tensor_parallel=4,
    quantization=None,
)
print(config)
# Output: {'model_memory_gb': 140.0, 'per_gpu_memory_gb': 55.0, ...}

See vLLM production guide for deep dive.

Horizontal Pod Autoscaling

Scale pods based on custom metrics (queue depth, GPU utilization).

Custom Metrics with Prometheus

yaml

# servicemonitor.yaml - Export vLLM metrics
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vllm-metrics
  namespace: ml-serving
spec:
  selector:
    matchLabels:
      app: vllm-llama2
  endpoints:
  - port: http
    path: /metrics
    interval: 15s

python

# metrics_exporter.py - Export custom metrics to Prometheus
from prometheus_client import Gauge, generate_latest
from fastapi import FastAPI
import asyncio

app = FastAPI()

# Custom metrics
queue_depth = Gauge("vllm_queue_depth", "Number of pending requests")
gpu_utilization = Gauge("vllm_gpu_utilization", "GPU utilization percentage")
tokens_per_second = Gauge("vllm_tokens_per_second", "Generation throughput")

async def collect_metrics():
    """Collect metrics from vLLM."""
    while True:
        # Get queue depth from vLLM
        queue_size = await get_vllm_queue_depth()
        queue_depth.set(queue_size)
        
        # Get GPU utilization
        gpu_util = await get_gpu_utilization()
        gpu_utilization.set(gpu_util)
        
        # Get tokens/sec
        tps = await get_tokens_per_second()
        tokens_per_second.set(tps)
        
        await asyncio.sleep(10)

@app.on_event("startup")
async def startup():
    asyncio.create_task(collect_metrics())

@app.get("/metrics")
async def metrics():
    return generate_latest()

HPA Configuration

yaml

# hpa.yaml - Autoscale based on queue depth
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-hpa
  namespace: ml-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama2-70b
  
  minReplicas: 1
  maxReplicas: 8
  
  metrics:
  # Scale on queue depth
  - type: Pods
    pods:
      metric:
        name: vllm_queue_depth
      target:
        type: AverageValue
        averageValue: "10"  # Target 10 requests per pod
  
  # Scale on GPU utilization
  - type: Pods
    pods:
      metric:
        name: vllm_gpu_utilization
      target:
        type: AverageValue
        averageValue: "80"  # Target 80% GPU usage
  
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scale down
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    
    scaleUp:
      stabilizationWindowSeconds: 30  # Scale up quickly
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30

bash

# Install Prometheus Adapter for custom metrics
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus-adapter prometheus-community/prometheus-adapter \
  --namespace monitoring \
  --set prometheus.url=http://prometheus-server.monitoring.svc

# Apply HPA
kubectl apply -f hpa.yaml

# Monitor autoscaling
kubectl get hpa -n ml-serving -w

Cluster Autoscaler for GPUs

Cluster Autoscaler provisions GPU nodes when pods are pending.

Cluster Autoscaler Configuration

yaml

# cluster-autoscaler.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
      - name: cluster-autoscaler
        image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.28.0
        command:
        - ./cluster-autoscaler
        - --cloud-provider=aws
        - --namespace=kube-system
        - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/ml-cluster
        - --balance-similar-node-groups
        - --skip-nodes-with-system-pods=false
        - --scale-down-delay-after-add=5m
        - --scale-down-unneeded-time=10m
        - --expander=priority
        
        env:
        - name: AWS_REGION
          value: us-west-2
        
        resources:
          requests:
            cpu: 100m
            memory: 300Mi
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cluster-autoscaler
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-autoscaler
rules:
- apiGroups: [""]
  resources: ["events", "endpoints"]
  verbs: ["create", "patch"]
- apiGroups: [""]
  resources: ["pods/eviction"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["pods/status"]
  verbs: ["update"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["watch", "list", "get", "update"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets", "statefulsets"]
  verbs: ["watch", "list", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-autoscaler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-autoscaler
subjects:
- kind: ServiceAccount
  name: cluster-autoscaler
  namespace: kube-system

Priority Expander for GPU Nodes

yaml

# priority-expander-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-expander
  namespace: kube-system
data:
  priorities: |-
    10:
      - .*-gpu-spot.*
    5:
      - .*-gpu-ondemand.*
    1:
      - .*

This prioritizes spot GPU instances (cheapest) over on-demand.

bash

# Deploy Cluster Autoscaler
kubectl apply -f cluster-autoscaler.yaml
kubectl apply -f priority-expander-config.yaml

# Monitor autoscaling
kubectl logs -f -n kube-system deployment/cluster-autoscaler

# Watch nodes scale
watch kubectl get nodes -l gpu=true

Batch Inference Workloads

Batch jobs maximize GPU utilization for non-real-time workloads.

Kubernetes Job for Batch Inference

yaml

# batch-inference-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: batch-inference-embeddings
  namespace: ml-batch
spec:
  parallelism: 4  # Run 4 pods in parallel
  completions: 4
  
  template:
    spec:
      nodeSelector:
        workload: llm-inference
      
      tolerations:
      - key: nvidia.com/gpu
        operator: Equal
        value: "true"
        effect: NoSchedule
      
      restartPolicy: Never
      
      containers:
      - name: batch-worker
        image: my-registry/batch-inference:latest
        
        command:
        - python
        - batch_worker.py
        - --input-bucket
        - s3://data/input/
        - --output-bucket
        - s3://data/embeddings/
        - --model
        - sentence-transformers/all-MiniLM-L6-v2
        
        resources:
          requests:
            nvidia.com/gpu: 1
            memory: 16Gi
            cpu: 8
          limits:
            nvidia.com/gpu: 1
            memory: 16Gi
        
        env:
        - name: WORKER_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name

python

# batch_worker.py
import torch
from transformers import AutoTokenizer, AutoModel
import boto3
from pathlib import Path

def process_batch(
    input_bucket: str,
    output_bucket: str,
    model_name: str,
    worker_id: str,
) -> None:
    """Process batch of documents."""
    
    # Load model
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModel.from_pretrained(model_name).to(device)
    
    # S3 client
    s3 = boto3.client("s3")
    
    # List files to process (shard by worker_id)
    response = s3.list_objects_v2(Bucket=input_bucket, Prefix=f"shard-{worker_id}/")
    
    for obj in response.get("Contents", []):
        key = obj["Key"]
        
        # Download
        local_path = Path(f"/tmp/{key}")
        local_path.parent.mkdir(exist_ok=True, parents=True)
        s3.download_file(input_bucket, key, str(local_path))
        
        # Process
        with open(local_path) as f:
            texts = f.readlines()
        
        # Generate embeddings in batches
        embeddings = []
        batch_size = 32
        
        for i in range(0, len(texts), batch_size):
            batch = texts[i:i+batch_size]
            
            inputs = tokenizer(batch, padding=True, truncation=True, return_tensors="pt").to(device)
            
            with torch.no_grad():
                outputs = model(**inputs)
                batch_embeddings = outputs.last_hidden_state.mean(dim=1).cpu().numpy()
                embeddings.extend(batch_embeddings)
        
        # Upload results
        output_key = key.replace("input/", "embeddings/") + ".npy"
        import numpy as np
        np.save(f"/tmp/{output_key}", np.array(embeddings))
        s3.upload_file(f"/tmp/{output_key}", output_bucket, output_key)
        
        print(f"✓ Processed {key} → {output_key}")

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument("--input-bucket")
    parser.add_argument("--output-bucket")
    parser.add_argument("--model")
    args = parser.parse_args()
    
    import os
    worker_id = os.environ["WORKER_ID"]
    
    process_batch(args.input_bucket, args.output_bucket, args.model, worker_id)

bash

# Submit batch job
kubectl apply -f batch-inference-job.yaml

# Monitor progress
kubectl get jobs -n ml-batch -w
kubectl logs -n ml-batch job/batch-inference-embeddings --follow

Cost Optimization Strategies

Reduce GPU costs 50-80% with smart scheduling.

Spot Instances for Batch Workloads

yaml

# spot-nodegroup.yaml
nodeGroups:
  - name: gpu-spot
    instanceType: g5.12xlarge
    minSize: 0
    maxSize: 20
    
    # 100% spot instances
    instancesDistribution:
      onDemandBaseCapacity: 0
      onDemandPercentageAboveBaseCapacity: 0
      spotAllocationStrategy: capacity-optimized
      spotInstancePools: 4
    
    labels:
      workload: batch-inference
      lifecycle: spot
    
    taints:
    - key: spot
      value: "true"
      effect: NoSchedule

Node Termination Handler

yaml

# spot-termination-handler.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: spot-termination-handler
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: spot-termination-handler
  template:
    metadata:
      labels:
        app: spot-termination-handler
    spec:
      nodeSelector:
        lifecycle: spot
      
      containers:
      - name: handler
        image: amazon/aws-node-termination-handler:v1.20.0
        
        env:
        - name: ENABLE_SPOT_INTERRUPTION_DRAINING
          value: "true"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

Cost Analysis

python

# cost_calculator.py
from dataclasses import dataclass

@dataclass
class GPUPricing:
    """GPU instance pricing."""
    instance_type: str
    on_demand_hourly: float
    spot_hourly: float
    gpu_count: int
    gpu_memory_gb: int

PRICING = {
    "g5.xlarge": GPUPricing("g5.xlarge", 1.006, 0.35, 1, 24),
    "g5.2xlarge": GPUPricing("g5.2xlarge", 1.212, 0.42, 1, 24),
    "g5.12xlarge": GPUPricing("g5.12xlarge", 5.672, 1.98, 4, 96),
    "p4d.24xlarge": GPUPricing("p4d.24xlarge", 32.77, 11.47, 8, 320),
}

def calculate_monthly_cost(
    instance_type: str,
    avg_instances: float,
    spot_percentage: float = 0.8,
) -> dict:
    """Calculate monthly GPU costs."""
    
    pricing = PRICING[instance_type]
    hours_per_month = 730
    
    # Split on-demand and spot
    on_demand_instances = avg_instances * (1 - spot_percentage)
    spot_instances = avg_instances * spot_percentage
    
    # Calculate costs
    on_demand_cost = on_demand_instances * pricing.on_demand_hourly * hours_per_month
    spot_cost = spot_instances * pricing.spot_hourly * hours_per_month
    total_cost = on_demand_cost + spot_cost
    
    # Compare to 100% on-demand
    full_on_demand_cost = avg_instances * pricing.on_demand_hourly * hours_per_month
    savings = full_on_demand_cost - total_cost
    savings_pct = (savings / full_on_demand_cost) * 100
    
    return {
        "instance_type": instance_type,
        "avg_instances": avg_instances,
        "spot_percentage": spot_percentage,
        "monthly_cost": total_cost,
        "on_demand_cost": on_demand_cost,
        "spot_cost": spot_cost,
        "savings_vs_on_demand": savings,
        "savings_percentage": savings_pct,
    }

# Example: 4 average instances, 80% spot
result = calculate_monthly_cost("g5.12xlarge", avg_instances=4, spot_percentage=0.8)
print(f"Monthly cost: ${result['monthly_cost']:,.0f}")
print(f"Savings: ${result['savings_vs_on_demand']:,.0f} ({result['savings_percentage']:.1f}%)")

# Output:
# Monthly cost: $10,502
# Savings: $6,062 (36.6%)

Integrate with cloud infrastructure cost optimization.

Production Monitoring

Monitor GPU utilization, throughput, and costs.

Prometheus + Grafana Stack

yaml

# monitoring-stack.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
# Install via Helm
# helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring

Custom Dashboards

yaml

# grafana-dashboard-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: vllm-dashboard
  namespace: monitoring
data:
  vllm-dashboard.json: |
    {
      "dashboard": {
        "title": "vLLM GPU Serving",
        "panels": [
          {
            "title": "Requests per Second",
            "targets": [
              {
                "expr": "rate(vllm_requests_total[5m])"
              }
            ]
          },
          {
            "title": "GPU Utilization",
            "targets": [
              {
                "expr": "DCGM_FI_DEV_GPU_UTIL"
              }
            ]
          },
          {
            "title": "Tokens per Second",
            "targets": [
              {
                "expr": "vllm_tokens_per_second"
              }
            ]
          },
          {
            "title": "Queue Depth",
            "targets": [
              {
                "expr": "vllm_queue_depth"
              }
            ]
          }
        ]
      }
    }

Alerting Rules

yaml

# prometheus-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: vllm-alerts
  namespace: monitoring
spec:
  groups:
  - name: vllm
    interval: 30s
    rules:
    - alert: HighQueueDepth
      expr: vllm_queue_depth > 100
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High request queue depth"
        description: "Queue depth is {{ $value }}, consider scaling up"
    
    - alert: LowGPUUtilization
      expr: avg(DCGM_FI_DEV_GPU_UTIL) < 20
      for: 30m
      labels:
        severity: info
      annotations:
        summary: "Low GPU utilization"
        description: "Average GPU utilization {{ $value }}%, consider scaling down"
    
    - alert: GPUOutOfMemory
      expr: DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE > 0.95
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "GPU memory exhausted"

Deploy monitoring with observability services.

Related implementation guides:

Primary references: official documentation, official documentation, official documentation, official documentation.

Deploy LLMs on Kubernetes Decision Table

Decision	Prefer the simpler path when	Add operational complexity when
Architecture	One component can own the contract and state	Independent scaling or fault isolation is required
Rollout	Offline replay covers the meaningful cases	Live behavior requires shadow traffic and a canary
Recovery	A failed operation is safe to repeat	Partial effects require idempotency or reconciliation
Measurement	One service objective represents user impact	Quality, latency, and cost need separate gates

Operating Deploy LLMs on Kubernetes as a System

The implementation is only one part of Deploy LLMs on Kubernetes. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship Deploy LLMs on Kubernetes behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our Deploy LLMs on Kubernetes engineering support.

Operating Deploy LLMs on Kubernetes as a System

Define the failure model

Measure the decision, not just the component

Roll out with reversible steps

Capacity and cost controls

Production readiness review

Frequently Asked Questions

What GPU should I use for LLM serving?

A10G (24GB) for small models (<13B), A100 40GB for medium (13-33B), A100 80GB or H100 for large (70B+). For production, prefer A100/H100 for better performance/watt.

Should I use spot instances for real-time serving?

Use spot for batch jobs, on-demand for real-time. Spot instances can be interrupted with 2-minute notice—unacceptable for user-facing APIs.

How do I prevent GPU idle time?

Implement queue-based autoscaling (scale when queue depth > threshold) and batch inference for background workloads. Target 80-90% utilization.

What's the right HPA metric?

Use queue depth (requests waiting) over GPU utilization. Queue depth is a leading indicator; GPU utilization lags. Scale when queue > 10-20 requests per pod.

How do I handle model loading time?

Use readiness probes with long initialDelaySeconds (5-10 min for 70B models). Keep at least 1 replica running to avoid cold starts.

Can I run multiple models on one GPU?

Yes with vLLM LoRA adapters or multi-model serving, but throughput per model drops. Prefer dedicated pods per model for production.

Conclusion

Deploying LLMs on Kubernetes with GPU autoscaling enables efficient production serving:

vLLM provides highest throughput with PagedAttention
GPU node pools isolate LLM workloads with proper scheduling
HPA scales pods based on queue depth and utilization
Cluster Autoscaler provisions GPU nodes on-demand
Spot instances reduce batch inference costs 60-70%
Monitoring tracks throughput, latency, and GPU efficiency

GPU autoscaling reduces costs 50-80% while maintaining SLAs.

At HinterBuild, we deploy production LLM infrastructure:

Free consultation

Book a free consultation call on LLM deployment on Kubernetes

30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.

Book a meeting

Keep reading

MLOpsSeptember 14, 2026 · 9 min read

GPU Scheduling in Kubernetes: Complete NVIDIA Guide for ML

Learn gpu scheduling in kubernetes through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

AI SystemsSeptember 14, 2026 · 9 min read

QLoRA: Fine-Tune 70B Models on Single GPU with 4-bit

QLoRA guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Read post

AI SystemsSeptember 14, 2026 · 9 min read

OpenAI-Compatible API with vLLM on Kubernetes

OpenAI-Compatible API with vLLM on Kubernetes guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable,.

Read post

DevOpsSeptember 14, 2026 · 12 min read

Crossplane for AI Infrastructure: Kubernetes-Native IaC for

Learn crossplane for ai infrastructure through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

The Deployment Challenge: Why LLM Serving is Different

Kubernetes GPU Node Setup

EKS GPU Node Group (AWS)

GKE GPU Node Pool (GCP)

vLLM Model Serving

vLLM Deployment

vLLM Configuration Tuning

Horizontal Pod Autoscaling

Custom Metrics with Prometheus

HPA Configuration

Cluster Autoscaler for GPUs

Cluster Autoscaler Configuration

Priority Expander for GPU Nodes

Batch Inference Workloads

Kubernetes Job for Batch Inference

Cost Optimization Strategies

Spot Instances for Batch Workloads

Node Termination Handler

Cost Analysis

Production Monitoring

Prometheus + Grafana Stack

Custom Dashboards

Alerting Rules

Deploy LLMs on Kubernetes Decision Table

Operating Deploy LLMs on Kubernetes as a System

Define the failure model

Measure the decision, not just the component

Roll out with reversible steps

Capacity and cost controls

Production readiness review

Operating Deploy LLMs on Kubernetes as a System

Define the failure model

Measure the decision, not just the component

Roll out with reversible steps

Capacity and cost controls

Production readiness review

Frequently Asked Questions

What GPU should I use for LLM serving?

Should I use spot instances for real-time serving?

How do I prevent GPU idle time?

What's the right HPA metric?

How do I handle model loading time?

Can I run multiple models on one GPU?

Conclusion

Related articles

GPU Scheduling in Kubernetes: Complete NVIDIA Guide for ML

QLoRA: Fine-Tune 70B Models on Single GPU with 4-bit

OpenAI-Compatible API with vLLM on Kubernetes

Crossplane for AI Infrastructure: Kubernetes-Native IaC for