HinterBuild logoHinterBuild
Back to blog
MLOps · 9 min read

GPU Scheduling in Kubernetes: Complete NVIDIA Guide for ML

Learn gpu scheduling in kubernetes through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Muhammad Abdul Sami, author

Muhammad Abdul Sami

· 9 min read

  • Kubernetes
  • DevOps
  • MLOps
  • Observability

Table of Contents:

GPU Scheduling Challenges: Why Default Kubernetes Fails

Short answer: Kubernetes treats GPUs as opaque resources—it can't handle GPU topology, partial allocation, or multi-tenancy. The fix is NVIDIA device plugin with time-slicing, MIG, and custom scheduling policies.

A ML platform ran 200 training jobs/day on 40 GPUs. Jobs queued for hours despite GPUs sitting idle because small jobs couldn't share GPUs with large jobs. We implemented GPU time-slicing and MIG—10 jobs now run concurrently per GPU. Queue time dropped from 4 hours to 15 minutes.

Key Takeaways:

  • NVIDIA Device Plugin exposes GPUs as Kubernetes resources
  • Resource requests ensure pods get exclusive GPU access
  • Node taints prevent non-GPU workloads from wasting GPU nodes
  • Time-slicing enables GPU oversubscription for bursty workloads
  • MIG provides hardware isolation for multi-tenant GPU sharing
  • Topology awareness optimizes multi-GPU placement

For production ML systems, GPU scheduling is critical infrastructure.

NVIDIA Device Plugin Setup

NVIDIA Device Plugin exposes GPUs as schedulable resources in Kubernetes.

Installation on EKS/GKE

bash
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.5/nvidia-device-plugin.yml

# Verify installation
kubectl get daemonset -n kube-system nvidia-device-plugin-daemonset

# Check GPU nodes
kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

Custom Configuration

yaml
# nvidia-device-plugin-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
  namespace: kube-system
data:
  config.yaml: |
    version: v1
    flags:
      migStrategy: mixed  # Support both MIG and full GPU
      failOnInitError: true
      nvidiaDriverRoot: /
      gdsEnabled: false
      mofedEnabled: false
    
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4  # Allow 4 pods per GPU
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      nodeSelector:
        accelerator: nvidia
      
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      
      priorityClassName: system-node-critical
      
      containers:
      - name: nvidia-device-plugin
        image: nvcr.io/nvidia/k8s-device-plugin:v0.14.5
        
        args:
        - --mig-strategy=mixed
        - --pass-device-specs=true
        - --config-file=/etc/nvidia/config.yaml
        
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
        - name: config
          mountPath: /etc/nvidia
        
        securityContext:
          privileged: true
      
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins
      - name: config
        configMap:
          name: nvidia-device-plugin-config
bash
# Apply custom configuration
kubectl apply -f nvidia-device-plugin-config.yaml

# Restart device plugin
kubectl rollout restart daemonset -n kube-system nvidia-device-plugin-daemonset

Connect to Kubernetes platform engineering.

Resource Requests and Limits

Request GPUs to ensure exclusive access and proper scheduling.

Basic GPU Request

yaml
# gpu-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: gpu-training-job
spec:
  restartPolicy: Never
  
  containers:
  - name: trainer
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    
    command:
    - python
    - train.py
    
    resources:
      requests:
        nvidia.com/gpu: 1  # Request 1 GPU
        memory: 16Gi
        cpu: 8
      limits:
        nvidia.com/gpu: 1  # Limit to 1 GPU
        memory: 16Gi

Multi-GPU Request

yaml
# multi-gpu-training.yaml
apiVersion: v1
kind: Pod
metadata:
  name: distributed-training
spec:
  restartPolicy: Never
  
  containers:
  - name: trainer
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    
    command:
    - torchrun
    - --nproc_per_node=4
    - train.py
    
    resources:
      requests:
        nvidia.com/gpu: 4  # Request 4 GPUs on same node
        memory: 64Gi
        cpu: 32
      limits:
        nvidia.com/gpu: 4
        memory: 64Gi
    
    env:
    - name: NCCL_DEBUG
      value: INFO

GPU Selection by Type

yaml
# gpu-type-selection.yaml
apiVersion: v1
kind: Pod
metadata:
  name: inference-a10
spec:
  nodeSelector:
    nvidia.com/gpu.product: NVIDIA-A10G  # Require A10G GPUs
  
  containers:
  - name: inference
    image: vllm/vllm-openai:latest
    
    resources:
      requests:
        nvidia.com/gpu: 1
      limits:
        nvidia.com/gpu: 1

Validation Script

python
# validate_gpu_allocation.py
import subprocess
import json

def check_gpu_allocation() -> dict:
    """Verify GPU allocation in pod."""
    
    # Check CUDA devices visible
    import torch
    gpu_count = torch.cuda.device_count()
    
    if gpu_count == 0:
        return {"status": "error", "message": "No GPUs detected"}
    
    # Get GPU details
    gpus = []
    for i in range(gpu_count):
        props = torch.cuda.get_device_properties(i)
        gpus.append({
            "device_id": i,
            "name": props.name,
            "total_memory_gb": props.total_memory / 1e9,
            "compute_capability": f"{props.major}.{props.minor}",
        })
    
    # Run nvidia-smi
    result = subprocess.run(
        ["nvidia-smi", "--query-gpu=index,name,memory.total", "--format=csv,noheader"],
        capture_output=True,
        text=True,
    )
    
    return {
        "status": "success",
        "gpu_count": gpu_count,
        "gpus": gpus,
        "nvidia_smi": result.stdout,
    }

if __name__ == "__main__":
    result = check_gpu_allocation()
    print(json.dumps(result, indent=2))

Node Affinity and Taints

Taints prevent CPU workloads from stealing GPU nodes.

Taint GPU Nodes

bash
# Taint all GPU nodes
kubectl get nodes -l accelerator=nvidia -o name | \
  xargs -I {} kubectl taint node {} nvidia.com/gpu=true:NoSchedule

# Verify taints
kubectl describe nodes -l accelerator=nvidia | grep Taints

Pod Toleration

yaml
# gpu-pod-with-toleration.yaml
apiVersion: v1
kind: Pod
metadata:
  name: gpu-workload
spec:
  # Tolerate GPU node taint
  tolerations:
  - key: nvidia.com/gpu
    operator: Equal
    value: "true"
    effect: NoSchedule
  
  # Schedule on GPU nodes
  nodeSelector:
    accelerator: nvidia
  
  containers:
  - name: worker
    image: my-gpu-image
    resources:
      requests:
        nvidia.com/gpu: 1
      limits:
        nvidia.com/gpu: 1

Node Affinity for GPU Types

yaml
# gpu-node-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: inference
  template:
    metadata:
      labels:
        app: inference
    spec:
      # Advanced affinity rules
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              # Require A100 or A10G
              - key: nvidia.com/gpu.product
                operator: In
                values:
                - NVIDIA-A100-SXM4-40GB
                - NVIDIA-A10G
          
          preferredDuringSchedulingIgnoredDuringExecution:
          # Prefer A100
          - weight: 100
            preference:
              matchExpressions:
              - key: nvidia.com/gpu.product
                operator: In
                values:
                - NVIDIA-A100-SXM4-40GB
      
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      
      containers:
      - name: inference
        image: my-inference-image
        resources:
          requests:
            nvidia.com/gpu: 1
          limits:
            nvidia.com/gpu: 1

GPU Time-Slicing for Oversubscription

Time-slicing allows multiple pods to share one GPU through time-multiplexing.

Enable Time-Slicing

yaml
# time-slicing-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
  namespace: kube-system
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: false
        resources:
        - name: nvidia.com/gpu
          replicas: 8  # Each GPU can be shared by 8 pods
        
        # Create separate resource for exclusive access
        - name: nvidia.com/gpu-exclusive
          replicas: 1
bash
# Apply time-slicing config
kubectl apply -f time-slicing-config.yaml

# Restart device plugin
kubectl rollout restart daemonset -n kube-system nvidia-device-plugin-daemonset

# Verify time-sliced GPUs available
kubectl describe node <gpu-node> | grep nvidia.com/gpu
# Should show: nvidia.com/gpu: 32 (if 4 GPUs x 8 replicas)

Use Time-Sliced GPU

yaml
# time-sliced-inference.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-time-sliced
spec:
  replicas: 16  # Can run 16 replicas on 4 GPUs (4x4)
  selector:
    matchLabels:
      app: inference-ts
  template:
    metadata:
      labels:
        app: inference-ts
    spec:
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      
      containers:
      - name: inference
        image: my-inference-image
        
        resources:
          requests:
            nvidia.com/gpu: 1  # Gets 1/8 of a physical GPU
            memory: 4Gi
            cpu: 2
          limits:
            nvidia.com/gpu: 1
            memory: 4Gi

Time-Slicing Performance Test

python
# benchmark_time_slicing.py
import torch
import time
from concurrent.futures import ThreadPoolExecutor

def benchmark_inference(model, input_size=(1, 3, 224, 224), iterations=100):
    """Benchmark inference latency."""
    
    model = model.cuda()
    model.eval()
    
    # Warmup
    for _ in range(10):
        with torch.no_grad():
            dummy_input = torch.randn(*input_size).cuda()
            _ = model(dummy_input)
    
    # Benchmark
    torch.cuda.synchronize()
    start = time.perf_counter()
    
    for _ in range(iterations):
        with torch.no_grad():
            dummy_input = torch.randn(*input_size).cuda()
            _ = model(dummy_input)
    
    torch.cuda.synchronize()
    elapsed = time.perf_counter() - start
    
    avg_latency_ms = (elapsed / iterations) * 1000
    throughput = iterations / elapsed
    
    return {
        "avg_latency_ms": avg_latency_ms,
        "throughput_inferences_per_sec": throughput,
    }

# Run benchmark
from torchvision.models import resnet50
model = resnet50(pretrained=False)

results = benchmark_inference(model, iterations=100)
print(f"Avg latency: {results['avg_latency_ms']:.2f} ms")
print(f"Throughput: {results['throughput_inferences_per_sec']:.2f} inferences/sec")

# Expected on time-sliced GPU:
# - Latency increases 2-4x vs dedicated GPU
# - Throughput per pod decreases proportionally
# - But total cluster throughput increases

Use cases for time-slicing:

  • Development/testing environments
  • Inference workloads with low GPU utilization (<30%)
  • Bursty workloads with idle periods
  • Cost-sensitive non-production workloads

Multi-Instance GPU (MIG)

MIG provides hardware-level isolation on A100/A30/H100 GPUs.

Enable MIG on Nodes

bash
# SSH to GPU node
# Enable MIG mode (requires reboot)
sudo nvidia-smi -i 0 -mig 1

# Reboot
sudo reboot

# After reboot, create MIG instances
# Split A100 into 7 instances (1g.5gb each)
sudo nvidia-smi mig -cgi 19,19,19,19,19,19,19 -C

# Verify MIG instances
nvidia-smi mig -lgi

# Output:
# +----+----------------+-------+-----------+
# | ID | GPU Instance   | Name  | Placement |
# +====+================+=======+===========+
# |  0 | GI ID:  0      | 1g.5gb| 0         |
# |  1 | GI ID:  1      | 1g.5gb| 1         |
# ...

MIG Device Plugin Configuration

yaml
# mig-device-plugin-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
  namespace: kube-system
data:
  config.yaml: |
    version: v1
    flags:
      migStrategy: mixed  # Support MIG and full GPU
    
    sharing:
      mig:
        strategy: mixed
    
    resources:
      - pattern: "*"
        devices: all

Use MIG Instance

yaml
# mig-training-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: mig-training
spec:
  template:
    spec:
      restartPolicy: Never
      
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      
      containers:
      - name: trainer
        image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
        
        command:
        - python
        - train.py
        
        resources:
          requests:
            nvidia.com/mig-1g.5gb: 1  # Request 1g.5gb MIG instance
            memory: 8Gi
            cpu: 4
          limits:
            nvidia.com/mig-1g.5gb: 1
            memory: 8Gi

MIG vs Time-Slicing Comparison

python
# compare_mig_vs_timeslicing.py
from dataclasses import dataclass

@dataclass
class IsolationMethod:
    name: str
    isolation: str
    memory_isolation: bool
    compute_isolation: bool
    overhead: str
    use_case: str

methods = [
    IsolationMethod(
        name="Dedicated GPU",
        isolation="Hardware",
        memory_isolation=True,
        compute_isolation=True,
        overhead="None",
        use_case="Production inference, training",
    ),
    IsolationMethod(
        name="MIG",
        isolation="Hardware",
        memory_isolation=True,
        compute_isolation=True,
        overhead="5-10%",
        use_case="Multi-tenant production, strict SLAs",
    ),
    IsolationMethod(
        name="Time-Slicing",
        isolation="Time-multiplexed",
        memory_isolation=False,
        compute_isolation=False,
        overhead="20-50%",
        use_case="Dev/test, bursty workloads",
    ),
]

import pandas as pd
df = pd.DataFrame([vars(m) for m in methods])
print(df.to_string(index=False))

MIG advantages:

  • True memory isolation (no OOM from neighbors)
  • Predictable performance
  • ECC memory per instance
  • Quality of service guarantees

MIG limitations:

  • Only on A100/A30/H100
  • Fixed instance sizes (1g, 2g, 3g, 4g, 7g)
  • Requires GPU reset to change configuration
  • Not supported by all CUDA applications

Multi-Tenant GPU Isolation

Isolate GPUs by namespace and enforce resource quotas.

Namespace Resource Quotas

yaml
# namespace-quota.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: team-ml
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: team-ml
spec:
  hard:
    requests.nvidia.com/gpu: "8"  # Max 8 GPUs
    limits.nvidia.com/gpu: "8"
    requests.memory: "256Gi"
    requests.cpu: "128"
  
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values:
      - high
      - medium
---
apiVersion: v1
kind: LimitRange
metadata:
  name: gpu-limits
  namespace: team-ml
spec:
  limits:
  - max:
      nvidia.com/gpu: "4"  # Max 4 GPUs per pod
      memory: "128Gi"
    min:
      nvidia.com/gpu: "0"
      memory: "1Gi"
    default:
      memory: "16Gi"
    defaultRequest:
      memory: "16Gi"
    type: Container

Priority Classes for GPU Scheduling

yaml
# priority-classes.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-high-priority
value: 1000000
globalDefault: false
description: "High priority GPU workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-medium-priority
value: 100000
globalDefault: false
description: "Medium priority GPU workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-low-priority
value: 10000
globalDefault: true
description: "Low priority GPU workloads (preemptible)"

Preemptible GPU Jobs

yaml
# preemptible-training.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: preemptible-training
  namespace: team-ml
spec:
  template:
    spec:
      priorityClassName: gpu-low-priority  # Can be preempted
      
      restartPolicy: OnFailure
      
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      
      containers:
      - name: trainer
        image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
        
        command:
        - python
        - train.py
        - --checkpoint-interval=100  # Frequent checkpoints for preemption
        
        resources:
          requests:
            nvidia.com/gpu: 2
            memory: 32Gi
          limits:
            nvidia.com/gpu: 2
            memory: 32Gi

GPU Topology Awareness

Optimize multi-GPU placement for interconnect bandwidth.

Topology Discovery

bash
# Check GPU topology
nvidia-smi topo -m

# Output shows NVLink/PCIe connections
#      GPU0  GPU1  GPU2  GPU3  NIC0
# GPU0   X   NV12  NV12  NV12  SYS
# GPU1  NV12   X   NV12  NV12  SYS
# GPU2  NV12  NV12   X   NV12  SYS
# GPU3  NV12  NV12  NV12   X   SYS

Topology-Aware Scheduling

yaml
# topology-aware-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: distributed-training-nvlink
spec:
  # Ensure all GPUs on same node (for NVLink)
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: distributed-training
        topologyKey: kubernetes.io/hostname
  
  containers:
  - name: trainer
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    
    command:
    - torchrun
    - --nproc_per_node=4
    - train.py
    
    resources:
      requests:
        nvidia.com/gpu: 4
      limits:
        nvidia.com/gpu: 4
    
    env:
    - name: NCCL_DEBUG
      value: INFO
    - name: NCCL_IB_DISABLE
      value: "0"
    - name: NCCL_NET_GDR_LEVEL
      value: "5"

Bandwidth Benchmark

python
# benchmark_gpu_interconnect.py
import torch
import torch.distributed as dist
import time

def benchmark_all_reduce(size_mb: int, iterations: int = 100):
    """Benchmark all-reduce bandwidth."""
    
    if not dist.is_initialized():
        dist.init_process_group(backend="nccl")
    
    rank = dist.get_rank()
    world_size = dist.get_world_size()
    
    # Create tensor
    num_elements = (size_mb * 1024 * 1024) // 4  # float32
    tensor = torch.randn(num_elements).cuda()
    
    # Warmup
    for _ in range(10):
        dist.all_reduce(tensor)
    
    torch.cuda.synchronize()
    
    # Benchmark
    start = time.perf_counter()
    for _ in range(iterations):
        dist.all_reduce(tensor)
    torch.cuda.synchronize()
    elapsed = time.perf_counter() - start
    
    # Calculate bandwidth
    data_size_mb = size_mb * iterations
    bandwidth_gbps = (data_size_mb / 1024) / elapsed
    
    if rank == 0:
        print(f"All-reduce bandwidth: {bandwidth_gbps:.2f} GB/s")
        print(f"Expected: ~300 GB/s (NVLink), ~25 GB/s (PCIe)")
    
    dist.destroy_process_group()

if __name__ == "__main__":
    benchmark_all_reduce(size_mb=128, iterations=100)

Integrate with AI platform architectures.

Related implementation guides:

Primary references: official documentation, official documentation, official documentation, official documentation.

GPU Scheduling in Kubernetes Decision Table

DecisionPrefer the simpler path whenAdd operational complexity when
ArchitectureOne component can own the contract and stateIndependent scaling or fault isolation is required
RolloutOffline replay covers the meaningful casesLive behavior requires shadow traffic and a canary
RecoveryA failed operation is safe to repeatPartial effects require idempotency or reconciliation
MeasurementOne service objective represents user impactQuality, latency, and cost need separate gates

Operating GPU Scheduling in Kubernetes as a System

The implementation is only one part of GPU Scheduling in Kubernetes. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship GPU Scheduling in Kubernetes behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our GPU Scheduling in Kubernetes engineering support.

Frequently Asked Questions

How many GPUs can Kubernetes schedule per node?

Limited by hardware. Typical configs: 1 GPU (g4dn), 4 GPUs (g5.12xlarge), 8 GPUs (p4d.24xlarge). Kubernetes has no architectural limit.

Should I use time-slicing or MIG?

Use MIG for multi-tenant production (strict isolation). Use time-slicing for dev/test or cost-sensitive workloads where isolation isn't critical.

Can I dynamically reconfigure MIG?

No—MIG requires GPU reset. Plan MIG profiles at cluster setup. For dynamic workloads, use time-slicing or dedicated GPU pools.

How do I prevent GPU idle time?

Implement job queuing (Kubeflow, Argo Workflows), autoscaling to zero when idle, and time-slicing for small workloads.

What's the overhead of GPU time-slicing?

20-50% per pod depending on workload contention. Total cluster throughput increases despite per-pod overhead.

How do I monitor GPU utilization?

Use DCGM exporter with Prometheus and Grafana. Track DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, and DCGM_FI_PROF_SM_ACTIVE.

Conclusion

GPU scheduling in Kubernetes enables efficient ML infrastructure:

  • NVIDIA Device Plugin exposes GPUs as schedulable resources
  • Resource requests ensure exclusive GPU allocation
  • Taints and tolerations prevent CPU workloads from wasting GPU nodes
  • Time-slicing enables oversubscription for bursty workloads
  • MIG provides hardware isolation for multi-tenant systems
  • Topology awareness optimizes multi-GPU placement

Proper GPU scheduling increases utilization 2-5x while maintaining isolation.

At HinterBuild, we build production GPU infrastructure:

Contact us for GPU scheduling consulting.

Free consultation

Book a free consultation call on GPU scheduling & Kubernetes

30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.

Book a meeting

Keep reading

Related articles

MLOps · 12 min read

Deploy LLMs on Kubernetes: Complete GPU Autoscaling Guide

Learn deploy llms on kubernetes through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

AI Systems · 9 min read

QLoRA: Fine-Tune 70B Models on Single GPU with 4-bit

QLoRA guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Read post

AI Systems · 9 min read

OpenAI-Compatible API with vLLM on Kubernetes

OpenAI-Compatible API with vLLM on Kubernetes guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable,.

Read post

DevOps · 12 min read

Crossplane for AI Infrastructure: Kubernetes-Native IaC for

Learn crossplane for ai infrastructure through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post