DevOpsSeptember 14, 2026 · 14 min read

ArgoCD for ML Model Deployments: Production GitOps Patterns

Q: Can I roll back a model instantly with ArgoCD?

Yes. ArgoCD keeps the manifests of previous syncs, so argocd app rollback <app <revision or a git revert redeploys the prior version in seconds, and Argo Rollouts' undo does the same for a Rollout. The pods still need to load the old model, so "instant" means "as fast as your startup time" — for large models, blue-green keeps the old version warm so the cutover is truly immediate.

Deploy ML models with ArgoCD GitOps: declarative manifests, MLflow registry sync, Kustomize overlays, Argo Rollouts canaries, and automated rollbacks.

Muhammad Abdul Sami

September 14, 2026 · 14 min read

MLOps
Kubernetes
DevOps
CI/CD
Observability

ArgoCD for ML model deployments solves the problem that kills most model serving setups: nobody can say with certainty which model version is running in production, how it got there, or how to get the previous one back. GitOps makes the Git repository the single source of truth for every serving Deployment, HPA, and traffic split, and ArgoCD continuously reconciles the cluster toward it. This guide covers the architecture, MLflow registry integration, Kustomize environment overlays, Argo Rollouts canaries with metric-driven rollback, and the failure modes we have hit running it for Kubernetes platform engineering clients.

Key Takeaways:

GitOps makes models infrastructure-as-code — every model version, replica count, and traffic weight is a Git commit you can diff and revert

Keep model weights out of Git: the registry (MLflow, S3) owns artifacts, Git owns the pointer and the serving config

Argo Rollouts AnalysisTemplates turn Prometheus accuracy and error-rate queries into automatic rollback triggers

Kustomize overlays per environment prevent staging/production drift without duplicating manifests

Model pods need longer readiness windows than web services — tune probes or canaries will roll back on cold starts

Set ignoreDifferences on /spec/replicas or ArgoCD and the HPA will fight each other indefinitely

Table of Contents:

Why GitOps for ML Models
ArgoCD Architecture for ML
Model Registry Integration
Declarative Model Serving
Multi-Environment Strategy
Automated Rollbacks & Canary
Monitoring & Observability
Production Deployment Patterns
Frequently Asked Questions

Why GitOps for ML Models: Beyond Manual kubectl

Short answer: ML model deployments fail because teams use manual scripts, inconsistent environments, and zero rollback strategy. GitOps with ArgoCD provides declarative infrastructure, automated deployments, and instant rollbacks for production model serving.

A fintech company deployed 40+ fraud detection models using custom Python scripts—every deployment required manual coordination, production outages happened weekly, and rollbacks meant SSH into nodes. We migrated them to ArgoCD with declarative model configs. New deployment process: push to Git, automated validation, zero-downtime rollout. Production incidents dropped 85%.

Push vs Pull: Why ArgoCD Deploys Differently

Most CI pipelines push to the cluster: a GitHub Actions job runs kubectl apply with cluster credentials. ArgoCD pulls: an in-cluster controller watches Git and applies what it finds. The difference matters more for ML than for stateless web services, because model deployments are changed by more people (data scientists, platform engineers, on-call) and through more channels (training pipelines, hotfix scripts, notebooks).

Approach	Source of truth	Drift detection	Rollback	Cluster credentials in CI
Manual `kubectl` / scripts	Whoever ran it last	None	Re-run old script, hope it still works	Yes, on laptops
CI push (`kubectl apply` in a job)	Git, until someone edits live	None between runs	Re-run pipeline at old commit	Yes, in CI secrets
GitOps pull (ArgoCD)	Git, enforced continuously	Continuous; `selfHeal` reverts live edits	`git revert` or `argocd app rollback`	No — controller runs in-cluster

The audit trail falls out for free: every production change is a commit with an author, a reviewer, and a diff. For regulated ML (credit, fraud, healthcare) that is often the reason the migration gets funded, not the operational benefits.

For production AI systems, treating model deployments like application deployments is non-negotiable. If you are still deciding how to version the models themselves, read our DVC and MLflow model versioning guide first — GitOps assumes an immutable, addressable artifact already exists.

ArgoCD Architecture for ML Model Serving

ArgoCD synchronizes Git repository state to Kubernetes cluster state—continuously reconciling declared vs actual infrastructure. The ArgoCD documentation describes the reconciliation loop in detail; what follows is the repository layout and Application spec we use for model serving.

yaml

├── applications/
│   ├── fraud-detection-app.yaml     # ArgoCD Application
│   ├── recommendation-app.yaml
│   └── sentiment-app.yaml
├── models/
│   ├── fraud-detection/
│   │   ├── deployment.yaml          # Model serving config
│   │   ├── service.yaml
│   │   ├── hpa.yaml                 # Autoscaling
│   │   └── kustomization.yaml
│   ├── recommendation/
│   │   ├── deployment.yaml
│   │   └── ...
│   └── base/
│       └── model-serving-base.yaml  # Shared configs
└── environments/
    ├── staging/
    │   └── kustomization.yaml       # Staging overlays
    └── production/
        └── kustomization.yaml       # Production overlays

Core components:

Application - ArgoCD resource pointing to Git repo + cluster target
Manifests - Kubernetes resources (Deployments, Services, ConfigMaps)
Sync Policy - Automated vs manual sync, self-heal, prune
Health Check - Custom model serving health definitions

yaml

# applications/fraud-detection-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: fraud-detection
  namespace: argocd
spec:
  project: ml-models
  
  source:
    repoURL: https://github.com/company/ml-gitops
    targetRevision: main
    path: models/fraud-detection
    
    # Kustomize build
    kustomize:
      namePrefix: fraud-
      commonLabels:
        app: fraud-detection
        team: ml-platform
  
  destination:
    server: https://kubernetes.default.svc
    namespace: ml-production
  
  syncPolicy:
    automated:
      prune: true        # Delete resources not in Git
      selfHeal: true     # Revert manual changes
      allowEmpty: false
    
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true   # Delete old resources after new ones ready
    
    retry:
      limit: 5
      backoff:
        duration: 5s
        maxDuration: 3m
        factor: 2
  
  # Health checks for model serving
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # Ignore HPA replica changes

Sync Policy Decisions That Matter for Model Serving

Three settings in the spec above are where ML teams get burned:

prune: true deletes any resource ArgoCD created that is no longer in Git. Correct for Deployments; dangerous if someone hand-creates a PersistentVolumeClaim holding a cached model and forgets to commit it. Keep model caches in resources that are in Git, or in a separate Application with prune: false.
selfHeal: true reverts live edits within seconds. That is the point of GitOps, but it also means an on-call engineer's emergency kubectl scale will be undone. The fix is not to disable it — it is to make the emergency path a fast Git commit (a one-line replica change on a protected branch with a single required reviewer).
ignoreDifferences on /spec/replicas is mandatory when an HPA manages replicas. Without it, ArgoCD sees the HPA's scale-up as drift, resets replicas to the Git value, and the HPA scales up again — a loop that shows as permanent OutOfSync and constant pod churn.

For many models, use the app-of-apps pattern: one root Application points at the applications/ directory, and each model gets its own child Application. Sync waves (argocd.argoproj.io/sync-wave annotations) let you deploy shared resources — the model-server base ConfigMap, the ServiceMonitor — before the models that depend on them.

Deploy ArgoCD on Kubernetes infrastructure with proper RBAC. If your serving images are packaged as Helm charts rather than Kustomize, ArgoCD handles both — our Helm charts from scratch guide covers the chart side.

Model Registry Integration: MLflow + ArgoCD

Separate model artifacts from deployment config—model weights in MLflow/registry, configs in Git. Git is not a blob store: a 2 GB weights file per commit makes clones unusable within weeks, and Git LFS only partially helps. The registry owns the artifact and its lineage (run ID, training data hash, metrics); Git owns the pointer to it and everything about how it is served.

Note that the stages API used below (Production, Staging) is deprecated in recent MLflow releases in favour of model version aliases (@champion, @challenger). The pattern is identical — resolve an alias to a concrete version, write the version into the manifest — only the client call changes.

python

# sync-model-registry.py
"""Sync latest model versions from MLflow to ArgoCD configs."""
import mlflow
from mlflow.tracking import MlflowClient
import yaml
from pathlib import Path
from datetime import datetime, timezone

class ModelRegistrySync:
    """Sync MLflow registry to ArgoCD manifests."""
    
    def __init__(self, tracking_uri: str, git_repo_path: Path):
        mlflow.set_tracking_uri(tracking_uri)
        self.client = MlflowClient()
        self.repo_path = git_repo_path
    
    def get_production_model(self, model_name: str) -> dict:
        """Get latest production model version."""
        versions = self.client.get_latest_versions(
            name=model_name,
            stages=["Production"]
        )
        
        if not versions:
            raise ValueError(f"No production version for {model_name}")
        
        latest = versions[0]
        
        return {
            "name": model_name,
            "version": latest.version,
            "run_id": latest.run_id,
            "artifact_uri": latest.source,
            "metrics": mlflow.get_run(latest.run_id).data.metrics,
        }
    
    def update_deployment_manifest(
        self,
        model_name: str,
        model_info: dict,
    ) -> None:
        """Update ArgoCD deployment with new model version."""
        manifest_path = (
            self.repo_path / "models" / model_name / "deployment.yaml"
        )
        
        with manifest_path.open() as f:
            deployment = yaml.safe_load(f)
        
        # Update model version in container env
        containers = deployment["spec"]["template"]["spec"]["containers"]
        
        for container in containers:
            if container["name"] == "model-server":
                # Update env vars
                env_vars = {e["name"]: e for e in container["env"]}
                env_vars["MODEL_VERSION"] = {
                    "name": "MODEL_VERSION",
                    "value": str(model_info["version"])
                }
                env_vars["MODEL_URI"] = {
                    "name": "MODEL_URI",
                    "value": model_info["artifact_uri"]
                }
                env_vars["MODEL_METRICS_F1"] = {
                    "name": "MODEL_METRICS_F1",
                    "value": str(model_info["metrics"].get("f1_score", "N/A"))
                }
                
                container["env"] = list(env_vars.values())
                
                # Update image tag with model version
                base_image = container["image"].split(":")[0]
                container["image"] = f"{base_image}:{model_info['version']}"
        
        # Add annotation for tracking
        annotations = deployment["metadata"].setdefault("annotations", {})
        annotations["ml.hinterbuild.com/model-version"] = str(model_info["version"])
        annotations["ml.hinterbuild.com/synced-at"] = datetime.now(timezone.utc).isoformat()
        
        # Write updated manifest
        with manifest_path.open("w") as f:
            yaml.safe_dump(deployment, f, default_flow_style=False)
        
        print(f"✓ Updated {model_name} to version {model_info['version']}")
    
    def sync_all_models(self, model_names: list[str]) -> None:
        """Sync all production models."""
        for model_name in model_names:
            try:
                model_info = self.get_production_model(model_name)
                self.update_deployment_manifest(model_name, model_info)
            except Exception as e:
                print(f"✗ Failed to sync {model_name}: {e}")

# Usage
sync = ModelRegistrySync(
    tracking_uri="https://mlflow.company.com",
    git_repo_path=Path("/path/to/ml-gitops-repo")
)

sync.sync_all_models([
    "fraud-detection",
    "recommendation-engine",
    "sentiment-classifier"
])

# Commit and push changes
# git add models/*/deployment.yaml
# git commit -m "feat: sync model versions from registry"
# git push origin main
# ArgoCD automatically detects and deploys

Automated pipeline: MLflow transition to Production → GitHub Action runs sync script → ArgoCD deploys.

Two details make this robust in practice. First, the sync script should open a pull request, not push to main, so a bad registry transition cannot reach production without a human glance at the diff. Second, pin the artifact by run ID or content hash, not by version number alone — a re-registered "version 7" that points at different weights is a silent break that only a hash catches.

Connect with data pipelines for end-to-end ML workflows.

Declarative Model Serving with Kustomize

Use Kustomize for environment-specific configurations without duplication. The base manifest defines the serving contract — ports, probes, metrics endpoint, resource shape — and each environment overlay patches only what differs.

yaml

# models/base/model-serving-base.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: model-server
  template:
    metadata:
      labels:
        app: model-server
    spec:
      containers:
      - name: model-server
        image: company/model-server:latest
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 9090
          name: metrics
        
        env:
        - name: MODEL_NAME
          value: OVERRIDE
        - name: MODEL_VERSION
          value: OVERRIDE
        - name: MODEL_URI
          value: OVERRIDE
        
        resources:
          requests:
            cpu: 500m
            memory: 2Gi
          limits:
            cpu: 2000m
            memory: 4Gi
        
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5

---
apiVersion: v1
kind: Service
metadata:
  name: model-server
spec:
  selector:
    app: model-server
  ports:
  - name: http
    port: 80
    targetPort: 8080
  - name: metrics
    port: 9090
    targetPort: 9090

---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: model-server
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: inference_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"

Environment overlays:

yaml

# environments/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: ml-staging

bases:
  - ../../models/base

namePrefix: staging-

commonLabels:
  environment: staging

replicas:
  - name: model-server
    count: 1  # Lower replicas in staging

resources:
  - ingress-staging.yaml

patchesStrategicMerge:
  - model-config-staging.yaml

# model-config-staging.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
spec:
  template:
    spec:
      containers:
      - name: model-server
        resources:
          requests:
            cpu: 250m      # Lower resources
            memory: 1Gi
          limits:
            cpu: 1000m
            memory: 2Gi

yaml

# environments/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: ml-production

bases:
  - ../../models/base

namePrefix: prod-

commonLabels:
  environment: production

replicas:
  - name: model-server
    count: 3  # High availability

resources:
  - ingress-production.yaml
  - pdb.yaml  # Pod Disruption Budget

patchesStrategicMerge:
  - model-config-production.yaml

# pdb.yaml - prevent all pods being evicted
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: model-server-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: model-server

Test locally before pushing:

bash

# Validate Kustomize build
kustomize build environments/staging/
kustomize build environments/production/

# Dry-run apply
kubectl apply --dry-run=client -k environments/staging/

Integrate with cloud infrastructure automation.

Multi-Environment Promotion Strategy

Progressive promotion: dev → staging → production with validation gates. Promotion is a Git operation — copying or re-pointing the production overlay at the version that passed staging — and the gates run before that commit is merged.

yaml

# .github/workflows/promote-model.yaml
name: Promote Model to Production

on:
  workflow_dispatch:
    inputs:
      model_name:
        description: 'Model to promote'
        required: true
        type: choice
        options:
          - fraud-detection
          - recommendation-engine
          - sentiment-classifier
      from_environment:
        description: 'Source environment'
        required: true
        type: choice
        options:
          - staging
      to_environment:
        description: 'Target environment'
        required: true
        type: choice
        options:
          - production

jobs:
  validate-staging:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run validation tests
        run: |
          # Check model metrics in staging
          python scripts/validate-model-metrics.py \
            --model ${{ inputs.model_name }} \
            --environment staging \
            --min-accuracy 0.92 \
            --min-f1 0.88
      
      - name: Load test staging endpoint
        run: |
          # Ensure staging handles production-like load
          k6 run \
            --vus 100 \
            --duration 5m \
            tests/load-test.js \
            --env MODEL=${{ inputs.model_name }} \
            --env ENVIRONMENT=staging
  
  promote:
    needs: validate-staging
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Copy staging config to production
        run: |
          MODEL="${{ inputs.model_name }}"
          
          # Copy deployment manifest
          cp "environments/staging/${MODEL}-deployment.yaml" \
             "environments/production/${MODEL}-deployment.yaml"
          
          # Update production-specific settings
          python scripts/apply-production-overrides.py \
            --file "environments/production/${MODEL}-deployment.yaml"
      
      - name: Create promotion PR
        uses: peter-evans/create-pull-request@v5
        with:
          commit-message: "feat: promote ${{ inputs.model_name }} to production"
          title: "Promote ${{ inputs.model_name }} to Production"
          body: |
            ## Model Promotion
            
            - Model: ${{ inputs.model_name }}
            - From: staging
            - To: production
            
            ### Staging Metrics
            [Auto-populated from validation]
            
            ### Checklist
            - [ ] Metrics exceed production thresholds
            - [ ] Load test passed
            - [ ] Security scan clean
            - [ ] Rollback plan documented
          branch: promote-${{ inputs.model_name }}-prod
          labels: model-promotion,production

Manual approval gate before production merge protects against bad models.

For multi-cluster deployments, replicate across regions with ArgoCD ApplicationSets.

Automated Rollbacks & Canary Deployments

Instant rollback when model performance degrades. Argo Rollouts replaces the standard Deployment with a Rollout resource that supports weighted canaries and metric-driven analysis; ArgoCD manages the Rollout like any other manifest.

Choosing a Rollout Strategy for Models

Strategy	Extra capacity needed	Time to full rollout	Detects bad model before users?	Best for
Rolling update	~1 pod	Minutes	No	Low-risk config changes
Blue-green	2x (both versions live)	Seconds to cut over	Only via smoke tests	Large models where warm-up is slow, instant revert required
Canary (Argo Rollouts)	1 canary pod or a % of fleet	30-60 min with pauses	Yes, on live traffic metrics	Most production models
Shadow / mirror	1x duplicate fleet	Days (offline comparison)	Yes, with zero user exposure	High-stakes models, new architectures

Blue-green doubles GPU cost for the duration of the overlap, which is why we default to canaries for GPU-served models and reserve blue-green for CPU models with slow warm-up. For shadowing, see our shadow mode deployment guide.

The ML-specific catch with analysis-driven rollback is label latency. A fraud model's true accuracy is not known until chargebacks arrive weeks later. The model_accuracy metric in the template below therefore has to be a proxy — agreement with the previous version, prediction distribution drift, or accuracy on a labelled replay set — not ground truth. Pick the proxy deliberately and document what it can and cannot catch.

yaml

# Progressive canary with Argo Rollouts
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: fraud-detection-model
spec:
  replicas: 5
  
  strategy:
    canary:
      steps:
      - setWeight: 10    # Route 10% traffic to new version
      - pause:
          duration: 5m   # Observe metrics
      - setWeight: 25
      - pause:
          duration: 10m
      - setWeight: 50
      - pause:
          duration: 15m
      - setWeight: 100   # Full rollout
      
      # Automatic rollback on metrics
      analysis:
        templates:
        - templateName: model-accuracy-check
        startingStep: 1   # Run after first step
        
  revisionHistoryLimit: 5  # Keep last 5 versions for rollback
  
  selector:
    matchLabels:
      app: fraud-detection
  
  template:
    metadata:
      labels:
        app: fraud-detection
        version: v2.3.0
    spec:
      containers:
      - name: model-server
        image: company/fraud-detection:v2.3.0
        ports:
        - containerPort: 8080

---
# Analysis template for automated rollback
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: model-accuracy-check
spec:
  metrics:
  - name: accuracy
    interval: 1m
    count: 5
    successCondition: result >= 0.92  # Minimum accuracy
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          avg_over_time(
            model_accuracy{
              model="fraud-detection",
              version="v2.3.0"
            }[1m]
          )
  
  - name: error-rate
    interval: 1m
    count: 5
    successCondition: result < 0.01  # Max 1% errors
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(
            model_errors_total{
              model="fraud-detection",
              version="v2.3.0"
            }[1m]
          )) / 
          sum(rate(
            model_requests_total{
              model="fraud-detection",
              version="v2.3.0"
            }[1m]
          ))

Manual rollback:

bash

# Rollback to previous version
kubectl argo rollouts undo fraud-detection-model

# Rollback to specific version
kubectl argo rollouts undo fraud-detection-model --to-revision=3

# Check rollout status
kubectl argo rollouts status fraud-detection-model

Combine with observability tooling for real-time metrics.

Monitoring & Observability for ML GitOps

Track deployment health AND model performance. Deployment health (sync status, pod readiness, rollout phase) tells you whether ArgoCD did what Git said. Model performance (latency, error rate, prediction drift) tells you whether what Git said was a good idea. You need both, and they usually live in the same Prometheus instance.

yaml

# ServiceMonitor for Prometheus scraping
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: model-server-metrics
  namespace: ml-production
spec:
  selector:
    matchLabels:
      app: model-server
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

# Model-specific alerts
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: ml-model-alerts
  namespace: ml-production
spec:
  groups:
  - name: model-performance
    interval: 30s
    rules:
    
    # Accuracy degradation
    - alert: ModelAccuracyDrop
      expr: |
        model_accuracy < 0.92
      for: 5m
      labels:
        severity: critical
        team: ml-platform
      annotations:
        summary: "Model accuracy below threshold"
        description: "{{ $labels.model }} accuracy is {{ $value }}"
    
    # High prediction latency
    - alert: ModelLatencyHigh
      expr: |
        histogram_quantile(0.95,
          rate(model_inference_duration_seconds_bucket[5m])
        ) > 0.5
      for: 3m
      labels:
        severity: warning
      annotations:
        summary: "Model p95 latency > 500ms"
    
    # Deployment sync failures
    - alert: ArgoSyncFailure
      expr: |
        argocd_app_sync_status{sync_status="OutOfSync"} == 1
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "ArgoCD sync failed for {{ $labels.name }}"

Python client for metrics:

python

from prometheus_client import Counter, Histogram, Gauge
import time

# Metrics
model_requests = Counter(
    "model_requests_total",
    "Total model inference requests",
    ["model", "version", "status"]
)

model_latency = Histogram(
    "model_inference_duration_seconds",
    "Model inference latency",
    ["model", "version"]
)

model_accuracy = Gauge(
    "model_accuracy",
    "Current model accuracy",
    ["model", "version"]
)

class InstrumentedModelServer:
    """Model server with Prometheus metrics."""
    
    def __init__(self, model_name: str, version: str):
        self.model_name = model_name
        self.version = version
        self.labels = {"model": model_name, "version": version}
    
    async def predict(self, features: dict) -> dict:
        """Make prediction with instrumentation."""
        start = time.perf_counter()
        
        try:
            # Run inference
            prediction = await self._run_inference(features)
            
            # Record success
            model_requests.labels(
                **self.labels,
                status="success"
            ).inc()
            
            return prediction
        
        except Exception as e:
            # Record failure
            model_requests.labels(
                **self.labels,
                status="error"
            ).inc()
            raise
        
        finally:
            # Record latency
            duration = time.perf_counter() - start
            model_latency.labels(**self.labels).observe(duration)
    
    async def update_accuracy_metric(self, accuracy: float) -> None:
        """Update current accuracy metric."""
        model_accuracy.labels(**self.labels).set(accuracy)

Deploy comprehensive monitoring across all ML infrastructure.

Production Deployment Patterns

Pattern 1: Blue-Green Model Deployment

yaml

# Deploy both versions, switch traffic atomically
apiVersion: v1
kind: Service
metadata:
  name: fraud-detection
spec:
  selector:
    app: fraud-detection
    version: blue  # Switch to 'green' for instant cutover
  ports:
  - port: 80
    targetPort: 8080

---
# Blue deployment (current production)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fraud-detection-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: fraud-detection
      version: blue
  template:
    metadata:
      labels:
        app: fraud-detection
        version: blue
    spec:
      containers:
      - name: model
        image: company/fraud-detection:v2.2.0

---
# Green deployment (new version)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fraud-detection-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: fraud-detection
      version: green
  template:
    metadata:
      labels:
        app: fraud-detection
        version: green
    spec:
      containers:
      - name: model
        image: company/fraud-detection:v2.3.0

# Switch traffic: kubectl patch svc fraud-detection -p '{"spec":{"selector":{"version":"green"}}}'

Pattern 2: Multi-Model Serving

yaml

# Single deployment, multiple models via config
apiVersion: v1
kind: ConfigMap
metadata:
  name: model-configs
data:
  models.yaml: |
    models:
      - name: fraud-detection
        version: v2.3.0
        path: s3://models/fraud/v2.3.0
        replicas: 3
        resources:
          memory: 4Gi
      
      - name: recommendation
        version: v1.5.0
        path: s3://models/recommend/v1.5.0
        replicas: 2
        resources:
          memory: 8Gi

---
# Model server loads all models
apiVersion: apps/v1
kind: Deployment
metadata:
  name: multi-model-server
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: server
        image: company/model-server:latest
        volumeMounts:
        - name: config
          mountPath: /config
      volumes:
      - name: config
        configMap:
          name: model-configs

Pattern 3: A/B Testing Models

yaml

# Istio VirtualService for traffic splitting
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: fraud-detection
spec:
  hosts:
  - fraud-detection.ml-production.svc.cluster.local
  http:
  - match:
    - headers:
        x-user-segment:
          exact: "premium"
    route:
    - destination:
        host: fraud-detection
        subset: v2-new-model
      weight: 100
  
  - route:  # Default: 90% v1, 10% v2
    - destination:
        host: fraud-detection
        subset: v1-stable
      weight: 90
    - destination:
        host: fraud-detection
        subset: v2-new-model
      weight: 10

Failure Modes We See in ML GitOps Repositories

The patterns above fail in predictable ways. These are the ones that show up repeatedly in audits:

image: company/model-server:latest in a base manifest. ArgoCD sees no diff when the tag is re-pushed, so nothing deploys — or worse, a node restart pulls a different image than its neighbours. Pin image digests or version tags; let the sync script bump them.
Readiness probes tuned for web services. A 7B-parameter model can take 90 seconds to load onto a GPU. With initialDelaySeconds: 10 the pod is killed before it ever becomes ready, the canary step fails, and Rollouts aborts a perfectly good model. Use a startup probe with a generous failureThreshold and keep the readiness probe tight for steady state.
Secrets committed as plain values. Registry tokens and S3 credentials in deployment.yaml are the most common finding. Use External Secrets Operator or Sealed Secrets so Git holds only references.
Environment overlays that drift into full copies. When a staging patch grows to 200 lines, the base is no longer the base. Refactor into components or accept that the environments are different applications.
GPU scheduling left to defaults. Canary pods that request a GPU on a full cluster stay Pending, and the rollout hangs forever. Our GPU scheduling on Kubernetes guide covers node pools, taints, and priority classes that make canaries schedulable.

Integrate patterns with Kubernetes platform engineering. For LLM-scale serving — where the model itself is the scaling bottleneck — see our guide to deploying LLMs on Kubernetes with GPU autoscaling.

Frequently Asked Questions

How do I handle model artifacts too large for Git?

Store model weights in object storage (S3/GCS) or a model registry such as MLflow, and keep only the artifact URI and version in Git. An init container or the serving framework downloads the weights on pod startup, and a persistent volume or node-local cache avoids re-downloading on every restart. Git LFS is not a good fit: it still bloats clones and gives you none of the lineage a registry provides.

Can I roll back a model instantly with ArgoCD?

Yes. ArgoCD keeps the manifests of previous syncs, so argocd app rollback <app> <revision> or a git revert redeploys the prior version in seconds, and Argo Rollouts' undo does the same for a Rollout. The pods still need to load the old model, so "instant" means "as fast as your startup time" — for large models, blue-green keeps the old version warm so the cutover is truly immediate.

How do I validate models before production with ArgoCD?

Use AnalysisTemplates in Argo Rollouts to query Prometheus during a canary and abort automatically when accuracy proxies, error rate, or latency cross thresholds. Before that, gate the promotion PR on offline evaluation and a load test in staging. The combination catches both the model that is wrong and the model that is right but too slow.

What about multi-cluster ML deployments?

Use ArgoCD ApplicationSets with a cluster generator to stamp the same model Application onto every registered cluster, with per-cluster overlays for region, GPU type, or replica count. A single Git commit then fans out to all regions, and the ApplicationSet controller reports per-cluster sync status. Keep the artifact in a bucket each region can reach with low latency.

How does ArgoCD integrate with CI/CD for ML?

CI (GitHub Actions, GitLab CI) trains, evaluates, and registers the model, then updates the manifest in the GitOps repo; ArgoCD notices the commit and deploys. Training and deployment stay separate pipelines with Git as the handoff, so a training failure can never partially deploy. Our AI CI/CD pipeline guide covers the evaluation gates that belong in the CI half.

Do I need Istio for model serving?

No. Argo Rollouts can shift canary traffic with a plain Kubernetes Service by adjusting replica ratios, and NGINX or Gateway API ingress supports weighted routing without a mesh. Istio becomes worth its complexity when you need header-based routing (A/B by user segment), mTLS between services, or per-request traffic mirroring — see do you actually need a service mesh.

Should GitOps manage training jobs as well as serving?

Usually not. Training jobs are imperative, run-once workloads with parameters that change every run, which fits an orchestrator (Argo Workflows, Kubeflow, Airflow) better than a reconciliation loop. Let GitOps own the long-lived serving state and the orchestrator's own deployment, and let the orchestrator own the jobs.

Conclusion

GitOps with ArgoCD transforms ML model deployments from manual, error-prone processes to declarative, automated infrastructure:

Declarative configs—all models defined as code in Git
Automated deployments—push to deploy, zero manual steps
Instant rollbacks—revert to previous versions in seconds
Multi-environment promotion—validate in staging, promote to production
Observability built-in—track deployments AND model performance
Production patterns—blue-green, canary, multi-model serving

For production ML infrastructure, GitOps is the standard.

At HinterBuild, we build production-grade ML deployment platforms:

Free consultation

Book a free consultation call on GitOps for ML & model deployment

30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.

Book a meeting

Resources:

Keep reading

DevOpsSeptember 11, 2026 · 10 min read

Zero Downtime Deployments: Practical Production Guide

Learn zero downtime deployments through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

DevOpsSeptember 14, 2026 · 12 min read

Crossplane for AI Infrastructure: Kubernetes-Native IaC for

Learn crossplane for ai infrastructure through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

DevOpsSeptember 11, 2026 · 12 min read

Helm Charts from Scratch: No Fluff, Production Guide

Learn helm charts from scratch through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

AI SystemsSeptember 14, 2026 · 9 min read

QLoRA: Fine-Tune 70B Models on Single GPU with 4-bit

QLoRA guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Read post

Why GitOps for ML Models: Beyond Manual kubectl

Push vs Pull: Why ArgoCD Deploys Differently

ArgoCD Architecture for ML Model Serving

Sync Policy Decisions That Matter for Model Serving

Model Registry Integration: MLflow + ArgoCD

Declarative Model Serving with Kustomize

Multi-Environment Promotion Strategy

Automated Rollbacks & Canary Deployments

Choosing a Rollout Strategy for Models

Monitoring & Observability for ML GitOps

Production Deployment Patterns

Pattern 1: Blue-Green Model Deployment

Pattern 2: Multi-Model Serving

Pattern 3: A/B Testing Models

Failure Modes We See in ML GitOps Repositories

Frequently Asked Questions

How do I handle model artifacts too large for Git?

Can I roll back a model instantly with ArgoCD?

How do I validate models before production with ArgoCD?

What about multi-cluster ML deployments?

How does ArgoCD integrate with CI/CD for ML?

Do I need Istio for model serving?

Should GitOps manage training jobs as well as serving?

Conclusion

Related articles

Zero Downtime Deployments: Practical Production Guide

Crossplane for AI Infrastructure: Kubernetes-Native IaC for

Helm Charts from Scratch: No Fluff, Production Guide

QLoRA: Fine-Tune 70B Models on Single GPU with 4-bit