MLOpsSeptember 14, 2026 · 12 min read

ML Model Versioning: Complete DVC & MLflow Guide for

Q: What's the difference between DVC and MLflow?

DVC tracks large files (data, models) Git can't handle. MLflow logs experiments (params, metrics, runs). Use both together—DVC for artifacts, MLflow for experiment metadata.

Q: Should I use MLflow or Weights & Biases?

MLflow is open-source and self-hosted. W&B has better UI but requires their cloud. For production control, prefer MLflow deployed on your infrastructure.

Q: How do I version datasets that change frequently?

Use DVC snapshots—each dvc add creates a new version. Reference specific versions in training code with Git commit SHAs.

ML Model Versioning guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Muhammad Abdul Sami

September 14, 2026 · 12 min read

LLM
LLM Serving
Evaluation
Cost Optimization

Table of Contents:

The Versioning Problem
DVC for Data & Model Artifacts
MLflow Experiment Tracking
Model Registry Patterns
Reproducibility Framework
CI/CD Integration
Production Deployment
Frequently Asked Questions

The Versioning Problem: Why ML Models Need Git-Like Tracking

Short answer: ML models fail in production because teams can't reproduce training runs, track which data produced which model, or roll back broken deployments. The fix is treating models as versioned artifacts with full lineage tracking.

An ML team trained 47 candidate models over 3 months. When their deployed model degraded, they couldn't identify which training run produced it or what data it used. No data hashes, no hyperparameter logs, no lineage. We implemented DVC + MLflow versioning—every model now has complete reproducibility from raw data to deployed weights.

Key Takeaways:

Version everything—code, data, models, and dependencies together

DVC tracks large files Git can't handle (datasets, model weights)

MLflow logs experiments—params, metrics, artifacts, and lineage

Model registry separates staging/production with approval gates

Reproducibility requires capturing full environment state

CI/CD integration automates model validation and deployment

For production AI systems, model versioning is infrastructure, not documentation.

DVC for Data & Model Artifacts

DVC (Data Version Control) is Git for large files. Track datasets and model weights without bloating repositories.

Initial Setup

bash

pip install dvc[s3]  # Or [gs], [azure] for other clouds

# Initialize in Git repo
git init
dvc init

# Configure remote storage (S3 example)
dvc remote add -d storage s3://my-bucket/dvc-store
dvc remote modify storage region us-west-2

# Commit DVC config
git add .dvc/config
git commit -m "Configure DVC remote"

Track Training Data

python

# data_pipeline.py
from pathlib import Path
import pandas as pd
from sklearn.model_selection import train_test_split

def prepare_training_data(raw_data_path: Path, output_dir: Path) -> None:
    """Prepare training dataset with version control."""
    # Load raw data
    df = pd.read_csv(raw_data_path)
    
    # Clean and transform
    df = df.dropna()
    df['features'] = preprocess_features(df)
    
    # Split
    train, test = train_test_split(df, test_size=0.2, random_state=42)
    
    # Save versioned datasets
    output_dir.mkdir(exist_ok=True, parents=True)
    train.to_parquet(output_dir / "train.parquet")
    test.to_parquet(output_dir / "test.parquet")
    
    # Log data statistics
    stats = {
        "train_samples": len(train),
        "test_samples": len(test),
        "feature_count": len(df.columns),
    }
    
    import json
    (output_dir / "data_stats.json").write_text(json.dumps(stats, indent=2))

# Prepare data
prepare_training_data(
    raw_data_path=Path("raw/data.csv"),
    output_dir=Path("data/processed"),
)

bash

# Track with DVC
dvc add data/processed/train.parquet
dvc add data/processed/test.parquet

# DVC creates .dvc files (tiny metadata)
# Actual data stored in remote (S3)
git add data/processed/*.dvc data/processed/data_stats.json
git commit -m "Add training data v1.0"

# Push data to remote
dvc push

# Teammates pull data
dvc pull

Track Model Artifacts

python

# training.py
from pathlib import Path
import joblib
from sklearn.ensemble import RandomForestClassifier
import hashlib

def train_model(data_dir: Path, output_dir: Path, hyperparams: dict) -> None:
    """Train model with artifact tracking."""
    import pandas as pd
    
    # Load versioned data
    train = pd.read_parquet(data_dir / "train.parquet")
    X_train = train.drop("target", axis=1)
    y_train = train["target"]
    
    # Train
    model = RandomForestClassifier(**hyperparams)
    model.fit(X_train, y_train)
    
    # Save model
    output_dir.mkdir(exist_ok=True, parents=True)
    model_path = output_dir / "model.pkl"
    joblib.dump(model, model_path)
    
    # Calculate model hash for lineage
    model_hash = hashlib.sha256(model_path.read_bytes()).hexdigest()[:8]
    
    # Save metadata
    metadata = {
        "hyperparams": hyperparams,
        "data_version": (data_dir / "train.parquet.dvc").read_text().split("md5: ")[1].split("\n")[0][:8],
        "model_hash": model_hash,
    }
    
    import json
    (output_dir / "metadata.json").write_text(json.dumps(metadata, indent=2))

train_model(
    data_dir=Path("data/processed"),
    output_dir=Path("models/rf-v1"),
    hyperparams={"n_estimators": 100, "max_depth": 10},
)

bash

# Track model with DVC
dvc add models/rf-v1/model.pkl
git add models/rf-v1/*.dvc models/rf-v1/metadata.json
git commit -m "Train random forest v1 (100 trees, depth 10)"
dvc push

DVC Pipeline for Reproducibility

yaml

# dvc.yaml - Define reproducible ML pipeline
stages:
  prepare_data:
    cmd: python data_pipeline.py
    deps:
      - raw/data.csv
      - data_pipeline.py
    outs:
      - data/processed/train.parquet
      - data/processed/test.parquet
    params:
      - prepare.test_size
      - prepare.random_state

  train_model:
    cmd: python training.py
    deps:
      - data/processed/train.parquet
      - training.py
    outs:
      - models/rf-v1/model.pkl
    params:
      - train.n_estimators
      - train.max_depth
    metrics:
      - models/rf-v1/metrics.json:
          cache: false

  evaluate:
    cmd: python evaluate.py
    deps:
      - models/rf-v1/model.pkl
      - data/processed/test.parquet
    metrics:
      - reports/metrics.json:
          cache: false

yaml

# params.yaml - Hyperparameters
prepare:
  test_size: 0.2
  random_state: 42

train:
  n_estimators: 100
  max_depth: 10

bash

# Run full pipeline
dvc repro

# DVC automatically:
# - Tracks dependencies
# - Skips unchanged stages
# - Caches outputs
# - Records metrics

# View metrics
dvc metrics show

# Compare experiments
dvc metrics diff

Connect DVC to data pipeline infrastructure and cloud storage.

MLflow Experiment Tracking

MLflow logs experiments, compares runs, and serves models. Essential for production ML.

Setup MLflow Tracking Server

bash

# Install MLflow
pip install mlflow

# Start local server
mlflow server \
  --backend-store-uri sqlite:///mlflow.db \
  --default-artifact-root ./mlruns \
  --host 0.0.0.0 \
  --port 5000

For production, deploy on Kubernetes infrastructure:

yaml

# mlflow-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mlflow-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mlflow
  template:
    metadata:
      labels:
        app: mlflow
    spec:
      containers:
      - name: mlflow
        image: ghcr.io/mlflow/mlflow:v2.10.0
        ports:
        - containerPort: 5000
        env:
        - name: BACKEND_STORE_URI
          value: postgresql://user:pass@postgres:5432/mlflow
        - name: DEFAULT_ARTIFACT_ROOT
          value: s3://ml-artifacts/
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: aws-creds
              key: access-key-id
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-creds
              key: secret-access-key

Track Training Runs

python

# training_with_mlflow.py
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score
import pandas as pd

# Configure MLflow
mlflow.set_tracking_uri("http://mlflow-server:5000")
mlflow.set_experiment("customer-churn-prediction")

def train_with_tracking(data_dir: Path, hyperparams: dict) -> None:
    """Train model with full MLflow tracking."""
    
    with mlflow.start_run(run_name=f"rf-{hyperparams['n_estimators']}-trees"):
        # Log hyperparameters
        mlflow.log_params(hyperparams)
        
        # Log data version
        mlflow.log_param("data_version", "v1.2")
        mlflow.log_param("data_path", str(data_dir))
        
        # Load data
        train = pd.read_parquet(data_dir / "train.parquet")
        test = pd.read_parquet(data_dir / "test.parquet")
        
        X_train = train.drop("target", axis=1)
        y_train = train["target"]
        X_test = test.drop("target", axis=1)
        y_test = test["target"]
        
        # Train
        model = RandomForestClassifier(**hyperparams)
        model.fit(X_train, y_train)
        
        # Evaluate
        y_pred = model.predict(X_test)
        y_proba = model.predict_proba(X_test)[:, 1]
        
        metrics = {
            "accuracy": accuracy_score(y_test, y_pred),
            "f1": f1_score(y_test, y_pred),
            "roc_auc": roc_auc_score(y_test, y_proba),
        }
        
        # Log metrics
        mlflow.log_metrics(metrics)
        
        # Log model
        mlflow.sklearn.log_model(
            model,
            "model",
            registered_model_name="churn-predictor",
            signature=mlflow.models.infer_signature(X_train, y_train),
        )
        
        # Log feature importance plot
        import matplotlib.pyplot as plt
        feature_importance = pd.DataFrame({
            "feature": X_train.columns,
            "importance": model.feature_importances_,
        }).sort_values("importance", ascending=False).head(10)
        
        plt.figure(figsize=(10, 6))
        plt.barh(feature_importance["feature"], feature_importance["importance"])
        plt.xlabel("Importance")
        plt.title("Top 10 Features")
        plt.tight_layout()
        plt.savefig("feature_importance.png")
        mlflow.log_artifact("feature_importance.png")
        
        # Log confusion matrix
        from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
        cm = confusion_matrix(y_test, y_pred)
        disp = ConfusionMatrixDisplay(cm)
        disp.plot()
        plt.savefig("confusion_matrix.png")
        mlflow.log_artifact("confusion_matrix.png")
        
        # Log training dataset stats
        mlflow.log_dict({
            "train_samples": len(train),
            "test_samples": len(test),
            "positive_rate": float(y_train.mean()),
        }, "data_stats.json")
        
        print(f"✓ Run logged to MLflow: {mlflow.active_run().info.run_id}")
        print(f"  Metrics: {metrics}")

# Hyperparameter tuning with tracking
from sklearn.model_selection import ParameterGrid

param_grid = {
    "n_estimators": [50, 100, 200],
    "max_depth": [5, 10, 15],
    "min_samples_split": [2, 5],
}

for params in ParameterGrid(param_grid):
    train_with_tracking(Path("data/processed"), params)

Compare Experiments

python

# compare_runs.py
import mlflow
from mlflow.tracking import MlflowClient

client = MlflowClient("http://mlflow-server:5000")

# Get experiment
experiment = client.get_experiment_by_name("customer-churn-prediction")

# Query runs
runs = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    order_by=["metrics.roc_auc DESC"],
    max_results=10,
)

# Display results
import pandas as pd
results = []
for run in runs:
    results.append({
        "run_id": run.info.run_id[:8],
        "n_estimators": run.data.params.get("n_estimators"),
        "max_depth": run.data.params.get("max_depth"),
        "roc_auc": run.data.metrics.get("roc_auc"),
        "f1": run.data.metrics.get("f1"),
        "accuracy": run.data.metrics.get("accuracy"),
    })

df = pd.DataFrame(results)
print("\nTop 10 runs by ROC AUC:")
print(df.to_string(index=False))

# Best run
best_run = runs[0]
print(f"\nBest run: {best_run.info.run_id}")
print(f"Parameters: {best_run.data.params}")
print(f"Metrics: {best_run.data.metrics}")

# Load best model
best_model = mlflow.sklearn.load_model(f"runs:/{best_run.info.run_id}/model")

Model Registry Patterns

Model Registry manages model lifecycle from training to production.

Register Models

python

# model_registry.py
import mlflow
from mlflow.tracking import MlflowClient

client = MlflowClient("http://mlflow-server:5000")

def promote_model_to_registry(run_id: str, model_name: str) -> str:
    """Register model from run."""
    # Register model
    model_uri = f"runs:/{run_id}/model"
    model_details = mlflow.register_model(model_uri, model_name)
    
    version = model_details.version
    print(f"Registered {model_name} version {version}")
    
    return version

def promote_to_staging(model_name: str, version: str) -> None:
    """Promote model to staging."""
    client.transition_model_version_stage(
        name=model_name,
        version=version,
        stage="Staging",
        archive_existing_versions=True,
    )
    
    # Add description
    client.update_model_version(
        name=model_name,
        version=version,
        description=f"Promoted to staging on {datetime.now().isoformat()}"
    )
    
    print(f"✓ {model_name} v{version} promoted to Staging")

def promote_to_production(model_name: str, version: str) -> None:
    """Promote model to production after approval."""
    # Validation check
    if not validate_model_quality(model_name, version):
        raise ValueError("Model failed validation checks")
    
    # Promote
    client.transition_model_version_stage(
        name=model_name,
        version=version,
        stage="Production",
        archive_existing_versions=True,
    )
    
    # Tag production deployment
    client.set_model_version_tag(
        name=model_name,
        version=version,
        key="deployment_time",
        value=datetime.now().isoformat(),
    )
    
    print(f"✓ {model_name} v{version} deployed to Production")

def validate_model_quality(model_name: str, version: str) -> bool:
    """Validate model meets quality thresholds."""
    # Get model version
    model_version = client.get_model_version(model_name, version)
    run = client.get_run(model_version.run_id)
    
    # Check metrics
    metrics = run.data.metrics
    
    required_metrics = {
        "roc_auc": 0.75,
        "f1": 0.70,
        "accuracy": 0.80,
    }
    
    for metric, threshold in required_metrics.items():
        if metrics.get(metric, 0) < threshold:
            print(f"✗ {metric}={metrics.get(metric):.3f} below threshold {threshold}")
            return False
    
    print(f"✓ All quality checks passed")
    return True

# Workflow
best_run_id = "abc123def456"
version = promote_model_to_registry(best_run_id, "churn-predictor")
promote_to_staging("churn-predictor", version)

# After staging validation
promote_to_production("churn-predictor", version)

Load Models by Stage

python

# inference.py
import mlflow

def load_production_model(model_name: str):
    """Load latest production model."""
    model_uri = f"models:/{model_name}/Production"
    model = mlflow.sklearn.load_model(model_uri)
    return model

def predict(features: dict) -> dict:
    """Predict using production model."""
    model = load_production_model("churn-predictor")
    
    import pandas as pd
    X = pd.DataFrame([features])
    
    prediction = model.predict(X)[0]
    probability = model.predict_proba(X)[0][1]
    
    return {
        "prediction": int(prediction),
        "probability": float(probability),
        "model_name": "churn-predictor",
        "model_stage": "Production",
    }

# Use in API
result = predict({"tenure": 24, "monthly_charges": 65.0, "contract": "Month-to-month"})
print(result)

Integrate with backend API engineering for model serving.

Reproducibility Framework

Full reproducibility requires versioning code, data, models, environment, and randomness.

Environment Management

yaml

# environment.yml - Conda environment
name: ml-training
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.11
  - pip
  - pip:
    - scikit-learn==1.4.0
    - pandas==2.1.4
    - mlflow==2.10.0
    - dvc[s3]==3.40.0

dockerfile

# Dockerfile - Reproducible training environment
FROM python:3.11-slim

# Install system deps
RUN apt-get update && apt-get install -y git

# Copy requirements
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Set working directory
WORKDIR /workspace

# Copy code
COPY . .

# Run training
CMD ["dvc", "repro"]

Deterministic Training

python

# reproducible_training.py
import random
import numpy as np
import torch

def set_seed(seed: int = 42) -> None:
    """Set seeds for reproducibility."""
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    
    # Deterministic operations
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

def train_reproducible(seed: int = 42) -> None:
    """Train with full reproducibility."""
    set_seed(seed)
    
    with mlflow.start_run():
        # Log seed
        mlflow.log_param("seed", seed)
        
        # Log git commit
        import subprocess
        git_commit = subprocess.check_output(
            ["git", "rev-parse", "HEAD"]
        ).decode().strip()
        mlflow.log_param("git_commit", git_commit)
        
        # Log Python version
        import sys
        mlflow.log_param("python_version", sys.version)
        
        # Log package versions
        import pkg_resources
        packages = {pkg.key: pkg.version for pkg in pkg_resources.working_set}
        mlflow.log_dict(packages, "requirements.json")
        
        # Train model (deterministic)
        model = train_model()
        
        # Log model with all context
        mlflow.sklearn.log_model(model, "model")

CI/CD Integration

Automate model validation and deployment.

GitHub Actions Workflow

yaml

# .github/workflows/ml-pipeline.yml
name: ML Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  train:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
      
      - name: Pull data with DVC
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          dvc remote modify storage access_key_id $AWS_ACCESS_KEY_ID
          dvc remote modify storage secret_access_key $AWS_SECRET_ACCESS_KEY
          dvc pull
      
      - name: Run training pipeline
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_TRACKING_URI }}
        run: |
          dvc repro
      
      - name: Validate model quality
        run: |
          python validate_model.py
      
      - name: Push artifacts
        run: |
          dvc push
  
  deploy:
    runs-on: ubuntu-latest
    needs: train
    if: github.ref == 'refs/heads/main'
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Promote to staging
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_TRACKING_URI }}
        run: |
          python promote_model.py --stage staging
      
      - name: Run integration tests
        run: |
          pytest tests/integration/
      
      - name: Promote to production
        if: success()
        run: |
          python promote_model.py --stage production

Connect to AI CI/CD pipeline practices.

Production Deployment

Deploy versioned models with monitoring.

Model Serving API

python

# model_server.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import mlflow
from prometheus_client import Counter, Histogram, generate_latest

app = FastAPI()

# Metrics
prediction_counter = Counter("predictions_total", "Total predictions")
prediction_latency = Histogram("prediction_latency_seconds", "Prediction latency")

class PredictionRequest(BaseModel):
    features: dict

class PredictionResponse(BaseModel):
    prediction: int
    probability: float
    model_version: str

# Load model on startup
model = None
model_version = None

@app.on_event("startup")
async def load_model():
    """Load production model."""
    global model, model_version
    
    model_name = "churn-predictor"
    model_uri = f"models:/{model_name}/Production"
    
    model = mlflow.sklearn.load_model(model_uri)
    
    # Get version
    from mlflow.tracking import MlflowClient
    client = MlflowClient()
    prod_versions = client.get_latest_versions(model_name, stages=["Production"])
    model_version = prod_versions[0].version if prod_versions else "unknown"
    
    print(f"✓ Loaded {model_name} v{model_version}")

@app.post("/predict", response_model=PredictionResponse)
@prediction_latency.time()
async def predict(request: PredictionRequest):
    """Predict endpoint."""
    try:
        import pandas as pd
        X = pd.DataFrame([request.features])
        
        prediction = int(model.predict(X)[0])
        probability = float(model.predict_proba(X)[0][1])
        
        prediction_counter.inc()
        
        return PredictionResponse(
            prediction=prediction,
            probability=probability,
            model_version=model_version,
        )
    
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get("/metrics")
async def metrics():
    """Prometheus metrics."""
    return generate_latest()

@app.get("/health")
async def health():
    """Health check."""
    return {"status": "healthy", "model_version": model_version}

Deploy with Kubernetes platform engineering and monitor with observability systems.

Primary references: official documentation, official documentation, official documentation, official documentation.

ML Model Versioning Decision Table

Decision	Prefer the simpler path when	Add operational complexity when
Architecture	One component can own the contract and state	Independent scaling or fault isolation is required
Rollout	Offline replay covers the meaningful cases	Live behavior requires shadow traffic and a canary
Recovery	A failed operation is safe to repeat	Partial effects require idempotency or reconciliation
Measurement	One service objective represents user impact	Quality, latency, and cost need separate gates

Operating ML Model Versioning as a System

The implementation is only one part of ML Model Versioning. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship ML Model Versioning behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our ML Model Versioning engineering support.

Operating ML Model Versioning as a System

Define the failure model

Measure the decision, not just the component

Roll out with reversible steps

Capacity and cost controls

Production readiness review

Frequently Asked Questions

What's the difference between DVC and MLflow?

DVC tracks large files (data, models) Git can't handle. MLflow logs experiments (params, metrics, runs). Use both together—DVC for artifacts, MLflow for experiment metadata.

Should I use MLflow or Weights & Biases?

MLflow is open-source and self-hosted. W&B has better UI but requires their cloud. For production control, prefer MLflow deployed on your infrastructure.

How do I version datasets that change frequently?

Use DVC snapshots—each dvc add creates a new version. Reference specific versions in training code with Git commit SHAs.

Can I rollback to previous model versions?

Yes—promote previous version to Production in MLflow registry, or load by version: models:/model-name/3.

How do I handle model drift in production?

Monitor prediction distributions and performance metrics. When drift detected, retrain with recent data, version as new model, validate, then promote. See data flywheel strategies.

What if training takes hours?

Cache intermediate artifacts with DVC, use incremental training if possible, or implement shadow mode deployment to validate before full retraining.

Conclusion

ML model versioning with DVC and MLflow enables production reproducibility:

DVC tracks artifacts—data and models Git can't handle
MLflow logs experiments—params, metrics, lineage
Model registry manages staging/production lifecycle
Reproducibility requires versioning code, data, models, environment
CI/CD automation validates and deploys models safely
Production serving loads versioned models by stage

Model versioning is infrastructure for reliable ML systems.

At HinterBuild, we build production ML infrastructure:

Free consultation

Book a free consultation call on ML model versioning & experiment tracking

30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.

Book a meeting

Keep reading

AI SystemsSeptember 14, 2026 · 9 min read

Model Routing in Production: Automatic Selection for Cost

Learn model routing in production through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

AI SystemsSeptember 8, 2026 · 10 min read

Model Distillation for LLMs: How to Build Smaller, Smarter

Learn model distillation for llms through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

AI SystemsSeptember 8, 2026 · 9 min read

LLM Routing: How to Pick the Cheapest Model That Works

LLM Routing guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Read post

AI SystemsSeptember 14, 2026 · 10 min read

vLLM in Production: PagedAttention, Continuous Batching, and

vLLM in Production guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Read post

The Versioning Problem: Why ML Models Need Git-Like Tracking

DVC for Data & Model Artifacts

Initial Setup

Track Training Data

Track Model Artifacts

DVC Pipeline for Reproducibility

MLflow Experiment Tracking

Setup MLflow Tracking Server

Track Training Runs

Compare Experiments

Model Registry Patterns

Register Models

Load Models by Stage

Reproducibility Framework

Environment Management

Deterministic Training

CI/CD Integration

GitHub Actions Workflow

Production Deployment

Model Serving API

ML Model Versioning Decision Table

Operating ML Model Versioning as a System

Define the failure model

Measure the decision, not just the component

Roll out with reversible steps

Capacity and cost controls

Production readiness review

Operating ML Model Versioning as a System

Define the failure model

Measure the decision, not just the component

Roll out with reversible steps

Capacity and cost controls

Production readiness review

Frequently Asked Questions

What's the difference between DVC and MLflow?

Should I use MLflow or Weights & Biases?

How do I version datasets that change frequently?

Can I rollback to previous model versions?

How do I handle model drift in production?

What if training takes hours?

Conclusion

Related articles

Model Routing in Production: Automatic Selection for Cost

Model Distillation for LLMs: How to Build Smaller, Smarter

LLM Routing: How to Pick the Cheapest Model That Works

vLLM in Production: PagedAttention, Continuous Batching, and