HinterBuild logoHinterBuild
Back to blog
DevOps · 12 min read

Crossplane for AI Infrastructure: Kubernetes-Native IaC for

Learn crossplane for ai infrastructure through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Muhammad Abdul Sami, author

Muhammad Abdul Sami

· 12 min read

  • Kubernetes
  • DevOps
  • MLOps
  • Observability

Table of Contents:

Why Crossplane for AI Infrastructure: Beyond Terraform

Short answer: AI teams waste weeks provisioning infrastructure—GPU nodes, vector databases, object storage, model registries—using fragmented tools. Crossplane unifies all infrastructure as Kubernetes resources with declarative compositions, self-service, and native GitOps integration.

A fintech ML team spent 3-4 weeks provisioning infrastructure for each new model: Terraform for cloud resources, Helm for Kubernetes apps, manual IAM setup, disconnected monitoring. We migrated them to Crossplane—data scientists request infrastructure via YAML, everything provisions automatically in 15 minutes. Infrastructure lead time dropped 95%.

Key Takeaways:

  • Kubernetes-native IaC—manage cloud resources as CRDs with kubectl
  • Declarative compositions—define reusable AI infrastructure templates
  • Self-service for ML teams—data scientists provision without DevOps tickets
  • Multi-cloud abstraction—same interface for AWS, GCP, Azure
  • GitOps integration—ArgoCD manages both apps AND infrastructure

For production AI systems, treating infrastructure as Kubernetes resources eliminates tool sprawl.

Crossplane Architecture for AI Platforms

Crossplane extends Kubernetes with cloud provider APIs—create S3 buckets, RDS instances, GKE clusters via kubectl.

┌─────────────────────────────────────────────────┐
│          Data Scientist / ML Engineer           │
└────────────────────┬────────────────────────────┘
                     │ kubectl apply -f ml-env.yaml
                     ▼
┌─────────────────────────────────────────────────┐
│         Crossplane (Kubernetes Cluster)         │
│                                                 │
│  ┌──────────────────────────────────────────┐  │
│  │  Composite Resource (XR)                 │  │
│  │  kind: MLEnvironment                     │  │
│  └─────────────┬────────────────────────────┘  │
│                │ Composition                    │
│                ▼                                │
│  ┌──────────────────────────────────────────┐  │
│  │  Managed Resources (MRs)                 │  │
│  │  - GKE Cluster (GPU nodes)               │  │
│  │  - Cloud SQL (metadata)                  │  │
│  │  - GCS Bucket (artifacts)                │  │
│  │  - Weaviate (vector DB)                  │  │
│  └─────────────┬────────────────────────────┘  │
└────────────────┼────────────────────────────────┘
                 │ Reconciliation
                 ▼
┌─────────────────────────────────────────────────┐
│       Cloud Providers (AWS, GCP, Azure)         │
│  - Compute (GPU instances)                      │
│  - Storage (S3, GCS, Blob)                      │
│  - Databases (RDS, Cloud SQL)                   │
│  - Networking (VPC, Load Balancers)             │
└─────────────────────────────────────────────────┘

Core concepts:

  • Provider - Plugin connecting Kubernetes to cloud API (AWS, GCP, Helm)
  • Managed Resource (MR) - Low-level cloud resource (S3Bucket, GKECluster)
  • Composite Resource (XR) - High-level abstraction (MLEnvironment, VectorDB)
  • Composition - Template mapping XR to multiple MRs
yaml
kubectl create namespace crossplane-system

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

helm install crossplane \
  crossplane-stable/crossplane \
  --namespace crossplane-system \
  --create-namespace

# Install cloud providers
kubectl crossplane install provider crossplane/provider-gcp:v0.28.0
kubectl crossplane install provider crossplane/provider-aws:v0.46.0
kubectl crossplane install provider crossplane/provider-helm:v0.15.0

# Configure GCP provider
kubectl create secret generic gcp-creds \
  --from-file=creds=./gcp-credentials.json \
  --namespace crossplane-system

cat <<EOF | kubectl apply -f -
apiVersion: gcp.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  projectID: my-ml-project
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: gcp-creds
      key: creds
EOF

Deploy on Kubernetes infrastructure.

GPU Cluster Composition for ML Workloads

Define reusable GPU cluster template for training and inference.

yaml
# compositions/gpu-cluster-composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: gpu-cluster-gcp
  labels:
    provider: gcp
    cluster-type: gpu
spec:
  writeConnectionSecretsToNamespace: crossplane-system
  
  compositeTypeRef:
    apiVersion: ml.hinterbuild.com/v1alpha1
    kind: GPUCluster
  
  resources:
  # GKE Cluster with GPU nodes
  - name: gke-cluster
    base:
      apiVersion: container.gcp.crossplane.io/v1beta2
      kind: Cluster
      spec:
        forProvider:
          location: us-central1
          initialNodeCount: 1
          
          # Enable features
          addonsConfig:
            gcePersistentDiskCsiDriverConfig:
              enabled: true
          
          # Autopilot for easier management
          enableAutopilot: false
          
          # Networking
          ipAllocationPolicy:
            useIpAliases: true
          
          # Logging
          loggingService: logging.googleapis.com/kubernetes
          monitoringService: monitoring.googleapis.com/kubernetes
    
    patches:
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.name
      toFieldPath: metadata.name
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.region
      toFieldPath: spec.forProvider.location
  
  # GPU Node Pool
  - name: gpu-node-pool
    base:
      apiVersion: container.gcp.crossplane.io/v1beta1
      kind: NodePool
      spec:
        forProvider:
          clusterSelector:
            matchControllerRef: true
          
          autoscaling:
            enabled: true
            minNodeCount: 0
            maxNodeCount: 10
          
          config:
            machineType: n1-standard-8
            
            # Attach GPUs
            guestAccelerator:
            - type: nvidia-tesla-t4
              count: 1
            
            # GPU drivers
            diskSizeGb: 100
            oauthScopes:
            - https://www.googleapis.com/auth/cloud-platform
            
            # Taints for GPU nodes
            taints:
            - effect: NO_SCHEDULE
              key: nvidia.com/gpu
              value: "true"
            
            labels:
              workload-type: ml-training
              gpu-type: t4
    
    patches:
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.gpuType
      toFieldPath: spec.forProvider.config.guestAccelerator[0].type
      transforms:
      - type: map
        map:
          t4: nvidia-tesla-t4
          v100: nvidia-tesla-v100
          a100: nvidia-tesla-a100
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.minNodes
      toFieldPath: spec.forProvider.autoscaling.minNodeCount
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.maxNodes
      toFieldPath: spec.forProvider.autoscaling.maxNodeCount
  
  # GCS Bucket for artifacts
  - name: artifact-bucket
    base:
      apiVersion: storage.gcp.crossplane.io/v1alpha1
      kind: Bucket
      spec:
        forProvider:
          location: US
          storageClass: STANDARD
          
          # Lifecycle rules
          lifecycle:
            rule:
            - action:
                type: Delete
              condition:
                age: 90  # Delete old artifacts
    
    patches:
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.name
      toFieldPath: metadata.name
      transforms:
      - type: string
        string:
          fmt: "%s-artifacts"

---
# XRD (Schema definition)
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: gpuclusters.ml.hinterbuild.com
spec:
  group: ml.hinterbuild.com
  names:
    kind: GPUCluster
    plural: gpuclusters
  
  claimNames:
    kind: GPUClusterClaim
    plural: gpuclusterclaims
  
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              parameters:
                type: object
                properties:
                  name:
                    type: string
                    description: Cluster name
                  region:
                    type: string
                    description: GCP region
                    default: us-central1
                  gpuType:
                    type: string
                    description: GPU type
                    enum: [t4, v100, a100]
                    default: t4
                  minNodes:
                    type: integer
                    description: Min GPU nodes
                    default: 0
                  maxNodes:
                    type: integer
                    description: Max GPU nodes
                    default: 10
                required:
                - name

Usage by data scientists:

yaml
# ml-team requests GPU cluster
apiVersion: ml.hinterbuild.com/v1alpha1
kind: GPUClusterClaim
metadata:
  name: fraud-training-cluster
  namespace: ml-team
spec:
  parameters:
    name: fraud-training
    region: us-central1
    gpuType: t4
    minNodes: 1
    maxNodes: 5
  
  # Cluster ready when provisioned
  compositionSelector:
    matchLabels:
      provider: gcp
      cluster-type: gpu

# kubectl apply -f cluster-claim.yaml
# Wait 10-15 minutes for provisioning
# kubectl get gpuclusterclaim fraud-training-cluster

Integrate with cloud infrastructure automation.

Vector Database Provisioning

Declarative vector database for RAG systems.

yaml
# compositions/vector-db-composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: weaviate-gcp
  labels:
    provider: gcp
    database: weaviate
spec:
  compositeTypeRef:
    apiVersion: ml.hinterbuild.com/v1alpha1
    kind: VectorDatabase
  
  resources:
  # GKE cluster for Weaviate
  - name: weaviate-namespace
    base:
      apiVersion: kubernetes.crossplane.io/v1alpha1
      kind: Object
      spec:
        forProvider:
          manifest:
            apiVersion: v1
            kind: Namespace
            metadata:
              name: weaviate
  
  # Weaviate via Helm
  - name: weaviate-helm
    base:
      apiVersion: helm.crossplane.io/v1beta1
      kind: Release
      spec:
        forProvider:
          chart:
            name: weaviate
            repository: https://weaviate.github.io/weaviate-helm
            version: 16.8.0
          
          namespace: weaviate
          
          values:
            replicas: 3
            
            resources:
              requests:
                cpu: 2000m
                memory: 4Gi
              limits:
                cpu: 4000m
                memory: 8Gi
            
            # Persistence
            storage:
              size: 100Gi
              storageClassName: ssd
            
            # Modules
            modules:
              text2vec-openai:
                enabled: true
              text2vec-cohere:
                enabled: true
              qna-openai:
                enabled: true
            
            # Authentication
            authentication:
              apikey:
                enabled: true
            
            # Monitoring
            monitoring:
              enabled: true
    
    patches:
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.replicas
      toFieldPath: spec.forProvider.values.replicas
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.storageSize
      toFieldPath: spec.forProvider.values.storage.size
  
  # Cloud SQL for metadata
  - name: metadata-db
    base:
      apiVersion: database.gcp.crossplane.io/v1beta1
      kind: CloudSQLInstance
      spec:
        forProvider:
          databaseVersion: POSTGRES_14
          region: us-central1
          settings:
            tier: db-custom-2-8192  # 2 vCPU, 8GB RAM
            diskSize: 50
            diskType: PD_SSD
            
            backupConfiguration:
              enabled: true
              startTime: "02:00"
            
            ipConfiguration:
              ipv4Enabled: true
              authorizedNetworks:
              - name: allow-gke
                value: 0.0.0.0/0  # Replace with actual GKE CIDR
    
    patches:
    - type: FromCompositeFieldPath
      fromFieldPath: spec.parameters.name
      toFieldPath: metadata.name
      transforms:
      - type: string
        string:
          fmt: "%s-metadata"

---
# XRD for VectorDatabase
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: vectordatabases.ml.hinterbuild.com
spec:
  group: ml.hinterbuild.com
  names:
    kind: VectorDatabase
    plural: vectordatabases
  
  claimNames:
    kind: VectorDatabaseClaim
    plural: vectordatabaseclaims
  
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              parameters:
                type: object
                properties:
                  name:
                    type: string
                  replicas:
                    type: integer
                    default: 3
                  storageSize:
                    type: string
                    default: "100Gi"
                required:
                - name

# Usage
---
apiVersion: ml.hinterbuild.com/v1alpha1
kind: VectorDatabaseClaim
metadata:
  name: rag-vector-db
  namespace: ml-production
spec:
  parameters:
    name: production-rag
    replicas: 3
    storageSize: "200Gi"

Connect to RAG & LLM systems.

Model Registry Infrastructure

MLflow registry with object storage backend.

yaml
# compositions/mlflow-composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: mlflow-platform
spec:
  compositeTypeRef:
    apiVersion: ml.hinterbuild.com/v1alpha1
    kind: ModelRegistry
  
  resources:
  # PostgreSQL backend for MLflow tracking
  - name: mlflow-db
    base:
      apiVersion: database.gcp.crossplane.io/v1beta1
      kind: CloudSQLInstance
      spec:
        forProvider:
          databaseVersion: POSTGRES_14
          region: us-central1
          settings:
            tier: db-custom-4-16384
            diskSize: 100
            diskType: PD_SSD
            
            backupConfiguration:
              enabled: true
              pointInTimeRecoveryEnabled: true
  
  # GCS bucket for model artifacts
  - name: artifact-storage
    base:
      apiVersion: storage.gcp.crossplane.io/v1alpha1
      kind: Bucket
      spec:
        forProvider:
          location: US
          storageClass: STANDARD
          
          versioning:
            enabled: true
          
          lifecycle:
            rule:
            - action:
                type: SetStorageClass
                storageClass: NEARLINE
              condition:
                age: 30
            - action:
                type: Delete
              condition:
                age: 365
  
  # MLflow server deployment
  - name: mlflow-server
    base:
      apiVersion: helm.crossplane.io/v1beta1
      kind: Release
      spec:
        forProvider:
          chart:
            name: mlflow
            repository: https://community-charts.github.io/helm-charts
            version: 0.7.19
          
          namespace: mlflow
          
          values:
            serviceType: LoadBalancer
            
            backendStore:
              postgres:
                enabled: true
                # Connection details from CloudSQL secret
            
            artifactRoot:
              gcs:
                enabled: true
                # Bucket name from composition
            
            resources:
              requests:
                cpu: 1000m
                memory: 2Gi
              limits:
                cpu: 2000m
                memory: 4Gi
            
            # Authentication
            extraEnvVars:
            - name: MLFLOW_TRACKING_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mlflow-auth
                  key: username
            - name: MLFLOW_TRACKING_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mlflow-auth
                  key: password

# Usage
---
apiVersion: ml.hinterbuild.com/v1alpha1
kind: ModelRegistryClaim
metadata:
  name: company-model-registry
  namespace: ml-platform
spec:
  parameters:
    name: mlflow-production
    region: us-central1

Integrate with data pipelines for end-to-end workflows.

Multi-Cloud ML Platform Composition

Abstract cloud providers for portable AI infrastructure.

yaml
# compositions/ml-platform-multi-cloud.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: ml-platform-aws
  labels:
    provider: aws
spec:
  compositeTypeRef:
    apiVersion: ml.hinterbuild.com/v1alpha1
    kind: MLPlatform
  
  resources:
  # EKS cluster
  - name: eks-cluster
    base:
      apiVersion: eks.aws.crossplane.io/v1beta1
      kind: Cluster
      spec:
        forProvider:
          region: us-west-2
          version: "1.28"
          
          roleArnSelector:
            matchLabels:
              role: eks-cluster
  
  # GPU node group
  - name: gpu-nodes
    base:
      apiVersion: eks.aws.crossplane.io/v1alpha1
      kind: NodeGroup
      spec:
        forProvider:
          region: us-west-2
          clusterNameSelector:
            matchControllerRef: true
          
          instanceTypes:
          - p3.2xlarge  # Tesla V100
          
          scalingConfig:
            minSize: 1
            maxSize: 10
            desiredSize: 2
  
  # S3 for artifacts
  - name: artifact-bucket
    base:
      apiVersion: s3.aws.crossplane.io/v1beta1
      kind: Bucket
      spec:
        forProvider:
          region: us-west-2
          
          versioningConfiguration:
            status: Enabled

---
# Same interface, GCP implementation
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: ml-platform-gcp
  labels:
    provider: gcp
spec:
  compositeTypeRef:
    apiVersion: ml.hinterbuild.com/v1alpha1
    kind: MLPlatform
  
  resources:
  # GKE cluster
  - name: gke-cluster
    base:
      apiVersion: container.gcp.crossplane.io/v1beta2
      kind: Cluster
      spec:
        forProvider:
          location: us-central1
  
  # GPU node pool
  - name: gpu-nodes
    base:
      apiVersion: container.gcp.crossplane.io/v1beta1
      kind: NodePool
      spec:
        forProvider:
          config:
            machineType: n1-standard-8
            guestAccelerator:
            - type: nvidia-tesla-v100
              count: 1
  
  # GCS bucket
  - name: artifact-bucket
    base:
      apiVersion: storage.gcp.crossplane.io/v1alpha1
      kind: Bucket
      spec:
        forProvider:
          location: US

---
# Usage: same YAML works on AWS or GCP
apiVersion: ml.hinterbuild.com/v1alpha1
kind: MLPlatformClaim
metadata:
  name: my-ml-platform
  namespace: ml-team
spec:
  parameters:
    name: ml-prod
    gpuType: v100
  
  # Select provider
  compositionSelector:
    matchLabels:
      provider: gcp  # Change to 'aws' for AWS

Deploy multi-cloud with cloud infrastructure expertise.

Observability Stack Automation

Provision monitoring alongside ML infrastructure.

yaml
# compositions/observability-composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: ml-observability
spec:
  compositeTypeRef:
    apiVersion: ml.hinterbuild.com/v1alpha1
    kind: ObservabilityStack
  
  resources:
  # Prometheus Operator
  - name: prometheus-operator
    base:
      apiVersion: helm.crossplane.io/v1beta1
      kind: Release
      spec:
        forProvider:
          chart:
            name: kube-prometheus-stack
            repository: https://prometheus-community.github.io/helm-charts
            version: 55.0.0
          
          namespace: monitoring
          
          values:
            prometheus:
              prometheusSpec:
                retention: 30d
                storageSpec:
                  volumeClaimTemplate:
                    spec:
                      accessModes: ["ReadWriteOnce"]
                      resources:
                        requests:
                          storage: 100Gi
            
            grafana:
              enabled: true
              adminPassword: CHANGE_ME
              
              dashboardProviders:
                dashboardproviders.yaml:
                  apiVersion: 1
                  providers:
                  - name: 'ml-dashboards'
                    folder: 'ML Models'
                    type: file
                    options:
                      path: /var/lib/grafana/dashboards/ml
  
  # Loki for logs
  - name: loki
    base:
      apiVersion: helm.crossplane.io/v1beta1
      kind: Release
      spec:
        forProvider:
          chart:
            name: loki-stack
            repository: https://grafana.github.io/helm-charts
            version: 2.9.11
          
          namespace: monitoring
          
          values:
            loki:
              persistence:
                enabled: true
                size: 50Gi
            
            promtail:
              enabled: true
  
  # Jaeger for tracing
  - name: jaeger
    base:
      apiVersion: helm.crossplane.io/v1beta1
      kind: Release
      spec:
        forProvider:
          chart:
            name: jaeger
            repository: https://jaegertracing.github.io/helm-charts
            version: 0.71.0
          
          namespace: monitoring
          
          values:
            collector:
              enabled: true
            query:
              enabled: true
            agent:
              enabled: true

# Usage
---
apiVersion: ml.hinterbuild.com/v1alpha1
kind: ObservabilityStackClaim
metadata:
  name: ml-monitoring
  namespace: ml-platform
spec:
  parameters:
    prometheusRetention: 30d
    lokiStorage: 50Gi

Connect to observability services.

Production Patterns & Best Practices

Pattern 1: Environment-Based Compositions

yaml
# Different compositions for dev/staging/prod
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: ml-env-dev
  labels:
    environment: dev
spec:
  # Smaller, cheaper resources for dev
  resources:
  - name: cluster
    base:
      spec:
        forProvider:
          machineType: n1-standard-4  # Smaller
          initialNodeCount: 1

---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: ml-env-prod
  labels:
    environment: production
spec:
  # Production-grade resources
  resources:
  - name: cluster
    base:
      spec:
        forProvider:
          machineType: n1-standard-16  # Larger
          initialNodeCount: 3
          enableAutopilot: true

# Usage
---
kind: MLEnvironmentClaim
metadata:
  name: my-environment
spec:
  compositionSelector:
    matchLabels:
      environment: production  # or 'dev'

Pattern 2: Self-Service with Namespaces

yaml
# Namespace-scoped claims
apiVersion: v1
kind: Namespace
metadata:
  name: ml-team-fraud

---
# Team provisions their own infrastructure
apiVersion: ml.hinterbuild.com/v1alpha1
kind: GPUClusterClaim
metadata:
  name: fraud-training
  namespace: ml-team-fraud  # Isolated per team
spec:
  parameters:
    name: fraud-cluster
    gpuType: t4
    maxNodes: 5
  
  # Resource quotas enforced
  compositionSelector:
    matchLabels:
      tier: standard

# RBAC limits team to their namespace
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ml-team-fraud-admin
  namespace: ml-team-fraud
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin
subjects:
- kind: Group
  name: ml-team-fraud
  apiGroup: rbac.authorization.k8s.io

Pattern 3: GitOps with ArgoCD

yaml
# ArgoCD manages Crossplane resources
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ml-infrastructure
  namespace: argocd
spec:
  project: ml-platform
  
  source:
    repoURL: https://github.com/company/ml-infra
    path: crossplane/claims
    targetRevision: main
  
  destination:
    server: https://kubernetes.default.svc
    namespace: ml-platform
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Combine with GitOps patterns.

Related implementation guides:

Primary references: official documentation, official documentation, official documentation, official documentation.

Crossplane for AI Infrastructure Decision Table

DecisionPrefer the simpler path whenAdd operational complexity when
ArchitectureOne component can own the contract and stateIndependent scaling or fault isolation is required
RolloutOffline replay covers the meaningful casesLive behavior requires shadow traffic and a canary
RecoveryA failed operation is safe to repeatPartial effects require idempotency or reconciliation
MeasurementOne service objective represents user impactQuality, latency, and cost need separate gates

Operating Crossplane for AI Infrastructure as a System

The implementation is only one part of Crossplane for AI Infrastructure. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship Crossplane for AI Infrastructure behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our Crossplane for AI Infrastructure engineering support.

Operating Crossplane for AI Infrastructure as a System

The implementation is only one part of Crossplane for AI Infrastructure. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship Crossplane for AI Infrastructure behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our Crossplane for AI Infrastructure engineering support.

Frequently Asked Questions

How does Crossplane compare to Terraform?

Crossplane is Kubernetes-native with continuous reconciliation, while Terraform is CLI-based with state files. Crossplane enables GitOps, self-service via CRDs, and unified app+infra management. Use Crossplane when you want Kubernetes-style declarative infrastructure.

Can I migrate existing Terraform to Crossplane?

Yes—use provider-terraform to wrap Terraform modules as Crossplane compositions. Gradually migrate resources to native Crossplane providers (provider-aws, provider-gcp).

What about state management?

Crossplane stores state in Kubernetes etcd, not separate state files. Backup etcd or use managed Kubernetes for disaster recovery.

How do I handle secrets?

Use External Secrets Operator to sync secrets from cloud secret managers (AWS Secrets Manager, GCP Secret Manager) into Kubernetes. Never commit secrets to Git.

Does Crossplane support all cloud resources?

Coverage varies by provider. AWS provider covers 900+ resources, GCP ~400, Azure ~600. For missing resources, use provider-terraform as a bridge.

How do I test compositions?

Use Crossplane CLI (crossplane beta render) to validate compositions locally, then deploy to dev cluster for integration testing before production.

Conclusion

Crossplane transforms AI infrastructure provisioning from fragmented tools to unified Kubernetes-native automation:

  • Declarative infrastructure—define GPU clusters, databases, storage as YAML
  • Self-service platform—ML teams provision without DevOps tickets
  • Multi-cloud abstraction—portable definitions across AWS, GCP, Azure
  • GitOps integration—ArgoCD manages apps AND infrastructure
  • Reusable compositions—platform team builds, data scientists consume
  • Kubernetes-native—kubectl for everything

For production ML platforms, Crossplane eliminates infrastructure bottlenecks.

At HinterBuild, we build self-service ML infrastructure platforms:

Contact us for infrastructure automation consulting.

Free consultation

Book a free consultation call on Crossplane & AI infrastructure automation

30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.

Book a meeting

Keep reading

Related articles

AI Systems · 9 min read

OpenAI-Compatible API with vLLM on Kubernetes

OpenAI-Compatible API with vLLM on Kubernetes guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable,.

Read post

MLOps · 9 min read

GPU Scheduling in Kubernetes: Complete NVIDIA Guide for ML

Learn gpu scheduling in kubernetes through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

MLOps · 12 min read

Deploy LLMs on Kubernetes: Complete GPU Autoscaling Guide

Learn deploy llms on kubernetes through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

DevOps · 14 min read

ArgoCD for ML Model Deployments: Production GitOps Patterns

Deploy ML models with ArgoCD GitOps: declarative manifests, MLflow registry sync, Kustomize overlays, Argo Rollouts canaries, and automated rollbacks.

Read post