Saga Pattern for Distributed Transactions

Table of Contents:

• Why Distributed Transactions Break
• Saga Pattern Fundamentals
• Choreography vs Orchestration
• Saga State Machine Schema
• Orchestrated Saga Implementation
• Compensating Transactions
• Failure Modes and Recovery
• Choreography Example
• Saga vs 2PC vs Outbox
• Production Operations Playbook
• Frequently Asked Questions

Why Distributed Transactions Break

Short answer: The saga pattern manages distributed transactions as a sequence of local transactions with compensating actions — avoiding two-phase commit (2PC), which does not survive real-world network partitions and cloud database limits.

A checkout flow touches payment, inventory, shipping, and notification services. Wrapping them in a single ACID transaction across Postgres, Stripe, and a warehouse API is impossible. 2PC (two-phase commit) promises atomicity but blocks on coordinator failure, performs poorly across regions, and is unsupported by most SaaS APIs.

Sagas accept eventual consistency: each step commits locally. If a later step fails, earlier steps run compensating transactions (refund payment, release inventory) rather than rolling back a global lock.

We design sagas in backend API engineering projects wherever microservices or bounded contexts must coordinate without a monolith database. Combined with the outbox pattern and background job workers, sagas survive partial failures that would corrupt naive async workflows.

Key Takeaways:

• Each saga step is a local transaction — no global lock
• Every forward action has a defined compensation
• Prefer orchestration when workflows exceed 3–4 steps
• Persist saga state — never rely on in-memory orchestration alone

Saga Pattern Fundamentals

A saga is a long-lived transaction split into steps:

PlaceOrder → ReserveInventory → ChargePayment → CreateShipment → SendConfirmation
                ↓ fail              ↓ fail
         ReleaseInventory      RefundPayment

Properties:

• Forward steps perform business actions (each commits independently)
• Compensating steps undo or logically reverse forward steps
• Idempotency required on every step — retries and duplicates are normal
• Saga log records current state for recovery after crashes

Sagas pair naturally with CQRS: command handlers start sagas; read models update when saga completes or compensates.

Choreography vs Orchestration

Choreography (Event-Driven)

Each service listens for events and publishes the next event. No central coordinator.

OrderSvc --OrderPlaced--> InventorySvc --InventoryReserved--> PaymentSvc ...

Pros: Loose coupling, no orchestrator to deploy
Cons: Hard to visualize flow, difficult debugging, cyclic dependencies creep in

Orchestration (Central Coordinator)

A saga orchestrator executes steps in order, calls services, handles failures, triggers compensations.

Pros: Explicit state machine, easier testing and observability
Cons: Orchestrator is a single point of logic (not necessarily SPoF if stateless + DB-backed)

Production default: Orchestration for checkout, onboarding, provisioning, and multi-tenant migrations. Choreography for simple notify chains.

Our data pipeline integrations often orchestrate multi-system ETL sagas with explicit rollback stages.

Saga State Machine Schema

Persist saga state in Postgres — orchestrator processes crash mid-flight:

CREATE TYPE saga_status AS ENUM (
    'running', 'completed', 'compensating', 'compensated', 'failed'
);

CREATE TYPE saga_step_status AS ENUM (
    'pending', 'running', 'completed', 'failed', 'compensated', 'skipped'
);

CREATE TABLE sagas (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    saga_type       TEXT NOT NULL,
    idempotency_key TEXT NOT NULL UNIQUE,
    status          saga_status NOT NULL DEFAULT 'running',
    current_step    INT NOT NULL DEFAULT 0,
    context         JSONB NOT NULL DEFAULT '{}',
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE saga_steps (
    id          BIGSERIAL PRIMARY KEY,
    saga_id     UUID NOT NULL REFERENCES sagas(id),
    step_index  INT NOT NULL,
    step_name   TEXT NOT NULL,
    status      saga_step_status NOT NULL DEFAULT 'pending',
    request     JSONB,
    response    JSONB,
    error       TEXT,
    started_at  TIMESTAMPTZ,
    finished_at TIMESTAMPTZ,
    UNIQUE (saga_id, step_index)
);

CREATE INDEX idx_sagas_running ON sagas (updated_at) WHERE status IN ('running', 'compensating');

Enqueue saga ticks as background jobs claimed via FOR UPDATE SKIP LOCKED.

Orchestrated Saga Implementation

Step Definitions (Python)

from dataclasses import dataclass
from typing import Callable, Awaitable

@dataclass
class SagaStep:
    name: str
    forward: Callable[[dict], Awaitable[dict]]
    compensate: Callable[[dict], Awaitable[None]] | None = None

CHECKOUT_SAGA = [
    SagaStep("reserve_inventory", reserve_inventory, release_inventory),
    SagaStep("charge_payment", charge_payment, refund_payment),
    SagaStep("create_shipment", create_shipment, cancel_shipment),
    SagaStep("send_confirmation", send_confirmation, None),  # no compensation needed
]

Orchestrator Core

async def advance_saga(conn, saga_id: str) -> None:
    saga = await conn.fetchrow("SELECT * FROM sagas WHERE id = $1 FOR UPDATE", saga_id)
    if saga["status"] not in ("running", "compensating"):
        return

    steps = CHECKOUT_SAGA if saga["saga_type"] == "checkout" else []
    ctx = json.loads(saga["context"])
    idx = saga["current_step"]

    if saga["status"] == "running":
        if idx >= len(steps):
            await conn.execute(
                "UPDATE sagas SET status = 'completed', updated_at = NOW() WHERE id = $1",
                saga_id,
            )
            return
        step = steps[idx]
        await conn.execute(
            "UPDATE saga_steps SET status = 'running', started_at = NOW() WHERE saga_id = $1 AND step_index = $2",
            saga_id, idx,
        )
        try:
            result = await step.forward(ctx)
            ctx.update(result)
            await conn.execute(
                """
                UPDATE saga_steps SET status = 'completed', response = $3::jsonb, finished_at = NOW()
                WHERE saga_id = $1 AND step_index = $2
                """,
                saga_id, idx, json.dumps(result),
            )
            await conn.execute(
                "UPDATE sagas SET current_step = $2, context = $3::jsonb, updated_at = NOW() WHERE id = $1",
                saga_id, idx + 1, json.dumps(ctx),
            )
            await enqueue_saga_tick(conn, saga_id)
        except Exception as exc:
            await conn.execute(
                "UPDATE saga_steps SET status = 'failed', error = $3, finished_at = NOW() WHERE saga_id = $1 AND step_index = $2",
                saga_id, idx, str(exc)[:2000],
            )
            await conn.execute(
                "UPDATE sagas SET status = 'compensating', updated_at = NOW() WHERE id = $1",
                saga_id,
            )
            await enqueue_saga_tick(conn, saga_id)

    elif saga["status"] == "compensating":
        await run_compensations(conn, saga_id, steps, ctx, idx)

Each step publishes domain events via outbox after local success — downstream CQRS projections stay decoupled.

Go Saga Step Example

func ChargePayment(ctx context.Context, sctx SagaContext) (SagaContext, error) {
    idempotencyKey := fmt.Sprintf("saga:%s:charge", sctx.SagaID)
    charge, err := stripeClient.Charge(ctx, stripe.ChargeParams{
        Amount:   sctx.TotalCents,
        Customer: sctx.StripeCustomerID,
        IdempotencyKey: idempotencyKey,
    })
    if err != nil {
        return sctx, err
    }
    sctx.PaymentIntentID = charge.ID
    return sctx, nil
}

func RefundPayment(ctx context.Context, sctx SagaContext) error {
    if sctx.PaymentIntentID == "" {
        return nil // nothing to compensate
    }
    _, err := stripeClient.Refund(ctx, sctx.PaymentIntentID)
    return err
}

External APIs (Stripe, Shippo) require idempotency keys derived from saga ID + step name.

Compensating Transactions

Compensations are not database ROLLBACK — they are business operations that semantically undo forward steps.

Compensation Rules

async def run_compensations(conn, saga_id, steps, ctx, failed_idx):
    completed = await conn.fetch(
        """
        SELECT step_index FROM saga_steps
        WHERE saga_id = $1 AND status = 'completed'
        ORDER BY step_index DESC
        """,
        saga_id,
    )
    for row in completed:
        step = steps[row["step_index"]]
        if not step.compensate:
            continue
        try:
            await step.compensate(ctx)
            await conn.execute(
                "UPDATE saga_steps SET status = 'compensated' WHERE saga_id = $1 AND step_index = $2",
                saga_id, row["step_index"],
            )
        except Exception as exc:
            await conn.execute(
                "UPDATE sagas SET status = 'failed', updated_at = NOW() WHERE id = $1",
                saga_id,
            )
            await alert_oncall(saga_id, step.name, exc)
            return
    await conn.execute(
        "UPDATE sagas SET status = 'compensated', updated_at = NOW() WHERE id = $1",
        saga_id,
    )

Some compensations are impossible (email sent, physical pick started). Design pivot to manual intervention — saga status failed with ops dashboard, not infinite retry.

Document compensation semantics in runbooks alongside observability dashboards.

Failure Modes and Recovery

Orchestrator Crash Mid-Step

Step marked running but never completed. Recovery job resets stale steps:

UPDATE saga_steps SET status = 'pending', started_at = NULL
WHERE status = 'running' AND started_at < NOW() - INTERVAL '10 minutes';

UPDATE sagas SET updated_at = NOW()
WHERE id IN (SELECT DISTINCT saga_id FROM saga_steps WHERE status = 'pending');
-- Re-enqueue saga tick jobs

Duplicate Step Execution

Always check idempotency before forward/compensate:

async def charge_payment(ctx: dict) -> dict:
    existing = await payment_repo.find_by_idempotency(f"saga:{ctx['saga_id']}:charge")
    if existing:
        return {"payment_id": existing.id}
    # proceed with charge

Partial Compensation Failure

Saga enters failed — human ops completes refund or inventory fix. Never auto-retry compensation blindly on non-idempotent external APIs.

Timeout vs Failure

Distinguish slow (extend timeout, poll provider) from failed (compensate). Payment APIs often need async status polling as a sub-step.

For AI workflow steps (LLM classification, content generation), wrap with structured output validation and treat validation failure as compensatable business errors — not transient retries.

Saga vs 2PC vs Outbox

Composition: Each saga step uses local ACID + outbox for events. Sagas coordinate across services; outbox guarantees delivery from each service.

Not every workflow needs a saga. Single-service transactions with background jobs suffice for most CRUD apps.

Choreography Example

Simple notification chains suit choreography — no central orchestrator:

OrderPlaced → InventoryReserved → PaymentProcessed → ShipmentCreated
     ↑              ↑                    ↑
  OrderSvc      InventorySvc         PaymentSvc

Each service writes local state + outbox event. Downstream services consume and react:

async def on_order_placed(event: dict) -> None:
    reservation_id = await inventory_service.reserve(event["orderId"], event["items"])
    await publish_outbox("inventory", reservation_id, "InventoryReserved", {
        "orderId": event["orderId"],
        "reservationId": reservation_id,
    })

When choreography breaks down: Payment fails after inventory reserves — who triggers ReleaseInventory? Without an orchestrator, you need compensating events (PaymentFailed) and careful contract design across teams. After the third "who publishes the compensate event?" meeting, adopt orchestration.

Choreography works when:

• Steps are linear with clear event ownership
• Compensations are rare or handled manually
• Team shares event schema registry and event-driven architecture conventions

For onboarding flows with 7+ steps and branching (KYC pass/fail, regional rules), orchestration with persisted saga state wins.

Production Operations Playbook

Dashboards

Track per saga_type:

• Running count, completion rate, compensation rate
• p95 duration (start → completed/compensated)
• Stuck sagas (running > 30 min)
• Failed compensations requiring manual intervention

Runbook: Stuck Saga

1. Query saga log: SELECT * FROM saga_steps WHERE saga_id = $1 ORDER BY step_index
2. Identify last completed step and error on failed step
3. Check external provider status (Stripe dashboard, WMS API)
4. If provider succeeded but local state failed → manual reconcile + mark step completed
5. If provider failed → trigger compensation or resume from failed step with idempotency
6. Never delete saga rows — audit requirement

Runbook: Compensation Storm

Multiple sagas compensating simultaneously may overwhelm refund APIs. Implement compensation rate limiting — separate queue with lower concurrency than forward steps.

INSERT INTO background_jobs (queue, job_type, payload)
VALUES ('saga-compensate', 'saga_tick', '{"sagaId": $1}');
-- Dedicated worker pool: max 5 concurrent compensations

Testing Matrix

Load-test saga orchestrator with connection pooling tuned for short transactions — long-held locks on saga rows block concurrent ticks for the same saga ID (expected) but should not block unrelated sagas.

Integrate saga visibility into customer support tools — "payment processing" status reduces tickets better than generic errors. API design should expose saga state on resource endpoints.

Primary references: official documentation, official documentation, official documentation.

Operating Saga Pattern for Distributed Transactions as a System

The implementation is only one part of Saga Pattern for Distributed Transactions. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship Saga Pattern for Distributed Transactions behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our Saga Pattern for Distributed Transactions engineering support.

Frequently Asked Questions

What is the saga pattern?

The saga pattern implements distributed transactions as a sequence of local transactions. If a step fails, compensating transactions undo previous steps instead of rolling back a global lock.

Saga vs two-phase commit (2PC)?

2PC blocks resources until all participants vote commit. Sagas commit each step immediately and compensate on failure — better suited for microservices and external APIs.

What is a compensating transaction?

A compensating transaction semantically reverses a completed saga step — e.g., refund a charge, release reserved inventory. It is not a database ROLLBACK.

Choreography or orchestration — which should I use?

Use orchestration for complex workflows with branching, 4+ steps, or cross-team ownership. Use choreography for simple event chains with mature contracts and few steps.

Are sagas eventually consistent?

Yes. Between steps, the system is in an intermediate state. UI should reflect pending states (payment processing, awaiting shipment) not binary success/failure only.

How do I test sagas?

Unit-test each forward/compensate pair with idempotency cases. Integration-test failure injection at each step index. Assert saga log states and downstream side effects.

Can sagas run synchronously in the API request?

Avoid long synchronous sagas — external API latency multiplies. Start saga, return 202 Accepted with saga ID, poll or subscribe for completion. Orchestrator ticks via job queue.

How do sagas relate to CQRS and outbox?

CQRS command handlers often start sagas. Each step commits locally and writes outbox events for read-side projections and notifications. See CQRS guide and outbox guide.

What is semantic lock in sagas?

A semantic lock marks an aggregate as "in saga" — rejecting conflicting commands until the saga completes or compensates. Example: block duplicate checkout while payment saga runs. Implement via status column (pending_saga) on the aggregate, not database row locks held for minutes.

When should I use Temporal or Cadence instead of custom sagas?

Managed workflow engines (Temporal, Cadence) excel when workflows are long-running (days/weeks), require complex timers, or need built-in visibility. Custom Postgres-backed sagas suit teams wanting minimal infrastructure and workflows completing in minutes. Many teams start custom, migrate to Temporal when operational burden grows.

Conclusion

The saga pattern is how production systems handle distributed transactions without 2PC:

• Local commits per step with explicit compensations
• Orchestrated state machines for complex flows
• Persisted saga log + idempotent steps for crash recovery
• Outbox and job queues as the execution and messaging layer

Design compensations before forward steps. Instrument saga duration, compensation rate, and stuck running states. Accept intermediate consistency — and make it visible to users.

At HinterBuild:

• Backend API Engineering
• Data Pipelines & Integrations
• Cloud Infrastructure & DevOps
• Observability & Monitoring

Schedule a consultation for distributed workflow architecture review.