Outbox Pattern for Guaranteed Message Delivery

Table of Contents:

• The Dual-Write Problem
• Outbox Pattern Overview
• Outbox Table Schema
• Writing to the Outbox in Transactions
• Relay Worker Implementation
• Idempotency at Consumers
• Outbox vs Change Data Capture
• Integration with CQRS and Sagas
• Monitoring and Alerting
• Production Case Study: Payment Events
• Frequently Asked Questions

The Dual-Write Problem

Short answer: The outbox pattern solves the dual-write problem — when you must persist domain data and publish a message atomically, but databases and brokers cannot share a single transaction.

The failure mode is familiar: you commit an order to Postgres, then publish OrderCreated to Kafka. The broker times out. Did the message send? Retry risks duplicate events. Skip risks lost notifications — no email, no inventory update, no webhook.

Or the reverse: message publishes, database rolls back. Downstream systems act on an order that never existed.

We implement the transactional outbox pattern in nearly every backend API engineering project that publishes domain events. Combined with background job workers using FOR UPDATE SKIP LOCKED, it delivers at-least-once messaging with operational simplicity — often without Kafka on day one.

Key Takeaways:

• Write events to an outbox table in the same DB transaction as domain data
• Relay worker publishes and marks rows processed — never publish inside the request handler
• Consumers must be idempotent — duplicates will happen
• Monitor outbox lag (oldest unpublished row age) as an SLO

Outbox Pattern Overview

┌─────────────┐     same txn      ┌──────────────┐
│   API /     │ ────────────────► │   Postgres   │
│   Command   │   orders + outbox │   (outbox)   │
└─────────────┘                   └──────┬───────┘
                                         │ poll SKIP LOCKED
                                         ▼
                                  ┌──────────────┐
                                  │ Relay Worker │
                                  └──────┬───────┘
                                         │ publish
                                         ▼
                                  ┌──────────────┐
                                  │ Kafka / SNS  │
                                  │ / webhooks   │
                                  └──────────────┘

Steps:

1. Business transaction inserts/updates domain rows and inserts outbox row(s)
2. Transaction commits — both succeed or both roll back
3. Relay worker claims unpublished outbox rows
4. Worker publishes to message broker or calls webhook
5. Worker marks row published (or deletes after retention window)

No dual write. One atomic commit.

Outbox Table Schema

CREATE TYPE outbox_status AS ENUM ('pending', 'processing', 'published', 'failed');

CREATE TABLE outbox (
    id              BIGSERIAL PRIMARY KEY,
    aggregate_type  TEXT NOT NULL,
    aggregate_id    TEXT NOT NULL,
    event_type      TEXT NOT NULL,
    payload         JSONB NOT NULL,
    metadata        JSONB NOT NULL DEFAULT '{}',
    status          outbox_status NOT NULL DEFAULT 'pending',
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    published_at    TIMESTAMPTZ,
    locked_at       TIMESTAMPTZ,
    locked_by       TEXT,
    attempts        INT NOT NULL DEFAULT 0,
    last_error      TEXT
);

CREATE INDEX idx_outbox_pending
    ON outbox (created_at ASC, id ASC)
    WHERE status = 'pending';

-- Optional: dedupe at publish time
CREATE UNIQUE INDEX idx_outbox_dedupe
    ON outbox (aggregate_type, aggregate_id, event_type, (payload->>'idempotencyKey'))
    WHERE (payload->>'idempotencyKey') IS NOT NULL;

Include correlation_id and causation_id in metadata for distributed tracing across data pipeline integrations.

Writing to the Outbox in Transactions

Never call HTTP or Kafka inside a database transaction. Insert the outbox row instead.

Python (asyncpg)

async def confirm_order(conn, order_id: int) -> None:
    async with conn.transaction():
        updated = await conn.fetchrow(
            """
            UPDATE orders SET status = 'confirmed', updated_at = NOW()
            WHERE id = $1 AND status = 'placed'
            RETURNING id, customer_id, total_cents
            """,
            order_id,
        )
        if not updated:
            raise OrderNotFoundError(order_id)

        await conn.execute(
            """
            INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload, metadata)
            VALUES ('order', $1, 'OrderConfirmed', $2::jsonb, $3::jsonb)
            """,
            str(order_id),
            {
                "orderId": order_id,
                "customerId": updated["customer_id"],
                "totalCents": updated["total_cents"],
                "idempotencyKey": f"order-confirmed:{order_id}",
            },
            {"correlationId": get_correlation_id()},
        )

Go

func ConfirmOrder(ctx context.Context, tx pgx.Tx, orderID int64) error {
    var customerID int64
    var totalCents int
    err := tx.QueryRow(ctx, `
        UPDATE orders SET status = 'confirmed', updated_at = NOW()
        WHERE id = $1 AND status = 'placed'
        RETURNING customer_id, total_cents`, orderID).Scan(&customerID, &totalCents)
    if err != nil {
        return err
    }
    payload, _ := json.Marshal(map[string]any{
        "orderId": orderID, "customerId": customerID, "totalCents": totalCents,
        "idempotencyKey": fmt.Sprintf("order-confirmed:%d", orderID),
    })
    _, err = tx.Exec(ctx, `
        INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
        VALUES ('order', $1, 'OrderConfirmed', $2::jsonb)`,
        fmt.Sprint(orderID), payload)
    return err
}

API returns 200 immediately. Side effects happen when relay publishes and CQRS projections consume events.

Relay Worker Implementation

The relay uses the same SKIP LOCKED claim pattern as job queues:

WITH next_row AS (
    SELECT id FROM outbox
    WHERE status = 'pending'
    ORDER BY created_at ASC, id ASC
    FOR UPDATE SKIP LOCKED
    LIMIT 1
)
UPDATE outbox o
SET status = 'processing', locked_at = NOW(), locked_by = $1, attempts = attempts + 1
FROM next_row WHERE o.id = next_row.id
RETURNING o.*;

Python Relay with SNS

import boto3
import json

sns = boto3.client("sns")
TOPIC_ARN = "arn:aws:sns:us-east-1:123456789:domain-events"

async def publish_outbox_row(row) -> None:
    message = {
        "eventType": row["event_type"],
        "aggregateType": row["aggregate_type"],
        "aggregateId": row["aggregate_id"],
        "payload": json.loads(row["payload"]),
        "metadata": json.loads(row["metadata"]),
        "outboxId": row["id"],
    }
    sns.publish(
        TopicArn=TOPIC_ARN,
        Message=json.dumps(message),
        MessageAttributes={
            "eventType": {"DataType": "String", "StringValue": row["event_type"]},
        },
    )

async def relay_loop(pool, worker_id: str) -> None:
    while True:
        async with pool.acquire() as conn:
            async with conn.transaction():
                row = await conn.fetchrow(CLAIM_OUTBOX_SQL, worker_id)
            if not row:
                await asyncio.sleep(0.2)
                continue
            try:
                await publish_outbox_row(row)
                await conn.execute(
                    """
                    UPDATE outbox SET status = 'published', published_at = NOW()
                    WHERE id = $1
                    """,
                    row["id"],
                )
            except Exception as exc:
                await conn.execute(
                    """
                    UPDATE outbox SET status = 'pending', last_error = $2,
                        locked_at = NULL, locked_by = NULL
                    WHERE id = $1
                    """,
                    row["id"], str(exc)[:2000],
                )

Go Relay with Kafka

func relayOnce(ctx context.Context, pool *pgxpool.Pool, producer *kafka.Writer, workerID string) error {
    tx, err := pool.Begin(ctx)
    if err != nil {
        return err
    }
    var row OutboxRow
    err = tx.QueryRow(ctx, claimOutboxSQL, workerID).Scan(/* fields */)
    if err == pgx.ErrNoRows {
        tx.Rollback(ctx)
        return nil
    }
    if err != nil {
        tx.Rollback(ctx)
        return err
    }
    if err := tx.Commit(ctx); err != nil {
        return err
    }
    msg, _ := json.Marshal(row)
    err = producer.WriteMessages(ctx, kafka.Message{
        Topic: "domain-events",
        Key:   []byte(row.AggregateType + ":" + row.AggregateID),
        Value: msg,
    })
    if err != nil {
        return markOutboxFailed(ctx, pool, row.ID, err)
    }
    return markOutboxPublished(ctx, pool, row.ID)
}

Run multiple relay instances — SKIP LOCKED prevents duplicate claims. Publishing may still duplicate on crash after publish but before mark; consumers handle idempotency.

Alert when MAX(created_at) FILTER (WHERE status = 'pending') exceeds SLA (e.g., 30 seconds).

Idempotency at Consumers

Outbox guarantees at-least-once delivery to the broker. Network retries and crash timing produce duplicates at consumers.

CREATE TABLE processed_events (
    idempotency_key TEXT PRIMARY KEY,
    processed_at    TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

async def handle_order_confirmed(conn, event: dict) -> None:
    key = event["payload"]["idempotencyKey"]
    inserted = await conn.fetchval(
        """
        INSERT INTO processed_events (idempotency_key)
        VALUES ($1) ON CONFLICT DO NOTHING RETURNING idempotency_key
        """,
        key,
    )
    if not inserted:
        return  # duplicate — safe skip

    await send_confirmation_email(event["payload"])
    await enqueue_inventory_job(conn, event["payload"]["orderId"])

This mirrors idempotency patterns in background job processing.

For LLM-triggered side effects, validate outputs with structured output schemas before acting on event payloads.

Outbox vs Change Data Capture

CDC captures row changes from the WAL. It works when events map 1:1 to table mutations. Outbox wins when events are richer than row diffs — OrderConfirmed with computed fields, privacy-redacted payloads, or intent-level semantics.

Hybrid: outbox for domain events, CDC for data pipeline integrations syncing analytics warehouses.

The transactional inbox pattern ( cousin to outbox ) deduplicates inbound messages at consumers — store processed message IDs, skip duplicates. Use outbox for outbound, inbox for inbound, when integrating with external partners who may retry webhook delivery.

Schema evolution: add schema_version to outbox payload JSON. Consumers switch on version during rolling deploys. Never mutate published event shapes in place — append new event types instead.

Integration with CQRS and Sagas

CQRS Projections

Outbox events feed projection workers that rebuild read models. Command handler writes order + outbox; relay publishes; projection consumer updates order_summary_read. See CQRS in practice.

Saga Orchestration

Saga steps often start from outbox events:

1. PaymentCaptured outbox event published
2. Inventory service consumes, reserves stock
3. On failure, saga emits CompensatePayment via its own outbox

See saga pattern for distributed transactions.

Ordering Guarantees

Use aggregate ID as Kafka partition key to preserve order per order/customer/account. Cross-aggregate ordering is neither guaranteed nor required in most domains.

Distributed workflows spanning multiple services combine outbox-per-service with saga orchestration — each local commit outboxes an event; sagas coordinate the cross-service sequence.

Monitoring and Alerting

Outbox failures are silent until downstream systems stop receiving events. Required metrics:

-- Dashboard query: outbox lag in seconds
SELECT EXTRACT(EPOCH FROM (NOW() - MIN(created_at))) AS lag_seconds
FROM outbox WHERE status = 'pending';

-- Failed publish rate (last hour)
SELECT COUNT(*) FROM outbox
WHERE status = 'failed' AND updated_at > NOW() - INTERVAL '1 hour';

Log every publish with outbox_id, event_type, and aggregate_id. Correlate with consumer logs via shared correlation_id in metadata.

Relay workers belong on the same observability stack as APIs — distributed traces should show API → outbox insert → relay publish → consumer handler as one trace when possible.

During deploys, relays drain pending rows before shutdown (graceful termination). New relay version picks up unprocessed rows — no message loss if Postgres outbox is the source of truth.

For high-volume systems, partition outbox by month and archive published partitions to cold storage. Unpublished rows never archive.

Production Case Study: Payment Events

A fintech API processed card captures synchronously — POST /capture called Stripe, updated ledger, and emitted webhook to merchant endpoints inline. Stripe latency spikes caused API timeouts; worse, some requests timed out after Stripe succeeded, leading to duplicate capture retries and double charges.

Outbox migration:

1. capture endpoint commits ledger update + outbox row (PaymentCaptured) — returns 200 in ~40ms
2. Relay publishes to SNS → SQS fan-out
3. Webhook worker consumes SQS, delivers merchant callbacks with retries
4. Reconciliation job compares Stripe state vs ledger nightly

async def capture_payment(conn, payment_id: str) -> None:
    async with conn.transaction():
        row = await conn.fetchrow(
            "UPDATE payments SET status = 'captured' WHERE id = $1 AND status = 'authorized' RETURNING *",
            payment_id,
        )
        if not row:
            raise PaymentError("invalid state")
        await conn.execute(
            """
            INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
            VALUES ('payment', $1, 'PaymentCaptured', $2::jsonb)
            """,
            payment_id,
            {"paymentId": payment_id, "amount": row["amount_cents"], "merchantId": row["merchant_id"]},
        )

Outcomes after 60 days:

• API p99 latency: 2.1s → 85ms
• Duplicate capture incidents: eliminated (idempotent Stripe keys + outbox dedupe)
• Merchant webhook delivery: 99.7% within 30s (async retries)

The merchant-facing SLA shifted from "capture is synchronous" to "capture confirmed immediately; webhook within 30 seconds" — contract change communicated upfront.

This integrates with background job processing for webhook delivery workers using SKIP LOCKED claiming when merchants prefer pull-based retry queues over push webhooks.

Backend API engineering reviews often catch dual-write paths in payment flows — outbox is the standard remediation.

Common mistake: Developers publish to Kafka first, then commit DB — inverted order loses events on DB rollback and creates ghost messages on DB failure after publish. Always commit outbox first; relay publishes second. The relay is the only component that talks to the broker for domain events.

For multi-region deployments, outbox rows commit in the primary region; relays publish to a globally replicated topic. Consumers must tolerate regional failover duplicates — idempotency keys are mandatory, not optional.

Primary references: official documentation, official documentation, official documentation, official documentation.

Operating Outbox Pattern for Guaranteed Message Delivery as a System

The implementation is only one part of Outbox Pattern for Guaranteed Message Delivery. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship Outbox Pattern for Guaranteed Message Delivery behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our Outbox Pattern for Guaranteed Message Delivery engineering support.

Frequently Asked Questions

What is the outbox pattern?

The outbox pattern stores outbound messages in a database table within the same transaction as business data. A separate relay process reads the outbox and publishes to a message broker, ensuring messages are never lost when the database commit succeeds.

Is outbox exactly-once delivery?

No. Outbox provides at-least-once delivery to the broker. Exactly-once end-to-end requires idempotent consumers and often broker idempotence keys. Design consumers to tolerate duplicates.

Where should the relay worker run?

As a dedicated background worker process — separate from API servers — with multiple instances for availability. Same deployment patterns as job queue workers.

Should I delete outbox rows after publishing?

Archive or delete after retention (7–30 days). Keeping published rows aids debugging and replay. Partition by month if volume is high.

Can I use outbox without Kafka?

Yes. Relay can call webhooks, SQS, Redis pub/sub, or enqueue background jobs directly — outbox is broker-agnostic.

How do I replay outbox events?

Reset selected rows to pending or insert new outbox rows with a replay flag. Consumers must remain idempotent. Prefer rebuilding projections from event log snapshots for large replays.

What metrics should I monitor?

Outbox lag (oldest pending age), publish error rate, relay throughput, and consumer lag downstream. Page when lag exceeds business SLA.

Outbox vs two-phase commit (2PC)?

2PC coordinates distributed transactions across databases — fragile and rarely used in cloud-native systems. Outbox achieves practical consistency with eventual delivery — simpler and more operable.

How many relay workers do I need?

Start with 2–3 relay instances for availability. Scale when outbox lag grows — relays are CPU and network bound, not database bound for typical event sizes. A single relay handles 500–2000 events/second on modest hardware with batch publish optimizations.

Should outbox payloads be large?

Keep payloads small — IDs and changed fields, not full aggregate snapshots. Consumers fetch details from authoritative store if needed. Large JSON blobs bloat outbox tables and slow relay throughput. Reference data pipeline integrations for bulk analytics export, not outbox.

Conclusion

The outbox pattern is the production answer to reliable messaging from transactional systems:

• One commit for domain data and events — no dual write
• Relay workers with SKIP LOCKED for concurrent, safe publishing
• Idempotent consumers for at-least-once reality
• Composes with CQRS, sagas, and job queues

Skip publishing inside request handlers. Write to the outbox, let relays do the network I/O, and instrument lag before your users notice missing emails.

At HinterBuild:

• Backend API Engineering
• Data Pipelines & Integrations
• Cloud Infrastructure & DevOps

Schedule a consultation for reliable messaging architecture review.