Event-Driven Architecture: When to Use It and How to

Table of Contents:

• What Event-Driven Architecture Actually Means
• When to Use EDA (and When Not To)
• Core Patterns: Pub/Sub, Event Sourcing, CQRS
• Message Broker Selection
• The Outbox Pattern (Production Essential)
• Consumer Design: Idempotency and Ordering
• Observability for Event Systems
• Frequently Asked Questions

What Event-Driven Architecture Actually Means

Short answer: Event-driven architecture (EDA) decouples services by having them communicate through events — immutable records of something that happened — rather than synchronous request-response calls.

When a customer completes an order, a synchronous system calls the inventory service, then the billing service, then the notification service in sequence. An event-driven system publishes an OrderCompleted event; inventory, billing, and notifications each subscribe independently and react on their own timeline.

We have built event-driven architectures for order processing, data pipeline orchestration, and AI agent workflows at HinterBuild. The pattern excels at decoupling and scale — but it introduces complexity that synchronous APIs avoid entirely. This guide covers when that trade-off is worth it and how to implement EDA without creating an unmaintainable message spaghetti.

Key Takeaways:

• Use EDA when services need loose coupling and can tolerate eventual consistency
• The outbox pattern is non-negotiable for reliable event publishing from databases
• Every consumer must be idempotent — at-least-once delivery is the default guarantee
• Start with a simple pub/sub topology; add event sourcing and CQRS only when justified

When to Use EDA (and When Not To)

Not every system needs events. The decision framework we use on backend API engineering engagements:

Use Event-Driven Architecture When

Skip EDA When

The Honest Trade-Off

Event-driven architecture trades debuggability and consistency for scalability and decoupling. A synchronous call chain fails loudly in one place. An event system fails silently when a consumer stops processing — unless you build observability from day one.

Our data pipelines and integrations team often starts clients on synchronous APIs and introduces events when a specific bottleneck (notification delays, inventory sync lag) justifies the complexity.

Core Patterns: Pub/Sub, Event Sourcing, CQRS

Pattern 1: Pub/Sub (Start Here)

The simplest event-driven architecture pattern. A publisher emits events to a topic; subscribers receive copies and process independently.

Order Service  →  [order.completed]  →  Inventory Service
                                      →  Billing Service
                                      →  Notification Service

from dataclasses import dataclass, asdict
import json
import boto3

@dataclass
class OrderCompletedEvent:
    event_type: str = "order.completed"
    order_id: str = ""
    customer_id: str = ""
    total_cents: int = 0
    timestamp: str = ""

async def publish_order_completed(order: Order) -> None:
    event = OrderCompletedEvent(
        order_id=order.id,
        customer_id=order.customer_id,
        total_cents=order.total_cents,
        timestamp=datetime.utcnow().isoformat(),
    )
    sns = boto3.client("sns")
    sns.publish(
        TopicArn=settings.ORDER_EVENTS_TOPIC_ARN,
        Message=json.dumps(asdict(event)),
        MessageAttributes={
            "event_type": {"DataType": "String", "StringValue": event.event_type}
        },
    )

# Consumer — inventory service
async def handle_order_completed(event: dict) -> None:
    order_id = event["order_id"]

    # Idempotency check — see idempotency guide
    if await already_processed(order_id, "inventory.reserve"):
        return

    await reserve_inventory(order_id)
    await mark_processed(order_id, "inventory.reserve")

Pattern 2: Event Sourcing

Instead of storing current state, store the sequence of events that led to it. Current state is derived by replaying events.

Use when: Full audit trail is a legal requirement, you need temporal queries ("what was the balance on March 1?"), or complex domain logic benefits from event replay.

Skip when: Simple CRUD, team unfamiliar with event sourcing, or read patterns do not benefit from replay.

# Event store — append-only log
class EventStore:
    async def append(self, stream_id: str, event: DomainEvent) -> None:
        await self.db.execute(
            """
            INSERT INTO events (stream_id, event_type, payload, version, created_at)
            VALUES ($1, $2, $3,
                (SELECT COALESCE(MAX(version), 0) + 1 FROM events WHERE stream_id = $1),
                NOW())
            """,
            stream_id, event.event_type, event.payload,
        )

    async def load_stream(self, stream_id: str) -> list[DomainEvent]:
        rows = await self.db.fetch(
            "SELECT * FROM events WHERE stream_id = $1 ORDER BY version",
            stream_id,
        )
        return [DomainEvent.from_row(r) for r in rows]

# Rebuild state from events
async def get_account_balance(account_id: str) -> int:
    events = await event_store.load_stream(f"account:{account_id}")
    balance = 0
    for event in events:
        if event.event_type == "funds.deposited":
            balance += event.payload["amount_cents"]
        elif event.event_type == "funds.withdrawn":
            balance -= event.payload["amount_cents"]
    return balance

Pattern 3: CQRS (Command Query Responsibility Segregation)

Separate write models (commands that produce events) from read models (projections optimized for queries).

Command → Write Model → Events → Projections → Read Model (optimized for queries)

CQRS pairs naturally with event sourcing but can stand alone. A write-side PostgreSQL database handles commands; a read-side Elasticsearch index serves search queries, updated by event consumers.

Production gotcha: Read models lag behind writes (eventual consistency). If your UI shows "order placed" but search still returns empty for 2 seconds, users notice. Design UX for eventual consistency or use read-your-writes patterns.

Message Broker Selection

Choosing the wrong broker creates operational pain for years. Here is the decision matrix we use:

When to Choose Kafka

Choose Apache Kafka when you need:

• Event replay (reprocess last 30 days of events)
• Stream processing (Kafka Streams, Flink)
• Throughput above 50,000 events/second sustained
• Multiple consumer groups reading the same events independently

Deploy Kafka on Kubernetes platform engineering infrastructure with proper monitoring — under-provisioned Kafka clusters fail catastrophically under partition rebalance.

When to Choose SQS

Choose Amazon SQS when you need:

• Simple point-to-point or fan-out messaging
• Managed service with zero operational overhead
• AWS-native integration (Lambda triggers, SNS fan-out)
• Throughput under 10,000 messages/second per queue

# SQS consumer with visibility timeout handling
import boto3

sqs = boto3.client("sqs")

async def poll_and_process(queue_url: str) -> None:
    while True:
        response = sqs.receive_message(
            QueueUrl=queue_url,
            MaxNumberOfMessages=10,
            WaitTimeSeconds=20,          # long polling
            VisibilityTimeout=60,         # time to process before re-delivery
        )
        for message in response.get("Messages", []):
            try:
                event = json.loads(message["Body"])
                await process_event(event)
                sqs.delete_message(
                    QueueUrl=queue_url,
                    ReceiptHandle=message["ReceiptHandle"],
                )
            except Exception as e:
                logger.error(f"Processing failed: {e}")
                # message returns to queue after visibility timeout

The Outbox Pattern (Production Essential)

The most common event-driven architecture failure: you write to the database and publish an event in the same request — but one succeeds and the other fails.

1. INSERT order into database     ✅
2. Publish order.completed event  ❌ (network timeout)
→ Order exists but no downstream service knows about it

The transactional outbox pattern solves this by writing the event to an outbox table in the same database transaction as the business data. A separate relay process publishes events from the outbox.

async def create_order(order_data: OrderCreate, session: AsyncSession) -> Order:
    async with session.begin():
        # Business write
        order = Order(**order_data.dict())
        session.add(order)
        await session.flush()  # get order.id

        # Outbox write — same transaction
        outbox_event = OutboxEvent(
            aggregate_id=order.id,
            event_type="order.completed",
            payload=json.dumps({
                "order_id": order.id,
                "customer_id": order.customer_id,
                "total_cents": order.total_cents,
            }),
            status="pending",
        )
        session.add(outbox_event)
    # Both committed atomically
    return order

# Outbox relay — runs as background worker
async def relay_outbox_events() -> None:
    while True:
        events = await db.fetch(
            """
            SELECT * FROM outbox_events
            WHERE status = 'pending'
            ORDER BY created_at
            LIMIT 100
            FOR UPDATE SKIP LOCKED
            """
        )
        for event in events:
            try:
                await message_broker.publish(event.event_type, event.payload)
                await db.execute(
                    "UPDATE outbox_events SET status = 'published' WHERE id = $1",
                    event.id,
                )
            except Exception as e:
                logger.error(f"Relay failed for event {event.id}: {e}")
                await db.execute(
                    "UPDATE outbox_events SET retry_count = retry_count + 1 WHERE id = $1",
                    event.id,
                )
        await asyncio.sleep(1)

FOR UPDATE SKIP LOCKED prevents multiple relay workers from processing the same event — critical when running multiple relay instances for high availability.

Our backend API engineering team implements outbox tables as standard in every event-publishing service.

Consumer Design: Idempotency and Ordering

At-Least-Once Delivery Is the Default

Every major message broker delivers messages at least once. Network failures, consumer crashes, and visibility timeout expiry all cause redelivery. Your consumers must handle duplicate events gracefully.

See our complete guide on idempotency in distributed systems for implementation patterns.

async def process_with_idempotency(event: dict) -> None:
    idempotency_key = f"{event['event_type']}:{event['order_id']}"

    # Atomic check-and-set in Redis
    was_set = await redis.set(
        f"idempotency:{idempotency_key}",
        "1",
        nx=True,       # only set if not exists
        ex=86400,      # 24-hour TTL
    )
    if not was_set:
        logger.info(f"Duplicate event skipped: {idempotency_key}")
        return

    await handle_business_logic(event)

Event Ordering

Kafka guarantees ordering within a partition. SQS standard queues do not guarantee ordering. SQS FIFO queues guarantee ordering within a message group.

If order.created must arrive before order.shipped, use:

• Kafka with order_id as partition key
• SQS FIFO with order_id as message group ID
• Or design consumers to tolerate out-of-order events (preferred when possible)

Dead Letter Queues

Every consumer needs a dead letter queue (DLQ) for messages that fail after maximum retries.

MAX_RETRIES = 5

async def process_with_dlq(message: dict, retry_count: int) -> None:
    try:
        await handle_event(message)
    except NonRetryableError as e:
        await send_to_dlq(message, reason=str(e))
    except RetryableError as e:
        if retry_count >= MAX_RETRIES:
            await send_to_dlq(message, reason=f"Max retries exceeded: {e}")
        else:
            raise  # broker will redeliver

Monitor DLQ depth with observability and monitoring — a growing DLQ is the earliest signal of consumer bugs or schema mismatches.

Observability for Event Systems

Synchronous systems fail loudly. Event-driven systems fail quietly — a consumer stops processing and nobody notices until downstream data is stale.

Required Observability

Distributed Tracing Across Events

Propagate trace context through event payloads:

from opentelemetry import trace
from opentelemetry.propagate import inject, extract

tracer = trace.get_tracer(__name__)

async def publish_with_trace(event: dict) -> None:
    carrier = {}
    inject(carrier)  # inject trace context into carrier dict
    event["_trace"] = carrier
    await broker.publish(event)

async def consume_with_trace(event: dict) -> None:
    ctx = extract(event.get("_trace", {}))
    with tracer.start_as_current_span("process_order_completed", context=ctx):
        await handle_event(event)

Implement end-to-end event tracing through our observability and monitoring practice — including consumer lag dashboards and DLQ alerting.

Production Case Study: Order Processing Migration

A logistics client migrated from synchronous order processing (5 sequential API calls, 2.8s P99 latency) to event-driven architecture with SNS fan-out and SQS consumers.

Before: Order API → Inventory API → Billing API → Shipping API → Notification API (serial chain)

After: Order API publishes order.completed → 4 independent SQS consumers process in parallel

Results:

• P99 order confirmation latency: 2.8s → 180ms (user sees confirmation immediately)
• End-to-end processing (all downstream complete): 4.2s → 1.1s (parallel consumers)
• One consumer failure no longer blocks the entire chain
• Added outbox pattern after a 4-hour incident where 847 orders were created but never billed

Lesson learned: Implement the outbox pattern on day one, not after the first dual-write failure.

Related implementation guides:

• Event Driven Ai Pipelines Kafka Sqs
• Evaluation Driven Development Ai

Primary references: official documentation, official documentation, official documentation, official documentation.

Frequently Asked Questions

What is event-driven architecture?

Event-driven architecture (EDA) is a software design pattern where services communicate by producing and consuming events — records of state changes — rather than making direct synchronous API calls to each other.

When should I use event-driven architecture instead of REST?

Use EDA when multiple services need to react to the same action independently, when you need to absorb traffic spikes, or when eventual consistency is acceptable. Use REST when you need immediate synchronous responses or strong consistency.

What is the difference between event-driven architecture and message queues?

Message queues are the transport mechanism. Event-driven architecture is the design pattern that uses message queues (or logs) to decouple services. EDA describes the architecture; SQS, Kafka, and RabbitMQ are the infrastructure.

What is the outbox pattern and why do I need it?

The outbox pattern writes events to a database table in the same transaction as business data, then a relay process publishes them. Without it, dual writes (database + message broker) can fail independently, causing data inconsistency.

How do I handle duplicate events in consumers?

Implement idempotency keys — store processed event IDs in Redis or a database table and skip events that have already been handled. See our idempotency guide.

Should I use Kafka or SQS?

SQS for AWS-native simplicity and moderate throughput. Kafka for high-throughput stream processing, event replay, and multiple independent consumer groups. Most teams should start with SQS and migrate to Kafka when replay or throughput requirements demand it.

What is CQRS and do I need it?

CQRS separates write models (commands) from read models (queries). You need it when read and write patterns diverge significantly — e.g., writes to PostgreSQL, reads from Elasticsearch. Most applications do not need CQRS initially.

How do I debug event-driven systems?

Use distributed tracing (OpenTelemetry) propagated through event payloads, monitor consumer lag and DLQ depth, and maintain an event catalog documenting every event type, schema, and owning team.

Conclusion

Event-driven architecture delivers real decoupling and scale — but only when implemented with the patterns that prevent its characteristic failure modes:

• Outbox pattern for reliable publishing
• Idempotent consumers for at-least-once delivery
• Dead letter queues for poison messages
• Observability for consumer lag and processing latency
• Start simple — pub/sub before event sourcing

At HinterBuild:

• Backend & API Engineering — outbox implementation, consumer design, API migration
• Data Pipelines & Integrations — event-driven data flows, Kafka and SQS architecture
• Observability & Monitoring — consumer lag, DLQ alerting, distributed tracing
• Cloud Infrastructure & DevOps — managed messaging infrastructure on AWS

Schedule a consultation for event architecture review.