Idempotency in Distributed Systems: Implementation Guide

Table of Contents:

• Why Idempotency Is Non-Negotiable
• Idempotency Keys: The Standard Pattern
• Implementation Patterns by Storage Layer
• HTTP Method Semantics and Idempotency
• Idempotency in Event Consumers
• Payment and Financial Operations
• Testing Idempotent Systems
• Frequently Asked Questions

Why Idempotency Is Non-Negotiable

Short answer: Idempotency guarantees that executing the same operation multiple times produces the same result as executing it once — essential because networks fail, clients retry, and message brokers redeliver.

In distributed systems, duplicate requests are not edge cases — they are the default. A mobile client retries on timeout. A load balancer sends the same request to two servers. An SQS message gets delivered twice because the consumer crashed before acknowledging. Without idempotency, every duplicate becomes a double charge, a duplicate order, or a corrupted state.

We implement idempotency in distributed systems on every backend API engineering engagement. This guide covers the patterns that prevent duplicate side effects — from idempotency keys to dedup tables to Redis-backed atomic checks.

Key Takeaways:

• Every write endpoint that has side effects needs idempotency protection
• Idempotency keys (client-generated UUIDs) are the industry standard
• Store the key + response together — replay the cached response on duplicate
• Event consumers need the same idempotency guarantees as HTTP endpoints
• TTL your idempotency records — but long enough to cover retry windows (24–72 hours)

Idempotency Keys: The Standard Pattern

An idempotency key is a unique identifier the client generates and sends with each request. The server stores the key and the response — if the same key arrives again, the server returns the cached response without re-executing the operation.

How It Works

Client                          Server
  │                               │
  │── POST /payments ────────────►│  Idempotency-Key: abc-123
  │                               │  1. Check: key abc-123 exists?
  │                               │  2. No → process payment, store result
  │◄── 201 {payment_id: "p_1"} ─│  3. Store: abc-123 → {payment_id: "p_1"}
  │                               │
  │── POST /payments (retry) ────►│  Idempotency-Key: abc-123
  │                               │  1. Check: key abc-123 exists?
  │                               │  2. Yes → return cached response
  │◄── 201 {payment_id: "p_1"} ─│  (same response, no double charge)

Client Implementation

import uuid
import httpx

async def create_payment(amount_cents: int, customer_id: str) -> dict:
    idempotency_key = str(uuid.uuid4())

    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.example.com/v1/payments",
            json={"amount_cents": amount_cents, "customer_id": customer_id},
            headers={"Idempotency-Key": idempotency_key},
            timeout=30.0,
        )
        response.raise_for_status()
        return response.json()

Client rules:

• Generate a new UUID for each distinct operation
• Reuse the same key when retrying the same operation
• Store the key locally until the response is confirmed (mobile apps: persist to disk)

Server Implementation (PostgreSQL)

from fastapi import FastAPI, Header, HTTPException, Depends
from sqlalchemy.ext.asyncio import AsyncSession
import json

@app.post("/v1/payments")
async def create_payment(
    payment_data: PaymentCreate,
    idempotency_key: str = Header(..., alias="Idempotency-Key"),
    db: AsyncSession = Depends(get_db),
):
    existing = await db.execute(
        select(IdempotencyRecord).where(
            IdempotencyRecord.key == idempotency_key,
            IdempotencyRecord.endpoint == "POST /v1/payments",
        )
    )
    record = existing.scalar_one_or_none()

    if record:
        if record.status == "completed":
            return json.loads(record.response_body)
        elif record.status == "processing":
            raise HTTPException(status_code=409, detail="Request in progress")
        elif record.status == "failed":
            # Allow retry of failed operations with same key
            pass

    # Acquire lock — prevent concurrent duplicate processing
    try:
        lock_record = IdempotencyRecord(
            key=idempotency_key,
            endpoint="POST /v1/payments",
            status="processing",
            request_body=payment_data.json(),
        )
        db.add(lock_record)
        await db.commit()
    except IntegrityError:
        await db.rollback()
        raise HTTPException(status_code=409, detail="Request in progress")

    try:
        payment = await process_payment(payment_data)
        response = {"payment_id": payment.id, "status": payment.status}

        lock_record.status = "completed"
        lock_record.response_body = json.dumps(response)
        lock_record.response_status = 201
        await db.commit()
        return response

    except Exception as e:
        lock_record.status = "failed"
        lock_record.error_message = str(e)
        await db.commit()
        raise

Idempotency Record Schema

CREATE TABLE idempotency_records (
    key             VARCHAR(255) NOT NULL,
    endpoint        VARCHAR(255) NOT NULL,
    status          VARCHAR(20) NOT NULL DEFAULT 'processing',
    request_body    JSONB,
    response_body   JSONB,
    response_status INT,
    error_message   TEXT,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at      TIMESTAMPTZ NOT NULL DEFAULT NOW() + INTERVAL '72 hours',
    PRIMARY KEY (key, endpoint)
);

CREATE INDEX idx_idempotency_expires ON idempotency_records (expires_at);

The composite primary key (key, endpoint) allows the same idempotency key to be used across different endpoints — matching Stripe's behavior.

Our backend API engineering team ships idempotency middleware as standard in every payment and order API.

Implementation Patterns by Storage Layer

Pattern 1: Database Dedup Table (Strongest Guarantee)

Best for: Financial operations, order creation, any operation where correctness is paramount.

Pros: ACID guarantees, survives restarts, auditable Cons: Adds latency (~2–5ms per check), requires cleanup job for expired records

Already shown above in the PostgreSQL implementation. Add a nightly cleanup job:

async def cleanup_expired_idempotency_records():
    await db.execute(
        "DELETE FROM idempotency_records WHERE expires_at < NOW()"
    )

Pattern 2: Redis Atomic Check-and-Set (Fastest)

Best for: High-throughput APIs, event consumer dedup, non-financial operations.

import redis.asyncio as redis

redis_client = redis.from_url("redis://cache.internal:6379")

async def process_idempotent(
    idempotency_key: str,
    handler: callable,
    ttl_seconds: int = 86400,
) -> dict:
    lock_key = f"idempotency:{idempotency_key}"

    # Atomic SET NX — only succeeds if key does not exist
    was_set = await redis_client.set(lock_key, "processing", nx=True, ex=ttl_seconds)

    if not was_set:
        cached = await redis_client.get(f"result:{idempotency_key}")
        if cached:
            return json.loads(cached)
        raise ConflictError("Request in progress")

    try:
        result = await handler()
        await redis_client.set(
            f"result:{idempotency_key}",
            json.dumps(result),
            ex=ttl_seconds,
        )
        await redis_client.set(lock_key, "completed", ex=ttl_seconds)
        return result
    except Exception:
        await redis_client.delete(lock_key)
        raise

Pros: Sub-millisecond check, automatic TTL expiry Cons: Not durable (Redis restart loses state), no audit trail

Use Redis for rate limiting and idempotency together — same Redis cluster, different key prefixes.

Pattern 3: Natural Idempotency (Database Constraints)

Best for: Operations where the business key itself prevents duplicates.

async def create_subscription(customer_id: str, plan_id: str) -> Subscription:
    try:
        sub = Subscription(customer_id=customer_id, plan_id=plan_id)
        db.add(sub)
        await db.commit()
        return sub
    except IntegrityError:
        # UNIQUE constraint on (customer_id, plan_id) prevents duplicate
        await db.rollback()
        return await db.execute(
            select(Subscription).where(
                Subscription.customer_id == customer_id,
                Subscription.plan_id == plan_id,
            )
        ).scalar_one()

Pros: Zero additional infrastructure, enforced at database level Cons: Only works when a natural unique key exists, error handling is coarse

Pattern Comparison

Our recommended production pattern: Redis for the fast path (check-and-set), PostgreSQL for the durable record (async write after processing). If Redis misses (restart), the database catches the duplicate.

HTTP Method Semantics and Idempotency

HTTP methods have defined idempotency semantics — but the real world violates them constantly.

POST is the danger zone. Every POST that creates a resource, charges money, or triggers a workflow needs explicit idempotency protection. GET, PUT, and DELETE are idempotent by HTTP specification — but verify your implementation does not introduce side effects (e.g., a GET that also increments a view counter is not idempotent).

PUT vs POST Idempotency

# PUT is naturally idempotent — same URL, same body, same result
@app.put("/v1/customers/{customer_id}")
async def update_customer(customer_id: str, data: CustomerUpdate):
    customer = await db.get(Customer, customer_id)
    if not customer:
        raise HTTPException(status_code=404)
    customer.name = data.name
    customer.email = data.email
    await db.commit()
    return customer  # calling twice produces same state

# POST is NOT idempotent — needs explicit protection
@app.post("/v1/orders")
async def create_order(
    data: OrderCreate,
    idempotency_key: str = Header(..., alias="Idempotency-Key"),
):
    # ... idempotency check required ...

Idempotency in Event Consumers

Event-driven systems deliver messages at-least-once by default. Every consumer is a duplicate request waiting to happen. See our event-driven architecture guide for the full messaging context.

Event Idempotency Pattern

async def handle_order_completed(event: dict) -> None:
    event_id = event["event_id"]  # publisher-assigned unique ID

    # Dedup check
    if await is_event_processed(event_id):
        logger.info(f"Skipping duplicate event: {event_id}")
        return

    async with db.begin():
        # Process within transaction
        await reserve_inventory(event["order_id"])
        await update_order_status(event["order_id"], "processing")

        # Mark as processed in same transaction
        await db.execute(
            """
            INSERT INTO processed_events (event_id, event_type, processed_at)
            VALUES ($1, $2, NOW())
            ON CONFLICT (event_id) DO NOTHING
            """,
            event_id, event["event_type"],
        )

Key design decisions:

1. Use the publisher's event ID, not a hash of the payload (payloads can be re-serialized differently)
2. Mark processed in the same transaction as the business logic — atomic commit or rollback
3. ON CONFLICT DO NOTHING handles the race where two consumer instances process the same event simultaneously

Idempotency for Sagas and Compensating Actions

In saga patterns, compensating actions (refunds, rollbacks) must also be idempotent:

async def compensate_payment(saga_id: str, payment_id: str) -> None:
    idempotency_key = f"compensate:{saga_id}:{payment_id}"

    if await is_processed(idempotency_key):
        return

    await refund_payment(payment_id)
    await mark_processed(idempotency_key)

Build idempotent event consumers as part of our data pipelines and integrations practice.

Payment and Financial Operations

Financial operations demand the strongest idempotency guarantees. A double charge is not a minor bug — it is a regulatory incident.

Stripe-Style Idempotency

Stripe's API accepts an Idempotency-Key header on all POST requests. We implement the same pattern:

class IdempotencyMiddleware:
    """ASGI middleware for automatic idempotency on POST/PUT/PATCH."""

    IDEMPOTENT_METHODS = {"POST", "PUT", "PATCH"}
    TTL_HOURS = 72

    async def __call__(self, scope, receive, send):
        if scope["method"] not in self.IDEMPOTENT_METHODS:
            return await self.app(scope, receive, send)

        headers = dict(scope.get("headers", []))
        key = headers.get(b"idempotency-key", b"").decode()

        if not key:
            # Require idempotency key for financial endpoints
            if scope["path"].startswith("/v1/payments"):
                return await self.send_error(send, 400, "Idempotency-Key required")
            return await self.app(scope, receive, send)

        cached = await self.store.get(key)
        if cached:
            return await self.send_cached_response(send, cached)

        # Process and cache
        response = await self.app(scope, receive, send)
        await self.store.set(key, response, ttl=self.TTL_HOURS * 3600)

Idempotency Key Lifecycle

72-hour TTL covers client retry windows, network partition recovery, and mobile offline scenarios. Stripe uses 24 hours; we recommend 72 for systems with mobile clients that may retry after extended offline periods.

Monitor idempotency hit rates with observability and monitoring — a sudden spike in cache hits may indicate a client bug sending duplicate requests.

Testing Idempotent Systems

Idempotency bugs hide until production traffic arrives. Test them explicitly.

Test Cases

import pytest
import uuid

@pytest.mark.asyncio
async def test_duplicate_request_returns_same_response(client):
    key = str(uuid.uuid4())
    payload = {"amount_cents": 5000, "customer_id": "cust_123"}
    headers = {"Idempotency-Key": key}

    response1 = await client.post("/v1/payments", json=payload, headers=headers)
    assert response1.status_code == 201
    payment_id_1 = response1.json()["payment_id"]

    response2 = await client.post("/v1/payments", json=payload, headers=headers)
    assert response2.status_code == 201
    payment_id_2 = response2.json()["payment_id"]

    assert payment_id_1 == payment_id_2  # same payment, not two

@pytest.mark.asyncio
async def test_concurrent_duplicate_requests(client):
    key = str(uuid.uuid4())
    payload = {"amount_cents": 5000, "customer_id": "cust_123"}
    headers = {"Idempotency-Key": key}

    # Fire 10 concurrent requests with same key
    responses = await asyncio.gather(*[
        client.post("/v1/payments", json=payload, headers=headers)
        for _ in range(10)
    ])

    payment_ids = {r.json()["payment_id"] for r in responses if r.status_code == 201}
    assert len(payment_ids) == 1  # exactly one payment created

@pytest.mark.asyncio
async def test_different_keys_create_different_resources(client):
    payload = {"amount_cents": 5000, "customer_id": "cust_123"}

    response1 = await client.post(
        "/v1/payments", json=payload,
        headers={"Idempotency-Key": str(uuid.uuid4())},
    )
    response2 = await client.post(
        "/v1/payments", json=payload,
        headers={"Idempotency-Key": str(uuid.uuid4())},
    )

    assert response1.json()["payment_id"] != response2.json()["payment_id"]

Chaos Testing

In staging, inject duplicate requests at the load balancer level and verify no duplicate side effects. Tools like Toxiproxy can simulate network retries that trigger client-side duplicate sends.

Related implementation guides:

• Saga Pattern Distributed Transactions

Primary references: official documentation, official documentation, official documentation, official documentation.

Operating Idempotency in Distributed Systems as a System

The implementation is only one part of Idempotency in Distributed Systems. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship Idempotency in Distributed Systems behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our Idempotency in Distributed Systems engineering support.

Frequently Asked Questions

What is idempotency in distributed systems?

Idempotency means an operation produces the same result whether executed once or multiple times. In distributed systems, it prevents duplicate charges, duplicate orders, and corrupted state when networks fail and clients or message brokers retry.

What is an idempotency key?

An idempotency key is a unique identifier (typically a UUID) that the client sends with a request. The server stores the key and response — duplicate requests with the same key return the cached response without re-executing the operation.

Which HTTP methods need idempotency keys?

POST always needs idempotency keys for operations with side effects. PATCH needs them when the patch triggers side effects. GET, PUT, and DELETE are idempotent by HTTP specification (verify your implementation).

How long should I store idempotency keys?

24–72 hours for most APIs. Stripe uses 24 hours. Systems with mobile clients that may retry after extended offline periods should use 72 hours. Run a cleanup job to delete expired records.

Should I use Redis or PostgreSQL for idempotency?

PostgreSQL for financial operations (ACID guarantees, audit trail). Redis for high-throughput non-financial APIs (sub-millisecond checks). Hybrid (Redis fast path + PostgreSQL durable record) for production systems that need both speed and durability.

How does idempotency relate to exactly-once delivery?

True exactly-once delivery is impossible in distributed systems. Idempotency achieves effectively-once processing — messages may be delivered multiple times, but side effects happen only once.

Do event consumers need idempotency?

Yes. Message brokers (SQS, Kafka, RabbitMQ) deliver at-least-once by default. Every event consumer must check whether an event has already been processed before executing business logic.

What happens when an idempotency key expires?

The server treats the request as new and executes the operation again. Design TTL to exceed maximum client retry windows. For financial operations, consider indefinite storage with periodic archival.

Conclusion

Idempotency in distributed systems is not optional — it is the foundation that makes retries, event redelivery, and network failures safe.

The patterns that prevent duplicate side effects:

• Idempotency keys on every POST with side effects
• Store key + response together; replay cached response on duplicate
• Atomic check-and-set in Redis or database for concurrent duplicate protection
• Same-transaction dedup in event consumers
• Test with concurrent duplicates — not just sequential retries

At HinterBuild:

• Backend & API Engineering — idempotency middleware, payment APIs, order systems
• Data Pipelines & Integrations — idempotent event consumers, saga patterns
• Observability & Monitoring — idempotency hit rate tracking, duplicate detection alerts
• Cloud Infrastructure & DevOps — Redis cluster setup for dedup and caching

Schedule a consultation for distributed systems architecture review.