HinterBuild logoHinterBuild
Back to blog
Backend Systems · 10 min read

Webhook Design for Reliability at Scale: Production Patterns

Webhook Design for Reliability at Scale guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable.

Muhammad Abdul Sami, author

Muhammad Abdul Sami

· Updated · 10 min read

  • APIs
  • Architecture
  • Performance
  • Testing

Table of Contents:

What Makes Webhook Reliability Hard

Short answer: Webhook reliability fails when receivers are slow, down, or reject valid requests — solved with exponential backoff retries, idempotency keys, signature verification, circuit breakers, and dead-letter queues.

If you searched "webhook design best practices", you're replacing manual polling with event-driven architecture, integrating with third-party APIs (Stripe, GitHub), or building webhook infrastructure for your backend system.

Key Takeaways:

  • Webhooks fail due to network timeouts, receiver downtime, rate limits, and bugs
  • At-least-once delivery requires retries + idempotency (receivers must handle duplicates)
  • Exponential backoff prevents retry storms (1s → 2s → 4s → 8s → 16s → 32s → fail)
  • HMAC signature verification prevents replay attacks and spoofed events
  • Circuit breakers automatically disable failing receivers (avoid wasting resources)
  • Dead-letter queues (DLQ) capture permanently failed deliveries for manual investigation

This guide covers webhook design for reliability at scale with production code (Python, Go), retry patterns, security implementation, and the observability framework we deploy at HinterBuild for event-driven architectures.

Webhook Architecture Fundamentals

Webhook Lifecycle

┌──────────────┐       ┌──────────────┐       ┌──────────────┐
│   Event      │──────▶│  Webhook     │──────▶│   Queue      │
│   Source     │       │  Producer    │       │  (Redis/SQS) │
└──────────────┘       └──────────────┘       └──────┬───────┘
                                                      │
                                                      ▼
                                              ┌───────────────┐
                                              │ Worker Pool   │
                                              │ (Async send)  │
                                              └──────┬────────┘
                                                     │
                     ┌───────────────────────────────┼───────────────────┐
                     ▼                               ▼                   ▼
              ┌─────────────┐              ┌─────────────┐      ┌─────────────┐
              │  Receiver A │              │  Receiver B │      │  Receiver C │
              │  (success)  │              │  (timeout)  │      │  (500 error)│
              └─────────────┘              └──────┬──────┘      └──────┬──────┘
                                                  │                    │
                                                  ▼                    ▼
                                          ┌────────────────┐   ┌────────────────┐
                                          │ Retry Queue    │   │ Retry Queue    │
                                          │ (exp backoff)  │   │ (exp backoff)  │
                                          └────────────────┘   └────────────────┘

Component Responsibilities

ComponentResponsibilityScaling
Event SourceTrigger events (order created, payment succeeded)Horizontal (app servers)
ProducerValidate event, enqueue for deliveryStateless (scales with traffic)
QueueDurably store pending deliveriesManaged (Redis Streams, SQS, Kafka)
Worker PoolFetch from queue, HTTP POST to receiver, handle retriesHorizontal (add workers for throughput)
DLQStore permanently failed deliveriesAppend-only (investigate later)

For background job patterns, see our background processing guide.

Retry Logic and Exponential Backoff

Never retry immediately. Exponential backoff prevents overwhelming failing receivers.

Exponential Backoff Algorithm

python
import time
import random
import requests
from typing import Optional

def send_webhook_with_retry(
    url: str,
    payload: dict,
    signature: str,
    max_attempts: int = 6,
    base_delay: int = 1
) -> bool:
    """
    Send webhook with exponential backoff retry.
    
    Retry schedule:
    - Attempt 1: immediate
    - Attempt 2: 1s delay
    - Attempt 3: 2s delay
    - Attempt 4: 4s delay
    - Attempt 5: 8s delay
    - Attempt 6: 16s delay
    Total time: ~31 seconds
    """
    
    for attempt in range(1, max_attempts + 1):
        try:
            response = requests.post(
                url,
                json=payload,
                headers={
                    'Content-Type': 'application/json',
                    'X-Webhook-Signature': signature,
                    'X-Webhook-Delivery-ID': payload['delivery_id'],
                    'X-Webhook-Attempt': str(attempt)
                },
                timeout=10  # 10 second timeout
            )
            
            # Success: 2xx status code
            if 200 <= response.status_code < 300:
                return True
            
            # Permanent failure: 4xx (except 429 rate limit)
            if 400 <= response.status_code < 500 and response.status_code != 429:
                print(f"Permanent failure {response.status_code}, not retrying")
                return False
            
            # Retry: 5xx or 429
            print(f"Attempt {attempt} failed: HTTP {response.status_code}")
            
        except (requests.Timeout, requests.ConnectionError) as e:
            print(f"Attempt {attempt} failed: {type(e).__name__}")
        
        # Last attempt — don't wait
        if attempt == max_attempts:
            return False
        
        # Exponential backoff with jitter
        delay = base_delay * (2 ** (attempt - 1))
        jitter = random.uniform(0, 0.3 * delay)  # ±30% jitter
        time.sleep(delay + jitter)
    
    return False

Why jitter? Prevents thundering herd when many webhooks fail simultaneously (e.g., receiver restarts). 30% jitter spreads retries over time.

Retry Decision Matrix

HTTP StatusActionReason
200-299✅ SuccessDelivery confirmed
408 Timeout🔄 RetryTemporary network issue
429 Rate Limit🔄 Retry (respect Retry-After)Receiver overloaded
500-599🔄 RetryReceiver error, may recover
400, 401, 403, 404, 410❌ Fail (don't retry)Invalid request or unauthorized
Network timeout🔄 RetryConnection failure
DNS failure❌ Fail (DLQ)Invalid URL

Asynchronous Retry with Celery

python
# tasks/webhooks.py — Celery task with automatic retries
from celery import shared_task
import requests

@shared_task(
    bind=True,
    max_retries=6,
    default_retry_delay=1,  # Base delay in seconds
    autoretry_for=(requests.Timeout, requests.ConnectionError),
    retry_backoff=True,  # Exponential backoff
    retry_backoff_max=32,  # Max delay 32 seconds
    retry_jitter=True
)
def send_webhook(self, url: str, payload: dict, signature: str):
    """Celery task with built-in exponential backoff"""
    response = requests.post(
        url,
        json=payload,
        headers={
            'X-Webhook-Signature': signature,
            'X-Webhook-Delivery-ID': payload['delivery_id']
        },
        timeout=10
    )
    
    # Raise exception to trigger retry
    if response.status_code >= 500 or response.status_code == 429:
        raise self.retry(countdown=2 ** self.request.retries)
    
    # Permanent failure
    if 400 <= response.status_code < 500:
        raise ValueError(f"Permanent failure: HTTP {response.status_code}")
    
    return response.status_code

# Usage
send_webhook.delay(
    url='https://example.com/webhooks',
    payload={'event': 'order.created', 'order_id': 12345},
    signature='hmac_sha256_here'
)

For production queue patterns, see background job processing.

Idempotency and Deduplication

At-least-once delivery guarantees duplicates. Receivers must handle duplicate events safely.

Idempotency Keys

python
# webhooks/producer.py — Generate delivery ID for deduplication
import uuid
import hashlib

def create_webhook_delivery(event_type: str, payload: dict) -> dict:
    """Create webhook delivery with idempotency key"""
    
    # Unique delivery ID (UUID v4)
    delivery_id = str(uuid.uuid4())
    
    # Optional: Content-based idempotency key (for exact duplicate detection)
    content_hash = hashlib.sha256(
        f"{event_type}:{payload['resource_id']}".encode()
    ).hexdigest()[:16]
    
    return {
        'delivery_id': delivery_id,  # Unique per delivery attempt
        'idempotency_key': content_hash,  # Same for identical events
        'event_type': event_type,
        'timestamp': datetime.utcnow().isoformat(),
        'payload': payload
    }

# Example webhook
webhook = create_webhook_delivery(
    event_type='order.created',
    payload={'order_id': 12345, 'total': 99.99}
)
# {
#   'delivery_id': 'a3b5c7d9-1234-5678-9abc-def012345678',
#   'idempotency_key': 'e4f1a2b3c4d5e6f7',
#   'event_type': 'order.created',
#   'timestamp': '2026-09-11T14:23:45Z',
#   'payload': {'order_id': 12345, 'total': 99.99}
# }

Receiver Idempotency Check

python
# receiver/webhooks.py — Idempotent webhook handler
from fastapi import FastAPI, Header, HTTPException
import redis

app = FastAPI()
redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)

@app.post("/webhooks/orders")
async def receive_order_webhook(
    payload: dict,
    x_webhook_delivery_id: str = Header(...),
    x_webhook_signature: str = Header(...)
):
    """Idempotent webhook receiver"""
    
    # 1. Verify signature (see security section)
    if not verify_signature(payload, x_webhook_signature):
        raise HTTPException(status_code=401, detail="Invalid signature")
    
    # 2. Check if already processed (idempotency)
    cache_key = f"webhook:processed:{x_webhook_delivery_id}"
    if redis_client.exists(cache_key):
        print(f"Duplicate delivery {x_webhook_delivery_id}, skipping")
        return {"status": "ok", "message": "Already processed"}
    
    # 3. Process event (database transaction)
    try:
        # Example: Create order record
        order_id = payload['payload']['order_id']
        total = payload['payload']['total']
        
        # Database insert (must be idempotent if retry after partial failure)
        # Use UPSERT or INSERT ... ON CONFLICT DO NOTHING
        await db.execute("""
            INSERT INTO orders (id, total, created_at)
            VALUES ($1, $2, NOW())
            ON CONFLICT (id) DO NOTHING
        """, order_id, total)
        
        # 4. Mark as processed (30-day TTL prevents unbounded growth)
        redis_client.setex(cache_key, 30 * 86400, "1")
        
        return {"status": "ok", "order_id": order_id}
        
    except Exception as e:
        # Return 500 to trigger retry
        raise HTTPException(status_code=500, detail=str(e))

Critical: Always check delivery_id before processing. Store processed IDs for 30 days (matches retry window).

For idempotency in distributed systems, see our idempotency guide.

Webhook Security: Signature Verification

Never trust webhook payloads without signature verification. Prevents replay attacks and spoofing.

HMAC Signature Generation (Sender)

python
# webhooks/security.py — HMAC SHA-256 signature
import hmac
import hashlib
import json

def generate_signature(payload: dict, secret: str) -> str:
    """Generate HMAC-SHA256 signature for webhook payload"""
    
    # Canonical JSON representation (consistent ordering)
    canonical_payload = json.dumps(payload, sort_keys=True, separators=(',', ':'))
    
    # HMAC-SHA256
    signature = hmac.new(
        secret.encode('utf-8'),
        canonical_payload.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()
    
    return signature

# Example
payload = {'event': 'order.created', 'order_id': 12345}
secret = 'whsec_a1b2c3d4e5f6g7h8i9j0'  # Unique per receiver
signature = generate_signature(payload, secret)
# 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

Signature Verification (Receiver)

python
# receiver/verify.py — Verify webhook signature
import hmac
import hashlib
import json
from fastapi import HTTPException

def verify_webhook_signature(
    payload: dict,
    received_signature: str,
    secret: str,
    tolerance_seconds: int = 300  # 5 minutes
) -> bool:
    """Verify HMAC signature and timestamp freshness"""
    
    # Verify timestamp (prevent replay attacks)
    payload_timestamp = datetime.fromisoformat(payload['timestamp'])
    age_seconds = (datetime.utcnow() - payload_timestamp).total_seconds()
    
    if age_seconds > tolerance_seconds:
        raise HTTPException(status_code=401, detail="Webhook too old")
    
    # Recompute signature
    canonical_payload = json.dumps(payload, sort_keys=True, separators=(',', ':'))
    expected_signature = hmac.new(
        secret.encode('utf-8'),
        canonical_payload.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()
    
    # Constant-time comparison (prevents timing attacks)
    return hmac.compare_digest(expected_signature, received_signature)

# Usage in FastAPI endpoint
@app.post("/webhooks")
async def receive_webhook(
    payload: dict,
    x_webhook_signature: str = Header(...)
):
    secret = get_webhook_secret(payload['sender_id'])  # From database
    
    if not verify_webhook_signature(payload, x_webhook_signature, secret):
        raise HTTPException(status_code=401, detail="Invalid signature")
    
    # Process webhook...

Signature Header Formats

Different providers use different header formats:

ProviderHeader FormatAlgorithm
StripeStripe-Signature: t=timestamp,v1=signatureHMAC-SHA256
GitHubX-Hub-Signature-256: sha256=signatureHMAC-SHA256
TwilioX-Twilio-Signature: signatureHMAC-SHA1 (legacy)
HinterBuildX-Webhook-Signature: signatureHMAC-SHA256

Always use HMAC-SHA256 (SHA-1 is deprecated).

Circuit Breakers for Failing Receivers

Don't waste resources retrying permanently broken receivers. Circuit breakers automatically disable failing endpoints.

Circuit Breaker State Machine

          ┌─────────────┐
          │   CLOSED    │──── Success
          │ (normal)    │◀────┘
          └──────┬──────┘
                 │ Failure threshold (5 failures)
                 ▼
          ┌─────────────┐
          │    OPEN     │──── All requests fail fast
          │  (broken)   │
          └──────┬──────┘
                 │ After timeout (30s)
                 ▼
          ┌─────────────┐
          │ HALF-OPEN   │──── Test with 1 request
          │  (testing)  │
          └──────┬──────┘
                 │
      ┌──────────┴──────────┐
      │                     │
   Success               Failure
      │                     │
      ▼                     ▼
┌─────────────┐      ┌─────────────┐
│   CLOSED    │      │    OPEN     │
└─────────────┘      └─────────────┘

Python Circuit Breaker Implementation

python
# webhooks/circuit_breaker.py — Circuit breaker for webhook delivery
from enum import Enum
from datetime import datetime, timedelta
import threading

class CircuitState(Enum):
    CLOSED = "closed"  # Normal operation
    OPEN = "open"      # Failing, reject immediately
    HALF_OPEN = "half_open"  # Testing recovery

class CircuitBreaker:
    def __init__(
        self,
        failure_threshold: int = 5,
        timeout_seconds: int = 30,
        success_threshold: int = 2
    ):
        self.failure_threshold = failure_threshold
        self.timeout_seconds = timeout_seconds
        self.success_threshold = success_threshold
        
        self.state = CircuitState.CLOSED
        self.failures = 0
        self.successes = 0
        self.last_failure_time = None
        self.lock = threading.Lock()
    
    def call(self, func, *args, **kwargs):
        """Execute function with circuit breaker protection"""
        
        with self.lock:
            # Check if circuit should transition to HALF_OPEN
            if self.state == CircuitState.OPEN:
                if datetime.now() - self.last_failure_time > timedelta(seconds=self.timeout_seconds):
                    print("Circuit transitioning to HALF_OPEN")
                    self.state = CircuitState.HALF_OPEN
                    self.successes = 0
                else:
                    raise CircuitBreakerOpenError("Circuit breaker is OPEN")
        
        # Attempt request
        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise e
    
    def _on_success(self):
        """Handle successful request"""
        with self.lock:
            if self.state == CircuitState.HALF_OPEN:
                self.successes += 1
                if self.successes >= self.success_threshold:
                    print("Circuit transitioning to CLOSED")
                    self.state = CircuitState.CLOSED
                    self.failures = 0
            elif self.state == CircuitState.CLOSED:
                self.failures = 0  # Reset failure counter
    
    def _on_failure(self):
        """Handle failed request"""
        with self.lock:
            self.failures += 1
            self.last_failure_time = datetime.now()
            
            if self.state == CircuitState.HALF_OPEN:
                print("Circuit transitioning back to OPEN")
                self.state = CircuitState.OPEN
            elif self.failures >= self.failure_threshold:
                print(f"Circuit OPEN after {self.failures} failures")
                self.state = CircuitState.OPEN

class CircuitBreakerOpenError(Exception):
    pass

# Usage
circuit_breaker = CircuitBreaker(failure_threshold=5, timeout_seconds=30)

def send_webhook(url, payload):
    try:
        return circuit_breaker.call(requests.post, url, json=payload, timeout=10)
    except CircuitBreakerOpenError:
        print("Circuit breaker OPEN, skipping delivery")
        # Move to DLQ or alert operations
        return None

Circuit Breaker with Redis (Distributed)

python
# webhooks/distributed_circuit_breaker.py — Shared state in Redis
import redis
from datetime import datetime, timedelta

class DistributedCircuitBreaker:
    def __init__(self, redis_client: redis.Redis, receiver_url: str):
        self.redis = redis_client
        self.key_prefix = f"circuit_breaker:{receiver_url}"
    
    def is_open(self) -> bool:
        """Check if circuit breaker is OPEN"""
        state = self.redis.get(f"{self.key_prefix}:state")
        if state == b"OPEN":
            # Check if timeout elapsed
            opened_at = self.redis.get(f"{self.key_prefix}:opened_at")
            if opened_at:
                age = datetime.now() - datetime.fromisoformat(opened_at.decode())
                if age > timedelta(seconds=30):
                    self.redis.set(f"{self.key_prefix}:state", "HALF_OPEN")
                    return False
            return True
        return False
    
    def record_success(self):
        """Record successful delivery"""
        state = self.redis.get(f"{self.key_prefix}:state")
        if state == b"HALF_OPEN":
            # Transition to CLOSED
            self.redis.delete(f"{self.key_prefix}:state")
            self.redis.delete(f"{self.key_prefix}:failures")
        else:
            self.redis.delete(f"{self.key_prefix}:failures")
    
    def record_failure(self):
        """Record failed delivery"""
        failures = self.redis.incr(f"{self.key_prefix}:failures")
        if failures >= 5:
            self.redis.set(f"{self.key_prefix}:state", "OPEN")
            self.redis.set(f"{self.key_prefix}:opened_at", datetime.now().isoformat())
            print(f"Circuit breaker OPEN for {self.key_prefix}")

# Usage
redis_client = redis.Redis(host='localhost', port=6379)
cb = DistributedCircuitBreaker(redis_client, 'https://example.com/webhooks')

if not cb.is_open():
    try:
        send_webhook(url, payload)
        cb.record_success()
    except Exception:
        cb.record_failure()
else:
    print("Circuit breaker OPEN, skipping delivery")

Deploy distributed circuit breakers with cloud infrastructure services.

Delivery Guarantees: At-Most-Once vs At-Least-Once

Delivery Guarantee Comparison

GuaranteeDuplicates?Lost Events?Use Case
At-most-once❌ No✅ Yes (on failure)Analytics, metrics (lossy OK)
At-least-once✅ Yes❌ NoFinancial transactions, orders
Exactly-once❌ No❌ NoTheoretical (requires distributed transactions)

Production recommendation: At-least-once with idempotency on receiver side. Exactly-once is impractical at scale.

At-Least-Once Implementation

python
# webhooks/delivery.py — At-least-once delivery with DLQ
import asyncio
from typing import Optional

async def deliver_webhook_at_least_once(
    delivery_id: str,
    url: str,
    payload: dict,
    signature: str,
    max_attempts: int = 6
) -> bool:
    """
    At-least-once delivery guarantee.
    - Retries on failure (exponential backoff)
    - Moves to DLQ after max attempts
    - Returns True if delivered or DLQ'd
    """
    
    for attempt in range(1, max_attempts + 1):
        try:
            response = await send_webhook_http(url, payload, signature, attempt)
            
            if 200 <= response.status_code < 300:
                await record_success(delivery_id, attempt)
                return True
            
            # Permanent failure (4xx)
            if 400 <= response.status_code < 500 and response.status_code != 429:
                await move_to_dlq(delivery_id, f"HTTP {response.status_code}")
                return True  # Don't retry
            
        except (asyncio.TimeoutError, ConnectionError) as e:
            await record_retry(delivery_id, attempt, str(e))
        
        if attempt < max_attempts:
            await asyncio.sleep(2 ** (attempt - 1))  # Exponential backoff
    
    # Max retries exceeded → DLQ
    await move_to_dlq(delivery_id, "Max retries exceeded")
    return True

async def move_to_dlq(delivery_id: str, reason: str):
    """Move failed delivery to dead-letter queue"""
    await db.execute("""
        INSERT INTO webhook_dead_letter_queue
        (delivery_id, url, payload, reason, failed_at)
        SELECT delivery_id, url, payload, $2, NOW()
        FROM webhook_deliveries
        WHERE delivery_id = $1
    """, delivery_id, reason)
    
    # Alert operations
    await alert_slack(f"Webhook {delivery_id} moved to DLQ: {reason}")

For distributed transaction patterns, see our SAGA pattern guide.

Monitoring and Observability

You can't fix what you don't measure. Track delivery success rate, latency, and failure reasons.

Key Metrics

python
# monitoring/webhook_metrics.py — Prometheus metrics
from prometheus_client import Counter, Histogram, Gauge

# Delivery outcomes
webhook_sent_total = Counter(
    'webhook_sent_total',
    'Total webhook deliveries attempted',
    ['receiver', 'event_type']
)

webhook_success_total = Counter(
    'webhook_success_total',
    'Successful webhook deliveries',
    ['receiver', 'event_type']
)

webhook_failure_total = Counter(
    'webhook_failure_total',
    'Failed webhook deliveries',
    ['receiver', 'event_type', 'failure_reason']
)

# Latency
webhook_delivery_duration_seconds = Histogram(
    'webhook_delivery_duration_seconds',
    'Webhook delivery latency',
    ['receiver', 'event_type'],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0]
)

# Queue depth
webhook_queue_depth = Gauge(
    'webhook_queue_depth',
    'Number of webhooks pending delivery'
)

# Circuit breaker state
webhook_circuit_breaker_state = Gauge(
    'webhook_circuit_breaker_state',
    'Circuit breaker state (0=closed, 1=open, 2=half-open)',
    ['receiver']
)

# Usage
import time

def send_webhook_with_metrics(receiver: str, event_type: str, url: str, payload: dict):
    webhook_sent_total.labels(receiver=receiver, event_type=event_type).inc()
    
    start = time.time()
    try:
        response = requests.post(url, json=payload, timeout=10)
        duration = time.time() - start
        
        webhook_delivery_duration_seconds.labels(
            receiver=receiver,
            event_type=event_type
        ).observe(duration)
        
        if 200 <= response.status_code < 300:
            webhook_success_total.labels(receiver=receiver, event_type=event_type).inc()
        else:
            webhook_failure_total.labels(
                receiver=receiver,
                event_type=event_type,
                failure_reason=f"http_{response.status_code}"
            ).inc()
    
    except requests.Timeout:
        webhook_failure_total.labels(
            receiver=receiver,
            event_type=event_type,
            failure_reason="timeout"
        ).inc()

Alerting Rules (Prometheus)

yaml
# prometheus/webhook_alerts.yml
groups:
- name: webhook_delivery
  interval: 30s
  rules:
  
  # Alert if success rate drops below 95%
  - alert: WebhookDeliveryFailureRateHigh
    expr: |
      (
        rate(webhook_failure_total[5m]) /
        rate(webhook_sent_total[5m])
      ) > 0.05
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Webhook delivery failure rate above 5%"
      description: "Receiver {{ $labels.receiver }} has {{ $value | humanizePercentage }} failure rate"
  
  # Alert if queue depth grows unbounded
  - alert: WebhookQueueBacklog
    expr: webhook_queue_depth > 10000
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "Webhook queue backlog above 10k"
      description: "{{ $value }} webhooks pending delivery"
  
  # Alert if circuit breaker opens
  - alert: WebhookCircuitBreakerOpen
    expr: webhook_circuit_breaker_state == 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Webhook circuit breaker OPEN"
      description: "Receiver {{ $labels.receiver }} circuit breaker is OPEN"

Deploy webhook observability with monitoring services.

Webhook Provider API Design

Good webhook providers let receivers manage subscriptions, view delivery logs, and test endpoints.

REST API for Webhook Management

python
# api/webhook_subscriptions.py — Webhook subscription CRUD
from fastapi import FastAPI, HTTPException, Depends
from pydantic import BaseModel, HttpUrl
from typing import List

app = FastAPI()

class WebhookSubscription(BaseModel):
    url: HttpUrl
    events: List[str]  # ['order.created', 'order.updated']
    secret: str  # Auto-generated HMAC secret
    enabled: bool = True

@app.post("/webhooks/subscriptions")
async def create_subscription(subscription: WebhookSubscription, user_id: str = Depends(get_current_user)):
    """Create webhook subscription"""
    
    # Generate webhook secret
    secret = generate_webhook_secret()
    
    subscription_id = await db.fetchval("""
        INSERT INTO webhook_subscriptions
        (user_id, url, events, secret, enabled, created_at)
        VALUES ($1, $2, $3, $4, $5, NOW())
        RETURNING id
    """, user_id, str(subscription.url), subscription.events, secret, True)
    
    return {
        "id": subscription_id,
        "secret": secret,  # Show secret ONCE on creation
        "url": subscription.url,
        "events": subscription.events
    }

@app.get("/webhooks/subscriptions")
async def list_subscriptions(user_id: str = Depends(get_current_user)):
    """List all webhook subscriptions"""
    
    rows = await db.fetch("""
        SELECT id, url, events, enabled, created_at
        FROM webhook_subscriptions
        WHERE user_id = $1
        ORDER BY created_at DESC
    """, user_id)
    
    return [dict(row) for row in rows]

@app.delete("/webhooks/subscriptions/{subscription_id}")
async def delete_subscription(subscription_id: int, user_id: str = Depends(get_current_user)):
    """Delete webhook subscription"""
    
    await db.execute("""
        DELETE FROM webhook_subscriptions
        WHERE id = $1 AND user_id = $2
    """, subscription_id, user_id)
    
    return {"status": "deleted"}

@app.post("/webhooks/subscriptions/{subscription_id}/test")
async def test_webhook(subscription_id: int, user_id: str = Depends(get_current_user)):
    """Send test webhook to validate receiver endpoint"""
    
    subscription = await db.fetchrow("""
        SELECT url, secret FROM webhook_subscriptions
        WHERE id = $1 AND user_id = $2
    """, subscription_id, user_id)
    
    if not subscription:
        raise HTTPException(status_code=404, detail="Subscription not found")
    
    # Send test event
    test_payload = {
        'event': 'test',
        'test': True,
        'timestamp': datetime.utcnow().isoformat()
    }
    
    signature = generate_signature(test_payload, subscription['secret'])
    response = requests.post(
        subscription['url'],
        json=test_payload,
        headers={'X-Webhook-Signature': signature},
        timeout=10
    )
    
    return {
        "status_code": response.status_code,
        "response_time_ms": response.elapsed.total_seconds() * 1000,
        "success": 200 <= response.status_code < 300
    }

Delivery Log API

python
@app.get("/webhooks/deliveries")
async def list_deliveries(
    subscription_id: int,
    user_id: str = Depends(get_current_user),
    limit: int = 50,
    offset: int = 0
):
    """List webhook delivery attempts with status"""
    
    rows = await db.fetch("""
        SELECT
            d.id,
            d.event_type,
            d.created_at,
            d.status,
            d.http_status_code,
            d.attempts,
            d.next_retry_at,
            d.last_error
        FROM webhook_deliveries d
        JOIN webhook_subscriptions s ON d.subscription_id = s.id
        WHERE s.id = $1 AND s.user_id = $2
        ORDER BY d.created_at DESC
        LIMIT $3 OFFSET $4
    """, subscription_id, user_id, limit, offset)
    
    return [dict(row) for row in rows]

@app.post("/webhooks/deliveries/{delivery_id}/retry")
async def retry_delivery(delivery_id: str, user_id: str = Depends(get_current_user)):
    """Manually retry failed webhook delivery"""
    
    delivery = await db.fetchrow("""
        SELECT d.* FROM webhook_deliveries d
        JOIN webhook_subscriptions s ON d.subscription_id = s.id
        WHERE d.id = $1 AND s.user_id = $2
    """, delivery_id, user_id)
    
    if not delivery:
        raise HTTPException(status_code=404)
    
    # Re-enqueue for delivery
    await enqueue_webhook_delivery(delivery)
    
    return {"status": "queued"}

For API design patterns, see our API design guide and backend engineering services.

Receiver Best Practices

Fast Response Times

python
# receiver/fast_response.py — Return 200 immediately, process async
from fastapi import FastAPI, BackgroundTasks

app = FastAPI()

@app.post("/webhooks/orders")
async def receive_webhook(payload: dict, background_tasks: BackgroundTasks):
    """
    Return 200 immediately, process in background.
    Prevents sender timeout while doing slow work.
    """
    
    # Validate signature (fast)
    if not verify_signature(payload):
        return {"error": "Invalid signature"}, 401
    
    # Check idempotency (fast Redis lookup)
    if already_processed(payload['delivery_id']):
        return {"status": "ok", "duplicate": True}
    
    # Schedule background processing
    background_tasks.add_task(process_webhook, payload)
    
    # Return 200 immediately
    return {"status": "accepted"}

async def process_webhook(payload: dict):
    """Slow processing happens after response sent"""
    # Database writes, external API calls, etc.
    await create_order(payload)
    await send_confirmation_email(payload)
    await update_analytics(payload)

Graceful Degradation

python
# receiver/graceful_degradation.py — Handle dependency failures
@app.post("/webhooks")
async def receive_webhook(payload: dict):
    """Degrade gracefully if dependencies fail"""
    
    try:
        # Critical path: Store webhook for processing
        await save_webhook_to_database(payload)
        
        # Non-critical: Update cache (ignore failures)
        try:
            await update_cache(payload)
        except Exception as e:
            log.warning(f"Cache update failed: {e}")
        
        # Non-critical: Send Slack notification (ignore failures)
        try:
            await notify_slack(payload)
        except Exception as e:
            log.warning(f"Slack notification failed: {e}")
        
        return {"status": "ok"}
    
    except DatabaseError:
        # Critical failure — return 500 to trigger retry
        return {"error": "Database unavailable"}, 503

Production Implementation Examples

Complete Go Webhook Worker

go
// worker/webhook_worker.go — Production webhook delivery worker
package main

import (
    "bytes"
    "context"
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "net/http"
    "time"
)

type WebhookDelivery struct {
    ID          string                 `json:"id"`
    URL         string                 `json:"url"`
    Payload     map[string]interface{} `json:"payload"`
    Secret      string                 `json:"secret"`
    Attempt     int                    `json:"attempt"`
    MaxAttempts int                    `json:"max_attempts"`
}

func (w *WebhookDelivery) Send(ctx context.Context) error {
    // Generate signature
    payloadBytes, _ := json.Marshal(w.Payload)
    signature := generateSignature(payloadBytes, w.Secret)
    
    // Create HTTP request
    req, err := http.NewRequestWithContext(ctx, "POST", w.URL, bytes.NewBuffer(payloadBytes))
    if err != nil {
        return err
    }
    
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("X-Webhook-Signature", signature)
    req.Header.Set("X-Webhook-Delivery-ID", w.ID)
    req.Header.Set("X-Webhook-Attempt", fmt.Sprintf("%d", w.Attempt))
    
    // Send with timeout
    client := &http.Client{Timeout: 10 * time.Second}
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    
    // Check response
    if resp.StatusCode >= 200 && resp.StatusCode < 300 {
        return nil  // Success
    }
    
    // Permanent failure
    if resp.StatusCode >= 400 && resp.StatusCode < 500 && resp.StatusCode != 429 {
        return fmt.Errorf("permanent failure: HTTP %d", resp.StatusCode)
    }
    
    // Retry
    return fmt.Errorf("temporary failure: HTTP %d", resp.StatusCode)
}

func generateSignature(payload []byte, secret string) string {
    h := hmac.New(sha256.New, []byte(secret))
    h.Write(payload)
    return hex.EncodeToString(h.Sum(nil))
}

func processWebhookQueue(ctx context.Context) {
    for {
        select {
        case <-ctx.Done():
            return
        default:
            // Fetch from queue (Redis, SQS, etc.)
            delivery, err := fetchNextDelivery(ctx)
            if err != nil {
                time.Sleep(1 * time.Second)
                continue
            }
            
            // Send webhook
            err = delivery.Send(ctx)
            if err != nil {
                // Schedule retry with exponential backoff
                delay := time.Duration(1<<delivery.Attempt) * time.Second
                scheduleRetry(delivery, delay)
            } else {
                markDelivered(delivery.ID)
            }
        }
    }
}

For production backend API patterns, see our engineering services.

Primary references: official documentation, official documentation.

Operating Webhook Design for Reliability at Scale as a System

The implementation is only one part of Webhook Design for Reliability at Scale. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.

Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.

Define the failure model

List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.

Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.

Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.

Measure the decision, not just the component

Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.

Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.

Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.

Roll out with reversible steps

Ship Webhook Design for Reliability at Scale behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.

Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.

Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.

Capacity and cost controls

Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.

Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.

Production readiness review

Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.

The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our Webhook Design for Reliability at Scale engineering support.

Frequently Asked Questions

What is the best retry strategy for webhooks?

Exponential backoff with jitter: 1s → 2s → 4s → 8s → 16s → 32s → dead-letter queue. Retry HTTP 5xx and timeouts; don't retry HTTP 4xx (except 429 rate limit). Total retry window: ~60 seconds over 6 attempts.

How do I prevent duplicate webhook processing?

Store delivery_id in Redis/database with 30-day TTL. Check before processing:

python
if redis.exists(f"webhook:processed:{delivery_id}"):
    return {"status": "duplicate"}
redis.setex(f"webhook:processed:{delivery_id}", 30*86400, "1")

Use INSERT ... ON CONFLICT DO NOTHING for database idempotency.

Should webhooks be synchronous or asynchronous?

Always asynchronous. Enqueue webhooks to a queue (Redis, SQS, Kafka) and process with worker pool. Never send webhooks inline with user requests — slow receivers block your API.

How do I secure webhooks without HTTPS?

You can't. HTTPS is mandatory for webhooks — prevents man-in-the-middle attacks. Use Let's Encrypt for free TLS certificates. Signature verification (HMAC) protects payload integrity but not confidentiality.

What should I log for webhook deliveries?

Log:

  • ✅ delivery_id, url, event_type, attempt, http_status, duration_ms, timestamp
  • ❌ Don't log full payload (may contain PII) — store hash instead

Store logs for 90 days for debugging and compliance.

How do I test webhook receivers before production?

Use webhook.site or ngrok for testing:

bash
# Expose local server to internet
ngrok http 8000
# Forwarding https://abc123.ngrok.io -> http://localhost:8000

# Use ngrok URL as webhook endpoint
curl -X POST https://abc123.ngrok.io/webhooks \
  -H "Content-Type: application/json" \
  -d '{"event": "test"}'

Conclusion

Webhook reliability at scale requires retry logic, idempotency, signature verification, circuit breakers, and dead-letter queues. The cost of unreliable webhooks is lost events, angry customers, and manual reconciliation.

Key Recommendations:

  • At-least-once delivery with exponential backoff (6 attempts over ~60s)
  • Idempotency on receiver side (store delivery_id for 30 days)
  • HMAC-SHA256 signatures with timestamp validation (prevent replay attacks)
  • Circuit breakers to disable failing receivers (5 failures → OPEN for 30s)
  • Dead-letter queues for permanently failed deliveries (manual investigation)

Build reliable webhook infrastructure with our backend API engineering services. We design event-driven architectures with durable delivery guarantees, observability, and security built-in.

Free consultation

Book a free consultation call on webhook architecture & event delivery

30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.

Book a meeting

Services

Keep reading

Related articles

AI Systems · 11 min read

Webhook Design for AI Pipelines: Reliability Patterns for

Build reliable webhook systems for AI pipelines with retry logic, idempotency, and validation. Production patterns from processing 50K+ AI webhooks daily.

Read post

Backend Systems · 10 min read

WebSockets vs SSE vs Long Polling: The Decision Guide

Learn websockets vs sse vs long polling through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post

Backend Systems · 11 min read

JWT vs Session Tokens: Which to Use

JWT vs Session Tokens guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.

Read post

Backend Systems · 12 min read

gRPC vs REST vs GraphQL: How to Choose the Right API

Learn grpc vs rest vs graphql through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.

Read post