Feature Flags in Production: Beyond On/Off
Learn feature flags in production through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Muhammad Abdul Sami
· Updated · 9 min read
- RAG
- Embeddings
- Vector Databases
- Evaluation
Table of Contents:
- Feature Flags Beyond Boolean Toggles
- Architecture Patterns
- Gradual Rollout and Canary Releases
- Targeting and Segmentation
- Kill Switches and Circuit Breakers
- A/B Testing and Experimentation
- Technical Debt Management
- Implementation Examples
- Scaling and Performance
- Observability and Metrics
- Decision Matrix
- Production Checklist
- Frequently Asked Questions
Feature Flags Beyond Boolean Toggles
Short answer: Feature flags in production enable gradual rollouts, A/B experiments, user targeting, and instant kill switches — far beyond simple on/off switches for incomplete features.
If you searched "feature flags in production", you're moving past "if enabled, show new UI" to progressive delivery and risk mitigation. At HinterBuild, our backend API engineering team deploys feature flags as infrastructure — every major change ships behind flags with gradual rollout and instant rollback.
Key Takeaways:
- Progressive delivery ships to 1% → 10% → 50% → 100% of users over days
- Kill switches disable features in <1 second without redeployment
- Targeting rules enable features for beta users, specific regions, or paid tiers
- Experimentation runs A/B tests without code changes
- Technical debt accumulates if flags aren't removed — set expiration policies
This guide covers feature flag architecture beyond boolean toggles, with production patterns for safe deployments, experimentation, and instant rollback.
Quick Use Case Matrix
| Use Case | Flag Type | Example |
|---|---|---|
| Work-in-progress feature | Release toggle | Hide UI until ready |
| Gradual rollout | Rollout toggle | 1% → 10% → 100% |
| A/B experiment | Experiment toggle | Test button color |
| Performance degradation | Kill switch | Disable expensive feature |
| Beta program | Permission toggle | Only beta users see |
| Regional compliance | Ops toggle | Disable feature in EU |
| Load shedding | Circuit breaker | Disable under high load |
| Paid feature | Entitlement toggle | Premium tier only |
Architecture Patterns
Centralized Feature Flag Service
┌─────────────────────────────────────────────┐
│ Feature Flag Service (Redis) │
│ ┌──────────────────────────────────────┐ │
│ │ new_checkout: 25% rollout │ │
│ │ dark_mode: enabled for beta users │ │
│ │ ml_recommendations: disabled (kill) │ │
│ └──────────────────────────────────────┘ │
└────────────────┬────────────────────────────┘
│
┌────────┴────────┐
▼ ▼
┌─────────┐ ┌─────────┐
│ API 1 │ │ API 2 │
│ Checks │ │ Checks │
│ flags │ │ flags │
└─────────┘ └─────────┘
Distributed Feature Flag Architecture
from launchdarkly import LDClient
from redis.asyncio import Redis
ld_client = LDClient(sdk_key="your-key")
redis = Redis.from_url("redis://localhost")
async def is_feature_enabled(user_id: str, feature: str) -> bool:
# 1. Check local cache (100µs)
cached = await redis.get(f"flag:{user_id}:{feature}")
if cached is not None:
return cached == "1"
# 2. Evaluate flag (1-5ms)
user = {"key": user_id}
enabled = ld_client.variation(feature, user, default=False)
# 3. Cache result (TTL 60s)
await redis.setex(f"flag:{user_id}:{feature}", 60, "1" if enabled else "0")
return enabled
Self-Hosted vs SaaS
| Factor | Self-Hosted (Unleash, Flagsmith) | SaaS (LaunchDarkly, Split.io) |
|---|---|---|
| Cost | Free (OSS) | $50–$1000/month |
| Latency | <1ms (Redis) | 5–50ms (API call) |
| Control | Full | Vendor-managed |
| Maintenance | Self-managed | Zero |
| Analytics | Custom | Built-in |
| Compliance | Data on-prem | Data with vendor |
Gradual Rollout and Canary Releases
Percentage-Based Rollout
# Feature flag with percentage rollout
import hashlib
def is_feature_enabled_for_user(user_id: str, feature: str, rollout_percentage: int) -> bool:
"""Consistent hash-based rollout."""
hash_input = f"{feature}:{user_id}".encode()
hash_value = int(hashlib.md5(hash_input).hexdigest(), 16)
bucket = hash_value % 100
return bucket < rollout_percentage
# Usage
if is_feature_enabled_for_user(user.id, "new_checkout", rollout_percentage=25):
return render_new_checkout()
else:
return render_old_checkout()
Rollout Schedule
# Feature rollout plan
feature: new_checkout
schedule:
- date: 2026-09-11
percentage: 1%
monitor: error rate, latency, conversion
- date: 2026-09-12
percentage: 5%
condition: error_rate < 0.5%
- date: 2026-09-13
percentage: 25%
condition: conversion_rate >= baseline
- date: 2026-09-14
percentage: 50%
- date: 2026-09-15
percentage: 100%
notify: product team
Automated Rollout with Metrics
from prometheus_client import Counter, Histogram
checkout_errors = Counter("checkout_errors_total", ["version"])
checkout_latency = Histogram("checkout_latency_seconds", ["version"])
async def gradual_rollout_automation():
"""Automatically increase rollout if metrics healthy."""
current_percentage = 1
while current_percentage < 100:
await asyncio.sleep(3600) # Wait 1 hour
# Check error rate
error_rate = get_error_rate("new_checkout")
baseline_error_rate = get_error_rate("old_checkout")
if error_rate > baseline_error_rate * 1.5:
# Errors too high — pause rollout
await alert_team(f"Rollout paused at {current_percentage}%: high error rate")
break
# Increase rollout
current_percentage = min(current_percentage * 2, 100)
await set_rollout_percentage("new_checkout", current_percentage)
await notify_team(f"Rollout increased to {current_percentage}%")
Pair with observability & monitoring for rollout health tracking.
Targeting and Segmentation
User Attribute Targeting
from typing import Dict, Any
class FeatureFlag:
def __init__(self, name: str, rules: list[Dict[str, Any]]):
self.name = name
self.rules = rules
def is_enabled(self, user: Dict[str, Any]) -> bool:
"""Evaluate targeting rules."""
for rule in self.rules:
if self._matches_rule(user, rule):
return rule["enabled"]
return False # Default: disabled
def _matches_rule(self, user: Dict[str, Any], rule: Dict[str, Any]) -> bool:
"""Check if user matches rule conditions."""
for condition in rule.get("conditions", []):
attribute = condition["attribute"]
operator = condition["operator"]
value = condition["value"]
user_value = user.get(attribute)
if operator == "equals" and user_value != value:
return False
elif operator == "in" and user_value not in value:
return False
elif operator == "greater_than" and user_value <= value:
return False
return True
# Flag configuration
dark_mode_flag = FeatureFlag(
name="dark_mode",
rules=[
{
"conditions": [
{"attribute": "tier", "operator": "in", "value": ["premium", "enterprise"]},
{"attribute": "country", "operator": "equals", "value": "US"},
],
"enabled": True,
},
{
"conditions": [
{"attribute": "beta_user", "operator": "equals", "value": True},
],
"enabled": True,
},
]
)
# Usage
user = {"id": "123", "tier": "premium", "country": "US", "beta_user": False}
if dark_mode_flag.is_enabled(user):
return render_dark_mode()
Segment-Based Targeting
# Define user segments
BETA_USERS = {"user_123", "user_456", "user_789"}
ENTERPRISE_CUSTOMERS = {"acme_corp", "globex", "initech"}
def get_user_segments(user_id: str, org_id: str) -> set[str]:
"""Compute user's segments."""
segments = {"all_users"}
if user_id in BETA_USERS:
segments.add("beta_users")
if org_id in ENTERPRISE_CUSTOMERS:
segments.add("enterprise")
# Add computed segments
user_data = get_user_data(user_id)
if user_data["signup_date"] < "2025-01-01":
segments.add("early_adopters")
return segments
# Flag evaluation
def is_feature_enabled_for_segments(feature: str, user_segments: set[str]) -> bool:
feature_segments = get_feature_segments(feature)
return bool(user_segments & feature_segments) # Intersection
Geographic Targeting
from fastapi import Request
async def get_user_country(request: Request) -> str:
"""Extract country from request."""
# CloudFlare header
country = request.headers.get("CF-IPCountry")
if country:
return country
# GeoIP lookup
ip = request.client.host
return geoip_lookup(ip)
@app.get("/features")
async def get_features(request: Request, user_id: str):
country = await get_user_country(request)
features = {
"dark_mode": is_feature_enabled(user_id, "dark_mode"),
"new_checkout": is_feature_enabled(user_id, "new_checkout"),
}
# EU-specific: disable feature for compliance
if country in ["DE", "FR", "GB"] and not is_compliant("new_checkout"):
features["new_checkout"] = False
return features
Kill Switches and Circuit Breakers
Instant Kill Switch
from redis.asyncio import Redis
redis = Redis.from_url("redis://localhost")
async def check_kill_switch(feature: str) -> bool:
"""Check if feature is killed (disabled globally)."""
killed = await redis.get(f"kill_switch:{feature}")
return killed == b"1"
async def kill_feature(feature: str, reason: str):
"""Instantly disable feature across all servers."""
await redis.set(f"kill_switch:{feature}", "1")
await redis.set(f"kill_reason:{feature}", reason)
await alert_team(f"🔴 Feature '{feature}' killed: {reason}")
# Middleware to check kill switches
@app.middleware("http")
async def kill_switch_middleware(request: Request, call_next):
# Check if route is killed
route = request.url.path
if await check_kill_switch(route):
return JSONResponse(
status_code=503,
content={"error": "Feature temporarily disabled"}
)
return await call_next(request)
Circuit Breaker Pattern
from enum import Enum
from datetime import datetime, timedelta
class CircuitState(Enum):
CLOSED = "closed" # Normal operation
OPEN = "open" # Feature disabled
HALF_OPEN = "half_open" # Testing recovery
class CircuitBreaker:
def __init__(self, feature: str, threshold: int = 5, timeout: int = 60):
self.feature = feature
self.threshold = threshold # Failures to trigger open
self.timeout = timeout # Seconds before half-open
self.failures = 0
self.state = CircuitState.CLOSED
self.last_failure = None
async def call(self, func, *args, **kwargs):
"""Execute function through circuit breaker."""
if self.state == CircuitState.OPEN:
# Check if timeout elapsed
if datetime.now() - self.last_failure > timedelta(seconds=self.timeout):
self.state = CircuitState.HALF_OPEN
self.failures = 0
else:
raise CircuitBreakerOpen(f"Circuit open for {self.feature}")
try:
result = await func(*args, **kwargs)
# Success — reset if half-open
if self.state == CircuitState.HALF_OPEN:
self.state = CircuitState.CLOSED
await notify_team(f"Circuit breaker closed for {self.feature}")
return result
except Exception as e:
self.failures += 1
self.last_failure = datetime.now()
if self.failures >= self.threshold:
self.state = CircuitState.OPEN
await kill_feature(self.feature, f"Circuit breaker opened: {e}")
raise
# Usage
ml_circuit = CircuitBreaker("ml_recommendations", threshold=5, timeout=300)
@app.get("/recommendations")
async def get_recommendations(user_id: str):
try:
return await ml_circuit.call(fetch_ml_recommendations, user_id)
except CircuitBreakerOpen:
# Fallback to simple recommendations
return fetch_simple_recommendations(user_id)
Load Shedding with Feature Flags
from prometheus_client import Gauge
current_load = Gauge("api_current_load", "Current API load")
@app.middleware("http")
async def load_shedding_middleware(request: Request, call_next):
load = current_load._value.get()
# Disable expensive features under high load
if load > 0.8: # 80% capacity
request.state.load_shedding = True
request.state.disable_features = ["ml_recommendations", "image_processing"]
return await call_next(request)
@app.get("/search")
async def search(query: str, request: Request):
results = basic_search(query)
# Conditionally add ML ranking
if not getattr(request.state, "load_shedding", False):
results = await ml_rank_results(results)
return results
Integrate with rate limiting strategies for comprehensive load control.
A/B Testing and Experimentation
Simple A/B Test
import random
def assign_variant(user_id: str, experiment: str) -> str:
"""Assign user to A/B test variant consistently."""
hash_input = f"{experiment}:{user_id}".encode()
hash_value = int(hashlib.md5(hash_input).hexdigest(), 16)
if hash_value % 2 == 0:
return "control"
else:
return "variant"
@app.get("/checkout")
async def checkout(user_id: str):
variant = assign_variant(user_id, "checkout_button_color")
if variant == "control":
button_color = "blue"
else:
button_color = "green"
# Track assignment
await track_event("experiment_assigned", {
"user_id": user_id,
"experiment": "checkout_button_color",
"variant": variant,
})
return {"button_color": button_color}
Multi-Armed Bandit
from typing import Dict
class MultiArmedBandit:
"""Epsilon-greedy bandit for A/B/C/D testing."""
def __init__(self, variants: list[str], epsilon: float = 0.1):
self.variants = variants
self.epsilon = epsilon
self.counts: Dict[str, int] = {v: 0 for v in variants}
self.values: Dict[str, float] = {v: 0.0 for v in variants}
def select_variant(self) -> str:
"""Select variant (explore or exploit)."""
if random.random() < self.epsilon:
# Explore: random variant
return random.choice(self.variants)
else:
# Exploit: best variant
return max(self.values, key=self.values.get)
def update(self, variant: str, reward: float):
"""Update variant statistics."""
self.counts[variant] += 1
n = self.counts[variant]
# Incremental average
old_value = self.values[variant]
self.values[variant] = old_value + (reward - old_value) / n
# Usage
button_color_bandit = MultiArmedBandit(["blue", "green", "red", "yellow"])
@app.get("/checkout")
async def checkout(user_id: str):
variant = button_color_bandit.select_variant()
return {"button_color": variant, "user_id": user_id}
@app.post("/checkout/complete")
async def checkout_complete(user_id: str, variant: str, purchased: bool):
"""Update bandit with conversion result."""
reward = 1.0 if purchased else 0.0
button_color_bandit.update(variant, reward)
return {"status": "ok"}
Statistical Significance
from scipy import stats
def calculate_significance(control: dict, variant: dict) -> dict:
"""Calculate A/B test statistical significance."""
# control/variant: {"conversions": int, "samples": int}
control_rate = control["conversions"] / control["samples"]
variant_rate = variant["conversions"] / variant["samples"]
# Z-test for proportions
pooled = (control["conversions"] + variant["conversions"]) / (control["samples"] + variant["samples"])
se = (pooled * (1 - pooled) * (1/control["samples"] + 1/variant["samples"])) ** 0.5
z_score = (variant_rate - control_rate) / se
p_value = 1 - stats.norm.cdf(abs(z_score))
return {
"control_rate": control_rate,
"variant_rate": variant_rate,
"lift": (variant_rate - control_rate) / control_rate,
"p_value": p_value,
"significant": p_value < 0.05,
"confidence": 1 - p_value,
}
# Example
result = calculate_significance(
control={"conversions": 120, "samples": 1000},
variant={"conversions": 150, "samples": 1000}
)
# {"lift": 0.25, "p_value": 0.02, "significant": True}
Compare with system design patterns for experimentation infrastructure.
Technical Debt Management
Flag Lifecycle
from datetime import datetime, timedelta
from enum import Enum
class FlagStatus(Enum):
ACTIVE = "active"
DEPRECATED = "deprecated"
EXPIRED = "expired"
class FeatureFlagMetadata:
def __init__(
self,
name: str,
created_by: str,
created_at: datetime,
expires_at: datetime,
jira_ticket: str,
):
self.name = name
self.created_by = created_by
self.created_at = created_at
self.expires_at = expires_at
self.jira_ticket = jira_ticket
def status(self) -> FlagStatus:
if datetime.now() > self.expires_at:
return FlagStatus.EXPIRED
elif datetime.now() > self.expires_at - timedelta(days=7):
return FlagStatus.DEPRECATED
else:
return FlagStatus.ACTIVE
# Enforce expiration policy
NEW_FEATURE_FLAGS_TTL = timedelta(days=90)
def create_feature_flag(name: str, created_by: str) -> FeatureFlagMetadata:
metadata = FeatureFlagMetadata(
name=name,
created_by=created_by,
created_at=datetime.now(),
expires_at=datetime.now() + NEW_FEATURE_FLAGS_TTL,
jira_ticket=f"TECH-{random.randint(1000, 9999)}",
)
# Alert on expiration
schedule_alert(
when=metadata.expires_at - timedelta(days=7),
message=f"Feature flag '{name}' expires in 7 days. Remove from code."
)
return metadata
Automated Flag Removal Detection
#!/bin/bash
# detect_unused_flags.sh
# List all flags in config
FLAGS=$(cat feature_flags.json | jq -r '.[] | .name')
for flag in $FLAGS; do
# Search codebase for flag usage
COUNT=$(grep -r "$flag" src/ | wc -l)
if [ $COUNT -eq 0 ]; then
echo "⚠️ Unused flag detected: $flag"
echo " Safe to remove from config"
fi
done
Flag Cleanup Checklist
## Feature Flag Cleanup Checklist **Flag:** new_checkout **Owner:** @alice **Rollout completed:** 2026-09-15 **Expiration:** 2026-09-30 - [ ] Verify 100% rollout for 7+ days - [ ] No incidents related to feature - [ ] Remove flag checks from code - [ ] Default to new code path - [ ] Remove flag from config - [ ] Delete A/B test data (if applicable) - [ ] Update documentation - [ ] Deploy cleanup changes - [ ] Archive flag in dashboard
Track with data pipelines for flag usage analytics.
Implementation Examples
Go Implementation
package main
import (
"context"
"fmt"
"hash/fnv"
"github.com/redis/go-redis/v9"
)
type FeatureFlags struct {
redis *redis.Client
}
func NewFeatureFlags(redisURL string) *FeatureFlags {
rdb := redis.NewClient(&redis.Options{
Addr: redisURL,
})
return &FeatureFlags{redis: rdb}
}
func (ff *FeatureFlags) IsEnabled(ctx context.Context, userID, feature string) (bool, error) {
// Check kill switch
killed, err := ff.redis.Get(ctx, fmt.Sprintf("kill_switch:%s", feature)).Result()
if err == nil && killed == "1" {
return false, nil
}
// Get rollout percentage
rolloutStr, err := ff.redis.Get(ctx, fmt.Sprintf("rollout:%s", feature)).Result()
if err != nil {
return false, nil // Default: disabled
}
var rollout int
fmt.Sscanf(rolloutStr, "%d", &rollout)
// Consistent hash
h := fnv.New32a()
h.Write([]byte(feature + ":" + userID))
bucket := h.Sum32() % 100
return int(bucket) < rollout, nil
}
func (ff *FeatureFlags) SetRollout(ctx context.Context, feature string, percentage int) error {
return ff.redis.Set(ctx, fmt.Sprintf("rollout:%s", feature), percentage, 0).Err()
}
func (ff *FeatureFlags) KillFeature(ctx context.Context, feature string) error {
return ff.redis.Set(ctx, fmt.Sprintf("kill_switch:%s", feature), "1", 0).Err()
}
JavaScript/TypeScript Implementation
import { createClient, RedisClientType } from 'redis';
import { createHash } from 'crypto';
export class FeatureFlags {
private redis: RedisClientType;
constructor(redisURL: string) {
this.redis = createClient({ url: redisURL });
this.redis.connect();
}
async isEnabled(userID: string, feature: string): Promise<boolean> {
// Check kill switch
const killed = await this.redis.get(`kill_switch:${feature}`);
if (killed === '1') {
return false;
}
// Get rollout percentage
const rolloutStr = await this.redis.get(`rollout:${feature}`);
if (!rolloutStr) {
return false; // Default: disabled
}
const rollout = parseInt(rolloutStr, 10);
// Consistent hash
const hash = createHash('md5').update(`${feature}:${userID}`).digest('hex');
const bucket = parseInt(hash.slice(0, 8), 16) % 100;
return bucket < rollout;
}
async setRollout(feature: string, percentage: number): Promise<void> {
await this.redis.set(`rollout:${feature}`, percentage.toString());
}
async killFeature(feature: string): Promise<void> {
await this.redis.set(`kill_switch:${feature}`, '1');
}
}
// Express middleware
export function featureFlagMiddleware(ff: FeatureFlags) {
return async (req: Request, res: Response, next: NextFunction) => {
const userID = req.user?.id || 'anonymous';
req.features = {
newCheckout: await ff.isEnabled(userID, 'new_checkout'),
darkMode: await ff.isEnabled(userID, 'dark_mode'),
};
next();
};
}
Scaling and Performance
Performance Comparison
| Approach | Latency | Scalability | Cost |
|---|---|---|---|
| In-memory cache | 10–100µs | High | Free |
| Redis | 0.5–2ms | Very high | $5–50/month |
| SaaS API | 5–50ms | Infinite | $50–1000/month |
| Config file | 0µs | N/A (requires deploy) | Free |
Caching Strategy
from functools import lru_cache
import asyncio
# Two-tier caching: memory + Redis
class CachedFeatureFlags:
def __init__(self, redis_url: str):
self.redis = Redis.from_url(redis_url)
self._cache: Dict[str, bool] = {}
self._cache_ttl = 60 # seconds
async def is_enabled(self, user_id: str, feature: str) -> bool:
cache_key = f"{user_id}:{feature}"
# Tier 1: In-memory cache (10µs)
if cache_key in self._cache:
return self._cache[cache_key]
# Tier 2: Redis (1ms)
cached = await self.redis.get(f"flag:{cache_key}")
if cached is not None:
result = cached == b"1"
self._cache[cache_key] = result
return result
# Tier 3: Evaluate (5ms)
result = await self._evaluate_flag(user_id, feature)
# Cache in Redis and memory
await self.redis.setex(f"flag:{cache_key}", self._cache_ttl, "1" if result else "0")
self._cache[cache_key] = result
return result
async def _evaluate_flag(self, user_id: str, feature: str) -> bool:
# Complex evaluation logic here
pass
Batch Flag Evaluation
async def get_all_flags_for_user(user_id: str) -> Dict[str, bool]:
"""Fetch all flags in single call (reduce latency)."""
flags = [
"new_checkout",
"dark_mode",
"ml_recommendations",
"beta_features",
]
# Evaluate all flags concurrently
results = await asyncio.gather(*[
is_feature_enabled(user_id, flag) for flag in flags
])
return dict(zip(flags, results))
@app.get("/user/flags")
async def get_user_flags(user_id: str):
# Single endpoint call from frontend
return await get_all_flags_for_user(user_id)
Observability and Metrics
Feature Flag Metrics
from prometheus_client import Counter, Histogram
flag_evaluations = Counter(
"feature_flag_evaluations_total",
"Total feature flag evaluations",
["feature", "result"]
)
flag_latency = Histogram(
"feature_flag_latency_seconds",
"Feature flag evaluation latency",
["feature"]
)
async def is_feature_enabled_instrumented(user_id: str, feature: str) -> bool:
with flag_latency.labels(feature=feature).time():
result = await is_feature_enabled(user_id, feature)
flag_evaluations.labels(feature=feature, result=str(result)).inc()
return result
Dashboard Metrics
-- Feature flag adoption rate
SELECT
feature,
COUNT(*) FILTER (WHERE enabled = true) AS enabled_count,
COUNT(*) AS total_evaluations,
ROUND(100.0 * COUNT(*) FILTER (WHERE enabled = true) / COUNT(*), 2) AS adoption_rate
FROM feature_flag_events
WHERE timestamp > NOW() - INTERVAL '24 hours'
GROUP BY feature;
Integrate with observability & monitoring dashboards.
Decision Matrix
When to Use Feature Flags
| Scenario | Use Feature Flag? | Type |
|---|---|---|
| New feature (incomplete) | ✅ Yes | Release toggle |
| Risky refactor | ✅ Yes | Kill switch |
| A/B test | ✅ Yes | Experiment toggle |
| Bug fix | ❌ No | Deploy normally |
| Config change | ⚠️ Maybe | Ops toggle |
| Gradual rollout | ✅ Yes | Rollout toggle |
| Paid feature | ✅ Yes | Permission toggle |
Tool Selection
| Team Size | Recommendation | Why |
|---|---|---|
| 1–10 engineers | Redis + custom | Simple, free |
| 10–50 engineers | Unleash (OSS) | Self-hosted, full-featured |
| 50–200 engineers | LaunchDarkly | SaaS, no maintenance |
| 200+ engineers | Split.io or Optimizely | Enterprise support |
Production Checklist
Implementation
- Feature flags stored in centralized system (Redis, LaunchDarkly)
- Local caching to reduce latency (<1ms)
- Kill switches for critical features
- Gradual rollout capability (1% → 100%)
- User targeting and segmentation
- A/B testing framework
Operations
- Flag evaluation metrics tracked (Prometheus)
- Dashboard for non-engineers (product, ops)
- Alerts on circuit breaker triggers
- Rollout playbook documented
- Incident response: how to kill feature
Governance
- Flag expiration policy (90 days)
- Automated expiration alerts
- Unused flag detection
- Code review requires flag metadata
- Documentation: why flag exists, removal criteria
Security
- Flag changes audit logged
- Production flag changes require approval
- Secrets not stored in flags
- Rate limiting on flag API
Related implementation guides:
Primary references: official documentation, official documentation, official documentation, official documentation.
Operating Feature Flags in Production as a System
The implementation is only one part of Feature Flags in Production. A production design also needs an explicit contract for inputs, outputs, ownership, and failure behavior. Write that contract before selecting a library. It should identify which component validates input, where state lives, what may be retried, and which result is authoritative when two components disagree. This prevents a convenient prototype boundary from silently becoming the long-term architecture.
Start with a representative baseline. Capture request shape, traffic distribution, dependency latency, error classes, and the quality signal users actually care about. Averages hide the cases that cause incidents, so keep percentiles and segment measurements by workload type. Record the configuration and dataset version beside every result. Without that context, a faster or more accurate run cannot be reproduced and should not be used to approve a rollout.
Define the failure model
List failures by where they originate: invalid input, capacity exhaustion, dependency timeout, partial state change, malformed output, and semantically wrong output. Each class needs a different response. Validation errors should fail immediately. Transient dependency failures may be retried with a budget and jitter. An operation that may have committed must use an idempotency key or reconciliation step before retrying. A syntactically valid but incorrect result belongs in evaluation and review, not a blind retry loop.
Set a deadline for the complete operation and derive smaller budgets for each dependency. Local timeouts that add up to more than the caller's deadline merely create abandoned work. Propagate cancellation where the protocol supports it. Bound every queue, retry loop, context buffer, and concurrency pool; an unbounded safety mechanism becomes a second outage during overload.
Design a degraded mode before it is needed. Depending on the workload, that can mean returning a cached answer, selecting a simpler path, placing work in a durable queue, or asking for human review. The degraded response must be visible in telemetry and, where it changes meaning, visible to the caller. Silent fallback makes quality regressions almost impossible to diagnose.
Measure the decision, not just the component
Use three layers of signals. System metrics cover latency, throughput, saturation, and errors. Correctness metrics measure whether the result satisfies its contract. Business or user metrics show whether the system solved the intended problem. Improving only one layer can move the others backward, so release criteria should name acceptable movement for all three.
Attach a reason code to every route, rejection, fallback, and retry. Include version identifiers for configuration, code, model, schema, and data when relevant. Logs should let an engineer reconstruct a decision without storing secrets or raw personal data. Traces should cross process boundaries, while metrics should remain low-cardinality enough to operate reliably.
Alert on symptoms that require action, not every internal anomaly. A useful alert names the affected service objective, links to a runbook, and distinguishes a customer-visible incident from exhausted headroom. Dashboards serve a different purpose: they support diagnosis and capacity planning. Treating a dashboard as an alerting strategy leaves failures undiscovered until someone happens to look.
Roll out with reversible steps
Ship Feature Flags in Production behind a versioned interface and a kill switch. Begin with offline replay using production-shaped, privacy-safe samples. Then use shadow execution when duplicate work has acceptable cost and side effects can be suppressed. A small canary should exercise the real dependency graph before traffic expands. Compare the canary with the baseline by cohort rather than mixing both populations into one aggregate.
Promotion gates should be written before the rollout. Include a minimum sample size or observation window, maximum regression in tail latency and error rate, and a correctness threshold. Roll back automatically when a hard safety boundary is crossed; use manual review for ambiguous quality movement. Preserve enough evidence from both paths to explain why the gate passed or failed.
Configuration deserves the same discipline as code. Review changes, validate them before activation, keep an immutable history, and make rollback a single operation. If a deployment changes code and configuration together, record both versions. Otherwise an incident responder may roll back the binary while leaving the triggering configuration active.
Capacity and cost controls
Model capacity in units the bottleneck understands: concurrent connections, tokens, queue jobs, database transactions, GPU memory, or bytes in flight. Convert the expected traffic distribution into those units and include burst behavior. Then load-test the first constrained dependency, not merely the public endpoint. A system that accepts more work than it can finish within its deadline is overloaded even if CPU utilization looks comfortable.
Cost is also a reliability limit. Add per-request attribution, tenant or workflow budgets, and a global circuit breaker for unexpectedly expensive paths. Review unit economics at the same granularity as performance; a cheap median can conceal a small class of requests responsible for most spend. Optimize only after measuring, because reducing context, replicas, validation, or redundancy can trade visible cost for less visible risk.
Production readiness review
Before launch, ask an engineer who did not build the feature to follow the runbook through one simulated failure. Verify backups or checkpoints by restoring them, not by checking that a job reported success. Exercise credential rotation, dependency unavailability, bad configuration, and rollback. Assign an owner for each alarm and a date for reviewing thresholds after real traffic arrives.
The final architecture document should be short enough to remain current. Keep the decision, rejected alternatives, invariants, dependency contracts, dashboards, and rollback procedure. Link detailed experiments rather than pasting them into the document. Teams that need help turning this review into an operable service can use our Feature Flags in Production engineering support.
Frequently Asked Questions
Should every feature be behind a flag?
Not every feature. Use flags for: risky changes, gradual rollouts, experiments, and kill switches. Don't use for: bug fixes, trivial changes, or when cleanup cost exceeds benefit.
How do I prevent feature flag technical debt?
Set expiration dates on all flags. Alert when flags approach expiration. Automated detection of unused flags. Code review checklist: "Is this flag still needed?"
What is the performance impact of feature flags?
<1ms with Redis caching. Evaluate once per request, cache result. Batch evaluations when fetching multiple flags.
Can I do A/B testing with feature flags?
Yes, feature flags enable A/B testing by assigning users to variants consistently. Track conversions and analyze results to determine winner.
How do I roll back a feature instantly?
Kill switch: Set flag to disabled in Redis. All servers pick up change within seconds (cache TTL). No redeployment required.
Should I use feature flags for configuration?
Be cautious. Flags are for code changes, not runtime config. Use environment variables or config service for settings like API keys, timeouts, URLs.
What happens if Redis goes down?
Fail-safe default: If flag service unavailable, default to "disabled" (safe) or use last cached value. Monitor Redis availability.
How do I coordinate feature flags across microservices?
Centralized flag service: All services query same Redis/LaunchDarkly. Ensures consistent user experience across services.
Conclusion
Feature flags in production extend far beyond simple on/off toggles:
- Progressive delivery with gradual rollout and instant rollback
- Kill switches for emergency feature disablement (<1 second)
- A/B testing and experimentation without code changes
- Targeting rules for beta users, regions, and paid tiers
- Technical debt managed with expiration policies
Feature flags are infrastructure, not just development convenience — invest in proper tooling and governance.
At HinterBuild, we design progressive delivery systems for production workloads:
- Backend API Engineering
- Cloud Infrastructure & DevOps
- Observability & Monitoring
- Kubernetes Platform Engineering
Schedule a consultation for feature flag architecture review.
Free consultation
Book a free consultation call on feature flag architecture & progressive delivery
30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.
Book a meeting
Keep reading
Related articles
Rate Limiting Strategies Beyond Simple Counters
Rate Limiting Strategies Beyond Simple Counters guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable,.
Read post
Feature Stores for ML: When, Why, and How to Build
Feature Stores for ML guidance for engineers: compare architecture choices, avoid failure modes, and ship a measurable, reliable production implementation.
Read post
Monorepo vs Multi-Repo: Engineering Tradeoffs
Monorepo vs multi-repo comparison for repository strategy — with scaling patterns, CI/CD optimization, tooling analysis, and when to use each approach.
Read post
System Design for 10M Users: Practical Architecture Guide
Learn system design for 10m users through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Read post