Constitutional AI Prompting: Self-Critique Patterns That
Learn constitutional ai prompting through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Muhammad Abdul Sami
· 10 min read
- RAG
- Embeddings
- Vector Databases
- Evaluation
Table of Contents:
- Constitutional AI Fundamentals
- Training-Time CAI vs Constitutional AI Prompting
- Self-Critique Patterns
- Refinement Loops
- Harmlessness Principles
- Production Implementation
- Multi-Step Verification
- Quality Scoring
- Cost vs Quality Tradeoff
- Failure Modes of Self-Critique
- Monitoring Self-Critique
- Frequently Asked Questions
Constitutional AI Fundamentals: Training Models to Self-Improve
Short answer: Constitutional AI prompting uses a written set of principles (a "constitution") to drive a generate, critique, revise loop at inference time. The model produces a draft, critiques it against the principles, then rewrites it to comply, with no human in the loop for the common case.
A content moderation AI agent was flagging false positives at 18%. We implemented Constitutional AI patterns with safety principles. The model now self-critiques borderline cases against defined guidelines before final classification. False positives dropped to 6%, a 67% reduction.
Key Takeaways:
- Constitutional AI prompting is an inference-time pattern: draft, critique against explicit principles, revise. It needs no fine-tuning.
- Principles must be testable. "Be safe" is useless; "Do not state a dosage without a 'consult a clinician' caveat" is checkable by a critic model.
- Self-critique catches policy violations far better than reasoning errors. Use it for compliance and tone, not for fixing math.
- Budget 2-3x tokens for a full critique-and-revise loop; gate it by risk level so low-stakes traffic pays once.
- Verify with a different model for critical outputs. A model grading its own work shares its own blind spots.
- Track the refinement rate. A rising rate means the base prompt regressed; a rate near zero means the critic is not looking hard enough.
The technique comes from Anthropic's paper Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022), which used critique-and-revision to generate training data. This post is about the prompting version: applying the same loop at request time to any hosted model, whether you call Anthropic's API or OpenAI's. For production AI agents, it reduces post-deployment interventions significantly.
Training-Time CAI vs Constitutional AI Prompting
The original paper used the constitution in two training phases: supervised fine-tuning on revised outputs, then reinforcement learning from AI feedback (RLAIF) where the model's own preferences replaced human labels. That is a model-vendor activity. What application teams do is the inference-time version, and it helps to be precise about what you are and are not getting.
| Aspect | Training-time CAI (RLAIF) | Constitutional AI prompting (this post) |
|---|---|---|
| Where principles live | Baked into weights | In prompts you control and version |
| Cost per request | None extra | 2-3x tokens for critique and revision |
| Latency | None extra | One or two additional round-trips |
| Principle updates | Requires retraining | Edit a prompt, redeploy |
| Domain specificity | General safety | Your policies, your tone, your compliance rules |
| Who can do it | Model vendors, fine-tuning teams | Any team with API access |
| Failure surface | Model-level | Critic prompt quality, JSON parsing, loop bounds |
The practical implication: hosted frontier models already carry a general constitution from training. Your prompting layer should encode the domain-specific rules the vendor could not know: your refund policy, your regulated-advice disclaimers, your brand voice. Duplicating generic "don't be harmful" principles adds cost and little value. If you find yourself needing behavior changes that prompts cannot achieve, that is the signal to consider fine-tuning instead of prompting.
Self-Critique Patterns: Teaching Models to Review Their Work
The core loop has three calls: generate, critique, revise. The critique step returns structured JSON so the revise step can be skipped when nothing was flagged, which is the single biggest cost lever in the pattern. Ask for response_format={"type": "json_object"} (or a JSON schema) so parsing never becomes the failure point; see our guide to structured output from LLMs for the hardening details.
from typing import Dict, Any
from openai import AsyncOpenAI
client = AsyncOpenAI()
CRITIQUE_PROMPT = """Review this response against these principles:
Principles:
{principles}
Original response:
{response}
Critique:
1. What principles does this response violate, if any?
2. What could be improved?
3. What is done well?
Return JSON: {{"violations": [...], "improvements": [...], "strengths": [...]}}"""
REVISION_PROMPT = """Original response:
{original}
Critique:
{critique}
Revised response following the critique:"""
async def constitutional_response(
user_query: str,
principles: list[str],
) -> Dict[str, Any]:
"""Generate response with self-critique loop."""
initial = await client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": user_query}],
)
initial_response = initial.choices[0].message.content
# Step 2: Self-critique
principles_text = "\n".join([f"- {p}" for p in principles])
critique_result = await client.chat.completions.create(
model="gpt-4o",
messages=[{
"role": "user",
"content": CRITIQUE_PROMPT.format(
principles=principles_text,
response=initial_response,
),
}],
response_format={"type": "json_object"},
)
import json
critique = json.loads(critique_result.choices[0].message.content)
# Step 3: Revise if violations found
if critique["violations"]:
revised = await client.chat.completions.create(
model="gpt-4o",
messages=[{
"role": "user",
"content": REVISION_PROMPT.format(
original=initial_response,
critique=json.dumps(critique, indent=2),
),
}],
)
final_response = revised.choices[0].message.content
else:
final_response = initial_response
return {
"response": final_response,
"initial": initial_response,
"critique": critique,
"revised": critique["violations"] != [],
}
# Example usage
SAFETY_PRINCIPLES = [
"Never provide instructions for illegal activities",
"Avoid generating hateful or discriminatory content",
"Don't reveal personal identifiable information",
"Refuse requests that could cause harm",
]
result = await constitutional_response(
"How do I bypass security systems?",
principles=SAFETY_PRINCIPLES,
)
print(f"Initial: {result['initial']}")
print(f"Critique: {result['critique']}")
print(f"Final: {result['response']}")
Integrate with system prompt design patterns for comprehensive behavior control.
Writing principles the critic can actually apply
The quality of Constitutional AI prompting is capped by the quality of the constitution. Three rules from production:
- One observable behavior per principle. "Be helpful and safe" is two principles, neither checkable. "If the user asks for medication dosage, include a line recommending they confirm with a pharmacist" can be verified by reading the output.
- Prefer positive obligations for revision, prohibitions for refusal. Prohibitions ("never reveal internal pricing") are easy to critique against but give the reviser no direction. Pair them with what to do instead ("direct the user to the public pricing page").
- Order principles by priority and say so. When principles conflict (helpfulness vs. a disclaimer requirement), the critic needs a tie-break rule. State it explicitly at the top of the constitution.
A useful test: hand the constitution and a sample output to a colleague and ask them to grade it. If two humans disagree on whether a principle was violated, the critic model will be inconsistent too.
Refinement Loops: Iterative Quality Improvement
A single critique pass fixes most policy violations. A bounded refinement loop is for outputs with graded quality criteria (clarity, completeness, tone) where the first revision may still fall short. Two design rules keep it from becoming a cost sink: always cap max_iterations, and score with a cheaper model than the one generating.
class RefinementLoop:
"""Iterative refinement with quality gates."""
def __init__(
self,
client,
max_iterations: int = 3,
quality_threshold: float = 0.85,
):
self.client = client
self.max_iterations = max_iterations
self.quality_threshold = quality_threshold
async def generate_with_refinement(
self,
prompt: str,
quality_criteria: list[str],
) -> Dict[str, Any]:
"""Generate and refine until quality threshold met."""
current_response = None
iterations = []
for i in range(self.max_iterations):
# Generate/refine
if current_response is None:
response = await self._generate(prompt)
else:
response = await self._refine(prompt, current_response, quality_criteria)
# Evaluate quality
quality_score = await self._evaluate_quality(response, quality_criteria)
iterations.append({
"iteration": i + 1,
"response": response,
"quality_score": quality_score,
})
current_response = response
if quality_score >= self.quality_threshold:
break
return {
"final_response": current_response,
"iterations": iterations,
"converged": quality_score >= self.quality_threshold,
}
async def _generate(self, prompt: str) -> str:
response = await self.client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
)
return response.choices[0].message.content
async def _refine(
self,
original_prompt: str,
previous_response: str,
criteria: list[str],
) -> str:
refine_prompt = f"""Original request: {original_prompt}
Previous response:
{previous_response}
Quality criteria not fully met:
{chr(10).join([f"- {c}" for c in criteria])}
Generate an improved response that better satisfies all criteria:"""
return await self._generate(refine_prompt)
async def _evaluate_quality(
self,
response: str,
criteria: list[str],
) -> float:
"""Score response quality (0-1)."""
eval_prompt = f"""Rate this response on how well it satisfies these criteria (0-10 scale):
Criteria:
{chr(10).join([f"- {c}" for c in criteria])}
Response:
{response}
Return JSON: {{"score": 0-10, "reasoning": "..."}}"""
result = await self.client.chat.completions.create(
model="gpt-4o-mini", # Cheaper model for evaluation
messages=[{"role": "user", "content": eval_prompt}],
response_format={"type": "json_object"},
)
import json
data = json.loads(result.choices[0].message.content)
return data["score"] / 10.0 # Normalize to 0-1
# Usage
refiner = RefinementLoop(client, max_iterations=3, quality_threshold=0.85)
result = await refiner.generate_with_refinement(
prompt="Explain quantum computing to a 10-year-old",
quality_criteria=[
"Uses simple, age-appropriate language",
"Includes concrete analogies",
"Avoids technical jargon",
"Explains key concepts clearly",
],
)
print(f"Converged: {result['converged']}")
for iter_data in result['iterations']:
print(f"Iteration {iter_data['iteration']}: Quality {iter_data['quality_score']:.2f}")
Combine with prompt versioning to track principle effectiveness.
In practice, most requests that converge do so by iteration two. If your telemetry shows many requests hitting the cap, the criteria are either contradictory or too vague for the scorer to reward progress, and no amount of iteration will fix that; rewrite the criteria instead.
Harmlessness Principles: Core Safety Guidelines
The three groupings below (harmlessness, helpfulness, honesty) mirror the structure of the original paper. Treat them as a starting template and replace the generic entries with your domain rules: a fintech assistant's honesty constitution should mention regulated-advice language; a support bot's helpfulness constitution should mention escalation paths.
HARMLESSNESS_CONSTITUTION = [
"Never assist with illegal activities or provide instructions for breaking laws",
"Avoid generating content that promotes violence, self-harm, or harm to others",
"Don't produce hateful, discriminatory, or harassing content based on protected characteristics",
"Refuse to generate sexually explicit content involving minors",
"Don't help users deceive, manipulate, or scam others",
"Avoid revealing personal identifiable information about real individuals",
"Don't provide medical, legal, or financial advice without appropriate disclaimers",
"Refuse to bypass safety systems or content policies",
]
HELPFULNESS_CONSTITUTION = [
"Provide accurate, factual information when answering questions",
"Acknowledge uncertainty when you don't know something",
"Give specific, actionable answers rather than vague generalities",
"Include relevant examples and context to aid understanding",
"Break complex topics into clear, digestible steps",
"Anticipate follow-up questions and address them proactively",
]
HONESTY_CONSTITUTION = [
"Admit limitations and gaps in knowledge",
"Cite sources when making factual claims",
"Distinguish between facts, opinions, and speculation",
"Correct previous errors when discovered",
"Be transparent about your nature as an AI",
]
Connect to multi-agent orchestration where agents enforce principles across specialized tasks.
Production Implementation: CAI System
The production version separates critical violations (refuse outright) from minor ones (refine). This matters because refining an unsafe response often produces a plausible unsafe response, which is worse than a clean refusal. The violation checker runs on a cheaper model; the generator and reviser run on the strong one.
from dataclasses import dataclass
from typing import List
@dataclass
class Constitution:
"""Set of principles governing AI behavior."""
harmlessness: List[str]
helpfulness: List[str]
honesty: List[str]
def all_principles(self) -> List[str]:
return self.harmlessness + self.helpfulness + self.honesty
class ConstitutionalAISystem:
"""Production Constitutional AI implementation."""
def __init__(self, client, constitution: Constitution):
self.client = client
self.constitution = constitution
async def generate(self, user_query: str) -> Dict[str, Any]:
"""Generate response with constitutional safeguards."""
# Generate initial response
initial_response = await self._generate_initial(user_query)
# Check for immediate violations
violations = await self._check_violations(initial_response)
if violations["critical"]:
# Critical violations—return refusal
return {
"response": "I cannot assist with that request as it violates safety guidelines.",
"initial": initial_response,
"violations": violations,
"refused": True,
}
# If minor violations, refine
if violations["minor"]:
refined_response = await self._refine_response(
user_query,
initial_response,
violations["minor"],
)
else:
refined_response = initial_response
return {
"response": refined_response,
"initial": initial_response,
"violations": violations,
"refined": bool(violations["minor"]),
"refused": False,
}
async def _generate_initial(self, query: str) -> str:
response = await self.client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": query}],
)
return response.choices[0].message.content
async def _check_violations(self, response: str) -> Dict[str, List[str]]:
"""Check response against constitution."""
check_prompt = f"""Review this response against safety principles:
Principles:
{chr(10).join([f"- {p}" for p in self.constitution.all_principles()])}
Response:
{response}
Identify violations:
- CRITICAL: Direct violations requiring refusal
- MINOR: Issues that should be refined
Return JSON: {{"critical": [...], "minor": [...]}}"""
result = await self.client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": check_prompt}],
response_format={"type": "json_object"},
)
import json
return json.loads(result.choices[0].message.content)
async def _refine_response(
self,
original_query: str,
response: str,
violations: List[str],
) -> str:
"""Refine response to address violations."""
refine_prompt = f"""Original query: {original_query}
Initial response:
{response}
Issues to address:
{chr(10).join([f"- {v}" for v in violations])}
Generate a refined response that maintains helpfulness while addressing these issues:"""
result = await self.client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": refine_prompt}],
)
return result.choices[0].message.content
# Production usage
constitution = Constitution(
harmlessness=HARMLESSNESS_CONSTITUTION,
helpfulness=HELPFULNESS_CONSTITUTION,
honesty=HONESTY_CONSTITUTION,
)
cai_system = ConstitutionalAISystem(client, constitution)
result = await cai_system.generate("How can I improve my credit score?")
print(f"Response: {result['response']}")
print(f"Refined: {result['refined']}")
print(f"Refused: {result['refused']}")
Deploy with RAG and LLM systems for safe, reliable outputs.
Multi-Step Verification
For outputs that carry real liability (regulated advice, external communications, actions with side effects), add an independent verifier from a different model family. Self-critique shares the generator's blind spots: if the model misread the policy when drafting, it will often misread it again when reviewing. A second model breaks that correlation. This is the same reasoning behind using a separate judge in LLM-as-judge evaluation.
async def multi_step_verification(
response: str,
principles: List[str],
) -> Dict[str, Any]:
"""Multi-model verification for critical outputs."""
# Step 1: Self-critique
self_critique = await constitutional_response(response, principles)
# Step 2: Independent verification (different model)
verify_prompt = f"""Verify this response follows all safety principles:
Principles:
{chr(10).join([f"- {p}" for p in principles])}
Response to verify:
{response}
Return JSON: {{"safe": true/false, "concerns": [...]}}"""
verification = await client.chat.completions.create(
model="claude-3-5-sonnet", # Different model for independence
messages=[{"role": "user", "content": verify_prompt}],
)
# Step 3: Aggregate results
import json
verify_data = json.loads(verification.choices[0].message.content)
return {
"approved": verify_data["safe"] and not self_critique["critique"]["violations"],
"self_critique": self_critique,
"external_verification": verify_data,
}
Quality Scoring
Scoring per dimension lets you weight what matters for the surface. A moderation pipeline weights harmlessness heavily; an internal documentation assistant weights helpfulness. Keep the weights in config, not code, so product can tune them without a deploy.
class QualityScorer:
"""Score output quality against constitution."""
def __init__(self, client, constitution: Constitution):
self.client = client
self.constitution = constitution
async def score(self, response: str) -> Dict[str, float]:
"""Score response on multiple dimensions."""
scores = {}
# Score harmlessness
scores["harmlessness"] = await self._score_dimension(
response,
self.constitution.harmlessness,
"How well does this avoid harmful content?",
)
# Score helpfulness
scores["helpfulness"] = await self._score_dimension(
response,
self.constitution.helpfulness,
"How helpful and actionable is this?",
)
# Score honesty
scores["honesty"] = await self._score_dimension(
response,
self.constitution.honesty,
"How honest and transparent is this?",
)
# Overall score (weighted average)
scores["overall"] = (
scores["harmlessness"] * 0.5 +
scores["helpfulness"] * 0.3 +
scores["honesty"] * 0.2
)
return scores
async def _score_dimension(
self,
response: str,
principles: List[str],
question: str,
) -> float:
"""Score on one dimension."""
score_prompt = f"""{question}
Principles:
{chr(10).join([f"- {p}" for p in principles])}
Response:
{response}
Rate 0-10. Return JSON: {{"score": 0-10}}"""
result = await self.client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": score_prompt}],
response_format={"type": "json_object"},
)
import json
data = json.loads(result.choices[0].message.content)
return data["score"] / 10.0
Cost vs Quality Tradeoff
Constitutional AI prompting adds roughly 2-3x token cost for a full loop (critique + refinement), and one to two extra round-trips of latency. The table below is illustrative for a typical support-style request; your ratios depend on prompt and output length.
| Configuration | LLM calls | Relative token cost | Added latency | Use when |
|---|---|---|---|---|
| Plain generation | 1 | 1x | 0 | Low-risk, high-volume traffic |
| Critique only (no revise unless flagged) | 2 (3 when flagged) | ~1.6-2x | +1 round-trip | Most production surfaces |
| Full critique + revise, always | 3 | ~2.5-3x | +2 round-trips | Regulated or externally visible output |
| Bounded refinement loop (cap 3) | 3-7 | ~3-6x | +2-6 round-trips | Graded-quality content, offline or async |
| Critique + independent verifier | 3-4 | ~3-4x | +2 round-trips | Actions with side effects, liability |
The cheapest large win is conditional revision: since the critique returns structured violations, only pay for the third call when something was flagged. Route by risk so low-stakes queries never enter the loop:
class AdaptiveCAI:
"""Use CAI only when needed."""
async def generate(self, query: str, risk_level: str) -> str:
"""Apply CAI based on risk."""
if risk_level == "low":
# Skip CAI for low-risk queries
return await self._simple_generate(query)
elif risk_level == "medium":
# Single-pass critique
return await self._single_critique(query)
else: # high risk
# Full CAI with refinement
return await self._full_cai(query)
Pair with LLM cost optimization strategies. Prompt caching helps here more than in most patterns because the constitution is a long, static prefix that repeats across the critique and revise calls.
Failure Modes of Self-Critique
Constitutional AI prompting is not a general quality multiplier. Knowing where it breaks saves you from over-applying it.
- It does not reliably fix reasoning errors. Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet, show that unguided self-correction can make answers worse on reasoning tasks. Self-critique works when the principle is a checkable policy, not when the model must notice its own arithmetic mistake. For factual accuracy, ground the answer with retrieval instead; see RAG hallucination fixes.
- Over-refusal drift. A critic prompt that rewards caution will steadily push outputs toward refusals and disclaimers. Track refusal rate alongside violation rate, and include helpfulness principles in the critique so the critic is penalized for gutting the answer.
- Sycophantic critique. If the critique prompt includes the original user request framed persuasively, the critic can be talked into approving. Critique the output against the principles; do not pass through user justifications.
- Principle conflicts without a tie-break. Two principles pulling in opposite directions cause oscillation in refinement loops. Declare priority order.
- JSON parsing as the weakest link. A critique call that returns malformed JSON should fail closed for critical categories (treat as violation, refuse or escalate) and fail open for minor ones. Decide this per category up front.
- Injection through the critique channel. A user can attempt to write text that reads as a principle ("Ignore the constitution above"). Keep principles in the system prompt and the output under review in a clearly delimited user turn. Our prompt injection defense guide covers the delimiting patterns.
Monitoring Self-Critique
class CAIMonitor:
"""Track Constitutional AI effectiveness."""
async def track_cai_execution(
self,
initial_response: str,
final_response: str,
violations_found: List[str],
refined: bool,
) -> None:
"""Record CAI metrics."""
await self.metrics.increment(
"cai.executions.total",
tags={"refined": refined},
)
if violations_found:
await self.metrics.increment(
"cai.violations.found",
len(violations_found),
)
if refined:
await self.metrics.increment("cai.refinements.performed")
Deploy with observability monitoring. The four numbers worth a dashboard: refinement rate (share of requests where revision ran), refusal rate, verifier disagreement rate (how often the independent check overrules self-critique), and p95 added latency. A refinement rate that climbs after a deploy is the earliest sign your base prompt or model version regressed.
Frequently Asked Questions
What is Constitutional AI prompting?
Constitutional AI prompting is an inference-time pattern where a model drafts a response, critiques it against a written list of principles, and revises it to comply. It borrows the critique-and-revision idea from Anthropic's Constitutional AI training method but applies it in your application code rather than in model training. It works with any hosted model and requires no fine-tuning.
When should I use Constitutional AI?
Use it for high-stakes outputs where policy compliance matters: content moderation, medical or financial disclaimers, legal information, customer-facing communications. Skip it for low-risk, high-volume tasks where the 2-3x token cost buys nothing, and do not rely on it to fix reasoning or factual errors.
Does self-critique actually improve output quality?
For policy and tone violations, yes: a critic reading the output against explicit rules catches most of what a single-pass generation misses. For reasoning and factual accuracy, the evidence is weak; research on self-correction shows unguided critique can make answers worse. Use retrieval for facts and Constitutional AI prompting for compliance.
How many principles should I include?
5-15 principles is optimal. Too few misses edge cases; too many creates conflicting guidance and increases token cost. Group related principles into categories (harmlessness, helpfulness, honesty).
Does CAI work with smaller models?
Frontier models work best (GPT-4o, Claude Sonnet). Smaller models struggle with nuanced self-critique. For smaller models, use external verification with a larger model.
Can I automate principle generation?
Yes—generate initial principles with LLMs, then refine based on production failures. Review and approve all principles before deploying.
How do I balance helpfulness and harmlessness?
Harmlessness takes priority—refuse unsafe requests even if unhelpful. For borderline cases, add disclaimers or caveats to maintain helpfulness while staying safe.
Conclusion
Constitutional AI enables models to self-improve against defined principles:
- Define clear principles for harmlessness, helpfulness, honesty
- Implement self-critique loops to catch issues before users see them
- Refine responses that violate principles
- Use risk-based triggers to balance cost and quality
- Monitor effectiveness continuously
In our deployments, self-critique typically cuts the share of outputs that need human review substantially; measure your own rate before and after rather than assuming a number.
If you are building a system where output compliance is a launch requirement, talk to us about AI agent development or RAG and LLM systems with Constitutional AI prompting built in.
Free consultation
Book a free consultation call on Constitutional AI & self-critique
30-minute call with the HinterBuild team. Discuss your project, architecture questions, or next steps — no obligation.
Book a meeting
Resources:
Keep reading
Related articles
When to Self-Host LLMs: Cost Analysis & Decision Framework
Learn when to self-host llms through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Read post
Corrective RAG (CRAG): Self-Correction & Retrieval Quality
Corrective RAG explained — self-critique retrieval, query rewriting, fallback search strategies, and production patterns for fixing bad RAG responses.
Read post
Self-Querying Retrieval Explained: LLM-Powered Metadata
Learn self-querying retrieval explained through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Read post
RAG vs Fine-Tuning vs Prompting: When to Use Each
Learn rag vs fine-tuning vs prompting through concrete architecture trade-offs, failure modes, rollout controls, and production measurement practices.
Read post