AI Agent Memory: Short-Term vs Long-Term Architecture

AI agent memory is two separate systems that teams routinely collapse into one. Short-term memory is the conversation inside the context window; long-term memory is structured, retrievable state that lives outside the model. Most agents that "forget" are not suffering from a small context window. They are suffering from having no explicit design for either layer, so the context window silently does both jobs badly.

"Does it remember anything?" is the first question users ask about any AI agent. After building memory architectures for 6+ production AI agent systems at HinterBuild, the honest answer is: most agents forget, and the longer the session, the worse they perform. This guide covers what each memory type is, where each one fails, and the four-layer hybrid architecture we use to fix it.

Key Takeaways:

• Short-term memory is a token-budgeted context window with selective truncation, not "whatever fits until the model drops it"
• Long-term memory is structured storage plus vector retrieval outside the LLM; raw chat logs in a database do not count
• Retrieval is 80% of the work. Memory without a precise retrieval layer is just data storage
• Inject at most 3-5 long-term memories per request; more memory is not better memory, precision beats volume
• Give every memory a TTL or invalidation path. Stale preferences cause worse decisions than missing ones
• Log which memories were injected on every call so you can debug "why did the agent think that?"

Table of Contents:

• Why Agent Memory Fails in Production
• Short-Term Memory (Context Window)
• Long-Term Memory (Persistent Storage)
• Hybrid Memory Architecture
• Memory Comparison Table
• Implementation Code Examples
• Invalidation, Privacy, and Observability
• Frequently Asked Questions

Why AI Agent Memory Fails in Production

Short answer: Most AI agent memory systems get worse over time because teams treat the LLM context window as memory instead of building explicit short-term and long-term memory layers.

The failure is gradual, which is why it survives testing. A 5-turn demo works perfectly. A 40-turn real support session degrades because the early turns holding the order ID, the customer's stated constraint, or the tool result the agent needs are the first ones truncated. The agent then re-asks, re-fetches, or, worse, guesses.

The second failure is the opposite: teams bolt a vector database onto the agent, dump every message into it, and inject the top-10 hits on every call. Now the context is full of half-relevant fragments from three months ago, and the model's attention is spread across noise. The MemGPT paper framed this well: the context window should be treated like RAM, with an explicit OS-style paging strategy, not as a bottomless log.

Short-Term Memory: The Context Window

Short-term memory in AI agents is the model's ability to recall the current conversation: the last N turns within the context window.

How Context Window Memory Works

• Conversation history stored in the LLM context window
• Each turn appends tokens (user messages, assistant responses, tool results)
• When the window fills, old turns get truncated, often silently
• The model "remembers" only what fits in the remaining budget

The Production Problem

Current frontier models advertise 128K to 1M token windows (see the Anthropic model docs and OpenAI model docs for exact figures per model). That sounds generous until you account for:

• System prompts and tool definitions (often 5-15K tokens for a tool-heavy agent)
• Tool results (a single search or database call can return 2-10K tokens)
• Model output tokens
• Real sessions reaching 30-50+ turns in 45 minutes

There is also a quality problem that arrives before the hard limit. Long-context research such as Lost in the Middle showed that models recall information at the start and end of a long context far better than information buried in the middle. A large window does not mean uniform recall across it. Cost scales with input tokens too, so a bloated history is paid for on every single turn.

I watched an agent go from perfect recall to asking the same question three turns later after 45 minutes. The context window had bloated silently. Early turns containing the order ID were truncated.

Short-Term Memory Design Patterns

1. Token budgeting. Allocate fixed portions: e.g., 60% conversation history, 40% system + tools. Never let history exceed its budget. We cover the accounting in detail in token budget management.

2. Selective truncation. Drop least informative turns, not oldest. Keep last 5-7 turns, keep all tool results, drop filler user utterances.

3. Summary injection. Before truncating, compress dropped turns into 1-2 sentence summaries the model can reference.

from typing import TypedDict
import tiktoken

class ConversationManager:
    """Token-budgeted short-term memory for AI agents."""

    def __init__(self, max_history_tokens: int = 8000, keep_recent_turns: int = 7):
        self.max_history_tokens = max_history_tokens
        self.keep_recent_turns = keep_recent_turns
        self.encoder = tiktoken.encoding_for_model("gpt-4o")

    def count_tokens(self, messages: list[dict]) -> int:
        return sum(len(self.encoder.encode(m["content"])) for m in messages)

    def truncate_with_summary(self, messages: list[dict]) -> list[dict]:
        """Truncate oldest turns but preserve summary."""
        if self.count_tokens(messages) <= self.max_history_tokens:
            return messages
        recent = messages[-self.keep_recent_turns:]
        dropped = messages[:-self.keep_recent_turns]

        # Generate summary of dropped turns (call LLM or rule-based)
        summary = self._summarize_turns(dropped)
        summary_msg = {
            "role": "system",
            "content": f"Earlier conversation summary: {summary}"
        }

        return [summary_msg] + recent

    def _summarize_turns(self, turns: list[dict]) -> str:
        """Compress dropped turns into actionable summary."""
        key_facts = []
        for turn in turns:
            if "ORD-" in turn.get("content", ""):
                key_facts.append(f"User discussed order {turn['content'][:50]}")
        return "; ".join(key_facts) if key_facts else "General inquiry session."

4. Pin critical facts. Identifiers, constraints, and confirmed decisions should be extracted into working memory the moment they appear, so truncating the turn that introduced them does not lose them. This is the single cheapest fix for the "re-asks for the order ID" failure.

Need help implementing session management? Our backend API engineering team builds production memory layers with Redis and PostgreSQL.

Long-Term Memory: Beyond the Context Window

Long-term memory lets AI agents remember across sessions: user preferences, past orders, historical interactions. This lives outside the context window entirely.

How Long-Term Memory Works

• Database or vector store outside the LLM
• Query by user identity (user_id, session_id)
• Relevant memories injected into context at query time
• Can persist indefinitely with proper invalidation

The Production Problem

Many teams dump raw conversation logs into a database and call it memory. When the model needs something from 2 months ago, retrieval fails. The data exists, but the retrieval layer does not work.

Memory without retrieval is just data storage.

Long-Term Memory Design Patterns

1. Structured memory entries. Store key-value pairs with metadata, not raw chat dumps:

# Good: structured
{"user_id": "u-1234", "key": "preferred_contact", "value": "email", "updated": "2026-09-01"}

# Bad: unstructured dump
{"user_id": "u-1234", "content": "User said they like email maybe on Tuesday..."}

2. Vector search for semantic retrieval. Use embeddings for "find memories related to billing preferences":

async def retrieve_memories(user_id: str, query: str, top_k: int = 3) -> list[dict]:
    """Retrieve relevant long-term memories via vector similarity."""
    query_embedding = await embed(query)

    results = await vector_db.search(
        collection=f"memories_{user_id}",
        vector=query_embedding,
        top_k=top_k,
        filter={"importance": {"$gte": 0.5}}
    )

    return [
        {
            "content": r["content"],
            "category": r["metadata"]["category"],
            "relevance_score": r["score"]
        }
        for r in results
    ]

3. Importance weighting. Weight by recency, user-explicit importance, and frequency of use. A preference the user stated directly ("always ship to my office") should outrank a preference inferred from one past order.

4. Explicit forgetfulness. Invalidate stale memories. Users change preferences. Accounts close. Without invalidation, agents make decisions on 2-year-old data.

5. Selective injection. Load only memories relevant to the current task. Order query means order memories. Billing query means billing memories. Never flood context with everything.

6. Write memories deliberately. Extraction should happen at session end (or on explicit events like "remember that…"), through a dedicated LLM call with a fixed schema, not by embedding every message as it arrives. The full write/read pipeline is covered in our agent memory architectures guide.

Implement semantic retrieval with our RAG and LLM systems expertise.

Hybrid Memory Architecture (What We Use in Production)

After 6+ production deployments, we use a four-layer hybrid memory architecture:

Layer 1: Context Window (Short-Term)

• Last 5-7 turns with hard token budget (8,000 tokens max)
• All tool results from those turns preserved
• Middleware enforces budget before every LLM call

Layer 2: Working Memory (Session-Level)

• In-memory store for current session
• Temporary calculations, intermediate results, pinned identifiers
• Cleared when session ends

Layer 3: Persistent Memory (Database-Backed)

• Structured key-value entries with metadata
• Vector embeddings for semantic search
• Indexed by user_id and category (preferences, history, settings)
• Retrieved on-demand and injected into context

Layer 4: Knowledge Base (Optional RAG)

• Vector store for factual information (docs, policies)
• Updated separately from conversation memory
• Used when model needs external knowledge, not user history

Figure 1: Production AI agent architecture. Memory retrieval flows through structured tools, separate from model reasoning.

The Retrieval Workflow

When a user asks "What was the status of my last order?":

1. Query long-term memory. Structured lookup by user_id + vector search for order-related history
2. Check working memory. Did user mention order ID this session?
3. Monitor context budget. Will injected memories fit?
4. Truncate if needed. Summary injection for dropped short-term turns
5. Inject and call LLM. System instructions + memories + trimmed history
6. Validate recall. Model confirms what it remembers before acting

This pattern prevents the state desync failures that break production agents.

Where Framework Checkpointers Fit

If you build on LangGraph, its checkpointer persists the full graph state per thread, which gives you Layers 1 and 2 (conversation plus working state) for free and survives process restarts. It does not give you Layer 3: cross-thread, per-user memory with semantic retrieval still has to be designed. See stateful agents with LangGraph checkpoints for the persistence side.

Short-Term vs Long-Term Memory Comparison

When to Use Each

Storage Choices by Layer

pgvector is usually enough for per-user memory: the collection per user is small (hundreds of entries, not millions), so the index size and latency concerns that push teams toward dedicated vector databases rarely apply.

Implementation Code Examples

Complete Memory Middleware

import redis.asyncio as redis
from dataclasses import dataclass

@dataclass
class AgentMemoryContext:
    user_id: str
    session_id: str
    short_term: list[dict]
    working_memory: dict
    long_term: list[dict]

class HybridMemoryManager:
    """Production hybrid memory for AI agents."""

    def __init__(self, redis_url: str, vector_db, max_context_tokens: int = 8000):
        self.redis = redis.from_url(redis_url)
        self.vector_db = vector_db
        self.conversation_mgr = ConversationManager(max_context_tokens)
        self.max_long_term_inject = 3  # Never inject more than 3 memories

    async def build_context(self, user_id: str, session_id: str, query: str) -> AgentMemoryContext:
        # Layer 1: Short-term from Redis session
        short_term = await self._get_session_history(session_id)

        # Layer 2: Working memory
        working = await self.redis.hgetall(f"working:{session_id}") or {}

        # Layer 3: Long-term retrieval (filtered, not all)
        long_term = await retrieve_memories(user_id, query, top_k=self.max_long_term_inject)

        # Enforce token budget on short-term
        short_term = self.conversation_mgr.truncate_with_summary(short_term)

        return AgentMemoryContext(
            user_id=user_id,
            session_id=session_id,
            short_term=short_term,
            working_memory=working,
            long_term=long_term
        )

    def format_for_llm(self, ctx: AgentMemoryContext) -> list[dict]:
        """Format memory layers as LLM messages."""
        messages = []

        # Inject long-term memories as system context
        if ctx.long_term:
            memory_text = "\n".join(f"- {m['content']}" for m in ctx.long_term)
            messages.append({
                "role": "system",
                "content": f"Relevant user history:\n{memory_text}"
            })

        # Add working memory if present
        if ctx.working_memory:
            messages.append({
                "role": "system",
                "content": f"Session context: {ctx.working_memory}"
            })

        # Add truncated conversation history
        messages.extend(ctx.short_term)

        return messages

Deploy with observability and monitoring to track memory retrieval accuracy and context token usage.

Invalidation, Privacy, and Observability

Three concerns get skipped in most memory designs and then surface as incidents.

Invalidation Rules

Every persistent memory needs one of three exit paths: a TTL (session facts expire in hours, inferred preferences in months), an overwrite key (a new value for preferred_contact replaces the old one rather than coexisting with it), or an explicit delete triggered by user action or account events. Memories that can only be added and never removed drift toward being wrong.

Privacy and Data Retention

Long-term memory is personal data by definition. Store it under the same retention policy and access controls as the rest of the user record, keep it per-tenant (one collection or a hard user_id filter, never a shared index), and make "forget everything about me" a single delete, not a data-engineering project. Our multi-tenant RAG isolation guide covers the namespace patterns that apply here too.

What to Log

For every LLM call, record which memories were injected (IDs and relevance scores), how many history tokens survived truncation, and whether a summary was substituted. When a user reports "it forgot X" or "it thought Y for no reason", that trace is the difference between a five-minute fix and a week of guesswork. Wire these into your existing traces rather than a separate memory log; the approach is the same as in LLM tracing with OpenTelemetry.

Frequently Asked Questions

What is the difference between short-term and long-term memory in AI agents?

Short-term memory is conversation history within the LLM context window (current session). Long-term memory is persistent storage outside the model (database, vector store) that survives across sessions and is retrieved on-demand. They fail differently: short-term fails by truncation, long-term fails by imprecise retrieval.

How do I prevent AI agents from forgetting mid-conversation?

Implement token budgeting middleware that monitors context window usage, selectively truncates oldest turns, and injects summaries of dropped content before the hard limit is reached. Pin identifiers and confirmed decisions into working memory as soon as they appear so they survive truncation. Never rely on the model's default truncation.

Should I store all conversations as long-term memory?

No. Store structured, categorized memories (preferences, key facts, account data) with importance weights. Raw conversation dumps create retrieval noise and privacy liability. Retrieve only the top-3 to top-5 relevant memories per query.

What is the best database for AI agent long-term memory?

Use Redis for session and working memory (fast, TTL-friendly) and PostgreSQL + pgvector or a dedicated vector database for semantic long-term retrieval. Per-user memory collections are small, so pgvector is usually sufficient and keeps memory in the same transaction boundary as user data. Match storage to access pattern.

How does agent memory relate to RAG?

RAG retrieves external knowledge (docs, policies). Agent memory retrieves user-specific history and preferences. Production agents typically need both: RAG for facts, memory for personalization, and they should be separate stores with separate update cadences.

Can LangGraph, CrewAI, or AutoGen handle memory automatically?

Partially. Frameworks provide session state primitives such as LangGraph checkpointers, but production memory architecture (token budgeting, vector retrieval, invalidation) must be built at the application layer. See our framework comparison for what each one ships.

How much long-term memory should I inject per request?

Maximum 3-5 relevant memories per query. More overwhelms the model, wastes context tokens, and increases the chance of a stale memory overriding what the user just said. Filter by user_id, semantic relevance, and importance score before injection.

How do I validate memory retrieval is working?

After injection, have the model confirm what it remembers: "I see you prefer email contact. Is that still correct?" This catches retrieval errors before they cascade into wrong actions. Offline, build a small eval set of (query, expected memory IDs) pairs and track recall@3 as you change the retrieval logic.

Conclusion

AI agent memory is not one feature. It is two systems working together:

• Short-term: Token-budgeted context with proactive truncation, pinned facts, and summary injection
• Long-term: Structured storage with precision retrieval and hard limits on injection, not bulk dumps
• Invalidation: Every memory needs a TTL, an overwrite key, or a delete path
• Observability: Log what was injected on every call so memory bugs are debuggable

The teams that succeed design memory around what users need to remember, then fit essential pieces into the context window, not the other way around.

Schedule a consultation to design memory architecture for your agent system, or read about our AI agent development work.

Resources:

• MemGPT: Towards LLMs as Operating Systems (arXiv)
• Lost in the Middle: How Language Models Use Long Contexts (arXiv)
• Anthropic documentation
• LangGraph documentation
• pgvector on GitHub