Embeddings Explained: Complete Guide to Vector Search for

Table of Contents:

• What Are Embeddings?
• How Embeddings Work Under the Hood
• Embedding Model Comparison (2026)
• Chunking Strategies That Affect Retrieval Quality
• Vector Databases and Storage Options
• Building a Production Embedding Pipeline
• Common Mistakes and How to Fix Them
• Frequently Asked Questions

What Are Embeddings?

Short answer: Embeddings are numerical vectors that represent text (or images, code, audio) in a high-dimensional space where semantically similar content sits close together.

If you searched "embeddings explained", you likely need to understand how vector representations power semantic search, RAG systems, and AI agent memory. After building embedding pipelines for 15+ production deployments at HinterBuild, one pattern holds: retrieval quality is 80% embedding and chunking choices, 20% LLM generation.

Key Takeaways:

• Embeddings convert text into fixed-size float arrays (typically 384–3072 dimensions)
• Similar meaning → smaller distance in vector space (cosine similarity or dot product)
• Model choice, chunk size, and metadata filtering matter more than vector DB brand
• Re-embed your corpus when you change models — vectors are not interchangeable
• Always evaluate retrieval with domain-specific queries, not generic benchmarks

A support team asked why their RAG bot kept returning irrelevant policy sections. The LLM was fine. The embedding model was general-purpose, chunks were 4,000 tokens with no overlap, and queries about "returns" matched "shipping" because both lived in the same chunk. Fixing embeddings and chunking raised answer accuracy from 61% to 89% without changing the model.

This guide explains embeddings end-to-end: theory, model selection, chunking, vector databases, and production code.

How Embeddings Work Under the Hood

An embedding model is a neural network trained to map input text to a dense vector. The training objective teaches the model that related texts should produce nearby vectors.

The Math (Simplified)

Given text "How do I return a product?", the model outputs:

[0.023, -0.156, 0.891, ..., 0.044]  # 1536 dimensions for text-embedding-3-small

Compare against stored document vectors using cosine similarity:

similarity(A, B) = dot(A, B) / (||A|| × ||B||)

Scores range from -1 to 1. In practice, relevant matches score > 0.7 for same-domain content with good models.

Why Embeddings Beat Keyword Search

Keyword search matches exact tokens. Embeddings match meaning:

This semantic matching is what makes RAG retrieval work for natural language queries against unstructured docs.

Embedding Types

Dense embeddings — Single vector per input. Used in most RAG systems (OpenAI, Cohere, Voyage, BGE).

Sparse embeddings — High-dimensional vectors with mostly zeros (BM25-style + learned). Hybrid search combines sparse + dense for best recall.

Multi-vector (ColBERT-style) — Multiple vectors per document for token-level matching. Higher quality, higher storage cost.

For most production AI applications, start with dense embeddings. Add hybrid search when recall on technical terms is insufficient.

Embedding Model Comparison (2026)

Choosing an embedding model affects retrieval accuracy, latency, and cost. Here is how leading options compare for English-centric production RAG.

MTEB scores are approximate benchmarks — always run your own eval with domain queries.

Model Selection Criteria

1. Domain match — Legal, medical, and code domains benefit from specialized models (Voyage-code, domain-finetuned BGE). General models underperform on jargon-heavy corpora.

2. Dimension vs speed tradeoff — OpenAI supports dimension reduction (e.g., 3072 → 512) with minimal quality loss. Smaller vectors = faster search, less storage.

3. Context length — If chunks exceed 512 tokens, use models supporting 8K+ input (Cohere, Voyage, OpenAI v3). Truncated input = lost semantics.

4. Multilingual needs — Cohere embed-v4 and multilingual-e5-large handle 100+ languages. English-only models fail on mixed-language support tickets.

5. Deployment constraints — Cloud API (OpenAI, Cohere) vs self-hosted (BGE, Nomic) for data residency requirements. Self-hosted adds infrastructure overhead.

from openai import OpenAI

client = OpenAI()

def embed_texts(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
    """Batch embed texts with OpenAI API."""
    response = client.embeddings.create(
        input=texts,
        model=model,
        dimensions=1536,  # optional reduction for v3 models
    )
    return [item.embedding for item in response.data]

def embed_query(query: str) -> list[float]:
    return embed_texts([query])[0]
from sentence_transformers import SentenceTransformer

local_model = SentenceTransformer("BAAI/bge-large-en-v1.5")

def embed_local(texts: list[str]) -> list[list[float]]:
    return local_model.encode(texts, normalize_embeddings=True).tolist()

Run retrieval evals before committing. Swap models on a 200-query test set and compare recall@5.

Chunking Strategies That Affect Retrieval Quality

Chunking splits documents into embeddable units. Bad chunking destroys even the best embedding model.

Core Chunking Parameters

Chunking Strategies by Content Type

Documentation / wikis — Use structural splitting on headings (H1, H2). Keep sections intact. 512–800 tokens per chunk.

Legal / policy — Split on numbered clauses. Never split mid-sentence. Include section title in chunk metadata.

Code repositories — Split on function/class boundaries. Include file path and language in metadata. See MCP code retrieval patterns.

Support tickets / chat logs — One ticket = one or more chunks. Include resolution status in metadata for filtering.

PDFs with tables — Extract tables separately. Do not embed table rows mixed with prose — retrieval returns fragments.

from langchain_text_splitters import RecursiveCharacterTextSplitter

def chunk_documents(documents: list[dict], chunk_size: int = 512, overlap: int = 64) -> list[dict]:
    """Chunk documents with metadata preservation."""
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=overlap,
        separators=["\n\n", "\n", ". ", " ", ""],
        length_function=len,
    )

    chunks = []
    for doc in documents:
        texts = splitter.split_text(doc["content"])
        for i, text in enumerate(texts):
            chunks.append({
                "content": text,
                "metadata": {
                    **doc.get("metadata", {}),
                    "source": doc["source"],
                    "chunk_index": i,
                    "total_chunks": len(texts),
                }
            })
    return chunks

Semantic Chunking (Advanced)

Instead of fixed token counts, split when embedding similarity between adjacent sentences drops — indicating a topic shift. Higher quality, higher compute cost. Worth it for long unstructured documents.

Metadata: The Hidden Retrieval Lever

Store and filter on metadata during search:

• document_type, product_line, effective_date, access_level
• Pre-filter before vector search to reduce noise
• Critical for multi-tenant RAG systems

Our backend API engineering team treats chunk metadata schemas as first-class API contracts.

Vector Databases and Storage Options

Vector databases store embeddings and enable fast similarity search at scale. The right choice depends on scale, existing infrastructure, and operational requirements.

Vector Database Comparison

When to Use pgvector vs Dedicated Vector DB

Use pgvector when:

• You already run PostgreSQL in production
• Corpus is < 5M vectors
• You need transactional consistency (embed on insert in same transaction)
• Team knows SQL, not another query language

Use dedicated vector DB when:

• Corpus exceeds 10M vectors
• Sub-10ms p99 latency at scale is required
• Advanced hybrid search and reranking are built-in requirements
• Horizontal sharding is needed

import psycopg
from pgvector.psycopg import register_vector

async def upsert_chunks(conn: psycopg.AsyncConnection, chunks: list[dict], embeddings: list[list[float]]):
    register_vector(conn)
    async with conn.cursor() as cur:
        for chunk, embedding in zip(chunks, embeddings):
            await cur.execute(
                """
                INSERT INTO document_chunks (content, metadata, embedding, source)
                VALUES (%s, %s, %s, %s)
                ON CONFLICT (source, chunk_index) DO UPDATE
                SET content = EXCLUDED.content, embedding = EXCLUDED.embedding
                """,
                (chunk["content"], chunk["metadata"], embedding, chunk["metadata"]["source"])
            )

async def search_similar(conn, query_embedding: list[float], top_k: int = 5, filters: dict = None):
    register_vector(conn)
    filter_clause = ""
    params = [query_embedding, top_k]

    if filters:
        filter_clause = "AND metadata @> %s"
        params.insert(1, filters)

    async with conn.cursor() as cur:
        await cur.execute(
            f"""
            SELECT content, metadata, 1 - (embedding <=> %s) AS similarity
            FROM document_chunks
            WHERE 1=1 {filter_clause}
            ORDER BY embedding <=> %s
            LIMIT %s
            """,
            params,
        )
        return await cur.fetchall()

Deploy vector infrastructure with cloud infrastructure and DevOps best practices — backup embeddings, monitor index rebuild times, set memory limits.

Building a Production Embedding Pipeline

A production embeddings pipeline has five stages: ingest → chunk → embed → store → retrieve.

Stage 1: Document Ingestion

Pull from sources: S3, Google Drive, Confluence, GitHub, databases. Track document_id, version, last_modified. Only re-process changed documents.

Stage 2: Chunk and Enrich

Apply content-type-specific chunking. Attach metadata. Optionally generate chunk summaries for better retrieval (embed summary + content).

Stage 3: Batch Embedding

Embed in batches of 100–500 texts. Handle rate limits with exponential backoff. Cache embeddings by content hash — skip re-embedding unchanged chunks.

import hashlib
import asyncio
from typing import Callable

async def embed_with_cache(
    chunks: list[dict],
    embed_fn: Callable,
    cache: dict,
) -> list[list[float]]:
    """Embed only chunks whose content hash changed."""
    results = []
    to_embed = []
    to_embed_indices = []

    for i, chunk in enumerate(chunks):
        content_hash = hashlib.sha256(chunk["content"].encode()).hexdigest()
        if content_hash in cache:
            results.append(cache[content_hash])
        else:
            results.append(None)
            to_embed.append(chunk["content"])
            to_embed_indices.append(i)

    if to_embed:
        new_embeddings = embed_fn(to_embed)
        for idx, embedding in zip(to_embed_indices, new_embeddings):
            content_hash = hashlib.sha256(chunks[idx]["content"].encode()).hexdigest()
            cache[content_hash] = embedding
            results[idx] = embedding

    return results

Stage 4: Index and Store

Upsert to vector DB with metadata. Build indexes (HNSW, IVF) for approximate nearest neighbor search. Monitor index size and query latency.

Stage 5: Retrieve and Rerank

Two-stage retrieval improves quality:

1. Vector search — Top 20 candidates by cosine similarity
2. Reranker — Cross-encoder model (Cohere rerank, bge-reranker) scores query-chunk pairs, returns top 5

async def retrieve_with_rerank(query: str, vector_store, rerank_client, top_k: int = 5):
    query_embedding = embed_query(query)
    candidates = await vector_store.search(query_embedding, top_k=20)

    reranked = rerank_client.rerank(
        query=query,
        documents=[c["content"] for c in candidates],
        top_n=top_k,
    )

    return [candidates[r.index] for r in reranked.results]

Integrate retrieval into agent memory architecture for personalized context injection.

Monitor retrieval quality with observability and monitoring: track recall@k on sampled queries, log retrieval scores, alert on score distribution shifts.

Common Mistakes and How to Fix Them

Mistake 1: Mixing Embedding Models

Vectors from OpenAI and BGE are not comparable. Changing models requires full re-embedding of your corpus. Version your embedding model in metadata.

Mistake 2: Chunks Too Large

4,000-token chunks dilute semantic signal. Retrieval returns entire sections when users need one paragraph. Fix: 512–800 tokens with 64-token overlap.

Mistake 3: No Metadata Filtering

Searching 100K chunks globally when the user asks about "Product A" returns Product B docs with similar language. Fix: filter by product, date, access level before vector search.

Mistake 4: Skipping Retrieval Eval

Teams tune prompts endlessly while retrieval returns wrong context. Fix: build a 100-query eval set, measure recall@5, fix chunking/model before touching the LLM.

Mistake 5: Ignoring Freshness

Stale embeddings after doc updates cause hallucinations grounded in old content. Fix: event-driven re-embedding on document change, TTL on cached vectors.

These mistakes appear repeatedly in production agent failures. Fix retrieval before blaming the model.

Evaluating Retrieval Quality

Before changing embedding models or chunk sizes, build a labeled query set of 50–100 real user questions with expected source documents. Measure recall@5 (did the correct doc appear in top 5?) and MRR (mean reciprocal rank). Run this eval after every pipeline change — the same discipline used in LLM evaluation workflows.

Target recall@5 ≥ 0.85 before investing in prompt tuning. Teams that skip retrieval eval spend weeks optimizing generation while the wrong context arrives every time. Log retrieval scores in production and sample 1% of queries for human review monthly.

Primary references: official documentation, official documentation, official documentation, official documentation.

Frequently Asked Questions

What are embeddings in simple terms?

Embeddings are lists of numbers that represent the meaning of text. Similar meanings produce similar numbers, enabling computers to find related content even when exact words differ.

What is the difference between embeddings and tokens?

Tokens are text fragments the LLM processes for generation. Embeddings are numerical vectors used for search and similarity. Different models, different purposes — your RAG system uses both.

Which embedding model should I use in 2026?

For most English RAG: OpenAI text-embedding-3-small (cost-effective) or text-embedding-3-large (higher accuracy). For self-hosted: BGE-large-en-v1.5. For multilingual: Cohere embed-v4. Always eval on your data.

How big should chunks be for RAG?

512–800 tokens with 10–20% overlap is the production sweet spot for documentation. Code and legal content need structure-aware splitting, not fixed sizes.

Do I need a dedicated vector database?

Not always. pgvector works well under 5M vectors with existing Postgres. Dedicated vector DBs (Pinecone, Qdrant, Weaviate) matter at scale or when you need sub-10ms latency.

Can I use the same embeddings for search and LLM context?

Yes — that is the standard RAG pattern. Embed chunks for retrieval, inject retrieved text into the LLM prompt. The LLM does not use embedding vectors directly.

How often should I re-embed my documents?

Re-embed when document content changes (event-driven) or when you switch embedding models (full corpus re-embed). Do not re-embed unchanged content — cache by content hash.

What is hybrid search and when do I need it?

Hybrid search combines dense embeddings with sparse keyword matching (BM25). Use it when domain-specific terms (SKUs, error codes, legal citations) are missed by semantic search alone.

Conclusion

Embeddings explained in production terms:

• Embeddings map meaning to vectors — the foundation of semantic search and RAG
• Model choice and chunking determine retrieval quality more than vector DB brand
• Build eval sets with domain queries before tuning LLM prompts
• Re-embed on model changes; cache on content changes

At HinterBuild, we build embedding pipelines and RAG systems for production AI:

• RAG & LLM Systems
• AI Agent Development
• Backend API Engineering
• Cloud Infrastructure & DevOps

Schedule a consultation to design your vector search architecture.