Graph RAG with Neo4j: Complete Guide to Knowledge Graph

Table of Contents:

• What Is Graph RAG?
• When Flat RAG Fails and Graph RAG Wins
• Neo4j Knowledge Graph Architecture for LLMs
• Building the Graph: Entity Extraction Pipeline
• Hybrid Retrieval: Vector Search + Graph Traversal
• Production Graph RAG Pipeline in Python
• Performance, Cost, and Operational Tradeoffs
• Frequently Asked Questions

What Is Graph RAG?

Short answer: Graph RAG combines knowledge graphs with LLM retrieval — storing entities and relationships in a graph database like Neo4j, then traversing connected nodes to retrieve context that flat vector search misses.

If you searched "Graph RAG Neo4j LLM", you likely hit a wall with standard RAG pipelines: the user asks "Which compliance officer approved the vendor contract that caused the Q3 outage?" and your embedding retrieval returns three chunks about outages, three about vendors, and zero that connect them. Graph RAG answers multi-hop questions by following explicit relationships instead of hoping cosine similarity stitches facts together.

Key Takeaways:

• Graph RAG stores entities (people, products, policies) and relationships (approved, caused, depends_on) as first-class data
• Neo4j is the most common production graph store for LLM retrieval because Cypher queries compose naturally with LLM-generated plans
• Hybrid retrieval — vector search for entry points, graph traversal for context expansion — beats either approach alone
• Entity extraction quality determines graph quality; garbage in, garbage out applies doubly to knowledge graphs
• Graph RAG adds latency and infrastructure complexity — use it when questions require relationship reasoning, not for every corpus

At HinterBuild, we deploy Graph RAG when flat RAG & LLM systems plateau on recall for interconnected domains: supply chain, compliance, org charts, product dependencies, and incident postmortems. One legal-tech client saw multi-hop answer accuracy jump from 54% to 91% after adding a Neo4j layer — same LLM, same documents, better structure.

This guide covers Graph RAG architecture, Neo4j schema design, hybrid retrieval patterns, and production Python code.

When Flat RAG Fails and Graph RAG Wins

Short answer: Flat RAG fails on multi-hop and relationship-heavy queries because embeddings encode local semantic similarity, not explicit connections between distant facts.

Query Types That Need Graphs

The Multi-Hop Problem

Consider this corpus fragment spread across three documents:

Doc A: "Sarah Chen is VP of Platform Engineering."
Doc B: "Platform Engineering owns the payments microservice."
Doc C: "The payments microservice failed during the March 15 incident."

A user asks: "Who is responsible for the service that failed on March 15?"

Flat retrieval embeds the query and returns Doc C (incident) and maybe Doc A (Sarah) — but never connects them. Graph RAG extracts:

(Sarah:C Person)-[:MANAGES]->(Platform:Team)-[:OWNS]->(Payments:Service)-[:FAILED_IN]->(Incident:Event {date: '2026-03-15'})

One traversal returns the answer: Sarah Chen.

This is why teams debugging bad RAG results should ask: are failures due to missing chunks, or missing connections between chunks?

When NOT to Use Graph RAG

Graph RAG is the wrong tool when:

• Your corpus is unstructured FAQs with no entity overlap
• Questions are single-hop ("What is our return window?")
• You cannot maintain entity extraction quality at your document velocity
• Latency budget is under 500ms end-to-end

For those cases, invest in chunking, reranking, and retrieval evaluation before adding graph infrastructure.

Neo4j Knowledge Graph Architecture for LLMs

Short answer: A production Neo4j graph for LLM retrieval layers document nodes, entity nodes, relationship edges, and vector indexes on text-bearing nodes for hybrid entry-point search.

Core Schema Pattern

┌─────────────┐     MENTIONS      ┌─────────────┐
│  Document   │──────────────────▶│   Entity    │
│  (chunk)    │                   │  (Person,   │
└─────────────┘                   │   Product)  │
       │                          └──────┬──────┘
       │ has_embedding                  │
       ▼                                  │ RELATES_TO
  [vector index]                          ▼
                                   ┌─────────────┐
                                   │   Entity    │
                                   └─────────────┘

Node types:

• Document — chunked text with embedding vector and source metadata
• Entity — extracted people, organizations, products, policies, incidents
• Concept — optional higher-level groupings (departments, regions)

Relationship types:

• MENTIONS — Document → Entity (with confidence score)
• RELATES_TO — Entity → Entity (typed: MANAGES, OWNS, APPROVED, CAUSED)
• DERIVED_FROM — Entity → Document (provenance for citations)

Neo4j Vector Index Setup

Neo4j 5.11+ supports native vector indexes, enabling hybrid search without a separate vector database:

CREATE VECTOR INDEX document_embeddings IF NOT EXISTS
FOR (d:Document)
ON d.embedding
OPTIONS {indexConfig: {
  `vector.dimensions`: 1536,
  `vector.similarity_function`: 'cosine'
}};

CREATE INDEX entity_name IF NOT EXISTS FOR (e:Entity) ON (e.name);
CREATE INDEX entity_type IF NOT EXISTS FOR (e:Entity) ON (e.type);

For teams already running Postgres with pgvector, Neo4j can hold the graph while vectors live elsewhere — but co-locating vectors in Neo4j simplifies the backend architecture.

Cypher as LLM Tool

The LLM does not query Neo4j directly in production. Instead:

1. Retrieval planner — LLM generates a structured retrieval plan (entities to find, hops to traverse)
2. Query executor — Your API layer validates and runs parameterized Cypher
3. Context assembler — Formats graph results into LLM-ready context with citations

Never pass raw LLM output to session.run() without validation — Cypher injection is real. Use allowlisted relationship types and parameterized queries.

Building the Graph: Entity Extraction Pipeline

Short answer: Entity extraction converts unstructured documents into graph nodes and edges — the highest-risk step in Graph RAG because extraction errors propagate through every retrieval.

Extraction Approaches

LLM Entity Extraction with Structured Output

from pydantic import BaseModel, Field
from openai import OpenAI
from typing import Literal

client = OpenAI()

class Entity(BaseModel):
    name: str
    type: Literal["Person", "Organization", "Product", "Policy", "Incident", "Location"]
    confidence: float = Field(ge=0.0, le=1.0)

class Relationship(BaseModel):
    source: str
    target: str
    type: str
    confidence: float = Field(ge=0.0, le=1.0)

class ExtractionResult(BaseModel):
    entities: list[Entity]
    relationships: list[Relationship]

EXTRACTION_PROMPT = """Extract entities and relationships from this document chunk.
Only extract explicitly stated facts. Do not infer relationships not supported by the text.
Return JSON matching the schema."""

def extract_graph_elements(chunk_text: str, doc_id: str) -> ExtractionResult:
    response = client.beta.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": EXTRACTION_PROMPT},
            {"role": "user", "content": chunk_text},
        ],
        response_format=ExtractionResult,
    )
    return response.choices[0].message.parsed

Entity Resolution (Deduplication)

"Sarah Chen", "S. Chen", and "Sarah Chen (VP Engineering)" must merge into one node:

from difflib import SequenceMatcher

def resolve_entity(name: str, entity_type: str, existing_entities: dict) -> str:
    """Return canonical entity ID, merging near-duplicates."""
    key = f"{entity_type}:{name.lower().strip()}"

    for existing_key, entity_id in existing_entities.items():
        existing_name = existing_key.split(":", 1)[1]
        if entity_type in existing_key:
            ratio = SequenceMatcher(None, name.lower(), existing_name).ratio()
            if ratio > 0.85:
                return entity_id

    entity_id = f"ent_{len(existing_entities)}"
    existing_entities[key] = entity_id
    return entity_id

Production systems add embedding-based entity linking for fuzzy matches. Log merge decisions — incorrect merges create false graph paths that poison retrieval.

Incremental Graph Updates

Documents change. Your pipeline must:

1. Hash chunk content — skip unchanged chunks
2. On update: delete old MENTIONS edges for that document, re-extract, re-link
3. On delete: cascade-remove document node and orphan-check entities

Deploy extraction as an async worker on your cloud infrastructure, triggered by document change events from your CMS or object store.

Hybrid Retrieval: Vector Search + Graph Traversal

Short answer: Hybrid Graph RAG uses vector search to find relevant entry-point nodes, then graph traversal to expand context along relationship paths before sending results to the LLM.

The Three-Stage Retrieval Pattern

Stage 1 — Vector entry points: Embed the user query, find top-k similar Document or Entity nodes.

Stage 2 — Graph expansion: From entry points, traverse 1-3 hops along typed edges, collecting connected entities and their source documents.

Stage 3 — Context ranking: Score expanded nodes by relevance, deduplicate, truncate to context budget.

from neo4j import GraphDatabase
from openai import OpenAI

class GraphRAGRetriever:
    def __init__(self, neo4j_uri: str, neo4j_auth: tuple, openai_client: OpenAI):
        self.driver = GraphDatabase.driver(neo4j_uri, auth=neo4j_auth)
        self.client = openai_client

    def embed_query(self, query: str) -> list[float]:
        response = self.client.embeddings.create(
            input=query,
            model="text-embedding-3-small",
        )
        return response.data[0].embedding

    def vector_search(self, embedding: list[float], top_k: int = 5) -> list[str]:
        cypher = """
        CALL db.index.vector.queryNodes('document_embeddings', $top_k, $embedding)
        YIELD node, score
        RETURN node.id AS doc_id, score
        ORDER BY score DESC
        """
        with self.driver.session() as session:
            result = session.run(cypher, embedding=embedding, top_k=top_k)
            return [record["doc_id"] for record in result]

    def expand_graph(self, doc_ids: list[str], max_hops: int = 2) -> list[dict]:
        cypher = """
        MATCH (d:Document)-[:MENTIONS]->(e:Entity)
        WHERE d.id IN $doc_ids
        CALL apoc.path.subgraphAll(e, {
            maxLevel: $max_hops,
            relationshipFilter: 'RELATES_TO>|MENTIONS<'
        })
        YIELD nodes, relationships
        UNWIND nodes AS n
        WITH DISTINCT n
        WHERE n:Document OR n:Entity
        OPTIONAL MATCH (n)-[:DERIVED_FROM]->(src:Document)
        RETURN n.id AS id,
               labels(n)[0] AS type,
               coalesce(n.text, n.name) AS content,
               n.source_uri AS source
        LIMIT 50
        """
        with self.driver.session() as session:
            result = session.run(cypher, doc_ids=doc_ids, max_hops=max_hops)
            return [dict(record) for record in result]

    def retrieve(self, query: str, top_k: int = 5, max_hops: int = 2) -> list[dict]:
        embedding = self.embed_query(query)
        entry_docs = self.vector_search(embedding, top_k)
        expanded = self.expand_graph(entry_docs, max_hops)
        return expanded

LLM Context Assembly

Format graph results with explicit provenance so the LLM can cite sources:

def format_graph_context(nodes: list[dict]) -> str:
    sections = []
    for node in nodes:
        if node["type"] == "Document":
            sections.append(f"[Doc: {node['id']}]\n{node['content']}")
        elif node["type"] == "Entity":
            sections.append(f"[Entity: {node['id']} — {node['content']}]")
    return "\n\n---\n\n".join(sections)

Pair with constrained generation so answers include source node IDs. This reduces hallucination on graph-derived facts.

Query Planning for Complex Questions

For questions requiring explicit multi-hop reasoning, add a planning step:

PLAN_PROMPT = """Given this user question, identify:
1. Key entities to look up
2. Relationship types to traverse
3. Expected hop count

Question: {query}

Return JSON: {{"entities": [...], "relationships": [...], "max_hops": N}}"""

The planner output parameterizes Cypher templates — safer than free-form query generation. Log plans and results for retrieval evaluation.

Production Graph RAG Pipeline in Python

Short answer: A production Graph RAG system runs ingestion, extraction, indexing, retrieval, and generation as separate services with observability at each stage.

End-to-End Architecture

Document Store → Chunker → Entity Extractor → Neo4j Upsert
                                                    ↓
User Query → Embed → Vector Search → Graph Expand → Rerank → LLM
                                                    ↓
                                              Response + Citations

FastAPI Retrieval Endpoint

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()
retriever = GraphRAGRetriever(
    neo4j_uri="bolt://localhost:7687",
    neo4j_auth=("neo4j", "password"),
    openai_client=OpenAI(),
)

class QueryRequest(BaseModel):
    query: str
    top_k: int = 5
    max_hops: int = 2

class QueryResponse(BaseModel):
    answer: str
    sources: list[dict]
    retrieval_latency_ms: float

@app.post("/graph-rag/query", response_model=QueryResponse)
async def graph_rag_query(request: QueryRequest):
    import time
    start = time.perf_counter()

    context_nodes = retriever.retrieve(
        request.query,
        top_k=request.top_k,
        max_hops=request.max_hops,
    )
    context = format_graph_context(context_nodes)

    response = OpenAI().chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer using ONLY the provided graph context. Cite document IDs."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {request.query}"},
        ],
    )

    latency = (time.perf_counter() - start) * 1000
    return QueryResponse(
        answer=response.choices[0].message.content,
        sources=[{"id": n["id"], "source": n.get("source")} for n in context_nodes if n["type"] == "Document"],
        retrieval_latency_ms=latency,
    )

Deploy behind your existing backend API with rate limiting, auth, and request tracing. Store retrieval logs for offline eval.

Ingestion Worker

def ingest_document(doc_id: str, text: str, source_uri: str, retriever: GraphRAGRetriever):
    from langchain_text_splitters import RecursiveCharacterTextSplitter

    splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
    chunks = splitter.split_text(text)

    for i, chunk in enumerate(chunks):
        chunk_id = f"{doc_id}_chunk_{i}"
        embedding = retriever.embed_query(chunk)
        extraction = extract_graph_elements(chunk, chunk_id)

        upsert_chunk_and_entities(
            driver=retriever.driver,
            chunk_id=chunk_id,
            text=chunk,
            embedding=embedding,
            source_uri=source_uri,
            extraction=extraction,
        )

Run ingestion asynchronously — entity extraction is the bottleneck (200-500ms per chunk with LLM extraction).

Observability Checklist

Track these metrics in production:

Wire metrics into your observability stack alongside standard LLM evaluation dashboards.

Performance, Cost, and Operational Tradeoffs

Short answer: Graph RAG costs 2-4x more to build and operate than flat RAG, but delivers 20-40% accuracy gains on relationship-heavy query sets — measure ROI on your specific question distribution before committing.

Cost Breakdown

Neo4j AuraDB managed instances start around $65/month for small workloads; self-hosted on Kubernetes scales with your DevOps maturity.

Scaling Considerations

Graph size: Neo4j handles billions of relationships, but LLM retrieval typically traverses subgraphs of 10-100 nodes. Index entity names and types aggressively.

Extraction throughput: Batch extraction overnight for large backfills; stream-process incremental updates. Consider LLM routing to run extraction on cheaper models with spot-check verification on GPT-4o.

Stale graphs: If documents update faster than extraction runs, users get answers from outdated relationships. Target extraction lag under 5 minutes for operational docs.

Graph RAG vs Alternatives

Graph RAG and ColBERT solve different problems — ColBERT improves token-level matching; graphs encode explicit relationships. Some production systems use both.

Security Notes

Knowledge graphs concentrate sensitive relationships — org charts, approval chains, incident root causes. Apply:

• Row-level security on Neo4j nodes via tenant labels
• Audit logging on Cypher queries
• Prompt injection defenses on the retrieval planner

Never expose Neo4j Bolt directly to client applications.

Primary references: official documentation, official documentation, official documentation, official documentation.

Frequently Asked Questions

What is Graph RAG?

Graph RAG is a retrieval pattern that stores document entities and their relationships in a knowledge graph, then uses graph traversal combined with vector search to retrieve context for LLM generation. It excels at multi-hop questions that flat embedding search cannot connect.

Why use Neo4j for Graph RAG?

Neo4j is the most mature property graph database with native vector index support (5.11+), Cypher query language, and APOC procedures for subgraph traversal. Its ecosystem integrates cleanly with Python LLM stacks and production API layers.

How is Graph RAG different from standard RAG?

Standard RAG retrieves semantically similar text chunks via vector search. Graph RAG additionally follows explicit entity relationships — enabling answers that require connecting facts across multiple documents.

Do I need Graph RAG if I have good embeddings?

If your evaluation set shows high recall@5 on single-hop questions but low accuracy on multi-hop questions, embeddings alone will not fix it — you need structural retrieval. Run a retrieval eval segmented by query type before investing in graphs.

How accurate is LLM entity extraction?

Expect 85-95% precision with GPT-4o-class models on well-formatted business documents. Accuracy drops on noisy PDFs, scanned OCR, and ambiguous pronouns. Always add human review for high-stakes domains and log extraction confidence scores.

Can I combine Graph RAG with reranking?

Yes — retrieve graph-expanded candidates, then rerank with a cross-encoder before LLM generation. This is the highest-quality Graph RAG pattern for production.

How much does Graph RAG slow down queries?

Expect 800ms-3s end-to-end versus 200-800ms for flat RAG. Vector entry search adds ~100ms, graph traversal adds 50-200ms, entity extraction at ingest adds 200-500ms per chunk. Cache frequent query patterns to reduce P95 latency.

Should the LLM write Cypher queries directly?

Not in production. LLM-generated Cypher risks injection and unbounded traversals. Use LLMs to produce structured retrieval plans, then execute parameterized Cypher templates with allowlisted relationship types.

Conclusion

Graph RAG with Neo4j solves the multi-hop retrieval problem that flat vector search cannot:

• Extract entities and relationships at ingest time — graph quality equals extraction quality
• Use hybrid retrieval: vector search for entry points, graph traversal for context expansion
• Validate and parameterize all Cypher — never execute raw LLM query output
• Measure accuracy separately for single-hop and multi-hop query sets
• Deploy only when relationship reasoning justifies the infrastructure cost

At HinterBuild, we design Graph RAG architectures for production RAG & LLM systems:

• RAG & LLM Systems
• Backend API Engineering
• Cloud Infrastructure & DevOps
• AI Agent Development

Schedule a consultation to evaluate whether Graph RAG fits your query patterns.