RAG vs Knowledge Graphs vs Structured Folders: How Should AI Access Your Company's Documents?

Three legitimate strategies exist for giving AI access to your company's documents: vector RAG, knowledge graphs, and a structured folder hierarchy the agent navigates directly. RAG retrieves by similarity. Knowledge graphs retrieve by relationship. Structured folders let the agent read the way a person who knows the filing system would. For most SMEs, the third option wins.

What each approach actually is

Three different bets about how AI should find information.

Vector RAG (retrieve-augment-generate) is the default most teams have heard of. You chop documents into chunks, convert each chunk into a numerical vector with an embedding model, store the vectors in a database, and at query time fetch the chunks whose vectors sit closest to the query vector. The AI sees only those chunks. It is similarity search dressed up for language.

Knowledge graphs go the opposite direction. Instead of geometric proximity, you extract entities (people, companies, products, concepts) and the relationships between them, then store the result as a graph you can traverse. Microsoft's GraphRAG, announced February 13, 2024 and open-sourced on GitHub in July 2024, is the version most teams encounter. It uses an LLM to build the graph from your documents up front, then groups related entities into hierarchical communities for retrieval.

Structured folders are the quietest of the three and the one that has crept up on the rest. You organise documents into a well-named hierarchy with predictable filenames, then give the agent file tools: a way to list directories, search for patterns, and read specific files. There is no vector index and no graph. The agent navigates the same way you would, except faster and at scale. Claude Code is the reference implementation: as Vadim Geshel documented, Claude Code does not pre-index your codebase or use embeddings; it explores on demand with Glob, Grep and Read.

How vector RAG actually behaves in production

RAG is well-understood, widely tooled, and quietly brittle.

The chunking problem is the one nobody warns you about loudly enough. As Stack Overflow's December 2024 engineering blog put it, breaking documents into chunks is hard to do well. Boundaries get cut mid-thought, small chunks lose context, large chunks dilute relevance. The chunks that come back to the AI are fragments. The AI then writes confident prose around fragments.

The deeper problem is what embeddings encode. Vector search compresses meaning into geometric distance, which means it finds passages that look similar but cannot follow a chain of reasoning across documents. Ask "what changed between our 2024 and 2025 contracts with this client" and a vector store has to hope both versions sit close in embedding space. They usually don't, because the language is similar but the differences are exactly what you wanted to find.

A 2026 Prism Labs piece named the failure mode well: when a single document is revised, every semantically related embedding silently becomes a candidate for a confidently wrong answer. The index does not know which version is current. The model does not know either.

RAG still wins when your corpus is genuinely large (millions of documents), genuinely heterogeneous, and the query pattern is "find me something that mentions X." It is the right tool for haystack problems. It is the wrong tool for "tell me what we actually decided."

What knowledge graphs do that RAG can't

Knowledge graphs answer questions that span the corpus.

Microsoft built GraphRAG because vanilla RAG fails on what the original blog called "narrative private data": questions where the answer is not in any single passage but in how passages connect. Their canonical example is asking what the top themes are across a dataset. Vector search retrieves the most similar passages to the word "themes," which is useless. GraphRAG retrieves the community summaries it built during indexing, which is the actual answer.

The cost structure is the part teams underestimate. GraphRAG's indexing pipeline runs an LLM over every document to extract entities and relationships. Microsoft's own community blog explains that LLM-powered entity extraction consumes roughly 58% of total indexing tokens. Independent analyses report GraphRAG indexing costs running three to five times higher than vector indexing, and at query time one comparison clocked a single GraphRAG retrieval at 610,000 tokens versus under 100 for a lightweight variant. Microsoft's LazyGraphRAG, released January 2025, brings indexing cost down to roughly 0.1% of the original by deferring community summarisation until query time. The cost gap was real enough that Microsoft itself had to fix it.

Knowledge graphs win when relationships are the answer. Compliance investigations, fraud detection, multi-hop research across a literature corpus, anything where "who is connected to whom through what" is the question being asked. They lose, badly, on cost and complexity, when your real question is "what's in this PDF."

Why structured folders are the third option SMEs should consider first

For most companies under 200 people, a well-organised folder beats both.

The structured-folder approach treats your documents the way Claude Code treats a codebase. The agent has Glob (find files by name pattern), Grep (search inside files), and Read (open a specific file). It uses Glob and Grep to find the right files, then Reads only those. This is two-step retrieval: locate, then load. Vadim Geshel's analysis of Claude Code's design called this "the confirm step, not the discovery tool" approach to file reading, and it is why context windows stay small even on large repositories.

Anthropic's September 2025 context engineering post made the broader case explicit. The discipline is finding the smallest set of high-signal tokens that maximise the likelihood of the desired outcome. As context grows, "context rot" sets in: the model's ability to recall accurately drops before you ever hit the hard limit. A vector store dumps the top-k chunks regardless. A folder navigator only reads what it actually needs.

LlamaIndex captured the same shift in their January 2026 piece "Files Are All You Need," arguing that agents with simple file search tools can interleave search and read operations the way a human scans across files or scrolls inside one to solve a task. Simon Willison's February 2026 write-up on file-native agentic systems pushed the same direction, noting that filesystem context gives frontier models a measurable performance lift across thousands of SQL-generation experiments.

What this looks like in practice for an SME: your documents already live in folders. You rename them so a human can guess what's inside (2025-Q3-contract-acme-renewal.pdf, not final_final_v2.pdf). You write a one-page index at the root explaining what each folder holds. You give the agent file tools, either through Claude Code, Claude with the filesystem MCP, or a similar setup. That is the whole infrastructure.

When each one wins

The choice is rarely about smartness. It is about scale, shape, and budget.

Vector RAG fits when the corpus is genuinely large (think hundreds of thousands of documents and up), the query pattern is keyword-and-similarity ("find me passages about X"), and the documents are stable enough that re-indexing is rare. Public knowledge bases, customer support archives at large companies, legal discovery corpora. A 2025 study published as LaRA, with 2,326 test cases across four QA tasks, found no one-size-fits-all answer between RAG and long-context approaches: the right choice depends on model size, task type, and corpus shape.

Knowledge graphs fit when relationships across the corpus are the actual answer: fraud networks, regulatory exposure across business units, research synthesis where the value is connecting papers rather than finding one. Budget for the indexing pipeline, the ongoing graph maintenance, and the engineering team to run both.

Structured folders fit a surprising range of cases the other two over-engineer: any document set small enough that a human could navigate it (most SMEs, this is you), evolving documents where staying current matters more than scale, situations where the agent needs the actual file rather than a fragment, and any team that wants to ship something this quarter rather than next year. The cost is the discipline of naming things well. The payoff is that the audit trail is visible: you can see exactly which files the agent opened.

In our work with Danish SMEs at Ellypsis, the default starting point is structured folders. We move to RAG only when document volume genuinely exceeds what folder navigation handles, and we have not yet hit a client where a knowledge graph was the right first build. The teams that go straight to RAG usually spend three months on infrastructure to solve a problem that disciplined folder naming solves in a week.

If you want to understand the connectivity layer that makes any of these access patterns possible, start with what MCP is and why it matters for your business.