$cat graph-memory-for-ai-agents.md

Graph Memory for AI Agents: It's the Index, Not the Files

November 4, 202613 min readby MDflowview as .md

A stack of blank glowing emerald glass sheets on the left feeding two conduits: the upper one ending in a scattered cloud of disconnected points, the lower one entering a connected lattice of nodes and edges at a brightly lit junction, on a dark terminal grid background

Every agent framework has converged on the same answer for memory: write markdown files. AGENTS.md and CLAUDE.md for standing instructions, skill files for procedures, daily notes for what happened. It is readable, diffable, portable, and it works — right up until it doesn't.

Stephen Chin, who runs developer relations at Neo4j, opened the graph track at AI Engineer in July with a mascot called Crabby D and a blunt diagnosis of why. The talk is framed as a pitch for graphs, and it is. But the detail that makes it worth taking seriously is the one that undercuts the easy reading: both halves of his comparison were built from the same markdown files.

TL;DR — Graph memory for AI agents replaces a flat pile of re-read text with nodes, edges and traversal. The ceiling it fixes is not the file format — Chin's own A/B test built a vector store and a graph store from one identical markdown corpus, and the graph answered precisely where the vector store told him to go check the config himself. What breaks at scale is using your files as the retrieval mechanism, so the agent loads everything every turn. Keep markdown as the source of truth, which is what a workspace like MDflow is for, and treat the index over it as a separate, replaceable layer.

What is graph memory for AI agents?

Graph memory stores what an agent knows as entities and the named relationships between them, so the agent reaches an answer by traversing a path rather than by re-reading documents.

A node is a thing — a host, a person, a service, a decision. An edge is a specific, typed relationship: runs, reports to, is exposed through, supersedes. Properties hang off both. Crucially, embeddings can live on the nodes too, which is why the production pattern is not graph instead of vectors but both: a vector search to find where to start, then a traversal outward from there.

Chin's architecture on stage was exactly that hybrid, and he described it plainly: a vector search gets the seed nodes where the traversal starts, then a graph search pulls the nearest neighbours and ranks them by how related they are. In the demo's visualisation the seeds are blue, the one-hop neighbours grey, and the handful that actually won their way into the model's context are green.

That last detail is the part worth stealing even if you never touch a graph database. You can see which subgraph produced the answer. Memory becomes auditable — a property that provenance for AI agent memory is otherwise very hard to retrofit.

Why file-based memory hits a ceiling

The criticism is not that markdown is a bad format. It is that files are a bad index.

Chin's own numbers, offered as his observation rather than a benchmark: his agents load at least 100k tokens of context on every round. Not because 100k tokens are needed, but because the agent cannot know in advance which part will be, so it loads everything in the hope that something will be useful. He named three ways this bites:

The re-read tax. The same files are pulled into context turn after turn. At small scale with a strong model, you get the answer anyway and the waste is invisible. At large scale it is the whole bill.
Memory that resets. His framing was an agent that wakes up on a fresh set of memory files and remembers nothing from yesterday — a familiar failure if you have watched a long-running agent roll over to new daily notes.
Skill-selection failure. When procedures are files, the agent has to pick the right file, and then the right chain of files. A correct procedure that never gets loaded is indistinguishable from one that doesn't exist.

None of these are fixed by better prose in the files. They are properties of flat retrieval, which is the same conclusion reached from a different direction in search is not memory.

Same corpus, two indexes: what the demo actually showed

This is the experiment worth reporting carefully, because it isolates the variable.

Chin built a digital twin of his home lab — a handful of Proxmox servers — on a segmented VLAN, so the agent could only answer from memory and could not look up live host state. From one set of original markdown files he built two stores: a vector store (he cites pgvector, which ships with some agent setups out of the box, and LanceDB) and a graph store on cognee with a Neo4j backend. Then he asked both the same security questions.

"What is exposed to the WAN running end-of-life software?" The vector side came back unable to find specific details, suggesting he source the information separately. The graph side ran Cypher queries, traversed, and named the host — a Minecraft server with an out-of-date OS — and flagged it.

"What management ports are bound to 0.0.0.0?" The vector side told him to go check the services configuration and expect a pfSense rule. The graph side identified HAProxy and OpenVPN as the two genuinely WAN-exposed services, having found the router node and followed its relationships directly.

The honest caveats, because they matter: this is a vendor demo, two questions shown live, one person's home lab, and the speaker is a co-author of the GraphRAG book it promotes. No accuracy numbers were presented and you should not infer any.

What survives all of that is the thing the design controlled for. The data was identical. Only the index differed. The vector store's failures were not failures of the underlying notes — the facts were in there, in markdown, in both cases. They were failures of similarity search to assemble a chain of relationships. Chin's phrasing for the alternative is the line to remember: have the agent traverse, not re-read.

Why this matters — for developers and for AI agents

For developers, this reframes a decision that usually gets made by accident. "What format is our agent memory in?" and "how does our agent find things?" look like one question when both answers are "markdown files in a folder." Separating them is what lets you change the second without migrating the first. Chin also points out the practical on-ramp: you do not need to be a graph expert, because current models write Cypher and build entity extractors competently — what you need is a clear model of what you are trying to represent.

For AI agents, the gain is precision and the ability to answer at all. A ranked list of passages that merely resemble the question produces confident, wrong answers — Chin's example is pulling back a fact that is related in some way but is not the one you needed. Multi-hop reasoning chains don't work on similarity search, and they are expensive on relational databases, which is the gap graphs are actually built for.

Which applications benefit most

Infrastructure and asset inventories. Any question of the form "what depends on what, and which of those is a problem" is a traversal, not a lookup.
Enterprise records at scale. Chin's own boundary condition: when the corpus no longer fits in a one-million-token window, files stop being a viable index. A data centre, not a home lab.
Customer and entity networks. Financial services and anything else where the relationships between parties are the data.
Long-running personal assistants. The daily-notes reset problem compounds precisely in the use case the memory is for.
Audit-sensitive work. Where "which facts produced this answer" has to be answerable afterwards.

Conversely: a few hundred documents of prose about a few dozen topics, answered one document at a time, does not need any of this. The index you already have — folders and titles — is adequate, and a graph build is overhead you will maintain for nothing.

The failure mode nobody budgets for

One aside in the talk deserves more attention than it got. Chin notes that goose — Block's agent framework, now hosted by the Agentic AI Foundation under the Linux Foundation alongside MCP and AGENTS.md — treats memory as just another MCP server, with commands to remember, retrieve and forget.

That is genuinely good design. It is also, as he puts it, one step away from calling the forget command and wiping out your own memory.

This is the under-discussed risk of agent-writable memory: not that the agent stores something wrong, but that it destroys something right, in a single tool call, with no second actor in the loop. If memory is a tool surface, deletion is a tool call, and tool selection is probabilistic.

How MDflow fits

What already lines up today

MDflow is markdown-native: documents are plain markdown, stored as markdown, retrievable as markdown over MCP and the HTTP API, and served as raw .md twins with YAML frontmatter when shared. That makes it the substrate half of this architecture rather than the index half — and it is deliberately the half that outlives whatever you index it with. Chin's two stores were both projections of markdown files; MDflow is where those files can live without being trapped in one vendor's schema.

On the index question, MDflow occupies a specific and modest point on the spectrum: a curated structural index, hand-written rather than derived. Every folder carries a description that states what the documents inside it are for, folders nest, and mdflow_get_context ranks those descriptions above folder names and document titles before returning bodies. Reading a document returns its full path and its compounded_description — the workspace→folder cascade of context above it. That is a containment tree with human-authored semantics on every node, which is enough to stop an agent loading the whole corpus, and is explored further in folder descriptions as agent context.

What it is not, and this is the honest boundary: MDflow has no graph store, no embeddings, and no entity traversal. It will not answer "which host runs the service exposed through this router." For that class of question you would build a graph from an MDflow workspace — which is exactly the shape of Chin's pipeline, and why keeping the source of truth in portable markdown over an API matters more than which index you pick first.

On the forget problem, MDflow's answers are concrete rather than aspirational. A body replacement refuses to empty a document unless the caller explicitly passes confirmEmpty: true — enforced identically in the MCP server, the REST API and the web editor, so a client with an unloaded buffer cannot silently wipe a document. And version history, on Pro accounts, is deliberately not exposed through the API or MCP at all: an agent can overwrite a document, but it cannot reach the history, which means a human can always roll back what an agent destroyed.

Where we are headed — direction, not a dated commitment

The direction is better derived indexes over the same markdown, not a different storage format. Richer structural signals an agent can navigate without loading everything; retrieval that improves without asking anyone to re-file their documents. Embedding- or graph-backed retrieval is on the map as a layer above markdown, never as a replacement for it. Treat all of that as direction.

The bottom line

Graph memory for AI agents is a real fix for a real ceiling, and the ceiling is lower than most teams realise — it arrives when questions start needing chains of relationships rather than passages of prose. But the talk arguing for graphs quietly proved the more durable point: both of its stores were built from the same markdown files. The format was never the bottleneck. The index was.

So keep the substrate boring, portable and human-readable, and let the index be the thing you replace when you outgrow it. Start free, connect an AI agent, or read the API docs.

Frequently asked questions

What is graph memory for an AI agent?

Graph memory stores what an agent knows as nodes and edges — entities and the named relationships between them — instead of as a flat pile of text to be re-read. The agent reaches an answer by traversing from a starting point through those relationships, rather than by loading whole files and hoping the relevant part is in there. In practice most production setups are hybrid: a vector search finds the seed nodes, then a graph traversal pulls their neighbours and ranks them by how related they are.

Are markdown files bad as AI agent memory?

No, and the strongest evidence for that is in the talk making the criticism. Stephen Chin of Neo4j built both halves of his A/B demo — the vector store and the graph store — from the same original markdown files. Markdown is a good substrate: durable, diffable, human-readable, portable between tools. What breaks at scale is using the files as the retrieval mechanism too, so the agent loads everything every turn. The fix is to add an index over the files, not to stop writing files.

When do markdown memory files actually stop scaling?

When answering a question needs a chain of relationships rather than a passage of prose, and when the corpus no longer fits comfortably in a context window you are willing to pay for every turn. Chin's rule of thumb on stage was the size of the problem relative to a one-million-token window: a few documents about a few topics are fine as files, while a data centre inventory, a customer-and-entity graph in financial services, or any multi-hop question across thousands of related records is not.

Why can't vector search answer multi-hop questions?

Because similarity in vector space is not the same thing as a relationship. A vector index can tell you which chunks look like your question, but it has no representation of the fact that this host runs that service which is exposed through that router. A question whose answer requires following two or three such hops cannot be assembled from a ranked list of lookalike passages, which is why Chin's vector store answered one security question with a suggestion that he go check the configuration himself.

Is it risky to expose agent memory as an MCP tool?

It introduces a failure mode worth designing against: if remembering, retrieving and forgetting are all tool calls, the agent is one bad tool choice away from deleting its own memory. The mitigations are ordinary engineering rather than anything exotic — make destructive writes require explicit confirmation instead of being a single silent call, and keep a version history the agent cannot reach, so a human can roll back what an agent destroyed.