A Semantic Layer for AI Agents: Thin Agents, Smart Substrate
Every team in a large company is building agents right now. Each one starts the same way: figure out where the data lives, work out whether that copy is the right copy, wire it into prompts and code, ship. Then the next team does it again from scratch. Then a schema changes, and every agent has to be rewired by hand.
That is the problem Emil Eifrem, Neo4j's CEO, put on stage at AI Engineer in Thinner Agents on a Smarter Substrate — and the pattern he proposed as the fix has a name worth borrowing even if you never touch a graph database: thin agents on a smarter shared substrate.
TL;DR — A semantic layer for AI agents is a shared, governed layer that says what your concepts mean, where the data behind each one lives, and which sources are trustworthy. Without it every agent hand-wires its own data sources, nothing is reused, and no agent is smarter tomorrow than it is today. Eifrem's blueprint has three pillars: a business-facing ontology, a technical ontology of your actual data sources, and execution traces fed back from agent runs. He is also blunt that markdown files alone will not get you there — and he is right. Markdown is the right home for the meaning half: MDflow makes folder descriptions the primary retrieval signal so the context you write is the context agents get.
What is a semantic layer for AI agents?
A semantic layer for AI agents is a single shared place that maps business intent to data, so agents stop rediscovering the same wiring. It answers three questions that otherwise get re-answered, badly, inside every agent:
- What do we mean by "customer" here?
- Where does the data behind that concept actually live?
- Which of the six copies is the one I should trust, and may I read it?
Eifrem's simplification of an agent is two buckets. There is the business logic — interpret intent, plan, act, loop — and there are the data sources the agent needs to act on. The first bucket is what teams think they are building. The second is where the work actually goes.
His running example is a bank-account-opening agent. To verify identity it needs a government-issued ID, which in that organization resolves to two sources: motor vehicle records and a passport verification service. Wiring those in works fine. The problem starts with the second agent, and the tenth.
The four problems with thick agents
The talk names them precisely, and they are worth stating as a checklist because most teams recognize at least two:
| Problem | What it looks like in practice |
|---|---|
| Discovery from scratch | With one Postgres, data location is trivial. With a hundred databases plus Snowflake, Databricks, and S3 buckets, every team re-does the archaeology. |
| Trust and governance | Enterprises duplicate data. Is this the right copy? The current version? Am I allowed to read it? |
| DRY violations | The mapping from intent to source lives in code and prompts, scattered per agent. One change cascades into manual rewiring everywhere. |
| No learning | "When your agent wakes up tomorrow, it's not smarter than it was today." And there is no cross-agent learning at all, because nothing is shared. |
That last one is the quiet killer. A thick agent's hard-won knowledge — the DMV lookup works, the other path times out — is trapped in one team's prompt, discovered again by the next team next quarter.
The three pillars
Eifrem's blueprint is three layers that together turn hand-wiring into a lookup.
- A business-facing ontology. The key concepts in your organization and how they relate — customers, accounts, debit cards, transactions — expressed the way humans in your company talk about them. His example is deliberately unglamorous: you do not have an
f_name. You have a customer, and they have a first name. Ontologies have a reputation for being complicated; the core of one is not. - A technical ontology. The metadata of every real data asset: the fourteen Oracle databases, the Snowflake instances, the S3 buckets, their schemas, where they sit. Plus the mapping to pillar one — this concept's first name has a system of record, and over there it is a column called
f_name. - Execution traces. The runtime signal agents leave behind as they walk the layer: what I tried, in what context, and whether it worked. Scored and fed back, so an agent that has succeeded ten times with the DMV lookup is more likely to reach for it in a similar context next time.
Taken together, the three pillars address all four problems. Discovery becomes a query. Trust comes from two directions at once — top-down from curated human judgment ("this is the system of record") and bottom-up from traces ("this is what actually worked"). The mapping lives in one governed place, so a change cascades automatically. And the substrate accumulates, which is the only mechanism by which tomorrow's agent is better than today's.
"Just use markdown files" — the honest version
Eifrem addresses the obvious objection head-on, and his answer is the most useful sentence in the talk. Teams have tried to solve exactly this with a pile of markdown files and skills:
It is part of the solution, but it is not the solution.
He backs it with a line from swyx on the Latent Space podcast, roughly: learn your databases — you cannot vibe-code with just markdown files.
This is worth taking seriously rather than defending against, and the split falls in a fairly clean place:
Markdown is genuinely good at pillar one. Concepts, definitions, conventions, what belongs where, why a decision was made, which process to follow. This is prose written by humans for humans, that agents happen to read very well. It is versionable, diffable, greppable, and requires no modelling ceremony to start. Every coding agent already reads it — AGENTS.md, CLAUDE.md, llms.txt, spec files. It is the substrate agents already speak.
Markdown is a bad fit for pillars two and three. A catalog of a hundred databases with live schemas is not a document — it is metadata that changes under you and needs to be queried, joined, and traversed. Execution traces are worse: high-volume, append-only, aggregated into scores. Writing either into flat files means reimplementing a database badly.
So the honest architecture is not markdown vs. graph. It is markdown for meaning, a real store for structure and scale — and a clean seam between them. Which is exactly where most teams get it wrong in the other direction too: they build a beautiful catalog of schemas and no place at all for the human judgment that makes those schemas interpretable.
Why this matters for developers and AI agents
For developers, the payoff is that context stops being copy-paste. The moment two agents need the same knowledge, a shared substrate is the difference between one edit and n edits. It also converts a class of debugging from "read six prompts" to "read one map." And it makes governance expressible: trust, freshness, and access are attributes of a source in a shared layer, not tribal knowledge in a team's Slack.
For agents, a substrate is what makes thinness safe. An agent that can look up what a compliance check needs does not have to carry that knowledge in its system prompt — which means the prompt stays small, the agent stays cheap, and the same agent works when the underlying source changes. This is context engineering applied one level up: instead of optimizing what you stuff into a window, you optimize what the agent can reliably fetch at the moment it needs it.
There is a real caution here too. A shared substrate is a shared blast radius. If the meaning layer is wrong, every thin agent is confidently wrong in the same direction — which raises the value of provenance and version history on the layer itself.
Which applications benefit most
- Enterprises with many agents and many data sources — the talk's own scope. The pattern only pays off when there is duplication to eliminate; one app on one Postgres does not need it.
- Process-following agents — onboarding, KYC, claims, procurement. Eifrem encodes the business process in the ontology too, so the agent follows a path rather than improvising one.
- Regulated domains — banking, health, insurance, where "which source, and am I allowed" is not a nice-to-have.
- Data and analytics agents — text-to-SQL and BI copilots fail on ambiguity about what a metric means far more often than on SQL syntax.
- Engineering organizations — the small-scale version most readers actually have: specs, ADRs, runbooks, and API contracts that every coding agent should read the same way.
How MDflow fits
MDflow is a home for pillar one — the human-facing meaning layer — for the knowledge your organization writes down rather than stores in a database. It does not pretend to be pillars two and three, and it should not.
What already lines up today
Folder descriptions are a semantic layer you author in prose. In MDflow every folder carries a description stating what belongs inside it and what the documents there are for. This is not decoration: the mdflow_get_context tool over MCP ranks folder descriptions as its primary context signal, ahead of folder names and document titles, then returns the best-matching markdown bodies. The meaning you write is the meaning that steers retrieval — no vector database, no embedding pipeline, and nothing you cannot inspect and edit. That is why folder descriptions are the feature that matters most for agents.
Workspaces scope the layer. Each workspace has its own name and description and buckets folders per client, product line, or domain — so an agent can be pointed at one coherent slice instead of ranking across everything you have ever written.
One substrate, many clients. The same store is reachable from Claude, ChatGPT, Cursor, Codex, and Claude Code over the remote MCP server (OAuth sign-in or a Personal Access Token) and from anything else over the REST API. That is the "shared" in shared substrate: the same folder descriptions and the same documents, not one copy per tool.
Machine-readable by default. Documents are plain markdown, and every shared page has a raw .md twin that opens with YAML frontmatter — title, canonical URL, visibility — so a tool can read and cite it in one request. Discovery is published as llms.txt, docs.md, an OpenAPI spec, and an A2A agent card. Agents find the surface without a human wiring it in.
Governance and history on the layer itself. Shares carry roles, the Document Log names the actor on every write (automated · <token name> for API and MCP writes), and version history captures every change with a line-by-line diff and non-destructive restore. When a shared meaning layer goes wrong, you need to see who changed it and roll it back — that is the whole point.
What MDflow does not do, plainly: there is no technical ontology. MDflow does not catalog your Oracle instances, does not hold schemas, and does not map concepts to columns. There is no execution-trace pillar either — retrieval does not currently learn from which documents agents fetched or whether the answer worked. If your problem is a hundred databases, you want a graph or a catalog, with MDflow as the meaning layer beside it, not instead of it.
Where we are headed
Direction, not a dated commitment. The interesting frontier for a markdown-native store is making the meaning layer richer without making it ceremonial: descriptions that can express relationships between folders, not just membership; retrieval that can be inspected and tuned rather than trusted blindly; and eventually some form of feedback from what agents actually read and write, which is the honest local version of pillar three. Alongside that: scoped tokens, so a thin agent's reach into the substrate can be narrowed to the slice its job needs.
The bottom line
The pattern generalizes past enterprises with a hundred databases. Any time two agents need the same knowledge, you have a choice: duplicate it into both, or put it somewhere both can read. The first is faster today and compounds badly. The second is what makes an agent thin.
Eifrem's caution about markdown is fair, and it cuts both ways. Markdown will not catalog your data estate — but a catalog will not tell an agent what your team means by "done," which client a convention applies to, or why last quarter's decision went the way it did. That half has to be written by humans, and it is best written in plain markdown that both people and every model can read.
Start free · Connect an AI agent · Read the API docs
Frequently asked questions
What is a semantic layer for AI agents?
A semantic layer for AI agents is a shared, governed layer that sits between agents and the systems holding data, and answers three questions in one place: what the concepts in your organization mean, where the data behind each concept actually lives, and which of those sources can be trusted. Agents query the layer instead of each team hard-wiring data sources into prompts and code, so the wiring is written once and reused by every agent.
What does "thin agents on a smarter substrate" mean?
It is a design pattern from Neo4j's Emil Eifrem: keep the agent itself small — interpret intent, plan, act, loop — and move the durable parts (meaning, source-of-truth mappings, and accumulated experience) into a shared substrate every agent reads. The alternative is thick agents, where each one carries its own hand-wired copy of that knowledge, so nothing is shared, nothing is reused, and every change has to be applied in many places.
Can you build a semantic layer out of just markdown files?
Partly. Markdown is very good at the human-facing half — naming concepts, defining what belongs where, writing down the decisions and conventions agents need in order to interpret data correctly. It is not a substitute for a catalog of a hundred enterprise databases or for aggregate queries over live records. Eifrem's own framing is that markdown files are part of the solution but not the whole solution, and that is the honest answer: use markdown for meaning, use a database or graph for structure and scale.
Why do agents need a semantic layer instead of just better retrieval?
Because retrieval answers where similar text is, and agents need to know which source is authoritative. In an enterprise there are usually several copies of the same data at different freshness levels, and semantic similarity cannot tell you which one is the system of record or whether you are allowed to read it. A semantic layer encodes that judgment once, as curated knowledge, instead of leaving each agent to guess.
How does MDflow relate to a semantic layer?
MDflow covers the human-facing, business-ontology half of the pattern for the knowledge an organization writes down. Every folder carries a description that states what belongs inside it, and MDflow's mdflow_get_context tool ranks those descriptions as its primary signal before folder names and document titles, so the meaning you author is what steers retrieval. Workspaces bucket that by client or product line, and the whole store is reachable over MCP and a REST API. MDflow does not catalog your databases and does not learn from agent execution traces — for those, a graph or catalog is the right tool.
Further reading
- Thinner Agents on a Smarter Substrate: The Ontology-based Semantic Layer — Emil Eifrem, Neo4j (AI Engineer, July 2026) — the talk this post is built on.
- Neo4j: agent-ready data and the semantic layer — the vendor's own material on the pattern, including the graph implementation.
- What is a semantic layer for AI agents? — a vendor-neutral-ish overview of the same idea from the data-catalog side.
- MDflow: folder descriptions as agent context, context engineering for AI agents, from systems of record to systems of context, connecting an AI agent over MCP, and the REST API.