skip to content
$mdflow
$cat model-agnostic-ai-architecture.md

Model-Agnostic AI Architecture: Why Optionality Wins

14 min readby MDflowview as .md
An emerald wireframe document core wired to a ring of interchangeable modular sockets, one lifted mid-swap, on a dark terminal-grid background

Every AI product team has now lived the same month twice. A model gets upgraded, the per-token price is unchanged, and your bill triples because the new reasoning mode spends three times the output tokens. Or a new generation lands at 40% more than its predecessor — which is being deprecated in four months, so it is not really a choice. Meanwhile your revenue did not grow 40%, and someone has to eat the difference: your customers or your investors.

That framing comes from a talk at AI Engineer by Sarah Sachs, who leads the AI engineering teams at Notion and negotiates model contracts for a living. She called it Token Town — how to go from AI-rich to AI-poor — and her conclusion is the most useful architectural principle in applied AI right now: the ability to walk away is the product decision.

TL;DR — Model-agnostic AI architecture treats the model as a swappable part, not a structural dependency. It matters because price does not track capability, and both change monthly — so if you cannot switch, you absorb every repricing your supplier decides on. The playbook: build the harness for multiple models, measure cost per capability per second rather than per token, use open-weight models for the large middle of your traffic, and stop paying an LLM to do work a CPU can do. The lock-in almost nobody prices is the context layer — optionality is worthless if switching models means abandoning what the last one learned. Keep it in plain markdown you own, readable over open protocols. That is what MDflow is for. Start free.

What is model-agnostic AI architecture?

Model-agnostic AI architecture is a system design where the language model is a component you can replace without rewriting the system around it. Prompts, tool schemas, retrieval, evaluations, guardrails, and stored context all live in your layer; the model plugs into them. Swapping providers is a configuration change and an eval run, not a quarter of engineering work.

The honest test is a stopwatch question. If your primary provider raised prices 40% tomorrow, how many days until you are running production on someone else? If the answer is months, you are not model-agnostic — you are a tenant, and your landlord sets the rent.

Notion's version of this is visible in the product. Its AI runs on an auto-router that picks the model per task, alongside a menu that in the 3.6 release spans Opus 4.8, Grok 4.3, and GLM 5.2 — three labs, one of them open-weight — and external coding agents (Claude and Cursor) that users orchestrate side by side. Sachs calls the posture AI Switzerland: state-of-the-art models always available, none of them load-bearing. On stage she said the auto-router handles roughly three quarters of their traffic.

Why lock-in is the expensive decision

Because your supplier is also your competitor, and price does not correlate with capability.

The supplier point is structural, not cynical. When you resell tokens from a lab that also sells a first-party product to your customers, you are buying at a markup and reselling at a markup, competing against the vendor's own direct offering with a worse cost of goods sold. That is not a margin you can defend on token economics. You defend it on product: understanding your customers' workloads better than a general-purpose chat interface can.

The pricing point is subtler and more important. Frontier pricing behaves like the classic gas-station game — if you are second best, you only have to be marginally cheaper than the leader to take the rest of the market. So a model's price encodes its position relative to the leader, not its absolute capability. And leadership rotates: the best model for a given task changes provider on a scale of months. A company that has marketed itself as one lab's flagship customer, or traded optionality for a volume discount, is therefore shipping a non-frontier product a large fraction of the time — while paying for the privilege of not being able to leave.

This is why optionality is leverage rather than merely hygiene. Very few companies have the volume to negotiate on scale. What almost any company can have is a credible ability to move, which is the only thing that makes a price negotiable at all.

The model-agnostic playbook

Four moves, in rough order of payoff.

1. Build the harness for multiple models. Not necessarily per-thread switching — killing a warm cache mid-conversation is genuinely hard — but the harness itself should treat the model as a parameter. Prompts and tool definitions that assume one vendor's quirks are the real switching cost, and they accumulate silently.

2. Measure cost per capability per second, over whole trajectories. A single-call latency or per-token comparison will tell you the wrong thing. Notion's evaluation of web-search providers is a good example: on one call, its chosen provider was not the cheapest, but across complete search trajectories the economics inverted. The granularity of your evals is what licenses you to make the trade-off at all — and nobody outside your company can build that expertise for you.

3. Use open-weight models for the middle of the distribution. Open-weight models from Chinese labs — GLM, Kimi, MiniMax — have tracked the closed frontier at a roughly three-to-six-month lag for over a year and a half, at a fraction of the cost per task. The claim is not that they win benchmarks. It is twofold: most of your traffic does not need a frontier model, and a credible alternative is the only downward pressure on price available in what is effectively an oligopoly at the top. Corollary worth internalising: if a task is just barely out of reach for open weights today, it will likely be in reach in six months. Build as if it will be.

4. Stop paying an LLM to do work a CPU can do. You do not need a model to turn a CSV into a PDF, to run a deterministic SQL query, or to make a tool call a CLI can make. This is where teams become token-poor fastest, because the spend is invisible — it looks like "agent usage" rather than a line item. It also happens to improve determinism, which is the other thing your customers want.

There is a fifth, quieter move: send fewer tokens in the first place. Most retrieval systems are wasteful by construction — they hedge by stuffing the window, because nobody wrote down what is actually relevant. Well-organised context is a cost lever, not just a quality lever. We have made the curation argument at length elsewhere; the token-economics version is that a workspace where an agent can find the right three documents costs a fraction of one where it has to read thirty.

Why this matters for developers and AI agents

For developers, it changes what you consider technical debt. Coupling to a model shows up as prompts tuned to one family's failure modes, tool schemas shaped by one API, evals that only exist as vibes, and — the worst one — memory in a proprietary store. None of it fails a test. All of it is priced the day you want to leave. Model-agnostic architecture is mostly the discipline of keeping the durable assets (evals, context, tool contracts) in your own layer and the volatile one (the model) behind a boundary.

For AI agents, it is a question of what survives the model. An agent's usefulness is a function of the context it can reach, and models are replaced far more often than knowledge is. If a year of accumulated decisions, specs, and runbooks lives inside one vendor's assistant memory, the switching cost is not an integration — it is amnesia. Agents built on a portable context layer inherit everything the previous model learned, because the learning was never in the model to begin with.

There is also a security dimension that arrives with agentic scale. Simon Willison's lethal trifecta — access to private data, exposure to untrusted content, and the ability to communicate externally — describes exactly the configuration a capable agent stack converges on. The more autonomous the system, the less supervised the risk. Deciding which data an agent can see, and being able to scope it, is architecture, not a setting.

Which applications benefit most

  1. Applied AI products that resell inference — SaaS with AI features, where gross margin is directly exposed to every repricing upstream.
  2. High-volume, low-complexity workloads — inbox triage, classification, extraction, summarisation. Routing these to a frontier model is the single most common way to overpay.
  3. Coding agents and software factories — where multiple agents from different labs are already used side by side, and the orchestration layer has to be neutral by construction.
  4. Enterprise deployments with procurement constraints — where "we can move providers in a week" is a compliance answer, not just an engineering one.
  5. Anything with long-lived memory — assistants, research tools, knowledge bases. The longer the accumulated context, the more expensive lock-in becomes, because the asset being held hostage keeps growing.
  6. Regulated or self-hosted environments — where open-weight models are not a cost optimisation but the only deployable option.

The lock-in nobody prices: your context layer

You can be perfectly model-agnostic at the inference layer and still be completely locked in at the knowledge layer. This is the gap we care about most, because it is the one that compounds.

Ask the stopwatch question again, but about context. If you moved every workload to a different model tomorrow, what would come with you? The prompts, probably. The evals, if you wrote them down. But the year of decisions, specs, meeting notes, runbooks, and corrections that make your agents useful — where does that live? If the answer is "in the assistant's memory", it is not yours in any operational sense. Proprietary memory is the highest-switching-cost component in the stack precisely because it is the one that is supposed to accumulate.

A portable context layer has three properties: it is plain text you own, in a format every model already reads; it is reachable over an open protocol rather than one vendor's SDK; and it is written, not inferred, so it is auditable and immediately available rather than statistical and slow to warm up.

How MDflow fits

MDflow is the model-agnostic layer underneath the model. It stores the knowledge your agents work from as plain markdown you can export, diff, and read without us — and serves it to whichever model you happen to be using this quarter.

What already lines up today

Markdown-native, no wrapper. Documents are stored as plain markdown, and every shared document has a raw .md twin any tool, model, or CI job can fetch — this post included, at /blog/model-agnostic-ai-architecture.md. Markdown is the one format every model family already handles natively, which makes it the cheapest possible interchange layer between a knowledge base and a model you have not chosen yet.

Open protocols, not an SDK. MDflow exposes a remote MCP server — Claude and the ChatGPT app connect with an OAuth sign-in and no pasted token, while Cursor, Codex, and Claude Code authenticate with a Personal Access Token — plus a plain HTTP API with an OpenAPI spec, a VS Code extension, and an n8n node for the parts of the workflow that never needed a model at all. The same workspace is readable from all of them. Switching models does not change a single line of that wiring, which is the whole point of MCP as an interface.

Cheap retrieval, by design. Every folder carries a description stating what its documents are for, and mdflow_get_context ranks those descriptions above folder names and document titles — retrieval without a vector database, and without an embedding bill that has to be re-paid every time you change models. Workspaces scope a query to the right part of your work, so an agent reads three relevant documents instead of thirty plausible ones. That is a token-economics feature disguised as an organisation feature.

Agents write back. Because the MCP write tools are real, an agent that learns something — a decision reached, a runbook step that was wrong — can record it in the workspace instead of losing it at the end of the session. Knowledge accumulates in your store, not in a model's memory, which is precisely what makes the next model swap survivable.

Governance and provenance. Version history records every change with an actor, so agent-written content is reviewable rather than a silent update. Sharing, collections, and comments scope what a person, a client, or an agent can see — the practical answer to the private-data leg of the lethal trifecta — and client-side encryption covers documents that should never be readable by anything but you.

Where we are headed

Direction, not a dated commitment: serving a whole collection to an agent as one cross-linked bundle, so a spec, its decision log, and its glossary arrive together rather than as three retrievals; richer typed frontmatter so documents can carry status, owner, and freshness as signals an agent can weigh; and tighter review round-trips for agent-proposed edits before they land.

The bottom line

The market for models is young, fast, and deliberately opaque, and it reprices on a monthly cadence that no product's revenue matches. Model-agnostic AI architecture is how you stop absorbing that volatility — build the harness for multiple models, measure cost per capability per second across whole trajectories, put open weights under the large middle of your traffic, and take the LLM out of the jobs that never needed one.

Then finish the job at the layer below. Optionality at the model is only real if the knowledge underneath it is portable: plain markdown you own, reachable over open protocols, that outlives whichever model is currently the best deal.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What is model-agnostic AI architecture?

Model-agnostic AI architecture is a system design where the language model is a swappable component rather than a structural dependency. Prompts, tool schemas, retrieval, evaluations, and stored context live in your own layer and stay valid when you change models. The practical test is simple: if a provider raised prices 40% tomorrow, how many days would it take you to move? If the answer is months, you are not model-agnostic — you are a tenant.

Why is LLM vendor lock-in expensive?

Because model pricing does not track model capability, and both change on a monthly cadence. A reasoning upgrade can keep the same per-token price while spending three times as many output tokens; a new model generation can cost 40% more than the predecessor it deprecates. If you cannot walk, you absorb every one of those changes. Optionality is the leverage — the ability to credibly move is what makes a price negotiable at all, and most companies never have the scale to negotiate any other way.

Do open-weight models actually reduce AI costs?

For a large share of everyday traffic, yes. Open-weight models from Chinese labs — GLM, Kimi, MiniMax — have tracked the closed frontier at roughly a three-to-six-month lag for over 18 months, at a fraction of the price per task. The point is not that they beat frontier models; it is that most traffic does not need a frontier model, and having a credible alternative applies downward pressure on the prices you pay for the traffic that does.

How do you cut AI token costs without losing quality?

Route by task, not by default. Send the small share of genuinely hard work to the strongest model and everything else to cheaper tiers, measured on cost per capability per second across whole trajectories rather than single calls. Then remove the LLM entirely where it was never needed — CSV conversion, deterministic SQL, tool calls a CLI can make. And curate the context you send: a well-organised markdown workspace answers a question in a few thousand tokens where an unstructured dump takes tens of thousands.

What does model-agnostic mean for where you store context and memory?

It means your knowledge cannot live inside one vendor's proprietary memory. Model optionality is worthless if switching models means abandoning everything the last one learned. A portable context layer is plain text you own, in a format every model already reads, reachable over an open protocol like MCP or a plain HTTP API — so the substrate outlives the model on top of it. That is the design MDflow is built around: markdown documents, folder descriptions as retrieval context, and no proprietary wrapper.

Further reading

← all posts./start-free →