Prompt Caching: Why Agents Cost 10x More Without It

A year ago, optimising the cost of an LLM feature meant watching output tokens. Then reasoning models arrived and it meant watching reasoning tokens too. Both of those are now the small numbers on the invoice. For an agent, the line that moves the bill is whether prompt caching is hitting — and most teams have never measured it.
At AI Engineer in New York, George Cameron, co-founder of the independent benchmarking firm Artificial Analysis, put a cost build-up for a single agentic task on screen and walked down it. Answer tokens: cents. Reasoning tokens: cents. One model call: about 13 cents. Steps taken to finish the task: 165. Total: roughly $21.
Then the line that should change how you instrument your agent. Take the cache hit rate to zero — no prompt caching, or your own inference setup where nobody enabled it — and the same task costs over $200.
TL;DR — For agents, the cost equation has moved from output tokens to input tokens × cache hit rate × turns per task. Prompt caching is prefix-based: providers reuse the computed state of the unchanged tokens at the front of your prompt, at 0.1× the input price on Anthropic and up to 95% off on OpenAI. That makes cache hit rate a property of how you assemble context, not a setting you switch on — one changing byte near the front takes it to zero. The practical rule is to put stable, authored context first and volatile context last, which is a job for versioned plain markdown: exactly what MDflow stores, with raw
.mdtwins, folder descriptions as retrieval scope, and access over MCP and an HTTP API.
What prompt caching actually is
Prompt caching lets the provider skip recomputing a prefix of your prompt it has already processed, and charge you a fraction of the input price for those tokens. It is not a response cache: the model still runs, still reasons, still produces fresh output. What is reused is the internal state built from tokens that did not change.
Two things follow from "prefix", and they are the whole post.
First, the discount is large. Anthropic's documentation states that "cache read tokens are 0.1 times the base input tokens price", with newer models cheaper still — 0.05× on Opus 5.5 and Sonnet 5.5, 0.025× on Fable 5.1 and Mythos 5.1. OpenAI describes cached input as "discounted up to 95%", with most GPT-5.6-and-later models reading at 0.1× the ordinary input rate. Ten times cheaper is the conservative end.
Second, the discount is conditional in a way that is easy to lose by accident. OpenAI's guide is unambiguous: "Cache reuse requires the entire rendered prefix to match. If content or a relevant setting changes before a breakpoint, the prefix after that change cannot match the existing cache entry." Anthropic describes the same mechanism as a hierarchy — "cache prefixes are created in the following order: tools, system, then messages" — and spells out the consequence: "changes at each level invalidate that level and all subsequent levels."
The rest of the mechanics are worth knowing before you design around them:
Anthropic write 1.25× base input (5-min TTL), 2× (1-hour TTL)
read 0.1× base input (0.05× on Opus 5.5 / Sonnet 5.5)
TTL 5 minutes by default, refreshed free on each use
min 512 tokens on the current 5.x models
max 4 cache breakpoints per request
OpenAI on by default for supported models
write 1.25× input (30-min default), 2.5× (12-hour)
— GPT-5.6+ only; earlier models: no write charge
read up to 95% off; 0.1× on most GPT-5.6+ models
min 1,024 tokens on GPT-5.6 and later
watch usage.input_tokens_details.cached_tokens
Note the write premium. A cached token you never re-read costs more than a plain one. Caching is a bet that the prefix will come back — which, in an agent loop, it does, 165 times.
Why the cost build-up changed
Because an agent re-sends its entire context on every single turn, and takes a lot of turns. A chatbot answers once. An agent reads a file, calls a tool, reads the result, calls another tool — and each of those steps ships the accumulated history back through the model as input.
Cameron's figures come from AA-Briefcase v1.1, Artificial Analysis's agentic knowledge-work benchmark — "four multi-week knowledge work projects, comprising thousands of input files and 91 tasks in total", producing deliverables like spreadsheets, presentations and memos. It is a realistic shape for the agents people are actually shipping, and it is where the 165 steps and the $21 come from.
His summary of what to instrument is the part to copy down. A year ago the conversation was about output tokens and reasoning verbosity. For agents, the three numbers that drive cost are:
- Input tokens per call — how much context you ship each turn.
- Cache hit percentage — what fraction of those tokens get the 10× discount.
- Turns per task — how many times you pay for the first two.
They multiply. That is why the uncached number is not 30% worse but an order of magnitude worse, and why Artificial Analysis's own cost-per-task methodology splits token usage "across input, cache hit, cache write, reasoning, and answer token prices, including representative cache hit rates" rather than quoting a single input price.
One more lever sits alongside them, and it is the one most teams reach for first: reasoning effort. Cameron showed the same model, Opus 5.5, costing about $1 per task at low effort and about $21 at max. That is a real 20× — but notice it is partly the same mechanism, because reasoning effort changes the number of turns and the tokens carried forward into every later call.
Cache hit rate is a context design decision
This is the claim worth taking away: your cache hit rate is not a provider feature you enable, it is an emergent property of how your harness assembles the prompt. Both providers cache automatically. You can still run at a hit rate near zero without a single error in your logs, because nothing fails when a prefix stops matching — it just gets quietly expensive.
Here is how teams lose it, in rough order of frequency:
A timestamp, session id or "current date" near the top of the system prompt. One variable byte at position 40 invalidates everything behind it, forever, on every request. Anthropic's advice is to "place static content (tool definitions, system instructions, context, examples) at the beginning of your prompt" and put the breakpoint on the last block that stays identical.
A tool registry that is regenerated per request. OpenAI lists tool names, descriptions, schemas and ordering among the things that change the prefix; Anthropic's docs note that modifying tool definitions invalidates the entire cache. If your tool list comes out of a dictionary or a plugin scan, its order can change between processes while the content is identical — a cache miss with no visible cause. (Tool definitions alone can run to six figures of tokens in large deployments, so this one is expensive.)
Retrieval results injected above the conversation. This is the subtle one, and it is a direct conflict between two pieces of advice you have probably both been given. "Retrieve only what is relevant to this turn" minimises input tokens. "Keep the prefix stable" maximises cache hits. If fresh search results land at the front of the prompt, each turn writes a new cache entry and reads none of them.
The resolution is ordering, not choosing a side. Stable context first, volatile context last. A document the task will keep referring to belongs in the prefix, fetched once, in the same bytes, before the turns begin. A per-turn search result belongs after the conversation history, where it invalidates nothing behind it. The question to ask of every block you assemble is not "is this relevant?" but "will this be byte-identical next turn — and if not, is it as late in the prompt as it can be?"
A five-minute wall clock. Anthropic's default TTL is five minutes, "refreshed for no additional cost each time the cached content is used"; OpenAI's newer default is 30 minutes with a 12-hour option. So a stable prefix pays off inside a burst of activity and expires between them. An agent that thinks for ninety seconds between steps is fine. A human-in-the-loop workflow that waits for an approval over lunch pays the write premium again.
What caching does not fix
Caching changes the price of context you had a reason to send. It does not make irrelevant context harmless. This matters because "it's cached, so it's basically free" is a tempting conclusion and a wrong one.
A cached token is still billed — at 0.1×, a 100,000-token prefix across 165 turns is not a rounding error. More importantly, price is not the binding constraint on context size. Model accuracy degrades as inputs grow, non-uniformly and often well below the documented window limit, which is the subject of tool output eating your context window. Caching a pile of marginally relevant material buys you a cheaper way to confuse the model.
The honest framing: trim first, then stabilise what is left, then cache it. In that order.
Which applications benefit most
The payoff from caching scales with turns per task and with the size of the stable prefix. Roughly, in descending order:
- Long-horizon agents — coding agents, research agents, knowledge-work agents of the AA-Briefcase shape. Many turns over one accumulating context; the canonical case.
- Document and codebase question answering — a large, genuinely static corpus in the prefix and a short question at the end. The textbook prefix-cache win.
- High-volume classification and extraction — a long system prompt with instructions and few-shot examples, a tiny variable payload. Thousands of requests sharing one prefix.
- Customer-facing assistants with a fixed policy layer — product documentation, tone rules, escalation policy. Stable across every conversation in the fleet.
- Multi-agent systems with a shared brief — every sub-agent starts from the same context block, so one prefix amortises across the whole fan-out.
What benefits least: one-shot calls, prompts below the minimum cacheable length (512 tokens on current Anthropic models, 1,024 on GPT-5.6 and later), and anything whose context genuinely differs every time.
Cameron also offered a useful sorting rule for the other half of the cost decision — whether you need the expensive model at all. He splits tasks into unlimited intelligence tasks, where more capability always yields a better result ("analyse the drivers of cost in my business"), and limited intelligence tasks, where it does not: ask how much you spent on subscriptions last month and a cheap model with the right MCP servers returns the same number as a frontier model. For limited-intelligence tasks the question is only what the cheapest route to that ceiling is. That is the budget-side mirror of the deadline-side argument in pick the fastest model your latency budget allows — and Artificial Analysis's advice for production rollouts is to shortlist from the public Pareto, then build your own benchmark, which is what their Optima product automates.
How MDflow fits
To be clear about the boundary: MDflow does not cache anything. Caching happens inside the inference API. What a knowledge base determines is the two inputs you can control — how many tokens your context costs, and whether those tokens are byte-identical from one turn to the next.
What already lines up today
- Plain markdown, stored and returned verbatim. A document fetched by id is the same bytes until someone edits it. That is precisely the property a cacheable prefix needs, and it is the one a fresh vector search cannot offer, because a different top-k ordering is a different prefix. Markdown is also dense per token — no tag overhead, no JSON envelope around the prose.
- Raw
.mdtwins. Shared documents and collections are reachable as raw markdown (/share/<slug>.md,/share/d/<id>.md), so the bytes your harness puts in the prefix are the bytes a human reviewed, with no renderer in between. - Folder descriptions as retrieval scope.
mdflow_get_contextranks folder descriptions above names and titles, so you can pull the context for a topic rather than the whole workspace — the input-token lever, applied before the caching lever. - One copy, many surfaces. The same document is reachable over MCP with OAuth, over the HTTP API with a token, and from VS Code or n8n — so the prompt-assembly code and the person maintaining the context read the same file.
- Version history with per-actor diffs. When a cache hit rate drops, there are two candidate causes: the model or SDK changed, or the context changed. Line-by-line history, with API and MCP writes attributed as
automated · <token name>, answers the second one instead of leaving you to guess. - Comments and sharing. The stable half of your prefix is usually prose a non-engineer owns — policy, tone, domain rules. Review belongs on the document, not in a pull request they will never open.
Where we are headed
Direction, not a dated commitment: pinning a document to a specific version, so a deployment depends on the copy that was reviewed rather than whatever was current at fetch time — which is also, exactly, a stable cache prefix with a name you can audit. Alongside it, typed frontmatter so a build step can check status, owner and freshness as signals rather than conventions, and richer collection-level context so a whole prefix set can be handed to an agent as one unit.
The bottom line
The expensive part of an agent is no longer what it writes. It is what it re-reads, multiplied by how many times it re-reads it. Instrument input tokens, cache hit rate and turns per task — Cameron's three numbers — before you go shopping for a cheaper model, because a trajectory with a zero hit rate costs an order of magnitude more on the same model.
And treat the hit rate as something you design rather than something you enable. Caching rewards a prompt whose front end is boring: authored, reviewed, versioned context that is the same bytes today as yesterday, with the volatile material pushed to the back. That stable layer has to live somewhere both your code and your colleagues can read. Start free, connect an AI agent, or read the API docs.
Frequently asked questions
What is prompt caching and how much does it save?
Prompt caching lets a model provider reuse the computed state of an unchanged prefix at the front of your prompt instead of processing those tokens again. On Anthropic's published pricing, cache read tokens are 0.1 times the base input token price, which is the origin of the 10x figure, and newer models go further — 0.05 times on Opus 5.5 and Sonnet 5.5. OpenAI describes cached input as discounted up to 95 percent, with most GPT-5.6 and later models reading at 0.1 times the ordinary input rate. Writing to the cache costs more than a plain input token, so the saving only appears when the prefix is actually reused.
Why does caching matter more for agents than for chatbots?
Because an agent re-sends its whole history every turn and takes many turns. George Cameron of Artificial Analysis showed a cost build-up for a long-horizon agent task at AI Engineer in November 2026: around 13 cents per model call, 165 steps, roughly 21 dollars for the task. With a cache hit rate of zero, the same trajectory cost over 200 dollars. The multiplier is not the model, it is the number of times the same prefix gets re-read.
What is a cache hit rate and what breaks it?
A cache hit rate is the share of your input tokens that were served from cache rather than processed fresh. It breaks whenever anything at the front of the prompt changes, because caching is prefix-based. OpenAI's documentation states that cache reuse requires the entire rendered prefix to match, and lists tool names, descriptions, schemas and even tool ordering among the things that affect it. Anthropic describes the prefix as tools, then system, then messages, where changes at each level invalidate that level and all subsequent levels. A timestamp in the system prompt, a reordered tool registry or freshly retrieved snippets injected above the conversation will each take the hit rate to near zero.
Should I just put my whole knowledge base in the prompt and cache it?
No. A cached token is cheaper, not free — at 0.1 times the input price, 100,000 cached tokens per turn across 165 turns is still real money, and long inputs measurably degrade accuracy regardless of price. Caching changes the cost of context you had a reason to send. It does not make irrelevant context harmless.
What does prompt caching have to do with a markdown workspace?
A knowledge base does not cache anything — caching happens at the inference API. What it controls is whether the context you send is small and whether it is byte-identical from one turn to the next. A markdown document fetched by id returns the same bytes until someone edits it, which is what a cacheable prefix requires, whereas a fresh search on every turn returns a different set of snippets and silently destroys the prefix. MDflow stores plain markdown, serves raw .md twins, scopes retrieval with folder descriptions, and keeps version history so you can tell whether a cache hit rate dropped because the model changed or because someone edited the context.
Further reading
- Optimizing the Cost of Intelligence — George Cameron, Artificial Analysis, AI Engineer, November 2026. The talk this post is built on.
- Prompt caching — Anthropic's documentation; the source for the pricing multipliers, the
tools→system→messageshierarchy, and the TTL behaviour. - Prompt caching — OpenAI's guide; the source for the prefix-matching rule and the list of settings that change the prefix.
- AA-Briefcase v1.1 — the agentic knowledge-work benchmark the cost build-up comes from.
- Effective context engineering for AI agents — Anthropic, on treating context as a finite budget rather than a dumping ground.
- MDflow: tool output and the context window · pick the fastest model your latency budget allows · agent context comes in shapes, not queries · context engineering for AI agents · MCP docs · API docs · FAQ