$cat tool-output-context-window.md

Tool Output Is Eating Your Agent's Context Window

August 22, 202615 min readby MDflowview as .md

A chaotic emerald swarm of wireframe ribbons and blocks compressing through a narrow glowing aperture on a dark terminal grid, emerging as three clean evenly spaced bars

Every data platform team knows the same rhythm. Someone's Spark job fails at 3am, the stack trace is four hundred lines of Java, and the person best placed to read it is on a support rotation that never empties. Draško Profirović, a staff engineer at Pinterest, opened his AI Engineer talk in July 2026 with exactly that: the support queue as a never-ending stream, and the awkward truth that ranking one team's failing job against another team's deadline is a decision humans have to make and LLMs do not.

So Pinterest built Medic for Apache Spark, an agentic diagnostics tool you ask "why did this job fail?" and get back a research document with evidence and grounded fixes. The interesting part of the talk is not that it worked. It is what broke first — and it was not the model, and it was not the prompt.

TL;DR — The thing that fills an agent's context window is tool output, not your system prompt. Logs, time series, API dumps and directory listings arrive unbounded on every turn and scale with the system being inspected, not with the question being asked. The fix is at the tool boundary: rank and truncate before the model sees anything, offer a drill-down instead of a dump, render expensive data into a fixed-size form, and quarantine the noisy work in a sub-agent. None of that is tunable without a record-and-replay harness, and none of the domain knowledge behind it belongs in a string literal — keep runbooks and agent contracts as versioned markdown in something like MDflow.

What "tool output eating the context window" means

It means the largest, least predictable consumer of your agent's context is the data your tools hand back — and unlike the prompt, nobody wrote it, reviewed it, or capped it.

The asymmetry is easy to miss. A system prompt is a few thousand tokens, authored deliberately, changed under review. A tool result is whatever the downstream system happens to produce: a full log file, a hundred thousand metric points, an API response with forty fields you did not ask for. It arrives every turn, and it scales with the size of the thing being inspected rather than the complexity of the question.

Pinterest hit this immediately. In their words, large tool outputs from logs would quickly consume tokens and bring a halt to the agent's reasoning. Not degrade it. Halt it.

Anthropic's numbers on the same problem are worth sitting with. In Introducing advanced tool use, they report tool definitions alone consuming 134,000 tokens in large deployments — before a single result comes back. Their code execution with MCP write-up describes a workflow that dropped from roughly 150,000 tokens to about 2,000 when intermediate results were processed in the execution environment instead of being round-tripped through the model.

The obvious reply is: fine, windows keep growing. That reply does not survive contact with the data.

Why a bigger window is not the escape hatch

Because accuracy falls off long before the window fills.

Chroma's Context Rot study tested 18 frontier models across GPT-4.1, the Claude 4 family, Gemini 2.5 and Qwen3, and found every one of them degrading as input length grew — non-uniformly, and sometimes by 30 to 50 percent well below the documented limit. A 200K-token window can show serious accuracy loss at 50K tokens of input.

There is a second, sharper failure mode that shows up in diagnostics specifically. Pinterest found that logs are noisy and many of the exceptions in them are benign, so anchoring on the last exception is often wrong. Feed the model everything and you have not been neutral — you have handed it a pile of plausible red herrings and asked it to be disciplined. It will not be. It will find a story in the noise, and the story will be fluent and wrong.

That is the real cost. Not the token bill. A confidently misattributed root cause that a tired on-call engineer accepts at 3am.

The fix is at the tool boundary

Four patterns, all of them from the same talk, all of them generalising far past Spark.

1. Rank and truncate before the model sees anything

Pinterest started where everyone starts: regex heuristics to filter known-benign exceptions. It did not scale — each new failure mode meant another rule, and the rules fought each other.

What replaced it is the good idea. They built an exception classifier pipeline that learns which exceptions commonly appear in successful jobs, treats those as likely red herrings, and filters them from future analysis. Exceptions get fingerprinted, clustered, and ranked by content relevance and by how recently they occurred relative to job termination.

Note what the training signal is. Not a labelled corpus of "real" failures — the passing jobs you already have. An exception that shows up in jobs that finished fine is, by construction, not why this one didn't.

2. Offer a drill-down, not a dump

Having ranked the exceptions, Pinterest stopped letting the agent read logs at all. It got two tools instead:

get_top_k_exceptions()        → ranked, truncated, bounded
get_exception_details(id)     → full log context, one exception, on request

This is the pattern to steal. A tool that returns everything is a tool that has made a decision on the agent's behalf — the wrong one. A tool that returns a ranked summary plus a handle for drilling down lets the agent spend context on the one thing it decided was worth spending it on. Same information available; radically different default cost. It is the same instinct behind Anthropic's tool search tool, which they measured at an 85% token reduction with accuracy on Opus 4.5 rising from 79.5% to 88.1% — more accurate with less in the window.

3. Render expensive data into a fixed-size form

Raw time series are not context-window friendly. Feeding them straight in works in a demo and falls over on a job that ran for nine hours.

Pinterest's move here is genuinely clever: convert the raw series into graphs, collage them into a single annotated image — min and max called out, not unlike a Grafana dashboard — and attach that image to the conversation.

The payoff is not that images are cheaper. It is that images are predictable:

The image guarantees how many input tokens will be used for analysing any given Spark job, irrespective of its duration.

A nine-hour job and a nine-minute job cost the same to look at. And the signals that actually matter in that data are visual anyway — executors dropping to zero, long plateaus, bottlenecks, any resource behaviour inconsistent with healthy progress. You are not compressing the data so much as choosing the representation the question is actually asked in.

4. Quarantine the noisy work in a sub-agent

That metrics analysis does not happen in the main conversation. It happens in what Pinterest calls a quarantine sub-agent: it does the messy work in its own context window, summarises its findings, and returns only the summary to the parent.

This is the general form of all four patterns. Context isolation is a design boundary, not an optimisation. Anything whose output size is unbounded and whose useful signal is small belongs behind a sub-agent that eats the cost locally and reports a paragraph.

The part everyone skips: you cannot tune what you cannot replay

Here is the honest bit of the talk. Before any of the above, Pinterest's single ReAct agent was tuned by hand against production, and they say plainly why that failed: prompt tuning became unsustainable, adding detail in one area degraded behaviour in another, response quality was inconsistent, and end-to-end testing relied on manual runs against production data that got retentioned away. It was impossible to know whether a change had broken an earlier win.

So they built a record-and-replay harness:

record mode    agent calls real downstream systems
               → tool responses captured as fixtures
               → fixtures checked into the repo as code

playback mode  agent runs against fixtures
               → performs analysis, generates report
               → test suite grades the report against offline evals

The evals are small and specific. One of them checks that the agent suggested at most three fixes — a verbosity control, scored down when it offers more. That is a rubric, written by a person, about what a good report looks like. It is not a model capability question at all.

They also wired OpenTelemetry traces into Langfuse and read agent runs as waterfall diagrams, which is how you find out that a bad answer came from a bad tool call three steps earlier rather than a bad final inference.

Only after the harness existed did the log and metrics work pay off measurably. That ordering is the lesson. Curation without evaluation is just a different set of guesses.

Splitting tools splits the prompt too

Pinterest's last move was to replace the single ReAct agent with a multi-agent architecture on LangGraph's deep agents library — a triage agent that classifies the job's lifecycle state and generates failure hypotheses, research agents that validate each hypothesis in parallel and return a score and a root cause, a supervisor that picks the highest-confidence one, and a healer agent that produces remediations grounded in runbooks ingested into a vector database.

Two things fall out of that which are easy to under-read.

The first is that the architecture made scope cheap: extending Medic to also optimise Spark SQL jobs was, in effect, adding a new prompt. They tried making the flow more deterministic with LangGraph workflows and found it brittle compared to reason-and-act — so the structure went into decomposition, not into hard-coding the path.

The second is the bill. One prompt became a dozen: an instruction, an output shape, a rubric for what counts as good, an escalation rule, per agent. That is not code — that is writing, and the healer agent's raw material is runbooks written by the on-call engineers who own the system, not by the team building the agent.

In most organisations that content ends up scattered across a wiki page, a Slack thread, a Confluence doc and a few string literals. Then the runbook the agent quotes quietly stops matching the runbook anyone approved, and after a bad remediation nobody can answer the only question that matters: did the model change, or did the runbook?

Which applications benefit most

Infrastructure and data platform diagnostics — Spark, Flink, Trino, Kubernetes, CI. Enormous logs, tiny signal, expert knowledge that lives in runbooks.
Production incident response — the same shape, with a stricter clock and a stronger need for grounded, auditable suggestions.
Observability and APM copilots — metrics and traces are the canonical unbounded tool output; render-don't-dump applies almost verbatim.
Coding agents on large repositories — search results, test output and build logs are exactly this problem wearing different clothes.
Security triage — scanner output is mostly benign findings, which is the exception-classifier problem restated.
Support and customer operations agents — ticket history and CRM dumps blow the window fast, and the remediation knowledge is written by support leads.
Any MCP server you ship to third parties — you do not control the caller's context budget, so your response shape is your API's quality.

How MDflow fits

MDflow is the versioned markdown layer for the written half of this system — the runbooks, contracts, rubrics and eval criteria that a multi-agent architecture multiplies and that no vector database should be the only home for.

What lines up today:

Markdown-native storage. Runbooks, per-agent prompts, report rubrics and eval definitions are documents, not string literals. No proprietary block model in between the author and the agent.
Folder descriptions as retrieval context. A folder's description says what the documents inside it are for, and mdflow_get_context ranks descriptions above folder names and titles — so "Spark remediation runbooks, one per failure class" is a first-class retrieval signal rather than a filename you hope matches.
Bounded retrieval by design. mdflow_get_context returns ranked matching bodies rather than the whole workspace. It is the top-K-then-drill-down shape: retrieve what matches, then mdflow_get_document the one you actually need.
One source for agents and humans. Remote MCP over OAuth or a Personal Access Token from Claude, ChatGPT, Cursor and Codex; the HTTP API for services, cron jobs, CI and n8n. The healer agent fetches the same file the on-call engineer edits.
Version history with line-by-line diffs across editor, API and MCP writes — which is how you answer "did the model change, or did the runbook?" in seconds instead of an afternoon.
The document log naming the actor, including automated · <token name>, so a runbook an agent rewrote is itself visible.
Raw .md twins with frontmatter at every document URL, so a harness can fetch a runbook as plain markdown without parsing HTML.
Comments anchored to passages and collections for grouping — the review surface for content an SRE owns and an agent consumes.
Client-side encryption for incident playbooks and known-weak-subsystem notes you would rather not have sitting in plaintext.

Where we are headed — direction, not a dated commitment: pinning a written definition to a moment in time, so "which version of this runbook was in force when the agent proposed that fix?" is trivially answerable from the trace.

The bottom line

The prompt is the part you can see. The tool output is the part that grows behind your back, and it is the part that decides whether your agent reasons or stalls. Rank before you return. Offer a drill-down instead of a dump. Render unbounded data into a fixed-size representation. Quarantine the noisy work in a sub-agent. Build the record-and-replay harness before you tune, because otherwise you are trading one guess for another. And when decomposition multiplies one prompt into a dozen, keep the writing where both the agent and the person who owns the system read the same version of it.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What actually fills an AI agent's context window?

Tool output, not the prompt. A system prompt is a few thousand tokens and it is written once. Tool results are unbounded, arrive on every turn, and scale with the size of the thing being inspected — a log file, a time series, a directory listing, an API response. Anthropic measured tool definitions alone consuming 134,000 tokens in large deployments, before a single result came back. The prompt is the part you can see and control; the tool output is the part that grows behind your back.

Why doesn't a bigger context window solve this?

Because model accuracy degrades well before the window is full. Chroma's 2025 Context Rot study tested 18 frontier models and found every one got worse as input length grew, sometimes losing 30 to 50 percent of accuracy far below the documented limit — a 200K window can show serious loss at 50K tokens of input. A bigger window buys you room to fail more expensively, not more reliably.

What is a quarantine sub-agent?

A sub-agent that consumes an expensive, noisy input in its own context window and returns only a short summary to the parent. Pinterest used one for Spark metrics: the sub-agent turned raw time series into annotated graphs, reasoned about them, and returned findings — so the parent agent's context stayed healthy no matter how long the job ran. The pattern generalises to any tool whose output size is unbounded and whose useful signal is small.

How do you tune an agent's tool output without guessing?

Record and replay. Pinterest built a harness with two modes: record mode calls the real downstream systems and captures tool responses as fixtures checked into the repo, and playback mode runs the agent against those fixtures and grades the report against offline evals. That turns prompt tuning from intuition into a test suite — you can tell whether a change that improved one failure mode broke another.

Where should agent runbooks and remediation knowledge live?

In versioned markdown that both the agent and the humans who own the system read from the same place. Pinterest's healer agent draws remediations from ingested runbooks, and that content is written by on-call engineers, not by the team building the agent. Keeping it as markdown an agent fetches at run time over MCP or an HTTP API means the runbook the agent quotes is the runbook a human approved, and a diff answers the first question after a bad suggestion: did the model change, or did the runbook?