$cat discovery-is-not-derivation.md

Discovery Is Not Derivation: Why AI Agents Stall

November 4, 202615 min readby MDflowview as .md

A glowing emerald chain running across a dark grid, bright and complete for most of its length, then breaking at one missing link where the remaining links sit unlit

There is a chart in Uri Rolls's talk that is worth more than the benchmark it comes from. It shows models working through a long exploitation chain in an unfamiliar system, and it splits the chain into two phases. In the discovery phase, the models do well. They probe the services, read what they are allowed to read, and assemble nearly every fact the task depends on. Then the line stops.

Rolls, CEO of Arithmetic, presented it with Thom Wolf of Hugging Face at the AI Engineer World's Fair 2026 under the title "Training Frontier Models to Out-Think Hackers". The talk is nominally about cybersecurity. The part worth stealing is a measurement result that applies to every long-horizon agent you run, in any domain: the models had the information and could not use it.

TL;DR — Discovery is not derivation. Agents are good at collecting the facts a task depends on and bad at combining two of those facts into a conclusion neither one states. On Arithmetic's Mask Off benchmark, models discover nearly everything they need and then fail to derive the exploit — a gap you can only see because every step of the chain is graded deterministically rather than the task returning one pass or fail. The lever you control is not the model. It is whether the relationships the agent would otherwise have to infer are written down somewhere it reads, which is what a markdown workspace like MDflow is for.

What "discovery is not derivation" means

Discovery is gathering the facts. Derivation is the inference over facts you already hold. An agent that has read every relevant page and still cannot answer the question has not failed at retrieval. It has failed at the step after retrieval, and the two failures want completely different fixes.

This distinction is easy to lose because both failures look identical from the outside. The task fails. The trace is long. The obvious diagnosis — it needed more context — is the one that is almost always wrong here, and it is expensive, because the remedy is to shovel more tokens into a window that already contains the answer.

Mask Off, Arithmetic's benchmark, happens to separate the two cleanly. It focuses on access control, which Rolls justified on grounds that are hard to argue with: a foothold is the first door in any attack, and broken access control has been number one on the OWASP Top 10 since 2021, where it displaced injection after a decade. It held the top spot in the 2025 revision, which also folded SSRF into it and now carries the largest set of mapped CWEs of any category. Rolls put the industry built around it at roughly thirty billion dollars — his figure, offered on stage rather than cited.

The reason it matters for a reasoning benchmark is what these bugs are. They are not memory-safety mistakes in one file. They are logic breaks between components: one service checks a condition one way, another checks what it believes is the same condition a different way, and the disagreement between them is the vulnerability. No single file contains it. It exists only in the relationship.

The setup that makes the gap visible

Three design choices in Mask Off are worth copying regardless of what your agents do.

The environment is black-box. The agent gets no source code and no internet access, only the basic tooling it would need to work. It gets a running system of chained services — the illustrated environment ran Keycloak, Vault and a broker together — and has to form its own model of how they fit. This is a deliberate rejection of the read-the-code-and-find-the-bug format. It is also, usefully, a rejection of the retrieval shortcut: there is nothing to look up, so nothing can be retrieved in place of reasoning.

The vulnerabilities are the team's own zero-days. Arithmetic's researchers find them in widely deployed open-source software and submit them to maintainers for verification. Because they were not public when the environment was built, no model could have seen them in training. That is contamination control by construction rather than by filtering, and it is the strongest version of the argument in benchmark contamination: if the answer key exists anywhere online, your benchmark is measuring recall.

Every step has a deterministic grader. This is the one that produces the chart. There is a final binary grader — did an underprivileged user perform an action it was not allowed to perform — and there are deterministic graders on the intermediate steps that record how far along the chain the agent actually got.

Why a binary grader would have hidden all of this

On a sixteen-step task, every failing run returns the same value. That is the whole problem with end-state grading, stated as plainly as it deserves.

Rolls described the illustrated solution as roughly sixteen steps. Run a dozen models against it with a pass/fail grader and you get a dozen zeros. You cannot tell the model that flailed at step two from the model that got to step fifteen and stopped. You cannot tell whether last week's prompt change helped. You cannot tell whether the failure is tooling, planning, or reasoning — and those have nothing in common as fixes.

With step-level graders, the same dozen zeros become a shape. And the shape Arithmetic found is the finding: models complete the discovery phase almost entirely and then do not derive the exploitation. The information is in the context window. The conclusion is not drawn.

That is a much more specific and more actionable result than "the benchmark is hard". It says the bottleneck is one particular cognitive step, and it tells you which one.

A second detail makes the point sharper. In an environment Rolls called Falltime, a model ran continuously for more than three hours and did not solve the task. He was explicit that it was not blocked waiting on code to execute. It was reasoning the entire time — which rules out the comfortable explanations. It was not short of time, short of tokens, or short of attempts.

The snapshot shown on stage, which Arithmetic itself labels as a snapshot rather than a ranking of models: a single solve in the first column, and in the wider attempt budget only one public model solving at all, with GPT-5.5 shown making the decisive leap in a specific environment. Treat those as a reading from one day on one benchmark, not a leaderboard.

The leap itself

The shape of the missing inference is instructive, and it is worth stating at the level the talk stated it, which is the level of a design principle rather than a recipe.

In the illustrated task, the system contains two authorization checks that are supposed to agree about who the administrator is. One of them matches on a name. The other matches on an ID. Names, in this system, can be changed by the user. IDs cannot. Each check, read on its own, looks correct. The vulnerability lives in the gap between them, and the escalation follows from noticing that gap.

Watch a model attempt it and, per Rolls, you see something that looks like effort: probing widely, jumping between services, trying things. It reaches the check. It reads the check. It never makes the connection between a mutable field in one place and an identity decision in another.

And there is a compounding difficulty that makes this harder than a static puzzle: in a live permissioning system, every action you take changes the state of the world you are reasoning about. An agent cannot brute-force the space, because the space moves underneath it. It has to hold a model of the system and update that model as it acts.

That is the sense in which this is a world-model problem rather than a knowledge problem. It is also exactly why Wolf reached for an unexpected comparison.

The ARC-AGI-3 parallel

Wolf's framing was that Mask Off belongs next to ARC-AGI-3, which is not a security benchmark at all. ARC-AGI-3 drops models into small interactive environments with no instructions, no stated rules and no goal, and asks them to work out what the world is by acting in it — click here, something changes there, infer the rule.

His stated figure for model performance there was 1 to 2 percent. The published launch leaderboard from March 2026 was harsher still, reporting frontier models between 0.0 and 0.3 percent against a human baseline that the ARC Prize team reports as complete solve rates on first contact. Either number tells the same story. These are not models failing on obscure knowledge. They are models failing to build a dynamic model of a world they are standing in.

The two benchmarks share a structure: unfamiliar system, no instructions, actions that mutate state, and a solution that requires a leap rather than a lookup. One wears a game interface and one wears a service mesh. They are measuring something close to the same thing.

Which is the uncomfortable implication for anyone shipping agents. If the gap is this fundamental, it will not be closed by giving the agent more documents — and the long-horizon failures documented in long-horizon agent evals start to look like the same animal seen from a different angle.

Which teams this applies to

Security review agents. The most direct read. An agent reviewing a PR sees one file. The class of bug that matters most is the one between two files, two services, or two teams.
Agentic coding on large systems. Cross-service refactors fail the same way: the agent reads both sides correctly and misses the contract between them. The invariant that makes the change safe is usually not written anywhere.
Incident response and SRE agents. Discovery is easy — logs, metrics, traces are all retrievable. The derivation ("this retry storm is downstream of that config change") is the whole job.
Data and analytics agents. The agent finds both tables. Whether it can join them correctly depends on a relationship that lives in someone's head, which is the argument for a semantic layer.
Anyone running an eval that returns a single number. If your harness grades only the end state, you already have this gap and cannot see it. That is a tooling decision you can reverse this week, unlike the reasoning gap itself.

How MDflow fits

MDflow is not a security tool and it does not make a model reason better. It finds no vulnerabilities, runs no agents against your systems, grades no steps, and holds no benchmark environments. Arithmetic's harness does that work; nothing in a markdown workspace substitutes for it.

What this talk clarifies is a different and narrower lever. The models failed to derive a relationship between two components. A derivation is only necessary when nobody has written the thing down. The moment the rule exists as prose in a place the agent reads, the inference becomes a retrieval — and retrieval is the half these models are already good at.

Concretely: a sentence like "authorization decisions in this system key on the immutable subject ID; display names are user-mutable and are never an authorization key" is a one-line invariant. It is obvious to whoever designed the system and invisible to everyone else, including every agent that will ever read the code. MDflow is a markdown workspace built so that the person and the agent read the same copy of sentences like that one.

What already lines up today

Invariants as prose, in the format both audiences read. Documents are plain markdown with a raw .md twin an agent can fetch directly, frontmatter included. There is no proprietary layer between the rule a human reviews and the bytes an agent retrieves.

Folder descriptions put the rule in reach. Every folder carries a description of what belongs in it, and mdflow_get_context ranks those descriptions above folder names and document titles — a retrieval signal you wrote rather than one a model guessed. This is why folder descriptions beat file names.

The cascade matches how invariants actually scope. Folders nest, and every document returns its full path plus a compounded_description — the workspace-to-folder cascade of descriptions. A rule that governs every service states itself once at the workspace level and arrives with every document beneath it, which is the right shape for "this holds everywhere unless a child says otherwise".

One copy for the reviewer and the agent. The same workspace is reachable from Claude, ChatGPT, Cursor and Codex over the remote MCP server, and from CI, cron and n8n over the HTTP API. A review agent that pulls its invariants at run time cannot drift from the copy a human engineer reads, because there is no second copy.

Version history answers when the rule changed. Every save captures the previous version across every write path — editor, API and MCP — with line-by-line diffs and non-destructive restore. After an incident, "was this invariant written down before or after?" is a diff rather than an argument. (Pro, private to the owner, and deliberately not exposed over the API or MCP.)

Failure traces are worth keeping as documents. "The agent discovered the name check and the ID check and never connected them" is a reusable artifact — a named pattern your next review prompt and your next eval case can both start from. That is the same argument eval rubrics for AI agents makes about edge cases: the value is in the dated specifics, not the summary.

Comments sit next to the clause they concern. Passage-anchored comments on shared documents keep a disagreement about an invariant attached to the invariant, rather than scrolling away in a thread.

Where we are headed

Direction rather than a dated commitment: better ways to pin a specific version of a document to a specific run, so "which invariant was in force when this review passed" stops being archaeology, and richer structured retrieval over folder descriptions.

Stated limits: MDflow stores and serves the prose. It cannot tell you whether the prose is true, and an invariant that is written down but wrong is worse than one that is missing.

The bottom line

The result worth carrying out of this talk is not that cybersecurity is hard for models. It is that the failure sits after retrieval, not before it — and that you can only see this if your grader has something to say about a run that did not finish.

So: instrument the steps, because a binary grader turns every distinct failure into the same zero. Expect the gap to be the inference and not the information, before you reach for more context. And write down the relationships your system depends on, because an agent can only derive what nobody has stated, and the one lever you hold without touching the model is how many derivations you make unnecessary.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What does it mean that discovery is not derivation?

Discovery is collecting the facts a task depends on. Derivation is combining two of them into a conclusion that neither one states. On Arithmetic's Mask Off benchmark, Uri Rolls reported that models discover nearly all the information they need in an environment and then fail to derive the exploitation from it. The gap is not missing information. It is the inference step over information the agent already has in context.

Why does a pass/fail grader hide agent failures?

Because on a sixteen-step task, every failing run returns the same value. A binary grader cannot distinguish an agent that stopped at step two from one that reached step fifteen, so it cannot tell you whether a change helped. Mask Off pairs a final binary grader, which asks whether an underprivileged user performed an unauthorized action, with deterministic graders on the intermediate steps that record how far the agent actually got.

What is the Mask Off benchmark?

Mask Off is a cybersecurity benchmark from Arithmetic, presented by Uri Rolls with Thom Wolf of Hugging Face at the AI Engineer World's Fair 2026. It focuses on access control. Agents work in live black-box environments built around zero-day vulnerabilities Arithmetic's own researchers found in widely deployed open-source software, with no source code and no internet access, and must chain many steps to escalate privilege.

Can AI agents build a model of a system they cannot read?

Only partially, on current evidence. Thom Wolf drew the parallel to ARC-AGI-3, where models must infer the rules of an unfamiliar interactive environment and act on them, and cited a 1 to 2 percent model success rate; the published launch leaderboard from March 2026 reported frontier models between 0.0 and 0.3 percent. In both settings an action changes the state of the world, so an agent has to maintain and update its own model of a system rather than pattern-match against a static artifact.

How do you make an agent's missing derivation cheaper?

Write the invariant down where the agent reads. An agent can only derive a relationship between two components by inference if no one has stated it; once the rule exists as prose in a place the agent retrieves from, the inference becomes a lookup. This does not make a model smarter. It removes one derivation from the set the model has to perform unaided, which is the only lever you control without training.