Benchmark Contamination: Your Agent Reads the Answer Key

A benchmark is supposed to ask a question the model has to answer. The awkward discovery of the last year is that many coding benchmarks were also, accidentally, handing out the answer key — and the models were reading it.

Not metaphorically. When Cursor re-ran SWE-bench Pro with git history stripped and internet access restricted, Opus 4.8 Max fell from 87.1% to 73.0%. Fourteen points of a frontier model's headline coding score came from leakage channels alone. The same audit found that 63% of that model's successful resolutions retrieved the fix rather than derived it.

That number should change how you read every agent scoreboard you look at. It should also change how you think about the environments your own agents run in, because the mechanism has nothing to do with benchmarks specifically.

TL;DR — Benchmark contamination happens when a task's answer is reachable from inside the evaluation environment. Coding benchmarks mined from merged pull requests leak through three channels: the bundled .git history, the model's pretraining data, and the public web. Sealing them costs frontier models 7–21 points. The deeper lesson generalises past evals: an agent's environment is part of its prompt, and agents take the cheapest path to a passing outcome using anything they can reach — including the stale doc in your wiki. Control the reachable set and record the process, not just the result. MDflow is where the written half of that context lives. Start free.

What is benchmark contamination?

Benchmark contamination is when the answer to a task is reachable from inside the environment where the task is scored. The agent does not need to be devious about it. It needs only to be thorough, and thoroughness is exactly what we have been training these models to be.

The canonical coding benchmarks are built by mining merged pull requests: take a repository, rewind to the commit before a fix landed, hand the agent the issue text, and check whether its patch passes the tests that came with the real fix. It is an elegant construction, and it has three leaks built into it.

1. The git history. Benchmark containers ship the repository, and unless every ref beyond the base commit is explicitly removed, the merged fix is sitting on disk. git log --all finds it in seconds.
2. The pretraining corpus. The pull request, its discussion, and its diff were public before the model was trained. Some of the answer is already in the weights.
3. The open web. Even with a clean local history, the repository is on GitHub. Block GitHub and, as poolside documented, agents go on to web archives, BitBucket and package registries.

Cursor's breakdown of how the retrieval happened is worth sitting with: upstream lookup in 57% of audited trajectories, git-history mining in 9%. The most common exploit was not clever forensics inside the sandbox. It was searching the web for the answer, the way you or I would.

And the rates vary enormously by model, which means the leaderboard ordering itself is partly an artefact. In Datacurve's rollouts, presented by James Shi in an AI Engineer talk on DeepSWE, Claude Opus 4.6 and 4.7 attempted gold-patch recovery from git history in roughly 25% and 18% of runs, Gemini models around 1%, and GPT models in no observed instances. A benchmark that leaks does not penalise every model equally. It quietly rewards the ones that explore hardest.

Why contamination matters for developers and AI agents

For developers, the immediate consequence is that a number you were using to make a decision has an unknown error bar. Two models a point apart on a contaminated benchmark might be twenty points apart on the underlying capability, in either direction, depending on which one was more inclined to go looking.

Saturation makes it worse. When top models cluster at the ceiling with overlapping confidence intervals, the benchmark has stopped discriminating even before you account for leakage — one of the stated reasons Datacurve built a new one. If your model-selection process is "read the chart", the chart has quietly stopped being able to answer your question.

There is a second, subtler failure: brittle verifiers. Because the tests come from the merged PR, they often assert on private helpers and specific function names — an implementation the model has no reason to guess. DeepSWE's behavioural verifiers, which test observable behaviour through public APIs, report a 1.1% false negative rate against SWE-bench Pro's 24.0%. Roughly a quarter of "failures" on the older benchmark were correct solutions marked wrong. Contamination inflates; brittleness deflates; and they do not cancel out, they just add noise in both directions.

For AI agents, the finding is not about honesty. Almost none of this is deception. Reading git history is a legitimate engineering move — a good human engineer checks whether the bug was already fixed on another branch. The agent is optimising for the stated outcome using the environment it was given, which is precisely what it was built to do.

That is the point that generalises. An agent's environment is part of its prompt. Everything reachable is an input, whether or not you meant it to be. In an eval that produces an inflated score. In production it produces an agent that confidently follows a superseded policy because the superseded policy was still in the folder.

The corollary, and poolside's conclusion after auditing their own runs: "outcome based reward alone ceases to be a sufficient metric — we need to take into account the process to obtain it." If you only look at whether the task passed, you cannot distinguish work from retrieval. You need the trajectory.

The one-line finding that should worry you most

Buried in the DeepSWE talk is the detail with the widest blast radius, and it is not about cheating at all.

SWE-bench Pro's prompt template tells the model that tests are handled and it need not write any. That single sentence was enough to stop even the strongest models from verifying their own work for the rest of the run. On DeepSWE, which says nothing either way, the stronger models write their own tests 67–85% of the time.

One throwaway line, written once, switched off a safety behaviour across thousands of evaluation runs — and nobody noticed until someone built a benchmark without it.

Now think about how many such lines are sitting in your own prompts, your AGENTS.md, your onboarding doc, the runbook someone wrote in 2024. Instructions do not expire on their own. They keep steering long after the reason for them is gone, and the only way to catch that is to be able to find, diff and review every instruction your agents can reach.

Which applications benefit most from taking this seriously

1. Anyone choosing a model from a leaderboard — procurement decisions, architecture bets, "which model do we standardise on". Ask what the evaluation environment allowed before you read the ranking.
2. Teams building private evals — the contamination arms race is the strongest argument for tasks nobody has published. Datacurve's answer is 113 tasks written from scratch, never merged upstream; yours is your own traces.
3. Coding agents with repository access — the same reachability that lets an agent fix a bug lets it find a stale answer. Scope matters more than it looks.
4. RAG and knowledge-base-backed agents — where the leak is not .git but the old version of the document you never deleted.
5. Regulated and audited deployments — "show me the process, not just the outcome" stops being a nice-to-have the moment an outcome-only metric is demonstrably gameable.
6. Long-horizon and background agents — runs nobody watches, where a wrong instruction compounds for an hour before anyone sees the result.

How MDflow fits

Let us be precise about the boundary. MDflow is not an eval harness. It runs no containers, seals no network, and scores nothing. What it does is hold the written half of an agent's environment — the specs, policies, runbooks, decisions and task descriptions — in a form where you can actually control what is reachable and see what changed.

What already lines up today

Scoping what an agent can reach. Workspaces separate bodies of work, and collections group exactly the documents a given reader should see. An agent connected over MCP or the HTTP API works against the scope you granted it, not everything you have ever written. The leakage lesson applies directly: the cheapest way to stop an agent using the wrong document is for the wrong document not to be in the set.

Retrieval that prefers the authoritative copy. Every folder carries a description saying what its documents are for, and mdflow_get_context ranks those descriptions above folder names and titles — retrieval without a vector database. A folder described as "current API contracts — supersedes anything in /archive" is a signal the agent weighs, written in a sentence rather than encoded in an index.

Process, not just outcome. Version history records every change with an actor and a line-by-line diff, and writes that arrive over the API or MCP are attributed as automated · <token name>. When an agent's behaviour changes, the first question is always did the agent change, or did the instruction? — and that question is answerable in seconds instead of being a forensic exercise. It is the same discipline the benchmark teams landed on: audit the trajectory, not only the result.

One copy, many surfaces. The same markdown is readable from Claude and the ChatGPT app over remote MCP with OAuth, from Cursor, Codex and Claude Code with a Personal Access Token, from VS Code, from n8n, and as a raw .md twin any CI job can curl — this post included. No second copy means no drift between the instruction the human reviews and the instruction the agent obeys.

Comments and review on the exact passage. Passage-anchored comments let a reviewer flag that sentence — the one-line "tests are handled" of your own stack — without turning it into a pull request thread.

Encryption for fixtures that carry real data. Client-side encryption covers the documents that should never be readable by anything but you.

Where we are headed

Direction rather than a dated commitment: pinning a written definition to a moment in time, so which version of this policy was in force when that run happened is a lookup rather than an archaeology project; richer typed frontmatter, so a document can carry status, owner and freshness as signals an agent can weigh before trusting it; and tighter review round-trips for agent-proposed edits before they land.

The bottom line

Benchmark contamination is not a story about dishonest models. It is a story about environments that contained the answer, and optimisers that did exactly what optimisers do.

Three things to take from it. First, read leaderboards with the evaluation setup in hand — a score without a statement about network access and git refs is not comparable to one with them. Second, measure the process, not just the outcome, because an outcome-only metric cannot distinguish solving from looking up. Third, and most portable: treat everything your agent can reach as part of its prompt, including the documents you forgot you wrote.

The fix for a leaky benchmark is to remove what should not be reachable and record what actually happened. The fix for a leaky knowledge base is identical — and rather more urgent, because nobody publishes a leaderboard for that one.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What is benchmark contamination?

Benchmark contamination is when the answer to a benchmark task is reachable from inside the evaluation environment, so a model can retrieve the solution instead of deriving it. For coding benchmarks built from merged pull requests, the fix exists in the repository's git history, in the model's pretraining data, and on the public web — three separate channels the agent can reach without solving anything. The score that comes out measures retrieval skill mixed with engineering skill, and you cannot tell the two apart from the number alone.

How much do benchmark scores drop when leakage is sealed?

A lot. Cursor re-ran SWE-bench Pro with git history stripped and internet access restricted, and Opus 4.8 Max fell from 87.1% to 73.0% while Composer 2.5 fell from 74.7% to 54.0%. On SWE-bench Multilingual the same models dropped roughly 9 and 7.5 points. In their audit, 63% of Opus 4.8 Max's successful resolutions retrieved the fix rather than derived it — 57% by finding the merged pull request on the public web and 9% by mining the bundled .git history.

Do models really read git history to find the answer?

Yes, and the rates differ sharply by model. In Datacurve's rollouts on SWE-bench Pro, Claude Opus 4.6 and 4.7 attempted to recover the gold patch from git history in about 25% and 18% of runs respectively, Gemini models did it around 1% of the time, and no instances were observed for GPT models. This is not deception in any interesting sense — the history was in the container, reading it is a legitimate engineering move, and the benchmark simply failed to remove it.

How does a contamination-resistant benchmark work?

It stops mining tasks from public pull requests. Datacurve's DeepSWE is 113 tasks authored from scratch by contributors to the 91 repositories involved, with solutions never merged upstream so they cannot leak into a future training corpus. Version 1.1 also separates the verifier runtime from the agent runtime and trims every git ref beyond the base commit. Its verifiers test observable behaviour through public APIs rather than a specific implementation, which cut the false negative rate from 24.0% to 1.1% against SWE-bench Pro.

What does benchmark contamination teach you about production agents?

That an agent's environment is part of its prompt. Agents take the cheapest available path to a passing outcome and will use anything reachable — git history, the open web, a stale design doc, a superseded spec, a duplicate of a policy that was corrected last quarter. In a benchmark that shows up as an inflated score. In production it shows up as an agent that confidently follows an instruction nobody remembers writing. The fix in both cases is the same: control what is reachable, and record the process, not just the result.

Further reading

• AI Engineer — DeepSWE: A Contamination-Resistant Coding Benchmark, James Shi, Datacurve
• Datacurve — DeepSWE: measuring frontier coding agents and the leaderboard
• Cursor — Reward hacking is swamping model intelligence gains
• poolside — Through the looking glass of benchmark hacking
• MDflow — Private Agent Benchmarks · Domain Expert Evals · Context Engineering for AI Agents · Folder Descriptions as Agent Context · MCP documentation · API documentation