skip to content
$mdflow
$cat private-agent-benchmarks.md

Private Agent Benchmarks: From Traces to Simulations

17 min readby MDflowview as .md
An emerald wireframe stream of telemetry trace lines flowing into a sealed isometric glass cube, which replicates into a parallel grid of smaller cubes each marked with a tick or a cross, on a dark terminal grid background

Every model release ships with a chart. Pass rates on SWE-bench, on Terminal-Bench, on whatever computer-use benchmark is current. The chart is genuinely useful — for deciding which model to try first. It tells you almost nothing about whether the agent you have built on top of that model is safe to ship on Thursday.

Rustem Feyzkhanov, who leads the AI platform team at Snorkel AI, gave a talk at AI Engineer in July 2026 called From Agent Traces to Agent Simulations, arguing that the gap between those two questions is the thing most teams are missing. His company runs millions of agent simulations a month and treats benchmark construction as an engineering discipline rather than a research artefact. His three claims were blunt: every company needs its own benchmark, it has to be as close to production as you can make it, and it is not a static file — it is a data set your production traces keep feeding.

TL;DR — A private agent benchmark is a set of repeatable simulation tasks built from your own production traces, running against your own tools, policies and database snapshots. Traces find failures but cannot be re-run; simulations freeze the environment so you can compare model, prompt, harness and skill changes apples to apples — and on cost and latency, not just pass rate. A task is mostly files: an instruction, an environment definition, verifiers, and an oracle solution proving it is solvable. The written half of that — instructions, policies, personas, rubrics, failure patterns — is prose, and prose needs a home both people and agents can read, which is what a markdown workspace like MDflow is for.

What is a private agent benchmark?

A private agent benchmark is a set of repeatable simulation tasks, built from your own production traces, that run against a miniature version of your own environment. Not your domain approximated by someone else's. Your tools, your API responses, your database state, your policies, your escalation rules.

The distinction that makes this concrete is between a trace and a simulation.

A trace is a record of one real run: the input prompt, the actions the agent took, the tools it called, the output it produced. Traces are how you find out that something broke, and they are irreplaceable for that. But you cannot re-run one. The database has moved on, the tool versions changed, the user is gone. You can A/B test in production, but you can never fully compare apples to apples, because the world underneath moved between the two arms.

A simulation turns that trace into a task with a frozen environment. Now the only variable between two runs is the agent configuration you are deliberately changing — and you can run hundreds of them in parallel, offline, on a laptop's worth of patience.

That reframing is what unlocks the second thing production teams care about and public leaderboards mostly ignore. Feyzkhanov's point: model releases report pass rate, which makes sense, because pass rate is a statement about the frontier. When you ship an agent you also care about cost per solved task, latency, and number of retries — and those are only comparable when the environment holds still.

Public benchmarks are useful to orient yourself and build a prior. A private benchmark is what you use to ship.

Why private agent benchmarks matter

For developers

Because you are not shipping a model, you are shipping a system — and only a private benchmark can score the system. SWE-bench is about fixing GitHub issues. Terminal-Bench is about an agent in a shell. Neither knows your approval thresholds, your refund limits, your internal API's habit of returning 200 with an error body.

A private benchmark lets you test the full stack: the model, the thinking level, the prompt, the harness, the skills, the tool set. In production you do not care which model is better in the abstract. You care whether this configuration, on your work, gets to an acceptable answer often enough and cheaply enough.

That gives you three distinct uses, and they arrive in that order:

  1. Release — verify the thing works at all, handle edge cases, pick the model, debug the traces.
  2. Keep it right — wire the benchmark in as a release gate, so a change anywhere in the agent stack cannot silently regress.
  3. Optimise — tune for cost or latency once correctness is pinned, or use the runs as training data. Snorkel published exactly this loop for an insurance-underwriting environment, reporting a Qwen3-30B model moving from 33.3 to 41.4 on multi-turn evaluation and 24.6 to 30.4 overall on a tau-bench-family benchmark after training on environment-derived data.

There is a fourth benefit that shows up quietly. When you have a benchmark, you stop fixing everything in the prompt. The industry anti-pattern Feyzkhanov named is the prompt that has accreted a decade of NEVER do this and ALWAYS output that — each line a scar from one bad run. With a simulation you control the full stack, so you can put the fix where it belongs: context overload is a harness problem, a missing procedure is a skill problem, an inconsistent shape is a structured output problem. The prompt stops being the landfill.

For AI agents

Because reliability, not capability, is the thing agents are currently short of — and reliability is only visible across repeated runs. The tau-bench paper made this measurable with the pass^k metric: the probability that an agent solves the same task correctly on all k independent attempts. Its headline finding was that frontier function-calling agents solved under 50% of tasks on pass^1, and under 25% on pass^8 in the retail domain. Average success and consistent success are different products.

You cannot compute anything like pass^k from production traces, because you never get to run the same task twice. A simulation is the only place that number exists.

The second agent-specific reason is that a simulation is where a simulated user can live. You can snapshot a database and mock an API, but you cannot put a real customer in a test harness. So the user becomes another model, with its own persona prompt and context, supplying the realistic ambiguity, mid-conversation changes of mind and incomplete information that real people supply and scripted test fixtures never do.

Anatomy of a simulation task

A benchmark task is less exotic than it sounds. It is a directory of files.

The most widely adopted shape today is the Harbor format, from the team behind Terminal-Bench (now the official harness for Terminal-Bench 2.0, Apache-2.0 licensed):

my-task/
├── task.toml            # configuration: timeouts, difficulty, tags
├── instruction.md       # the natural-language task the agent sees
├── environment/
│   └── Dockerfile       # the container the agent works in
├── tests/
│   └── test.sh          # verifiers — the agent never sees these
└── solution/
    └── solve.sh         # oracle solution — proves the task is solvable

Four pieces do the work.

The environment has to be mini-production. Not production — you are not running a full stack per experiment — but close enough that the agent cannot tell. In practice that means a database snapshot rather than the live database, sidecar containers for API services and MCP tools, mocked upstreams, and the real policy documents the agent would read at run time. The rule of thumb is the one you already use for integration tests, applied to an agent.

The simulated user is an LLM with a persona. It has its own prompt and context, and its job is to behave like a person rather than a fixture.

Verifiers look at more than the output. This is where agent evaluation departs from ordinary testing. A coding test checks the code. An agent simulation has four surfaces to check: the final environment state (did the right rows change?), the trace (did it take a sane path, or get there by luck?), the artifacts (the files it produced), and the user replies. Deterministic assertions handle final state and tool calls well. LLM-as-a-judge — or harness-as-a-judge, or agent-as-a-judge — handles trace quality and planning. Subject-matter experts handle the residue: not every case, only the ones where the verifiers and the agent disagree. That is the expensive human attention, spent where it is actually informative.

The oracle solution proves the task is possible. It runs the same sequence as the agent, but with a known-good scripted solution in the agent's seat. It validates the task, not the agent. Skip it and you cannot tell a failing agent from an impossible task or a broken verifier — and when a benchmark is new, those two bugs are more common than agent failure.

For long-horizon work that spans hours, the pattern extends: break the task into steps, give each step its own prompt and its own verifiers, and terminate the simulation early once the agent has clearly gone off the rails.

What goes wrong when you build one

Benchmark development is its own craft, and the failure modes are specific:

  • The agent reward-hacks the simulation. It works out it is in a sandbox and games the verifier instead of doing the task. (This is the same simulation-awareness problem that broke long-horizon evals — an agent that knows it is being tested is no longer being tested.)
  • The task is too easy, or the verifiers are too broad. Everything passes, including runs that did the wrong thing.
  • The verifiers are wrong. Everything fails, and you spend a week debugging an agent that was right.
  • Variance is too high. The same config passes and fails at random, and no comparison you make is meaningful.

The countermeasure is to accept that a benchmark is software and treat it as such: its own repository, its own CI pipeline. Check that dependencies are pinned and base images resolve. Check that no fixtures are missing. Run the oracle and require it to pass. Run the agent N times and require the task to be solvable but not trivial. Tag the result simple / medium / hard from the observed pass rate, and only then approve it into the benchmark.

And keep a holdout. The classic 80/20 train/test split applies for the same reason it always did: if you tuned against every task you have, your number is a memory, not a measurement. Snorkel's own public Senior SWE-Bench does this in the open — 100 tasks drawn from real pull requests across 12 production repositories, 50 released publicly and 50 held back specifically to resist contamination.

Once the agent is live, two loops run side by side. One expands the benchmark: observability catches production failures, and the interesting ones become new tasks. The other runs experiments: new agent configs against the extended benchmark, recorded, and used as a release gate. The whole thing only works if those two loops are connected — which is a nice way of saying that your observability tool and your experiment tool need to be looking at the same catalogue of failure patterns.

Which applications benefit most

  1. Customer-facing agents with policy constraints — refunds, cancellations, eligibility, escalation. The policy is the hard part and it is not in any public benchmark.
  2. Multi-system enterprise workflows where success is a database state, not a sentence — underwriting, claims, procurement, onboarding.
  3. Internal coding and ops agents running against your conventions, your monorepo layout, your runbooks — where "did it follow our rules?" is the whole question.
  4. Long-horizon agents whose work spans hours or days and cannot be judged from a final answer alone.
  5. Cost-sensitive high-volume agents, where the real decision is whether a smaller model plus a better harness beats a frontier model, and only a controlled environment can answer it.
  6. Regulated domains that need to show not just the score but the exact task, environment and criteria that produced it.

How MDflow fits

MDflow is not an eval harness. It runs no containers, executes no verifiers, and scores nothing. That belongs to Harbor, to your CI, to Arize or Braintrust or Langfuse or whatever pipeline you already run.

What it addresses is the half of a benchmark that is not code. Look at the task directory again: instruction.md is a markdown file. So are the policies the agent must follow, the personas that drive simulated users, the rubrics an LLM judge is handed, the difficulty notes, and the failure-pattern log that decides which production traces become new tasks. In most teams that prose is scattered across a repo, a Confluence page, a Slack thread and a prompt string — and the copy the agent reads in production quietly stops matching the copy the benchmark tests against.

What already lines up today

Markdown, with a raw twin for agents. Task instructions and policies are stored as plain markdown with no proprietary layer, and every document has a raw .md twin an agent or a build step can fetch directly. The file your simulation harness pulls at task-build time is the file a person edits.

Version history answers "did the agent change, or did the task?" Every saved change captures the previous version across every write path — editor, HTTP API and MCP — with line-by-line diffs and non-destructive restore. When a benchmark number moves between two runs, that is the first question, and a diff answers it in seconds. (Version history is a Pro feature, private to the document owner, and deliberately not exposed over the API or MCP.)

Folder descriptions make the right policy retrievable. Every folder carries a description of what belongs inside it, and mdflow_get_context ranks those descriptions above folder names and document titles before returning bodies. A folder described as "Refund and escalation policy — authoritative; the production agent and the simulation environment both read these" is a retrieval signal you wrote deliberately, not one inferred from a filename. That is why folder descriptions beat file names.

One source for production and simulation. The same workspace is reachable from Claude, ChatGPT, Cursor and Codex over the remote MCP server with OAuth or a Personal Access Token, and from scripts, cron jobs, CI and n8n over the HTTP API. If the agent under test pulls its policy from the workspace at run time, the environment cannot drift from production, because there is no second copy to drift.

The Document Log shows who moved the goalposts. A cross-document activity feed records created, edited, shared and deleted events with the actor on every row, shown as automated · <token name> for anything arriving via API or MCP, with a side-panel diff on edited rows.

Tasks live inside the documents they concern. Because /tasks aggregates ordinary - [ ] checkbox lines out of markdown bodies, "add three edge-case tasks for tool-timeout handling" can sit inside the failure-pattern note that prompted it and still appear on a real list.

Collections and comments for the review loop. Group an instruction, its policy and its rubric into a collection and share it as one read-only link with the subject-matter expert reviewing disagreements; they can attach comments to a selected passage of the markdown source rather than starting a thread that scrolls away.

Encryption where the fixtures are sensitive. Simulation environments derived from production traces often carry real customer data. Documents can be client-side encrypted, which also means they are never scanned or indexed server-side.

Where we are headed

Direction, not a dated commitment: we are most interested in making a written definition easier to pin to a moment in time. For benchmarks specifically, "which version of the policy was in force when this task was scored" is the question we would most like to make trivial to answer — alongside richer structured retrieval over folder descriptions.

The bottom line

Public benchmarks tell you which model to try. A private agent benchmark tells you whether your agent is shippable — and it is the only place that questions like pass^k, cost per solved task, and "did this prompt change break anything" have answers at all.

Building one is less about infrastructure than it looks. A task is an instruction, an environment, some verifiers and an oracle. The environment is engineering; the instruction, the policy, the persona and the rubric are writing. Keep the writing somewhere versioned, where a human reviewer and the agent under test read the same file, and where a diff can tell you which one of them moved.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What is a private agent benchmark?

A private agent benchmark is a set of repeatable simulation tasks built from your own production traces, running against a mini version of your own tools, APIs, database state and policies. Public benchmarks like SWE-bench or Terminal-Bench measure a model against someone else's domain. A private benchmark measures your whole agent stack — model, harness, prompts, skills and tools — against the work your users actually send it.

Why are traces not enough to evaluate an AI agent?

Traces show you what happened once. They are excellent for spotting failures and edge cases in production, but they are not repeatable: the database state, the tool versions and the user all moved on. You cannot re-run a trace against a different model or prompt and compare apples to apples. A simulation freezes the environment so the only thing that changes between runs is the agent configuration you are testing.

What files make up a simulation benchmark task?

In the Harbor format, from the Terminal-Bench team, a task is a self-contained directory: task.toml for configuration, instruction.md for the natural-language task the agent sees, environment/Dockerfile for the container it runs in, tests/ for the verifiers, and an optional solution/ holding the reference or oracle solution. The agent sees the instruction and the environment; it never sees the verifiers or the oracle.

What is an oracle solution and why does a benchmark task need one?

An oracle solution is a reference script that a human author writes to prove the task is solvable at all, and that the verifiers pass when it is solved correctly. It validates the task, not the agent. Without one you cannot distinguish an agent that failed from a task that was impossible or a verifier that was simply wrong — and both of those bugs are common when a benchmark is new.

How does MDflow relate to agent benchmarks?

MDflow is not an eval harness and runs no containers or scoring. It holds the written half of a benchmark: task instructions, the policies the agent must follow, the personas that drive simulated users, verifier rubrics and the failure-pattern log that decides which traces become new tasks. Those are markdown documents with version history, and the same document can be retrieved by a person in an editor or by an agent over MCP or the HTTP API.

Further reading

← all posts./start-free →