Agent as a Judge: When Evals Need Their Own Agent

Evals stopped being optional about two years ago. What almost nobody noticed is that while teams were busy writing them, the thing being evaluated changed shape underneath the evals.

Aparna Dhinakaran, co-founder of Arize AI, opened the evals track at AI Engineer in July 2026 with the numbers behind that shift. Arize runs over 100 million evals a month; the average team on the platform runs about 12 different eval jobs, and the top teams run over 3,800 distinct evaluators. Plenty of measurement. And yet her own team kept shipping agent failures that none of it caught.

TL;DR — Agent as a judge makes the evaluator an agent: it reads the trace, follows the trajectory, inspects tool calls, and chooses what evidence to gather before returning a verdict. A fixed rubric can only score the finished output, which is the wrong surface when the failure is a stuck loop, a stale argument, or lost context halfway through. In the original research it aligned with human experts 90.4% of the time against 60.4% for LLM as a judge — at roughly 11× the latency — and, surprisingly, giving the judge memory made it worse. It does not replace deterministic checks or rubric judges; it is a third layer. All three layers are driven by written criteria, and written criteria need a home both people and agents can read — which is what MDflow is for. Start free.

What is agent as a judge?

Agent as a judge is an evaluation method in which the evaluator is itself an agent — it investigates the run rather than scoring its output. Instead of receiving a finished answer plus a rubric, the judge receives access to the trace and works: it follows the trajectory, reads intermediate steps, inspects tool calls and their arguments, and decides for itself what evidence it needs before committing to a verdict.

The distinction lands cleanly against its predecessor:

The idea has a formal origin. In Agent-as-a-Judge: Evaluate Agents with Agents (Zhuge et al., presented at ICML 2025, from a team spanning Meta AI and KAUST that includes Jürgen Schmidhuber), the authors argue that contemporary evaluation is inadequate for agentic systems for two reasons: it either "focuses exclusively on final outcomes — ignoring the step-by-step nature of the thinking done by agentic systems — or requires excessive manual labour." An agentic evaluator is the obvious escape from that dilemma, because it can look at the steps without a human having to read all of them.

Why fixed rubrics ran out of road

Dhinakaran's framing of the escalation is worth repeating, because it is the reason a rubric stopped being sufficient rather than merely imperfect:

1. 2023 — the thing being evaluated was an answer to a prompt.
2. 2024 — frontier models added tool calls, reasoning, and deep research.
3. Now — teams run loops over real-world data, with sub-agents kicked off on long-horizon tasks.

Each of those was a jump in complexity, and the failure surface jumped with it. Arize builds its own agent, Alex, which lives inside its UI and searches across enormous volumes of traces. As the frontier labs shipped capabilities, Arize added them to Alex — and then met the new failure modes personally. Alex would forget context. It would not know when something was already done. It would get stuck in loops.

None of those are output-quality defects. Run a rubric over the final answer and it may well pass: the answer is fluent, relevant, correctly formatted. The defect is in the middle of the run.

There is a second, structural problem. Alex generates a dynamic UI on every interaction — so every user produces a genuinely different trajectory. A fixed rubric presumes a stable shape to score against. When there is no stable shape, a fixed rubric is measuring the wrong object.

The failure catalogue that agentic judges are good at, and single-pass judges are structurally blind to, comes down to a handful of patterns: stuck loops, the same tool called repeatedly for no gain, stale arguments passed mid-trajectory, context lost partway through, silent tool failures papered over by a plausible final answer, and failed error recovery.

That last one deserves emphasis. A silent tool failure that the model smooths over is precisely the failure an output-only eval is built to miss.

What the research actually found

Being precise about the evidence matters more than the headline, because both the strong result and the awkward caveats are useful.

The paper introduced DevAI, a benchmark of 55 realistic AI code-generation tasks annotated with 365 hierarchical solution requirements — a structure that matters, because requirement-level judging is what an agentic evaluator can do and an output scorer cannot. Three code-generating systems were benchmarked on it: MetaGPT, GPT-Pilot and OpenHands.

Alignment with the human expert consensus (evaluating OpenHands):

Cost and time, which is where it gets interesting:

Read those two rows together. Against human evaluation, the saving is enormous — roughly 97% of both cost and time. Against an LLM judge, the token cost is effectively identical and the agentic judge is about eleven times slower. The "97% cheaper" line that circulates about this paper is true only of the human comparison, and citing it against LLM-as-a-judge inverts the actual trade-off.

The finding that should change your design

The framework was designed with eight modules: graph, locate, read, search, retrieve, ask, memory, and planning. The best-performing configuration used five. Search, planning, and memory were dropped — and memory did not merely fail to help, it actively degraded alignment, because an error in an early judgment propagated into a chain of subsequent errors.

The version that worked gathered fresh evidence for every requirement.

This is a genuinely counterintuitive result for anyone whose instinct is that more agentic capability is better. It also has a clean practical reading: a judge should not accumulate opinions; it should re-derive them. Which raises the obvious question — if the judge's state is deliberately disposable, where does the durable part live? The answer is the written criteria, outside the judge. We will come back to that.

Honest limits, stated plainly: this is one benchmark, on code generation, produced by the authors of both the benchmark and the method. In production there is no ground-truth consensus to align against at all, which is why human spot-checking does not go away.

Why agent as a judge is useful

For developers

You get a reason, not a rate. A pass rate tells you 8% of runs failed. A judge that read the trajectory tells you the agent called the same search tool nine times with the same arguments because the first call returned an empty list it treated as a transient error. One of those is a number to worry about; the other is a fix.

Findings you did not think to ask for. A rubric can only score criteria you wrote down in advance, which means it structurally cannot discover a failure mode. An investigating judge can flag the trajectory was inefficient without anyone having pre-registered efficiency as a criterion. Arize shipped this as Signal, a long-running agent that reads production traces, groups recurring failures into ranked issues with evidence, and — because it already holds the analysis — can open a pull request with a proposed fix.

Plain-language criteria instead of column mapping. A productised agentic judge takes its scoring instruction as prose and finds the relevant fields in the trace itself. That is a smaller detail than it sounds, and a significant one: the interface to your evaluator becomes a document, not a config schema.

For AI agents

A trace becomes a readable artefact. An agent-authored investigation of a trace is something a later agent can consume — which is what closes a self-improving loop rather than just reporting on one.

Requirement-level verdicts are actionable. "Failed requirement 3.2 of 7, here is the step where it went wrong" gives a coding agent a target. "Score: 0.71" gives it nothing.

Fresh evidence per criterion beats a growing context. The memory ablation is a design lesson for agents generally: state that accumulates without correction compounds its own errors. Re-reading a canonical written definition is safer than remembering your last interpretation of it.

Which applications benefit most

1. Long-horizon agents. Hours or days of work, where the interesting failures are mid-run and the final output is not diagnostic.
2. Agents that generate their own interface. Dynamic UI, generated dashboards, tool-composing agents — anything where each run has a different shape and there is no stable artefact to score.
3. Deep research and multi-step retrieval. The answer reads well; the question is whether the sources were actually consulted or quietly skipped.
4. Multi-agent and sub-agent systems. Handoffs are where context goes missing, and handoffs only exist in the trajectory.
5. Coding agents. The origin domain, with a natural fit: hierarchical requirements plus artefacts a judge can inspect directly.
6. High-volume production traffic. Too many traces for humans, too varied for a rubric — the gap a continuously running judge fills.
7. Regulated workflows. Where how a conclusion was reached is part of the compliance question, not just whether it was right.

How MDflow fits

MDflow is not an eval harness. It does not run judges, collect traces, score criteria, or host a leaderboard. That is Arize's business and your observability vendor's business.

What MDflow holds is the layer the whole method rests on. Look again at the two most striking results above. An agentic judge takes its instruction as prose. And the judge that worked threw away its memory and re-derived every verdict from a written requirement. Both point at the same artefact: the durable object in an eval system is not the judge and not the score — it is the written definition of what good means. That is a document, and documents are what MDflow does.

What already lines up today

Hierarchical requirements are a markdown structure, not a schema. DevAI's 365 annotations were hierarchical solution requirements, and hierarchy is what markdown natively is — nested headings, nested lists, one requirement per line where each has to be individually judgeable:

## 3. Retrieval behaviour

- [ ] 3.1 Every factual claim in the answer traces to a retrieved source
- [ ] 3.2 A source that returns empty is reported, never silently skipped
- [ ] 3.3 No tool is called twice with identical arguments in one run

That is a document a person can argue with in review, and the same document an agentic judge can be handed as its criteria. Because /tasks aggregates ordinary - [ ] lines out of document bodies, a requirement list is also a checklist you can work through without a second system.

One copy for the human reviewer and the judge. The same markdown is readable from Claude and the ChatGPT app over remote MCP with OAuth, from Cursor, Codex and Claude Code with a Personal Access Token, from VS Code, from n8n, and as a raw .md twin any CI job can curl — this post included. The rubric your reviewers debate and the rubric your judge is scored against being literally the same file is the cheapest reliability win in the whole stack, and the one teams most often skip.

Retrieval that finds the authoritative copy. Every folder carries a description of what its documents are for, and mdflow_get_context ranks those descriptions above folder names and titles — retrieval without a vector database. A folder described as "current eval criteria — supersedes anything under /archive" is a signal you wrote in a sentence, and it is exactly the signal a judge needs in order to not score against last quarter's rubric.

Discovered failure patterns can be written back. A judge like Signal exists to find failure modes nobody pre-registered. Those findings are only worth something if they become durable, reviewable prose rather than a closed ticket. Agents can create and update documents over MCP and the HTTP API, so a discovered pattern lands as a document in your workspace — and every such write is attributed as automated · <token name>, so you always know a judge wrote it rather than a person.

Version history answers the first question after any eval number moves. Every saved change captures the previous version across every write path with line-by-line diffs and non-destructive restore. When alignment drops, did the agent change or did the criteria change? stops being a forensic exercise. Criteria drift is not a hypothetical: it is what happens to every rubric that meets real outputs.

Comments land on the exact sentence. Passage-anchored comments let a reviewer dispute that criterion — the one that has been quietly failing every run for a month — without escalating to a pull request thread.

Encryption where the fixtures are real. Documents holding real customer traces or production data can be client-side encrypted, which also means they are never indexed server-side.

Where we are headed

Direction rather than a dated commitment: pinning a written definition to a moment in time, so which version of this rubric was in force when that run was judged is a lookup instead of an archaeology project; richer typed frontmatter, so a criteria document can carry status, owner and freshness as signals a judge weighs before trusting it; and tighter review round-trips for agent-proposed edits — the Signal-shaped workflow where a judge proposes a change to the criteria and a human approves it in place.

The bottom line

Agent as a judge is not a better rubric. It is an admission that the failure moved, from the output to the trajectory, and that measuring a trajectory requires an evaluator that can walk one.

Three things to take from the evidence. First, layer, do not replace: deterministic checks for anything with a correct answer, a rubric judge for subjective qualities you can state in advance, an agentic judge for what only shows up in a trace. Dhinakaran's own summary is that most teams do the first two, and the future is having all three. Second, respect the latency: same token cost, eleven times the wall clock, so deploy it where it buys you a finding rather than a faster score. Third, and most portable: the judge's memory is disposable, the written definition is not. The configuration that won gathered fresh evidence against a written requirement every single time.

Which means the artefact you should be most careful with is not your harness. It is the document that says what good means — and whether the human who reviews it and the agent that enforces it are reading the same copy.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What is agent as a judge?

Agent as a judge is an evaluation method where the evaluator is itself an agent: it reads the trace of the run under test, follows the trajectory, inspects tool calls and intermediate steps, and decides what evidence to gather before returning a verdict. LLM as a judge scores a finished output against a fixed rubric in a single pass. Agent as a judge investigates how the output was reached, which is where most agent failures actually live.

How accurate is agent as a judge compared to LLM as a judge?

On the DevAI benchmark in the original 2024 paper by Zhuge and colleagues, agent as a judge aligned with the consensus of human experts 90.44% of the time in the black-box setting and 92.07% with access to intermediate artefacts, against 60.38% and 70.76% for LLM as a judge. That result is from one benchmark, on code generation, run by the authors of both the benchmark and the method, so treat it as a strong signal rather than a settled number.

Does agent as a judge replace LLM as a judge?

No. Aparna Dhinakaran of Arize framed it as three layers that coexist: deterministic checks for anything with a correct answer, LLM as a judge for subjective qualities you can state as a fixed rubric, and agent as a judge for failures that only appear in a trajectory. Most teams run the first two today. Agent as a judge is roughly eleven times slower than a single-pass LLM judge, so it belongs where the extra latency buys you something a rubric cannot see.

Why did giving the judge memory make it worse?

Because judging errors chain. The paper designed eight modules and found the best-performing configuration used only five of them: graph, locate, read, retrieve and ask. Search, planning and memory were dropped, and memory actively degraded alignment because a wrong verdict early in a run biased the verdicts that followed. The practical lesson is that a judge should gather fresh evidence for every criterion, and that the durable state belongs in an external written definition rather than in the judge's own accumulated opinions.

What does MDflow do for agent evals?

MDflow is not an eval harness and scores nothing. It holds the written half that agentic judges depend on: the criteria, the hierarchical requirements, the failure-pattern notes a judge discovers, and the definition of done. Because an agentic judge takes instructions in plain language, the interface to it is a document, and MDflow keeps that document readable by both a human reviewer and the judge itself over MCP or the HTTP API, with version history showing exactly when a criterion changed.

Further reading

• AI Engineer — The Future of Evals: From LLM as a Judge to Agent as a Judge, Aparna Dhinakaran, Arize AI
• Zhuge et al. — Agent-as-a-Judge: Evaluate Agents with Agents (arXiv 2410.10934, ICML 2025) and the DevAI benchmark
• Arize — Agent-as-a-Judge: From LLM Judges to Agentic Evaluation
• MDflow — Eval Rubrics for AI Agents · Domain Expert Evals · Private Agent Benchmarks · From Signal to PR: the Self-Improving Agent Loop · MCP documentation · API documentation