---
title: "Eval Rubrics for AI Agents: Golden Sets and LLM Judges"
description: "Eval rubrics decide what your agent scores. Why the rubric — not the harness — is the bottleneck, how human agreement caps your judge, and what to write down."
author: "MDflow"
date: 2026-08-16
reading_time: "17 min"
canonical_url: https://mdflow.cz/blog/eval-rubrics-for-ai-agents
md_url: https://mdflow.cz/blog/eval-rubrics-for-ai-agents.md
---

# Eval Rubrics for AI Agents: Golden Sets and LLM Judges

*Published August 16, 2026 · 17 min read*


Most teams building agent evals think their problem is tooling. It usually is not. The harness is a weekend of work; the argument about whether *that particular output* was a pass can run for a month.

Two engineers from Google's YouTube Ads team — Daniel and Pratika, who work on image and video models for ads — gave a talk at AI Engineer in July 2026 about building production evals for a generative agent. Almost none of it was about infrastructure. It was about rubrics, rater agreement, golden sets and traces: the written artefacts that decide what a score actually means.

> **TL;DR** — An eval rubric is the written definition of pass, fail and edge case that both your human raters and your LLM judge are scored against. Start small and grade by intuition before scaling; a judge can never be more reliable than your own team's human-human agreement (about 81% in the MT-Bench study); pass/fail rates hide the reason, so collect rater explanations and read agent traces; and expect the rubric to drift as you see real outputs. All of that is prose, and prose needs a home that both people and agents can read — which is what a markdown workspace like [MDflow](https://mdflow.cz) is for.

## What is an eval rubric?

**An eval rubric is the written definition of what counts as a pass, a fail, and an edge case for a specific agent behaviour.** It is what a human rater reads before scoring, and what an LLM judge receives as its prompt. The harness computes numbers; the rubric decides what those numbers mean.

In practice it is a short document, and the useful ones look roughly like this:

```markdown
# Rubric: ad disclaimer handling

## Pass
- Any legally required disclaimer present in the source asset is
  present, legible, and unmodified in the output.

## Fail
- Disclaimer removed, cropped, obscured, or reworded — for any reason,
  including "it improved the composition".

## Edge cases (decided 2026-07-14, see incident #412)
- Disclaimer partially occluded in the SOURCE asset → pass. We do not
  penalise the agent for input defects.
- Agent relocates the disclaimer but keeps it legible → fail. Placement
  is part of the legal requirement.

## Rater notes
- Always record WHY, not just the verdict.
- Score brand safety and accuracy separately; an output can pass one
  and fail the other.
```

Notice what is doing the work. It is not the pass and fail lines — those are the easy part. It is the edge cases, dated and tied to the incident that forced the decision, and the instruction to record reasoning. That is the part every team rediscovers the hard way, and the part that is almost never written down.

## Why eval rubrics matter

### For developers

**Because a rubric is the only thing that makes an eval number comparable across time, people, and models.** The YouTube Ads team's framing was that agent reliability is a function of three things: the agent's capabilities, its guardrails, and its evals. The eval is what lets you prove a change helped, run ablations, and hill-climb on quality instead of vibes.

But that only works if the measuring stick holds still. If two raters interpret "brand safe" differently, or if the rubric silently shifts between March and June, your regression numbers are noise wearing a lab coat. The team's own experience was blunt: early on, engineers and raters kept disagreeing among themselves about whether a case should pass — and every one of those disagreements is a rubric that was not specific enough yet.

There is a second, less obvious cost. The talk showed a slide joking that writing the eval is the tiny part and humans arguing over the rubric is the enormous part. That is not a joke about process overhead. It is a description of where the actual product thinking happens. Arguing about the rubric *is* deciding what the product is supposed to do.

### For AI agents

**Because increasingly, the rater is a model — and a model can only be as calibrated as the rubric it is handed.** The standard reference point is [Zheng et al.'s MT-Bench paper](https://arxiv.org/abs/2306.05685), which found GPT-4 agreed with human raters roughly 85% of the time, against a human-human agreement of about 81%. That is often quoted as "LLM judges are as good as humans." The more useful reading is the second number: **81% is the ceiling.** A judge cannot be calibrated against a team that does not agree with itself.

And judges are less stable than a single agreement figure suggests. A 2026 preprint, ["The Coin Flip Judge?"](https://arxiv.org/abs/2606.13685), measured repeated identical evaluations and reported pairwise preferences flipping **13.6% of the time on average**, a first-position bias, cross-judge agreement of only 76% (κ = 0.51), and — the finding most likely to change how you run evals — that semantically equivalent prompt templates changed the majority outcome in **25% of tested cases**.

Rephrasing the rubric changes the score. Which means the rubric is not documentation *about* your eval. It is a versioned input to it, and it deserves the same treatment as code.

## What the YouTube Ads team actually learned

### Vibe first, scale later

**Grading a handful of outputs by hand is the correct first step, even though it does not scale.** This is the counterintuitive one. The instinct is to build the comprehensive eval set on day one. The team's experience was the opposite: early on, failure patterns are obvious by eye, prompt tweaks can produce large gains, and you may still want to change the architecture outright. A large golden set at that stage is a liability — you end up recalibrating the eval and the agent simultaneously, and the numbers swing wildly for reasons you cannot attribute.

So: start with a few core tasks, the primary things you want the agent to be good at. Expand as the patterns clarify. And **test the negatives** — checking that the agent did *not* do the forbidden thing is as important as checking that it did the required thing, and it is the half teams routinely skip.

This lines up with what [Hamel Husain has argued](https://hamel.dev/blog/posts/evals/index.html) about evals generally: read your system's actual outputs one by one, categorise the failures you see, and only then define metrics against those failures.

### Human agreement is a prerequisite, not a metric

**Before you scale to external raters or an LLM judge, get your own team to agree.** The practical instruments from the talk are unglamorous and effective:

| Instrument | What it fixes |
| --- | --- |
| A clear rubric with **concrete examples** | Raters coming back with "I'm not sure how to score this" |
| Required **explanations**, not just verdicts | A pass rate that tells you nothing about where to improve |
| **Multi-dimension** scoring (accuracy, brand safety, …) | Outputs that are genuinely good on one axis and bad on another |
| Monitoring **human-vs-LLM disagreement rates** | A judge quietly drifting away from what you meant |
| **Rater training**, treated as real work | Everything above, at scale |

The disagreement-monitoring point deserves emphasis. The team ran a sampling pipeline comparing how a human expert scored a case against how the LLM judge scored it, and watched the agreement rate as a trend. That is the difference between "we validated the judge once" and "we know the judge is still aligned this week."

### Pass/fail hides the reason

**The single sharpest story in the talk is an agent that was explicitly and repeatedly told that legal disclaimers can never be removed, and removed them anyway.** Not in the common case — in edge cases. The categorical pass rate never surfaced it, because on aggregate the number looked fine.

The trace surfaced it immediately. The agent detected the disclaimer, stated in its own reasoning that it had found one, and then removed it. The example they demonstrated was a public-parks ad carrying a "paid for by" line at the bottom right; the agent stripped it.

The lesson generalises past that one bug: **if you want to know what your agent is doing, read what it thought it was doing.** A score tells you a rule was violated. A trace tells you whether the model missed the instruction, misread the scope of the instruction, or understood it and traded it away against something else. Those are three completely different fixes.

### Patterns, not runs

**Do not patch the prompt because of one bad run.** This is the trap the team called out most directly, and it is the one that is hardest to resist, because a single vivid failure feels like a mandate. But these are non-deterministic systems. One failing example is one sample from a distribution.

The discipline is to make sure the golden set contains **multiple examples of each pattern you care about**, and to act on the failure rate for the pattern rather than on the individual case. Which in turn means the patterns themselves need to be named and written down somewhere — otherwise "we've seen this before" lives only in whoever happened to be on the review rotation that week.

Alongside this: keep a test set you use sparingly, refresh it with production data, and invest in online evals so the distribution you measure still resembles the one you serve.

### Expect the rubric to move

**A rubric written before you have seen real outputs is a hypothesis.** This is not just folklore — it is a documented finding. Shankar et al.'s [*Who Validates the Validators?*](https://arxiv.org/abs/2404.12272) (UIST 2024) named it **criteria drift**: people need criteria in order to grade outputs, but grading outputs is what teaches them what the criteria should be. Some criteria turn out not to be definable a priori at all; they depend on the specific outputs observed.

Which is exactly why the "vibe first" advice works, and exactly why the rubric needs version history. The interesting question about a rubric is rarely what it says today. It is what changed in July, and which incident caused it.

## Which applications benefit most

1. **Content-generating agents with legal or policy constraints** — ads, financial copy, health information, anything where "must never remove the disclaimer" is a real sentence someone will read back in a deposition.
2. **Multi-dimension agent outputs** where one artefact is scored on several independent axes (accuracy, safety, tone, brand fit) and a single pass/fail collapses information you need.
3. **Customer-facing support and sales agents**, where the rubric encodes discretion limits — what may be refunded, discounted, promised — and drifts every time policy changes.
4. **Coding agents on long migrations**, where "did it follow our conventions?" is a rubric question and the conventions live in prose, not in a linter.
5. **Any team that has outgrown one person's judgement** — the moment a second rater joins, every ambiguity in the rubric becomes a measurable disagreement.
6. **Regulated domains with audit requirements**, where you must show not only the score but the criteria in force at the time it was produced.

## How MDflow fits

**MDflow is not an eval harness.** It does not score outputs, run judges, store traces, or compute agreement statistics. That belongs to your eval platform — Arize, Braintrust, Langfuse, an internal pipeline, whatever you already run.

What MDflow addresses is the layer underneath: **the rubrics, rating guides, golden-set definitions, failure-pattern notes and launch criteria are prose**, and in most teams that prose is scattered across a Google Doc, a Slack thread, a prompt string in a repo, and someone's memory of what was decided in March. When the definition lives in four places, the human rater and the LLM judge are no longer scoring against the same thing.

### What already lines up today

**The rubric as a plain markdown document.** Rubrics are written for two audiences — a person who has to apply them and a model that receives them as a prompt — and markdown is the format both read natively. MDflow stores them with no proprietary layer in between, and every document has a [raw `.md` twin](/markdown-ai) that an agent can fetch directly.

**Version history answers "what changed, and when?"** Every saved change captures the previous version across every write path — editor, HTTP API, and MCP — with line-by-line diffs and non-destructive restore. Given criteria drift, this is the feature that matters most: when a regression appears between two eval runs, the first question is whether the agent changed or the rubric did, and a diff answers it in seconds. (Version history is a Pro feature, is private to the document owner, and is deliberately **not** exposed over the API or MCP.)

**Folder descriptions make the right rubric retrievable.** Every folder carries a description stating what belongs inside it, and [`mdflow_get_context`](/docs/mcp) ranks those descriptions **above** folder names and document titles before returning matching bodies. A folder described as *"Rating rubrics and edge-case decisions for ad-generation evals — authoritative; raters and judges both read these"* is a retrieval signal you wrote, not one a model inferred. That is [why folder descriptions beat file names](/blog/folder-descriptions-agent-context).

**One definition, both raters.** The same workspace is reachable from Claude, ChatGPT, Cursor and Codex over the [remote MCP server](/docs/mcp) with OAuth or a Personal Access Token, and from scripts, cron jobs, CI and n8n over the [HTTP API](/docs/api). A judge prompt that pulls the rubric from the workspace at run time cannot drift from the copy a human rater is reading, because there is no copy.

**Comments put rater disagreement next to the rule it concerns.** Share a rubric — by public link or privately by email — and readers can attach a comment to a selected passage of the markdown source, highlighted inline, with the owner moderating from the normal editor. There are deliberately no reply threads, which suits this use: an argument about an edge case should not live in a comment forever, it should be promoted into the rubric body. That is still far more recoverable than the same argument in a Slack thread that scrolls away.

**The Document Log shows who moved the goalposts.** A cross-document activity feed records created, edited, shared and deleted events with the actor on every row, shown as `automated · <token name>` for anything arriving through the API or MCP. Click an edited row for a side-panel diff. If an eval number moved on a Tuesday, this tells you whether a rubric moved with it.

**Collections group the whole eval pack.** A rubric is rarely alone — it travels with a rating guide, a golden-set definition, and the launch criteria. [Collections](/use-cases/sharing) let you share that set as one read-only link with a cross-functional team or an external rating vendor, without giving anyone access to the rest of the workspace.

**Tasks are lines in documents.** Because [`/tasks`](/blog/markdown-task-management) aggregates ordinary `- [ ]` checkbox lines out of markdown bodies, "add three more negative examples for the disclaimer pattern" can live inside the rubric it belongs to and still show up on a real task list.

**Encryption where the examples are sensitive.** Golden sets often contain real customer data. Documents can be [client-side encrypted](/blog/encrypted-notes-app), which also means they are never scanned or indexed server-side.

### Where we are headed

Direction, not a dated commitment: we are most interested in making a written definition **easier for an agent to pull at the moment it needs it** — richer structured retrieval over folder descriptions, and better ways to reference a specific version of a document rather than whatever is current. For evals in particular, "which version of the rubric produced this score" is the question we would most like to make trivial.

## The bottom line

Eval rubrics are the part of agent evaluation that no tool ships for you. The harness is commodity; the definition of a pass is your product thinking, and it is written in English. The YouTube Ads team's findings all point the same direction: start by grading a few outputs by hand, get your own team to agree before you scale, collect reasons rather than verdicts, read the traces, act on patterns rather than runs, and expect the criteria themselves to move as you learn.

All of that produces documents. Keep them somewhere with version history, where a human rater and an LLM judge read the same file, and where the diff can tell you which one of them changed.

[Start free](/login) · [Connect an AI agent](/docs/mcp) · [Read the API docs](/docs/api)

## Frequently asked questions

### What is an eval rubric?

An eval rubric is the written definition of what counts as a pass, a fail, and an edge case for a given agent behaviour. It is what a human rater reads before scoring an output, and it is what an LLM judge is handed as a prompt. The eval harness runs the scoring; the rubric decides what the score means. Almost every disagreement about an eval number is really a disagreement about the rubric.

### Should you write a big eval set before or after you start building the agent?

After. The YouTube Ads team at Google recommends starting with intuition-based grading of a small number of outputs, because at that stage failure patterns are obvious by eye and prompt or architecture changes are still cheap. Jumping to scaled human raters too early means recalibrating the eval and the agent at the same time. Start with a handful of core tasks, include negative cases, and expand the golden set as the failure patterns become clear.

### Why does human-human agreement matter for LLM-as-a-judge?

Because it is the ceiling. In the MT-Bench paper, GPT-4 agreed with human raters about 85% of the time, but human raters only agreed with each other about 81% of the time. If your own team cannot agree on whether an output passes, an LLM judge cannot be calibrated against them in any meaningful way. Getting the rubric to the point where two humans score the same output the same way is a prerequisite, not a nice-to-have.

### Why is a pass-fail rate not enough to debug an agent?

Because a rate tells you that something failed, not why. The YouTube Ads team described an agent that had been told repeatedly that legal disclaimers must never be removed, and it still removed them in edge cases. The categorical pass rate never surfaced it. The reasoning trace did: the agent detected the disclaimer, announced that it had found one, and removed it anyway. Rater explanations and agent traces are what turn a number into a fix.

### What does MDflow do for AI agent evals?

MDflow is not an eval harness and runs no scoring. It holds the prose that evals depend on: rubrics, rating guides, golden-set definitions, launch criteria and failure-pattern notes, as plain markdown documents with version history and line-by-line diffs. The same document a human rater reads can be retrieved by an LLM judge over MCP or the HTTP API, so both sides are scored against one definition instead of two copies that quietly drift apart.

## Further reading

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685) — Zheng et al., NeurIPS 2023; the source of the ~85% judge-human / ~81% human-human agreement figures
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://arxiv.org/abs/2404.12272) — Shankar et al., UIST 2024; the criteria-drift paper
- [The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation](https://arxiv.org/abs/2606.13685) — 2026 preprint on judge flip rates, position bias, and prompt-template sensitivity
- [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/index.html) — Hamel Husain, on reading outputs before defining metrics
- [Long-horizon agent evals](/blog/long-horizon-agent-evals) — the other half of the problem: what happens when an eval has to run for a simulated year
- [Agent observability: from production signal to PR](/blog/agent-observability-self-improving-loop) — what to do once a trace has told you what broke
- [Separate the task from the model](/blog/separate-the-task-from-the-model) — why specs and evals outlive the implementation between them
- [MDflow MCP documentation](/docs/mcp) and [HTTP API documentation](/docs/api)
</content>