$cat pairwise-evals-vs-absolute-scores.md

Pairwise Evals: Your Judge Is Scoring the Vibe

November 4, 202617 min readby MDflowview as .md

Two glowing emerald wireframe film frames facing each other across a bright central fulcrum on a dark grid, with a third unlit frame sitting apart in the background

Video generation got good enough to be free, and judging it did not. That gap is now the interesting part of the problem: the hard thing was never making a video, it was deciding whether the video you made is any good.

Maor Bril, who has been at Character.AI for a bit over two years, took the AI Engineer stage in July to describe what his team built to answer that question. The talk is nominally about video. The part worth stealing is a claim about evaluation protocol that generalises to anything a model produces — and that the research literature partly contradicts, in a way that makes the advice sharper rather than weaker.

TL;DR — Pairwise evals ask which of two outputs is better instead of putting one output on a 1-to-10 scale, and Character.AI found that relative verdicts generalise where absolute scores do not. But a COLM 2025 paper found pairwise preferences flip about 35% of the time under an irrelevant distractor, against 9% for absolute scores, and refuse to call a tie on deliberately matched pairs. Both findings have the same root cause: a judge that has not been told which axis to score will score gloss. Name the axes, write them down somewhere both your human raters and your judge prompt read from — which is what a markdown workspace like MDflow is for — and the protocol becomes a second-order choice.

What pairwise evals are

A pairwise eval asks a judge which of two outputs is better on a stated criterion, rather than asking it to place a single output on an absolute scale. No rater has to hold a shared definition of "a 7" in their head; they only have to express a preference. A ranking is reconstructed afterwards from many such comparisons.

The two prompt shapes look like this:

Absolute (pointwise):
  Given this criterion, score this response from 1 to 10.
  → 7

Pairwise (comparative):
  Given this criterion, which response is better: A or B? (C = tie)
  → B

Bril's argument for the second shape was about agreement, and he put it as a room test: ask a dozen people to rank one video from 1 to 10 on storytelling and you get a six, a five, a four and an eight. Show the same people two videos and ask which tells the better story, and most of them agree. That is his observation about human raters rather than a cited study, but it is the intuition behind every comparison-based ranking system, including Chatbot Arena, which fits a Bradley-Terry model to head-to-head votes precisely so that it never needs a shared absolute scale.

So Character.AI trained on pairs: A versus B, not 1 through 10.

Why frame-level metrics ran out

The tools inherited from the text and image era measure frames, and the properties that make a video good or bad exist across time.

CLIPScore compares a single frame against the prompt that produced it. LPIPS measures perceptual distance between frames, which is a reasonable proxy for drift. Combine them and you can answer "does this frame match the prompt" and "did the picture change too much between these two frames." You cannot answer the question anyone actually has, which is whether the thing told the story it was supposed to tell.

Bril listed what that leaves uncovered: does the story hold; does the physics make sense, so the character walks down the stairs rather than hovering; does the same character stay the same character across shots; does the pacing reflect that people take time to get from one place to another; and when audio is added, does the door-slam land on the frame where the door slams.

The research agrees, and more pointedly than the talk needed. Frame-based measures evaluate frames independently and so cannot capture temporal coherence, and CLIP-based temporal consistency can be gamed outright: a video that simply repeats one frame scores perfectly on it. Hold that thought, because it is about to reappear as a production incident.

The failure that names the problem

Character.AI's first attempt at a fast judge shipped, and it was wrong. Bril's phrasing is the useful one: it was wrong in a very confident way.

The example he put on screen was a four-second clip of a static character — the camera does not move at all — which his model scored 9.2 on camera work. Elsewhere it declared that the physics looked great, about ghosts hovering and people flying.

That is not a bug in the architecture. It is a statement about what the training data taught the model to look at. In his words, it scored the vibe rather than the axes: it had learned to detect coherent-looking video and the absence of generation artifacts — the gloss — and had learned nothing about whether a story was told.

The fix was in the dataset, not the protocol. He started pairing real footage against AI footage, which is the obvious move he had deliberately avoided first, because the obvious risk is that you train an AI detector instead of a quality model. Two controls kept that from happening:

Identical encoding on both sides, so no compression artifact could stand in for "this one is the AI one."
The same annotation method on both sides, so every axis was labelled the same way regardless of where the clip came from.

With those in place it worked. The lesson is not "use real footage." It is that the thing you must get right is what distinguishes the two sides of every pair — because whatever that is, in aggregate, is what your model will learn to measure.

What the research says about "don't score, compare"

Here the literature pushes back, and it is worth taking seriously rather than quoting only the half that agrees.

"Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation" (Tripathi, Wadhwa, Durrett and Niekum; arXiv April 2025, revised August 2025, listed as published at COLM 2025) ran a clean experiment. Take a response, add a distractor feature — defined in the paper as an extraneous feature irrelevant to the criterion being judged — to the response the judge originally liked less, and see whether the verdict moves. The quality did not change. Only the gloss did.

The three distractors were assertiveness, prolixity (more words, fancier words) and sycophancy. The judges were LLaMA 3.2-3B, LLaMA 3.3-70B, Qwen 2.5-3B and Qwen 2.5-72B.

The headline, in the paper's own words: "Pairwise preferences flip in about 35% of the cases, compared to only 9% for absolute scores." Broken out by distractor on MT-Bench (absolute / pairwise): assertiveness 9.0 / 35.5, prolixity 9.25 / 35.2, sycophancy 9.7 / 34.9. The effect held across model sizes.

The second result is the one that should change how you design a pair set. On a dataset built so that both responses were matched in quality by design, the judges still refused to call it:

Judge	Ties, pairwise	Ties, absolute
LLaMA 3.2-3B	2.4%	84.6%
LLaMA 3.3-70B	3.2%	93.2%
Qwen 2.5-3B	6.5%	86.5%
Qwen 2.5-72B	7.3%	88.0%

Asked to compare two things that were deliberately the same, a pairwise judge picked a winner more than nine times out of ten. The paper's conclusion is that absolute scoring is more robust to this manipulation and more honestly reflects intentional ties.

Now put the two sources side by side. The paper's distractors are gloss. Character.AI's V1 failure was gloss. Assertiveness, verbosity and sycophancy are to a text judge what cinematic polish and artifact-free rendering are to a video judge: the surface property that correlates with quality often enough to be learned in its place. One was measured in a lab and named distracted evaluation; the other was hit in production and called scoring the vibe. They are the same failure.

The protocol follows the size of the gap

The two findings only look contradictory if you ignore what was in each pair.

Character.AI manufactured its badness: real footage against AI footage, corrupted clips against clean ones, deliberately generated slop. Those pairs contain an enormous, genuine quality gap, and a relative verdict on a large real gap is easy and stable. The paper's pairs were the opposite case by construction — equal quality, differing only in an irrelevant feature — and that is exactly where a protocol that must name a winner will invent one.

A usable rule, stated as a design decision rather than a law:

Large, real gaps → compare. Model-versus-model, before-and-after a prompt change, a regression check against a known-good generation. Run both orderings and average, which is the standard mitigation for position bias.
Small or matched gaps, and anything with a ground truth → score. Instruction-following, factual correctness, policy conformance. A protocol able to say "these are the same" is worth more than a forced ranking.
Either way, score the axes you named. Both failure modes above are a judge filling an unspecified criterion with the most available signal. The protocol decides how the verdict is expressed; the rubric decides what gets measured. That is the argument of eval rubrics for AI agents, arriving here from a different direction.

Bril's own second takeaway says the same thing more bluntly: score the real axes you care about, and do not expect them to miraculously appear.

Put the eval inside the generation loop

The third takeaway is where the engineering lives. An eval that runs after generation tells you that you wasted the money; an eval inside the loop stops you spending it.

Long AI videos are not generated long. They are assembled from many short clips, which are themselves generated from keyframes. A character who drifts between keyframe one and keyframe two gets baked into every second of footage produced from them. Catching that drift at the still-image stage costs almost nothing. Catching it after a four-minute sequence is assembled costs the whole sequence.

The constraint on moving evaluation earlier is latency, and this is where the architecture gets interesting. Character.AI's calibrated setup — frame metrics, plus frontier models as judges, plus periodic human annotation sessions feeding corrections back into the judge prompts — produced good scores slowly and expensively. So they distilled that committee into a single small vision-language model (a Qwen model, per the Q&A, chosen partly because the team already had good results post-training that family). It scores a roughly 15-second clip in about three seconds. A larger model scored better and was rejected anyway, because the improvement did not justify the latency.

Two design details are worth copying regardless of domain:

The output is a reason, not a number. The model says why a clip scored low — an extra limb, broken physics, audio out of sync — because a score alone cannot drive a retry. On audio specifically, the approach described on stage was not sound recognition: the model learns to identify key frames, and if the prompt says a door slams, it checks for a spike in the audio at that timestamp. It does not know what the sound is. It knows something loud should happen there.

The pipeline became an agentic workflow. Fixed pipelines work when the use case is narrow, and users are not narrow — they arrive with their own characters, their own images, their own voice, and the pipeline drifts. Handing the agent tools to validate its own output lets it adapt and retry. That only works if the validator is genuinely independent of the generator, which is a constraint worth stating explicitly: the generator cannot be the validator. A fast distilled judge trained on separately annotated data qualifies. A second call to the same model that produced the clip does not.

Which teams this applies to

Generative media — video, image, audio, 3D. The frame-versus-story distinction is literal here, and the cost asymmetry between early and late detection is largest.
Agentic coding tools. The "gloss" is code that looks idiomatic. The axes are whether it compiles, passes, and does the thing asked.
Any product with an LLM judge in CI. If your gate is a judge, your gate inherits the judge's protocol bias. Quality gates for AI agents is the companion argument about layering them.
RAG and summarisation. Fluency is the distractor. A confident, well-structured summary of the wrong document scores well on every axis you forgot to name.
Teams with human raters in the loop. Both findings reduce to rater calibration, which is a documentation problem before it is a tooling problem. See domain expert evals.

How MDflow fits

MDflow is not an eval harness. It does not score outputs, run judges, store traces, hold video, or compute agreement statistics. That is your eval platform's job.

What this talk makes visible is a different gap. Almost every decision above is prose that has to stay stable across a model, a judge prompt, a human annotation session and a postmortem — and in most teams that prose is in a Slack thread, a slide, and a string literal in a repo. MDflow is a markdown workspace built so that an agent and a person read the same copy.

What already lines up today

The axis definitions, written once. "Pacing" means nothing to a judge until someone writes down what it means, with examples of a four that is not a six. Those definitions are markdown, every document has a raw .md twin an agent can fetch directly, and there is no proprietary layer between the prose a human reads and the bytes a judge prompt pulls.

The pairing protocol is a document, not folklore. "Identical encoding on both sides, same annotation method on both sides" is the methodological decision that kept Character.AI's model from becoming an AI detector. It is exactly the kind of rule that is obvious to whoever made it and invisible to whoever builds the next dataset. Write it down next to the axes it protects.

Version history answers "did the model change, or did the definition?" Every save captures the previous version across every write path — editor, HTTP API and MCP — with line-by-line diffs and non-destructive restore. When a score moves between two runs, that is the first question, and a diff answers it in seconds. (Version history is a Pro feature, private to the owner, and deliberately not exposed over the API or MCP.)

One definition, both raters. The same workspace is reachable from Claude, ChatGPT, Cursor and Codex over the remote MCP server, and from CI, cron and n8n over the HTTP API. A judge prompt that pulls its axis definitions at run time cannot drift from the copy a human annotator reads, because there is no second copy.

Folder descriptions make the right rubric retrievable. Each folder carries a description of what belongs in it, and mdflow_get_context ranks those descriptions above folder names and titles — a retrieval signal you wrote rather than one a model guessed. That is why folder descriptions beat file names.

The postmortem is a document, not a slide. "Scored 9.2 on camera work for a static shot" is the single most useful artifact Character.AI produced, because it is a concrete, re-runnable example of the failure. Keep those as cases in the workspace and your next dataset starts with the last one's known holes.

Where we are headed

Direction, not a dated commitment: the thing we most want to make trivial is pinning a specific version of a definition to a specific run, so "which rubric produced this score" stops being an archaeology exercise. Richer structured retrieval over folder descriptions is the other half of it.

Stated limits: MDflow computes nothing, gates nothing, and stores no media. It is where the criteria live, not where they run.

The bottom line

"Don't score, compare" is good advice with a condition attached. Compare when the gap between your two candidates is real and large, because a relative verdict on a large gap is the easiest judgment any rater can make. Score when the candidates might legitimately be equal, because a protocol that cannot say "tie" will manufacture a winner more than nine times in ten.

Neither protocol saves you from the actual failure, which is that an unspecified criterion gets filled with whatever surface signal is most available — assertiveness in text, polish in video. Name the axes. Write down how the pairs are constructed. Keep the postmortems. Then put the result close enough to generation that it can stop a mistake instead of reporting one.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What are pairwise evals?

A pairwise eval asks a judge which of two outputs is better on a stated criterion, instead of asking it to put a single output on a 1-to-10 scale. The individual verdicts are relative, so no rater has to hold a shared absolute scale in their head, and a ranking is recovered afterwards from many comparisons. It is the protocol behind preference-based ranking systems such as Chatbot Arena, which fits Bradley-Terry to head-to-head votes.

Is pairwise comparison better than absolute scoring for LLM judges?

It depends on how large the real quality gap is. A COLM 2025 paper by Tripathi, Wadhwa, Durrett and Niekum found that adding an irrelevant distractor feature to the weaker response flipped pairwise preferences in about 35 percent of cases against only 9 percent for absolute scores. On deliberately quality-matched pairs, pairwise judges called a tie just 2.4 to 7.3 percent of the time while absolute scoring gave identical scores 84.6 to 93.2 percent of the time. Pairwise is strong when the gap is genuine and large, and it invents a winner when the gap is not there.

What does it mean that an eval scores the vibe?

It means the judge learned to react to surface polish rather than to the property you named. Character.AI's first video quality model scored 9.2 on camera work for a shot in which the camera never moved, and praised the physics of hovering characters. The same failure has a formal name in the LLM judge literature: distracted evaluation, where an extraneous feature such as assertiveness, verbosity or sycophancy changes the verdict although the thing being judged did not change.

Why do frame-level video metrics miss bad video?

Because CLIPScore and LPIPS evaluate frames, and the properties people care about exist across time. Neither can represent story, pacing or whether a sound lands on the right beat. Worse, frame-consistency proxies reward stillness: research notes that a video repeating one frame can score perfectly on temporal consistency, which is close to the exact failure Character.AI hit in production.

Should evals run inside the generation loop?

Where a mistake compounds, yes. Long AI videos are assembled from many short clips, so a character that drifts between two keyframes gets baked into everything generated from them, and checking at the keyframe stage costs a fraction of regenerating a finished sequence. The constraint is latency, which is why Character.AI distilled a committee of metrics and frontier judges into a small vision-language model that scores a roughly 15-second clip in about three seconds and reports which axis failed rather than a bare number.