---
title: "AI Pull Request Review: Prove, Judge, Escalate"
description: "AI pull request review works when you split it three ways: scripts prove what is provable, an agent judges what needs context, and people own the protected calls."
author: "MDflow"
date: 2026-11-11
reading_time: "17 min"
canonical_url: https://mdflow.cz/blog/ai-pull-request-review
md_url: https://mdflow.cz/blog/ai-pull-request-review.md
---

# AI Pull Request Review: Prove, Judge, Escalate

*Published November 11, 2026 · 17 min read*


A pull request changes a schema in a production data pipeline. CI is green. The tests pass, the linter is happy, every check reports success. Is it safe to merge? That question is the whole case for **AI pull request review** — and it is the one question an AI reviewer is least equipped to answer by itself.

That gap — between *CI passed* and *this change was actually safe* — is the one Chetas Joshi's team on Robinhood's data platform has to close every day. In a talk published on 10 November 2026, [they described what happens to that gap](https://www.youtube.com/watch?v=MQiojLnK-E0) when coding assistants make pull requests cheap to open: the PRs arrive at machine speed, and the number of people who carry enough context to say what is safe to merge does not grow at all.

> **TL;DR** — Don't build *an* AI reviewer. Split review into three duties with three owners: deterministic **scripts prove** what a source of truth can prove, an **agent judges** what needs context and cannot be proven, and **people own** a fixed set of protected calls the agent is never allowed to clear. Robinhood reports about 60% of data pipeline PRs now merging with no human review, roughly 75% less manual review effort, and a miss rate near 2%. The part that decides whether this works is not the model — it is the written rules and context the agent reads, which get revised every time a human corrects it. That ruleset needs a durable, versioned home that both people and agents can read, which is what a markdown workspace like [MDflow](https://mdflow.cz) is for.

## What is AI pull request review?

**AI pull request review is using an agent to read a PR, reason about its risk, and either approve it or escalate it to a person.** The interesting design question is not whether an agent *can* do that. It is what you refuse to let it do.

The answer Robinhood landed on is three duties, three owners:

| Layer | Owner | Job | Failure mode it owns |
| --- | --- | --- | --- |
| 1 | Deterministic scripts | Prove facts against a live source of truth | Check fails → CI red, PR blocked, no model involved |
| 2 | Agent | Judge the risks a script cannot see | Scores risk, comments with reasoning, approves or escalates |
| 3 | Humans | Own the protected calls | Decides; the agent only surfaces |

Both layers review the same PR, and merging requires both: passing checks **and** an approving review. The scope here matters — these are PRs to data pipelines: schema changes, pipeline code, configuration, and grants that give someone access to data.

## Why scripts come before the agent

**Because some questions already have an authoritative answer, and a judgment call is the wrong tool for them.** Joshi's framing is blunt: use AI for judgment, not to rediscover facts a script can look up. That is partly a reliability argument and partly a cost one — every fact a script proves is review work that never reaches a model.

The four layer-one checks in their system:

```text
data contracts        → does the changed schema match the schema registry?
required tags         → does the table have an owner, and is PII labelled
                        in the catalog?
lineage blast radius  → does any registered downstream job break?
performance           → does it still run, and still make its freshness SLA,
                        at production data volume in staging?
```

Each one hits a live system the code alone cannot tell you about. If any fails, there is no debate: CI goes red like any other failing check and the PR is blocked until it is fixed.

Note what the third one can and cannot see. The lineage check walks every downstream job *the lineage tool has on record*. Its view of "downstream" is exactly as complete as that record — no more. Hold on to that.

## Why the agent comes after

**Because the registered picture is not the whole picture.** The agent's rules are configured per change type and written knowing what layer one already covers, so it spends its attention on what the scripts structurally cannot catch: query anti-patterns, sensitive-data asks, and consumers that were never registered anywhere.

Its toolset is deliberately narrow. It reads, searches and lists files across the repository. It cannot query the live catalog, the lineage tool, or production state — live facts stay layer one's job. It reads the PR description and the commit messages for intent, under a rule worth stealing outright: **intent never explains away a real finding.** If the description says it is safe and the code says otherwise, the code wins.

### The PR that was green and still wrong

The worked example from the talk is a two-part schema change: add a new column, and widen an existing one from `integer` to `bigint`. Widening a type is about the safest schema change there is.

Layer one cleared it completely. The new column really does exist upstream in the registry. The table has an owner and its PII is labelled. Every registered consumer of the widened column survives the change. Performance passes.

Then the agent walked the repository and found a job that lineage had never heard of. It reads the widened column, and it writes that value into *another* table — where the column is still an `integer`. The first value that arrives too big for an integer, that write fails.

A valid schema change, green in CI, passing every registered check, with a downstream job quietly waiting to break. The agent flagged the whole chain with its reasoning, scored the risk high, and escalated.

Why had lineage missed it? Because enterprise data estates are messy: workloads run across different query engines, each engine has to publish its own lineage, and if one does not publish, that relationship is simply absent. There is no single catalog that automatically sees everything — which is exactly the problem open standards like [OpenLineage](https://openlineage.io/) and [data contracts](https://datacontract.com/) exist to chip away at, and exactly why a registry-only view stays incomplete in the meantime.

## "None found" is not "none exist"

**The most transferable rule in the whole design is an honesty constraint on the agent.** The code walk is genuinely useful and it still cannot see everything. Ad-hoc queries someone runs by hand, dashboards, anything living outside the codebase — a repository search will never find those consumers no matter how carefully it looks.

So the agent is not permitted to report an absence as a fact. If it finds nothing, it does not get to say *there are no downstream consumers*. It says it **found none in the places it searched**. The agent never gets to claim an all-clear it cannot support.

This is a small wording rule with a large blast radius. An agent that reports "no consumers found" and an agent that reports "no consumers exist" produce the same text length and completely different decisions downstream — and the second one is the kind of confident absence claim that [routes a team around its own guardrails](/blog/why-ai-agents-route-around-guardrails).

## The calls the agent never gets to make

Most changes the agent clears on its own. But three categories are **flag, never clear** — no matter how confident it sounds:

1. **Grants touching sensitive data.** Who can reach sensitive data is a decision a person owns every time.
2. **Migrations that look like they change intent.** If a change alters what the data *means* rather than just how it is moved or stored, a person decides.
3. **Low-confidence verdicts.** The agent does not get to round up to "approve" just because nothing concrete came back wrong.

In all three the agent's job stops at surfacing the risk and its reasoning. This is the same separation-of-duties instinct behind [generator/validator separation](/blog/generator-cannot-be-the-validator) and [layered quality gates](/blog/quality-gates-for-ai-agents): the thing that produces a verdict should not also be the thing that is allowed to act on it unsupervised.

## Two kinds of mistake, and the one teams forget to count

**A reviewer has two failure modes, and measuring only one of them produces a useless gate.**

|  | Agent approved | Agent escalated |
| --- | --- | --- |
| **Fine PR** | ✅ correct | ❌ false escalation — costs a person's time |
| **Bad PR** | ❌ miss — the change ships | ✅ correct |

A gate that escalates everything avoids every miss and reduces review load by exactly zero. It has moved the work, not removed it. So false escalations have to be measured the same way misses are.

Robinhood's reported numbers, as of 31 August 2026 and self-reported by one team:

- **Miss rate: about 2%**, mostly from gaps in the rules rather than failures of judgment.
- **No clean false-escalation number** — but the dominant cause is identified: *missing context*.
- About **60%** of data pipeline PRs merge with no human review.
- About **75%** less manual review effort, worth roughly **132 engineering hours a month**.
- Incidents tied to these PRs down about **30%**.

The phrasing of the two error sources is the thing to carry away: **misses give you new rules; false escalations show you context the reviewer was missing.** Both outputs are prose, and both have to go somewhere a future review will actually read.

## The feedback loop is manual, and that is the honest part

Nothing here retrains itself. The agent posts its verdict and reasoning as review comments; a person replies agreeing or disagreeing and **explaining why**; those replies are collected daily; once a week the on-call reads them and updates the review rules by hand. Robinhood is explicit that this is manual today, and that an idea to let an incident-investigation agent help with the weekly pass exists but is not built.

Two examples show what each error type teaches.

**A correction (false escalation).** A column was marked as personal information in the source database, but the ingestion pipeline encrypts it before it lands in the lake — so the PR proposed labelling it non-sensitive there. The agent flagged it: upstream says PII, so the label looked wrong. The author pushed back and asked it to look at how similar columns on other tables were already classified rather than going purely off the upstream tag. A human agreed with the author. The next similar column stops costing an escalation — **but** the talk adds the caveat that this depends on that specific pipeline encrypting before the write. It is not a blanket rule that encryption makes data non-sensitive.

**A miss.** Columns in an open copy of some data were re-tagged as personal information, which moved them into a more restricted copy and removed them from the open one. The re-tag looked fine and merged. Nobody had checked whether anything downstream was still reading those columns from the open copy. Something was, and it broke. That became a rule: when a column is re-tagged as sensitive, search the codebase for anyone still reading it from the copy that is about to lose it.

They are also candid about the architecture they would change. Today the two layers are separate systems, triggered independently and running in parallel, and **neither sees the other's pass/fail or outage state**. The agent knows what the scripts are configured to cover, not what happened when they ran, and it cannot call a layer-one check as a tool when a judgment needs a live fact. A single entry point — a classifier that looks at the whole PR, with the layer-one checks exposed to the agent as callable tools — is a proposal, not something running.

## Which applications benefit most

1. **Data platform and pipeline changes**, where "the code is correct" and "this is safe to merge" are genuinely different questions.
2. **Infrastructure-as-code and access grants**, where the blast radius lives in a system the diff never mentions.
3. **Any repository where coding agents open PRs faster than the owners can read them** — which is most of them now.
4. **Regulated workflows** that must show not just a result but who approved it and on what basis.
5. **Monorepos with cross-team consumers**, where the person who can judge impact is three teams away.
6. **Migration programmes**, where mechanical changes and intent changes look identical in a diff.

## How MDflow fits

**MDflow does not review pull requests.** It is not a CI system, a code reviewer, or a policy engine, and it will not be. What this architecture needs — and what quietly decides whether it works — is a durable home for the layer that is neither script nor model: the written rules, the escalation policy, and the accumulated corrections that tell an agent what *this* organisation considers risky.

That layer is prose under constant revision. Every week it changes. Every miss adds to it. And it has two audiences that must read the same copy: the on-call engineer editing it, and the agent loading it.

### What already lines up today

**The rules stay in plain markdown, readable by both audiences.** Review rules, escalation policy, the "flag, never clear" list, the per-change-type guidance — these are documents. MDflow stores them as plain markdown with no proprietary layer, and every document has a [raw `.md` twin](/markdown-ai) with YAML frontmatter that an agent can fetch directly.

**Agents read the current version, not a stale copy.** A reviewer connects over the [remote MCP server](/docs/mcp) with OAuth or a named Personal Access Token, or over the [HTTP API](/docs/api) from CI. That matters here specifically: a ruleset edited weekly and copy-pasted into a prompt string starts drifting the day after it is written.

**Folder descriptions are the retrieval signal.** Every folder carries a description of what belongs in it, and `mdflow_get_context` ranks those descriptions **above** folder names and document titles before returning matching bodies. A folder described as *"Review rules for data pipeline PRs — authoritative; layer one already checks contracts, tags, lineage and performance"* tells the agent what not to re-derive. That is [why folder descriptions beat file names](/blog/folder-descriptions-agent-context).

**Version history makes a rule change reviewable.** Every saved change captures the previous version across every write path — editor, API and MCP — with line-by-line diffs and non-destructive restore. When a rule is added after a miss, the diff *is* the record of why the reviewer's behaviour changed. (Version history is a Pro feature, private to the document owner, and deliberately not exposed over the API or MCP — an agent can write, but it cannot rewrite the record of what it wrote.)

**The Document Log names the actor.** A cross-document activity feed records created, edited, shared and deleted events, and anything arriving over the API or MCP is attributed as `automated · <token name>` rather than blurring into "you" — the same discipline as [scoped tokens for agent authorization](/blog/agent-authorization-scoped-tokens).

**Comments anchor the weekly pass to the clause in question.** Readers of a shared document can attach a comment to a selected passage, highlighted inline. When the argument is about whether the encryption caveat belongs in rule 4, that is where the argument should live.

**Sharing crosses the team boundary without an account.** Public read-only links, [collections](/use-cases/sharing) that bundle several documents into one link, and per-person email-scoped access mean a consumer team can read the rules that govern their tables without being onboarded into your workspace.

### Where we are headed

Direction, not a dated commitment. The thing that fits this problem best is **referencing a document at a specific version** — so a review comment, an incident write-up, or an agent's own note can point at *the rules as they stood on 31 August*, not at whatever they say today. Alongside that, richer structured retrieval over folder descriptions, so an agent reviewing a grant reliably finds the policy that governs it rather than the one next to it.

## The bottom line

The instinct when review becomes a bottleneck is to put a model in front of it. The better move is to decide, explicitly, which of three things each question is: a **fact** a script can prove against a source of truth, a **judgment** that needs context, or a **call** that belongs to a person no matter what the model says.

Get that split right and the model stops being the risky part. What is left is the written context the agent reasons over — the rules, the exceptions, the corrections, the things your organisation learned the expensive way. That is the asset. Keep it in plain markdown, versioned, with a log of who changed what, and readable by the agent that has to apply it at three in the morning.

[Start free](/login) · [Connect an AI agent](/docs/mcp) · [Read the API docs](/docs/api)

## Frequently asked questions

### What is AI pull request review?

AI pull request review is the use of an agent to read a pull request, reason about its risk, and either approve it or escalate it to a person. In the design Robinhood's data platform team presented, it is deliberately only one of three layers: deterministic scripts prove facts against a source of truth, the agent judges what needs context and cannot be proven by a script, and a human owns a fixed set of protected decisions that the agent is never allowed to clear.

### Why not let the AI agent check everything?

Because some questions already have an authoritative answer, and spending a judgment call on them is both less reliable and more expensive. If a schema registry can say whether a column exists upstream, a script should ask it; a failed check turns CI red with no model involved. Robinhood's stated rule is to use AI for judgment, not to rediscover facts a script can look up — which also keeps the cost of running the reviewer down.

### What is a false escalation, and why measure it?

A false escalation is a fine pull request that the reviewer sent to a human anyway. It is the mirror image of a miss, which is a bad pull request the reviewer approved. Measuring only misses rewards a gate that escalates everything, and a gate that escalates everything has not reduced review load at all — it has just moved the work. Robinhood reported a miss rate of about 2% as of 31 August 2026 and no clean false-escalation number, while noting the dominant cause of false escalations was missing context.

### Which pull requests should an AI reviewer never approve on its own?

Three categories in the Robinhood design: grants that touch sensitive data, migrations that look like they change what the data means rather than how it is moved or stored, and any verdict the agent itself is not confident in. For those the agent's job stops at surfacing the risk and its reasoning. Everything outside those categories it can and does approve by itself — about 60% of data pipeline pull requests merged with no human review.

### Where should the rules an AI reviewer follows live?

Somewhere durable, versioned, and readable by both the on-call engineer who edits them and the agent that loads them — not inside a prompt string in a CI config. Each miss becomes a new rule and each false escalation reveals context the reviewer was missing, so the ruleset is a document under constant revision. MDflow keeps that kind of document as plain markdown with version history, a log naming which agent touched what, and MCP or HTTP API access so the reviewer reads the current version rather than a copy.

## Further reading

- [60% of Our Data PRs Now Merge With No Human Review](https://www.youtube.com/watch?v=MQiojLnK-E0) — Chetas Joshi, Robinhood, published 10 November 2026 (the source talk)
- [OpenLineage](https://openlineage.io/) — the open standard for collecting lineage across engines, which is the gap the code walk was compensating for
- [Data contracts](https://datacontract.com/) — background on the Open Data Contract Standard and its tooling, for the general shape of a schema-registry check (the talk names no specific standard)
- [State of AI-assisted Software Development 2025](https://dora.dev/research/2025/dora-report/) — DORA, on AI as an amplifier of an organisation's existing strengths and weaknesses
- [The Generator Cannot Be the Validator](/blog/generator-cannot-be-the-validator) — why the thing that writes an artefact must not be the only thing that judges it
- [Quality Gates for AI Agents](/blog/quality-gates-for-ai-agents) — the Swiss cheese model, and when a second gate buys you nothing
- [AI Code Verification: The New Bottleneck for Agents](/blog/ai-code-verification-loop) — the wider version of this problem
- [Behaviour specs for AI agents](/blog/behavior-specs-for-ai-agents) — writing down what "correct" means before an agent has to judge it
- [MDflow MCP documentation](/docs/mcp) and [HTTP API documentation](/docs/api)

