skip to content
$mdflow
$cat quality-gates-for-ai-agents.md

Quality Gates for AI Agents: The Swiss Cheese Model

16 min readby MDflowview as .md
Four emerald wireframe panels punched with scattered holes, standing one behind another in deep perspective, with a beam of light threading through a chain of aligned holes, on a dark terminal grid background

Most teams shipping agents have one validation step. The agent produces something, a check looks at it, and if the check passes, it goes out. When something bad reaches production anyway, the instinct is to improve the check: a sharper prompt, a bigger model, another criterion bolted onto the rubric.

Safety engineering settled this argument decades ago, in industries where the cost of being wrong was measured in lives. The answer is not a better check. It is more checks, arranged so their weaknesses do not overlap — and the discipline to accept that each one, individually, will keep letting things through.

TL;DR — Quality gates for AI agents should be designed like James Reason's Swiss cheese model: assume every gate has holes, stack several, and accept deliberate redundancy between them. The catch is that redundancy only pays off when the gates fail independently; three gates driven by the same model and the same prompt share their blind spots and protect you roughly as well as one. That makes each gate's criteria a first-class artifact — separately named, separately versioned, and readable by both the humans who argue about it and the agents that enforce it, which is exactly what MDflow stores.

What is a quality gate for an AI agent?

A quality gate is a check that sits between an agent producing an output and that output reaching a user, and that has the authority to block it.

That makes it a different thing from an eval, even when the two share a rubric. An eval runs over a batch, offline, and produces a number that tells you how the system is doing. A gate runs on one live output, in the critical path, and produces a verdict: ship, retry, or drop. Evals inform the roadmap. Gates decide what a customer sees today.

The distinction matters because the two have opposite failure preferences. An eval that is slightly miscalibrated gives you a slightly wrong number. A gate that is slightly miscalibrated either blocks good work — which costs you coverage — or waves bad work through, which costs you trust. Both are real costs, and which one you optimise against is a product decision, not a modelling one.

Jai Chopra and Soumya Gupta of Uber's computer vision team described a production system built around this distinction in their AI Engineer talk, Building Closed-Loop Evals for a Multimodal Agent at Scale (July 2026). The use case is food photography on Uber Eats: small merchants often lack the time, the know-how and the budget for professional shoots, and a photo is frequently the first signal a customer gets about a restaurant. The team put the delivery marketplace at roughly a $90B annual run rate across 10,000 cities, growing about 20% year over year, with millions of items added every month.

The constraint is what makes it interesting. Customers want photos that look real, and a meaningful fraction of them distrust anything that looks AI-generated. So the system has to improve images without inventing food that is not there, without flattening every merchant into the same house style, and without touching photos that were fine already.

Why one gate is never enough

Because every gate has holes, and you do not get to know where they are in advance.

This is the Swiss cheese model, proposed by the British psychologist James Reason in his 1990 book Human Error and introduced to a much wider audience through his 2000 BMJ paper Human error: models and management. Picture each defensive layer in a system as a slice of Swiss cheese. In an ideal world each slice is solid; in reality each one is full of holes, and the holes move. A hazard reaches the end only when the holes in every slice happen to line up at the same moment.

The model's other contribution is the split between active failures — the unsafe act at the sharp end — and latent conditions, the upstream decisions that sat dormant in the system until something else lined up with them. Aviation and healthcare both reorganised around this idea, and the reason it stuck is that it stops you from interrogating the last person in the chain and starts you asking why nothing earlier caught it.

Chopra named the model directly when explaining why Uber runs a final publish-ready QA step after the editing loop has already passed its own QA. There is redundancy between the two, and the redundancy is the point rather than an oversight. The final gate is more holistic, it applies policy checks the earlier one does not, and — the detail worth stealing — it will sometimes flag things that should have been caught upstream.

That last behaviour is the most useful output of a layered design, and almost nobody instruments it. A late gate catching an early gate's miss is not a nuisance. It is a free, continuously generated map of where the holes in your earlier slices are.

What the gates are actually catching

The failure modes worth gating against are more specific than "bad output", and the Uber examples are unusually concrete:

  1. Faithfulness — the agent added shrimp to a dish that had none. The edit is attractive and wrong.
  2. Completeness — the agent removed sauce that was in the original. Nothing was invented; something was quietly lost.
  3. Count mismatch under instruction pressure — an image showed six chicken wings while the dish name said eight pieces. If the router waves that through to an editing agent, the agent has every incentive to hallucinate two more wings to satisfy the description. The gate that catches this is upstream of the model that would have done the damage.
  4. Reward hacking — a creative edit was rejected by the QA agent, so the next attempt oversteered into a generic ceramic bowl. The pixels changed a lot; the output got worse in every way that mattered. The team called this a nugatory change: a difference large enough to look like work and too meaningless to count as any.
  5. Honest uncertainty — on an image where the wonton count could not be verified in either the input or the output, the verdict was unsure, and unsure was treated as a rejection in production.

Three of those five are failures the gate exists to catch on behalf of a later step, not the current one. That is a design hint: gates belong where the damage would be cheapest to prevent, which is usually earlier than where it would be most visible.

It is also worth being precise about the loop metric. Uber tracks pass@k through the enhance-and-retry loop. The name comes from the 2021 Codex paper, Evaluating Large Language Models Trained on Code, where k samples were drawn independently for a programming problem and the problem counted as solved if any one of them passed the unit tests. An agent loop uses the term for something subtly different: the attempts are sequential, and each one sees the gate's feedback on the last. Both curves rise with k, but they rise for different reasons — one measures the value of resampling, the other measures the value of the feedback. If your pass@k curve is flat after the first retry, your gate is producing verdicts but not usable critique.

The hole you cannot see: gates that share a blind spot

The Swiss cheese model carries an assumption that is easy to miss and expensive to get wrong: the holes in different slices have to be in different places.

Reliability engineering has a name for the violation — a common-cause failure, where a single shared factor knocks out several redundant components at once and the redundancy turns out to have been decorative. Nuclear and aerospace designers spend serious effort on this, because it is the standard way a system with four backups still fails.

Agent pipelines are unusually prone to it, and nothing in the architecture diagram shows it. If three gates are the same frontier model, reading overlapping criteria from the same prompt template, judging the same artifact with the same evidence, then they do not have independent holes. They have one hole, rendered three times. The pipeline diagram shows defence in depth; the behaviour is a single check with extra latency and triple the cost.

Independence has to be engineered deliberately, by varying something real between layers:

  • Different evidence. A gate that sees the original and the output can judge faithfulness. A gate that sees only the output cannot, but it can judge whether the result stands on its own. Neither is redundant with the other.
  • Different modality or model. Uber's team noted that weaknesses in the frontier image models — object coherence, physics plausibility — leaked upward into their application. A gate built on the same family inherits the blind spot.
  • Different question. "Is this faithful to the input?" and "does this comply with policy?" fail in different places. "Is this good?" asked three times fails in one place.
  • Different authority. The Uber team baked product, design, policy and legal input into the definition of a better image. Criteria from different stakeholders catch different things precisely because they were written by people with different fears.

The practical test is simple and uncomfortable: take the last three incidents that reached production and ask, for each gate, whether it had any real chance of catching them. If the answer is the same for every gate, you have one slice.

Why this matters for developers, and for agents

For developers, gates are where an agent stops being a demo. The spectrum runs from rigid rules — controllable, brittle, unable to cover a long-tail marketplace — to unconstrained agency, which is creative and unshippable. Gates are how you buy agency at the top of the pipeline by paying for constraint at the bottom. Uber's guardrail metric for the router is recall, chosen deliberately: a false positive costs compute and risks degrading an already-good image, while a false negative lets a bad image through. They made the trade explicit instead of optimising for accuracy and hoping.

For AI agents, gates are the only mechanism that converts "the model got it wrong" into something actionable. A rejected output with a reason attached is a training signal, a retry instruction and a diagnostic record at once. Uber feeds exactly this into an umbrella diagnoser agent that localises which component in the pipeline is responsible for a mismatch and triggers a tuning pipeline against a golden dataset of human labels. None of that works if the gate only emits a boolean.

Which is the quiet precondition under all of it: logging first. Their end-to-end orchestration writes a flat JSON record that technical and non-technical people can both read — the whole run in one structure, so a product manager can inspect a single case and an engineer can roll it up in aggregate. As Chopra put it, if you do not start with logging, you have nothing to optimise against, let alone a self-improving loop.

Which applications benefit most

  1. Anything published under someone else's name. Marketplace listings, merchant content, generated imagery, customer-facing copy — the brand risk is borne by a third party who did not choose your model.
  2. Agent-written code and configuration. Independent layers here are nearly free: a type checker, a test suite and a reviewing model fail in genuinely different places.
  3. Regulated outputs. Clinical, financial and legal workflows already assume layered review. The vocabulary of latent conditions and active failures is one auditors recognise.
  4. High-volume, long-tail systems. When inputs vary more than any rubric anticipates, redundancy is the only strategy that degrades gracefully.
  5. Self-improving pipelines. A loop that rewrites its own prompts needs a gate it did not write to tell it whether the rewrite helped.
  6. Multimodal pipelines. When the evidence is an image, a video or audio, "I cannot verify this" is a common and correct verdict, and it needs a layer that is allowed to say so.

How MDflow fits

Gates are a runtime concern, and MDflow is not a runtime. But every gate in a layered design is only as good as the criteria it enforces, and criteria are documents — which is precisely the half that tends to be handled worst.

The failure pattern is specific: the criteria for all three gates end up in one prompt template in the repo, or split between a prompt, a Notion page that design owns, and a legal requirement that exists only in a Slack thread. At that point gate attribution is impossible. When the final gate catches something the first gate should have caught, there is no single artifact to edit, and no way to answer the question that follows every regression — which criterion changed, and when.

What already lines up today:

  • Documents are plain markdown, so the rubric a design lead reviews and the rubric an agent reads at run time are the same file. No export step, no second copy that drifts.
  • One document per gate is a structure the product supports natively — folders group a pipeline's gates, and each gate's criteria get their own title, their own URL and their own history. That is what makes "this belongs to gate 3" a sentence with a referent.
  • Version history captures the previous version on every save, including writes made through the API and MCP rather than only in the editor, with a line-by-line markdown diff against the current document and non-destructive restore. History is private to you and kept for 30 days or the last 300 versions. After a regression, this is the audit trail: the faithfulness criterion changed here, and here is the diff.
  • Folder descriptions act as durable context, so the standing instructions shared across a family of gates live next to the documents instead of inside a prompt string. See folder descriptions as agent context.
  • mdflow_get_context and the wider MCP surface let a judge agent fetch the current rubric at run time instead of carrying a stale copy baked into its system prompt — 25 tools across documents, folders and workspaces, every call scoped server-side to the token's owner. MCP docs.
  • The HTTP API with a personal access token covers the same ground for pipelines that are not MCP clients, which is how a tuning job writes a diagnosis back into the knowledge base. API docs.
  • Raw .md twins with frontmatter give every rubric a machine-readable address, so the criteria your dashboard links to and the criteria your gate enforces are literally the same bytes.
  • Comments and Team workspaces give the "is this actually a violation?" argument — the one that involves product, design, policy and legal — a place to happen against the specific line in question, rather than vanishing into a thread.

Where we are headed, as direction rather than a dated commitment: richer structured frontmatter for documents that describe operational state, and better tooling for comparing two versions of an agent-facing document over time. The principle we will not trade away is that the file stays plain markdown you can walk away with.

The bottom line

The useful idea from seventy years of safety engineering is not that you should add more checks. It is that you should stop expecting any single check to work, and design for the moment when several fail at once.

For AI agents that means three things. Gates belong where the damage is cheapest to prevent, which is usually earlier than where it would be most visible. Redundancy between gates is a feature, and a late gate catching an early gate's miss is the most valuable signal the system produces — if you bother to record it. And redundancy is worth nothing unless the layers fail independently, which does not happen by itself when every gate is the same model reading the same prompt.

Underneath all of it sits an unglamorous requirement: each gate's criteria need to be a document someone can point at, argue about, version and hand to a model — not a string literal that three gates share and nobody owns.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What is a quality gate for an AI agent?

A quality gate is a check that sits between an agent producing an output and that output reaching a user, and that can block it. It differs from an eval in when it runs and what it does: an eval scores a batch offline to tell you how the system is doing, while a gate runs on a single live output and decides whether it ships. The same rubric often powers both, but a gate needs a verdict, not a score.

Why run a second quality gate if the first one already passed the output?

Because every gate has holes, and you do not know where they are. This is James Reason's Swiss cheese model, from safety engineering: each layer of defence is a slice with holes in different places, and an incident only reaches production when holes in every layer line up. Uber's computer vision team said explicitly that their final publish-ready check duplicates work done upstream, and that the redundancy is deliberate rather than an oversight.

When does adding more gates stop helping?

When the gates share a failure mode. Reliability engineering calls this a common-cause failure: redundancy only multiplies protection if the layers fail independently. Three gates that are the same model, reading overlapping criteria from the same prompt, with the same blind spot about the same edge case, are one slice of cheese wearing three hats. Varying the model, the modality, the criteria or the evidence each gate sees is what makes a second gate worth its latency.

What does pass@k measure in a self-correcting agent loop?

It measures the share of items that clear the gate by the kth attempt. The metric comes from the 2021 Codex paper, where k independent samples were drawn for a coding problem and the problem counted as solved if any sample passed the unit tests. Agent loops use the name slightly differently: the attempts are sequential and each one is conditioned on the gate's feedback from the last, so the curve shows how much the feedback is worth, not just how much resampling is worth.

Where should the criteria for each gate live?

In a separate, named, versioned document per gate rather than inside one shared prompt string. Gate attribution depends on it: when a late gate catches something an early gate should have caught, the fix belongs to one specific rubric, and you can only make that edit if that rubric is a thing you can point at. Versioning matters for the same reason, because the question after a regression is almost always which criterion changed and when.

Further reading

← all posts./start-free →