Agent Drift Detection: When the Baseline Moves

Your agent got slower this quarter. Nobody changed the prompt, the model is pinned, the tool definitions are the same, and every individual run looks fine when you open it. The dashboard is green. Users are complaining anyway.

This is the failure mode that observability tooling is worst at, because it is not a failure. It is a baseline that quietly moved while you were watching for spikes.

The clearest treatment of it I have seen this year came from a payments engineer, not an AI engineer. Ritvik Pandya leads a payments team at JP Morgan Chase, and his AI Engineer talk Learned Execution Graphs for Anomaly Detection & Drift in APIs is about real-time payment processing across a chain of services. Every structural idea in it transfers directly to agent systems, which are also a chain of steps with a shape, a baseline and a tendency to change without telling you.

TL;DR — Agent drift detection starts with one distinction: an anomaly is a bad run against a baseline you still trust, while drift is the baseline itself moving. Represent each run as an execution graph — a DAG of the steps it actually took — and you can compare shape, not just totals, which localises a regression to a node instead of a number. Drift splits three ways: structural (a step was added or removed), scale (same shape, more volume), and covariate (the input mix changed and the system is fine). Each needs a different response, and only one of them is a bug. Keep baselines per task type rather than global, treat a new task as a cold start with no baseline, and make the system deployment-aware so you can tell a regression from a release. And keep the baselines and runbooks themselves in versioned markdown — because after a bad run, the question you need to answer in minutes is did the model change, or did the document? MDflow is built for that half. Start free.

What is agent drift?

Agent drift is a permanent change in what "normal" means for your agent system. The latency, the step count, the tool-call sequence, the token spend, or the distribution of requests arriving has moved far enough that the numbers you alert against describe a system that no longer exists.

The distinction Pandya draws is worth stating in his own framing, because it is unusually concrete:

• You drive to the office every day and it takes an hour. One day it takes ninety minutes because of an accident. That is an anomaly. The baseline is still right.
• Months later you are drinking coffee at 4pm and realise the same drive has been taking eighty minutes for a while now. Nothing broke. Traffic in the area changed. That is drift. The baseline is wrong.

An anomaly is an incident. Drift is an accounting problem. Confusing them is expensive in both directions: alerting on drift generates noise until the team mutes the alert, and rebaselining an anomaly quietly accepts a regression as the new normal.

Agent systems drift faster than payment APIs, for reasons specific to how they are built:

Not one of those shows up in a git diff of your application code.

The execution graph: giving a run a shape you can compare

An execution graph is a short-lived DAG representing one request's actual path through the system — which steps ran, in what order, what context each passed to the next. Pandya is explicit that this is not a persistent property graph of the kind you would query in Neo4j. It exists for the life of a request, to be compared against a baseline.

That comparison is the whole point. A single latency number tells you the run was slow. A graph tells you where.

baseline    edge → auth → orchestrator → ┬ fx-rate ──┐
                                          ├ ledger ───┼→ notify   700ms
                                          └ fraud ────┘

observed    edge → auth → orchestrator → ┬ fx-rate ━━━━━┓  (1.9× baseline)
                                          ├ ledger ───┼→ notify  1400ms
                                          └ fraud ────┘

Two different readings fall straight out of the shape. If one node is slow, you have localised the problem to a service. If everything is slow by roughly the same factor, the cause is something shared — the host, the network, a common dependency — and chasing individual nodes is wasted effort.

An agent trace is already this graph. The OpenTelemetry GenAI semantic conventions model it explicitly, with invoke_agent spans nesting chat and execute_tool spans; the conventions moved into their own semantic-conventions-genai repository and are still marked Development, but the span shape has been stable enough to build on. What most teams do with those traces is read them one at a time when something breaks. The execution-graph framing says: derive a baseline shape per task type, and compare every run against it automatically.

For an agent, the baseline is richer than latency:

• the sequence of tool calls, and its length
• which tools appear at all
• how many retry loops fire (Pandya's advice: model every loop as its own node so it is countable)
• tokens in and out per step
• where the run terminates

A run that used to touch four tools and now touches nine has drifted in shape, and that is visible long before it shows up as a quality complaint.

Three kinds of drift, three different fixes

Once you have detected drift, categorising it is what tells you what to do — and only one of the three categories is a bug.

1. Structural drift

A node appeared or disappeared. Pandya's example is the coffee shop that starts asking whether you have a membership: a new step in a sequence you had memorised. In an agent system this is an MCP server adding a tool, a hook firing where none fired before, a guardrail inserting a check, a sub-agent being introduced.

The fix is not to suppress the alert. It is to update the baseline to include the new step, then keep alerting against the new shape. Structural drift that goes unrebaselined poisons every downstream comparison.

2. Scale drift

The shape is unchanged; a component is now handling more than it was provisioned for. This is the ordinary capacity conversation — scale the component out, make the call asynchronous, add a queue. For agents it usually surfaces as rate limits, provider queueing, or a retrieval index that got slow as the corpus grew.

3. Covariate drift

The most interesting one, and the one teams misdiagnose most often. The system is working exactly as designed; the input mix changed. Pandya's version: you launched expecting 60% domestic payments and 40% international, the product got popular abroad, and now the ratio has flipped. Nothing is broken. International requests legitimately take longer, so the blended average moved.

There are only two honest responses, and both start with the same realisation:

• Split the baseline — one per class of request, so domestic and international are never averaged together, or
• Raise the blended baseline to reflect the mix you actually have.

For agents this is the drift that matters most, because agent inputs are natural language and their distribution is completely outside your control. A support agent tuned on billing questions that is now mostly answering integration questions has not regressed. It is being asked a different job. The formal name for this is covariate shift, and the statistical machinery for detecting it is well established — maximum mean discrepancy (Gretton et al., JMLR 2012) and KL divergence are the two Pandya names, and both are implemented in open-source detectors like Alibi Detect.

The reason this category is worth naming out loud: if you treat covariate drift as a regression, you will "fix" a system that was never broken, and quite possibly make it worse for the users who now make up the majority.

Detecting drift without drowning in false alarms

Most drift-detection systems die of noise, not of blindness. The practical guidance from the talk is mostly about keeping the false-positive rate survivable.

Check in tiers. The first check should be cheap — Pandya's analogy is the agent at the airport who glances at your boarding pass and waves you through. Compare the run's total against the baseline; if it is inside the band, stop. Only escalate to node-level comparison and statistical tests when tier one fails. In an agent stack, tier one is total latency, step count and token spend. Tier two is the graph comparison. Tier three is an LLM-as-judge pass over the trajectory, which is the expensive one.

Never hold one global baseline. "All POST requests should complete in n ms" is the anti-pattern; it is the average of things that were never comparable. Hold a baseline per operation and per client class. The agent equivalent: a baseline per task type. A one-shot classification and a multi-hour research run measured against the same expected step count give you a number that is meaningless for both.

Treat new tasks as a cold start. A new endpoint has no history, so inheriting a generic baseline produces immediate false alarms. Let it accumulate its own before you alert on it.

Watch for a delayed node masquerading as a missing one. Pandya's sharpest operational point: if seven services emit telemetry and one is late, your graph has six nodes — which looks exactly like structural drift. Tail-based sampling helps here, because the decision to keep a trace is made once the trace is complete rather than at the first span, which is also why it is the right sampling strategy when what you care about is the shape of a whole request. Agent runs have the same hazard on a longer timescale: a sub-agent that has not returned yet is not a sub-agent that was skipped.

Make the system deployment-aware. Without knowing that a release just went out, drift detection tells you something changed but not what to do about it. With it, a shape change that starts at a deploy boundary is a rollback candidate rather than a mystery.

Insist on explainability. The line worth keeping: if you go to the doctor and they tell you your health score is 22, you have learned nothing. A drift alert that says "step 4 now appears in 80% of runs, up from 3%, starting Tuesday" is actionable. A composite score is not.

Then roll the fix out gradually. Once you know the drift type and the action, assess the risk, ship to 5–10% of traffic, verify, then go to 100%.

Which applications benefit most

1. Coding agents in CI — where step-count and retry-loop drift are the earliest signal that a model or tool update changed behaviour, well before pass rates move.
2. Customer-facing support agents — the highest covariate-drift exposure of any category, because the input distribution is set by your users and your marketing, not by you.
3. Long-horizon research and analysis agents — where a run that used to take twelve steps and now takes forty is a cost incident nobody notices until the invoice arrives.
4. Agents over MCP tool fleets — every server added to the fleet is structural drift for tool selection across the whole set.
5. Regulated workflows — finance, healthcare, legal — where "the behaviour changed and we cannot say when or why" is itself the finding.
6. Multi-agent pipelines — where the shape is the architecture, so structural drift and an architecture change are the same event.

How MDflow fits

Detection is a telemetry problem, and MDflow is not a telemetry vendor. But every drift investigation ends at the same fork, and that half is a documents problem.

After a bad run, you need to answer one question fast: did the model change, did the tools change, or did the instructions change?

That question is only answerable if the instructions have a history. In most teams the agent's runbooks, baselines, task definitions, tool guidance and escalation rules live in a wiki page, a Slack thread, a plan.md on someone's branch and a few string literals in the repo. When the behaviour shifts, there is no way to line up "the agent started doing this on Tuesday" with "someone edited the runbook on Tuesday."

What already lines up today:

• Documents are plain markdown, so the same file is the runbook a human approves and the context an agent reads. No export step, no format that only one side understands.
• Version history captures the previous version on every save — including writes made through the API and MCP, not just edits in the editor — and shows a line-by-line markdown diff against the current document. Restoring is non-destructive. History is private to you and kept for 30 days or the last 300 versions. That is the audit trail a drift investigation actually needs: the baseline document changed here, and here is the diff.
• Folder descriptions act as durable context, so the standing instructions for a class of task live next to the documents rather than inside a prompt string. See folder descriptions as agent context.
• mdflow_get_context and the wider MCP surface let an agent read the current runbook at run time instead of carrying a stale copy in its system prompt — 25 tools over documents, folders and workspaces, every call scoped server-side to the token's owner. MCP docs.
• The HTTP API with a personal access token covers the same ground for pipelines that are not MCP clients, which is how you would write a drift report back into the knowledge base from a monitoring job. API docs.
• Raw .md twins with frontmatter mean every document has a machine-readable address, so the baseline your dashboard links to and the baseline your agent reads are literally the same bytes.
• Comments and Team workspaces give the rebaselining decision a place to be argued and recorded, rather than being an untracked edit.

Where we are headed — direction, not a dated commitment: richer structured frontmatter for documents that describe operational state, and better tooling around comparing two versions of an agent-facing document over time. The principle we will not trade away is that the file stays plain markdown you can walk away with.

The bottom line

The reason agent drift is hard is not that it is subtle. It is that the industry's instinct is to alert on deviations, and drift is a deviation that means the alert is wrong rather than the system. Getting this right takes three things: a run representation with a shape you can compare, a taxonomy that separates structural from scale from covariate, and baselines held per task type instead of globally.

And it takes one unglamorous thing more — the documents that define what "normal" is need to be versioned as carefully as the code. Otherwise every drift investigation ends in a shrug.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What is agent drift?

Agent drift is a permanent change in the normal behaviour of an agent system — its latency, its step count, the shape of its tool-call sequence, or the mix of requests coming in — that makes the old baseline wrong. It is different from an anomaly, which is a single run that deviated from a baseline that is still correct. An anomaly asks what went wrong with this run. Drift asks whether the run was fine and the yardstick is out of date.

What is the difference between an anomaly and drift?

An anomaly is one deviation against a baseline you still trust: your commute took ninety minutes today instead of sixty, because of an accident. Drift is the baseline itself moving: your commute has quietly taken eighty minutes for the last three months, because traffic in the area changed. The response is opposite in each case. You fix an anomaly and keep the baseline. You investigate drift and then rebaseline, because continuing to alert against the old number produces noise, not signal.

What are the three kinds of drift in an agent system?

Structural drift is a change in the shape of the run — a step added or removed, a new tool in the sequence, a retry loop that was not there before. Scale drift is the same shape taking longer or costing more because volume grew past what a component was provisioned for. Covariate drift is the input mix changing while the system works exactly as designed, such as a support agent that used to see mostly billing questions now seeing mostly integration questions. Only the first is a code change, and only the third usually means the system is healthy and the baseline is not.

Why do per-endpoint baselines beat one global baseline?

Because a global baseline is the average of things that were never comparable, so it is simultaneously too loose for the fast path and too tight for the slow one. In Ritvik Pandya's AI Engineer talk on execution graphs at JP Morgan Chase, the guidance is to avoid a single number for all POST requests and instead hold a baseline per operation and per client class. For agents the equivalent is a baseline per task type — a one-shot classification and a multi-hour research run should never be measured against the same expected step count.

How does a knowledge base relate to agent drift?

Because the baseline and the runbook are documents, and they drift too. When an agent behaves differently after a change, the first question is whether the model changed, the tools changed, or the instructions changed — and that is only answerable if the instructions live in versioned markdown with a per-save history you can diff. MDflow keeps documents as plain markdown, captures the previous version on every save including writes made through the API and MCP, and shows a line-by-line diff against the current version, so a changed baseline is a reviewable commit rather than an untraceable edit.

Further reading

• Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase, AI Engineer (July 2026). The source talk.
• OpenTelemetry GenAI semantic conventions — invoke_agent, chat and execute_tool span shapes. Status: Development.
• OpenTelemetry sampling concepts — head-based versus tail-based, and why the latter matters when you care about whole-trace shape.
• A Kernel Two-Sample Test — Gretton, Borgwardt, Rasch, Schölkopf and Smola, JMLR 13 (2012). The MMD statistic.
• Alibi Detect: MMD drift detector — an open-source implementation.
• From Signal to PR: the self-improving agent loop — what to do once you have detected the regression.
• Private agent benchmarks: from traces to simulations — turning production traces into a repeatable release gate.
• Your agent architecture has a half-life of six months — the structural-drift problem at architecture scale.
• MDflow MCP server · HTTP API · FAQ