$cat behavior-specs-for-ai-agents.md

Behavior Specs for AI Agents: Opinions You Can Evaluate

November 6, 202621 min readby MDflowview as .md

A single glowing emerald document card at the top, its wireframe lines fanning down into a disciplined grid of identical agent nodes all tracing the same path, with one divergent path bending back toward the document, on a dark terminal-grid background

If you are reading this, your own productivity has probably gone up enormously in the last two years. Claude Code, Codex, Cowork — pick your tool. Now look at the company you work for. Has its output gone up by the same multiple? Almost certainly not, and not by a little.

That gap is the subject of a talk by Mitchell Troyanovsky, co-founder of Basis, which builds AI agents used end-to-end by accounting firms and raised $100M at a $1.15B valuation in February 2026. Published by AI Engineer on 4 November 2026, it is one of the few talks that treats the gap as an organizational problem with a concrete technical fix — a written artifact he calls a behavior spec — rather than a model-capability problem that will solve itself.

TL;DR — Individual productivity scales with better agents; company productivity scales with process, and a general-purpose agent hands process design to whichever employee is typing. The fix is to be opinionated — to decide how the work gets done and write it down. A behavior spec is that written opinion: a short statement of intent, kept separately from the prompt that implements it, that serves as both team alignment and a rubric for runs that have no ground truth. Outcome evals need a correct answer and cannot scale past a few thousand cases; behavior evals need only the rule and the trace, so they scale with production. That makes the written rule a load-bearing artifact — which is what a versioned, machine-readable markdown workspace like MDflow is for.

What is a behavior spec?

A behavior spec is a short written statement of how an agent should handle a specific situation, kept separately from the prompt that implements it. Troyanovsky calls it "a product spec for the agent era." It is not documentation written after the fact, and it is deliberately not a transcript of the system prompt. It is the intent, recorded before the context is written, so that a team can argue about the rule itself rather than about a paragraph of prompt text.

A concrete one from the accounting domain: when a bank reconciliation does not balance, the agent must explain the difference rather than post a balancing entry to force it to tie.

Notice what is interesting about that rule. It is not obviously correct. A different firm might reasonably want a plug entry below some threshold, with a note. Troyanovsky is explicit that the rule's content is not the point: "what you don't want is a customer getting one way one time and another way the other time." The spec exists to make the choice a choice — made once, by someone accountable for it — instead of an accident that varies run to run.

The second thing a behavior spec does is less obvious, and it is where the real leverage is. Because the rule is written in checkable language, it can be used as a rubric. More on that below.

Why individual speed does not become company speed

Organizations have always scaled non-deterministic workers, and they do it with process, not determinism. This is the reframe that makes the rest of the talk work.

Troyanovsky's observation is that humans are already expert at working with unpredictable systems — "it's just those systems are normally their co-workers, not their computers." Look at any firm. The person at the top has, in his words, "literally no idea about the exact day-to-day of the people at the very bottom," and yet trusts the work will come out right. Not because the people are deterministic. Because there are processes, and whoever designed them believed they generalize to good outcomes.

The process goes all the way down. Reconciling cash means getting the bank data, and getting the bank data has its own process — perhaps only "use the approved software to extract it." You would not expect a firm to let every employee Google extract PDF and use whatever they found. That is not a tooling preference. It is the thing that makes the firm's output predictable enough to sell.

Now hold that next to a coding-style agent product, where the interaction is: here is an agent, tell it what you want. Troyanovsky identifies two gaps.

The competency gap. Agents as we ship them today "outsource the complexity to the user." The user has to manage skills and memories, decide when to invoke the agent, and maintain a working theory of mind about it. The audience at an AI engineering conference finds this easy. The overwhelming majority of knowledge workers have no idea how to do any of it, and no reason to learn.

The process gap. If the agent does whatever each user tells it, then the method is set by whoever happens to be typing. That inverts the direction organizations depend on: "the process should be coming from senior leadership down, not from any individual employee deciding how they're going to do their job." Even in 2018, nobody would have accepted every employee inventing their own data-extraction method. An agent that accepts it has not removed the process — it has just stopped writing it down.

This is the actual answer to why the enterprise curve is flat. It is not that the models are not good enough. It is that a tool which multiplies individual improvisation does not compose into organizational output.

Opinions are the unit of reliability

The fix for both gaps is the same: have an opinion about how the work gets done. Opinions make an agent simple for the user, because they no longer have to supply the method. And they make it reliable, because the method stops varying.

Take Troyanovsky's example. An agent needs to read a PDF. There are effectively unlimited valid approaches: throw it at a vision model; write Python against an open-source OCR library; call a specialist OCR provider; OCR and then have a VLM check the work; have five VLMs check it; have a hundred and take the best. Tell a capable agent to "extract the PDF" and it will pick one. It will often pick a good one.

And the result will still not be reliable — "not because the process it did was bad, but because the process it did was not opinionated."

That sentence is worth reading twice, because it separates two things that usually get collapsed. Quality is about any single run. Reliability is about the distribution across runs. An agent that solves the problem brilliantly on Monday and differently on Tuesday has high average quality and no reliability at all, and reliability is what a firm is actually buying.

Which leads to a standard that is lower and much harder than the one most teams set: "nobody is looking for perfect. What you're looking for is reliability and consistency and trustworthiness." Troyanovsky puts the bar in a form that sounds like a joke and is not — it is fine for the agent to be unable to do something, as long as it is consistently unable to do it, and then improves, the way a person does. The failure mode to fear is not incapacity. It is the colleague who is a world expert one day and misses something basic the next. You cannot build a process on that person, and you cannot build one on that agent.

Opinions come from several places. Some are simply expertise: you know that an agent doing tax work is more accurate if it reads the primary source before answering, and you do not need an eval to tell you. You know an agent making a slide deck has fewer formatting bugs if it screenshots the deck before returning. And opinions are not only about accuracy — screenshotting costs latency, and if the formatting bug happens one time in a hundred you may rationally decline to pay for it. The opinion is a decision with trade-offs, which is exactly why it needs an owner and a record.

Your context is runtime training data

The behavior an agent exhibits is produced by everything in its context, not by one instruction. This is the engineering reason a behavior spec has to be a separate artifact rather than a comment in the prompt.

In a non-trivial agent, a single rule is carried by the system prompt, by progressive disclosure, by tool descriptions, by what the model learned in training, and by whatever the harness assembled this turn. There is usually no paragraph you can point at and say that is the rule. Writing two sentences into the system prompt and expecting them to govern is a tactic that works for narrow agents and quietly stops working as the system grows.

Troyanovsky's framing for this is the most portable idea in the talk: treat the context the agent sees while working as runtime training data. Not literally training, but effectively — this is the data that causes the behavior. "The word prompt," he says, "trivializes it a bit." Teams that adopt the reframe start caring about every token in the window the way they would care about a training set, which is roughly the correct amount.

The practical consequence: because the behavior lives across many layers, the statement of the behavior has to live outside all of them. That is the behavior spec. It survives a refactor that moves the rule from the system prompt into a tool description. It gives you something stable to review, to disagree with, and to score against.

One more thing follows from it. You should not write down everything. If the spec restates every instruction, you have duplicated state and created a second thing to keep in sync. You write down the opinions that matter most, and let the rest live in the implementation.

The behavior spec as a rubric

The hardest problem in production agent evaluation is that you cannot label production. This is where behavior specs stop being documentation and start paying for themselves.

Outcome-based evals — the traditional kind, with a right answer and a rubric for judging — are indispensable and do not scale. You may have a hundred cases, or a thousand. Production has hundreds of thousands of situations, and the number is growing fast, because as agents get better at long-horizon work the number of decisions inside a single trajectory grows with it. Troyanovsky's point is blunt: you cannot prove every trajectory in that distribution is correct, because you do not have a ground truth for every trajectory. So you cannot know, from outcome evals alone, whether your evals generalize to production.

But you can ask a different question of a trajectory — one that does not need a correct answer:

Did a situation arise in this run where behavior X was relevant?
If so, did the agent actually exhibit it?

Both are answerable from the written rule plus the trace. That is the whole trick, and the economics of it are what matter. Outcome cases are gated by human labelling. Behavior checks are gated only by how many behaviors you have written down, and you can run them against every trajectory you have — eval runs, staging, production. In Troyanovsky's words, you "cannot scale [outcome evals] anywhere near as fast as you can scale the ability to supervise what the behaviors of the agent are."

This does not prove the agent is right. It is explicitly weaker: evidence that the opinions you hold are being followed, from which you infer the system is more likely to generalize. But weak evidence over every run beats strong evidence over a thousand, and it is the only kind of evidence that grows at the same rate as your traffic.

Two conditions make it work. The rule has to be written in language precise enough that a judge — human or model — can decide "relevant / not relevant" and "followed / not followed" without inventing its own interpretation. And both the judge and the engineer have to be reading the same copy of the rule, which is a documents problem, not an eval-harness problem, and the one teams most reliably get wrong.

Why the behavior failed: forking and simulation

A failed behavior eval tells you the rule was broken; it does not tell you which token broke it. Troyanovsky describes two tools for closing that gap, one expensive and one nearly free.

Forking. Anyone who has used the web versions of Claude Code or Codex has seen the fork control under a message: branch the agent from that point with the same state. Doing it properly in a cloud system is genuinely hard — you have to reconstruct the database environment, the prompts, everything the agent could see. But once you have it, the debugging loop is direct. Take a trajectory that failed a behavior check. Go to the moment just before the agent made the offending decision. Fork. Then either ask the agent why it did that, or change the context and see whether the behavior appears. Compare that to an outcome eval, where all you know is that something was wrong somewhere in a five-hour run containing a thousand inference calls.

Simulation. The cheap version, and Troyanovsky's favorite: ask a coding agent to spin up a sub-agent that pretends to be your production agent. Hand it the real system prompt and the real skills, render everything faithfully, and tell it not to cheat. Then describe a situation and ask what it would do. If you can reproduce the failure, you can iterate on the context in seconds instead of minutes, without rebuilding an environment. It is strictly worse signal than a real fork. It is also available today, with nothing built — he used it for over a year before Basis had forking internally.

Both tools consume the same input: a written behavior precise enough to check. Without the spec there is no failing signal to investigate, and without the signal there is no loop. That is the dependency chain Troyanovsky closes on — opinions produce signal, signal produces improvement, and a system with no opinions cannot generate either.

Which applications benefit most

Regulated professional services — accounting, tax, audit, legal, clinical coding. The work has defensible-but-arbitrary conventions, a named person accountable for each, and reviewers who need to know the agent did it the firm's way. This is the home case.
Internal agents rolled out across a whole company, as opposed to tools adopted by individuals. The moment the user is not the author of the method, the method has to come from somewhere, and that somewhere is a document.
Long-horizon agents whose runs contain hundreds or thousands of decisions. Outcome labelling is hopeless at that length; per-decision behavior checks are the only supervision that scales.
Multi-model or multi-vendor systems. When you swap the model underneath, behavior evals are what tell you which of your opinions survived the swap — a question an aggregate score cannot answer.
Customer-facing agents where consistency is the product. Support, onboarding, intake. Two customers getting different treatment for the same case is a worse outcome than both getting the slightly suboptimal one.
Any team where the person who owns the rule does not write code. A compliance lead, a managing partner, a clinical director. If the rule only exists inside a prompt in a repository, its actual owner can neither review it nor correct it.

The inverse is worth stating too. If the work is genuinely novel each time — open-ended research, one-off analysis — you cannot enumerate the behaviors, and insisting on opinions early will just slow you down. Opinions are for work you intend to do repeatedly and sell.

How MDflow fits

MDflow runs no evals, forks no trajectories and simulates nothing. Those belong in your harness and your observability stack. What it addresses is the artifact all three of them consume, and which has no natural home in any of them: the written behavior spec itself.

That artifact has an awkward shape. It is read by at least three different consumers — the team agreeing on the opinion, the judge scoring runs against it, and whatever assembles the agent's context — and it is frequently written by someone who is not an engineer. In most teams it therefore ends up in four places at once: a Notion page for the humans, a string in the eval repo for the judge, a paragraph in the system prompt for the agent, and a Slack thread where the rule was actually decided. The failure is not dramatic. The copies simply drift, and one day the agent is being scored against a rule nobody has agreed to since March.

What already lines up today

Plain markdown, not a database. A behavior spec is a file. It stays diffable, greppable and readable by a human, with no schema migration between "we changed the rule" and "it shipped."
Raw .md twins. Every document is served as plain text/markdown at a stable URL with its frontmatter intact, so an eval harness or a judge prompt can curl the current rule directly — no client library, no scraping a rendered page.
Folder descriptions as retrieval context. A description like "behavior specs governing the reconciliation agent — these are the rubrics trajectory evals score against; changes require sign-off" is ranked above folder names and document titles by mdflow_get_context, so whoever asks about reconciliation rules lands on the live set rather than a superseded draft. More on that in folder descriptions as agent context.
One copy, many surfaces. The same document is reachable over MCP with OAuth, over the HTTP API with a token, and from VS Code or n8n — so the engineer, the partner who owns the rule and the pipeline that scores trajectories all touch one file instead of three copies.
Version history with per-actor diffs. When behavior changes in production, the first question is whether the model changed or the written rule changed. Line-by-line history, with API and MCP writes attributed as automated · <token name>, answers it instead of leaving you to guess.
Comments and sharing. Review happens on the document, by the non-engineer who owns the wording, rather than as a pull-request comment they will never see.
Team workspaces with per-workspace permissions. This is the direct answer to the process gap: a Team workspace can be read-only for most members and writable only by the people accountable for the method, so the opinion genuinely flows down rather than being edited by whoever ran the agent last. Every change lands in a Team audit feed, and agents reach the same documents through the Team MCP server and Team API.

Where we are headed

Direction, not a dated commitment: typed frontmatter so a spec can carry status, owner and freshness as signals a build step can check rather than conventions it must assume; pinning a document to a specific version so an eval run depends on the rule that was reviewed rather than whatever was current at fetch time; and richer collection-level context so a whole behavior-spec set can be handed to an agent or a judge as one unit.

The bottom line

Troyanovsky closes on the objection you have probably already formed: if you constrain the agent, won't it miss the better way? Maybe. But as he puts it, nobody wants their agent to Move 37 their tax return. They want it done reliably, at scale, in a way they can trust and their people know how to work with. Brilliance that arrives unpredictably is not an asset a firm can sell.

There is a trust model underneath that, and it is the same one you already use with people. When a colleague sends you a pull request, you decide how hard to look based on two things: the blast radius of the change, and how competent you believe that person to be. Agents earn review latitude on exactly those axes. Which means reliability is not a nice-to-have alongside capability — it is the mechanism by which an agent is allowed to do more.

For agents to spread, they have to be simple. For teams to trust them, they have to be reliable. Both come from the same place: deciding how the work should be done, writing it down, and then checking that it happened. The opinions are the thing that survives — the model will be replaced, the harness will be rewritten, and the written rule is what gets carried forward. Give it a home that is versioned, reviewable by the person who owns it, and readable by the machines that have to follow it. Start free, connect an AI agent, or read the API docs.

Frequently asked questions

What is a behavior spec for an AI agent?

A behavior spec is a short written statement of how an agent should handle a particular situation, kept separately from the prompt that implements it. It records the intent — for example, that when a reconciliation does not balance the agent must explain the difference rather than post a balancing entry — so the team can agree on the rule, and so the rule can later be used as a rubric to check whether real runs actually followed it. It is deliberately not a transcript of every instruction in the system prompt, only the opinions that matter most.

How is a behavior spec different from a system prompt?

The system prompt is one of several things that produce a behavior; the behavior spec states what the behavior should be. In a production agent the same rule is carried by the system prompt, progressive disclosure, tool descriptions and the model's own training, so there is rarely one paragraph you can point at and call the rule. Writing the intent down separately gives you something stable to align the team on and to score runs against, even as the implementation moves between those layers.

What is a behavior eval and how is it different from an outcome eval?

An outcome eval asks whether the final answer was right, which requires a ground truth. A behavior eval asks two questions about a run that may have no ground truth at all: did a situation arise where this behavior was relevant, and if so did the agent exhibit it? Because the second kind needs only the written rule and the trace, it scales to the hundreds of thousands of situations production produces, where hand-labelled outcome cases cannot.

Why does individual AI productivity not translate into company productivity?

Because organizations scale through process, and a general-purpose agent hands process design to each individual user. Mitchell Troyanovsky of Basis describes two gaps. The competency gap: the agent outsources complexity to the user, who now has to manage skills, memory and when to invoke it. The process gap: how the work gets done becomes whatever each employee decided, which is the opposite of how firms have always scaled — process flows down from senior leadership, not up from individual improvisation.

Where should a behavior spec live?

Somewhere versioned, reviewable by the non-engineer who owns the rule, and readable by machines at run time — because the same document has to serve three readers: the team agreeing on the opinion, the judge scoring trajectories against it, and whatever builds the agent's context. A shared markdown workspace such as MDflow fits that shape: plain markdown with line-by-line version history, comments for review, and retrieval over MCP or an HTTP API so a judge or build step can fetch the current text rather than a copy that has quietly drifted.