skip to content
$mdflow
$cat state-machine-ai-agents.md

Don't Let the LLM Drive: State Machines for AI Agents

14 min readby MDflowview as .md
A row of emerald wireframe hexagonal chambers joined by directional arrows, one lit from within by a glowing cube, with an overhead control rail of mechanical levers reaching down into each chamber, on a dark terminal grid background

You have shipped a multi-step agent. It demos beautifully. Then a real user gets in, and halfway through the flow the agent decides it is finished. Or it skips a step. Or it quietly loops back to one it already completed. The demo never showed you that, because the demo was you, behaving.

The reflex is to prompt harder. Add a rule. Add another rule. Add a "you MUST NOT declare the task complete until every step has been performed" in capital letters.

Ornella Bahidika and Joel Allou of Microsoft gave a short, unusually blunt talk at AI Engineer in July 2026 arguing that this never works, because it is aimed at the wrong layer. They build Ace, a live AI voice tutor that runs a full lesson start to finish. Their claim: reliability was never a prompting problem — it is a control problem. The fix is not a better prompt. It is taking the control flow out of the model.

TL;DR — Multi-step agents fail because the model is asked to hold the plan, its position in the plan, and the task all at once. Model the workflow as an explicit state machine in code: each state calls the model with one narrow contract, the harness validates the response and decides what comes next. The model proposes, the harness decides. This buys you reliability, auditability, and usually a cheaper model — but it also splits one system prompt into a dozen per-step contracts, and that prose needs a home both your code and your reviewers can read, which is what a markdown workspace like MDflow is for.

What is a state machine for an AI agent?

A state machine for an AI agent is an explicit set of named steps, written in ordinary code, that owns the question of where the agent is and what comes next. The model is invoked once per step, given only what that step needs, and asked for one thing back. The harness validates the answer, updates the state, and selects the next step.

In Ace, a lesson is six states:

intro → teach → check → grade → advance → wrap

Each state sends the model a narrow contract — do this one thing, return it in this shape — and the harness does the rest. Bahidika's framing is the cleanest version of this idea I have heard:

The model is the talent, and the harness is the director.

A talented actor delivers a line brilliantly. They are not the person who remembers that this is take three of six, that the previous scene needs reshooting, and that the crew breaks in twenty minutes. Asking one entity to do both is a staffing mistake, not a talent problem.

So Ace stopped asking. Three questions in particular were engineered outside the model entirely:

  1. Is the lesson done?
  2. Did the student actually get it right?
  3. What comes next?

Every one of those is a control decision. None of them is a language task. The model never decides where the lesson is — as Allou put it, it proposes, but ultimately it is the harness that decides.

This is not a new idea so much as a rediscovered one. Anthropic's Building Effective Agents draws exactly this line: workflows are systems where models and tools are orchestrated through predefined code paths, while agents are systems where the model dynamically directs its own process. Most teams reach for the second because it demos better, when the first is what their problem actually needs. HumanLayer's 12-Factor Agents makes it Factor 8 — own your control flow — with the same advice: own the loop yourself, set explicit conditions for looping, retrying and halting, and do not let the model self-regulate.

Why "prompt it harder" doesn't fix it

Because the control flow lives in the prompt, and a prompt is not an enforcement mechanism.

When one context window has to hold the plan, the agent's position in the plan, the rules for advancing, the conversation so far, and the actual task, position is the cheapest thing for the model to lose. It is not a fact the model was told once and can look up; it is a running count it has to maintain across a growing context. Models are excellent at delivering a line. They are genuinely bad at remembering they are on step three of six.

And the failure is silent. A model that skips a step does not raise an exception — it produces a fluent, confident, well-formed response for the wrong step. There is nothing to catch, because nothing outside the model was ever checking.

A state machine changes the shape of the failure. If step four returns something that does not satisfy step four's contract, validation fails, and you can retry that one step, fall back, or escalate to a human. The blast radius of a bad model call is one state, not the whole run.

It also makes the run auditable. Ace's logs show harnessing for each concern separately: which section to speak about, what to draw on the whiteboard, when to clear the queue, how to end the lesson. When something goes wrong you can point at a state and a contract, rather than reading a transcript and guessing what the model was thinking.

Why this lets you drop to a smaller model

Model size mostly buys planning and long-context recall — and a state machine removes both jobs from the model.

This was the most concrete result in the talk. Ace runs on Claude Haiku 4.5, not a frontier Opus-class model. Not because Haiku is as capable in general, but because the harness had already absorbed everything Haiku is worse at. Each call is a single scoped action with exactly the input that action needs. There is no plan to hold, no position to track, no ambiguity about what to do next.

For a live voice tutor, that is the whole product. A tutor that is brilliant but takes four seconds to respond is not a tutor. A cheap, fast, reliable model inside a well-built harness beat an expensive, slow, smart model deciding everything for itself — on reliability and on cost and on latency.

The general form of the trade: you can pay for intelligence at inference time, or you can pay for structure once, in code. Structure is cheaper per run, deterministic, testable, and it does not regress when a provider updates a model.

Which applications benefit most

The pattern generalises well beyond voice tutoring. The strongest candidates:

  1. Voice and real-time agents. Latency budgets are brutal and there is no scrollback for a user to correct a wrong turn. Scoped calls to a fast model win.
  2. Onboarding and intake flows. KYC, patient intake, insurance claims, employee onboarding. The steps are known, the order matters, and skipping one is a compliance incident rather than a bad UX.
  3. Ops runbooks and incident response. A runbook already is a state machine. Handing it to an open-ended agent throws away the one artefact you had that encoded the correct order.
  4. Coding agents on migrations. Sweeping mechanical changes over a codebase have a natural per-unit loop — pick target, apply, verify, commit. Let code pick the target, not the model. (More on that in loop engineering.)
  5. Multi-step support and escalation. "Did we actually resolve it, or did the agent decide it was done" is precisely the decision you do not want the model making about its own work.
  6. Anything regulated or reversible-at-cost. If a wrong step means a refund, a shipment, or a disclosure, the step boundary is where your controls belong.

The inverse also holds: genuinely open-ended research, exploration and brainstorming have no known step order, so imposing one just makes them worse. Use the model's autonomy where the path is unknown, and take it away where the path is known.

How to tell whether your agent needs this

Allou offered a heuristic worth stealing: if the reliability of your agent is somewhat of a coin flip, take the control flow out of the model.

Three sharper diagnostics:

  • Does it fail differently on identical input? If the same request produces a different shape of run each time — not a different wording, a different sequence — the sequence is not owned by anything.
  • Is your system prompt mostly prohibitions? A prompt that has accreted a layer of "NEVER declare complete before…" and "ALWAYS check before…" is a control-flow spec written in the least enforceable language available.
  • Can you name the states? If you can list them on a whiteboard in under a minute, they belong in code. If you genuinely cannot, you may have a real agent problem rather than a workflow problem.

How MDflow fits

MDflow is not an orchestration framework. It runs no state machines and executes nothing. That belongs in your own code, or in LangGraph, or the Agents SDK, or whatever runtime you already have.

What it addresses is the side effect nobody warns you about: splitting one agent into a state machine splits one prompt into a dozen.

Before, you had a system prompt. After, you have a per-state contract for intro, teach, check, grade, advance and wrap — each with its own instruction, its own output shape, its own rubric for what "good" means, and its own escalation rule. The teaching content for the teach state. The grading rubric for the grade state. The tone and the boundaries for the wrap state. That is not code. That is writing — usually written by someone who is not the engineer, like a curriculum designer, a compliance officer or a support lead.

In most teams that writing ends up scattered across a repo, a Confluence page, a Slack thread and a few string literals, and the copy the agent runs quietly stops matching the copy anyone reviewed.

What already lines up today

Markdown, with a raw twin for agents. Per-state instructions and rubrics are plain markdown with no proprietary layer, and every document has a raw .md twin your harness or build step can fetch directly. The file the state machine loads is the file a person edits.

A folder per state machine, described for retrieval. Every folder carries a description of what belongs inside it, and mdflow_get_context ranks those descriptions above folder names and document titles before returning bodies. A folder described as "Lesson state contracts — authoritative; the Ace harness loads these at run time" is a retrieval signal you wrote on purpose, not one inferred from a filename. That is why folder descriptions beat file names.

Version history answers "which step regressed?" Every saved change captures the previous version across every write path — editor, HTTP API and MCP — with line-by-line diffs and non-destructive restore. When pass rates on the grade state drop, the first question is whether the model changed or the contract did, and a diff answers it in seconds. (Version history is a Pro feature, private to the document owner, and deliberately not exposed over the API or MCP.)

One source for the harness and for the humans. The same workspace is reachable from Claude, ChatGPT, Cursor and Codex over the remote MCP server with OAuth or a Personal Access Token, and from your own services, cron jobs, CI and n8n over the HTTP API. If the harness pulls each state's contract from the workspace, there is no second copy to drift.

The Document Log shows who changed a contract. A cross-document activity feed records created, edited, shared and deleted events with the actor on every row, shown as automated · <token name> for anything arriving via API or MCP, with a side-panel diff on edited rows.

Collections and comments for the non-engineers. Group the six state contracts into a collection and share it as one read-only link with the subject-matter expert who owns the content; they can attach comments to a selected passage of the markdown rather than starting a thread that scrolls away.

Tasks live inside the document they concern. Because /tasks aggregates ordinary - [ ] checkbox lines out of markdown bodies, "tighten the advance criteria after the timeout bug" can sit inside the state contract that prompted it and still appear on a real list.

Encryption where the content is sensitive. Intake scripts, clinical protocols and internal escalation policies can be client-side encrypted, which also means they are never scanned or indexed server-side.

Where we are headed

Direction, not a dated commitment: we are most interested in making a written definition easier to pin to a moment in time — so "which version of this state's contract was in force during that run" is trivial to answer — alongside richer structured retrieval over folder descriptions.

The bottom line

Reliability in multi-step agents is not something you prompt your way into. It is something you build around the model. Name the states, write one narrow contract per state, validate what comes back, and let code decide what happens next. You will usually find the model you needed was smaller than the one you were paying for.

Then keep the contracts somewhere both halves of the system can read: start free, connect an AI agent, or read the API docs.

Frequently asked questions

What is a state machine for an AI agent?

A state machine for an AI agent is an explicit set of named steps, written in ordinary code, that decides which step the agent is on and which step comes next. The model is called once per step with a narrow contract — do this one thing, return it in this shape — and the harness validates the response, updates the state and picks the next step. The model contributes content; it never contributes control flow.

Why do multi-step AI agents skip steps or loop forever?

Because the control flow lives in the prompt. When an agent has to hold the plan, its position in the plan, the rules for advancing and the actual task in one context window, position is the easiest thing to lose. It declares itself done early, silently skips a step, or re-enters one it already completed. Adding more instructions rarely fixes it, because the problem is not that the model misunderstood the rules — it is that nothing outside the model is enforcing them.

Can a smaller model run a multi-step agent?

Often yes, once the harness carries the state. Model size mostly buys you the ability to plan and hold long context. A state machine removes both jobs: each call is a single, well-scoped action with only the input that step needs. Microsoft's Ace voice tutor ran a full lesson on Claude Haiku 4.5 rather than a frontier model precisely because the harness, not the model, decided what came next — cutting cost and latency without losing reliability.

When should I take control flow out of the model?

The practical test is a coin flip: if the same input produces the right end-to-end behaviour roughly half the time, the problem is control, not prompting, and you should move the decisions into code. Also move them when a wrong step is expensive or irreversible, when you need the same run to be auditable, or when latency and cost matter enough that you would rather make five cheap scoped calls than one expensive open-ended one.

Where should the per-step instructions for a state machine agent live?

In version-controlled markdown that both the harness and a human reviewer read from the same place. Turning one system prompt into a dozen per-step contracts turns prompt engineering into a content management problem: each step needs its own instruction, its own output shape and its own rubric. Keeping those as markdown documents — retrievable at run time over MCP or an HTTP API, with diffs when a step regresses — is exactly what a markdown workspace like MDflow is for.

Further reading

← all posts./start-free →