---
title: "Pick the Fastest Model Your Latency Budget Allows"
description: "A deadline makes model quality non-monotonic: past the budget, a better answer scores zero. Pick the fastest model that fits, then buy capability back in code."
author: "MDflow"
date: 2026-11-05
reading_time: "15 min"
canonical_url: https://mdflow.cz/blog/latency-budget-model-selection
md_url: https://mdflow.cz/blog/latency-budget-model-selection.md
---

# Pick the Fastest Model Your Latency Budget Allows

*Published November 5, 2026 · 15 min read*


Most model-selection advice assumes you are climbing. Start with something cheap, measure quality, climb the capability ladder until the answers are good enough, then stop. It is a sound procedure, and it quietly assumes the thing that makes it sound: that a slightly late answer is still worth something.

In a real-time system it is not. **The moment your system has a hard deadline, the search runs the other way** — and the cleanest statement of that reversal came from a six-minute talk at the AI Engineer World's Fair in July 2026, where Joel Allou and Ornella Bahidika of Microsoft explained why the voice tutor they built runs on a small model *on purpose*.

Their framing of the constraint is the part worth keeping: "our budget was never IQ, it's milliseconds."

> **TL;DR** — A deadline makes model quality **non-monotonic**: past the budget, a better answer scores zero, so the usual climb-until-good-enough search inverts. The rule Allou and Bahidika gave is to pick the fastest model your **latency budget** allows and spend the remaining effort on scaffolding — moving state, sequencing and judgement out of the model and into code, then handing the model a freshly assembled summary each turn. The scaffolding is the price, but you pay it once, in code, rather than on every turn. What it leaves behind is a pile of authored prose — scenario catalogues, state definitions, success rubrics, summary templates — that outlives whichever model you picked, which is the layer [MDflow](https://mdflow.cz) is built for: plain markdown with raw `.md` twins, folder descriptions as retrieval context, and the same copy reachable over [MCP](/docs/mcp) and an [HTTP API](/docs/api).

## What a latency budget actually is

**A latency budget is the hard deadline between a user finishing their input and your system producing its first output — divided across every stage that has to run in between.** The model does not get the budget. It gets the remainder.

In a cascaded voice pipeline the stages are roughly network transit, speech-to-text, the model's time to first token, and text-to-speech first audio. Vendor breakdowns of those stages vary enough that you should treat any published table as directional and measure your own: [Deepgram](https://deepgram.com/blog/voice-agent-architecture-stt-llm-tts-pipeline-design) and [Soniox](https://soniox.com/wiki/voice-agent-latency-budget) both put the model's first-token wait as the largest and most variable line, while other guides set a much tighter first-token target of 200–300 ms with prefix caching. The one thing every source agrees on is the ordering: **the model is the biggest controllable line in the budget, and the number that matters is time to *first* token, not total generation.**

Allou and Bahidika gave their own figure rather than a general one: their system needed to start talking in about **950 milliseconds**. That is the whole deadline, not the model's share of it.

(The upstream question — how the pipeline decides you have *stopped* speaking in the first place — is a separate machine with its own failure modes, covered in [voice agent turn-taking](/blog/voice-agent-turn-taking). This post is about what goes in the model slot once that budget is fixed.)

## Why a deadline makes capability non-monotonic

**Because quality that arrives late has no value, and a deadline turns "better" into a step function.**

This is the piece of the argument that generalises past voice, and it is worth stating as arithmetic. Without a deadline, the value of a model's answer is a smooth increasing function of its quality; more capability is always weakly better, so climbing is rational. With a deadline, value is quality *multiplied by whether you made it*. Past the budget that multiplier is zero. A frontier model that reasons for a second and returns a beautiful answer scores exactly the same as a crash.

Bahidika made the point perceptually rather than economically, which is more useful: "when a voice agent pauses for even a second, your brain says it's dead." The user does not experience a slow but thoughtful tutor. They experience a broken app, and they say "hello?" over the top of your model's first token.

What makes this genuinely awkward is that **the capability you are paying for is the thing that breaks the deadline.** Allou was blunt about it:

> The model is really good at reasoning. You can give it a problem, in this case a lesson, and it can reason through it… But that is actually precisely the problem. Because the reasoning can take a couple of seconds, and those seconds are really valuable when you are building voice applications.

Not a tradeoff between two independent properties. The same mechanism, counted twice — once as an asset, once as a liability.

Their demo made it concrete. The same question, asked twice. On a frontier reasoning model — [Claude Opus 4.7](https://www.anthropic.com/news) — you watch the model think, and the answer lands a couple of seconds later. On a far smaller model in the same family, [Claude Haiku 4.5](https://www.anthropic.com/news/claude-haiku-4-5), with the scaffolding in place, the answer comes back in about **900 milliseconds** — inside the budget, and, as Allou put it, feeling "almost instant… because all of the smart parts have already happened prior to the model actually speaking."

## What they took away from the model

**The scaffolding is defined by subtraction: list everything the model is currently deciding, and move each item that is not a language task into code.**

In their system the model does not decide what happens next in the lesson. It does not track what the student knows. It does not work out what to explain next. All of that runs outside, in a state machine with a layer on top that infers mastery, and the result is handed to the model as a **summary, assembled fresh every turn**. Bahidika's description of what is left: "one thing it's really good at — talking."

That is a smaller job than it sounds, and a much better-specified one. The model is no longer holding a plan, a position in the plan, a rubric and a conversation at once; it is turning one paragraph of state into one paragraph of speech. (The reliability half of that same architecture — why pulling control flow out of the model stops agents from skipping steps and declaring themselves finished — is the subject of the speakers' other talk, covered in [state machines for AI agents](/blog/state-machine-ai-agents).)

The pattern worth stealing is the sorting rule. For each decision your agent currently makes, ask whether it is a *language* task or a *control* task. "Phrase this encouragingly" is language. "Has the student mastered this?", "what comes next?", "are we done?" are control. Control tasks in a model are paid for in milliseconds and variance; control tasks in code are paid for once and then run in microseconds, deterministically, with a test suite.

## The price: scaffolding is capital, not rent

To their credit, the speakers did not pretend this is free. Allou:

> A small model like Haiku 4.5, if it doesn't have any scaffolding, tends to drift on long structure and really needs strict rules in order to be able to stay organised. So the scaffolding piece is the price. But the good thing is you pay it once, and in code — not on every single turn.

That sentence is the whole economic argument, and it is an amortisation argument rather than a quality one. **Inference-time reasoning is rent: you pay it on every turn, in latency and in tokens, forever, and the price changes when your provider changes the model. Scaffolding is capital: you pay it once, in engineering time, and it keeps working.**

Which also tells you when the trade is *bad*. Scaffolding only amortises if you run the loop often and the path is roughly known. For genuinely open-ended work — research, exploration, a one-off analysis where you cannot enumerate the states — there is nothing to amortise, and buying reasoning by the turn is the correct purchase. The rule is not "small models are better." It is: **let the deadline pick the model, and let the volume justify the scaffolding.**

Allou's generalisation at the end of the talk is the line to put on the wall:

> Pick the fastest model that your latency budget allows, and then spend the rest of your time actually building the scaffolding.

And the consequence: "in those cases, the model is the smallest part of the system."

## Which applications benefit most

The inversion applies wherever a late answer is a failed answer, or where per-turn cost is multiplied by something large:

1. **Voice agents and live tutoring.** The original case. There is no scrollback, no spinner, and silence reads as breakage rather than effort.
2. **Interactive UI copilots.** Inline completions, command palettes, autocomplete and smart suggestions — anything where the user's next keystroke is the deadline, and an answer that lands after it is discarded by definition.
3. **High-volume classification and routing.** Moderation, triage, intent detection, spam. Here the multiplier is not latency but volume: per-turn reasoning cost times millions of turns, against a scaffolding cost you pay once.
4. **Real-time control and telemetry.** Alert enrichment, anomaly triage, trading and bidding systems where the decision window is fixed externally and nobody is waiting for your model to feel ready.
5. **On-device and edge agents.** The same arithmetic, with the budget set by a frame or a battery rather than a conversation — covered separately in [on-device AI agents](/blog/on-device-ai-agents-compute-budget).
6. **Anything with a conversational interface over a known workflow.** Intake, onboarding, booking, support flows. The steps are enumerable, which is exactly the condition that makes scaffolding amortise.

The inverse holds too. If you cannot name the states, you do not yet have a scaffolding problem — you have a research problem, and a frontier model thinking for four seconds is the right tool.

## How MDflow fits

**MDflow is not in the loop.** It runs no state machines, orchestrates nothing, and an agent working inside a 950-millisecond budget should not be making a network call to a hosted markdown workspace mid-turn. Being clear about that is the point: the sub-second path belongs to your own code.

What MDflow addresses is the artefact this architecture produces, which is easy to miss because it does not look like infrastructure. **When you move thinking out of the model, it does not disappear — it turns into writing.**

Look at what the scaffolding in the talk actually needs in order to exist. Someone had to enumerate every scenario a lesson can be in. Someone had to define what mastery means, concretely enough for code to infer it. Someone had to decide what belongs in the per-turn summary and what does not, and in what shape, because that summary *is* the prompt now. None of that is code. It is prose with hard consequences — and it is frequently written by someone who is not the engineer: a curriculum designer, a clinician, a compliance lead, a support manager.

In most teams that material scatters across a repo, a wiki page, a Slack thread and a few string literals, and then the copy the agent runs quietly stops matching the copy anyone reviewed.

### What already lines up today

- **Plain markdown, not a database.** A scenario catalogue or a summary template is a file. It stays diffable, greppable and readable by a human, with no schema migration between "we changed the wording" and "it shipped."
- **Raw `.md` twins.** Every document is served as plain `text/markdown` at a stable URL with its frontmatter intact, so the build step that bakes scaffolding copy into your service can `curl` it directly — no client library, no scraping a rendered page.
- **Folder descriptions as retrieval context.** A description like *"per-state summary templates for the lesson engine — these are compiled into the prompt builder; edit with care"* is ranked above folder names and document titles by `mdflow_get_context`, so an agent or teammate asking about summary templates lands on the live set rather than an archived draft. More on that in [folder descriptions as agent context](/blog/folder-descriptions-agent-context).
- **One copy, many surfaces.** The same document is reachable over [MCP](/docs/mcp) with OAuth, over the [HTTP API](/docs/api) with a token, and from [VS Code](/docs/vscode) or [n8n](/docs/n8n) — so the engineer who reads it, the designer who writes it and the pipeline that ships it all touch one file.
- **Version history with per-actor diffs.** When behaviour changes, the first question is whether the model changed or the written context changed. Line-by-line history, with API and MCP writes attributed as `automated · <token name>`, answers it instead of leaving you to guess.
- **Comments and sharing.** Review happens on the document, by the non-engineer who owns the wording, rather than as a pull-request comment they will never see.

### Where we are headed

Direction, not a dated commitment: typed frontmatter so a template can carry status, owner and freshness as signals a build step can check rather than conventions it must assume; pinning a document to a specific version so a deployment depends on the copy that was reviewed rather than whatever was current at fetch time; and richer collection-level context so a whole scaffolding set can be handed to an agent as one unit.

## The bottom line

A deadline changes what "good model" means. Without one, climb until the answers are acceptable. With one, the ceiling is set by milliseconds, and capability past the deadline is worth nothing at all — so pick the fastest model that fits, then buy the capability back in code, where you pay for it once instead of on every turn.

The part that survives is not the model and not the plumbing. It is the written definition of the job the model is no longer doing: the states, the rubrics, the summary your harness assembles each turn. Give it a home that is versioned and readable by both your code and the person who wrote it. [Start free](/login), [connect an AI agent](/docs/mcp), or [read the API docs](/docs/api).

## Frequently asked questions

### What is a latency budget for an AI agent?

A latency budget is the hard deadline between a user finishing their input and the system producing its first output, split across every stage that has to run in between. In a voice agent that means network transit, speech-to-text, the model's time to first token, and text-to-speech first audio. The model does not get the whole budget, only what is left after the other stages are subtracted, and what counts is time to first token rather than total generation time, because streaming lets the next stage start early.

### Should I pick the smartest model or the fastest model?

It depends on whether your system has a deadline. Without one, climb the capability ladder until quality is acceptable and stop. With one, the search runs the other way: a deadline makes quality non-monotonic, because an answer that arrives after the budget expires scores zero no matter how good it is. Joel Allou and Ornella Bahidika of Microsoft put the rule plainly at AI Engineer in July 2026 — pick the fastest model that your latency budget allows, then spend the rest of your time building the scaffolding around it.

### Why does a frontier reasoning model fail at voice?

Because thinking time is dead air. The speakers said their system needed to start talking in about 950 milliseconds, and a frontier model that reasons for a full second has already lost the room regardless of answer quality. Bahidika's framing was perceptual rather than technical: when a voice agent pauses for even a second, your brain says it is dead. The reasoning that makes a frontier model good at hard problems is precisely what makes it unusable in a sub-second loop.

### What is agent scaffolding and what does it cost?

Scaffolding is the machinery around the model that does the jobs the model is no longer asked to do — tracking state, deciding what comes next, judging whether a step succeeded, and assembling the summary handed to the model each turn. The speakers were explicit that it is not free: a small model without scaffolding drifts on long structure and needs strict rules to stay organised. The trade is that you pay for scaffolding once, in code, rather than paying for reasoning on every single turn.

### What does a fast, small model need from a knowledge base?

Everything the scaffolding hands it. If the harness writes a fresh summary for the model on every turn, the material behind that summary — the scenario catalogue, the state definitions, the rubrics for what counts as success, the summary templates themselves — is authored prose that outlives the model you picked. It has to be versioned, reviewable by the non-engineers who often write it, and retrievable by the code that assembles the prompt. That written layer is what a markdown workspace is for, not the sub-second loop itself.

## Further reading

- [Your Voice Agent Doesn't Need a Frontier Model](https://www.youtube.com/watch?v=fnLBmfsI_Fg) — Joel Allou and Ornella Bahidika, Microsoft, AI Engineer World's Fair, July 2026. The talk this post is built on.
- [Voice agent architecture: STT, LLM, TTS pipeline design](https://deepgram.com/blog/voice-agent-architecture-stt-llm-tts-pipeline-design) — Deepgram, on where the stages sit and why they have to overlap.
- [Voice agent latency budget](https://soniox.com/wiki/voice-agent-latency-budget) — Soniox's per-stage breakdown; useful as a shape, not as a constant.
- [Building Effective Agents](https://www.anthropic.com/engineering/building-effective-agents) — Anthropic, on the line between predefined code paths and model-directed control.
- [12-Factor Agents](https://github.com/humanlayer/12-factor-agents) — HumanLayer; Factor 8, *own your control flow*, is the same argument from the reliability side.
- [Claude Haiku 4.5](https://www.anthropic.com/news/claude-haiku-4-5) — the small fast model used in the talk's demo.
- MDflow: [state machines for AI agents](/blog/state-machine-ai-agents) · [voice agent turn-taking](/blog/voice-agent-turn-taking) · [on-device AI agents](/blog/on-device-ai-agents-compute-budget) · [tiny language models](/blog/tiny-language-models-prompting-floor) · [MCP docs](/docs/mcp) · [API docs](/docs/api) · [FAQ](/faq)