Voice Agent Turn-Taking: The 200 ms Problem

Two voice agents. Same model, same prompt, same system message. In the first, you start saying "I want to fly to —" and change your mind halfway through, and the agent keeps talking for another two seconds while you sit there trying to get a word in. In the second, the agent notices you inside 200 milliseconds and backs off.

Nothing about the model explains the difference. Chintan Agrawal and Daniel Wirjo, both solutions architects on the AWS startups team, opened their AI Engineer talk in July 2026 with exactly that comparison to make the point: turn-taking is an audio engineering problem, not an LLM problem. You can have a perfect model, a perfect prompt, and a completely broken conversation.

TL;DR — Voice agent turn-taking is the machinery that decides when the user has stopped speaking and when the agent should shut up. It lives in three small components around the model — voice activity detection, end-of-turn detection, and an interruption handler — and it comes in three levels: a local silence timer, an STT service that emits turn events, or your own turn model running on top of a local VAD. Humans hand off turns in about 200 ms; cascaded pipelines calling cloud APIs run 800–1,300 ms, and roughly two thirds of that is speech-to-text plus model time-to-first-token. Which makes context discipline a latency feature — and a reason to keep an agent's knowledge somewhere retrievable like MDflow rather than pasted into a system prompt that decays after twenty turns.

What is turn-taking in a voice agent?

Turn-taking is how a voice agent decides that you have finished speaking and it is now its turn — and how it gets out of the way when you start speaking over it. It is not something the language model does. It happens in the audio pipeline in front of the model, and it is usually the reason a demo that reads perfectly in a transcript feels awful on a phone call.

The standard cascaded pipeline is familiar:

  audio in → STT → LLM → TTS → audio out

What is often missing from that mental picture are the three pieces that actually govern the conversation:

VAD is the tiny one at the front. End-of-turn detection watches the VAD signal plus the audio features and makes the actual "respond now or wait" call. The interruption handler fires when the user talks over the agent: it stops text-to-speech playback and cancels the in-flight generation so the pipeline is clean for new input within roughly 50 ms.

The 200 millisecond constraint

Two hundred milliseconds is not a product target someone picked — it is roughly how fast humans hand off turns to each other, everywhere. Stivers and colleagues sampled ten languages from traditional indigenous communities to major world languages in a 2009 PNAS study and found the same underlying pattern in all of them: a strong avoidance of overlapping talk and a modal transition gap clustered around 200 ms, despite genuine cultural variation in style.

The implications for a voice agent are unforgiving. Around 800 ms, things start to feel off. By about 1.5 seconds, users assume the connection dropped and hang up. An HR chat agent gets five seconds to respond and nobody minds. Voice gets none of that slack.

And the honest state of play is that nobody is close. Agrawal cited 755 ms as the best measured voice-to-voice figure he had seen published for a cascaded pipeline in 2026 — roughly four times slower than a human turn transition. Here is where a typical budget goes:

  mic capture + encode        ~40 ms     physics
  network + jitter buffer     ~52 ms     physics
  STT + endpointing          ~300 ms     ← lever
  LLM time to first token  500–650 ms    ← dominant bottleneck
  TTS + network out + play  85–120 ms
  ─────────────────────────────────────
  total                  ~1,100–1,300 ms

Those numbers come from production measurements by Kwindla Hultman Kramer, co-founder of Daily and creator of Pipecat. His team has demonstrated around 500 ms total voice-to-voice by co-locating every model in one GPU cluster and eliminating the network hops — an achievable floor if you are willing to buy the infrastructure. For everyone else calling cloud APIs, 800 to 1,300 ms is the real range.

STT and the LLM together eat about two thirds of the budget. They are the only two levers that move the needle meaningfully. Everything else is rounding error or physics.

Which is the whole reason turn-taking matters so much. If you cannot close the latency gap, the next best thing is to make sure the agent never feels like it is stepping on you — because a 900 ms response that arrives at the right moment reads far better than a 700 ms response that arrives while you are still mid-sentence.

The three levels of turn detection

Agrawal framed the design space as three levels. The striking thing in the demo is that all three are the same Pipecat pipeline — the code is nearly identical, and only the configuration of one component changes. The behaviour is completely different.

Level 1 — a local VAD and a silence timer

Silero VAD is the default here, and plenty of production systems still run on nothing else. It is a genuinely tiny model — around 309,000 parameters and a couple of megabytes on disk — that takes raw audio, converts it to spectral features via a short-time Fourier transform, runs a few convolutional layers plus an LSTM for cross-frame memory, and emits a speech probability. It processes 32 ms chunks in well under a millisecond on a single CPU thread.

At this level, one parameter is your entire user experience: minimum silence duration.

# level 1 — you own the silence decision
SileroVADAnalyzer(params=VADParams(stop_secs=0.3))

Set it low and the agent snaps — it cuts people off while they are still thinking. Set it high and the agent becomes so patient that users wonder whether the call is still connected. There is no universally right answer: a sales agent might want 200 ms, while a domain where users need a moment to think might want 1,000–1,200 ms.

The limitation is structural, not tunable. A VAD answers an acoustic question. It cannot answer a semantic one. Four situations produce an identical 400 ms silence:

1. A completed sentence.
2. An incomplete thought.
3. A thinking pause.
4. A backchannel acknowledgement ("mm-hm").

The intent is completely different in each. The VAD sees the same thing in all four.

Level 2 — let the STT service tell you

Several speech-to-text providers now do turn detection inside the same websocket that carries the transcription, emitting an end-of-turn event when they think the turn is done. Deepgram's Flux is built specifically for this, folding end-of-turn detection into the recognition model itself and reporting median end-of-turn detection under 300 ms; Cartesia does the same in its streaming STT. Agrawal put both in the 250–300 ms P50 range in their testing.

These work well, because the service has both the full audio signal and the linguistic context — far more information than a VAD will ever have.

The trade-off is transparency. When it works it works great. When it misfires and cuts someone off, there is no log line telling you why. The decision was made on someone else's server and you live with it.

Level 3 — local VAD plus your own turn model

Keep a small VAD running locally for the basic "is there speech" signal, then add a semantic turn model that runs during the silence and decides whether the pause meant I'm done or I'm still thinking, using prosody and intonation rather than words.

Smart Turn is the open one: BSD-2 licensed, open data and open training code, weights on Hugging Face, 23 languages, around 8 MB for the CPU build, and fast enough to need no GPU — Daily reports roughly 12 ms inference on a modern CPU and about 60 ms on a cheap AWS instance.

The accuracy is the part people find surprising. Agrawal quoted 58.9% recall and 68.4% precision for v3.2 in their June 2026 measurements. Roughly six times out of ten the model catches a finished sentence quickly. The other four times it is not confident enough.

That is fine, and understanding why is the important part. The VAD timer is still running underneath as a safety net. If the turn model does not fire, the silence timer kicks in at its configured threshold anyway. You are never stuck waiting. You get a fast response when the model is confident and a slightly slower but safe one when it is not. It degrades to level 1 rather than to broken.

Barge-in: not every interruption is an interruption

The other direction — the user speaking while the agent is talking — has a mechanical half and a judgement half.

The mechanical half is largely solved by the frameworks. The VAD picks up speech in about 32 ms, the pipeline flushes in roughly 15 ms, TTS stops, the LLM call is cancelled, and the user never hears the agent talk over them.

The judgement half is not solved, and it is where the user experience actually lives. In a normal conversation, if someone says "yeah" while you are speaking, you do not stop — they are agreeing. If they say "okay, wait, no, that was wrong," you stop immediately. Sorting those apart is a classification problem:

• Stop everything — a correction or an objection. The user has new information and the current utterance is now wrong.
• Keep going, maybe quieter — a filler or a backchannel. They are following along.
• Ignore completely — a cough, a door, background noise.

Most production systems today stop on every one of them. That is why false interruptions matter more than they sound: when an agent cuts people off incorrectly, they are measurably more likely to demand a human. Turn-taking failures do not show up as bad transcripts. They show up as escalation rate.

Which applications benefit most

1. Phone support and contact centres, where a false interruption converts directly into a request for a human and a blown deflection rate.
2. Outbound sales and scheduling agents, where a snappy 200 ms cadence is the whole illusion and a 1.5 second pause ends the call.
3. Healthcare intake and triage, where users need long thinking pauses and an impatient silence timer talks over someone mid-symptom.
4. In-car and hands-free assistants, where background noise makes naive VAD thresholds fire constantly.
5. Language learning and accessibility tools, where non-native or slower speech patterns break silence timers tuned on fluent speakers.
6. Drive-through and kiosk ordering, where corrections mid-utterance ("actually, make that a large") are the normal case rather than an edge case.

How MDflow fits

Let us be precise: MDflow is not in the real-time audio path. It is not a VAD, it is not an STT service, and nothing about it will move your P50. Anyone selling you a knowledge base as a latency fix is selling you something.

What it does touch is the part of the budget nobody talks about in a turn-taking talk — and Agrawal raised it anyway, because it bites in production.

After 15 or 20 turns, models start ignoring parts of the system prompt. They get verbose. They go off script. In voice, that is fatal — you cannot just dump a wall of text on someone. So you need context pruning or session resets.

That is the same problem the rest of agent engineering has been circling for two years, arriving from a different direction. In a chat agent, a bloated system prompt costs you money and a bit of quality. In a voice agent it costs you the two things you have least of: time-to-first-token, which is already your dominant bottleneck, and instruction adherence across a long call. Context discipline stops being hygiene and becomes a latency feature.

What already lines up today

• Markdown-native storage. The policies, product facts, escalation rules and scripts a voice agent speaks from are prose. They live in MDflow as plain markdown that a person edits in a browser — no export step between the version the compliance team reviews and the version the agent reads.
• Retrieval that returns few documents, not everything. mdflow_get_context takes a topic and ranks folder descriptions above folder names and document titles, then returns the best-matching markdown bodies. You write the description; nothing is embedded, so nothing goes stale after an edit. For a voice agent, the useful property is that it is designed to hand back a handful of relevant documents rather than a dump — which is exactly what you want to fetch before the call, or between turns, and never mid-utterance.
• Compounded folder descriptions. A folder inherits its parents' descriptions, so "escalation rules" nested under "EU / regulated" carries that qualification with it. Scoping context by folder is a cheap way to keep a call's prompt small.
• Three ways in, one copy of the truth. MCP for agent frameworks, a plain HTTP API with a Personal Access Token for anything else, and raw .md twins of shared documents with YAML frontmatter for the simplest possible fetch. Your Pipecat process and your reviewer are reading the same file.
• Version history and comments. When the escalation rate moves on a Thursday, "did the pipeline change or did the script change?" is answerable rather than a group chat argument.
• Team workspaces. Support leads, compliance and the engineers building the pipeline can share a workspace with per-workspace permissions instead of a prompt string pasted into three repos.

Where we are headed

Direction, not a dated commitment: we are interested in making retrieval tighter — the ability to ask for a slice of a document rather than a whole body, so an agent operating under a hard latency budget can pull the two paragraphs it needs. And in richer provenance on retrieved context, so an agent that quotes a policy on a recorded call can say which version it read. Both serve the same goal: making the retrieved payload smaller and more accountable, which is what a voice pipeline needs and what a chat agent quietly wants too.

The bottom line

Turn-taking is the part of a voice agent that decides whether it feels like a conversation, and it lives entirely outside the model. Start at level 1 and tune one number honestly against your domain. Move to level 2 or 3 when the silence timer starts costing you escalations — and know that level 3 buys you portability and visibility, not magic accuracy, because it degrades gracefully to the timer underneath it.

Then look at the other two thirds of your budget. Speech-to-text and time-to-first-token are where the milliseconds actually are, and the size of what you put in front of the model is one of the few parts of that you fully control. Keeping the agent's knowledge retrievable — small, versioned, and outside the prompt — is the unglamorous half of making a voice agent fast.

Start free · Connect an AI agent over MCP · Read the API docs

Frequently asked questions

What is turn-taking in a voice agent?

Turn-taking is how a voice agent decides that you have finished speaking and it is now its turn to reply, and how it backs off when you start speaking over it. It is handled by three small components sitting around the model — voice activity detection, end-of-turn detection, and an interruption handler — not by the language model itself. Two agents running the identical model and prompt can feel completely different depending on how those three are configured.

Why is 200 milliseconds the target for voice agents?

Because that is roughly how fast humans hand off turns to each other. Stivers and colleagues measured turn transitions across ten languages in a 2009 PNAS study and found a modal gap of about 200 ms everywhere, despite large cultural differences in style. Anything much slower reads as hesitation, and by around 1.5 seconds users assume the line has dropped. Cascaded voice pipelines calling cloud APIs typically land at 800 to 1,300 ms voice-to-voice, so turn-taking has to do the work that raw speed cannot.

What is the difference between VAD and end-of-turn detection?

Voice activity detection answers a purely acoustic question — is someone speaking right now — and it answers it in milliseconds. End-of-turn detection answers a semantic one: was that pause the end of a thought, or a breath in the middle of one. A VAD sees a 400 ms silence after a completed sentence and a 400 ms silence after the word "um" as exactly the same signal. That is why a silence timer alone either cuts people off or leaves dead air.

What is barge-in handling in a voice agent?

Barge-in handling is what happens when the user starts talking while the agent is still speaking. Mechanically the pipeline must flush fast — stop text-to-speech playback and cancel the in-flight model generation within tens of milliseconds — so the user never hears the agent talk over them. The harder part is judgement: a "yeah" or a cough is not an interruption, while "no, wait, that's wrong" is. Most production systems today stop on every sound, which is why false interruptions are a common source of escalations to a human.

How does MDflow relate to voice agents?

MDflow is not in the real-time audio path. It holds the knowledge the agent speaks from — policies, product facts, escalation rules, scripts — as plain markdown documents that a person edits in a browser and an agent retrieves over MCP or the HTTP API. Because two thirds of a voice agent's latency budget is speech-to-text plus model time-to-first-token, context has to be small and precise rather than a wall of pasted text, and MDflow's folder descriptions and mdflow_get_context are built to return the few relevant documents instead of everything.

Further reading

• Voice Agents That Handle Interrupts — Chintan Agrawal and Daniel Wirjo, AWS, at AI Engineer (July 2026). The talk this post is built on.
• Universals and cultural variation in turn-taking in conversation — Stivers et al., PNAS 2009. The source of the 200 ms figure.
• Pipecat — the open-source voice agent framework used in all three demos.
• Silero VAD — the level 1 voice activity detector.
• Smart Turn and smart-turn-v3 on Hugging Face — the open semantic turn detection model.
• Deepgram Flux end-of-turn configuration — a level 2 implementation and its tuning parameters.
• Context engineering for AI agents — why what you put in front of the model is a design decision.
• Tool output and the context window — the same budget problem, from the tool-calling side.
• MDflow MCP docs · MDflow API docs · FAQ