On-Device AI Agents Run on a 16-Millisecond Budget

Most conversations about agent capability are about intelligence — which model, how many parameters, what benchmark score. Move the agent onto a phone and intelligence stops being the binding constraint. You run out of milliseconds first.

Shafik Quoraishee and Joanne Song of The New York Times gave a talk at AI Engineer in July 2026 on local agentic systems for mobile games, and the most transferable thing in it was not a technique. It was an accounting exercise: before you ask whether the agent is smart enough, work out what it is allowed to spend.

TL;DR — On-device AI agents are constrained by space, time and energy long before they are constrained by model quality. A 60fps app has ~16.6 ms per frame; a measured edge NPU produced 6.9 tokens/second, about 145 ms per token — nine frames for one token. So a real-time local agent cannot put a language model in its inner loop. What it can afford is cheap deterministic machinery per frame plus precomputed written context it reads rather than derives. The two hardest items on NYT's own roadmap were long-term memory of one person's habits and a shared state language across apps — both of which are documents, not model capabilities. Keeping that document readable by people and agents alike is what MDflow is for. Start free.

What is an on-device AI agent's compute budget?

An on-device agent's compute budget is the fixed allowance of space, time and energy it must fit inside while the app it lives in keeps running. On a server, all three are elastic — you buy more. On a handheld device they are hard walls, and the agent shares them with the product.

The three, as the NYT team framed them:

The useful move in the talk was treating these as a constraint graph with deliberately asymmetric penalties rather than a single resource ceiling. Time overruns were penalised hard, because a missed frame is something the user feels immediately. Space was soft-constrained, because borrowing a little extra memory is usually invisible. That asymmetry is a design decision, and it is the kind of decision that only becomes available once you have written the budgets down explicitly instead of discovering them in a crash report.

Energy is the one most teams underestimate, because it is the budget that moves. It is not a constant you can measure once in a benchmark run and design against.

The history this sits in

AI in games is old, and the trajectory matters for understanding what "agentic" adds.

1. Symbolic (1980s) — the ghosts in Pac-Man are a finite state machine: conditional, hand-authored, entirely deterministic.
2. Reinforcement learning — AlphaGo and AlphaZero, then the sample-efficiency line through EfficientZero. EfficientZero V2 (Wang et al., 2024) beat DreamerV3 on 50 of 66 tasks across Atari 100k, Proprio Control and Vision Control — state of the art at learning from very little data.
3. Agentic — a language model reasons over the game state in context, through tool calls, with no reward signal to grind out.

The distinction that matters for budgeting: RL changes the weights; an agentic system changes nothing and reasons every time. RL front-loads all its cost into training and leaves a cheap, fast policy behind. An agentic loop pays at every single decision — which is exactly the cost that a 16-millisecond frame cannot absorb.

That is not an argument against agentic systems on device. It is an argument about where in the loop they belong.

The arithmetic nobody likes

The gap between a frame budget and on-device token generation is roughly three orders of magnitude. It is worth doing the multiplication once, because it settles a lot of architectural argument.

A 60fps app has about 16.6 ms per frame. Practical mobile guidance splits that into roughly 8–10 ms of CPU, 8–10 ms of GPU, and 2–3 ms of headroom for OS overhead and throttling. The agent is asking for a slice of an allowance that is already fully committed.

Now the other side. In LLM Inference at the Edge (Tummalapalli, Arayakandy, Pal and Kundan, March 2026), a Hailo-10H NPU generated 6.9 tokens/second at under 2 W, against an RTX 4050 at 131.7 tokens/second and 34.1 W sustained.

At 6.9 tokens/second:

1 token   ≈ 145 ms   ≈  8.7 frames
40 tokens ≈   5.8 s  ≈  350 frames

A single token costs nearly nine frames. A short plan costs six seconds of wall time. There is no prompt engineering that closes that gap.

Two further findings from the same paper make it worse, and they are the ones that do not show up in a quick benchmark:

• The iPhone 16 Pro "loses nearly half its throughput within two iterations" under sustained load. Thermal management, not raw capability, is the primary mobile constraint.
• The Galaxy S24 Ultra hits a hard OS-enforced GPU frequency floor that terminates inference entirely.

Underneath both sits memory bandwidth. Mobile devices run at roughly 50–90 GB/s against a data-centre GPU's 2–3 TB/s — a 30–50× gap that is decisive because decode is memory-bound: the weights are streamed for every token generated, so the compute units idle waiting on memory. Offloading to an NPU does not automatically fix it either; autoregressive decode issues many small workloads and accrues scheduling overhead without enough token-level parallelism to amortise it, which can cancel the expected energy win.

The models are genuinely improving against these constraints. Google's Gemma 3n uses per-layer embeddings to keep only ~2 GB (E2B) or ~3 GB (E4B) of core weights in accelerator memory, and starts responding around 1.5× faster on mobile than Gemma 3 4B. That is real progress on the space budget. It does not turn 145 ms per token into 16.

The budget is a thing you can learn

If the budget is the constraint, the budget itself becomes something to optimise — and there is now research treating it that way.

In Finding the Time to Think: Learning Planning Budgets in Real-Time RL (Muppidi, Darwish, Cope, Henriques and Foerster, Oxford, August 2026), the authors start from the observation that in real-time settings the environment advances while the agent deliberates — so thinking longer is not free, it costs you world state. They train a lightweight gating policy on top of a frozen AlphaZero planner to decide, per state, how much planning to buy.

Learned state-dependent budgets against the best fixed budget:

The headline is not the percentages, it is the shape of the result: knowing when to think cheaply is worth more than thinking better. The policies also transferred to real hardware across GPU classes and frame rates without retraining, which is the property you need for anything shipping to a fleet of different phones.

What the New York Times actually built

Worth stating the disclaimer the team led with, because it is load-bearing: NYT puzzles are made by people, the games ship with no AI features, and this is experimental work on solvability and playability rather than generation.

The agentic Space Invaders player runs a compact loop on an on-device model: perceive the scene, predict where the attacking ships go next, decide whether a bullet or a ship is close enough to warrant action, act, then loop. Nothing exotic — but note that every one of those stages is a place where you either spend a model call or you do not.

The Mini Crossword solver is a constraint-satisfaction agent over a satisfaction graph. It places words, and when the constraints get too crowded or too wrong it backtracks efficiently rather than exploring blindly. This is the cheap-machinery-instead-of-inference pattern in its purest form: a graph and a backtracking rule do the work that a language model would pay tokens for.

The accessibility half is the more interesting product

Song's half of the talk reframed the whole thing, and it is where local agents earn their keep.

Traditional accessibility in games is a fixed state model: rigid, hand-authored, a toggle or a static menu. Deterministic, predictable, and — the actual flaw — blind to real-time need. A static "easy mode" cannot fix a crossword grid that remains fundamentally blind to the person holding the phone.

The grounding is WCAG 2.2 and its four human-centric principles: perceivable, operable, understandable, robust. The interesting signal is what comes next: the WCAG 3.0 working draft replaces binary pass/fail with graded bronze, silver and gold conformance scored across outcomes. Accessibility is moving from a checkbox to a scale — and still a working draft, with a final standard not expected before 2028.

The team's argument is that gameplay should make the same move. Instead of modes, dials that rise and fall with live context:

• Input tolerance — how forgiving interactions are around physical constraints.
• Step granularity — how far complex sequences get broken down.

The agent's entire job is to calculate those dials continuously, reading signals an on-device model can actually see: gaze tracking for search friction, shaky taps for motor difficulty, handwriting recognition as an alternative input path. And then to act rather than watch — monitoring the focus path, flagging a keyboard trap in a broken dialogue and injecting an exit route live, or measuring the interface on the fly and resizing controls that violate target-size rules. As Song put it, the agent rewrites the layout to adapt to the human rather than the other way around.

Accessibility and difficulty stop being separate features and become two ends of one continuously tuned dial. That is a product you cannot build with a cloud round trip in the loop, for the same reason you cannot build it with a static menu.

Their roadmap is mostly not a model problem

The talk closed on what still needs building for local agents to genuinely understand games. Five items:

1. Faster — plans and decisions inside a 16 ms frame.
2. Predictive models — see what a layout change will do before making it.
3. Long-term memory — learning one specific person's habits and needs over time.
4. A shared state language — so one agent works across multiple games instead of being rebuilt per release.
5. Better chips and honest benchmarks — real measurement that the agent actually helps.

Items 1, 2 and 5 are hardware and research. Items 3 and 4 are documents. A durable per-person profile and a shared vocabulary across applications are not capabilities you get from a larger model; they are written artefacts that have to live somewhere, survive app updates, and be readable by more than one consumer. That is a substrate problem wearing a model-capability costume.

Why compute budgets matter

For developers

Budgets change the architecture, not just the tuning. Once you accept 145 ms per token against a 16.6 ms frame, a set of decisions resolve themselves:

• The inner loop gets deterministic machinery — state machines, constraint graphs, small classifiers. Not a language model.
• Model calls move off the critical path: between turns, on idle, on a background thread, or ahead of time.
• Anything derivable in advance gets derived in advance, somewhere with no budget, and shipped down as data.
• Every input the agent consumes gets a cost. A vector search you cannot afford is not a retrieval strategy.

Measure under sustained load, not in a demo. A benchmark that runs for ten seconds measures a device that has not warmed up yet. The iPhone result above — half the throughput by the second iteration — is what your users get, and it is invisible in a cold-start number.

Battery is a product requirement. An agent that measurably shortens a session is a feature users disable, however good its decisions were.

For AI agents

Precomputed written context is the cheapest context an edge agent can consume. This is the whole practical consequence, and it generalises well past games.

An agent that must decide inside a frame cannot afford to infer that this user prefers larger tap targets and slower prompts — that inference is tokens, and tokens are frames and battery. It can absolutely afford to read it. A few hundred bytes of plain text is a file read: no embedding pass, no index, no round trip, no decode.

That makes the division of labour clear. The expensive work — noticing a pattern across weeks of sessions, deciding what it means, writing it down — happens where there is no budget: a server, a desk machine, overnight. The on-device agent consumes the conclusion. It reads a profile, not a history.

Which is also why NYT's fourth roadmap item matters more than it looks. A shared state language means the written form is stable enough that a different app, a different agent, and a human reviewer all read the same thing. The moment each app invents its own opaque profile format, you are back to rebuilding from scratch per release — exactly the problem that item was listed to solve.

Which applications benefit most

1. Real-time interactive apps — games, editors, instruments, anything with a frame budget and a user watching it.
2. Accessibility and adaptive interfaces — the signals are continuous and intimate (gaze, tap accuracy, hesitation), which is both a latency argument and a privacy argument for keeping them local.
3. Privacy-sensitive assistants — health, journaling, finance, messaging, where the compelling version of the feature is the one where the data never leaves the device.
4. Field and offline work — inspections, logistics, subway tunnels, aircraft. Connectivity is not a dependency you can design around.
5. Robotics and embedded control — the frame budget is a control loop, and the consequences of missing it are physical.
6. High-frequency personal automation — anything that runs constantly in the background, where per-invocation cloud cost and battery drain compound into something untenable.

How MDflow fits

What already lines up today

MDflow is a Markdown-native workspace built to be read by both people and agents, which turns out to be the right shape for the cheap half of an edge agent's context — with one honest caveat stated up front: MDflow is hosted. It is not an on-device store, and an agent working inside a frame budget should never be making a network call to it. The realistic pattern is that a small set of documents syncs down and gets read locally, the way any other asset would.

What that buys you today:

• Plain Markdown, not a database. A profile or a rule set is a file small enough to read directly and parse with anything. No embedding model, no index to build, no SDK. On a device where a single token costs nine frames, "parse a short text file" is the only retrieval budget that comfortably fits.
• Folder descriptions as retrieval signal. MDflow ranks folder descriptions above names and titles in mdflow_get_context, so a sentence like "current accessibility profiles — supersedes anything in /archive" steers a reader without a vector index. Free-text semantics, no inference cost.
• Raw .md twins. Every document is fetchable as plain Markdown with its frontmatter intact, so a sync job, a CI step or a build script can curl it and ship it into a bundle. No client library on the device side.
• One copy, many surfaces. MCP with OAuth for Claude and the ChatGPT app, a PAT for Cursor, Codex and Claude Code, plus VS Code and n8n. The desk-side agent that writes the profile and the human who reviews it are editing the same file the device build reads — that is the practical version of a shared state language.
• Version history with attribution. Per-actor line-by-line diffs, and API or MCP writes attributed as automated · <token name>. When behaviour shifts between Tuesday and Friday, you can answer the only useful diagnostic question quickly: did the model change, or did the written context?
• Client-side encryption. A behavioural profile — gaze patterns, tap accuracy, accessibility needs — is among the most sensitive data a product can hold. Encryption happens in the browser before upload, so the server stores ciphertext.
• A mobile app that already assumes bad connectivity. The iOS app queues share-sheet captures on the device and uploads them when the network returns, so capture works offline or signed out.

Where we are headed

Direction, not a dated commitment. The items that matter for this use case are offline reading and editing on mobile, so a synced set is fully usable with no connection at all; typed frontmatter carrying status, owner and freshness as signals a reader can weigh rather than conventions it has to guess; and pinning a written definition to a moment in time, so a device build can depend on a specific version of a profile rather than whatever was current at fetch. The throughline is unchanged: keep one copy of the written context, readable by a person, a desk agent and a build step alike.

The bottom line

On-device agents are a budgeting discipline before they are an intelligence problem. Space, time and energy are fixed, shared with the product, and — in the case of energy — shrinking while you use them. A 16.6 ms frame against 145 ms per token is not a gap you optimise away; it is a constraint you design around, by keeping cheap deterministic machinery in the inner loop and spending model calls rarely, elsewhere, and preferably in advance.

The corollary is the part worth carrying into any edge project: the cheapest thing an on-device agent can consume is a conclusion somebody already wrote down. Derive expensively where there is no budget, write it down in a form a person can also read, and let the device read rather than reason. NYT's two hardest roadmap items — a durable per-person memory and a shared state language across apps — are that same insight stated as a wish list.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What limits on-device AI agents the most?

Not model quality — resource budgets. An on-device agent competes for three fixed allowances at once: space, meaning the model weights plus state history plus planning artefacts must fit alongside whatever the app itself needs; time, because an interactive app at 60 frames per second has only about 16.6 milliseconds per frame and the agent's decision has to fit inside it or the interface visibly stutters; and energy, because sustained inference drains a battery and triggers thermal throttling. Measurements of edge inference under sustained load show an iPhone 16 Pro losing nearly half its throughput within two iterations, which means the budget also shrinks while you are using it.

How slow is an on-device language model compared to a frame budget?

Roughly three orders of magnitude. A 60fps app has about 16.6 ms per frame. In the arXiv paper LLM Inference at the Edge, a Hailo-10H NPU generated 6.9 tokens per second, which is about 145 ms per token — close to nine frames for a single token. A forty-token plan would cost about six seconds, or some 350 frames. That arithmetic is why real-time on-device agents do not put a language model in the inner loop; they use cheap deterministic machinery per frame and spend model calls rarely and off the critical path.

Should an on-device agent reason about user preferences or read them?

Read them. Inferring a preference costs tokens, and tokens on a phone cost frames and battery. Reading a short precomputed file costs a file read. This is why the useful design split is to do the expensive derivation somewhere with no budget — a server, a desk machine, overnight — write the conclusion down in a compact form, and let the on-device agent consume the conclusion. Precomputed written context is the cheapest context an edge agent can have.

What did the New York Times build with local agents?

Experimental work, not shipped game features — the team was explicit that their puzzles are made by people and the games contain no AI features. Shafik Quoraishee and Joanne Song demonstrated an agentic Space Invaders player running a perceive, predict, decide, act loop on an on-device model, and a constraint-satisfaction agent that fills the Mini Crossword by backtracking over a satisfaction graph. The second half of the talk was about accessibility: an agent that continuously tunes dials like input tolerance and step granularity to the player's live context, audits layouts against WCAG, and can detect a keyboard focus trap and inject an exit route on the device.

Does MDflow run on-device?

No, and it is worth being precise about that. MDflow is a hosted Markdown workspace, so an on-device agent would sync a small set of documents down and read them locally rather than query MDflow inside a frame. What MDflow provides is the shape an edge agent can afford: plain Markdown files small enough to read directly, folder descriptions that act as retrieval signals without a vector index, raw .md twins any client can fetch and parse with no SDK, and version history that answers whether the model changed or the written context did. Offline reading and editing on mobile is roadmap direction, not a shipped feature.

Further reading

Primary sources

• Local Agentic Theory For Mobile Games — Shafik Quoraishee & Joanne Song, The New York Times, AI Engineer, July 2026
• LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load — Tummalapalli et al., March 2026
• Finding the Time to Think: Learning Planning Budgets in Real-Time RL — Muppidi et al., Oxford, August 2026
• EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data — Wang et al., 2024
• Introducing Gemma 3n — Google Developers Blog
• WCAG 2.2 and the WCAG 3.0 working draft — W3C

From the MDflow blog

• Local LLMs in 2026: The Frontier Gap Is Closing
• Privacy-Preserving AI Agents
• State Machines for AI Agents
• Portable AI Memory
• The MDflow mobile app

MDflow docs

• MCP server · HTTP API · Markdown for AI · FAQ