---
title: "Tiny Language Models: Where Prompting Stops Working"
description: "Tiny language models under 1B parameters cannot be prompted into reliability — they have to be fine-tuned. Why DRAM forces the choice, and what the fine-tune costs."
author: "MDflow"
date: 2026-11-04
reading_time: "19 min"
canonical_url: https://mdflow.cz/blog/tiny-language-models-prompting-floor
md_url: https://mdflow.cz/blog/tiny-language-models-prompting-floor.md
---

# Tiny Language Models: Where Prompting Stops Working

*Published November 4, 2026 · 19 min read*


Most of the attention in AI engineering goes to the top of the size ladder. The interesting engineering is increasingly at the bottom, among **tiny language models** small enough to ship inside an app — where the constraint is not how smart the model is but whether it fits in memory at all, and where the familiar workflow of writing a good prompt simply stops producing a working feature.

Cormac Brick, a tech lead on Google's AI Edge team, made that break explicit at AI Engineer in July 2026 in a talk called *Why Large? Tiny LMs & Agents on Edge/Robotics*. The useful part is not the benchmark numbers. It is the line he draws across the size ladder: above it, you prompt a model and ship; below it, you fine-tune or you get nothing. Two completely different jobs, separated by about a billion parameters.

> **TL;DR** — Between roughly 1B and 4B parameters, a small language model is something you *buy*: zero-shot prompt it and ship the feature. Below about a billion, prompting stops working, and the only lever left is a fine-tune on a synthetic data set — which means the capability becomes something you *author*. The durable artifact is no longer a prompt; it is the written definition of the task that generates the data, plus everything the model is too small to memorise and must be handed on every turn. [MDflow](https://mdflow.cz) is a versioned markdown workspace built for exactly that layer — plain markdown with raw `.md` twins, folder descriptions as retrieval context, and the same copy reachable over [MCP](/docs/mcp) and an [HTTP API](/docs/api).

## What tiny language models are

**A tiny language model is roughly 50 million to 500 million parameters — small enough to ship inside an application binary and run in under 2 GB of RAM.** That is one to two orders of magnitude below the models most people mean by "small", and the gap matters more than the names suggest.

| Tier | Size | Memory | How you get behaviour | Where it lives |
| --- | --- | --- | --- | --- |
| Small | ~1B–4B | ~4 GB+ device | Zero-shot prompting, LoRA adapters | Phone OS, high-end laptops, pricier robotics |
| Tiny | ~50M–500M | under ~2 GB | Fine-tuning on task-specific data | App binaries, browsers, IoT boards, consumer robotics |

The small tier is already in your pocket. A model of that class ships with Android's AI core on high-end phones and with Apple Intelligence; some app vendors bundle their own. Brick's point about this tier is that it is *easy*: "you just need to zero-shot prompt in order to get it to work", and if your product can live inside its memory budget, your engineering journey stops there.

The tiny tier exists because for a great many products, the journey cannot stop there.

## Why DRAM, not compute, sets the ceiling

**The binding constraint on edge AI in 2026 is memory cost, not arithmetic.** This is the reversal that makes the rest of the argument necessary, and it is not a modelling problem at all — it is a bill-of-materials problem.

Memory got expensive, fast. Raspberry Pi has raised board prices repeatedly through 2026, every time because of memory. Its February announcement of [more memory-driven price rises](https://www.raspberrypi.com/news/more-memory-driven-price-rises/) blamed "an unprecedented rise in the cost of LPDDR4 memory" caused by competition for fab capacity from the AI build-out, reported that some parts had more than doubled in cost over a single quarter, and scaled the increase purely by memory density — $10 on 2 GB boards, $60 on 16 GB ones. Brick noted the same pressure on the other side of the market: some phone manufacturers are shipping *less* RAM this year than last. Every gigabyte your model needs is a gigabyte on every unit you ever sell.

Here is the arithmetic that follows, using the numbers Brick gave for the E2B model in the [Gemma 4](https://deepmind.google/models/gemma/) family — the member built for phones and edge devices. Google's AI Edge team squeezes it with a mix of 2-bit, 4-bit and 8-bit quantization down to about 2.9 bits per weight, plus tricks like per-layer embeddings. The result:

```text
weights in memory          ~841 MB
+ runtime and KV cache     → ~2 GB active RAM
+ OS and the rest of the app
= a 4 GB+ device, as a rule of thumb
```

A 4 GB floor sounds modest until you price it across a product line. It puts a small model within reach of laptops, phones and higher-end electronics, and out of reach of lower-tier browsers, most of the IoT market and consumer robotics.

Two second-order effects make the squeeze tighter than the headline number.

**The model is usually not the main feature.** Brick described it as a common pattern: the thing you want the model to do "is like one tiny thing in a corner that needs to run while everything else in the system is running". Your AI feature does not get the device's memory budget. It gets whatever is left after the product the user actually bought.

**Decode speed is memory-bound.** On a Raspberry Pi, Brick measured the 2B-class model at about 7.6 tokens per second of decode with the team's own runtime, [LiteRT-LM](https://developers.google.com/edge/litert-lm/overview). A roughly 270M-parameter model on the same board: about 45 tokens per second. There are fewer weights to read from memory for every token produced, and on a memory-starved board that is the whole story. The same model reaches around 24 tokens per second on a Jetson Orin Nano and, on a Qualcomm IoT board with an NPU, close to 4,000 tokens per second of prefill at about 31 tokens per second of decode — enough, at roughly 1,120 tokens for a high-resolution image, for about three frames per second of visual input with usable decode left over.

So the pressure is consistent in one direction: smaller is not a compromise you accept, it is the thing that decides whether the product exists.

## Where prompting stops working

**Below roughly a billion parameters, prompt engineering stops being a reliable way to get behaviour out of a model.** This is the line in Brick's talk that generalises past edge hardware, and the most interesting claim in it.

Above the line, the playbook is prompting, occasionally with LoRA adapters, and it is "somewhat robust" even at function calling and agent skills. Below the line, the playbook changes completely: you either find an off-the-shelf fixed-task model that already does what you need, or you fine-tune.

Google says the same thing in its own words. The [FunctionGemma](https://blog.google/technology/developers/functiongemma/) release in December 2025 — a 270M-parameter model built on Gemma 3 270M and tuned for function calling — describes the model as "designed to be molded, not just prompted", and gives the reason plainly: you need "the consistent, deterministic behavior that comes from fine-tuning on specific data, rather than the variability of zero-shot prompting."

The accompanying number is the one to remember. On Google's Mobile Actions evaluation — turning natural language like *"create a calendar event for lunch tomorrow"* into the right OS call — fine-tuning took the model from a 58 percent baseline to 85 percent. Brick reported a comparable figure for his team's own mobile actions model: over 86 percent reliability across about ten output functions, from arbitrary free text.

Read those two numbers as a boundary condition rather than a benchmark. A 58 percent baseline is a model that is not shippable at any amount of prompt iteration. The gap between it and 85 percent was not closed by better instructions. It was closed by data.

That is the reframe worth carrying out of the talk:

- **Above the prompting floor, capability is something you buy.** A better base model ships and your feature improves for free.
- **Below it, capability is something you author.** The artifact that makes the model useful is a data set, and a data set is downstream of a written definition of the task.

## The fixed-task playbook

**The route to a shippable tiny model is a loop with exactly one expensive step, and that step is building the data.** Brick laid the sequence out as a repeatable playbook, and it is worth following in order, because the first step frequently ends it.

**1. Check whether a stock fixed-task model already does the job.** Automatic speech recognition, vision and embeddings are solved at this size and need no training at all. Apple's [FastVLM](https://huggingface.co/apple/FastVLM-0.5B) at 0.5B parameters, converted for LiteRT, gives a device a first level of visual awareness with hardware acceleration; small ASR and text-embedding models are equally stock. If a fixed-task model covers your feature, you are done and the rest of this list is wasted effort.

**2. Pick the smallest plausible base model and measure it on the real device first.** Footprint and decode speed, on the actual board, before any training. Brick's starting points were Gemma 3 at 270M for general-purpose work and FunctionGemma for function-calling patterns, which carries extra pre-training for the shape of the task. If the base model is already too slow or too large unfine-tuned, no amount of training will fix it.

**3. Write the task down.** This is the step that is easy to skip and expensive to skip. Before any data exists, somebody has to state what the model must do: the catalogue of functions it can emit, the argument schemas, the phrasings real users produce, what counts as out of scope, what it should refuse, and how it should behave when the input is ambiguous. Nothing downstream is better than this description.

**4. Generate the synthetic data set.** Brick's working range is 10,000 to 10 million synthetically generated samples, depending on the task. Google published the `google/mobile-actions` data set and a fine-tuning notebook so the Mobile Actions result can be reproduced end to end.

**5. Fine-tune, then verify on a held-out set.** The shape to expect is the FunctionGemma shape: a mediocre baseline moving to a usable number, measured on data the model has not seen.

**6. Re-run when a better base model appears.** This is where the economics land. Brick's own summary of the outcome was that work which would need a 2B or 4B model with prompting can be matched — "the same or greater quality" — by a much smaller fine-tuned model, on a much wider set of devices, with much better responsiveness, provided you are willing to invest in the data set.

Note what survives each turn of that loop. The weights do not: a new base model drops and you retrain. The data set largely does. The written definition of the task definitely does, and it is the thing that regenerates everything below it.

## The interface this unlocks: speaking instead of navigating

**The highest-value thing a fine-tuned tiny model buys you is voice-to-function-calling on devices with no room for a good interface.** Brick called this out as the key use case, and the reasoning is UX rather than ML: small devices tend to require settings menus, and settings menus are genuinely difficult for a lot of people. Put a tiny ASR model in front of a fine-tuned function-caller and the menu collapses into a sentence.

The second example from the talk shows the same architecture doing something less obvious. The team built an offline voice dictation app whose entire backbone is two fine-tuned Gemma models in the low hundreds of millions of parameters: an ASR engine, and a "text policing" engine that cleans up the *ums* and *ahs* and biases transcription toward names and vocabulary that matter to that specific user. Highly accurate dictation used to be a server feature behind a subscription. Here it runs locally, with no network, on two models that fit in a phone app.

Brick also noted the browser case: Chrome's built-in [Summarizer and Proofreader APIs](https://developer.chrome.com/docs/ai/built-in-apis) reach a far wider set of users when the feature can be delivered by a model small enough for the machines those users actually have.

## What the tiny model does not carry

**The smaller the model, the larger the job of everything around it.** This is the inversion that the size-ladder framing tends to hide, and it is the part worth designing for.

A 270M-parameter model has no room for your world. It does not know your function catalogue. It does not know your users' names, your product's jargon, your internal terminology. It has no memory of where a workflow has got to. Every one of those things has to arrive from outside, on every single turn, assembled by code you wrote. That is exactly the architecture a second AI Engineer talk described from the other direction: Microsoft's Joel Allou and Ornella Bahidika built a voice tutor on a small model by extracting *all* the reasoning into a state machine and handing the model a summary each turn — the subject of [state machines for AI agents](/blog/state-machine-ai-agents). Their rule was to pick the fastest model the latency budget allows and spend the rest of the time building the scaffolding, because the scaffolding is paid once in code rather than on every turn.

Three categories of input have to come from somewhere, and in practice they come from prose:

- **The catalogue of actions** — not only the function signatures, which live in code, but what each one *means*, when it applies and when it must not, which is the part that determines whether the generated training data covers reality.
- **The vocabulary** — the names, product terms and domain jargon that a biasing or personalisation layer needs in order to stop a tiny ASR model from guessing.
- **The state summary** — a compressed description of where things stand, written by your harness in the shape the model was trained to consume.

And there is a fourth consumer of that prose, which Brick named as the direction he most wants to go: having an agent generate the synthetic data set for you. The moment that happens, the written definition of the task stops being documentation for humans and becomes machine input — read at build time by the generator, read at run time by the harness, revised whenever the task changes. It needs to be versioned, retrievable, and readable by a model without scraping a web page.

## Which applications benefit most

1. **Consumer hardware and entry-level robotics**, where the bill of materials rules out 4 GB of RAM and the alternative is no intelligence at all.
2. **Phone apps with an AI corner feature**, where the model must coexist with the product the user actually opened.
3. **Browser-delivered features**, where reach is capped by the weakest machine you want to support.
4. **Voice-first interfaces for devices with no good screen** — appliances, wearables, industrial panels, anything whose settings menu is a usability problem.
5. **Privacy-bound and offline workflows**, where the data cannot leave the device and the feature must work with no reception.
6. **High-volume, low-margin features**, where cheap per-call tokens still multiply into a serious bill at consumer scale.

## How MDflow fits

**MDflow does not fine-tune models, generate training data, or run anything on a device.** It holds the layer that the playbook above keeps returning to: the written definition of the task, the catalogue, the vocabulary, the edge cases — the prose that both a generator agent and your runtime harness have to read, and that outlives every base model you train on.

### What already lines up today

**Plain markdown, with a raw twin for the machine.** Documents are stored as markdown and served as a [raw `.md` file](/markdown-ai) with frontmatter intact. A data-generation pipeline fetches the task definition as bytes rather than parsing HTML, and the copy a human edits is the copy the generator reads.

**Folder descriptions as the retrieval signal.** Every folder carries a description of what belongs in it, and `mdflow_get_context` ranks those descriptions **above** folder names and document titles. A folder described as *"the action catalogue for the on-device assistant: one document per function, with argument schemas, example phrasings and refusal cases"* is a retrieval hook you wrote rather than one a model inferred — the argument in [folder descriptions as agent context](/blog/folder-descriptions-agent-context).

**The same copy from every surface.** The workspace is reachable from Claude, ChatGPT, Cursor and Codex over the [remote MCP server](/docs/mcp), and from CI, cron and pipeline code over the [HTTP API](/docs/api) with a personal access token. A training-data job and the engineer reviewing the spec do not get to drift apart, because there is no second copy to drift to.

**Version history answers which spec produced which checkpoint.** Every save captures the previous version across every write path — editor, API and MCP — with line-by-line diffs and non-destructive restore. When a fine-tune regresses, "what changed in the task definition between these two runs" is a diff rather than an argument. (Pro, private to the owner, and deliberately not exposed over the API or MCP.)

**Domain vocabulary is a document, not a config file.** The names and jargon a biasing layer needs are edited by the people who know them. Shared documents with passage-anchored comments let a domain expert correct a phrasing list without opening a pull request, which is the same argument [domain expert evals](/blog/domain-expert-evals) makes about who gets to define good.

**Encryption for the parts that are proprietary.** Client-side encryption is available for documents whose contents — an internal action catalogue, customer vocabulary — should not sit in plaintext in a vendor's database.

### Where we are headed

Direction rather than a dated commitment: better ways to pin a *specific version* of a document to a specific run, so "which task definition produced this checkpoint" stops being archaeology, and richer structured retrieval over folder descriptions for pipelines that assemble context programmatically.

**Stated limits:** MDflow stores and serves the prose. It cannot tell you whether the task definition is complete, and a fine-tune generated from a spec that omits half the real inputs will be confidently wrong at 270M parameters just as it would be at 270 billion.

## The bottom line

The size ladder has a seam in it. Above roughly a billion parameters, a model is a component you prompt and ship, and the next release makes your feature better for free. Below it — where the DRAM budget on real consumer hardware is pushing more and more products — prompting stops producing reliable behaviour, and the capability becomes something you author out of a written task definition and a synthetic data set.

That is a shift in what you own. The weights are disposable; you will retrain them when the next base model lands. The description of the task is not, and neither is everything the model is too small to memorise and must be handed on every turn. Both are prose, both are read by machines, and both need a home with a version history.

[Start free](/login) · [Connect an AI agent](/docs/mcp) · [Read the API docs](/docs/api)

## Frequently asked questions

### What is a tiny language model?

A tiny language model is roughly 50 million to 500 million parameters — one or two orders of magnitude smaller than the 1B to 4B models that ship inside phone operating systems. At that size the weights fit in a few hundred megabytes, the model can be shipped inside an application binary rather than downloaded, and it runs in under 2 GB of RAM, which puts it within reach of low-end browsers, IoT boards and consumer robotics instead of just high-end phones and laptops.

### When should I fine-tune a small model instead of prompting a bigger one?

When the task is fixed and narrow, and when the device cannot afford the bigger model. Between roughly 1B and 4B parameters, zero-shot prompting usually gets you a working feature and you should stop there. Below about 1B, prompting stops producing reliable behaviour and a fine-tune on task-specific data is the only remaining lever. Google's own FunctionGemma release states this directly: you need the consistent behaviour that comes from fine-tuning on specific data rather than the variability of zero-shot prompting.

### How much synthetic data do I need to fine-tune a tiny model for one task?

Cormac Brick of Google's AI Edge team put the working range at 10,000 to 10 million synthetically generated samples, depending on the task, and reported that this is enough to take a tiny model to high reliability on a single fixed job — matching or beating what a 2B to 4B model does with prompting alone. Google reports that fine-tuning took a 270M-parameter model from a 58 percent baseline to 85 percent accuracy on its Mobile Actions evaluation.

### Why is RAM, not compute, the limiting factor for on-device AI?

Because memory is what you have to buy on every unit, and memory got expensive. Raspberry Pi has raised board prices repeatedly through 2026, each time blaming the cost of LPDDR4 memory — by February the increase scaled with memory density alone, from 10 dollars on 2 GB boards to 60 dollars on 16 GB ones — and some phone makers are shipping less RAM than the year before. A 2B-class model needs around 2 GB of active RAM once weights, runtime and KV cache are counted, which implies a 4 GB device before the operating system and the rest of your application. Decode speed is memory-bound too: fewer weights to read per token is why a 270M model decodes several times faster than a 2B one on the same board.

### What does a tiny model need from my knowledge base?

Everything it does not have room to memorise. A 270M-parameter model carries no world knowledge, no catalogue of your functions, no record of your users' vocabulary and no memory of where a workflow has got to. All of that has to arrive from outside on every turn, and the written definitions behind it — the function catalogue, the phrasings, the edge cases — are also what a generator agent reads when it produces the synthetic training set. The smaller the model, the more load the context layer carries.

## Further reading

- [*Why Large? Tiny LMs & Agents on Edge/Robotics*](https://www.youtube.com/watch?v=hacEQHHhu2Q) — Cormac Brick, Google AI Edge, AI Engineer, July 2026. The talk this post is built on.
- [FunctionGemma: Bringing bespoke function calling to the edge](https://blog.google/technology/developers/functiongemma/) — Google, December 2025. The 58-to-85 percent Mobile Actions result and the "molded, not just prompted" framing.
- [Gemma](https://deepmind.google/models/gemma/) — the open model family, including the Gemma 4 E2B and E4B models built for phones and edge devices, and Gemma 3 270M.
- [LiteRT-LM](https://developers.google.com/edge/litert-lm/overview) — Google's on-device runtime, and the [AI Edge Gallery](https://github.com/google-ai-edge/gallery) app for trying these models on a phone.
- [More memory-driven price rises](https://www.raspberrypi.com/news/more-memory-driven-price-rises/) — Raspberry Pi on the DRAM squeeze.
- [Don't Let the LLM Drive: State Machines for AI Agents](/blog/state-machine-ai-agents) — the scaffolding half of the same argument.
- [Local LLMs in 2026: The Frontier Gap Is Closing](/blog/local-llms-frontier-gap) — what is happening at the other end of the size ladder.
- [On-Device AI Agents Run on a 16-Millisecond Budget](/blog/on-device-ai-agents-compute-budget) — the latency side of the same constraint.
- [Folder Descriptions as Agent Context](/blog/folder-descriptions-agent-context) — how MDflow makes written context retrievable.