Local LLMs in 2026: The Frontier Gap Is Closing
Every prediction about AI capability for the last three years has pointed up and to the right — bigger models, bigger clusters, bigger bills. There is a second trend line running underneath it that gets far less attention, and it points down: the amount of hardware needed to reach any given level of capability keeps collapsing.
At AI Engineer, Ahmad Osman of Osmantic gave a talk called The Desktop Frontier built entirely on that second line. His headline prediction: within roughly 18 months, GLM-5.2-class intelligence will run on a single RTX 5090 with 32 GB of VRAM. He called the estimate conservative. He also noted that in December he predicted frontier-quality inference on a single RTX Pro 6000 — and that it landed by March.
TL;DR — Local LLMs are closing the gap with frontier models much faster than the headline model releases suggest, because capability per parameter is improving on its own exponential. The densing law — published in Nature Machine Intelligence — finds that capability density doubles roughly every 3.5 months, so the parameter count needed for a given capability halves about every quarter. The practical proof: Qwen3.6-27B, a dense 27B model that fits on one consumer GPU, outscores a 397B mixture-of-experts on coding benchmarks. If the model is going to be the part of your stack that churns every few months, the part that must not churn is your context — which is an argument for keeping it in plain markdown behind an open protocol like MDflow does, rather than inside any one vendor's model.
What is the densing law?
The densing law is an empirical trend showing that the capability density of large language models — capability per parameter — doubles roughly every 3.5 months. It was introduced by Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng, Xu Han, Zhiyuan Liu and Maosong Sun, first on arXiv in December 2024 and later published in Nature Machine Intelligence. Evaluated across five widely used benchmarks, the maximum capability density observed doubles on roughly a 3.5-month cadence.
Read it the other way round and it is startling: every three and a half months, a model with half the parameters can match what was state of the art before. The same paper reports inference cost reductions of up to two orders of magnitude following from that.
Osman's term for the same idea is impact per parameter, and he uses it as a lens on hardware rather than benchmarks: what capability did this require last year, what did it need to run on, and what does it need now?
The receipts, in model releases
The trend is easiest to see as a sequence of releases rather than a curve.
| Model | Released | Size | What it took to run |
|---|---|---|---|
| Llama 2 70B | Jul 2023 | 70B dense | A multi-GPU rig |
| Llama 3.1 405B | Jul 2024 | 405B dense | Server-class hardware |
| gpt-oss-120b | Aug 2025 | 120B MoE | One high-end GPU; first widely usable open tool-caller |
| Qwen3.5-397B-A17B | Feb 2026 | 397B MoE, 17B active | Workstation / small server |
| Qwen3.6-27B | Apr 2026 | 27B dense | One consumer GPU |
| GLM-5.2 | Jun 2026 | 744B MoE, ~40B active | Workstation-class (DGX Station or 8× RTX Pro 6000) |
The row that matters is the second-to-last. Qwen3.6-27B, released in April 2026 under Apache 2.0, outscores Alibaba's own 397B mixture-of-experts on coding benchmarks — 77.2 vs 76.2 on SWE-bench Verified, 59.3 vs 52.5 on Terminal-Bench 2.0 — while being roughly fifteen times smaller in total parameters. At Q4_K_M quantization it lands near 17 GB, which is one consumer GPU.
That is the densing law made concrete, and it is worth being precise about the mechanism. As Osman puts it: it is not that small models are beating big models — it is that newer, more efficient models are beating older, less efficient ones. Architecture work compounds. GLM-5.2, the MIT-licensed 744B MoE Z.ai released on 13 June 2026, shares one attention indexer across every four transformer layers to cut per-token compute at its full 1M-token context. NVIDIA's Nemotron 3 family runs the majority of pretraining multiply-accumulates in NVFP4, a 4-bit floating-point format, on a 25T-token corpus — pushing the efficiency gain back into training, not just inference.
None of this says the gap disappears. There will probably always be a frontier that open weights trail. The claim is narrower and more useful: the gap is measured in months, and the hardware floor keeps falling.
Why local LLMs matter
For developers
The economics invert. Today's cloud tokens are cheap partly because they are subsidised by an infrastructure land grab. Osman's blunt framing — why fund someone else's data center on a subscription, when the same capability is arriving on hardware you can buy outright? That is a real question for any team whose agent workloads run continuously rather than in bursts. A rig that looks expensive against one month of API spend looks different against thirty-six.
Hardware stops depreciating on the usual curve. His sharpest observation is a second-order one: as models get denser, existing hardware gets more valuable, because it can run things it could not run last year. An RTX 3090 — a 2020 Ampere card — still trades near or above its original price. He is not selling his.
Nothing gets taken away. No deprecated model endpoint, no changed rate limit, no silent behaviour shift on a Tuesday, no refusal you cannot appeal. If you have ever had a production prompt regress because a provider updated a model behind a stable name, you already know why this matters.
Data never leaves. For regulated work, that is not a preference, it is the requirement that decides whether the project happens at all.
For AI agents
Agents change the calculus more than chat does, in two specific ways.
Agents burn tokens on a completely different scale. A coding agent doing a real task issues dozens of tool calls, re-reads files, retries, and carries a large context every turn. That is exactly the workload where a fixed hardware cost beats a per-token cost — and exactly the workload where per-request latency and rate limits hurt most.
Parallelism becomes affordable. Osman's illustration: the eight RTX 3090s it once took to load Llama 2 70B can now run something like fifteen parallel Qwen3.6-27B agents. Fan-out patterns that are prohibitive at frontier API prices become routine when the marginal cost of an extra agent is electricity.
And crucially, tool calling is no longer the blocker. gpt-oss-120b was, in Osman's telling, the first open model that reliably did it. Everything since has been better. Which matters because tool calling is the entire substrate agents run on — including the Model Context Protocol.
Which applications benefit most
- Coding agents. The highest-volume, most repetitive agent workload there is, and the one where 27B-class open models are now genuinely competitive. Also the workload most sensitive to latency, which local inference wins outright.
- Regulated and air-gapped environments. Healthcare, defence, legal, finance. Not "we would prefer not to send this to a vendor" — "we are not permitted to."
- High-volume classification and extraction. Triage, tagging, routing, enrichment. Millions of small calls where per-token pricing dominates the budget and a specialised small model is often better than a general large one.
- Personal and small-team knowledge work. Assistants over your own notes, email and documents, where the entire value proposition depends on the data being sensitive enough that you would rather it stayed put.
- Edge and offline deployments. Field devices, ships, factories, anywhere the network is the least reliable component in the system.
- Fine-tuned domain specialists. NVFP4-class training efficiency pushes the cost of a specialised model down far enough that a small team can justify one.
How MDflow fits
MDflow does not run models, host inference, or care which model you use. That is precisely the point of including it in this conversation.
Here is the strategic read on the densing law: if capability density doubles every 3.5 months, the model layer of your stack has a half-life measured in months. You will swap models — from a frontier API to a local 27B, from that to whatever ships in October, from that to a fine-tune. Anything you couple tightly to a specific model is a liability. The thing that has to survive every swap is the context layer: the notes, specs, decisions, policies and project knowledge your agents read before they do anything useful.
So the design question is not "which model?" It is: what does a context layer look like if you assume the model underneath it will be replaced four times a year? Three properties fall out of that, and they are the ones MDflow is built on.
What already lines up today
Plain markdown, not a proprietary store. Documents are .md files with nothing between the author and the reader. A local model reading your context does not need an SDK, a connector or a parser — markdown is already the native input format for every model on the list above. Every shared document also has a raw .md twin with YAML frontmatter, so a local harness can curl a URL instead of implementing a client.
A local stdio MCP server, not just a hosted one. MDflow publishes a runnable local MCP server (server.mjs) alongside the hosted remote endpoint. That matters here specifically: if you are running inference on your own machine to keep it local, routing your context through a process on that same machine is the consistent choice. It is the same tool surface either way.
MCP works with local models because MCP is a harness protocol, not a model feature. Claude Code, Cline, Continue, OpenCode and their relatives hold the MCP client and hand tool definitions to whatever model you point them at. Swap GPT-5.5 for Qwen3.6-27B behind the same harness and your MDflow workspace is still there, unchanged, with the same mdflow_get_context call.
Folder descriptions do retrieval work that small models need more, not less. Each folder carries a human-written description of what belongs in it, and mdflow_get_context ranks those descriptions above folder names and document titles before returning matching bodies. A 27B model has less headroom to recover from a bad retrieval than a frontier model does — which makes a deliberately-written retrieval signal more valuable on local hardware, not less.
Client-side encryption with a local decryptor. Documents can be encrypted in the browser so the server stores ciphertext, and mdflow-crypt is a local command-line tool that decrypts them on your own machine. If your reason for running models locally is that the content is sensitive, the store holding that content should be able to make the same promise.
Every surface, one workspace. The REST API with a Personal Access Token or OAuth, the VS Code extension, the n8n node, the web clipper, the iOS app, plus llms.txt, an agent card and an OpenAPI spec for discovery. A hybrid setup — local model for the bulk work, frontier API for the hard cases — reads and writes the same documents from both.
What MDflow is not. It is not a local-first desktop app, and it does not run on your hardware end to end; the workspace is hosted. If your requirement is a fully air-gapped system with no network dependency, MDflow is not that, and no amount of framing changes it. What it is: a model-neutral, format-neutral context layer that a local model can read exactly as well as a hosted one — and whose contents are plain markdown you can export and walk away with at any time.
Where we are headed
Direction, not a dated commitment. The interesting work for a context layer in a world of cheap, swappable local models is making retrieval legible and portable: clearer signals about which documents an agent actually read before acting, freshness cues so stale context announces itself, and scoped tokens so a local agent reaches only the folders its task needs. The through-line is the same — the context layer should outlive every model that reads it.
The bottom line
The story the industry tells about itself is a scaling story, and it is true. The quieter story is that capability keeps getting denser, on its own exponential, and that one is what determines whether the model doing your work lives in a data center or under your desk. A dense 27B beating a 397B MoE on coding benchmarks, eighteen months after 405B models needed server racks, is not an anomaly — it is the trend line behaving normally.
Which means the model is the fastest-moving, most replaceable component in your stack. Plan accordingly. Buy the hardware if the maths works, keep your options open on weights, and put your actual asset — the accumulated context that makes an agent useful in your specific situation — somewhere that does not care which model shows up next quarter.
Start free · Connect an AI agent · Read the API docs
Frequently asked questions
Can a local LLM match a frontier model in 2026?
Not exactly, but the gap has narrowed enormously. GLM-5.2, an MIT-licensed 744B-parameter mixture-of-experts model with roughly 40B active per token and a 1M-token context, is open-weight and competitive with closed frontier models on several benchmarks — it still needs serious hardware. The more useful data point for most people is Qwen3.6-27B, a dense 27B model released in April 2026 that outscores Alibaba's own 397B mixture-of-experts on coding benchmarks and fits in about 17 GB at 4-bit quantization, which is one consumer GPU.
What is the densing law of LLMs?
The densing law is an empirical trend published by Chaojun Xiao and colleagues, first on arXiv in December 2024 and later in Nature Machine Intelligence. It measures capability density — capability per parameter — and finds that the maximum capability density of large language models doubles roughly every 3.5 months. In practice the parameter count required to hit a given level of performance halves about every quarter, which is why last year's data-center model becomes this year's desktop model.
What hardware do you need to run a good local LLM?
For a strong general-purpose coding and reasoning model in 2026, one modern consumer GPU with 24–32 GB of VRAM runs a 27B-class dense model at 4-bit quantization comfortably. Frontier-scale open-weight models like GLM-5.2 still need a workstation-class machine such as an NVIDIA DGX Station with 784 GB of coherent memory, at roughly $85,000–$125,000. The trend line is that the requirement keeps falling.
Do local models work with MCP?
Yes — because MCP support lives in the harness, not in the model. Claude Code, Cline, Continue, OpenCode and similar tools speak the Model Context Protocol and pass tool definitions to whatever model you point them at. The model only needs reliable tool calling, which open-weight models have had since gpt-oss-120b. The same MCP server serves a local model and a frontier model without changes.
If I run models locally, where should my notes and context live?
Somewhere model-neutral and format-neutral, because the model is the part you will keep swapping. Plain markdown files reachable over an open protocol survive every model change. MDflow stores documents as plain markdown, exposes them over MCP and a REST API, publishes a local stdio MCP server you can run on your own machine, and supports client-side encryption with a local command-line decryptor so the plaintext never has to leave your hardware.
Further reading
- The Desktop Frontier — Ahmad Osman, Osmantic (AI Engineer, July 2026) — the talk this post is built on, including the RTX 5090 prediction.
- Densing law of LLMs — Xiao et al., Nature Machine Intelligence. The capability-density result. arXiv preprint.
- GLM-5.2 is the new leading open-weights model — Artificial Analysis' independent evaluation.
- Qwen3.6-27B on Hugging Face — weights, benchmarks and licence for the dense 27B model.
- Inside NVIDIA Nemotron 3 — NVFP4 native 4-bit pretraining, and why training efficiency matters as much as inference efficiency.
- MDflow: model-agnostic AI architecture, portable AI memory, context engineering for AI agents, client-side encryption, and connecting an agent over MCP.