Skill Novelty: Stop Writing What the Model Already Knows

You wrote a skill file. You ran it against a handful of tasks, the agent did better with it than without, and that settled the question. So you kept adding to it. Nobody ever asked the more awkward follow-up: which lines did the work? Skill novelty is the name for that question, and for most skills the honest answer is a small minority of them.
It is the idea at the centre of an AI Engineer talk by Nikolay Petrov of Intel, published on 11 November 2026. The title is the finding: a large share of a typical skill file is material the model could already produce on its own.
TL;DR — A skill is supposed to teach a model something it does not know, but almost nobody checks whether it does. You can check it: split the skill into small items, ask the model about each one with and without the skill loaded, and have a judge from a different model family grade both runs. The result sorts every item into four buckets — already known, newly taught, actively harmful, or missed entirely. The sharpest finding is the gap between grading leniently and strictly: the Opus model tested answered about 70% of items at the level of the general idea, and about 22% on exact details. Concepts are in the weights. Specifics are not. Keep the specifics — your parameters, your thresholds, your conventions — somewhere durable and agent-readable; MDflow is a reasonable home for that half, and is explicitly not the measuring instrument.
What is skill novelty?
Skill novelty is the share of a skill file that a model could not already produce without it. It is a measurement, not a style rule, and the method is more interesting than the number.
Start by splitting the skill into small, individually checkable items. The talk borrows this decomposition from FActScore (Sewon Min et al., EMNLP 2023), which breaks long-form generated text into atomic facts and scores each one against a reliable knowledge source. Here the same move is applied to instructions instead of generations: each item becomes a question paired with the answer the skill says you should get. Some items are general propositions; others are hard contracts about exactly what the agent should do. A skill decomposes into roughly 25 to 30 of them.
Then run two arms for every item. One with the skill loaded into context, one without. Ask the model the question in both, and have a judge — ideally from a different model family, or a panel — grade the answers. Two design details matter: the questions must not leak their own answers, and you decide deliberately whether the model gets tools and web access, depending on whether you want to probe what it knows internally or how it behaves in practice.
What comes back is a two-by-two of four cases:
| Without the skill | With the skill | What it means |
|---|---|---|
| Correct | Correct | Already known. The skill is paying rent in context for nothing. |
| Correct | Wrong | Harmful. The skill made the model worse at something it could do. |
| Wrong | Correct | Novel. This is the knowledge you are actually buying. |
| Wrong | Wrong | Missed. The skill meant to teach it and did not land. |
The harmful quadrant is the cheapest thing in the whole exercise to act on, because removing those items is a strict improvement with no trade-off. It is also rare in well-worn skills: across nearly 400 items drawn from Anthropic's published skills, the talk reports roughly one such case. Smaller, less-used skills showed more of them.
And the cost of running this is low enough not to be a planning problem. The talk puts 29 skills at around $14 in Sonnet and Haiku calls, with Sonnet acting as judge.
Why skill novelty matters for developers and AI agents
Because the usual eval tells you that a skill helps, and novelty tells you which parts of it help. Those are different questions, and only the second one lets you make the skill smaller without guessing.
The talk's worked illustration of the first question is the ordinary two-arm task eval: without the skill the agent solves three of five tasks, with it five of five. That is a real signal and worth having. But it is a verdict on the whole file. When you then want the skill to be cheaper, or you want to know whether it is still earning its place against a newer model, a single aggregate score gives you nothing to act on. You are back to changing lines and hoping.
For developers
The finding that should change how you write is the gap between two grading strictnesses. Graded leniently — does the model have the general idea? — the Opus model tested answered about 70% of skill items. Graded strictly — does it have the exact parameters, values and steps? — about 22%. The transcript we worked from gives no specific Opus version and no per-skill breakdown, so treat these as the shape of the result rather than a benchmark figure.
The shape is what matters, and it is consistent with how these models behave everywhere else. Frontier models generalise concepts well and specifics badly. The paragraph in your skill explaining what exponential backoff is, or why idempotency keys exist, is almost certainly redundant — the model has read more about those than you have. The line naming your retry ceiling, your idempotency window, your escalation path when the third attempt fails, is not redundant and never will be, because it was never in the training data.
That is a writing instruction disguised as a measurement. Cut the explanation. Keep the numbers.
There is a second, quieter consequence: novelty is relative to the model. Smaller models in the talk showed lower understanding of the same skills than larger ones, which means a skill that is 70% redundant for a frontier model may be genuinely load-bearing for the small one you route cheap work to. If you run a mixed fleet — and most teams now do, as we argued in latency budgets and model selection — "this skill is mostly redundant" is a claim that needs a model attached to it.
For AI agents
For the agent, every redundant line is attention spent on a restatement of something it already had. This is the same argument we made in context engineering: context is a budget, not a warehouse, and curation beats volume. Novelty measurement is what turns that principle into an edit list.
It also fits the way skills are meant to load. Anthropic's agent skills design is built on progressive disclosure — a short description at startup, the full body only when a task matches, bundled files later still. Progressive disclosure controls when you pay for a skill. Novelty controls how much there is to pay for once it loads. The two compose, and neither substitutes for the other.
One honest caveat the talk makes and we will repeat: not every skill is trying to teach anything. Steering or process skills exist to produce a repeatable flow of actions, not to add knowledge, and they will score low on novelty by design. Measuring them against this yardstick and then cutting them would be a mistake. Novelty applies to knowledge skills.
Which applications benefit most
- Shared skill libraries. The moment more than one team loads the same skill, its token cost multiplies by every invocation while its author sees none of it. Libraries are where a redundant paragraph gets expensive.
- Long-lived repository documents. The instruction files that accrete in a repo over a year are the textbook case — each addition was justified at the time, and nobody has ever subtracted. We looked at the family of these files in AGENTS.md, CLAUDE.md, SKILL.md and llms.txt explained.
- Mixed-model and model-switching shops. If the same skill feeds a frontier model and a small local one, you need the per-model picture before trimming anything.
- Internal platform teams publishing skills to the rest of the company. Novelty gives them a review criterion sharper than "looks helpful", which is what skill review usually degenerates into.
- Products shipping agent-facing documentation. If an agent meets your product through a skill, the useful half of that skill is your specifics — endpoints, limits, error semantics — not your explanation of what an API is.
How MDflow fits
Measurement is not the hard part of this in the long run. The hard part is that the surviving material — the specifics, the ones no model will ever have in its weights — has to live somewhere durable, readable by agents, and editable by the people who know the numbers.
What lines up today
MDflow stores everything as plain markdown, which is the format skill files are already written in. There is no export step between a document in a workspace and something an agent can read.
Folder descriptions are the part that matters most here. A folder's description states what belongs in it, and MDflow's retrieval ranks those descriptions as the primary signal — above folder names and document titles — when an agent asks for context on a topic. That is a natural home for the kind of scoping a skill needs: this folder holds the retry and rate-limit conventions for our payment integrations. Topic retrieval itself is an MCP tool — mdflow_get_context — while the HTTP API gives an agent the explicit route instead: list folders, list documents, fetch a body, all with a Personal Access Token.
Version history (Pro) is the quietly useful one for this workflow. If novelty measurement tells you to cut 40% of a document, you want to be able to see what it said before — and to put a line back when the next model turns out not to know it after all. Worth stating plainly: version history is private to the document owner and is deliberately not exposed through the API or MCP server, so an agent cannot read or rewrite a document's past.
Every shared document also has a raw markdown twin that begins with YAML frontmatter — title, canonical URL, markdown URL, visibility — so a tool can cite and read it in a single request. Combined with the discovery files (llms.txt, the agent card, the OpenAPI spec), an agent can find the material and read it without a scraping step.
What MDflow does not do, and we are not going to imply otherwise: it does not measure skill novelty. There is no judge, no two-arm harness, no scoring. It is where the text lives, not the instrument that tells you which text to keep. The measuring belongs in your eval framework; the surviving prose belongs somewhere a human can edit it and an agent can read it.
Where we are headed
Direction rather than a dated commitment, and deliberately vague about timing because none of it is scheduled: the folder description is the primitive we keep coming back to, and we think there is more to do in making what a folder is for carry further into what an agent retrieves. The other thread is the round trip — MDflow already edits markdown in place in GitHub, Google Drive and OneDrive, and skills tend to live in repositories, so the gap between where a skill is version-controlled and where its prose is edited is a gap worth narrowing. Treat both as interests, not commitments.
The bottom line
A skill that improves your eval score is not thereby a good skill — it is a file with some useful lines in it and an unknown quantity of filler. Novelty measurement is a cheap way to tell the two apart: decompose into items, run both arms, judge with a different family, read the quadrants. Then cut the already-known, delete the harmful, and validate with the evals you already had.
The durable lesson survives whatever the next model knows. Write down what the model cannot know. Your thresholds, your conventions, your reasons, your exceptions. The explanation of the general concept has a shelf life measured in model releases; the specifics do not.
Start free · Connect an AI agent · Read the API docs
Frequently asked questions
What is skill novelty?
Skill novelty is the share of a skill file's content that a model could not already produce without it. You measure it by splitting the skill into small checkable items, asking the model about each one twice — once with the skill in context and once without — and comparing. Items the model answers correctly either way are redundant; items it only answers with the skill loaded are the novel part you are actually paying context for.
How much of a typical SKILL.md does the model already know?
It depends entirely on how strictly you grade. In Nikolay Petrov's AI Engineer talk, the Opus model tested answered roughly 70% of skill items correctly when graded leniently on the general idea, but only about 22% when graded strictly on exact parameters, values and steps. So most of a skill is redundant as concept explanation and still load-bearing as specifics.
Can a skill make an agent worse?
Yes, though it appears to be rare in mature skills. The two-arm comparison has a quadrant for items the model answered correctly without the skill and got wrong with it loaded. Across nearly 400 items drawn from Anthropic's published skills the talk found roughly one such case, but smaller and less-used skills showed more. Those items are the cheapest thing to fix because removing them strictly improves the skill.
Does measuring novelty replace running evals?
No. Novelty is a signal about what the model already knows, not proof that cutting a line is safe. The talk is explicit that it gives you a starting direction for optimisation, and that every change still has to be validated by running the skill's actual task evals. Treat novelty as the thing that tells you where to look and evals as the thing that tells you whether you were right.
Does MDflow measure skill novelty?
No. MDflow is a markdown workspace, not an evaluation framework: it has no judge, no two-arm harness and no scoring. What it does is hold the text — the skills, runbooks and folder descriptions that survive the cut — as plain markdown that agents read over MCP or the HTTP API, with version history on Pro so you can see what a document said before you trimmed it.
Further reading
- Your SKILL.md Is 70% Stuff the Model Already Knows — Nikolay Petrov (Intel), AI Engineer, 11 November 2026. The source for this post. The talk also demonstrates a framework for evaluating agent context; the transcript we worked from does not render its name clearly enough for us to cite it here.
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation — Sewon Min et al., EMNLP 2023. The atomic-decomposition method the novelty measurement borrows.
- Equipping agents for the real world with Agent Skills — Anthropic, on skills and progressive disclosure.
- Effective context engineering for AI agents — Anthropic, on treating context as a finite budget.
- AGENTS.md, CLAUDE.md, SKILL.md, llms.txt Explained — what each agent-facing file is for.
- Context Engineering for AI Agents: Curation Beats a Bigger Prompt — the principle this post makes measurable.
- Skills Are the New SDKs — why teachability is becoming a product property.
- MDflow MCP setup and the HTTP API reference — how agents read a workspace.