$cat separate-the-task-from-the-model.md

Separate the Task From the Model: Specs, Code, Evals

August 16, 202616 min readby MDflowview as .md

A fixed emerald wireframe contract frame with input and output ports held still above a conveyor of interchangeable dark model cubes, one lifted mid-swap, on a dark terminal-grid background

Every other week there is a new model, and with it a new technique that is going to change everything. Recursive language models. A new optimiser. A better agent loop. If you are the kind of engineer who wants to try all of them, you have probably noticed the tax: each experiment touches the prompts, the parsing, the retries, and half the code around them.

Maxime Rivest and Isaac Miller made the argument against paying that tax at AI Engineer in July 2026, in a talk about DSPy with a deliberately awkward title — The Unreasonable Effectiveness of Separating the Task from the Model. The claim underneath it is small and load-bearing: in ordinary programming, when you repeat something often enough, you make it a function. AI programs deserve the same treatment, and almost none of them get it.

TL;DR — Separate the task from the model and everything inside the boundary — prompt wording, model choice, harness, agent loop — becomes a disposable implementation detail you can search over. What stays is the contract: specs (what should happen), code (what must happen), and evals (what good looks like). Two of those three are prose and examples rather than source code, and they are read by models at runtime, so they need a durable, agent-readable home. MDflow is a versioned markdown workspace built for exactly that half — reachable over MCP and an HTTP API, with folder descriptions as retrieval context and an audit trail on every change.

What "separate the task from the model" means

It means defining an AI task the way you define a function — a name, typed inputs, typed outputs, a contract — and treating everything inside that boundary as implementation you are free to replace. Functions are reusable, composable, testable and optimisable. You can package one and hand it to someone else who only needs to know the contract to use it as a black box. None of those properties are available to a pipeline whose interface is a prompt string.

DSPy — from the Stanford NLP group, now Declarative Self-improving Python — is the most developed version of this idea. A signature declares the input and output interface. A module is an implementation strategy over that signature. An optimiser searches over implementations against a metric. Rivest's own first program was mundane: extracting tax values from farm invoices so he could do his taxes.

import dspy

dspy.configure(lm=dspy.LM("openai/gpt-5-mini"))

class ExtractTaxes(dspy.Signature):
    """Extract all taxes from the invoice. If the invoice is illegible, output 0."""
    invoice: str = dspy.InputField()
    total_tax: float = dspy.OutputField()

class InvoiceTaxes(dspy.Module):
    def __init__(self):
        super().__init__()
        self.extract = dspy.Predict(ExtractTaxes)        # vanilla
        self.recheck = dspy.ChainOfThought(ExtractTaxes) # same contract, more reasoning

    def forward(self, invoice: str):
        out = self.extract(invoice=invoice)
        if not out.total_tax:
            out = self.recheck(invoice=invoice)          # requirement, not a hope
        if out.total_tax < 0:
            raise ValueError("negative tax — escalate to a human")
        return out

The line to notice is the one setting the model, at the top, entirely outside the signature. The task description does not know or care which model runs it.

That boundary does not restrict you to small inputs and outputs. The same shape holds for here is my whole inbox and a new email, draft a reply, or here is a spec and a repository, produce a pull request. Those are repeated tasks too. Fix the contract and the inside can be a single prompt today, an agent with tools next month, and a recursive scaffold after that, without anything outside changing.

The three things that specify a task

A signature alone is not enough to specify a task — it is a type, not a goal. Before ChatGPT existed, DSPy's creator Omar Khattab had already landed on the set of three that is:

Artifact	The question it answers	Its natural form
Specs	What should happen	Natural language — instructions plus the input/output signature
Code	What must happen	Deterministic constraints, assertions, control flow
Evals	What good looks like	Examples and metrics

The third one is the one people skip, and the talk's defence of it is the best part. Rivest asked his father, on the farm, how he knew a tree was a maple. His father could not tell him. He could not give instructions and he certainly could not give code. It took years of examples.

That is not a story about botany. It is a description of the long tail of every real specification — the latent, hard-to-articulate part of good that is the reason internships exist. Instructions are enormously efficient where they work. Give a friend the rules and they can play the board game tonight; make them learn from examples alone and it is a long night. But instructions run out, and where they run out you need examples.

Get all three down and something changes qualitatively: the goal is now specified well enough that a machine can search for the implementation. That is what optimisers do. DSPy started by having code find few-shot examples that made weak base models behave. Then models got good enough to rewrite instructions, and the optimisers moved up a level — GEPA (Agrawal et al., an ICLR 2026 oral) evolves prompts by reflecting on textual feedback rather than following policy gradients, beating GRPO by up to 20% while using up to 35× fewer rollouts, and lifting a plain chain-of-thought program on MATH from 67% to 93%. The trajectory is consistent: each year, more of the implementation is delegated away.

Why separating the task from the model is useful

For developers

You get to try techniques without paying integration cost for each one. When recursive language models landed — Alex L. Zhang, Tim Kraska and Omar Khattab at MIT CSAIL, a paradigm that keeps a huge context in a REPL and lets the model recursively sub-query it — the DSPy answer was to make it a module. It might help your long-context task. It might not. The point is that finding out is one line and your signature is unchanged.

You get a credible exit from any model. DSPy's case-study list credits Shopify with cutting yearly costs by roughly 550× on structured metadata extraction across all Shopify shops, by moving from an expensive model to a cheap one while keeping the same evals and the same business logic. The saving comes from the swap; the fixed boundary is what made the swap a configuration change rather than a rewrite. This is the concrete mechanism behind model-agnostic architecture — optionality is only real if exercising it is cheap.

You get a failure surface that is inspectable. A constraint expressed in code either held or it did not, and the traceback names it. A constraint expressed in English inside a system prompt competes for attention with everything else in the context and degrades probabilistically. Both are "rules"; only one is testable.

For AI agents

An agent given a contract has a smaller, better-posed problem. Return an object with these fields, satisfying these constraints, scored by this metric is a target. Do your best with this task is not. The same asymmetry shows up in long-horizon agent evals, where the runs that fail are usually the ones with no stated definition of done.

The contract is what makes an agent's output reusable by another agent. If a task is a black box with a declared interface, an orchestrating agent can compose it without reading its internals — the same property that makes functions composable in ordinary code, and the reason a well-specified task survives being handed between models.

The evals are a hill the system can climb on its own. Once good is measurable, the loop from production signal back to improvement can be mostly automatic — which is the same architecture as an agent observability loop that turns signal into a pull request, with the metric standing in for the human reviewer's judgement on the easy cases.

Which applications benefit most

High-volume extraction and classification — invoices, metadata, support tickets, document parsing. Fixed contracts, brutal cost sensitivity, and a measurable definition of correct. This is where 550× lives.
LLM-as-judge and evaluation pipelines — a judge is a task with an unusually crisp signature, and it is the component you most want to be able to swap models under without re-tuning everything downstream.
Agentic coding — spec plus repository in, pull request out is a repeated task, which is precisely the argument for spec-driven development.
RAG and retrieval pipelines — the retrieval strategy is the implementation detail par excellence, and it changes more often than the question your users are asking.
Multi-step workflows in regulated domains — where what must happen genuinely has to be code, not a hopeful sentence in a prompt.
Anything you run at scale on a frontier model — because the whole value of the discipline is being able to answer "could a cheaper model do this?" with an experiment instead of an argument.

The part that never gets automated

Miller closed on the question everyone asks: what happens when we have AGI? His answer is the most quotable thing in the talk. If you asked Albert Einstein to help you with your emails, he would probably ask what an email is. Intelligence is not the same as being all-knowing.

A perfect model still will not know your task, your customers, the four incompatible things your company means by "revenue", or which of your colleagues needs the caveat spelled out. That gap — last-mile learning — does not close as models improve, because it was never a capability problem. It is a context problem, and the context is yours.

Which is why the durable artifact is the specification, not the prompt. DSPy 4 previews two directions that lean further into this: dspy.Flex, which lets the optimiser learn the code of a harness rather than only its instructions, and qualitative learning, which tries to build evals out of the textual feedback already present in production — traces, user actions, product analytics — instead of a dataset someone hand-labelled as a proxy for reality. Both make more of the implementation disposable. Neither writes your specification for you.

How MDflow fits

Of the three artifacts, exactly one has an obvious home. Constraint code lives in the repository beside the pipeline it guards. Specs and evaluation criteria usually should not, and teams discover this the hard way: they are read by models at runtime, referenced from several repositories, revised by people who do not open pull requests, and they change on a different cadence than the code around them. So they end up in a Notion page, a Slack thread, and three prompt strings that have quietly diverged.

MDflow is a versioned markdown workspace for that half. It is not an optimiser and does not run your evals — DSPy, your CI and your eval harness do that.

What already lines up today

Markdown is the native unit, and the model reads the same bytes you do. A task specification, a scoring rubric, a set of worked examples of good — these are documents. Every one is also served as a raw .md endpoint with YAML frontmatter, so an eval job in CI can fetch the current rubric over plain HTTP with no client library, which matters when egress is allowlisted.

Folder descriptions are declared intent, and they outrank filenames. Each folder carries a description of what belongs in it, and mdflow_get_context ranks those descriptions above folder names and document titles before returning matching bodies. A folder described as "Task contracts — one document per task, with the signature, the constraints and the current definition of a passing result" is a retrieval signal you wrote on purpose, which is why folder descriptions beat file names.

Version history is what makes a rubric safe to change. Every saved change on every write path captures the previous version, with line-by-line diffs and non-destructive restore. When a benchmark number moves you can see whether the system changed or the definition of good did. It is deliberately Pro, private to the document owner, and not exposed over the API or MCP — an agent cannot quietly rewrite the record of its own edits.

The Document Log answers "which run proposed this?" A cross-document activity feed records created, edited, shared and deleted events with the actor on every row, shown as automated · <token name> for anything arriving through the API or MCP. Give each optimisation pass its own Personal Access Token and the provenance of a proposed spec change is a row, not an archaeology project.

Qualitative learning needs a review queue, and checkboxes are one. /tasks aggregates ordinary - [ ] lines from every markdown body, so "decide whether hedged answers should score as failures" — written by an automated pass reading production feedback — becomes something a person can prioritise and tick off, with the document remaining the source of truth.

Every runtime reads the same workspace. The same documents are reachable from Claude, ChatGPT, Cursor and Codex over the remote MCP server with OAuth or a Personal Access Token, and from CI, cron, an orchestration DAG or n8n over the HTTP API — plus a local stdio server, a VS Code extension and an iOS app. The nightly optimiser run and the domain expert who decides what good means are looking at the same file.

Where we are headed

Direction, not a dated commitment. Two things about this use case interest us. The first is richer structured retrieval over folder descriptions, so an agent asking "what is the current acceptance criterion for invoice extraction?" reliably lands on the one canonical document rather than the three most lexically similar ones. The second is narrower agent credentials: a Personal Access Token today reads and writes everything its owner owns, which is enough for attribution but is not least privilege — and an automated pass proposing rubric changes on a schedule is exactly the case that wants a token scoped to one folder.

The bottom line

The industry's default posture is to own the prompt and rent the model. Separating the task from the model inverts it: own the contract, rent the implementation. Specs, code and evals are the part that compounds; the prompt string, the model name and this month's scaffold are the part you should be able to throw away without ceremony.

The catch is that two of those three artifacts are not source code, and putting them in a chat log or an untracked page quietly undoes the whole discipline — you cannot swap a model with confidence if nobody can find the current definition of correct. Write the contract down where every agent and every engineer reads it. Then let the implementation churn, which it will.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What does it mean to separate the task from the model?

It means defining an AI task the way you define a function — a name, typed inputs, typed outputs, and a contract about what it must do — and treating everything inside that boundary as a swappable implementation detail. The model, the prompt wording, the harness, whether it is a single call or an agent loop, whether it uses tools or retrieval: all of that is implementation. When the boundary is fixed, you can change the internals freely and the rest of your system does not notice.

What are specs, code and evals in an AI program?

They are the three artifacts that together fully specify a task. Specs are what should happen, written in natural language — instructions and the input/output signature. Code is what must happen, expressed as deterministic constraints: retry with more reasoning if the first extraction is empty, refuse a negative value, escalate to a human. Evals are what good looks like, expressed as examples and metrics, because some judgements cannot be written as either instructions or rules. With all three in place a goal is specified well enough to be optimised automatically.

Is DSPy just prompt engineering with extra steps?

No, it is the opposite bet. Prompt engineering treats the prompt string as the artifact you own and maintain. DSPy treats the prompt as a compiler output — something an optimiser generates from your signature, your constraints and your metric. You maintain the contract and the evaluation; the wording is regenerated when the model changes. That is why swapping models becomes a re-run rather than a rewrite.

Does separating the task from the model actually save money?

It is what makes the saving reachable. DSPy's own case-study list credits Shopify with reducing yearly costs by roughly 550 times on structured metadata extraction across all Shopify shops, by moving from an expensive model to a cheap one while keeping the same evals and business logic. The saving comes from the model swap; the fixed task boundary is what made the swap a configuration change instead of a rewrite of every prompt in the pipeline.

Where should AI task specifications and evals live?

Constraint code belongs in the repository, beside the pipeline it guards. Specs and evaluation criteria usually do not, because they are read by models at runtime, referenced from several repositories, edited by people who do not open pull requests, and revised on a different cadence than code. They need a versioned, agent-readable home with an audit trail — a markdown workspace an agent can fetch over MCP or plain HTTP is the practical form, so the same definition of what good looks like is in front of every model call.