Agent Environments and Rollouts: Evals Meet RL

In 2018, a colleague sent you a four-hundred-line pull request and you made a coffee and read it. The names were good. The refactors were tidy. You left a comment about extensibility. That was the dance, and eight years later it reads like a period piece.

Alex Shaw of the Laude Institute opened his AI Engineer talk in July 2026 with that nostalgia and then named what actually ended. Software engineering, he suggested, was the era when you knew what the code would do before you ran it. Swap one regex for one model call and that guarantee is gone — the program is probably better, and you can no longer say with confidence what it prints on the millionth run.

TL;DR — Agent development is closer to machine learning than to software engineering, so it needs machine learning's tooling. The unit of work is the rollout: an environment (an instruction, a sandbox, a verifier) is handed to an agent, which produces a trajectory, which a verifier turns into a reward. That one primitive powers evals, RL, prompt optimisation and batch production work alike. The executable half of an environment belongs in git; the written half — what the task means, why the reward is scored that way, what you learned — belongs somewhere versioned and retrievable like MDflow.

What is an agent environment?

An agent environment is three things bundled together: an instruction, a sandbox, and a verifier. That is the whole definition, and its smallness is the interesting part.

You need some way to tell the agent what it is supposed to do — the instruction. You need somewhere for it to do the work, and since agents act on computers, that somewhere is a virtual computer — the sandbox. And you need some way to decide whether it actually did the thing, within some time limit or other stopping condition — the verifier, which may be a test script, a rubric, or another agent that comes in and looks around.

In Harbor, the framework Shaw's team built out of Terminal-Bench, an environment is a directory:

my-task/
  instruction.md        # what the agent is asked to do
  task.toml             # configuration and metadata
  environment/          # Dockerfile, compose file, or a prebuilt image
  tests/test.sh         # the verifier
  solution/solve.sh     # a reference solution

Two details matter more than they look. First, the instruction is a markdown file — Harbor pulled it out of the YAML config it used to live inside, precisely because instructions are prose that humans read and argue over. Second, the verifier writes its score to a file inside the container rather than printing something a framework has to parse, which means non-binary rewards work and the task stops depending on the harness that runs it.

The result is that an environment is portable. Because this layout has become something close to a convention, environments pass between teams and stay interoperable — which is what turns a benchmark from a leaderboard into an asset.

What is a rollout?

A rollout is one run of one agent against one environment. The loop is short enough to write out:

1. start the sandbox
2. hand the sandbox to the agent
3. the agent works until a stopping condition  → trajectory
4. hand the sandbox to the verifier
5. the verifier writes a reward                → reward
6. stop the sandbox

Aggregate the rewards across a dataset of environments and you have an eval result. That is it. Real frameworks add flavours on top — multi-step rollouts, verification in a separate sandbox, artifact collection, a simulated user pushing back mid-task — but they are all built on those six lines.

The reason this simple shape deserves a name is that it is universal. Once you can perform a rollout, the same primitive serves several jobs that used to look like different disciplines:

• Evaluation — aggregate the rewards.
• Supervised fine-tuning — keep the trajectories.
• Reinforcement learning — keep the rewards.
• Prompt and harness optimisation — feed the trajectory back as natural-language feedback, the way GEPA evolves prompts by reflecting on what went wrong rather than computing a gradient.
• Production batch work — what Shaw called agentic map-reduce: run a thousand agents across a thousand sandboxes, then reduce the results. Process a quarter of expense receipts. Read every pull request from last month and answer one question about them. Sweep your own coding sessions for the mistakes you kept correcting.

That last one was an emergent use. Nobody designed the eval framework to be a distributed job runner; people just started using it that way, because "run any agent, with any model, in any sandbox, on any task, ten thousand times in parallel" turns out to describe a lot more than evaluation.

Why agent development is machine learning, not software engineering

Because you no longer know what the program will do before you run it, so you have to characterise it empirically instead of reading it. François Chollet made the point about generated code: treat it as a black-box artifact whose behaviour and generalisation you manage through empirical evaluation, like any other ML model. Shaw's generalisation is that this is not a fact about generated code — it is a fact about agents.

Which means the machine learning toolbox maps over almost term for term:

The left column has a decade of mature products, libraries and platforms behind it. The right column is roughly where the left column was in 2015 — except that the number of people building agents already dwarfs the number who ever trained a model, and the gap keeps widening.

The practical consequence is a discipline shift. You cannot review your way to confidence any more. The 2018 move was to read the diff carefully. The 2026 move is to build the environment, run a few hundred rollouts, and look at the distribution.

Which teams need environments most

Every company that uses computers, which is nearly all of them — but the four shapes Shaw sees in practice are worth separating, because they call for very different environments:

1. Evaluate how well agents build your product. An eval built on your own internal codebase, not a public benchmark. Ramp SWE-Bench is the example he cites: a private benchmark built from real engineering problems inside Ramp, so the team can pick the coding agent and model that perform best on their code instead of maximising token spend on faith.
2. Evaluate how well agents use your product. If your product is moving toward a headless mode — and most are — then how well an agent can drive it is a product metric. Build the eval, then iterate on the product until the number goes up.
3. Evaluate how well agents power product features. The agent is inside the thing you ship, so its failure modes are your failure modes.
4. Evaluate how well agents automate internal processes. The unglamorous back office, where most of the actual hours are.

Satya Nadella's version of the advice, quoted in the talk, is the sharpest framing of why this is worth the effort: start with the eval that matters and your ability to grade the outcome, and then you can welcome all models. Once the eval is yours, the power moves to you. You stop trusting brands, public leaderboards and someone else's benchmark, and you start picking your own point on the cost-performance curve.

How MDflow fits

MDflow does not run sandboxes, score rollouts or train models. It holds the half of an environment that is not executable — and that half is where teams quietly lose the plot.

What already lines up today

The instruction is markdown, and markdown is what MDflow is. instruction.md gets committed into a task directory eventually, but it does not start there. It starts as prose someone drafts, that three people argue over, that gets rewritten twice after the first rollouts come back scoring things nobody meant to measure. MDflow stores plain markdown documents with real version history, so the drafting happens somewhere with a diff instead of in a Slack thread.

Rubrics and reward definitions are prose, not code. A test.sh says whether a file exists. It does not say why partial credit is 0.4, what counts as reward hacking on this task, or which failure modes you deliberately decided not to penalise. That reasoning is the thing new teammates need and the thing that evaporates. Keep it in the same document as the instruction it governs — and see eval rubrics for AI agents for what goes into one.

Folder descriptions make the corpus retrievable. MDflow ranks folder descriptions above folder names and document titles when an agent calls mdflow_get_context, so a folder described as "environment specs and scoring decisions for the internal coding benchmark" is findable by an agent that was only told to go read the scoring rules.

The agent reads the same file you edit. Twenty-five MCP tools and a REST API behind a Personal Access Token mean the harness-writing agent can fetch the instruction, and write back what it learned, without anyone exporting anything. Shared documents also serve a raw .md twin with YAML frontmatter, so an external runner can curl a spec it does not have credentials for.

MDflow is itself an example of category two. The llms.txt index, the agent card, the OpenAPI spec and the raw markdown twins exist so that agents can use the product without a human in the loop — which is the surface you would point an environment at if you wanted to measure how well they actually do. We build for agents on purpose.

Decisions become tasks in the document that explains them. Because /tasks aggregates ordinary - [ ] checkbox lines out of markdown bodies, "decide whether task 14's verifier is measuring the wrong thing" can live inside the section it concerns and still show up on a real list.

Where we are headed

Direction, not a dated commitment. The thing we most want to make trivial is pinning a written definition to a moment in time — so that "which version of the instruction was in force when this run scored 0.72" is one click rather than an archaeology exercise. Richer structured retrieval over folder descriptions is the other active line of work.

The bottom line

The 2018 skill was reading a diff and knowing what it would do. That skill has not become worthless, but it has stopped being sufficient, because the interesting parts of your system are now black boxes you can only understand by running them a great many times and looking at the distribution.

The unit of that understanding is the rollout, and the asset behind it is the environment. Environments are cheap to run and expensive to write — not because the Dockerfile is hard, but because deciding what "done" means for a fuzzy task, and defending that decision six months later, is hard. Put the container and the test script in git. Put the instruction, the rubric and the reasoning somewhere versioned, retrievable and shared, where the person revising the rules and the agent reading them are looking at the same file.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What is an agent environment?

An agent environment is three things bundled together: an instruction telling the agent what to do, a sandbox for it to do the work in, and a verifier that inspects the sandbox afterwards and produces a reward. In frameworks like Harbor an environment is just a directory on disk, which is why environments can be shared between teams and reused across evaluation, training and production.

What is a rollout in agent development?

A rollout is one run of one agent against one environment. Start the sandbox, hand it to the agent, let the agent work until a stopping condition, pass the sandbox to the verifier, collect a reward, stop the sandbox. The agent leaves behind a trajectory — the full record of what it did. Aggregating rewards across many rollouts is what produces an eval result.

Why is agent development more like machine learning than software engineering?

Because you no longer know what the program will do before you run it. Traditional software is deterministic, so reading the code tells you its behaviour. An agent's performance is a black-box artifact you can only characterise empirically, by running it many times and measuring outcomes — the same discipline machine learning has always required.

Do evals and reinforcement learning use different infrastructure?

No, and that is the point. Evaluation aggregates rewards across rollouts. Supervised fine-tuning consumes the trajectories those same rollouts produced. Reinforcement learning consumes the rewards. Prompt optimisers like GEPA consume the trajectories as natural-language feedback. One rollout primitive feeds all of them, which is why environments are the asset worth investing in.

Where should the instruction and rubric for an agent environment live?

The executable parts of an environment — the container definition, the test script — belong in version control next to the code they test. But the prose around them does not: what the task actually means, why the reward is scored the way it is, which model won last quarter and why. MDflow keeps that written half as versioned markdown that a person edits in an editor and an agent retrieves over MCP or the HTTP API, so both are reading the same file.

Further reading

• Everything Is a Rollout — Alex Shaw, Laude Institute, AI Engineer, July 2026
• Harbor and its documentation — the open-source environment format, rollout framework and registry
• Terminal-Bench — the terminal-task benchmark Harbor grew out of
• GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning — Agrawal et al., the trajectory-as-feedback optimiser
• Private agent benchmarks: from traces to simulations — how to source the environments in the first place
• Eval rubrics for AI agents — what goes inside the verifier when the answer is not a test script
• Long-horizon agent evals — what breaks when a rollout runs for weeks instead of minutes
• Spec-driven development: where your markdown specs should live — the same argument, one layer up
• MDflow MCP documentation and HTTP API documentation