$cat process-data-vs-state-data.md

Process Data vs State Data: What AI Learns From

September 5, 202614 min readby MDflowview as .md

A single flat emerald wireframe document sheet on the left, and on the right the same sheet exploded into a branching trail of layered sheets and connected nodes

Ask most organisations for their data and you get a database dump. Rows in a CRM, closed tickets, approved documents, the final version of the deck. That is the output of the work. The work itself — the four options someone considered, the one they rejected and why, the constraint that turned out to matter halfway through — was never written anywhere, or was written in a Slack thread that scrolled away.

That gap is the subject of Sean Cai's AI Engineer talk State of Data. Cai is an independent analyst who covers the data markets that frontier labs buy from, and the reframe he opens with is the useful part for anyone building on top of those models: the valuable data is not the saved file. It is the trajectory.

TL;DR — State data is the final artefact: the row, the ticket, the approved doc. Process data is the trajectory that produced it — the sequence of decisions, the drafts, the rejected options, the rationale. Models learn general competence from state data and real expertise from process data, which is why AI got good at coding first: GitHub is a process-data machine that also happens to store code. Almost no other profession has one. Most of your organisation's process data is being discarded right now, by tools that keep only the last save. A workspace like MDflow keeps the trail — version history, a per-actor change log, anchored comments — in plain markdown that both people and agents can read.

What is process data vs state data?

State data records what the answer was. Process data records how someone got to it.

Cai's framing is that when people say "our data", they nearly always mean state-based data: the rows in an ERP, the final output, the saved file. It is what a system persists once the thinking is over. Process-based data is the other thing — the trajectory, the reasoning trace, the sequence of decisions that gets a professional from a blank page to a finished work output.

	State data	Process data
What it is	The final artefact	The trajectory that produced it
Example	An approved architecture doc	The three architectures rejected, and why
Example	A closed support ticket	What the engineer tried first and ruled out
Example	A signed contract	Which clauses were fought over and conceded
Answers	What is true?	How does a competent person decide?
Where it lives	Databases, document stores	Usually nowhere

The distinction matters because of what each one teaches. A model trained on finished artefacts learns to produce things that look like finished artefacts. A model trained on trajectories learns the judgment — which is the part that transfers when the next problem is not the same as the last one.

This is also why the "just point the agent at our data warehouse" plan disappoints so reliably. The warehouse is pure state. It contains every decision your company ever made and no record of why any of them were made.

Verifier's Law: why coding matured first

Domains become tractable for AI in the order their verification infrastructure was built, not in the order of their economic value.

Cai leans on Verifier's Law, Jason Wei's formulation: the ease of training AI to solve a task is proportional to how verifiable the task is. All tasks that are possible to solve and easy to verify will be solved by AI. Wei lists five properties that make a task easy to verify — objective truth, fast to verify, scalable to verify, low noise, and a continuous rather than binary reward.

Cai decomposes verifiability into three axes that are easier to apply to a whole profession:

Asymmetry of verification — how hard is it to break the task into checkable steps?
Veracity of verification — how much consensus is there about what correct even means?
Proliferation of verification — how often does the real world hand you fresh examples of verified work?

Score coding on those three and you get high, high, high. Unit tests decompose correctness objectively. The community broadly agrees on what working code is. And public repositories supply effectively unlimited examples — with commit messages attached as free reasoning traces. Cai's point is that software did not get solved first because it is easy. It got solved first because Web 2.0 accidentally built it a process-data pipeline and called it version control.

Now look at where the money is trying to go next: finance, healthcare, law, biology, security, taste. These score low on veracity — practitioners genuinely disagree about what correct means — and low on proliferation, because the verified examples exist only inside private enterprise workflows. No public system ever captured them. That is why AI companies are suddenly interested in buying your Slack history and your Jira logs: those are the closest thing to a process-data trail that most professions ever produced, and they were produced by accident.

The ordering has held so far. Code, then search, then finance, then healthcare and law — roughly the order in which the three axes decline.

Type 1 vs type 2 data, and the benchmark problem

Captured work beats manufactured examples, and most of the market ships the second while marketing the first.

Cai splits training data on a quality axis:

Type 1 — pure capture of real workflows. Commits, session replays, the actual trail of an actual decision, with minimal shaping by non-experts.
Type 2 — contrived. Hire domain experts, sit them in an arbitrary setting, have them manufacture examples.

Type 2 is fine for getting a model off the floor; when a model reads at a first-grade level, anyone can teach it. Type 1 is what carries it from roughly competent to genuinely expert, because the realism is inherited from the work itself rather than reconstructed by someone imagining the work. And it has a structural property Cai stresses: a type 1 dataset is only durably available if you partner with a live business, because the frontier keeps moving. A dead startup's code base stops generating new trajectories the day it dies.

The uncomfortable corollary is his description of the dominant benchmark recipe. Hire experts. Have them generate plausible tasks. Cherry-pick the ones where the model diverged. Package the result as a hard North Star benchmark — and then sell the data to climb that same benchmark. He calls it Goodhart's law with a profit motive: once the measure is the target and the target is set by people who are not the true domain experts, it stops measuring anything real.

There is a second, more technical failure he names. Contrived benchmarks test a single isolated in-distribution question, so they cannot tell you whether a model sustains correct reasoning across a long dependent episode. Add cross-harness and cross-infrastructure differences — the same model scored under a different scaffold — and a single benchmark number is one sample from a distribution nobody measured. In his own private finance benchmarks he reports two frontier models landing within three points of each other on the same long-horizon task while failing in opposite directions: one nailing the arithmetic and missing the methodology, the other the reverse. The headline number hid the entire finding.

His advice to researchers is one line worth stealing: stop outsourcing your definition of realism to the same vendors you buy your evals from. Letting the test writer grade the test is not evaluation.

Why this matters if you are not training a model

You probably are not buying RL environments. The argument still lands, because the same asymmetry governs whether an agent can do your work.

For developers. Every agent you deploy is doing a small-scale version of what the labs are doing: learning your domain from whatever context you hand it. If all you can hand it is state — the current file, the current schema, the current runbook — it will produce plausible output and repeat mistakes your team ruled out two years ago. Process data is what stops that. A rejected-alternatives section in a design doc is worth more to an agent than three more pages describing what shipped.

For AI agents. Agents are unusually bad at inferring intent from artefacts, because the artefact is the one thing that survived the argument. Given a config file, an agent cannot tell which values are load-bearing and which are leftovers. Given the change history and the note explaining why the timeout was raised, it can. Process data is the difference between an agent that can edit your system and one that can only extend it.

For the moat question. Cai's closing line is that your moat is not the data — it is the pipeline into real-world work, plus the infrastructure to keep learning from it as the models improve underneath you. A static dump depreciates. A running capture appreciates. That is the same conclusion we reached from a different direction in Your moat is your data model.

Which applications benefit most

The teams with the most to gain from capturing process data are the ones whose work is high-judgment and poorly instrumented:

Engineering organisations — architecture decisions, incident post-mortems, and the long tail of "we tried that, here's why it didn't work".
Regulated professions — legal, compliance, clinical and financial work, where the reasoning behind a conclusion is the deliverable and the conclusion alone is nearly worthless.
Consultancies and agencies — where the methodology is the product but only the final deck is ever archived.
Research teams — negative results, discarded hypotheses, and the reasons an approach was abandoned, none of which reach the paper.
Support and on-call — the diagnostic path, not just the resolution code.
Any team running more than one agent — because a second agent has no way to learn from the first one's mistakes unless those mistakes were written down somewhere both can read.

How MDflow fits

MDflow is a markdown workspace for people and AI agents. It does not capture keystrokes or tool-call telemetry, and it is not an RL environment. What it does is keep the written layer of process data instead of throwing it away — which, for most knowledge work, is where the reasoning actually lives.

What already lines up today:

Version history captures the previous version on every saved change, across every write path — the in-app editor, the HTTP API and the MCP server alike. You get a line-by-line markdown diff between any earlier version and the current text, and a non-destructive restore that preserves the current text as a new entry. That is a trajectory, not a snapshot: the document's own history of what it used to say.
The Document Log at /log is a cross-document activity feed that answers "what did agents and integrations do to my documents?" Every row carries an actor — you for browser actions, automated · <token name> for anything arriving over the API or MCP — so giving each agent its own token turns the log into a per-agent trace. Click an edited row for the diff.
Anchored comments on shared documents attach the argument to the passage it is about, and survive after the text they discussed was edited away, kept as orphaned notes rather than deleted. The disagreement is process data; most tools resolve it into nothing.
Folder descriptions state why a body of work exists, and mdflow_get_context ranks them above folder names and document titles when assembling context for an agent. This is the cheapest process-data capture there is — one paragraph explaining what a folder is for.
Plain markdown, addressable both ways. Every document has a raw .md twin with frontmatter, and the workspace is discoverable through llms.txt, an agent card and an OpenAPI spec. Process data that only one product can read is that product's cache, not your asset.
Agents can write back. An agent that finishes an investigation can deposit what it learned as a document rather than dropping it when the session ends — see letting agents write to your knowledge base.

Where we are headed — direction, not a dated commitment. The obvious gaps are on the retrieval side: history is currently private to the document owner and deliberately not exposed over the API or MCP, which keeps the trail safe but also keeps it invisible to the agent that would most benefit from reading it. Making a document's history legible to an agent you trust, without turning it into an exfiltration surface, is the interesting design problem. Beyond that, richer structure around decisions themselves — so "what did we rule out, and when did that change?" is a query rather than an archaeology exercise.

The bottom line

The models are going to keep getting better at everything that has a verifier. What they will not do is infer the reasoning your organisation never wrote down. State data is abundant and nearly worthless; process data is scarce, appreciating, and currently being deleted by every tool that keeps only the last save.

You do not need a data strategy to start. You need somewhere the reasoning goes that is not a chat thread, and a tool that keeps the trail.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What is the difference between process data and state data?

State data is the final saved artefact — the row in the ERP, the closed ticket, the approved document. Process data is the trajectory that produced it: the sequence of decisions, the intermediate drafts, the options considered and rejected, and the reasoning behind the choice. State data tells a model what the answer was. Process data tells it how a competent person got there, which is the part that transfers to a new problem.

What is Verifier's Law?

Verifier's Law is Jason Wei's observation that the ease of training AI to solve a task is proportional to how verifiable the task is, and that all tasks that are possible to solve and easy to verify will eventually be solved by AI. He lists five properties that make a task easy to verify: objective truth, fast verification, verification that scales to many candidate solutions at once, low noise, and a continuous reward rather than a pass/fail bit.

Why did AI get good at coding before it got good at law or finance?

Because software had already built the verification infrastructure. Unit tests give objectively checkable correctness, the community broadly agrees on what working code means, and public repositories supply effectively unlimited fresh examples with commit messages attached as free reasoning traces. Law, finance, biology and healthcare score low on all three: correctness is contested, verification is slow and expert-bound, and the examples sit locked inside private workflows that no public system ever captured.

What is type 1 versus type 2 data?

Type 1 data is captured from real work as it actually happened — commits, session replays, the trail of a real decision. Type 2 data is contrived: experts are hired, sat in an artificial setting, and asked to manufacture examples. Type 2 is enough to teach a model the basics of a domain. Type 1 is what carries it from competent to expert, because the realism is inherited from the work itself rather than reconstructed by someone imagining the work.

How do you capture process data in a document workspace?

Write the reasoning down where the artefact lives, and keep the trail. In MDflow that means version history capturing every saved change with a line-by-line diff and non-destructive restore, the Document Log at /log recording who or which agent changed what, anchored comments preserving the argument next to the passage it is about, and folder descriptions stating why a body of work exists. None of that requires new behaviour from the people doing the work — it just requires a tool that does not throw away everything except the last save.