skip to content
$cat document-extraction-for-ai-agents.md

Document Extraction for AI Agents: Where Accuracy Is Won

15 min readby MDflowview as .md
An emerald wireframe scene on a dark terminal grid: a skewed pie-chart disc, a warped lattice of fused empty cells and a leaning stack of blank panels feeding through a faceted crystal prism and emerging as neat parallel bars on a flat plane

Ask an engineer why their retrieval system gave a wrong answer and you will usually get a debugging session about the answering side: the prompt, the reranker, the chunk size, whether a bigger model would have got it. It is the most instrumented part of the stack and the easiest part to change.

It is also, frequently, not where the error happened.

TL;DR — Document extraction for AI agents is the step that converts PDFs, spreadsheets and decks into the text a model reads, and it silently caps everything downstream. Mis-parsed tables and charts do not surface as errors — they surface as confident wrong answers, because the model never sees the original. Published benchmarks show OCR noise cascading into retrieval and generation, and chart and diagram content disappearing in conversion. The practical conclusion from teams operating at scale: buy the extraction layer, build the eval that measures it, and store the result as markdown you control. MDflow is where that markdown can live once it is clean.

This post is built on the AI Engineer conversation "Why Datasite Stopped Building Its Own Parsing" (published 4 November 2026), in which Raj Bakhru, global CTO of Datasite, is interviewed by Adit Abraham, co-founder and CEO of Reducto — the vendor Datasite moved to. The operational numbers below are Bakhru's own, described on stage, and they are attributed as such; the interview is a customer story told alongside its vendor, so the general lesson is the part worth taking, not the recommendation. The supporting research is independent and cited separately.

What document extraction for AI agents actually is

Document extraction is the step that turns a file into the text an agent reads.

It sits at the very front of the pipeline, before chunking, before embedding, before retrieval, before the prompt. A PDF goes in; markdown, or something like it, comes out. Everything after that point operates on the output and has no access to the input. That one-way boundary is the whole reason extraction quality is structural rather than incremental: a retrieval system cannot recover information that was destroyed before it was indexed.

The ICCV 2025 study OCR Hinders RAG made this measurable. The authors built OHRBench from 8,561 document images across seven real-world domains with 8,498 question-and-answer pairs, and separated extraction errors into two kinds:

  • Semantic noise — the text is simply wrong. A character misread, a number transposed, a word dropped.
  • Formatting noise — the text is right but its structure is not. A table flattened into prose, a two-column page interleaved line by line, a heading demoted into body text.

Their headline conclusion is uncomfortable and worth quoting carefully: evaluating the OCR solutions then in use for RAG, they report that none is competent for constructing high-quality knowledge bases. Not "some are better than others" — none cleared the bar. And both noise types degraded downstream performance, with the degree of noise tracking the drop in RAG performance.

"It's all edge cases"

Bakhru's description of building this layer in-house is the most vivid part of the talk, and the phrase he lands on is that it is all edge cases — "literally everything is an edge case."

Three of his examples are worth keeping, because each one names a different failure mode:

  • A pie chart the model read wrong because the shade of blue in the legend did not correlate precisely with the shade in the slice. His observation about that case is the sharpest thing in the interview: hand the same model the same data as a bar chart and it gets the answer right. The model was not the variable. The representation was.
  • Merged cells in financial spreadsheets, which he describes as the most complicated documents in the world — hard enough for a human to read, let alone a parser deciding which header governs which column.
  • File-format long tail — .xlsb, macro-enabled workbooks, template files. Each one a small amount of work, and the list does not end.

The independent research lines up with each of these. A 2025 benchmark of market-research deliverables, Meet Your New Client: Writing Reports for AI, converted PDF and PowerPoint reports to markdown and measured end-to-end question answering. Its finding: text is reliably extracted, while significant information is lost from complex objects like charts and diagrams. That is the pie chart, generalised and quantified.

And on tables, a January 2026 study, An Empirical Investigation of Robustness in Large Language Models under Tabular Distortions, distorted tables that were otherwise in canonical form and measured what happened. Even frontier models showed a minimum 22% accuracy drop under distortion. The authors' conclusion is the part that should worry anyone running a document pipeline: the models lack an inherent ability to detect and correct subtle distortions. Told explicitly, in a system prompt, that the table might be malformed, they partially adjust — "though not consistently or completely."

A bad parse is worse than a missing document

Those two findings combine into the thing that makes extraction errors expensive rather than merely annoying.

A document that fails to ingest is a visible failure. Retrieval comes back empty, the agent says it does not know, and somebody investigates.

A document that ingests incorrectly produces none of that. The retriever finds the chunk, the chunk looks like a table, the model reads it fluently and answers with full confidence. The one party that could notice the discrepancy — a reader comparing against the original — is not in the loop. The agent cannot notice, because for the agent the extracted text is the document.

This is also why extraction errors are so hard to attribute after the fact. The symptom shows up as "the agent got the revenue number wrong", which looks like a model problem, a retrieval problem, or a prompting problem. All three are cheap to tinker with and none of them is the cause. Teams can spend a long time optimising the answering side of a pipeline whose ceiling was set before the first embedding was computed.

Bakhru puts the inversion plainly, and it is the single sentence worth taking from the whole talk:

"A lot of people think that all the work needs to be done on the other side, the answering side, and most of the work actually is on the extraction side. You get that right, and everything else can follow."

The build-versus-buy decision, as an allocation question

Datasite's scale makes the trade-off unusually legible. On stage, Bakhru described a platform where roughly 60% of transactions above $100 million are run, around 55,000 data rooms a year, billions of pages of content, and over 500,000 monthly active users — with bankers occasionally dropping a terabyte of documents in at a moment's notice.

He and his co-founder had built their own parsing and ingestion pipeline at their previous startup, Blue Flame, being, in his words, scrappy and trying to save every dollar. It worked — until it did not, and the "did not" arrived as support tickets from some of the world's largest private equity firms, each one a new edge case.

His framing of the decision is not about capability. It is about allocation:

"What do I want my limited set of high-quality engineers working on? It is not the support cases of making sure that we're extracting content properly when that's already been solved."

The arithmetic behind that is specific. He puts the engineering org at about 250 people and the firm's growth ambition at roughly $200 million of incremental revenue a year, pursued through five major initiatives. He wants those 250 engineers spread across the five, and as few as possible on the layers underneath — ideally none.

There is a second-order observation in the same answer that is worth flagging on its own: he describes a barbell effect in hiring under agentic coding. The engineers who get the most leverage from AI coding tools are the ones with strong mental models of codebases and long experience, who are now, as he puts it, 10x'd over their normal 10x. Juniors also gain, because the models can generate the documentation that never existed. The middle, he says, is "so-so." Whatever one makes of that, it sharpens the allocation argument: if your most leveraged people are also your scarcest, spending them on a solved problem is the most expensive choice available.

The eval is the part you cannot outsource

The one thing Datasite did build is the measurement. Bakhru describes an internal benchmark they call VDR bench: a representative data room of hundreds of documents, a few hundred questions with known answers, re-run whenever anything in the stack changes. Their bar is that it must beat dropping the same files into a frontier lab's own product — Claude Projects, OpenAI Projects — because if it does not, the platform is not adding value.

That benchmark is also what makes their parsing claim checkable rather than rhetorical: he says their data science team measured recall improving by about 22% from the change in parsing. Treat the specific figure as what it is — a customer's own measurement, described on stage next to the vendor, on a benchmark that has not been published. The transferable part is the method. They know the number because they built an eval on their own documents, with their own questions, and ran it on every change.

Without that, "our extraction is good enough" is a belief, not a finding. With it, extraction quality becomes a dial you can turn deliberately — and the first thing most teams discover when they build one is that the answering side was never their binding constraint.

Which applications benefit most from getting extraction right

The payoff concentrates where meaning lives in layout, and where being confidently wrong is expensive:

  1. Financial and transaction analysis — spreadsheets, models and statements, where a shifted column is a wrong number rather than a wrong word.
  2. Legal and contract review — defined terms, numbered clauses, cross-references and schedules whose structure is the meaning.
  3. Regulated diligence and compliance — where an answer has to be traceable to a source, so a silently reshaped source breaks the audit trail as well as the answer.
  4. Scientific and technical literature — figures, captions and multi-column layouts that linearise badly.
  5. Healthcare records — mixed scans, forms and tables, often with the critical value in a cell rather than a sentence.
  6. Internal knowledge bases built from legacy document stores — decades of decks, PDFs and exports, where nobody is going to re-author the originals.

For anything that is already plain prose in plain text, extraction is close to free. The further a document sits from that, the more of its accuracy is decided before a model is involved at all.

How MDflow fits

Let us be precise about the boundary, because it matters for whether any of this is useful advice.

MDflow does not parse PDFs, spreadsheets or slide decks. It is not an OCR engine, and nothing in this post should be read as a claim that it replaces one. Extraction is an upstream layer, and a specialised one — which is exactly the talk's point about build versus buy.

What lines up today is the other half of the problem: where clean, extracted text lives afterwards, and whether anything is lost on the way out.

  • Markdown is the handoff format. It is what extraction tools emit and what models read most reliably, and MDflow stores it as plain markdown rather than as a proprietary document model. There is no second lossy conversion between the parser's output and what an agent eventually reads. Documents are also served as raw .md twins with frontmatter intact, so an agent can fetch the canonical text directly rather than scraping a rendered page.
  • A pipeline can write straight in. The HTTP API and the MCP servers, authenticated with a personal access token or OAuth, let an ingestion job create and update documents programmatically. Whatever you run your extraction with, the output has somewhere to land that your agents, your editor and your teammates all read from. Markdown import handles the bulk case, including dragging a whole folder tree in at once.
  • Folder descriptions carry the context the file name lost. Each folder has a description of what its documents are for, and mdflow_get_context ranks those descriptions above folder names and titles. A folder described as "quarterly financials, extracted from the board decks — figures verified against the source PDFs" tells a retrieving agent something the extracted text itself never will. That is the structure extraction cannot recover, supplied by hand, once.
  • The original can sit beside the extraction. Attachments let you keep the source PDF, workbook or deck on the document itself, as a link in the markdown, so a human can check a suspicious number against the original. Attachments are a UI feature — they are not part of the API or the MCP servers in this release — but they close the verification loop that extraction errors otherwise leave open.
  • Capture-time extraction, where it applies. The Web Clipper converts a page to markdown at the moment you save it, and its Interpreter can run natural-language extraction prompts against the page using your own provider key. For web content, that is the extraction step, done once, into the format you keep.

And the case that needs no extraction at all is worth stating: a document written in markdown from the start never goes through this pipeline. There is no pie-chart legend to misread and no merged cell to misattribute, because the authoring format and the agent's reading format are the same file. That is not available for the terabyte of legacy PDFs — but it is available for everything you write from here on.

Where we are headed. The direction — not a dated commitment — is to make the context around a document richer and more machine-readable, so that what a document is and how it got there travels with it rather than living in someone's head. The principle underneath does not move: plain markdown you own, exportable in full, with no lock-in on the way out.

The bottom line

The answering side of an AI pipeline is where the attention goes because it is where the knobs are. The extraction side is where the ceiling is.

A document that fails to ingest announces itself. A document that ingests wrong does not — it produces a fluent, confident, incorrect answer from a model that has no way to know the table it just read was reshaped on the way in. Published benchmarks put numbers on both halves of that: extraction noise cascades through retrieval and generation, and even frontier models lose a fifth of their accuracy on distorted tables without noticing.

So buy the layer that is already solved, spend your scarce engineers on the problem that is yours, build the eval that tells you which is which — and keep the clean output in a format that will not need converting again.

Start free · Connect an AI agent · Read the API docs

Frequently asked questions

What is document extraction for AI agents?

Document extraction is the step that turns a PDF, spreadsheet, slide deck or scan into the text an agent actually reads — usually markdown. It sits before chunking, embedding and retrieval, so every later stage inherits whatever it produced. If a table arrives with its columns shifted, no amount of retrieval tuning recovers the original numbers.

Why does extraction quality matter more than the answering model?

Because errors introduced at extraction are invisible downstream. The model never sees the source document, only the extracted text, so a mis-parsed table looks exactly like a correct one. The ICCV 2025 OHRBench study found that OCR noise cascades through retrieval and generation, and concluded that none of the OCR solutions it evaluated was competent for building high-quality RAG knowledge bases.

What are the hardest documents to extract correctly?

Anything whose meaning lives in layout rather than word order: financial spreadsheets with merged cells, charts where the legend carries the labels, multi-column scans, macro-enabled and template workbook formats, and slide decks. Plain prose extracts reliably. A 2025 benchmark of market-research deliverables found text was reliably extracted from PDF and PowerPoint while significant information was lost from complex objects like charts and diagrams.

Should we build our own document parsing pipeline?

Build it only if parsing is the product. Datasite's CTO Raj Bakhru described moving off a self-built pipeline precisely because the edge cases were endless and the support burden consumed engineers he wanted on revenue work. The decision is less about capability than about where your scarce senior engineering time earns the most.

How does markdown help with document extraction?

Markdown is the format most extraction tools emit and the format language models read most reliably, so it is the natural boundary between the two. Storing the extracted result as plain markdown also means there is no second lossy conversion on the way out, and a document authored in markdown to begin with skips extraction entirely.

Further reading