The Benchmark Where the Best AI Model Still Fails 85% of the Time on a PDF

Picture a benefits administrator with a 40-page PDF open on one monitor and an employee's question on the other: "If I have the PPO plan and see an out-of-network specialist, what's my actual out-of-pocket cap after the deductible?" The answer is in there. It's just split across a table on page 12, a footnote on page 14 that excludes certain specialist visits, and an amendment on page 38 that quietly overrides the number on page 12. A person who knows how to read benefits packets can find it in a few minutes. A new paper called GDP.pdf gave this exact kind of question to seven frontier multimodal models. The best one got 15% of the benchmark's 100 questions right. The worst got 1%.

That's not a benchmark where models are close to solved and squeezing out the last few points. That's a benchmark where the field is barely on the board. And the reason is worth understanding, because it says something uncomfortable about how we've been measuring "AI can read documents now."

Why your OCR benchmark is lying to you

Most document AI benchmarks test one skill at a time. There's a benchmark for reading text off a scanned page (OCR). One for understanding where things sit on a page, like which line item belongs to which column (layout analysis). One for reading values off a bar chart (chart reasoning). One for answering questions about a table (table QA). One for general "look at this document and answer a question" tasks (document VQA).

A model can score well on all five of these in isolation and still fail at the thing a real professional actually needs: opening a specific, messy, real-world PDF and answering a specific, consequential question about it. The GDP.pdf authors put it plainly in the paper's framing: a high score on any one isolated capability doesn't tell you whether the model can handle a realistic question about a real document. Reading a table correctly doesn't help if the model doesn't notice that a footnote three pages later excludes half the rows. Reading a chart correctly doesn't help if the model doesn't cross-reference it against the paragraph that explains what the chart is actually measuring.

This is the same trap OCR vendors have been navigating for years: an engine that nails a clean, standalone invoice can still choke the moment that invoice is a leaked photo of a photocopy with a coffee ring on it, or spans multiple pages with a continuation footer, or has a table that jumps a column halfway down. Isolated-skill benchmarks measure the clean case. Real work lives in the messy case.

How they built a benchmark that's hard to fake

The clever part of GDP.pdf isn't the documents. It's the filter they used to decide which questions were worth including.

The team recruited working professionals across ten fields, real people who deal with benefits packets, leases, datasheets, clinical guidelines, and construction plans as part of their actual jobs, and had them write questions about real documents from their domain. But writing a hard-sounding question isn't the same as writing a question that actually separates capable models from incapable ones. So they added a gate: a candidate question only made it into the final benchmark if at least two frontier multimodal models failed it in a way that mattered. Not a stylistic quibble about phrasing. A wrong answer, missed evidence that should have changed the answer, or a fabricated claim the document doesn't support.

That two-model-failure rule matters more than it might sound. It means every question in the benchmark has already been proven, empirically, to trip up current AI systems in a specific and meaningful way. There's no filler in there, no softball question that every model gets right and that just pads the score. Each of the 100 items is there because it beat at least two strong models first.

They also didn't just grade pass or fail. Every question comes with a rubric of atomic criteria, small, checkable requirements the answer needs to hit, so the paper can report two different numbers: a graded rubric score (partial credit for getting some of the pieces right) and a strict task-level pass rate (did the model get the whole thing right, full stop). And every item is tagged against a taxonomy of eleven capabilities across three tiers, covering things like text extraction and grounding (can you find the right passage and quote it accurately), table and chart comprehension, cross-referencing across pages, spatial reasoning (where things sit relative to each other on the page), and abstention (does the model know when a document simply doesn't support an answer, and say so, instead of guessing).

That abstention piece deserves its own callout. A lot of document AI evaluation only tests whether the model gets the right answer when there is one. GDP.pdf also checks whether the model will fabricate an answer when the document doesn't actually contain the information, which is arguably the more dangerous failure mode in a professional setting. Nobody gets hurt by a model saying "I can't find that in this document." People get hurt by a confident, wrong number in a benefits calculation or a lease clause.

The numbers that should worry you

The headline result is stark: across seven frontier models, the best one passed 15% of the 100 items on the strict task-level metric. The worst passed 1%. For a field that has spent the last two years claiming near-human performance on document understanding, a 15% ceiling on questions written by actual domain professionals about actual professional documents is a hard number to explain away.

It's worth sitting with what that gap represents. These aren't models failing at OCR in the classic sense, they can mostly read the text on the page fine. They're failing at the reasoning layer on top of it: connecting a number in a table to a caveat in a footnote, noticing that a later amendment supersedes an earlier clause, counting symbols correctly on a floor plan, or realizing that a chart's axis doesn't mean what a quick glance suggests it means. That's the layer where professional PDFs actually live, and it's the layer general-purpose multimodal models haven't been trained to handle well, because most training and evaluation data doesn't look like a 40-page benefits packet with three amendments stapled to the back.

The six ways models keep losing

The paper traces most of the failures back to a fairly small, recognizable set of loss patterns, and this is the most useful part for anyone building with these tools day to day:

• Misaligned tables. The model reads a value from the wrong row or column, especially when a table spans a page break or has merged cells.
• Misread charts. The model gets the general shape right but misreads a specific value, an axis label, or which series a data point belongs to.
• Skipped footnotes and exclusions. The model finds the headline number but misses the caveat that changes it, the classic benefits-packet trap from the opening example.
• Miscounted floor-plan symbols. On construction and architectural documents, the model undercounts or overcounts repeated symbols (outlets, fixtures, doors) that a human would tally by hand.
• Scan noise. Real-world documents aren't clean PDFs exported from Word. They're photocopies of photocopies, faxed forms, and phone photos of physical documents, and image quality degrades model accuracy more than people assume.
• Superseding amendments. A document says one thing on page 5 and an amendment on page 30 changes it. Models tend to answer from whichever version they encounter first rather than resolving the conflict.

None of these are exotic edge cases. They're the ordinary texture of professional paperwork. A lease has addenda. A datasheet has a revision history. A clinical guideline has an update memo. If your benchmark doesn't include documents shaped like this, it isn't measuring the thing people actually need.

Why it matters for anyone building or choosing OCR

If you're building a product on top of OCR or document AI, or evaluating vendors to pick one, GDP.pdf is a useful reality check against marketing claims. "State of the art on document VQA" and "can answer a real question about a real 40-page policy document with a footnote that changes the answer" are different claims, and this paper shows the gap between them is enormous, not marginal.

The practical takeaways:

• Don't trust single-capability benchmarks as a proxy for end-to-end reliability. A model that tops a table-QA leaderboard can still miss the footnote that overrides the table. If a vendor only quotes isolated-skill scores, ask what happens when the skills have to compose.
• Multi-model retry earns its keep here. Since the failure modes are inconsistent across models (misaligned tables for one, missed amendments for another), routing the same document to a second or third engine when the first answer looks shaky is not a luxury, it's close to necessary for anything with real stakes attached.
• Abstention is a feature to test for, not assume. Before you trust a model's answer on a document, check whether it will actually say "not in this document" when the answer isn't there, rather than confidently inventing one. GDP.pdf's abstention tier exists because this is currently a weak point across the board.
• Treat scan quality as part of the pipeline, not an afterthought. Scan noise was one of the six recurring loss patterns. If your ingestion pipeline doesn't clean up or flag low-quality scans before they hit the model, you're inheriting an error source you could partially engineer away.
• Budget for human review on anything with an amendment, exclusion, or revision history. These are exactly the documents where current models fail most, and exactly the documents where a wrong answer is most likely to cost someone money or health coverage.

What this benchmark doesn't tell you

A few honest caveats. A hundred questions is a small benchmark. It's enough to make a strong point about the shape of the failure (six recurring patterns across ten professional domains is a meaningful signal), but it's not enough to give you a precise, stable percentage you should quote as "the" accuracy number for any given model on this class of task. Small benchmarks are also more sensitive to which specific documents and questions got selected, even with the two-model-failure filter in place.

The two-model-failure selection rule is smart for surfacing genuinely hard, meaningful questions, but it also means the benchmark is explicitly constructed to be hard for today's models. That's the point, but it means the 15%/1% numbers describe the ceiling on adversarially selected questions, not a random sample of "questions professionals typically ask." Real-world accuracy on the full mix of easy and hard questions a benefits administrator or construction estimator asks day to day will be higher than 15%, just not as high as the marketing usually implies.

And seven models, evaluated once, on one hundred items, is a snapshot rather than a trend line. Models are moving fast, and this specific 15% ceiling will likely rise. What's more durable is the taxonomy: the eleven capabilities and the six loss patterns are a genuinely useful checklist for evaluating any document AI system, today's or next year's, because they describe structural gaps in how these models handle grounded, multi-part reasoning over real paperwork, not just a scoreboard that will be stale in six months.

The honest summary: professional PDFs are harder than the benchmarks made it look, the industry has been grading on the wrong axis, and GDP.pdf is a solid first attempt at grading on the right one.