Why AI Document Readers Choke in the Middle of Long PDFs

Picture a 40-page vendor contract. You upload it to your favorite AI tool and ask what the termination clause says. The clause is on page 22. The model answers fast and sounds sure of itself. It also happens to be wrong, and if you dig into what it actually did, it grabbed language from the boilerplate on page 3 instead.

That's not a hypothetical edge case. According to a new paper called SynthDocBench, it's close to a predictable failure mode. The middle third of a long document is the hardest part for five of the six vision-language models the researchers tested closely, and most models get steadily worse the deeper into a document you ask them to look. If you build products on top of these models, or you're picking between OCR and document-AI engines for a real workflow, this is worth twenty minutes of your attention.

The problem with grading on a single messy exam

Most vision-language model (VLM) benchmarks for documents, things like DocVQA, ChartQA, and MMLongBench-Doc, do a reasonable job of asking "can this model read this document and answer a question about it?" What they don't do well is tell you why a model failed.

Real documents vary along several dimensions at once: how long they are, how complicated the layout is (single column of text versus a dense financial table versus a form with checkboxes), what kind of content is embedded (plain text, tables, charts, scanned images), and how hard the question is. When a model gets a wrong answer on an existing benchmark, all four of those factors are tangled together. Was the model confused because the document was long? Because it had an unusual layout? Because the answer needed table math instead of Word find-and-replace? You genuinely cannot tell, because the benchmark documents were scraped from the real world and nobody controlled for any of it.

That's the gap SynthDocBench tries to close. Instead of collecting real documents and hoping the variation averages out, the authors generate documents from scratch and dial each factor independently, the way you'd design a controlled experiment in a lab rather than observe whatever shows up in the wild.

Building documents like a science experiment, not a scrape

Here's the mechanism, in plain terms. The researchers built an LLM pipeline that generates entire documents end to end: content, structure, and visual layout together. Every document is assigned one of six layout archetypes (think: report-style, form-style, mixed table-and-text, and so on), and then the paper varies document length, layout complexity, what mix of text/tables/charts/images appears, and what type of question gets asked, each one independently of the others. That's what "combinatorial design" means here: instead of one knob that mixes everything together, you get several knobs you can turn one at a time while holding the rest fixed.

There's a clever wrinkle. Forty percent of the time, the generation process deliberately overrides the "expected" pattern for a document, for example putting a chart somewhere layout conventions wouldn't usually put one. The point is to stop models from gaming the test. If a model learns "answers about revenue are always near the top of page 1" because that's how real annual reports tend to be laid out, it can score well without actually reading the document, it's pattern-matching on document conventions rather than doing comprehension. The 40% random override breaks that shortcut, because the model can't assume the document is laid out the "normal" way.

The other headline feature is length. SynthDocBench documents run substantially longer, and with more structural variety, than what DocVQA, ChartQA, or MMLongBench-Doc typically include. That matters because a lot of the interesting failures in this paper only show up once documents get long, and shorter benchmarks would have missed them entirely.

Longer documents make models measurably worse

The first finding is the least surprising but still worth stating plainly: accuracy drops as documents get longer. This isn't shocking on its own (everything gets harder as context grows), but the rate and sharpness of the decline is the point. Because SynthDocBench controls length independently of layout and content type, the researchers can attribute the drop specifically to length rather than to "longer documents also tend to have messier layouts," which is the confound every prior benchmark carried.

For anyone building a product that processes multi-page documents (leases, medical records, financial statements, technical manuals), this is a reminder that a demo working well on a two-page sample tells you very little about how it'll hold up on the 60-page version a real customer uploads.

The middle-of-the-document blind spot

This is the paper's most interesting result. The researchers split each document into thirds (early, middle, late) and looked at where models made mistakes. In five of the six models tested this way, the middle third was the hardest part of the document to answer questions about. Not the end, where you might expect models to run out of steam. The middle.

They also measured what they call the "Early-to-Late trend," basically, does accuracy on early-document questions versus late-document questions get worse as you move through the text. Five of six models showed a negative trend, meaning they did worse on later material than earlier material, and the steepest decline hit 8.3 percentage points.

If this pattern sounds familiar, it should. Researchers studying long-context text-only language models have documented something similar for years, often called "lost in the middle": models are good at using information from the start and end of a long input and worse at using information buried in the center. SynthDocBench is showing that the same failure shows up in visual document understanding too, not just plain text. That's a meaningful extension, because a lot of people assumed multimodal document models would behave differently since they're processing layout and images, not just a token stream. They don't behave differently. The weak spot travels with the architecture.

Practically, this means a model reading a 30-page insurance policy is disproportionately likely to miss something on pages 12 through 20, compared to something on page 2 or page 28, even when nothing about those middle pages is objectively harder to read.

Charts fall apart exactly when you need them most

The third failure mode is about chart comprehension specifically. Reading a bar chart or line graph and answering a question about it is already a harder task than reading text, since the model has to map visual elements (bars, axes, legends) to numeric meaning. SynthDocBench found that this skill degrades badly once charts are embedded in long documents, worse than you'd predict just from the general length-related decline described above.

Think about where charts actually live in the documents people care about: quarterly earnings decks, scientific papers, government reports, sales dashboards exported to PDF. These are exactly the documents where a chart on page 15 carries the number someone is trying to extract, and exactly the setting where SynthDocBench shows the model is least reliable.

Why it matters for anyone building or choosing OCR

If you're evaluating OCR or document-AI engines for something beyond a toy demo, three practical takeaways come straight out of this paper:

• Test with your real document lengths, not short samples. A model that scores well on your five-page test PDF is not the same model once your users start uploading 50-page contracts. Ask any vendor (or run any open model) against documents at the length you'll actually see in production.
• Don't trust answers about the middle of long documents without spot-checking them. If a workflow depends on extracting something from the interior of a long file, that's precisely the zone this paper flags as least reliable. Consider chunking long documents and querying each chunk separately, or use retrieval to pull the relevant section before asking the model to read it, rather than dumping the whole document into one long context window.
• Chart-heavy long documents need extra scrutiny. If your use case involves reports with embedded charts and tables (financial filings, research papers, analytics exports), don't assume a model that handles isolated charts well will handle the same chart buried on page 20 of a report just as well. Validate charts specifically, separately from plain text extraction.

None of this is an argument to avoid VLMs for document work. It's an argument for testing document AI the way SynthDocBench tests it: broken into controlled pieces, not judged on vibes from a handful of examples.

What this benchmark can't tell you

The honest caveats matter here. SynthDocBench is fully synthetic, generated by an LLM pipeline rather than sourced from real-world scans, filings, or forms. Synthetic documents are useful precisely because you can control every variable, but that control comes at a cost: an LLM-generated "invoice" or "report" might have its own subtle statistical fingerprint that differs from how real invoices and reports are actually laid out and phrased. The paper's own claim, that current models may be overfitting to benchmark artifacts, cuts both ways. It's plausible that some of what SynthDocBench measures is itself a new kind of artifact, just a more carefully controlled one.

The evaluation also covers seven frontier VLMs, which is a meaningful sample but still a snapshot of models available in mid-2026. Model behavior on long-context tasks has been moving fast, and it's worth checking whether newer releases have specifically targeted the "lost in the middle" weakness before assuming it applies uniformly going forward.

Still, the core contribution holds up regardless of those caveats: by controlling length, layout, modality, and question type independently, the researchers isolated failure modes that messier real-world benchmarks were structurally incapable of surfacing. The middle-of-document blind spot in particular is a genuinely new and useful finding, not just a repackaged version of something we already knew. If you're building anything that reads long documents for a living, it's worth designing your evaluation pipeline, and your product's chunking strategy, with that blind spot in mind.