A better test for OCR models

Built to separate models

We started with 80 hard pages. Then we tried 20,000 balanced 20-page mixes. We kept the mix that best matched the 80-page results.

Across 400 balanced mixes, the same top model won 99% of the time.

We did not just pick the 20 hardest pages. That made too many models fail on the same pages. A low score is not useful if it cannot tell you which model is better.

Twenty pages are small enough for us to rerun the full model catalog, but the calibration keeps the result close to the larger test. This lets us update the benchmark without losing the differences that decide rank.

What we test

We test the things people actually upload: typed pages, rough scans, tables, multi-column documents, handwriting, receipts, and screenshots. The ranking uses four hard groups below, with five pages in each.

How scoring works

Each page has its own checks. We score the four groups separately, then average them. A model cannot win on tables alone and ignore everything else.

That matters because the math pages have more checks than the table pages. We want both to count, without letting one document type decide the whole result.

Returning some text is not enough. A model has to pass the checks that matter for that page. The basic check that only asks whether a model returned text does not add to its quality score.

Complete runs only

We only rank a model after it completes all 20 pages. A partial run, or a score carried over from an older test, does not get a place on the leaderboard. It stays unranked until we have a full result.

We also show a 95% confidence range. When two ranges overlap, they share a tier. That is a more honest result than claiming a winner from a small score gap.

The other 11 images

The 20 hard pages decide rank. We also run 11 everyday images: typed text, handwriting, receipts, invoices, tables, dense text, and a website screenshot. They show what each model can read in more familiar situations, but they do not change rank.

Ten of those images have a complete answer key. We compare the model's text with that answer. The screenshot has expected lines instead. This gives each model page useful evidence without mixing an easy-document score into the main ranking.

A result you can check

We keep the test pages, expected answers, scoring code, and source output for each result. Before a run starts, we check that the test files match the published version.

If we change an image, an answer, or the scoring rules, it becomes a new benchmark version. We do not quietly mix old and new results in one leaderboard.

Test pages come from AI2's olmOCR-bench (ODC-BY-1.0).