CC-OCR Put Nine Top AI Models Through 7,058 Real Documents, and Most Failed the Basics

Feed a state-of-the-art vision-language model a photo of a Thai restaurant menu shot at a slight angle under fluorescent lighting, and ask it to pull out the dish names and prices. A surprising number of them will start confidently, then get stuck. Not stuck as in "wrong answer." Stuck as in repeating the same three words over and over until the output cuts off, like a skipping record.

That's one of the findings buried in CC-OCR, a benchmark released in December 2024 by a team primarily from Alibaba's Qwen group, alongside South China University of Technology. The paper tested nine prominent large multimodal models, including GPT-4o and Gemini, on 7,058 real, annotated images across four categories of text-reading tasks. The headline result: models we think of as broadly capable at "reading" images have specific, predictable failure modes that a simple accuracy score never surfaces. If you're building anything on top of AI-powered document processing, that distinction matters more than the leaderboard rank.

A real receipt from the CORD dataset — the kind of angled, real-world document CC-OCR uses to test models. Source: CORD (Park et al., NAVER CLOVA, 2019), licensed CC BY 4.0.

What OCR benchmarks usually get wrong

Optical character recognition, pulling readable text out of an image, used to mean running a specialized model trained on one narrow job: reading printed English text on a scanned page, say, or reading license plates. Large multimodal models (LMMs) changed the framing. These are the same family of model that can describe a photo or answer questions about a chart, and they can also read text embedded in images, because they were trained on huge piles of image-text pairs scraped from the internet, PDFs, and screenshots.

The problem is that "read the text" turns out to hide a lot of different, harder sub-skills once you leave clean scanned pages. Real documents have text at odd angles. They have tables where the meaning lives in the grid structure, not just the words. They mix languages within a single line. They're handwritten, or water-damaged, or photographed at a weird angle on someone's phone.

Most existing OCR benchmarks before CC-OCR tested a narrow slice of this problem. Some focused purely on scene text (street signs, storefronts). Others focused purely on document parsing (turning a PDF into structured Markdown or HTML). Few tested multiple languages beyond English and Chinese. Almost none combined "does the model find the right region of the image" (grounding) with "does the model produce accurate text" as separate, measurable things. That gap is what CC-OCR set out to close, and it's why the paper is worth reading even if you'll never train a model yourself. It tells you what to actually check before you trust an OCR pipeline with your documents.

How CC-OCR is built: four tracks, not one score

CC-OCR splits the evaluation into four tracks, each targeting a different real-world scenario:

Multi-scene text reading. This is classic "text in the wild": street signs, product packaging, screenshots, receipts. The challenge isn't reading text in isolation, it's finding the text at all when it's small, angled, or embedded in a busy background, then reading it accurately.

Multilingual text reading. The benchmark spans languages beyond the usual English/Chinese pair that dominates OCR research, testing whether a model that reads English signage well can also handle, say, Japanese vertical text or Arabic right-to-left script.

Document parsing. This is the "PDF to structured text" job that's increasingly central to AI pipelines: taking a scanned document and converting it into Markdown, HTML, or another structured format that preserves tables, headers, and reading order, not just a flat text dump.

Key information extraction. Given a document like an invoice or a form, pull out specific fields, the total amount, the date, the vendor name, rather than transcribing everything indiscriminately. This is the task that most directly maps to what businesses actually want from OCR: not "read me the whole page" but "tell me the invoice number."

Across these four tracks, the benchmark includes 39 subsets. A striking design choice: 41% of the 7,058 images come from real applications and were released for the first time in this paper, rather than being repackaged from older academic datasets. That matters because older benchmarks tend to get memorized. A model that's seen a dataset's images during training will look artificially strong on it. Fresh, real-world images make the test harder to game and closer to what a production system will actually encounter.

The evaluation also separates two things that most OCR scoring conflates: whether the model finds the right location in the image (grounding, usually scored by how well a predicted bounding box overlaps the true one) and whether the extracted text is accurate. A model can nail the text and still be graded down if it can't point to where that text came from, which turns out to be a common and telling failure.

Why receipts and rotated signs break state-of-the-art models

The paper tested nine prominent LMMs, spanning both proprietary models (the GPT-4o and Gemini families) and open-source ones, and the results split cleanly into three recurring weak spots.

Text grounding is worse than raw reading accuracy suggests. A model can transcribe the text on a sign correctly while being unable to draw an accurate box around where that text sits in the image. This sounds like a minor technical gap, but it's the difference between "the model read the invoice total" and "the model can tell your downstream system exactly where the invoice total is, so you can highlight it, redact it, or cross-check it against a template." For any pipeline that does more than transcription, that gap is the whole ballgame.

Multi-orientation text is a consistent blind spot. Rotated, vertical, or otherwise non-horizontal text trips up models that handle standard left-to-right horizontal text just fine. This lines up with something anyone who's dealt with scanned documents already knows: real-world scans are never perfectly aligned. A receipt photographed at an angle, a form scanned upside down, a vertically-set Japanese sign, these aren't edge cases in production, they're Tuesday.

Hallucination via repetition is the most unusual finding. On harder or more visually cluttered images, some models get caught in a loop, repeating a word, phrase, or line over and over rather than producing a coherent, bounded output. This isn't a case of the model getting the wrong answer; it's the model getting stuck in a degenerate output pattern entirely. It's a known failure mode in language generation more broadly (models sometimes fall into repetition loops when uncertain what should come next), but seeing it show up specifically on OCR tasks, where you'd expect the "correct" output to be tightly constrained by what's literally on the page, is a useful warning. Confidence-based filtering won't necessarily catch this, because the model isn't expressing uncertainty in a way a naive downstream check would notice. It's just repeating itself with total conviction.

None of the nine models dominates across all four tracks. A model that's strong on English scene text can be mediocre on multilingual documents; a model that parses documents into clean Markdown can struggle with key information extraction on the exact same source images. That's the practical headline: "best OCR model" isn't a single, transferable title. It depends heavily on what your documents actually look like.

Why it matters if you're building on OCR

If you're choosing an OCR engine, or building a product that leans on one, three things from this paper should change how you evaluate options.

First, test on your actual documents, not a vendor's demo examples. CC-OCR's own contribution is partly a rebuke of narrow benchmarking: a model that scores well on clean, horizontal, English-only scene text tells you almost nothing about how it will handle a rotated, multilingual, table-heavy invoice from an actual customer. If your product touches receipts, foreign-language documents, or handwritten forms, benchmark against those specifically, because general leaderboard rank won't predict it.

Second, separate "did it read the text" from "can it tell me where the text is." If your use case involves highlighting extracted fields, building structured data from documents, or doing any kind of human review workflow, grounding accuracy is not a nice-to-have, it's core functionality, and it's exactly the dimension CC-OCR shows models are weakest on relative to raw text accuracy.

Third, watch for repetition loops in production, not just wrong answers. A model that hallucinates a plausible-but-incorrect total is a known, manageable risk; you can flag low-confidence extractions and add a review step. A model that gets stuck repeating a phrase until it hits a token limit is a different failure entirely, one that can silently eat your API budget on a single document and produce output that's obviously broken but easy to miss in an automated pipeline if you're not specifically checking for it. Building a length or repetition check into your post-processing step isn't overkill, it's a direct response to a documented failure mode.

This is also the practical argument for testing more than one engine against the same document rather than betting everything on a single model. Since no model tested in CC-OCR dominated across all four tracks (multi-scene reading, multilingual reading, document parsing, key information extraction), the safest bet for a production system is treating engine choice as document-type-dependent rather than a one-time decision, and having a fallback path when your primary engine's specific weaknesses show up on a given input.

What the paper doesn't settle

CC-OCR is a snapshot of nine models as of late 2024, and the field moves fast: newer multimodal models have shipped since, and some may have specifically addressed the grounding and repetition issues this paper surfaces. The benchmark also doesn't dig deeply into why repetition hallucination happens, it documents the pattern without offering a fix, which leaves practitioners to build their own detection and mitigation logic rather than relying on a documented cure.

It's also worth remembering that a benchmark, however comprehensive, is still a fixed set of 7,058 images. Real documents keep surprising you in ways no fixed test set fully captures: a new invoice template, a language combination nobody thought to include, a scanner artifact unique to one client's hardware. CC-OCR is a genuinely useful map of where current models tend to struggle. It's not a substitute for testing against your own documents before you ship.

That's really the throughline of the whole paper: general-purpose "the model can read images" capability and reliable, production-grade OCR are related but distinct things, and the gap between them shows up in specific, testable ways, orientation, grounding, and runaway repetition, rather than a single dial you can crank up.