An AI Read a 2,000-Year-Old Seal and Just Made Up a Poem

A seal, a proverb, and a model that made up its own poem

Here's the setup. Researchers show a vision-language model a photo of a 2,000-year-old Chinese seal. The carved text is a real classical proverb, something like "no worldly flavor outlasts the bitterness of a root vegetable" (世間滋味賞來無過菜根甜). One of the strongest OCR-specialist models in the test read it as "美酒歸來無人問和賞," a different string of characters that also happens to sound like a plausible line of classical poetry. A general-purpose model did something stranger: it locked onto a five-character fragment, "賞來無人花," and just kept repeating it, spinning out a loop that reads fluently and says nothing the seal actually says.

Neither model paused. Neither flagged the passage as unclear. Both handed back confident, grammatical, historically-plausible-sounding Chinese, and both were wrong. That's the core finding of Ancient-Bench, a new benchmark out of South China University of Technology and Huawei that tests how well today's AI models can read text carved, cast, brushed, and printed on actual historical Chinese artifacts. And it's a useful case study for anyone who assumes "AI can read old documents now" is a settled question.

Annotated Oracle Bone Inscription - Heji 29990 and 30174 - Rain Divination — by Quinlan718 via wikimedia, CC0 1.0.

Why this isn't really one OCR problem

Ask someone to build "Chinese OCR" and they'll probably picture a printed page: consistent font, dark ink, clean background. That's the easy end of this benchmark. The hard end is a set of artifacts that don't share a writing system in any way a modern reader would recognize.

Chinese script has gone through roughly seven major forms over its history, and Ancient-Bench covers all of them: oracle bone script (carved into turtle shells and animal bones by Shang dynasty diviners, over 3,000 years ago), bronze script (cast into ritual vessels), seal script, clerical script, regular script, cursive script, and running script. Each is a genuinely different set of character shapes, not just a font change. A character that's a simple, well-known glyph in regular script might be an abstract pictograph in oracle bone script that only a handful of specialists can read.

Layer onto that nine different physical media, because the writing surface changes what "text" even looks like: oracle bones and bronzes (short, dense inscriptions, often chipped or corroded), bamboo and wooden slips (thin strips bound into books, prone to warping and fading), silk manuscripts (fragile fabric, frequently damaged), seals (text carved in mirror image, sometimes in relief), stone steles and cliff inscriptions (rubbings taken from weathered rock, with cracks and erosion baked into the image), woodblock-printed books, and ink calligraphy. Read a Han dynasty land contract on a bamboo slip and a Ming dynasty cursive poem on a hanging scroll, and you're really looking at two different recognition problems that happen to use the same underlying language, three-plus millennia apart.

Existing benchmarks, the paper argues, mostly pick one of these axes and stop. Some cover several script types but only in printed books. Some cover oracle bones in depth but nothing else. One earlier attempt at a multi-medium set, MCHDoc, tried to stitch several media together, but it pulled from existing single-medium datasets with no shared labeling rules, so the numbers weren't really comparable across categories. Ancient-Bench's pitch is to build one dataset that's honest about all three dimensions at once: medium, script, and time period.

Deciding what "correct" even means

Before you can grade a model on ancient text recognition, you have to decide what a correct transcription looks like, and that turns out to be a real design problem, not a formality.

Take a character carved on an oracle bone. It might not exist in modern Unicode at all. Scholars have a standard practice called transcription (隶定): converting an archaic character form into its equivalent in a later, readable script, the way you might explain that an old handwritten symbol "is" a modern letter even though it doesn't look like one. Ancient-Bench's annotators had to decide, character by character, whether a given archaic form has an agreed modern equivalent, whether it should be flagged with an "unrecognizable" placeholder because no equivalent exists, or whether the source institution's own museum label (which often simplifies aggressively for a general audience) can be trusted at all. The team found that museum annotations frequently normalize variant or traditional characters into modern simplified forms for readability, which quietly erases distinctions that actually matter for historical research: two characters that were later merged into one simplified form, for instance, carried different meanings in the original texts.

The paper settles this with three annotation standards. Symbol standardization defines how to mark repeated characters, damaged or illegible passages, and characters that exist in the image but can't be typed in any character set. Character standardization sets the rules for handling those archaic and variant forms, favoring what-you-see-is-what-you-get over silent simplification. Parsing standardization covers layout: reading order (vertical Chinese text runs top to bottom, right to left; some artifacts run differently), line breaks, interlinear notes, and which regions of a printed book page are actual text versus margin notes or binding artifacts that should be ignored.

It's not a glamorous part of the paper, but it's arguably the most useful contribution: a shared, checkable definition of "ground truth" for a category of documents where reasonable experts can otherwise disagree about the transcription. Twenty trained annotators worked on this for six months, cross-checking against four established paleographic reference databases, with a separate reviewer verifying the final labels.

The numbers: nobody breaks 60%, and some tasks barely function

The final dataset holds 2,700 images, sourced from 14 national-level museums and libraries, split across the nine media types (400 calligraphy pieces down to 200 cliff inscriptions) and covering roughly 1200 BCE to 1911 CE. The researchers then ran 33 models against it: closed models like GPT-5, Gemini, and Claude; open models like Qwen and Kimi; and a set of specialist document-OCR systems built specifically for text extraction rather than general conversation. Each model's output got scored against the standardized transcriptions using two metrics: F1 (how many predicted characters match the ground truth, penalizing both missed and invented characters) and normalized edit distance, a character-level measure of how many single-character edits it would take to turn the model's guess into the correct answer.

The best model overall, ByteDance's Doubao-seed-2-0-lite, hit 52.08% edit distance and 57.86% F1. Read that plainly: the strongest system tested gets, on average, well under three out of five characters right when you ask it to read an ancient Chinese artifact. Every other model does worse. The best dedicated OCR specialist, HunyuanOCR, tops out at 46.11% F1, meaningfully behind general-purpose chat models like Doubao, Gemini, and the open-weight Kimi K2.6 (56.62% F1). That's a real reversal of what you'd expect: models built and marketed specifically for document parsing lose to general-purpose assistants on this task.

Performance also splits hard by artifact type, and the split tells you something useful. Oracle bone script is close to unsolved: even Gemini, the best performer on that specific task, manages only 15.75% F1. Bronze inscriptions aren't much better, at 38.48% for the top model. Both categories involve dense, ancient glyph forms with no modern equivalent that a model can pattern-match against. Seals are their own trap: because seal inscriptions commonly quote classical poetry, models default to their training-data priors and produce fluent-sounding poetry that isn't what's actually carved, which is exactly the seal example from the opening. Meanwhile stone steles, which tend to preserve well-formed clerical or regular script in decent condition, hit 88.88% F1 for the top model, closer to a genuinely usable result.

The gap between those numbers (under 16% on oracle bones, near 89% on steles) is the real finding here, more than any single leaderboard score. "Ancient Chinese OCR" isn't one capability a model has or lacks. It's dozens of narrower capabilities, and current systems have some of them and not others.

The hallucination problem is a UX problem, not just an accuracy problem

The paper builds a separate calligraphy subset specifically to study hallucination, and it identifies two distinct failure patterns worth knowing about if you're evaluating any OCR system on messy input, not just ancient Chinese text.

The first is what the authors call prior-driven semantic completion: when strokes are ambiguous or deformed (which is normal in cursive calligraphy), the model falls back on its language model half rather than its vision half, and produces text that sounds right instead of text that's actually there. This is the seal-poem failure from the opening, generalized. The second is cropping and detection-induced over-recognition: when the system's own region-detection step draws a slightly wrong box around the text, it picks up neighboring strokes or background noise and turns them into extra, invented characters, stitching them into a fluent but fabricated reading.

Both failure modes share a property that should worry anyone building a product on top of these models: the output looks exactly as confident, exactly as well-formatted, whether it's right or completely invented. There's no visible hedge, no lower confidence score surfaced to the user, nothing that distinguishes "I read this correctly" from "I'm pattern-matching to something plausible." For a casual user, that's the difference between a tool that's occasionally wrong and a tool that's occasionally wrong in a way you can't detect without independently knowing the answer already.

Why it matters if you're building or choosing OCR

If your documents look like the easy end of this benchmark (printed pages, modern handwriting, decent scans), none of this changes your calculus much; mainstream OCR and general VLMs already handle that reasonably well. But this paper is a clean, measured demonstration of where that confidence breaks down, and the pattern generalizes past ancient Chinese artifacts to any OCR use case with degraded, unusual, or out-of-distribution source material: old handwriting, faded receipts, damaged forms, non-Latin scripts with limited training data.

Three takeaways worth carrying into a real evaluation:

General-purpose VLMs currently beat specialist OCR models on genuinely hard, unfamiliar input. That's a bit uncomfortable if you've built a pipeline around a dedicated OCR model on the assumption that specialization wins, and it's worth re-testing that assumption on your actual documents rather than a marketing benchmark.

Confidence and correctness are not linked in these systems yet. On this benchmark, models produce fluent, wrong answers just as readily as fluent, right ones, especially when the input is ambiguous. If your product surfaces OCR output directly to a user without some form of verification or confidence signal, degraded or unusual documents are where that gap turns into a support ticket, or worse.

Aggregate accuracy numbers hide category collapse. A model claiming "OCR works" needs its own version of this paper's per-medium breakdown before you trust it on your specific document type, because a strong average can be built on top of one category that's near-solved and another that's barely functioning.

What the paper doesn't claim

The authors are upfront about two limits. First, Ancient-Bench annotates text at the line or region level, not with character-level bounding boxes, so it's good for scoring end-to-end recognition but not for evaluating fine-grained text detection. They've flagged that as a planned addition. Second, and more interesting: since the source images come from public museum and library archives, some of this material may already be sitting inside the training data of the very models being tested. That's a familiar problem for any benchmark built from public sources, and it cuts in one direction, if anything nudging the reported scores upward rather than making the results look artificially harder. Given that even the best models still fail badly on oracle bones and bronze inscriptions, that caveat doesn't do much to soften the paper's main point: reading 3,000 years of Chinese artifact text is still, for now, a problem current AI hasn't solved.