The OCR Benchmark Built to Make GPT-5 Fail

Take a screenshot of a scam ad you've probably seen: a "customer service" phone number rendered in a wobbly, semi-transparent font, laid over a photo of a smiling person, colors chosen so the text almost disappears into the background. You read it in half a second without trying. Feed that same image to GPT-4o and ask it to transcribe the number, and there's a good chance it gets it wrong, or can't find the text at all.

That gap, between what a human eye does automatically and what a state-of-the-art vision model can't reliably do, is the subject of a new paper called ArmorOCR. The researchers built a benchmark of images specifically designed to be readable by people and hard for models, tested a stack of current large multimodal models (LMMs) against it, and found some genuinely rough numbers: GPT-4o scored 23.2% on their adversarial OCR test. GPT-5 did better but still landed at 30%. Even Gemini 2.5 Flash, decent at general OCR, managed 32.3%. Then they trained a model specifically for this problem and got it to 55.7%, more than double GPT-4o's score, using a base model a fraction of the size of these frontier systems.

What "grounded" adversarial OCR actually means

Most OCR benchmarks test recognition: here's an image, what does the text say? That's already hard when text is stylized or degraded, but it's only half the job. A model can sometimes guess the right answer without actually finding the text, by leaning on context. If a sign in a photo of a coffee shop probably says something coffee-related, a model might produce a plausible answer while pointing at the wrong pixels entirely, or no pixels at all.

The paper calls this out directly: answer accuracy alone doesn't prove a model can actually locate text. So they frame the task as "grounded OCR perception," meaning a model has to do two things together: find the region where the text lives (localization, usually a bounding box) and read what's inside it (recognition). Get the box wrong and it doesn't matter if the guessed word happens to be correct. This is the harder, more honest version of the task, and it's the one that matters for real applications like redacting sensitive fields on a scanned form or flagging where a piece of manipulated text sits in an image.

"Adversarial" here doesn't mean someone is running a formal attack algorithm against the model's weights. It means the text itself is visually hostile: warped, hidden in a busy background, drawn with dots instead of strokes, blended into an AI-generated image so seamlessly that the boundary between "text" and "picture" barely exists.

Inside AdvSpot: 390 images built to trip up a model

The benchmark, called AdvSpot, is small by modern dataset standards: 390 images. But every one is annotated at the region level, and each was chosen to represent one of 13 specific ways text can dodge a model's eye, grouped into 5 categories.

Spatial manipulation covers text that's been physically twisted: rotated at odd angles, mirrored, or shrunk down to a handful of pixels. This is the kind of thing a person barely notices when reading a tilted street sign, but it throws off models trained mostly on horizontal, left-to-right text blocks.

Glyph variation is about the letterforms themselves. Stylized glyphs split, occlude, or truncate strokes, think of a logo where half a letter is hidden behind another shape but your brain fills in the gap anyway. Handwritten text, with its irregular spacing and personal quirks, falls in this category too.

Image degradation is the everyday stuff: motion blur, glare off a phone camera, film grain, an overexposed photo, compression artifacts from being screenshotted and re-shared a dozen times. Nothing exotic here, just the reality of images that weren't shot in a studio.

Contextual blending is where things get genuinely interesting. It includes low-contrast text (same color family as the background, so it's technically visible but visually camouflaged), pattern overlays (dots or lines drawn over the text like a watermark), and what the authors call AIGC fusion text, text baked directly into an AI-generated image so it looks like a natural part of the scene rather than a caption slapped on top. This last category turned out to be the single hardest thing in the whole benchmark. Before training, most models scored close to zero on it.

Visual encoding is the most abstract group: characters represented through symbol arrangements, dot density, or line orientation rather than conventional strokes. This is closer to a puzzle than to reading, and it's the other category where baseline models basically flatlined.

How they actually trained a model to get better at this

ArmorOCR's training runs in two stages, and the logic behind each one is worth walking through, because the first stage in particular is a clever workaround for a problem that shows up constantly in machine learning: you can't cheaply produce ground-truth for the hard version of a task, but you might already have signal from an easier version of it.

The first stage is called On-Policy Self-Distillation. Here's the trick: for each adversarial training image, the researchers also generate several transformed versions, resized, rotated, flipped, or otherwise cleaned up, and a separate judge model picks whichever transformed version is easiest to read correctly. That easier view becomes a kind of privileged hint. During training, the student model only ever sees the original hard image and has to produce its own answer on its own, current policy (that's the "on-policy" part: it's grading itself, not a frozen snapshot from earlier in training). A teacher process then compares the student's answer against what it could deduce from the easier transformed view plus the real ground-truth text, and nudges the student's internal probabilities toward the better answer. The nudging uses two different penalty types depending on which part of the output is being corrected: a softer one for the model's reasoning steps, a firmer one for the actual predicted text. In plain terms, the model is taught to read a distorted photo by comparing what it says against what it should have said if it had seen a slightly less distorted version of the same photo.

The second stage is reinforcement learning through Group Relative Policy Optimization, or GRPO, a technique that's become popular for fine-tuning language and vision-language models because it doesn't need a separate reward model, it compares a group of the model's own sampled outputs against each other and reinforces whichever ones scored best. ArmorOCR applies this with four distinct reward signals tuned to four sub-tasks: how well predicted boxes overlap ground-truth boxes (for pure localization), how close predicted text is to the correct string using edit distance (for pure recognition), a combined score for doing both at once (full spotting), and a simple correctness check for visual question answering about the text. Training this way, rather than optimizing one blended metric, forces the model to actually get good at localization and recognition as separate skills instead of finding a shortcut that only satisfies one blended score.

All of this ran on top of Qwen3-VL-8B-Instruct, a mid-sized open model, using around 50,000 synthesized training examples in stage one and 70,000 harder cases in stage two.

The numbers, and where the gains actually show up

The headline result is that jump from GPT-4o's 23.2% to ArmorOCR's 55.7% on AdvSpot's VQA task, and a similar pattern on localization, where ArmorOCR hit 63.3% IoU (a standard measure of box overlap) against 47.5% for the best baseline, Qwen3-VL-235B, a model roughly 30 times larger.

The gains aren't evenly spread, which is the more useful thing to know if you're trying to judge whether this generalizes. On AIGC fusion text, the category where text is dissolved into an AI-generated background, ArmorOCR beat the strongest competing model by 65%. That's the category the paper's own error analysis flagged as nearly unsolvable beforehand, so a large jump there is meaningful evidence the two-stage training approach is doing real work rather than just memorizing benchmark quirks. Symbol-based and dot-pattern encoding remained the toughest nut even after training, the paper is upfront that these stay "highly challenging."

On other adversarial OCR benchmarks the researchers didn't design themselves, AdvOCR and SmuggleBench, ArmorOCR edged out specialized prior methods but by much smaller margins (56.0% versus 55.0%, and 17.1% versus 16.4%). That's worth sitting with: the biggest gains show up on the benchmark the authors built and tuned their training data toward, and shrink on benchmarks built by other people. That's not a red flag by itself, most specialized training does best on its home turf, but it's a reason to treat the 55.7% headline number as a ceiling for this specific kind of adversarial text rather than a universal adversarial-OCR score.

The other number that matters is what didn't change: general OCR performance on standard benchmarks like OCRBench stayed roughly flat, with no extra general-purpose training data added. Specializing a model for a narrow, hard problem without wrecking its everyday performance is the less flashy but arguably more important result here. A lot of fine-tuning work trades general competence for a spike on one benchmark, and this one didn't.

Why it matters for anyone building or choosing OCR

If you're picking an OCR engine or an LMM to handle document pipelines, receipts, ID verification, or content moderation, this paper is a useful reality check on a blind spot most benchmarks don't cover. Standard OCR leaderboards mostly test clean documents and reasonably well-photographed natural scenes. None of that tells you what happens when a user uploads a glare-heavy phone photo of a receipt, a screenshot with a semi-transparent watermark over the numbers, or an AI-generated image with a phone number melted into the wallpaper. Those aren't edge cases anymore. Glare and blur happen on every mobile upload. Watermarked and AI-blended text shows up constantly in scam content, spoofed documents, and moderation-evasion attempts, which is exactly the kind of thing a trust-and-safety pipeline needs to catch.

The practical takeaway isn't "switch to ArmorOCR," since it's a research checkpoint built on an 8-billion-parameter open model, not a hardened production API. It's that model size and general OCR benchmark rank are weak predictors of how a system handles genuinely adversarial input. A smaller model trained specifically for grounding and adversarial robustness beat models many times its size on this task. If your use case touches fraud detection, content moderation, or any pipeline where someone has an incentive to make text hard for a machine to read, general-purpose OCR scores won't tell you what you need to know. You'd want to test against something closer to AdvSpot's categories: warped and rotated text, low-contrast overlays, glare, and text blended into generated imagery, before trusting a model's numbers on a clean benchmark.

What's still shaky here

A 390-image benchmark is small enough that a handful of unusual examples could swing category-level scores meaningfully, and the paper doesn't report variance across multiple runs, so treat individual category numbers as directional rather than precise. The training and evaluation data are synthesized rather than pulled from real-world scam or fraud content, which is reasonable for building a controlled benchmark but means real adversarial text in the wild, created by people actively trying to evade a specific model, may look different from what AdvSpot captures. And the categories where ArmorOCR still struggles, symbol and dot-pattern encoding, are arguably the most deliberately adversarial ones in the set, closer to a visual cipher than to distorted natural text. Solving the easier 80% of adversarial OCR while the hardest 20% stays near zero is real progress, but it's not the same as solving adversarial OCR.

Still, the core contribution holds up: a benchmark that actually measures whether a model can find hard-to-read text, not just guess at it, and a training recipe that meaningfully closes the gap without wrecking everyday performance. That combination, a harder eval plus a fix that doesn't cost you elsewhere, is rarer in this field than it should be.