GOT-OCR2.0: A 580M-Parameter Model That Reads Text, Formulas, Sheet Music, and Charts

Feed a photo of a handwritten chemistry worksheet into most OCR tools and you'll get a wall of garbled Latin characters where the molecular structures should be. Feed it a page of sheet music and you'll get nothing at all, because sheet music isn't "text" as far as the software is concerned. That's the gap a group of researchers from StepFun and the University of Chinese Academy of Sciences set out to close with a model called GOT, short for General OCR Theory. Their claim: one 580-million-parameter model can read plain text, math and chemical formulas, tables, charts, sheet music, and even geometric shapes, and output each one in a format you can actually use, markdown, LaTeX, SMILES notation, whatever fits.

That's a wide net for something smaller than a lot of open-weight chatbots. Worth digging into how, and whether it holds up.

Figure from General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model (Wei et al., arXiv:2409.01704), licensed CC BY-SA 4.0.

Why "just run OCR on it" stopped being good enough

The OCR most of us grew up with, think Tesseract, or the scanner software bundled with an office printer, works by breaking a document into stages: detect where the text lives, crop those regions, recognize the characters in each crop, then stitch the results back into reading order. The paper calls this OCR-1.0, and it's been good enough for typed English text on a white background for decades.

It falls apart the moment "text" stops meaning plain sentences. A table isn't just text, it's text with row and column relationships that get lost the second you flatten it into a stream of characters. A chemical structure diagram encodes bonds and ring shapes that have no character-level representation at all. Sheet music is symbols with pitch and duration, arranged in a way no line-by-line reading order can capture. Multi-stage pipelines built for line-by-line English don't bend well to this. Every content type has ended up with its own bespoke recognition system, so a company that needs to digitize invoices, textbooks, and scientific papers is running three separate specialized systems, each with its own bugs, latency, and integration cost.

The more recent alternative is to bolt OCR onto a large multimodal language model like GPT-4V, treat the whole thing as "describe what's in this image." That works, in a general way, but general-purpose vision-language models weren't trained with OCR precision as the target, so they tend to hallucinate on dense text, choke on very long documents, and cost far more compute per page than a purpose-built recognizer needs to.

GOT's pitch is a middle path: purpose-built for reading, but unified across content types, and end-to-end rather than a chain of separately-trained modules.

How GOT actually works: a small eye and a bigger reader

Architecturally GOT is refreshingly plain: an image encoder feeds a text decoder. No detection stage, no separate recognition stage, no downstream parser stitching things together. One forward pass, image in, text out.

The encoder is the more interesting half. It's a roughly 80-million-parameter vision transformer, initialized from Meta's Segment Anything Model, that takes a 1024x1024 image and compresses it down to just 256 tokens before handing them to the decoder. That's a high compression ratio for the density of information on, say, a full page of a scanned book. The bet the authors are making is that a document image, however visually complex, doesn't need billions of pixels' worth of representation once you've picked out the shapes that matter for reading it back. Contrast that with how far behind a typical scene-understanding vision encoder would fall, trying to compress a photo of a busy street into that few tokens without losing the objects in it. Documents apparently compress more cleanly than street scenes.

The decoder is a roughly 500-million-parameter language model built on Qwen, given a long context window so it can hold an entire page of extracted text without losing track of the first paragraph by the time it reaches the last.

Training happens in three stages. First the encoder is pretrained alongside a small, throwaway decoder just to teach it to see text-shaped things. Then the real decoder gets bolted on and the two are trained together on a broad mix of documents. Finally the whole system gets fine-tuned per task, using different prompts to switch behavior, plain text extraction versus markdown-formatted tables versus LaTeX for formulas versus SMILES strings for molecules. The prompt is effectively the mode switch: same weights, different instructions, different output format.

Figure from General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model (Haoran Wei et al., arXiv:2409.01704), licensed CC BY-SA 4.0.

Reading a receipt versus reading a chemistry textbook

One detail I like: GOT supports what the authors call interactive OCR. You can point at a region by coordinates, or by color, and ask the model to read just that part. That sounds like a small feature, but it's the difference between "extract everything and hope the layout parser gets the boundaries right" and "tell the model exactly what you want and get only that back." Anyone who's tried to pull a single line item out of a cluttered invoice knows how often layout-based extraction points at the wrong box.

The model also handles documents that don't fit in one glance. Dynamic resolution kicks in for oversized images, posters, engineering drawings, anything too big to downsample without losing legibility, by tiling the image and processing pieces separately before merging results. And multi-page OCR lets you feed in several pages at once and get one coherent output back, rather than gluing together per-page outputs yourself and hoping page breaks don't split a sentence in half.

None of this required exotic new theory. It required training data nobody had bothered to build: synthetic documents rendered from LaTeX and HTML for math and tables, scene-text images for street signs and product labels, and generated sheet music and molecular diagrams paired with their symbolic ground truth. Building a matched training set for chemistry diagrams and sheet music is unglamorous work, and it's plausibly the actual unlock here more than any single architectural trick.

What the results say, and where to stay skeptical

On standard OCR benchmarks the paper reports GOT beating larger general-purpose vision-language models, including systems with several billion parameters, despite GOT weighing in at 580 million. That's a real result worth taking seriously: a specialist model with a fraction of the parameters outperforming generalist heavyweights on the specific job of reading documents accurately. It lines up with something the OCR field has learned repeatedly: a model trained end-to-end for a narrow, well-defined task tends to beat a much bigger model that's good at everything except that task.

On formatted outputs, tables kept as markdown, formulas kept as LaTeX, molecules kept as SMILES, the paper reports GOT holding up well against dedicated single-purpose tools built for exactly one of those tasks. Handling all of them from one model, without a router deciding which specialist to call, is the headline claim, and the benchmark numbers back it up for the cases tested.

What the paper is honest about, and what's worth flagging for anyone considering this in production: benchmark performance on curated test sets doesn't guarantee it on the genuinely messy documents that show up in the real world, coffee-stained receipts, faxed forms, photos taken at an angle under bad lighting. The paper's own scene-text results are weaker than its document-image results, which tracks with a pattern seen across the OCR field generally: clean, well-lit document scans are the easy case, and photos of text in the physical world are the hard case that keeps improving slowly, year over year. And a model built around a fixed set of output formats, markdown, LaTeX, SMILES, kern notation for music, will need retraining or fine-tuning to add a new content type. It's unified, not infinitely extensible out of the box.

Why it matters if you're building or buying OCR

If you're evaluating OCR engines for a real product, the practical takeaway isn't "GOT is the best engine," it's what GOT demonstrates about where the field is heading: purpose-built, end-to-end models can match or beat much larger general-purpose systems on document understanding, at a fraction of the compute cost. That matters directly for cost-per-page and latency, both real line items if you're processing thousands of documents a day.

It also validates something worth remembering when comparing engines: "OCR" isn't one task. A tool that's excellent at typed English invoices can be mediocre at handwritten forms, tables, or scientific notation, and the gap often only shows up once you test the specific document type you actually need. GOT's unified approach to text, tables, formulas, and specialized notation is a useful reminder to test any OCR engine, this one included, against the actual documents your business handles, not just a generic benchmark. A model that tops a leaderboard on printed text can still fall over on your particular receipts or your particular scanned contracts.

And practically: if your pipeline needs to handle documents with mixed content, plain text plus tables plus the occasional formula, a single unified model like GOT is architecturally simpler to integrate than stitching together a text OCR engine, a table extractor, and a formula recognizer with three different APIs and three different failure modes. Fewer moving parts usually means fewer places for a production pipeline to break.