Microsoft's KOSMOS-2.5: One OCR Model, Two Very Different Jobs

Picture two people staring at the same scanned invoice. One is a court reporter, transcribing every word and noting exactly where on the page it sits, down to the pixel. The other is an editor, rebuilding the document from scratch as clean markdown: this is a table, this is a header, this is a bolded total. Same input, completely different output shape, completely different downstream use.

Most OCR systems pick one of those jobs and specialize. Microsoft's KOSMOS-2.5, described in a September 2023 paper from the company's research division, is trained to do both from the same 1.3-billion-parameter model, switching between them with nothing more than a different prompt. That's the whole pitch, and it's a more interesting pitch than it sounds at first, because those two tasks usually pull a model in opposite directions.

U.S. IRS Form 1040 individual income tax return, 1941; public domain (U.S. Government work), via Wikimedia Commons.

The two jobs OCR systems don't usually share

Traditional OCR gives you text plus coordinates: "the word 'Total' sits at pixel (412, 88)." That's what you need if you're building a searchable PDF, redacting sensitive fields, or feeding a downstream layout parser. It's precise and spatial, but it doesn't understand structure. It doesn't know that a block of text is a table row versus a caption versus a footnote.

Image-to-markdown generation is the opposite instinct. You feed in a screenshot of a document and get back ## Section Header, a | Column | Column | table, a **bold** total line. It captures structure and style, which is exactly what you want if you're piping a document into an LLM, a search index, or a wiki. But it usually throws away exact pixel positions, because markdown doesn't have a coordinate system.

KOSMOS-2.5 treats these as two flavors of the same underlying skill, "machine reading of text-intensive images," and trains one model to do either, selected by prompt. Ask it for OCR and you get spatially grounded text blocks. Ask it for markdown and you get structured, styled output. Under the hood it's the same weights, the same shared decoder, just steered by which task token you feed it.

How a language model learns to see a page

KOSMOS-2.5 is built on a familiar recipe from the vision-language model playbook: a vision encoder looks at the image, a resampler compresses that into a manageable number of tokens, and a decoder-only autoregressive transformer generates output one token at a time, the same way GPT models generate text. The twist is what it's asked to generate. Instead of just captioning "a photo of a document," it's trained to output either coordinate-tagged text spans or markdown syntax, depending on the prompt it's given.

Autoregressive here just means the model predicts the next token based on everything it has generated so far, the standard approach for both language and, increasingly, structured document output. What makes this different from a plain OCR pipeline is that the model isn't running a separate detection step (find text regions) and then a separate recognition step (read what's in each region). It's doing both in one continuous generation pass, which is closer to how a large language model writes a paragraph than how a classic OCR engine scans a page.

The training diet is the part that actually explains the performance: 357.4 million document pages, pulled from a mix of sources spanning scanned documents, web-rendered pages, and other text-dense image types. That's a genuinely large corpus for this kind of specialized pretraining, and it's the reason the model can handle document types well beyond clean printed pages. Scale of training data, more than architectural cleverness, tends to be the boring but true answer for why these models generalize.

Grading the model: two new benchmarks nobody had before

One underappreciated contribution of this paper is procedural rather than architectural: the authors built two new benchmarks, OCREval and MarkdownEval, specifically because existing OCR benchmarks didn't test what they cared about. If you're claiming a model does spatial OCR and markdown generation well, you need evaluation sets that actually measure both, and apparently those didn't exist at a useful scale before this.

On those benchmarks, KOSMOS-2.5 performs comparably to GPT-4o, OpenAI's flagship multimodal model, on document-level text recognition and image-to-markdown generation. That's a notable claim for a model roughly two orders of magnitude smaller than what GPT-4o is believed to be. It doesn't mean KOSMOS-2.5 beats GPT-4o across the board, and the paper isn't claiming that. It means that on the narrow, well-defined tasks of reading document images and converting them to structured text, a much smaller purpose-built model closes most of the gap with a general-purpose giant.

That's a pattern worth internalizing if you're choosing OCR tooling: general-purpose multimodal models are impressive generalists, but a smaller model trained specifically on hundreds of millions of document pages can match them on the document-reading task specifically, at a fraction of the compute cost per call.

The chat version: same brain, fine-tuned for questions

The paper doesn't stop at raw transcription. The authors fine-tune KOSMOS-2.5 into a variant called KOSMOS-2.5-CHAT, aimed at document understanding rather than pure transcription, meaning it can answer questions about a document image instead of just reading it out.

Evaluated across nine text-rich visual question-answering benchmarks, things like "what's the total on this receipt" or "who signed this form," KOSMOS-2.5-CHAT performs comparably to other state-of-the-art generalist models that are roughly five times larger, 1.3 billion parameters against 7 billion. That's the paper's headline efficiency claim, and it lines up with the broader theme: specialization on document data buys you a lot of the capability that raw parameter count would otherwise have to provide.

For anyone building a product on top of a model like this, that efficiency claim matters more than the raw accuracy number. A 1.3B model is dramatically cheaper to serve, faster to run, and easier to fine-tune further than a 7B one. If it gets you 90% of the way to the bigger model's document-QA performance, that's a real infrastructure decision, not just a research footnote.

Where this actually breaks: the honest limitations

The paper is upfront that this isn't a solved problem. A few things worth flagging for anyone evaluating whether to build on this kind of model:

• The 357.4-million-page training corpus, while large, is still a curated slice of document types. Document layouts the model hasn't seen much of during training, unusual scripts, heavily stylized fonts, extremely dense tables, handwriting, will predictably degrade both the spatial OCR output and the markdown reconstruction.
• Markdown generation involves a judgment call about structure. Two humans looking at a complex layout (nested tables, multi-column academic papers, forms with irregular field placement) might disagree about the "correct" markdown representation. That ambiguity makes MarkdownEval scores harder to interpret than a straightforward character-error-rate metric.
• Comparable to GPT-4o on these specific benchmarks doesn't mean comparable on everything. GPT-4o handles a much broader range of reasoning tasks around an image; KOSMOS-2.5 is purpose-built for reading and structuring text-dense images specifically.
• As with most single-model benchmarks, real-world performance depends heavily on how close your documents are to the training distribution. A receipt scanned on a phone in bad lighting is a different animal than a clean PDF export, and the paper's benchmark numbers won't tell you which one you're dealing with.

Why it matters for anyone building document pipelines

If you're building a product that has to turn document images into usable text, KOSMOS-2.5 is a useful data point on two fronts.

First, it's evidence that spatial OCR and structured markdown extraction don't have to be two separate systems bolted together. A single model, trained the right way, can serve both needs, which simplifies the engineering on your end: one model call instead of an OCR pass plus a separate layout-reconstruction pass.

Second, and more practically, it reinforces something worth remembering when you're comparing OCR engines: bigger isn't always the deciding factor. A 1.3B-parameter model trained on hundreds of millions of document pages closed most of the gap with GPT-4o on document reading, and matched 7B generalist models on document Q&A. If your use case is specifically "extract structured text from documents," a model built for exactly that job can outperform a much larger, more general model per dollar of compute, which is exactly the kind of tradeoff worth testing rather than assuming.

The practical takeaway isn't "use KOSMOS-2.5 specifically." It's that when you're picking an OCR or document-parsing engine, the question to ask isn't just how big the model is or how many benchmarks it tops. It's whether the model was actually trained on documents that look like yours, and whether it outputs the shape of data your pipeline needs, coordinates, markdown, or something else entirely, without forcing you to bolt a second system on top to get there.