Your Chatbot Can't Find the Chart That Has the Answer. This Paper Tries to Fix That.

Picture an investor-relations chatbot that has ingested a 40-page quarterly deck. Someone asks it: "What was our churn rate last quarter?" The number is real, and it's on page 23. But it's not written anywhere as a sentence. It's the height of one bar in a chart, next to three other bars, with a small percentage label rendered inside the graphic itself.

A retrieval system that works by reading extracted text will sail right past that page. There's no sentence to match against the question. This is the exact problem a new paper called ConceptFormer, from a team including researchers at Northeastern University and Tsinghua University, is trying to solve, and the way they solve it is more interesting than the usual "bigger model, more data" story.

The page with your answer doesn't always have the words for it

Before you can get an AI system to answer a question about a document, you need to find the right page. This step is called retrieval, and it's the backbone of retrieval-augmented generation (RAG): find the relevant evidence first, then hand it to a language model to actually answer the question. Get retrieval wrong and it doesn't matter how good the model reading the page is. It never sees the page.

For years, the standard way to search documents was to run OCR (optical character recognition) to pull out the text, then search that text the way you'd search any other corpus. This works fine for a plain-text contract. It falls apart for a slide deck, an annual report, or a scanned form, because OCR throws away everything that isn't text: the shape of a chart, the fact that a number sits in the top-right cell of a table, the visual emphasis of a bolded total. A lot of the actual meaning in a business document lives in that structure, not in the words.

The newer approach, which the paper calls visual retrieval, skips OCR entirely and has a vision-language model (a model that can process both images and text together, like GPT-4V or Qwen-VL) look at the page as an image and turn the whole thing into a vector, a long list of numbers that captures its meaning. The query gets turned into a vector too, and retrieval becomes a matter of finding which document vectors sit closest to the query vector in that shared space. This preserves layout and visual structure, which is a real improvement. But it creates a training problem: how do you teach the model which part of a busy page actually matters for a given question?

Recent attempts at that fine-grained training signal have leaned on two proxies. One is textual descriptions: have a strong model write out, in words, what the relevant evidence is. The other is localized visual regions: draw a bounding box around the part of the page that answers the question. Both have real flaws. A caption can't fully capture what a chart's shape or a table's layout is doing. A bounding box assumes the evidence lives in one clean rectangle, which breaks down the moment the answer depends on connecting a number in a chart to a caption two inches away, or a footnote at the bottom of the page.

Latent concepts: evidence without a caption or a box

ConceptFormer's core move is to stop forcing the evidence signal into either format. Instead of writing a caption or drawing a rectangle, it represents "what matters on this page for this query" as a set of continuous vectors it calls latent concepts.

Here's a way to think about it. A caption is a compressed description in English. A bounding box is a compressed description in pixel coordinates. Both formats throw away information in translation, because they're forcing something visual and relational into a fixed vocabulary. A latent concept skips the translation step. It's a vector that lives directly in the same mathematical space as the query and document embeddings, shaped by training to sit near whatever part of the document's representation is actually relevant. Nobody can read a latent concept out loud the way they can read a caption, but it doesn't need to lose the visual and semantic information the way a caption does either.

The other piece worth pulling out is dynamic sizing. During training, a strong vision-language model decides, for each query-document pair, how many concept tokens are needed to represent the relevant evidence. A simple lookup question (what's the total on this invoice) might need just one. A synthesis question that depends on both a chart and the paragraph explaining it might need several. That's a meaningful departure from feeding every example through a fixed-size representation regardless of how much evidence it actually requires.

These concepts only show up during training. They act as an intermediate target that guides the retriever's embedding space, pulling query and document representations together in a way that reflects the true evidence rather than a lossy proxy of it. At inference time, when someone actually types a question into a search box, the retriever just computes query and document embeddings and compares them directly. Nothing about the deployed system requires running a heavyweight vision-language model at query time, which matters a lot for anyone thinking about production cost.

The numbers: beating both camps at once

The results are reported as relative improvements in NDCG@10, a standard way to score how good a ranked list is. NDCG stands for normalized discounted cumulative gain. In plain terms: if the truly relevant page is ranked first, you get full credit; if it's ranked tenth, you get much less; if it's not in the top 10 at all, you get essentially none. "Normalized" means the score is scaled against the best possible ranking, so you get a number you can compare across different queries and datasets.

Across a set of visual document retrieval benchmarks, ConceptFormer posts a 16.7% relative improvement in average NDCG@10 over the strongest visual retrieval baseline it was tested against, and a 22.1% relative improvement over the strongest OCR-based text retrieval baseline. Those are relative gains against the best competing system in each category, not against some weak strawman, which is the more honest way to report this kind of result.

What I find more convincing than the headline numbers is the follow-up analysis. The paper reports that when you look at what the latent concepts actually attend to, they land on the genuinely relevant visual evidence: the right region of a chart, the right cell in a table. That's a check worth taking seriously, because it's easy for an auxiliary training signal to improve a benchmark score for reasons that have nothing to do with the story you're telling about it. Here, the concepts appear to be doing the job they were designed for: connecting fine-grained textual detail (the kind OCR is good at) with document-level visual structure (the kind pure text search misses), inside one retriever, without one approach cannibalizing the other.

Why it matters if you're building or choosing a document AI stack

If you're building search or RAG over anything that isn't plain text, this paper is a data point worth paying attention to, even if you never touch the code. The OCR-then-search pattern that a lot of teams default to because it's familiar is measurably behind vision-native retrieval on documents where the evidence lives in charts, tables, and layout, and that gap looks like it's widening rather than closing. If your documents are financial reports, product spec sheets, scanned forms, or slide decks, and your users ask questions whose answers sit inside a chart or a table rather than a sentence, a text-only pipeline is going to keep missing pages that a human would find instantly.

That doesn't mean OCR is dead. Plenty of documents genuinely are dense, well-structured text, and OCR-based search remains cheaper and easier to debug for those cases. But if you're evaluating engines or retrievers for document-heavy workloads with real visual structure, this is a good reminder to test with the kind of pages that actually break naive OCR: a chart with a number nobody wrote out as a sentence, a table where the answer requires knowing which column you're in, a form where a checkbox next to a line means something a text extractor can't see. Benchmark on those, not on clean paragraphs, because that's where the gap between old-style and vision-native retrieval actually shows up.

What's still fuzzy

A few things are worth flagging honestly. The training process depends on a strong vision-language model to generate the latent concept supervision, which means the quality of ConceptFormer's training signal is bounded by how good that teacher model is and adds real cost to the training pipeline, even though it doesn't touch inference cost. The latent concepts themselves are not human-readable by design, which is a fine trade for performance but a real cost for debugging: if the retriever gets something wrong in production, there's no caption or bounding box to inspect and ask "was this the problem."

The headline comparisons are also best-versus-best: ConceptFormer's strongest configuration against the strongest visual baseline and the strongest OCR baseline, each on the benchmarks where they were tested. That's a legitimate and common way to report results, but it's worth remembering that a single deployed system rarely gets to cherry-pick its best baseline comparison the way a paper can. And the abstract describes the benchmarks only as "diverse visual document retrieval benchmarks," so it's not clear from what's public how well this generalizes to the messiest real-world cases: poor scan quality, handwriting, heavily multilingual documents, or the kind of ugly, low-resolution photos of paper that show up constantly in actual production traffic rather than in curated benchmarks.

The code and data are public on GitHub, which is the right move and makes it possible for anyone to check these claims rather than take the abstract's word for it. For a field that's been leaning hard on "trust the benchmark," that's worth crediting on its own.