Picture a state DOT plan sheet for a highway interchange. A dimension line runs from a guardrail post to the edge of pavement, labeled "24'-0"." Nearby, a callout bubble points at a culvert with a note about pipe diameter. None of this has a natural reading order. The number 24 means nothing without the arrow it's attached to, and the arrow means nothing without the two points it connects. Now run that page through OCR. You get a bag of text: "24'-0"", "CULVERT", a pipe diameter, some station numbers, all correct as extracted strings, and all useless, because the thing that made them meaningful (which line touches which object, where on the page it sits) is gone.

That's the problem a new paper called PlanSightRAG is trying to solve, and it's a good example of a failure mode that's easy to miss if you think of OCR as a solved problem. It isn't solved for drawings. It's solved for documents that are mostly sentences.

Why civil plans break the standard document-AI playbook

Most retrieval-augmented generation (RAG) systems for documents follow the same recipe: OCR the page into text, chunk the text, embed the chunks, and let a language model retrieve and reason over them. That works reasonably well for contracts, invoices, and reports, because the information lives in the words and the words read in order.

Civil engineering standard plans don't work that way. A single sheet from a state Department of Transportation (DOT) plan set might combine:

A plan view (looking straight down) and a profile view (a side elevation) on the same page, at different scales
Dimension lines and leader lines that encode relationships purely through geometry
Symbols and hatching patterns that mean specific things by convention, not by any text label
Cross-references to other sheets, tables, and a separate specification book that defines the actual numeric limits

Strip the layout out with OCR and you've thrown away most of what makes the drawing readable. The authors put it plainly in their abstract: OCR-based automation "strips away the geometry and layout essential for interpreting these plans." Engineers still do compliance checking on this stuff by eye, sheet by sheet, cross-referencing against specification documents that can run hundreds of pages. It's slow, and it's exactly the kind of repetitive, error-prone task automation should be good at, except that the standard document-AI toolkit doesn't fit it.

The fix: stop converting the drawing into text at all

PlanSightRAG's core move is to skip OCR-based text extraction entirely for indexing and retrieval. Instead of converting a plan sheet to text and then searching the text, it treats each page as an image and searches images with images.

The retrieval backbone is a model called ColNomic-3B. To understand what that means, it helps to know the family it belongs to. Standard document retrieval takes a whole page and compresses it into a single vector (a list of numbers) that's supposed to summarize everything on it. That's fast but lossy: a page with twelve distinct callouts collapses into the same size vector as a page with one paragraph. ColNomic instead uses a multi-vector approach descended from a method called ColPali: it slices the image into a grid of patches and produces a separate vector for each patch, so a query can match against the specific region of the page where the relevant content actually sits, not just an average of the whole sheet. When you query the system (say, "what's the minimum clearance for this guardrail type"), it compares your query against every patch across every page and finds the strongest matches. This is why the geometry survives. You're never asking a model to describe the drawing in words first.

On top of that retrieval layer sits what the paper calls an agentic Planner-Retriever-Auditor-Synthesizer pipeline. That's four roles, each handled by a step in the pipeline rather than a single monolithic prompt:

Planner breaks down what the question is actually asking (which sheet types are relevant, what needs to be checked)
Retriever pulls the candidate plan pages and reference specification pages using the ColNomic multi-vector search
Auditor checks whether the retrieved evidence actually supports an answer, rather than passing along whatever came back
Synthesizer produces the final answer or compliance verdict, grounded in the retrieved pages

The other piece worth flagging is what they call MaxSim heatmaps. Since the retrieval model computes patch-level similarity scores, you can visualize which patches on a page drove the match. That produces a heatmap overlaid on the original drawing, showing you exactly where the system thinks the relevant dimension, symbol, or note is. This matters more than it might sound like at first. A compliance check that just returns "PASS" or "FAIL" is not something an engineer can sign off on. A compliance check that returns "FAIL, and here's the exact spot on the drawing that triggered it" is something you can actually review and trust, or catch when it's wrong.

What the numbers actually say

The team built a benchmark from real material: 1,898 pages of standard plans from five state DOTs, turned into 4,056 question-answer pairs. On zero-shot retrieval (meaning the model wasn't fine-tuned on this specific set of plans beforehand), PlanSightRAG hit 91.47% Recall@5. That metric means: for a given question, is the correct source page among the top 5 pages the system retrieves? Getting that right over 91% of the time on drawings this dense and unstructured is a solid result.

The more convincing number, to me, is what happened on a held-out Michigan DOT corpus that wasn't part of the original five states: 91.40% Recall@5, almost identical to the in-distribution result. Different states format their standard plans differently. A system that only works on the drawings it was tuned against isn't very useful to a DOT that wasn't in the training set. Generalizing this cleanly to a new state's plan format is the result that actually suggests the approach is picking up on general plan-reading structure, not memorizing a handful of layouts.

Where the paper gets more honest, and more interesting

The compliance-checking results are where I want to slow down, because the headline number needs an asterisk that the paper itself supplies.

On synthetic, parametrically-generated compliance drawings (drawings built with known, programmatically-controlled values so the correct answer is unambiguous), a pipeline built on Qwen2.5-VL-72B, a 72-billion-parameter vision-language model, reaches 100% verdict accuracy. But only under one specific condition: when the system is handed the rule threshold it needs to check against, already resolved to a number. In other words, if you tell the model "the maximum allowed value here is 6 inches," it correctly compares the drawing's dimension to 6 inches every single time.

That's a controlled ceiling, and the paper is upfront that it's a ceiling, not a demonstration of end-to-end compliance checking. It tells you the model can compare a number on a drawing to another number correctly, which is good to confirm but not the hard part of the task. The genuinely useful comparison is this: a non-VLM OCR baseline, with no vision model involved, already reaches 76.4% on that same setup. So a large chunk of this particular benchmark is solvable without understanding the drawing's geometry at all, just by extracting numbers and comparing them. The gap between 76.4% and 100% is real, but it's a narrower gap than "OCR versus a fancy vision system" might suggest at first glance, and it's worth remembering when a paper leads with a 100% figure.

The result I think is actually the paper's most important contribution is the smaller, less flashy one at the end: autonomous rule-grounding. Instead of being handed the compliance threshold, the system extracts the numeric limit directly from a specification corpus with no human supplying the rule ahead of time. That's the version of the task that resembles what compliance checking actually requires in practice: you don't get handed the rule number, you have to go find it in a spec book and then apply it correctly to the drawing. That's a much harder problem than comparing two pre-resolved numbers, and it's the one this system needs to be reliably good at to be worth deploying. The paper demonstrates it works, without giving a headline accuracy figure for it, which is itself a tell about how far this component still has to go before it's benchmark-ready.

Why it matters for anyone building or buying document AI

If you're building document pipelines that touch engineering drawings, architectural plans, electrical schematics, or anything else where meaning lives in the spatial relationship between marks rather than in the words themselves, this paper is a concrete argument for a rule worth internalizing: don't OCR first and ask questions later. Converting to text throws away information you can't easily reconstruct, and no amount of clever prompting on the OCR output gets it back.

For product teams evaluating OCR or document-AI vendors, the retrieval numbers here (91%+ Recall@5, holding up on an unseen state's plan format) are a genuinely useful bar to compare against if you're dealing with layout-heavy documents: maps, technical drawings, forms with complex tables, anything where "read it top to bottom" isn't how a human would approach the page.

And for anyone tempted to quote the 100% compliance-verdict number in a deck, read the fine print first. That number describes a system checking a value it was already told the rule for. The real test, autonomous rule-grounding from a spec corpus with no answer key handed over, is the one to watch, and it's also the one without a clean accuracy number yet. That's not a knock on the paper. Being honest about which part of your pipeline is proven and which part is a promising early demo is more useful to practitioners than a single triumphant percentage.

What's still missing

A few things are worth keeping in view. The benchmark's retrieval numbers come from Recall@5 on question-answer pairs built from five states' plans, which is a solid start but far short of the full diversity of DOT formatting conventions across all fifty states, let alone county and municipal standards that don't follow DOT templates at all. The compliance pipeline relies on a 72-billion-parameter model, which is not a lightweight thing to run in production, and the paper doesn't report cost or latency figures that would tell you whether this is practical to run at the volume a real DOT review office would need. And the synthetic compliance drawings, while a reasonable way to get a clean ground truth, are simpler than the messy, hand-annotated, decades-old plan sets that a lot of real infrastructure review still runs on.

None of that undercuts the core idea. Treating a plan sheet as an image to be searched, instead of a page to be transcribed, is the right instinct for this category of document, and the retrieval results back it up. The compliance-checking half of the story is earlier stage than the framing suggests, but the paper is refreshingly clear about exactly where that line sits, which is more than you can say for most benchmarks in this space.

Reading Highway Plans Without Losing the Drawing: A Look at PlanSightRAG