Skipping Vision Layers Works, Until You Ask It to Read Text

Ask a vision-language model to count the cars in a parking lot photo, and it runs the same amount of computation as when you ask it to transcribe a blurry receipt. Same encoder, same number of layers, same milliseconds and dollars spent, whether the task is "how many" or "what does this say." That should bother you a little, because those are obviously different jobs. Counting cars needs rough spatial layout. Reading a receipt needs fine detail on tiny characters. Nobody designed these models to treat both tasks identically. It's just what happens when you build one vision encoder and run it on everything.

A new paper by Tarun Tomar asks a blunt question: if a vision-language model already works, can you make it cheaper by skipping some of its internal vision processing, without retraining anything? And if so, does it help to skip different parts depending on whether the question is about text, counting, objects, colors, or spatial relationships? The answer is yes and also, in an important way, no. The paper finds real, transferable gains from smart pruning, and then finds that its most tempting idea (customizing the shortcut per task) collapses in exactly the case OpenOCR readers care about most: reading text.

Passport 8 — by Ken_Mayer via flickr, BY 2.0.

What's actually inside a vision-language model's "eyes"

A vision-language model (VLM) takes an image and a text prompt and produces a text answer. Before any language reasoning happens, the image passes through a vision encoder: a stack of transformer blocks, each one refining the visual representation a little more. Block 1 might pick out edges and textures. Block 12 might represent something closer to "this is a receipt total in the bottom-right region." By the time the image reaches the language model, it has passed through every block in the stack, in order, no matter what the question was.

That's the setup this paper attacks. The idea is to skip a fixed number of blocks, call it K, entirely at inference time. No fine-tuning, no retraining, just deciding "we're not going to run blocks 4, 9, 15, and 21 this time" and seeing what happens to accuracy. If accuracy barely drops while compute drops meaningfully, you've found free efficiency. Do this across a family of models and you've found something that might generalize.

The obvious first attempt is to rank blocks by how important they seem individually (remove one, measure the damage, repeat for every block) and then delete whichever K blocks look least important on their own. The paper calls this independent ranking, and it's the natural baseline anyone would reach for first. Another natural instinct is to just chop out a contiguous run of blocks, on the theory that middle layers are redundant. Both are cheap to compute. Both, it turns out, leave real accuracy on the table.

Breeding better routes instead of just ranking blocks

The problem with independent ranking is the same problem you'd run into evaluating basketball players one at a time and then picking your "best five" without ever checking if they play well together. Blocks interact. Removing block 9 might be harmless if block 10 is still there to cover for it, but harmful once block 15 is also gone. Importance isn't additive, so scoring blocks in isolation and combining the top picks misses interactions that only show up when you test combinations directly.

That's where the paper's main method comes in: an evolutionary search over which specific set of K blocks to skip. Instead of scoring blocks individually, it tests whole combinations (whole "routes" through the vision encoder) directly against a held-out set of 876 examples, and evolves toward better ones. Start with a population of candidate routes, keep the ones that score well, mutate and recombine them, repeat. It's the same logic breeders use to improve a crop: you don't decide in advance which single gene matters most, you select on real-world outcomes and let good combinations survive.

One detail matters here: the search is "source-balanced," meaning the held-out set it evaluates against deliberately mixes different task types (OCR, counting, object recognition, attributes, spatial relationships) in balanced proportions, rather than letting one category dominate. That's meant to stop the search from finding a route that's great at, say, counting and mediocre at everything else just because counting examples happened to be overrepresented.

The paper tests this on two models, Qwen2.5-VL-3B-Instruct and SmolVLM2-2.2B-Instruct, both mid-size open VLMs, and compares against independent ranking, contiguous removal, and random block selection at matched budgets (same number of blocks skipped, different choice of which ones).

The result that should make people rethink naive pruning

The headline number: on SmolVLM2, the evolutionary search's shared four-block route (one fixed set of four blocks to skip, used for every question regardless of task) beats independent ranking construction by 4.91 percentage points. That's a meaningful gap for a change that costs nothing at inference time beyond deciding which four blocks to leave out. And because this pattern shows up on both architectures tested, it's a decent signal that combinatorial search (checking whole routes together rather than scoring blocks one by one) is a genuinely better way to build these shortcuts than the obvious first approach.

If the paper stopped there, the takeaway would be simple: searching for good block combinations beats ranking them individually, and that's worth knowing if you're trying to cut inference cost on a VLM without retraining it.

Where the specialization idea gets tempting, and then breaks

The next question the paper asks is more ambitious: what if you don't use one fixed route for every question, but instead pick a different K-block route depending on what kind of question it is? If you know (or can guess) that a question is about OCR versus counting versus spatial reasoning, maybe the ideal blocks to skip are different for each. This requires a capability label, some signal telling the system which task category the current question falls into.

On Qwen2.5-VL, this pays off, at first glance. The six-block capability-specific policy beats the shared, one-size-fits-all route by 2.17 points overall, and almost the entire gain comes from a single category: a 7.10-point improvement on OCR specifically. That's a real result, and if it held up everywhere, it would be genuinely useful: use one generic route by default, swap to a specialized OCR route when you know you're reading text, and get a meaningful accuracy bump for free.

Except it doesn't hold up. When the paper checks the SmolVLM2 OCR-specific route against IIIT5K, a standard scene-text benchmark (photos of text in the wild: storefronts, road signs, product labels), that specialized route trails the plain shared route by 13.6 points. Not a rounding error. A specialized shortcut that looked like a win on the search's own held-out set turned out to actively hurt performance on a separate, sealed OCR test.

Why this happens is the important part. The capability-specific route was optimized against a limited pool of OCR-labeled examples from the selection set. It learned to skip blocks that happened to be unimportant for that particular slice of text-reading examples, not blocks that are unimportant for reading text in general. IIIT5K's scene text (curved signage, weird fonts, cluttered backgrounds) apparently needs some of exactly the visual processing the specialized route decided to throw away. The route overfit to a narrow definition of "OCR" rather than finding something that generalizes across what OCR actually looks like in the wild.

Why it matters for anyone building or choosing OCR

If you're building a document-processing pipeline, or evaluating which OCR engine or VLM backbone to trust, this paper is a useful cautionary tale about a specific kind of efficiency trick, not a reason to distrust VLM-based OCR generally.

The good news: block skipping via proper combinatorial search, not ad hoc pruning, is a legitimate way to cut inference cost on a vision-language model without retraining it, and the gains transferred across two different model families in this study. If a vendor tells you they've found a cheaper inference path through careful search rather than guesswork, that's plausible and worth taking seriously.

The catch: any time a provider tells you they've built a "task-specialized" or "OCR-optimized" fast path through a shared model, ask what it was validated against. This paper shows a specialized OCR route can look great on the exact benchmark used to build it, then lose over 13 points on a different, standard OCR test. That gap is exactly the kind of thing that would be invisible if you only checked the vendor's own reported numbers. If you're choosing between OCR engines, the practical move is the same one this paper implies: test on your own documents, especially the messy real-world kind (receipts, signage, handwriting, low-contrast scans), not just whatever benchmark a vendor's optimization was tuned against.

For anyone building rather than buying: this is a strong argument for keeping a single shared, well-tested route as the default and treating any task-specific fast path as an experimental add-on that needs its own held-out validation on a genuinely different test set before it ships, not just a stronger score on the set you used to find it.

What's still shaky here

This is a small-model, small-scope study, and it should be read that way. Both tested models are relatively compact (3B and 2.2B parameters), so it's an open question whether the same gap between shared and specialized routes shows up in larger production-grade VLMs. Only two architectures were tested, which is enough to show the pattern isn't a one-off fluke, but not enough to call it universal.

There's also a practical wrinkle the paper doesn't fully resolve: using a capability-specific route requires knowing the capability label at inference time, meaning something (a classifier, a user-provided hint, a router model) has to correctly tag each incoming question as OCR, counting, spatial, or whatever, before picking a route. Get that classification wrong and you're not just losing the specialization's benefit, you might be actively routing an OCR question through a route optimized for counting.

The most important limitation, though, is the one the paper is honest about in its own framing: combinatorial search reliably improves how you build a route, but capability labels don't reliably define a route that generalizes. That's a narrower and more useful claim than "task-specific pruning works," and it's the one worth remembering the next time someone pitches a model that's been specially optimized for reading text. Ask what it does on text that doesn't look like the text it was optimized on.