Stop Feeding Ten Pages to Vision LLMs: Inside ViSAR’s Training-Free Adaptive Retrieval

Ask a visual language model to locate information in a long document, and standard document retrieval architectures can do something remarkably wasteful. They score each page screenshot independently using a visual encoder, sort the list, and hand a fixed top five or top ten pages over to a heavy Large Vision-Language Model (LVLM).

If the query is simple, the evidence may sit on one page while the remaining retrieved pages add irrelevant visual context. The model still processes tables, headers, and layouts that may not help answer the question. Conversely, a rigid budget can omit evidence when a question requires information from several pages.

This dilemma stems from a fundamental design choice in modern visual document retrieval: fixed top-$k$ selection. Existing approaches commonly retrieve the same number of pages regardless of query complexity, even though determining how many pages a particular question needs has a direct effect on latency and answer quality.

A research paper published on September 2, 2026, by Adrien Mialland, Marc Plantevit, Julien Gallois, and Céline Robardet (INSA Lyon, CNRS, LIRIS, EPITA, and Lowit) introduces an elegant alternative: ViSAR (Visual Semantic Activation Retrieval). ViSAR dynamically determines the number of pages to retrieve at inference time without training the visual encoder. By analyzing interactions already present inside late-interaction embedding spaces, ViSAR reduced end-to-end question-answering latency by up to 58.7% in the reported experiments while maintaining or improving answer accuracy.

The Blind Spot Inside Late-Interaction Encoders

To understand how ViSAR works, look first at how visual document retrievers operate.

Traditional text-based retrieval requires Optical Character Recognition (OCR) to convert pixels into text strings before embedding them. That process can miss visual content and document-specific layout. Modern OCR-free visual retrievers, including ColPali and ColQwen2.5, instead represent a page screenshot as a set of visual patch embeddings in a shared space. The user query is similarly represented by multiple embeddings.

Relevance is calculated using late interaction, powered by the MaxSim operator introduced by ColBERT:

$$S_{Q, P^p} = \sum_{i=1}^m \max_{j \in {1, \dots, n_p}} \langle q_i, v_j^p \rangle$$

For every query embedding $q_i$, the retriever searches across visual patch embeddings $v_j^p$ on page $p$, takes the highest similarity, and sums those peak alignments.

Late interaction enables fine-grained query-page matching and offline page encoding. Yet it has two structural limitations:

1. Independent page evaluation: The retriever produces relevance scores for pages independently, without exploiting the semantic structure across pages.
2. Uniform semantic weighting: Every query and page embedding contributes uniformly to the standard late-interaction score.

Text retrieval has addressed related weighting questions using token-frequency statistics, learned importance estimates, or sparse representations. But those approaches rely on discrete token structures or additional training and do not directly extend to visual embeddings.

Consequently, late interaction produces a ranked list of scalar scores. Previous adaptive-$k$ approaches have relied on score heuristics, such as identifying the largest gap between consecutive scores or clustering score distributions. ViSAR instead uses the semantic structure encoded in late-interaction representations.

How ViSAR Operates: Decoding Semantic Activations

ViSAR does not alter the underlying encoder weights. Instead, it preserves multi-vector interactions that standard late interaction would otherwise collapse into a scalar score.

The architecture breaks down into four sequential stages: calculating query-to-page weights, deriving patch-level importance, constructing a query-conditioned page similarity matrix, and minimizing an adaptive partition cost function.

Query & Page Embeddings
         │
         ▼
[1. Query-to-Page] ──► Weights query semantics and page co-activations
         │
         ▼
[2. Page-to-Query] ──► Reverses MaxSim and derives patch-level weights
         │
         ▼
[3. Page-to-Page]  ──► Matches weighted visual patches across pages (N x N matrix)
         │
         ▼
[4. Adaptive-k]    ──► Evaluates coherence and leakage to pick optimal k*

1. Query-to-Page Interaction Weighting

Standard late interaction collapses the interaction between query vector $q_i$ and page $p$ into a sum immediately. ViSAR retains the full activation matrix:

$$A_{p,i} = \max_{j} \langle q_i, v_j^p \rangle$$

This value represents how strongly query semantic $i$ manifests anywhere on page $p$. ViSAR normalizes this activation by its mean across document pages ($\hat{A}_{p,i}$) and scales it by its normalized inter-page standard deviation:

$$\tilde{A}{p,i} = \hat{A}{p,i} \cdot \hat{\sigma}_i$$

A query embedding with similar activations across pages is downweighted, while one with stronger, spatially localized activations receives more emphasis.

From these modulated activations, ViSAR computes two weights:

• Query embedding weight ($w_i$): Formulated as $w_i = \log(N / (1 + a_i))$, where $a_i = \sum_p \tilde{A}_{p,i}$. This penalizes ubiquitous semantic content and emphasizes sparse activations.
• Page weight ($w_p$): Measures semantic co-activation. Pages where discriminative query semantics activate together receive greater importance.

2. Page-to-Query Interaction Weighting

Next, ViSAR reverses the direction of search. Instead of asking only how well a page matches the query, it evaluates how relevant each visual patch on a page is to the query.

For every visual patch $v_j^p$, ViSAR computes its relevance score $r_j^p$ by taking the maximum alignment across query vectors, modulated by the token and page weights calculated in step one:

$$r_j^p = \max_i \left[ \langle v_j^p, q_i \rangle \cdot \tilde{A}_{p,i} \cdot \hat{w}_i \cdot \hat{w}_p \right]^2$$

It then centers and thresholds the relevance:

$$w_j^p = \max(0, r_j^p - \text{mean}_{p,j}(r_j^p))$$

This thresholding produces inactive patches with zero weight. Some pages can become entirely inactive, and those pages need not contribute to the page-to-page similarity computation.

3. Page-to-Page Interaction Matrix

With weighted patches in hand, ViSAR measures cross-page relationships. For a source page $P^p$ and target page $P^{p'}$, it computes how well weighted patches on page $p$ match weighted patches on page $p'$:

$$S_j^{p \to p'} = \hat{w}j^p \cdot \max{j'} \left[ \langle v_j^p, v_{j'}^{p'} \rangle \cdot \hat{w}_{j'}^{p'} \right]$$

ViSAR averages the top $T=50$ patch interactions and takes the square root, yielding a directional similarity score:

$$\text{Sim}(p, p') = \sqrt{\frac{1}{T} \sum_{j \in \mathcal{T}} S_j^{p \to p'}}$$

Because source patches seek their best counterparts independently in the target page, this similarity is directional: $\text{Sim}(p, p') \neq \text{Sim}(p', p)$. Assembling these pairwise scores creates an $N \times N$ page-level similarity matrix conditioned on the user query.

4. Adaptive-k via Coherence and Leakage

To decide where to cut off retrieval, ViSAR ranks pages using their self-similarity score $s_p = \text{Sim}(p, p)$.

For every candidate cutoff $k$, the document is split into two groups: the candidate relevant set $R_k$ containing the top-$k$ pages, and the irrelevant set $I_k$ containing the remaining pages.

For each page $p \in R_k$, ViSAR evaluates two opposing forces:

• Internal Coherence ($c_k^p$): The average similarity between page $p$ and pages in the retrieved candidate set $R_k$.
• External Leakage ($l_k^p$): The average similarity between page $p$ and discarded pages in $I_k$.

The overall quality of candidate set size $k$ is scored using the cost function $J(k)$:

$$J(k) = \sum_{p \in R_k} w_p^s \left( c_k^p - \gamma l_k^p \right)$$

Here, $\gamma$ is a leakage penalty parameter, set to $10^5$ in the reported experiments. Finding the optimal page count $k^\star$ requires evaluating at most $N$ candidate sets rather than exhaustively evaluating $2^N$ possible subsets. The method also evaluates whether the minimum corresponds to a sharp transition in the cost function.

What the Data Shows: Leaner Sets and Faster Inference

The authors evaluated ViSAR on an NVIDIA A6000 GPU with 48 GB of memory across two multi-page benchmarks: MMLongBench and LongDocURL. Both datasets provide answer-evidence pages for evaluating page ranking, as well as document question-answering scenarios requiring textual and visual reasoning.

They tested three visual late-interaction encoders—ColQwen2.5, ColPali, and ColModernVBERT—alongside a text retriever using Tesseract OCR (ColBERTv2) and a single-vector visual retriever (VisRAG-Ret). Answer generation used Qwen2.5-VL-7B-Instruct, while Qwen2.5-14B-Instruct served as an LLM-as-a-judge evaluator using few-shot structured output.

Across both benchmarks, ground-truth evidence requires an average of 1.9 pages: the median is one page on MMLongBench and two pages on LongDocURL.

1. Retrieval Efficiency and Context Compaction

Table 1 shows how adaptive retrieval methods behave relative to the Oracle, defined as the smallest top-$k$ window in the standard late-interaction ranking that contains every evidence page:

Encoder	Method	MMLongBench Mean $k^\star$	MMLongBench Median	LongDocURL Mean $k^\star$	LongDocURL Median
ColQwen2.5	Oracle (Late-Int.)	8.3	2	10.7	3
	Score-Cluster	18.6	8	36.6	20
	Largest-Gap	15.4	2	28.7	3
	ViSAR (Ours)	4.7	3	7.9	5
ColPali	Oracle (Late-Int.)	8.8	2	12.4	3
	Score-Cluster	18.7	8	35.8	18
	Largest-Gap	16.2	3	24.7	2
	ViSAR (Ours)	5.3	3	8.1	6
ColModernVBERT	Oracle (Late-Int.)	10.5	2	13.0	3
	Score-Cluster	23.0	12	47.4	44
	Largest-Gap	20.0	4	39.3	4
	ViSAR (Ours)	7.5	4	13.5	11

The Oracle shows that retrieving all evidence pages can require a much larger window than the number of evidence pages alone would suggest. Because the underlying late-interaction ranking may place irrelevant pages ahead of evidence pages, including all evidence can require expanding the retrieval window.

Score-Cluster and Largest-Gap generally retrieve more pages than ViSAR. On MMLongBench with ColQwen2.5, ViSAR retrieves an average of 4.7 pages. It has lower recall than Largest-Gap (75.16% versus 81.12%) but higher precision (50.37% versus 45.13%), favoring a more compact retrieved set.

2. Downstream Question-Answering Accuracy

The reported results show that adaptive retrieval can improve answer accuracy while using an input budget of at most five or ten pages.

Retrieval Method	MMLongBench Max-5	MMLongBench Max-10	LongDocURL Max-5	LongDocURL Max-10
Fixed Top-$k$ Baselines
ColBERTv2 (OCR + Text)	24.51%	24.70%	47.18%	47.70%
M3DocRAG (ColPali)	34.86%	35.08%	59.31%	58.71%
VisRAG-Ret (Single-Vector)	34.48%	35.69%	57.29%	58.02%
ColQwen2.5 (Fixed Top-$k$)	35.04%	35.69%	59.79%	59.27%
Adaptive Retrieval (ColQwen2.5)
Largest-Gap	35.79%	35.88%	61.01%	60.89%
Score-Cluster	36.25%	35.97%	60.00%	59.83%
ViSAR (Ours)	36.53%	36.63%	61.06%	60.97%

Across 60 configurations combining three encoders, five LVLMs, two page budgets, and two datasets, ViSAR improved accuracy in 24 cases and maintained accuracy in the remaining 36. The paper reports no statistically significant decreases. With ColQwen2.5 and ColPali, the results generally trend upward, with statistically significant improvements on LongDocURL according to McNemar’s test ($p < 0.05$).

The largest improvements appeared with LVLMs the authors describe as more sensitive to longer contexts, consistent with ViSAR reducing both retrieved pages and irrelevant pages.

3. The Latency Dividend

The practical bottleneck in the evaluated visual document pipeline is LVLM generation time. Fixed top-$k$ retrieval always fills the LVLM input budget, while ViSAR adjusts the number of pages and uses the full budget only when needed.

ViSAR introduces retrieval overhead from patch weighting and cross-page similarity calculations. But the paper finds that this overhead contributes only marginally to total cost for most evaluated document sizes, while reduced generation latency dominates the end-to-end result.

MMLongBench Latency at Max-10 Budget:
Fixed top-10: [Retrieval][================ Generation ================]
ViSAR:        [Larger retrieval step][====== Reduced generation ======]
                                      End-to-end reduction: up to 58.7%

The reported end-to-end latency reduction reaches 58.7% on MMLongBench and 38.5% on LongDocURL at a Max-10 LVLM budget. ViSAR’s retrieval overhead grows with document size and becomes noticeable for the largest MMLongBench document, which contains 468 pages. The supplementary material proposes similarity-matrix approximations for such cases.

Matrix Geometry: When Sparsity Signals Correct Answers

One of the paper’s most compelling observations lies in the structure of the query-conditioned page similarity matrix $\text{Sim}(p, p')$.

The authors illustrate two contrasting cases:

• Sparse matrices: When query-relevant semantics are localized, relatively few pages share the relevant content. The matrix is sparse and the cost function $J(k)$ has a sharper minimum, supporting a more reliable retrieval boundary.
• Dense matrices: When query-relevant semantics are distributed across more pages, the matrix becomes denser. The corresponding minimum in $J(k)$ is shallower, making the stopping decision less reliable.

Across tested encoders on MMLongBench, and with a similar trend reported for LongDocURL, higher similarity-matrix sparsity correlates with higher answer accuracy.

This is a useful research direction rather than a completed confidence system. The authors suggest that matrix structure could become a label-free feedback signal for future retrieval strategies, including iterative query refinement or evidence selection, without requiring an additional model or training.

Engineering Caveats: The Boundaries of Training-Free Pruning

ViSAR is an algorithmic contribution with several practical constraints worth keeping in mind:

• Page-to-page computation grows with document size: ViSAR constructs a page-level similarity matrix and evaluates page-to-page interactions block-wise rather than as a dense tensor. Its inactive-page optimization avoids unnecessary computation, but the paper reports growing retrieval overhead as documents become larger. The overhead is most noticeable in its 468-page MMLongBench example.
• Encoder quality sets a limit: ViSAR relies on semantic structure already learned by the encoder. ColQwen2.5 and ColPali showed stronger results than ColModernVBERT. The paper attributes ColModernVBERT’s more modest gains in part to its smaller size—250M parameters versus 3B—and the resulting difficulty in separating semantics into compact retrieval sets.
• Encoding instability was observed: ColModernVBERT had numerical instability during encoding for 7.31% of LongDocURL queries. The authors report that the issue occurred independently of ViSAR, and those samples were excluded from evaluation.
• ViSAR favors compact retrieval: As the Oracle cutoff grows, ViSAR retrieves fewer pages on average, limiting irrelevant context but also trading recall for precision in some settings. Largest-Gap achieves the best aggregate F1 score on LongDocURL, while ViSAR provides more consistent adaptive behavior across different values of $k_{\text{Oracle}}$.

Why it matters for document pipelines

For teams studying document question answering, ViSAR offers an alternative to treating retrieval depth as a permanent constant.

Its central finding is that multi-vector visual retrievers contain more information than a final scalar late-interaction score exposes. ViSAR uses query-to-page activations, patch-level weights, and page-to-page semantic structure to choose a retrieval set tailored to the query.

The paper’s reported results point to three concrete implications:

1. More compact page sets: ViSAR retrieves fewer pages on average than the Oracle, Largest-Gap, and Score-Cluster methods in the reported comparisons.
2. Lower end-to-end latency: At a Max-10 budget, it reduces end-to-end RAG latency by up to 58.7% on MMLongBench and 38.5% on LongDocURL.
3. A signal for future retrieval-quality research: The relationship between similarity-matrix sparsity and answer accuracy suggests a potential feedback signal for later retrieval and evidence-selection methods.

If a visual document stack already uses a late-interaction encoder such as ColPali or ColQwen2.5, ViSAR provides a training-free way to investigate adaptive-$k$ retrieval. Its results suggest that the embedding geometry can help decide when a document pipeline has retrieved enough pages—without assuming that every query deserves the same fixed context window.