Teaching a 500M Vision Model for Navigation More Than Doubled Its OCR-Bench Score

A compact vision-language model trained for navigational assistance can improve its OCR-Bench score without a separate OCR-focused training stage. That is one of the most striking results in Small yet Assistive: Spatially-Aware Post-Training for Low Vision, a paper from researchers at IIT Mandi and Mohamed bin Zayed University of Artificial Intelligence.

Published on September 23, 2026, the paper describes work led by Rishabh Choudhary and his co-authors to build an on-device visual narrator for people with blindness and low vision (BLV). They took SmolVLM2-500M, a compact model with 507.5 million parameters, and subjected it to a post-training pipeline targeting spatial grounding, directional language, metric distances, and hazard detection.

When they evaluated the finished model on standard vision-language benchmarks, the result was striking. On OCR-Bench, the model's ANLS score climbed from 32.54% to 65.54%. That is a 101.5% relative improvement. On TextVQA, accuracy rose from 37.75% to 54.47%, a 44.2% gain.

The paper interprets these results as evidence that BLV-focused spatial post-training can transfer to general visual-text reasoning. The improvement came from training on spatially grounded video descriptions, not from a dedicated document-recognition stage.

The Polite Hallway Problem

If you ask a current vision-language model to describe a room, it may produce a generic caption: "A hallway with some doors and people."

That may be adequate for image-captioning benchmarks, but the paper argues that it is inadequate for physical navigation. Professional audio-description guidelines, established by the ITC and adopted by Netflix and the American Council of the Blind, prescribe present-tense, spatially grounded, hazard-first, objective narration.

An audio-description-compliant narrator might say: "A corridor extends approximately eight meters ahead. A partially open door is on your left, three meters away. A person walks toward you from the far end."

Large VLMs can generate high-quality audio-description-compliant narrations, but the paper notes that they cannot run on mobile devices. Small VLMs offer competitive latency, yet often lack the spatial detail, directional cues, and hazard awareness needed for BLV navigational assistance.

The researchers picked SmolVLM2-500M-Video-Instruct as their foundation. The base architecture consists of three parts:

• A SigLIP vision transformer tower with approximately 86.4 million parameters, processing 384×384-pixel inputs.
• A linear projection connector with LayerNorm and approximately 11.8 million parameters, mapping visual features into language-model tokens.
• A LLaMA-architecture SmolLM2 language decoder with approximately 409.3 million parameters, 32 blocks, and an 8,192-token context window.

Total parameter count is approximately 507.5 million. The paper uses "500M" as the model's class label rather than as a literal parameter total. The target was a compact model capable of producing spatially grounded BLV descriptions while remaining practical for on-device inference.

The Post-Training Assembly Line

The authors built a four-stage pipeline combining synthetic teacher supervision, phased supervised fine-tuning, rule-based reinforcement learning, and a recovery patch.

[4 Video Keyframes (LUV Delta)] + [BLV Prompt]
                     │
                     ▼
         ┌───────────────────────┐
         │ Phase 1: SigLIP Only  │  (First 25% of training steps)
         └───────────┬───────────┘
                     ▼
         ┌───────────────────────┐
         │ Phase 2: LoRA Unfreeze│  (All LoRA adapters; Vision LR halved)
         └───────────┬───────────┘
                     ▼
         ┌───────────────────────┐
         │   GRPO Reinforcement  │  (6 rule rewards: distance, hazards, no hedging)
         └───────────┬───────────┘
                     ▼
         ┌───────────────────────┐
         │   SFT Recovery Patch  │  (50 curated samples)
         └───────────┬───────────┘
                     ▼
         [Fold 3 LoRAs into Base Weights] -> Single 969 MB FP16 Checkpoint

Stage 1: Keyframes Without Extra Model Inference

To create training inputs, the team extracted four keyframes per video using perceptual color-space differencing in LUV space. They first subsampled each video to at most 64 evenly spaced frames, computed the mean absolute inter-frame difference, and selected the three highest-difference frames alongside the first frame.

This captured scene transitions and significant motion events without an additional model inference step. The paper contrasts the inexpensive LUV method with DINOv2, which it describes as adding approximately 86 million parameters and 50–100 milliseconds of overhead.

Stage 2: Distilling Gemma 4 with Phase-Switched SFT

To generate synthetic training data, the team used Gemma-4-31B-IT as a teacher. Its system prompt encoded professional audio-description guidance: present tense, safety-critical content first, explicit spatial positions with distances, observable features only, the environment type in the first sentence, hazard and obstacle flags, active voice, and a maximum of four sentences.

The teacher generated captions for 9,646 videos: 7,985 from Charades and 1,661 from AVCaps.

The phase switch addresses a training risk the authors identify as early overfitting to misaligned features. They implemented a single training run with two phases:

1. Visual Alignment, first 25% of steps: Only the SigLIP encoder and multimodal connector were trainable. LoRA adapters remained frozen so the visual representations could first align with the audio-description distribution.

2. Full Adaptation, remaining 75%: All LoRA adapters were unfrozen, and the vision encoder learning rate was halved to 5×10⁻⁶. A custom DifferentialLRTrainer managed the separate AdamW parameter groups.

Prompt tokens—including system instructions, user text, and image tokens—were masked to -100, so the causal-language-model loss was computed only over assistant response tokens.

Stage 3: GRPO with a Deterministic Reward

Supervised fine-tuning teaches the model the structure of audio description. The team then used Group Relative Policy Optimization (GRPO) to optimize BLV-specific behavior.

For each prompt, GRPO sampled four completions and normalized their rewards within the group. Instead of using a learned reward model, the authors constructed six deterministic, rule-based components that could be computed in under one millisecond per sample:

• Caution or hazard awareness: +0.40 when both reference and generation contain the caution item; -0.30 when the reference contains it but the generation omits it; -0.10 when the generation introduces it without reference support.
• Directional grounding: +0.08 for each word from {left, right, center, ahead, behind}, capped at +0.24.
• Metric distance: +0.20 for at least one matching metric distance.
• Structural economy: +0.16 for two to four sentences; -0.10 for more than six sentences.
• Hedging penalty: -0.20 for phrases such as "appears to be" or "I cannot."
• Environment identification: +0.10 when the first sentence names the scene type using one of fourteen environment terms.

Together, these rules define the paper's BLV-specific reward.

Stage 4: The Recovery Patch

The paper states that multi-stage post-training can induce catastrophic forgetting. To recover general descriptive quality while preserving BLV-specific spatial grounding, the authors applied a lightweight SFT patch: three epochs on 50 curated Gemma captions, using the GRPO checkpoint as the base and a fresh LoRA adapter with rank 64, alpha 128, learning rate 5×10⁻⁵, batch size 2, and gradient clipping at 1.0.

At inference time, the SFTv2, GRPO, and SFT-patch adapters were merged sequentially into the base weights, producing a unified model with no additional inference overhead.

The reported metrics show a trade-off rather than a universal improvement. In Table 8, MCF Ambience is 3.44 for the GRPO condition and 2.94 for the final patched condition. In Table 7, Action coverage falls from 88% to 62%. At the same time, the patched condition achieves the highest score on seven of eight LLM-judge sub-dimensions and produces the strongest results on many keyword and NLP measures.

How Navigation Post-Training More Than Doubled the OCR-Bench Score

The most striking result is what happened to the model's general visual capabilities.

Model	Parameters	OCR-Bench ANLS	TextVQA Accuracy	TextVQA ANLS
SmolVLM2-256M (base)	256M	16.49%	1.08%	3.09%
SmolVLM2-500M (base)	500M	32.54%	37.75%	47.52%
SmolVLM2-2.2B (base)	2.2B	25.44%	27.51%	29.64%
LLaVA-1.5-7B	7B	28.78%	47.79%	62.96%
PaliGemma-3B	3B	69.76%	76.09%	87.36%
Smol-VL-BLV (Ours)	500M	65.54%	54.47%	65.97%
Qwen2.5-VL-3B	3B	88.56%	81.12%	90.39%

The OCR-Bench score climbed from 32.54% for the baseline SmolVLM2-500M to 65.54% for Smol-VL-BLV. TextVQA accuracy rose from 37.75% to 54.47%.

The fine-tuned model outperformed the 2.2B-parameter SmolVLM2 baseline on OCR by roughly 40 percentage points. It also scored higher than LLaVA-1.5-7B on both OCR-Bench ANLS and TextVQA accuracy, although larger models such as Qwen2.5-VL-3B remained ahead overall.

Why does training on spatially grounded scene descriptions help a network read words?

The paper's discussion points to the content of the teacher data. The Gemma-generated descriptions included the state of the environment, nearby hazards, directional cues, a person's position, clothing colors, and movement. The authors argue that this spatially grounded supervision encourages the small model to build spatial maps and creates structural alignment that can sharpen its parsing of geometric boundaries in characters and text strings.

That is the paper's proposed explanation for the transfer. The reported experiments demonstrate the OCR and VQA gains, but do not isolate a particular attention mechanism or establish that navigation and OCR always rely on the same internal representations.

Dissecting the Navigational Gains

Across a 458-sample evaluation set, the authors measured BLV keyword coverage alongside standard NLP metrics. The cumulative results show distinct contributions from each training stage, with trade-offs across some dimensions.

Metric	Condition A (Base)	Condition B (SFT)	Condition C (+GRPO)	Condition D (+Patch)	Change A→D
Spatial Coverage	79%	81%	97%	100%	+21
Directional Cues	29%	51%	88%	100%	+71
Distance Mentions	45%	20%	22%	100%	+55
Obstacle Mentions	77%	85%	92%	96%	+19
Moving Hazards	16%	21%	17%	66%	+50
Step Changes	4%	7%	7%	5%	+1
BLEU-4	0.74	1.66	1.97	13.53	+12.79
ROUGE-L	10.74	17.66	17.93	30.49	+19.75

SFT established basic descriptive structure, lifting ROUGE-L from 10.74 to 17.66 and BLV Mean from 79% to 86%. GRPO then raised BLV Mean to 93%, obstacle coverage from 85% to 92%, and directional coverage from 51% to 88%. Its distance coverage remained 22%, however, and moving-hazard coverage fell from 21% to 17%.

The patch produced the strongest NLP scores and brought condition D's direction and distance coverage to 100%, obstacle coverage to 96%, and moving-hazard coverage to 66%. But it did not improve every measure: Action coverage fell from 88% after GRPO to 62%, and BLV Mean fell from 93% to 90%.

In a separate human evaluation, twenty sighted crowd-work raters reviewed five real-time demonstration videos. On a 0–5 scale, the final system received mean scores of 4.0 for Navigation Relevance, 3.5 for Caution/Alert Accuracy, 4.5 for Spatial and Directional Precision, 4.5 for Response Conciseness, and 4.0 for Overall Trustworthiness. These scores support the model's navigation-oriented assistance target, but they do not establish safe independent navigation.

What the OCR Transfer Supports

For lightweight multimodal research, the paper offers three narrower conclusions:

1. Spatially grounded supervision can transfer to visual-text benchmarks. The result does not show that this recipe will automatically improve every PDF, receipt, or document pipeline. It does show that, in this model and evaluation setup, BLV-focused post-training improved OCR-Bench and TextVQA alongside spatial description quality.

2. Online GRPO outperformed the tested offline preference methods in this setup. On the balanced evaluation set, DPO scored 3.89 on MCF and RLAIF-V DPO scored 3.88, both below the SFT v2 score of 4.38. GRPO reached 4.18 on MCF and 4.20 on NAF. The authors argue that its online reward was better suited to enforcing directional language, distance references, and hazard awareness.

3. Teacher-student distillation can transfer accessibility-oriented description behavior into a compact model. Gemma-4-31B-IT generated 9,646 audio-description-compliant captions, which the 507.5M-parameter student used as synthetic supervision. The paper presents this as evidence that strong teacher-generated data can induce more precise spatial grounding in a small VLM.

The sky train station was very poorly signposted — by shankar s. via flickr, BY 2.0.

Compressing Down to 442 Megabytes

Training the model used a single 48 GB RTX A6000 GPU. Running the result on a phone required a separate conversion and quantization process.

The authors merged their three trained LoRA adapters into the base weights, producing a 969 MB float16 HuggingFace checkpoint. Exporting it to GGUF separated the model into two deployment components:

• The language-model backbone: 783 MB in float16.
• The vision projector, mmproj.gguf: 191 MB in float16.

Standard Q4_K_M quantization reduced the language model to 290 MB, but the paper reports that uniform 4-bit compression degraded positional reasoning. The authors therefore used IQ4_NL importance-weighted, non-linear quantization.

First, they ran llama-imatrix on a calibration set of approximately 500 BLV captions. This recorded which weight channels were most frequently activated and produced an importance matrix for non-uniform bit allocation.

┌────────────────────────────────────────────────────────┐
│               Deployed Package (442 MB)                │
├────────────────────────────┬───────────────────────────┤
│ LM Backbone (251 MB)       │ Vision Projector (191 MB) │
│ - Self-Attention: Q5_K     │ - Native Float16          │
│ - Other Layers: IQ4_NL     │ - Unquantized             │
│ - Importance-Matrix Tuned  │                           │
└────────────────────────────┴───────────────────────────┘

Second, they applied a mixed-precision override: the self-attention projections (q_proj, k_proj, and v_proj) were quantized at the higher Q5_K precision, while most of the remaining network used IQ4_NL. The authors describe those attention mechanisms as important for preserving directional coherence.

The appendix reports that the override costs approximately 15 MB and produces a 251 MB language backbone. Paired with the 191 MB float16 vision projector, the complete on-device package totals 442 MB.

The Cold Reality of On-Device Latency

The authors deployed the 442 MB package on a Samsung Galaxy A55 with an Exynos 1480 processor and 8 GB of RAM. They compiled llama.cpp natively through Termux and ran the final configuration with four CPU threads.

In the final 21-frame benchmark, the model loaded into RAM in an average of 291 milliseconds, reached approximately 780 MB peak RAM, and produced no crashes or out-of-memory errors. Generation speed averaged 39.3 tokens per second, with generation itself taking approximately 1.4 seconds.

The bad news was the total time from visual input to completed description: 27.1 seconds.

Most of that time was prompt evaluation. Processing the visual input and prompt tokens on the A55 CPU took an average of 25.1 seconds before generation began.

Platform	Hardware Backend	TTFT (Prompt Eval)	Generation Speed	Total Latency
Cloud	Nvidia T4 GPU	1.39 s	187.42 tok/s	1.70 s
MacBook	Apple Silicon (Metal)	1.95 s	132.6 tok/s	2.41 s
Samsung A55	Exynos 1480 (CPU Only)	25.10 s	42.9 tok/s	27.10 s

The paper reports 39.3 tokens per second as the average in its 21-frame final benchmark and 42.9 tokens per second in the cross-hardware comparison, so those figures describe different benchmark summaries. In both cases, the A55's latency is dominated by CPU-only prompt evaluation.

The same quantized model completed the full multimodal run in 2.41 seconds on an Apple Silicon Mac using Metal and 1.70 seconds on a cloud T4 GPU. The authors attribute the A55's slower result to unavailable hardware acceleration support rather than to the model's generation speed alone.

The paper investigated several acceleration routes:

• Android NNAPI: The llama.cpp Android build does not expose an NNAPI execution provider. Its GPU path targets OpenCL and Vulkan, while Samsung's Exynos NNAPI driver frequently falls back to GPU for unsupported operations.
• ONNX Runtime Mobile: Hugging Face Optimum could not export SmolVLM2; its exporter raised KeyError: 'smolvlm2' because the model's SigLIP encoder and mid-sequence visual-token injection were not registered for export.
• MLC-LLM: The tested acceleration path supports LLaMA, Mistral, and Qwen models and targets Snapdragon hardware. SmolVLM2 was not included, so the Exynos device fell back to CPU.
• Vulkan and OpenCL: The Vulkan headers were available, but the vendor-specific Mali ICD loader was not. The corresponding proprietary OpenCL driver was also inaccessible from Termux.
• Samsung ONE SDK: Direct Exynos NPU programming required NDA registration and models in .nnc format produced from TFLite or ONNX. Access was not granted during the project.

The phone includes a dedicated NPU, but the tested Termux deployment could not use it. The result is a model that fits comfortably within the device's memory while relying on general-purpose CPU cores for the expensive visual-prefix computation.

The Blind Spots: What Still Fails

The authors are candid about the model's limitations. If you are designing systems for accessibility or critical spatial tasks, these gaps are vital to understand:

• Step changes remain rare: Condition D reached 100% coverage for directions but only 5% for step changes. The authors attribute this primarily to underrepresentation of stairs, curbs, elevation changes, and related scenarios in the source datasets.
• Moving hazards remain incomplete: Moving-hazard keyword coverage reached 66% in the final condition, leaving substantial room for improvement.
• Synthetic distance estimates: Distances such as "three meters ahead" are approximate visual judgments produced by the teacher model, not sensor-measured values.
• Limited scene coverage: The dataset contains some indoor and outdoor scenes, but the authors identify roads, crowded scenes, and other difficult navigation-related environments as areas for future work.
• Low vision versus full blindness: The work primarily targets low-vision users. The authors state that the system does not yet meet the reliability required for safe independent navigation by fully blind users.
• Sighted human evaluation: The reported human evaluation used twenty sighted crowd-work raters. The authors identify evaluation by BLV users as an important next step.

Looking Forward

Smol-VL-BLV shows that a compact VLM can be specialized for spatially grounded, navigation-oriented scene narration while remaining offline and deployable on a smartphone.

By pairing teacher-generated supervision with deterministic reinforcement learning, Choudhary and his team improved BLV-specific spatial grounding and general visual-text benchmark performance in a 507.5M-parameter model.

For OCR research, the important result is a measured transfer: OCR-Bench ANLS improved by 101.5% relative to the SmolVLM2-500M baseline, while TextVQA accuracy rose by 44.2%. The authors' explanation is that spatially grounded descriptions encourage better spatial maps and geometric parsing, but the paper presents this as a result and hypothesis rather than a universal law about OCR.

The authors plan to collect more data covering stairs, curbs, elevation changes, and moving hazards; introduce a dedicated step-change reward; evaluate crowded crosswalks and busy intersections; develop geometry-based spatial-correctness metrics; and add factuality checks for incorrect directions, false hazard claims, and fabricated distances.