SP-DocReader: Cutting Vary-600K OCR Error Rates in Half with Difference-Aware Self-Play

When more parameters make document reading worse

Fine-tuning a 4-billion-parameter vision-language model on tens of thousands of document pages sounds like a promising recipe for better transcription. In SP-DocReader's experiments, updating the language model and OCR module together did produce lower in-distribution character error rates. Yet on held-out document benchmarks, that strategy trailed a smaller adaptation. On IIT-CDIP, which contains scanned records, joint language-model and OCR tuning produced a character error rate of 16.87%. A 0.5-billion-parameter OCR module, trained on the same pages with its backbone frozen, clocked in at 14.42%. The visual encoder and its connector remained frozen in both configurations.

That counterintuitive gap is one finding in SP-DocReader, an OCR research paper published on October 8, 2026, by Wenjie Liao, Xiaohui Song, Liangjie Zhao, and Haonan Lu, affiliated with Guangdong OPPO Mobile Telecommunications and Adelaide University. Document transcription is fundamentally unforgiving. Producing a short answer about a page is a different task from reproducing its text, including small characters and numbers. The paper's examples show what can go wrong: misreading a constitutional amendment's repeal year as 1935 instead of 1933, turning "State legislatures" into "State legislators", or transcribing a mathematical dimension formula as dim S = dim A + IL instead of dim S = dim A + |L|.

SP-DocReader takes direct aim at these residual errors. The framework combines two ideas: a modular OCR branch that keeps the underlying vision-language model frozen, and an iterative self-play training scheme that selects loss positions from discrepancies between generated and reference transcriptions.

The outcome is striking. On Qwen3-VL-4B, three rounds of difference-aware self-play cut character error rate on the Vary-600K validation set from 2.40% to 1.11%, a relative error reduction of roughly 54% compared with two-epoch supervised fine-tuning. In a separate training-budget comparison, self-play using 30,000 reference pages outperformed supervised fine-tuning using 120,000 pages on a five-task composite score. That is one-quarter of the distinct training-reference pages, not a measurement of annotation cost or total computation.

Why standard supervised fine-tuning spreads its attention

To understand SP-DocReader's motivation, look at how standard supervised fine-tuning (SFT) trains document readers.

Under standard autoregressive fine-tuning, the model receives a document image alongside a prompt, and the training pipeline computes teacher-forced cross-entropy loss across the entire target text. Every target token receives supervision. If a reference contains 800 model tokens, the objective includes all 800 positions, although their losses and gradient contributions need not be equal.

A reader that already transcribes much of a page correctly may still struggle with a smaller set of remaining errors. Full-sequence SFT does not use a generated reading to identify those positions. SP-DocReader asks whether explicitly selecting the remaining disagreements can improve adaptation after the supervised stage.

Teacher forcing also differs from generation. During transcription, the model conditions each new token on its own previous output. During supervised training, it predicts the next reference token given the ground-truth prefix. SP-DocReader brings the model's generated readings into training, while still using complete ground-truth prefixes when scoring reference tokens.

The reported supervised trajectories motivate that change. On Qwen3-VL-4B, Vary-600K CER improves from 2.50% after one supervised epoch to 2.40% after two, then rises to 2.66% after three. InternVL shows a similar reversal on this metric. Those results support investigating a different objective; they do not establish that supervised fine-tuning universally stops helping after two epochs.

Borrowing self-play without a separate reward model

SP-DocReader builds on SPIN's iterative pairing of reference responses with outputs from a previous model checkpoint. An active model learns from those pairs, then becomes the opponent for the next round.

Document transcription has a useful structural advantage over open-ended chat: the training pages have reference transcriptions. The framework can locate disagreements with those references without training a separate reward model or collecting subjective preference labels.

SP-DocReader adapts that loop for page transcription:

1. The opponent generates a draft: At round t, the framework freezes the current checkpoint, designating it as the opponent ($\theta_t$). The opponent greedily generates one reading per page, stopping at EOS or 1,024 new tokens, excluding the prompt.
2. The framework compares the draft: It aligns the opponent's generated token sequence with the known reference transcription.
3. The framework selects discrepancies: It identifies unmatched positions in both sequences. These positions determine which token log-probabilities enter the training objective.
4. Targeted optimization updates the player: A dual-loss objective updates the trainable OCR module. One component compares opponent-relative score changes for reference and generated discrepancies. The other directly supervises unmatched reference tokens.
5. The opponent refreshes: After local optimization, the updated player becomes the next opponent ($\theta_{t+1}$), generating new readings and discrepancy masks.

Readings and masks remain fixed within each round. Refreshing them between rounds lets subsequent training respond to the updated model's remaining errors, although the procedure does not guarantee that every new discrepancy is harder or subtler than the last.

Surgical error correction: longest common subsequences and dual gradients

The core technical contribution of SP-DocReader lies in how it selects and scores discrepancies. The authors introduce two complementary mechanisms: Reading Discrepancy Masking (RDM) and Focused Fidelity Loss (FFL).

Reading Discrepancy Masking (RDM)

The framework aligns the ground-truth token sequence $y$ and the opponent's synthetic sequence $\hat{y}$ directly at the model-token level.

It computes a Longest Common Subsequence (LCS) using dynamic programming over suffix pairs. When tokens match, both sequence pointers advance. When tokens disagree, the algorithm advances the pointer that preserves the larger remaining suffix LCS length. On a tie, it advances the synthetic pointer while retaining the ground-truth position.

This deterministic alignment produces two position sets, each belonging to its own sequence:

• $\mathcal{E}_y$: Unmatched positions in the ground-truth sequence.
• $\mathcal{E}_{\hat{y}}$: Unmatched positions in the generated reading.

These are model-token discrepancies, not necessarily individual characters, words, or a unique diagnosis of each recognition error.

Crucially, SP-DocReader does not feed the unmatched tokens back as disconnected text fragments. Every selected token is scored using its complete original conditioning prefix. A reference token at position 450 still uses all preceding 449 reference tokens; a generated token uses its own generated prefix. The masks select direct loss terms, while matched tokens remain part of the contextual computation.

The Reading Discrepancy Loss ($\mathcal{L}_{RDM}$)

The framework computes a contrastive objective adapted from SPIN, with motivation from the expectation gap used in Integral Probability Metrics. For each sequence, it calculates the score change between the current player and the frozen opponent:

$$\Delta_t(z) = S_\theta(z, \mathcal{E}z) - S{\theta_t}(z, \mathcal{E}_z)$$

Here, $S_\theta$ sums token log-probabilities across the selected positions, using their full original prefixes. The sum is not length-normalized. The Reading Discrepancy Loss applies a logistic loss:

$$\mathcal{L}_{RDM}(\theta, \theta_t) = \mathbb{E}\left[\log\left(1 + \exp\left(\beta [\Delta_t(\hat{y}) - \Delta_t(y)]\right)\right)\right]$$

The objective favors a larger score increase over the opponent for ground-truth discrepancies than for generated discrepancies. It is a relative comparison, rather than a penalty based solely on the absolute probability of an incorrect token.

Focused Fidelity Loss (FFL)

The contrastive objective can decrease by lowering the synthetic score without increasing the ground-truth score. Suppressing an incorrect reading therefore does not, by itself, ensure stronger support for the reference.

To add that direct pressure, the authors introduce Focused Fidelity Loss ($\mathcal{L}_{FFL}$). FFL applies teacher-forced negative log-likelihood at unmatched ground-truth positions:

$$\mathcal{L}{FFL}(\theta) = - \mathbb{E}\left[\sum{k \in \mathcal{E}y} \log P\theta(y_k \mid y_{<k}, I, x)\right]$$

Matched positions receive no direct token-loss terms. They can still influence the selected losses through their role in the conditioning context, so masking does not mean that all internal gradients or computation disappear at matched positions.

The final objective combines both terms:

$$\mathcal{L}{total} = \mathcal{L}{RDM} + \lambda \mathcal{L}_{FFL}$$

The experiments use $\lambda = 0.5$ as the default weight for direct supervision.

The authors derive the combined gradient with respect to the trainable OCR parameters:

$$\nabla_\theta \mathcal{L}{pair} = -[\beta \sigma(-u) + \lambda]\nabla\theta s_y

• \beta \sigma(-u)\nabla_\theta s_{\hat{y}}$$

Here, $s_y$ and $s_{\hat{y}}$ are the masked reference and synthetic scores, and $u=\beta[\Delta_t(y)-\Delta_t(\hat{y})]$. Under gradient descent, the two terms provide pressure to raise the reference score and lower the synthetic score, with FFL adding weight to the reference contribution. The derivation explains those roles; it does not prove reduced gradient variance or automatic computational savings from masking.

The 0.5-billion parameter sweet spot

Model architecture choices matter alongside the loss functions. Rather than updating the language backbone, the authors build on DocVLM's modular reader.

The pipeline divides responsibilities:

• Frozen visual pathway: The visual encoder processes reduced-resolution pages. The standard protocol renders PDFs at 72 DPI when physical dimensions are available; raster-only pages have their longer edge capped at 842 pixels without upsampling.
• Frozen language backbone: The language model's weights remain unchanged, although derivatives through its computation are needed to train the OCR branch.
• External OCR engine: Fixed EasyOCR supplies recognized text and bounding boxes from the same reduced-resolution page.
• Trainable OCR module: A DocFormerV2-derived encoder ingests OCR text and layout, using LayTextLLM-style interleaving of projected box embeddings and text embeddings. Learnable queries compress its output into 64 OCR query representations, which a projection maps into the frozen VLM.

The OCR encoder, projection layer, and learnable queries together account for approximately 0.5 billion trainable parameters. This is an attached OCR module, not a 0.5-billion-parameter subset of Qwen3-VL-4B.

The authors compare it with an "OCR-free" variant that tunes the language model without the OCR branch, using 3.3 billion trainable parameters, and a "full-parameter" variant that tunes the language model and OCR module together, using 3.8 billion. Despite that label, the visual encoder and its connector remain frozen in all three variants.

The results reveal the tradeoff:

• In-distribution reading (Vary-600K): Full tuning achieves 0.98% CER, compared with 1.11% for modular tuning.
• Zero-shot academic pages (DocBank): Modular tuning achieves 9.05% CER, compared with 11.28% for full tuning.
• Zero-shot scanned documents (IIT-CDIP): Modular tuning reaches 14.42% CER, compared with 16.87% for full tuning.
• Document question answering (DocVQA): Modular tuning scores 84.89 ANLS, compared with 80.41 for full tuning.

The longer trajectories reinforce that contrast. From SFT-2 through five self-play rounds, full tuning's four-task zero-shot average falls from 75.92 to 73.60, while modular tuning's rises from 75.89 to 79.81.

These results favor modular adaptation on the evaluated zero-shot tasks. They do not establish that full tuning learned a particular rendering style, or that its general reasoning capabilities deteriorated. Freezing the backbone preserves its weights; the paper measures the resulting system's document performance, not preservation of every broader capability.

What the numbers show: halved error rates and a fourfold page-budget comparison

The authors evaluate two backbones, InternVL-3.5-4B and Qwen3-VL-4B, using 30,000 English and 30,000 Chinese training pages from Vary-600K for the main experiments. Standard evaluations use reduced-resolution inputs, with separate preprocessing rules for PDFs and raster images. A resolution study also tests other settings.

Primary Transcription and QA Benchmarks

The main comparison covers the low-resolution base model, two-epoch supervised fine-tuning (SFT-2), and three subsequent self-play rounds (SP-DR-3):

Backbone	Method	Vary-600K CER (%) ↓	DocBank CER (%) ↓	IIT-CDIP CER (%) ↓	DocVQA ANLS ↑	InfoVQA ANLS ↑
InternVL-3.5-4B	Base (low-res)	4.85	13.44	19.56	72.79	50.52
	SFT-2	2.12	11.60	17.83	73.01	48.15
	SP-DR-3	1.17	9.73	15.35	74.27	48.06
Qwen3-VL-4B	Base (low-res)	5.22	12.82	18.45	78.54	52.21
	SFT-2	2.40	10.95	16.71	81.22	55.34
	SP-DR-3	1.11	9.05	14.42	84.89	58.94

On Qwen3-VL-4B, self-play cuts in-distribution CER by more than half. DocBank CER drops by 1.90 percentage points, IIT-CDIP by 2.29 points, and DocVQA ANLS rises by 3.67 points. InternVL also improves on all three transcription benchmarks and DocVQA, but InfographicVQA slips from 48.15 to 48.06.

Here, "zero-shot" means no target-benchmark adaptation during the paper's supervised and self-play stages. It does not establish that pretrained components never encountered those datasets or related documents.

Additional Parsing Benchmarks

The reported aggregate scores also improve beyond the primary datasets:

• OmniDocBench (Text Edit Distance, lower is better): Qwen3-VL-4B improves from 0.347 to 0.291; InternVL improves from 0.366 to 0.312.
• OCRBench v2 (higher is better): Qwen3 rises from 49.1 to 55.3; InternVL rises from 46.5 to 52.0. The paper uses an equal-language mean of English and Chinese scores.
• CC-OCR (higher is better): Qwen3 rises from 65.4 to 71.0; InternVL rises from 62.7 to 67.9. The paper reports an equal-track mean.

These are aggregate improvements, not evidence that every underlying subtask improves.

The Data Efficiency Dividend

In the training-budget comparison, Qwen3-VL-4B with SP-DocReader uses 30,000 distinct reference pages and achieves a composite score of 82.46. SFT-2 using 120,000 pages scores 80.73.

That composite gives equal weight to Vary-600K, DocBank, and IIT-CDIP normalized edit similarity, scaled to 0–100, and DocVQA and InfographicVQA ANLS. It therefore includes both in-distribution and zero-shot performance.

Each budget starts from the same pre-task initialization and completes its own supervised stage. The smaller-budget model does not inherit adaptation on the larger dataset. The result supports better performance with one-quarter of the training-reference pages in this comparison, while leaving computation and annotation effort as separate questions.

Resolution Robustness

The paper evaluates fixed Qwen3-VL-4B checkpoints across four source-image settings. The DPI labels apply to PDFs with known physical dimensions; raster-only pages use corresponding long-edge caps without upsampling:

• 48 DPI: SFT-2 reaches 3.41% CER; SP-DocReader reaches 1.74%, about a 49% relative reduction.
• 72 DPI: SFT-2 reaches 2.40%; SP-DocReader reaches 1.11%, about a 54% reduction.
• 96 DPI: SFT-2 reaches 1.93%; SP-DocReader reaches 0.92%, about a 52% reduction.
• 150 DPI: SFT-2 reaches 1.52%; SP-DocReader reaches 0.71%, about a 53% reduction.

The relative reduction stays within a narrow range across these settings. Both the visual branch and EasyOCR receive the changed source image, so this measures the combined system's response to resolution rather than isolating the visual encoder.

What Components Actually Matter?

The Qwen3-VL-4B component study separates the loss contributions while retaining discrepancy masks in every variant:

• Baseline (SFT-2): 2.40% Vary-600K CER and 81.22 DocVQA ANLS.
• Without the contrastive RDM loss: FFL alone reaches 1.88% CER and 83.47 ANLS. This row still uses LCS-selected reference positions.
• Without FFL: The contrastive loss reaches 1.57% CER and 82.96 ANLS. It still provides a relative incentive to raise reference scores, but lacks FFL's additional direct supervision.
• Full SP-DocReader: The combined objective achieves the best result in all five reported columns, including 1.11% CER, 84.89 DocVQA ANLS, and 58.94 InfographicVQA ANLS.

Refreshing also looks useful in the plotted comparison. A refreshed second-round checkpoint reaches an average near 90.3 after two local epochs, exceeding the fixed-reading branch's average near 89.8 after five local epochs. That average combines Vary-600K NED, DocBank NED, and DocVQA ANLS. The refresh changes the opponent, generated readings, and masks together; the plot does not isolate their individual effects or establish equal total training cost.

Why it matters for real-world document engineering

For engineering teams building document-processing pipelines, the paper offers three practical directions to investigate:

1. Consider freezing the language backbone during OCR adaptation.

   Modular tuning performs better on the evaluated zero-shot tasks while updating fewer parameters. At SP-DR-3, its reported cumulative adaptation cost is 0.33 times the complete joint-tuning SFT-2 run, compared with 1.58 times for joint tuning at SP-DR-3. The accounting includes supervised initialization, preprocessing, generation, alignment, scoring, and optimization. This supports the tested modular configuration, rather than a universal rule against language-model tuning.

2. Test discrepancy-focused training before assuming more pages are necessary.

   The 30,000-versus-120,000-page result makes error-focused adaptation worth investigating. It does not demonstrate equivalent gains for invoices, shipping manifests, or any other specific production domain. Applying the idea requires reference transcriptions and evaluation on the documents the system will actually encounter.

3. Measure the complete reader's inference overhead.

   The paper reports approximately 10% end-to-end inference overhead relative to the backbone without OCR, measured at batch size one on an NVIDIA H20. The protocol includes page processing, external OCR, encoding, and generation after model loading. That is a useful reported result, but whether the overhead is acceptable depends on the deployment. It should not be treated as a hardware-independent latency guarantee.

Where the approach hits its limits

Despite strong benchmark results, SP-DocReader has clear boundaries:

• Single-page scope: The study focuses on single-page transcription, rather than demonstrating multi-page reading or structured extraction. It does not establish performance on cross-page tables or document-level dependencies.
• Fixed output budgets: Self-play generation and evaluation stop at EOS or 1,024 new tokens, with no continuation or stitching. That budget may limit longer-page coverage. The protocol scores capped transcriptions against complete references; reaching the cap alone does not prove an omission, and emitting EOS does not guarantee completeness.
• External OCR dependency: The modular reader uses EasyOCR text and boxes alongside visual features. The paper does not separately establish how reliably it recovers content missed entirely by the external engine.
• Limited language and model evidence: Training uses English and Chinese pages, and experiments cover two backbone families. The results do not establish generalization to other languages or architectures.
• Remaining recognition errors: The qualitative examples show improvements, not perfect transcription. In the constitutional example, SP-DocReader corrects the repeal year but still reads the passage date as December 16 rather than the displayed reference's December 18. Other examples retain spelling and symbol errors.

Within those boundaries, SP-DocReader shows that discrepancy-focused self-play can improve a modular document reader beyond its supervised starting point. The strongest evidence is concrete: lower transcription error across the three primary reading datasets on both backbones, better scores on most evaluated question-answering comparisons, and a favorable reference-page budget comparison. The backbone stays frozen, while the OCR module learns from the disagreements that remain.