Breaking the Hallucination Loop: How TOLA Cuts Diffusion Text Super-Resolution to One Step

Blurry text presents a particular problem for image restoration. In a degraded crop, a single missing or merged stroke can change the character a reader sees, even when the restored image looks sharp. Text image super-resolution therefore has to recover visual detail while preserving character identity.

Generic image super-resolution methods are not designed specifically for that second requirement. They can improve an image’s appearance without reliably restoring its transcription. The distinction matters most under severe degradation, when the input offers little evidence about a thin stroke or an ambiguous character.

Diffusion models offer a way to generate plausible high-frequency detail. For text, though, plausible detail is not enough: a crisp reconstruction of the wrong character is still an error.

A paper published on September 24, 2026, by Yike Xu, Yue Shi, Yong Guo, and Jiezhang Cao introduces TOLA (Text-aware One-step Latent Adaptation). It addresses two limitations the authors identify in diffusion-based text image super-resolution: the latency of iterative prediction and the risk that an incorrect text prior will repeatedly influence image restoration. In the paper’s measured setup, TOLA reduces average inference time from 9.9548 seconds for DiffTSR to 0.1319 seconds per image while improving both image-quality and recognition metrics.

Here is how the TOLA paper explains the failure mode, how its one-step path works, and what its evaluations establish.

The hallucination trap in iterative text diffusion

DiffTSR couples an Image Diffusion Model (IDM) with a Text Diffusion Model (TDM) through a Modulated Multimodal (MoM) module. Its reported inference route evaluates the IDM and TDM 200 times each to restore one low-resolution text crop.

That repetition has a measurable cost. On one NVIDIA RTX PRO 6000 GPU, the paper reports 9.9548 seconds per 128 × 512 image for DiffTSR, along with 59,213.970 G MACs for its complete restoration trajectory. Those figures come from synchronized, batch-size-one measurements on 100 fixed test images; they are not a measurement of a complete document-processing system.

The authors identify a second risk beyond latency. A text prior inferred from a degraded image can be wrong. If that prior repeatedly guides denoising, the evolving image and text predictions may reinforce the early error, producing a sharp but incorrect character. The paper presents this as a motivation for removing iterative image–text refinement, rather than tracing a specific mistaken character through individual denoising steps.

Its recognition diagnostics show the scale of the problem on CTR-TSR-Test ×4. Across 8,089 images, DiffTSR produced 18,469 character-level edits under the paper’s normalized TransOCR evaluation, including 14,657 substitutions. Among images whose predicted text prior was wrong, DiffTSR’s image-level exact-match transcription accuracy was 10.43%. These measurements show an association between unreliable priors and poor recognition; they do not establish that every edit was caused by feedback between the two models.

PRISM offers a nearby point of comparison. It uses a single image-reconstruction step, but its text-prior recovery still takes 16 Euler steps. TOLA’s route constructs its text condition once and omits TDM during inference.

Decoupling semantics from the denoising trajectory

TOLA’s central idea is that text guidance can be selected before image prediction, while remaining visual errors can be corrected afterward in the clean latent space. The two jobs do not need a shared iterative trajectory.

Its inference path has two parts:

1. Construct text guidance once. A frozen recognizer predicts tokens and token-wise confidence scores. TOLA weights the token embeddings by those scores before MoM builds the condition used by the image model.
2. Predict and correct the image latent. The adapted IDM makes one noise prediction. A compact Latent Residual Correction (LRC) module then adjusts the resulting clean-latent estimate before decoding.

For each input, TOLA runs TransOCR, the VAE encoder, MoM, the IDM U-Net, LRC, and the VAE decoder once each. TDM is not executed.

Confidence weighting for uncertain text

TOLA starts with a degraded low-resolution image $x_{LR}$ on a 128 × 512 canvas. A frozen TransOCR instance predicts text tokens and a confidence score $\alpha$ for each token. After character normalization and vocabulary mapping, TOLA weights the embeddings:

$$\alpha \odot \text{Emb}(\tilde{c})$$

The multiplication scales each predicted token’s embedding by its aligned confidence score. The paper specifies raw confidence weighting without calibration or a threshold. A low-confidence token therefore contributes a smaller embedding to MoM, which constructs the semantic condition $C_{cond}$ once.

This reduces the influence of uncertain OCR predictions; it does not guarantee that a wrong prediction will be ignored. Nor does the paper claim that confidence scores reliably identify every error. The visual latent also enters MoM’s conditioning computation, and the IDM receives the low-resolution latent alongside its noisy latent.

The ablation gives a concrete measure of the weighting’s contribution. Compared with uniform confidence values, the full confidence-weighted model improves TransOCR exact-match accuracy from 0.5258 to 0.5401, a gain of 1.43 percentage points. PSNR rises from 24.6159 to 24.8095 dB, while FID falls from 16.9615 to 12.6849. Those are results for the tested model and benchmark, not a general guarantee that uncalibrated confidence will prevent errors in other pipelines.

Single-step reconstruction from a noisy latent

The VAE encoder maps the degraded image to an unscaled three-channel latent $z_{LR}$. TOLA draws one Gaussian noise sample $\epsilon$ and constructs a noisy latent at the fixed, zero-indexed timestep $t = 999$ of DiffTSR’s 1,000-step linear schedule. Its schedule endpoints are $\beta_0 = 0.0015$ and $\beta_{999} = 0.0205$:

$$z_t = \sqrt{\bar{\alpha}t}z{LR} + \sqrt{1-\bar{\alpha}_t}\epsilon$$

This is a noisy version of $z_{LR}$, not a replacement of the input with pure noise. TOLA constructs $z_t$ directly at the selected timestep instead of traversing a reverse denoising chain.

The IDM U-Net receives the channel-wise concatenation $[z_t,z_{LR}]$ and MoM’s text condition $C_{cond}$. The authors begin with the official CTR-trained DiffTSR checkpoint, freeze its base weights, and train rank-4 LoRA adapters in eligible IDM and VAE-encoder layers. The LoRA scaling parameter is $\alpha=4$; here that symbol denotes a LoRA setting, separate from the token-confidence scores above.

The adapted IDM predicts noise $\hat{\epsilon}_{\theta}$ once. Because the inherited DiffTSR model predicts noise, TOLA computes an initial clean-latent estimate directly:

$$\hat{z}_0=\frac{z_t-\sqrt{1-\bar{\alpha}t}\hat{\epsilon}{\theta}}{\sqrt{\bar{\alpha}_t}}$$

No reverse update follows this calculation. The estimate goes straight to LRC.

Fixing structural distortions with Latent Residual Correction

One noise prediction can leave structured errors in the clean-latent estimate, particularly around thin strokes, boundaries, and ambiguous local patterns. TOLA addresses those errors with a small correction module instead of another iterative refinement process.

LRC first defines the residual between the low-resolution latent and the initial clean estimate:

$$r=z_{LR}-\hat{z}_0$$

It takes the six-channel concatenation $[z_{LR},r]$ and predicts a correction $\Delta r$. The module consists of a $1\times1$ projection from 6 to 32 channels, one residual-in-residual dense block containing three residual dense blocks, and a $1\times1$ projection back to 3 channels. Each dense block uses five densely connected $3\times3$ convolutions with growth width 16; residual scaling is 0.2.

The output projection is initialized to zero. At the start of training, LRC therefore leaves $\hat{z}_0$ unchanged. As it learns, its predicted correction changes the residual:

$$r_{\text{corr}}=r+\Delta r$$ $$\hat{z}{0,\text{corr}}=z{LR}-r_{\text{corr}}$$

The corrected latent is divided by the inherited scale $s=0.18215$ and passed through the frozen VAE decoder to produce $x_{SR}$.

LRC has 180,323 parameters. Together with 2,027,520 IDM LoRA parameters and 250,548 VAE-encoder LoRA parameters, TOLA optimizes and saves 2,458,391 adaptation parameters. That figure is a parameter count, not a file-size claim. Inference still uses the pretrained backbone.

Degraded image (x_LR)
   ├──> TransOCR ──> tokens + confidences ──> weighted embeddings ──┐
   │                                                                 │
   └──> VAE encoder ──> z_LR ──> q-sampling at t=999 ──> z_t          │
                            │                          │             │
                            └──────────> MoM <─────────┘ <───────────┘
                                          │
                                       C_cond
                                          │
                         [z_t, z_LR] ──> IDM U-Net
                                          │ one noise prediction
                                          v
                                  Clean latent estimate
                                          │
                               LRC residual correction
                                          │
                                   VAE decoder
                                          │
                                          v
                                 Restored image (x_SR)

The benchmark results: speed, fidelity, and independent checks

The authors evaluate TOLA on the 8,089-pair CTR-TSR-Test ×4 benchmark and on 200 paired real-image crops in RealCE-200. They also show qualitative results on RT50, a set of 50 real text inputs.

CTR-TSR-Test (×4 upscaling)

Under the paper’s unified evaluation protocol, TOLA beats DiffTSR on all nine reported CTR-TSR-Test metrics:

• Inference latency: 0.1319 seconds per image for TOLA versus 9.9548 seconds for DiffTSR, a 75.5× speedup in the measured batch-size-one setup.
• Restoration MACs: 297.497 G versus 59,213.970 G for the complete DiffTSR trajectory. These counts exclude VAEs and auxiliary text modules.
• PSNR: 24.8095 dB versus 20.5958 dB, a gain of 4.2137 dB.
• Image-quality metrics: SSIM rises from 0.6261 to 0.7235, while FID falls from 24.9120 to 12.6849.
• TransOCR recognition: Image-level exact-match accuracy rises from 0.4645 to 0.5401; normalized edit-distance similarity rises from 0.6519 to 0.7370.

The table also includes published PRISM numbers, but PRISM used a different BTL-test ×4 and PP-OCRv5 protocol. Its reported figures are a cross-protocol reference, so a direct numerical claim that TOLA beat PRISM on the same CTR-TSR-Test evaluation would be misleading.

The independent OCR cross-check

TransOCR has several roles in TOLA’s study: separate frozen instances provide conditioning and recognition-loss supervision, and TransOCR also supplies a primary evaluation metric. The authors therefore add PaddleOCR as an independent recognition check. PaddleOCR is not used for conditioning, training, validation, or checkpoint selection.

That check points in the same direction. PaddleOCR exact-match accuracy rises from 0.3970 for DiffTSR to 0.4626 for TOLA, a gain of 6.56 percentage points. PaddleOCR normalized edit-distance similarity rises from 0.5873 to 0.6585. An independent recognizer strengthens the recognition result, though it does not make the evaluation free of every possible dataset or protocol effect.

Real-image evaluation: RealCE-200 and RT50

For RealCE-200, the authors construct 200 paired crops from the RealCE validation split. The low-resolution and reference observations come from matched official 13 mm and 52 mm focal images, using the same text-annotation coordinates. The set is excluded from training, validation, and checkpoint selection.

TOLA leads the compared methods on all six RealCE-200 metrics. It reaches 19.2240 dB PSNR and 0.6199 SSIM, versus DiffTSR’s 17.6287 dB and 0.5970. PaddleOCR exact-match accuracy is 0.5100 for TOLA and 0.4300 for DiffTSR.

RT50 serves a different purpose. It contains 50 real inputs—17 Chinese, 17 English, and 16 numeric samples—selected to cover varied capture conditions. The paper presents qualitative comparisons on this set; it does not report a paired-reference score or a production field trial for RT50.

What the ablations actually show

The paper’s controlled variants help separate the contributions of adaptation, residual correction, and text conditioning.

First, LoRA carries much of the one-step adaptation, while LRC improves reconstruction. A frozen one-step path scores 20.0416 dB PSNR and 119.4517 FID. The LoRA-only variant reaches 24.4220 dB and 12.7420 FID. Adding LRC raises PSNR by 0.3847 dB, raises SSIM by 0.0127, and lowers LPIPS by 0.0121.

The recognition change from LRC alone is less certain. The paper’s paired analysis reports a small exact-match accuracy change whose McNemar test is not statistically significant ($p=0.584$). The evidence supports LRC’s image-reconstruction benefit more strongly than a separate recognition benefit.

Second, a parameter-matched plain correction performs worse than the tested LRC design. Replacing LRC with the paper’s plain residual-correction baseline lowers PSNR from 24.8095 to 23.9846 dB and raises FID from 12.6849 to 16.9340. That comparison shows the gain is not explained by parameter count alone. It does not isolate which individual feature of the dense-block design causes the difference.

Third, the selected timestep balances the reported metrics. In a fixed-checkpoint inference control, $t=949$ produces slightly higher PSNR than $t=999$—24.9307 versus 24.8067 dB—but worse LPIPS, DISTS, and FID. Setting $\epsilon=0$ also reduces PSNR and worsens FID relative to the default. The authors select $t=999$ for the strongest overall result among their tested settings.

What it means for document-processing pipelines

TOLA makes a narrow but useful advance: it demonstrates that diffusion-based text super-resolution can avoid an iterative text-and-image route while improving the paper’s measured restoration and recognition results. Three design lessons follow from the experiments.

1. One-step restoration changes the latency calculation

The measured drop from nearly ten seconds to 0.1319 seconds per crop is substantial. It makes TOLA more plausible for a targeted restoration stage than the compared 200-step DiffTSR route. End-to-end throughput will still depend on crop detection, batching, hardware, data movement, and the rest of an OCR system; the paper measures model inference on fixed images, not a deployed document pipeline.

2. Uncertain text guidance deserves less influence

TOLA’s confidence-weighted condition performs better than uniform weighting in the reported ablation. This supports reducing the influence of uncertain OCR tokens when they guide reconstruction. It also leaves an important boundary: weighting cannot recover information that the degraded image and recognizer both fail to provide.

3. A small adaptation state can change a large pretrained model

The authors train rank-4 LoRA adapters and LRC rather than retraining the DiffTSR backbone. Their 2.46-million-parameter adaptation state is small relative to the retained pretrained model, and they report approximately 55 hours of training on four NVIDIA RTX PRO 6000 GPUs. Whether that approach is cheaper for another team depends on access to the pretrained checkpoint, suitable data, and the cost of its own training and inference setup.

Where the evidence stops

TOLA’s diagnostics also show limits that matter when interpreting the result:

• Incorrect text priors remain difficult. When the predicted TransOCR prior matches the ground truth, TOLA’s image-level exact-match transcription accuracy is 0.9683. When it is wrong, accuracy is 0.2217, versus DiffTSR’s 0.1043 in that same wrong-prior group. Supplying ground-truth text as an oracle condition raises TOLA’s overall accuracy from 0.5401 to 0.6253. That gap shows room for better semantic guidance, without establishing which future recognizer or model would close it.

• The reported tests use a fixed canvas and defined datasets. Training and the unified paired evaluation use 128 × 512 images. The paper does not evaluate arbitrary document layouts or establish how an upstream crop detector would affect results.

• The latency result is hardware-specific. The 0.1319-second figure comes from one RTX PRO 6000 GPU at batch size one. In the same timing table, faster feedforward methods exist—for example, NAFNet takes 0.0324 seconds and TSRN takes 0.0148 seconds—though their restoration and recognition scores differ. The paper does not report CPU or serverless inference costs.

TOLA’s result is strongest on the question it directly tests: whether text-aware diffusion restoration needs repeated image and text predictions to achieve high quality. On the authors’ benchmarks, constructing confidence-weighted guidance once, making one IDM prediction, and correcting its latent residual delivers better measured results than the compared DiffTSR route at a fraction of its inference time.