Video Scene Text Editing Still Breaks: How ViTeX-Bench Exposes the Trade-Offs

Imagine a five-second video of someone carrying a cardboard box down a hallway. Now try to change the printed label on that box from "FRAGILE" to "STORAGE".

It sounds like a small request. Video generation has become increasingly realistic and controllable. Yet replacing seven letters on a moving surface while keeping the rest of the video intact brings together several problems that current editors still struggle to solve at once.

An editor may produce a polished clip with the wrong characters. Another may spell the replacement correctly in individual frames, but let the glyphs jitter as the video plays. Still another may preserve the scene and its motion while leaving the original text in place.

This breakdown of video scene text editing is the focus of ViTeX-Bench, a paper posted to arXiv on September 30, 2026, by Xinghao Chen, Xiangbo Gao, Jiongze Yu, Yuheng Wu, and Zhengzhong Tu from Texas A&M University. The authors built a 387-video benchmark dataset, designed a 13-metric evaluation protocol anchored by optical character recognition (OCR), evaluated eight baseline systems, and introduced an open-source reference editor called ViTeX-Edit-14B.

For anyone working in document AI, video understanding, or visual editing, the paper offers a sobering look at the gap between producing convincing video and making a precise, localized change.

The Trilemma: Why Replacing Video Text Is So Deceptively Hard

To understand the task, look at the three requirements every video text edit must satisfy simultaneously:

1. Character-level correctness: The model must render the requested target sequence. Dropped characters, substitutions, and plausible-looking squiggles can all defeat the edit.
2. Temporal consistency: The text must stay attached to the underlying surface. If the camera pans or the object tilts, the replacement must follow the scene's motion and perspective while maintaining a coherent appearance.
3. Edit locality: The surrounding video should remain unchanged. Background geometry, lighting, and moving subjects should survive the text replacement intact.

In static image editing, systems such as FLUX-Text can render accurate characters in individual frames. But video adds time. Independent frame edits provide no explicit temporal coupling, while propagating a first-frame edit can introduce fading or drift. Video-native editors model motion, but that does not guarantee control over an exact character sequence.

Existing resources address parts of this problem. General video editing benchmarks measure instruction following, visual quality, temporal consistency, and preservation. Earlier video text replacement work also exists, including STRIVE. ViTeX-Bench's contribution is a dedicated combination of paired training data and a frozen evaluation protocol that checks requested text over time alongside temporal quality and scene preservation.

The Hall of Failures: How Today's Editors Break

The authors evaluated eight baseline systems grouped into four editing families across a frozen test split of 157 real-world 720p clips, each containing 120 frames at 24 frames per second. Their results and selected examples expose four recurring failure modes.

1. Stable but Unchanged (Wan2.1-VACE-14B)

Wan2.1-VACE-14B is a mask-conditioned video editor. In the paper's "Change NEW to OLD" example, it returns a text region that remains essentially as it appeared in the source.

Across the evaluation, it achieved a sequence accuracy (SeqAcc) of 0.000 and a character accuracy (CharAcc) of 0.298. Its temporal text stability (TTS) was nevertheless 0.689. For context, the unedited source scored 0.317 CharAcc against the requested replacement strings, showing that partial character overlap can occur without a successful edit.

The paper does not establish why the model retains the source text. It establishes the practical failure: stable decoded text and low temporal error do not demonstrate that the requested replacement happened.

2. Per-Frame Correct, Temporally Chaotic (FLUX-Text)

What happens if you take an image scene-text editor such as FLUX-Text and run it independently on every frame?

Text correctness improves substantially over several video-based baselines. FLUX-Text leads the evaluated raw editors on mean text correctness, posting 0.528 SeqAcc and 0.737 CharAcc. But its independently generated frames exhibit substantial temporal instability.

On the text-crop Warp metric (Warp_c), which measures motion-compensated pixel error using source-video optical flow, FLUX-Text scores 13.01, compared with the unmodified source video's 1.27. Its text-crop flicker score (Flicker_c) reaches 14.81, approximately four times the source value of 3.68.

The paper's selected example captures the problem: characters can be legible in individual frames while their appearance changes inconsistently between adjacent frames.

3. Polished but Incorrect (Kling Video 3.0 Omni)

Kling Video 3.0 Omni represents the instruction-guided video-to-video family. The authors queried its public web interface with source videos and a fixed editing-instruction template.

Kling achieved the highest mean MUSIQ scores among the evaluated editors, including a full-frame score of 72.23, above the source video's 70.33. But the text measurements tell a different story: SeqAcc was 0.000 and CharAcc was 0.208. In the selected "SOC" to "COC" example, the output looks plausible but does not render the requested string correctly.

These results concern the tested configuration and evaluation split. Within that scope, they make the distinction clear: a high perceptual quality score does not establish successful text replacement.

4. The Drifting Morph (TextCtrl + AnyV2V)

Another approach is first-frame editing followed by propagation: edit frame 1 with TextCtrl, then use AnyV2V with an I2VGen-XL backbone to carry the edit through time.

The paper's selected example begins with a correct first-frame edit, but the rendered text and surrounding scene drift as the clip advances. Across the evaluation, the combination scored 0.057 SeqAcc and 0.308 CharAcc.

Its locality peak signal-to-noise ratio (PSNR-loc) was 21.08 dB, while locality LPIPS reached 0.225, the highest—and therefore worst—mean LPIPS distance among the evaluated raw editors. Propagating an edit does not, by itself, guarantee either persistent text correctness or preservation of the surrounding scene.

The ViTeX-Bench Protocol: 13 Metrics on Three Axes

To turn these failures into measurable comparisons, ViTeX-Bench distributes 13 metrics across three complementary axes. Each axis has one primary metric, and the authors compare their trade-offs through a Pareto set rather than a weighted aggregate score.

+------------------------------------------------------------------------+
|                         ViTeX-Bench Evaluation                         |
+------------------------------------------------------------------------+
| Axis 1: Text Correctness      Primary: SeqAcc (Exact Substring Match)   |
| Does the text read correctly? Diagnostics: CharAcc, TTS                 |
+------------------------------------------------------------------------+
| Axis 2: Visual & Temporal     Primary: Warp_c (Motion-Compensated MAE)  |
| Is the video natural/stable?  Diagnostics: Warp_f, Flicker_f/c,         |
|                              MUSIQ_f/c                                |
+------------------------------------------------------------------------+
| Axis 3: Edit Locality         Primary: DreamSim-loc (Perceptual Dist.)  |
| Does the surrounding scene    Diagnostics: PSNR, SSIM, LPIPS            |
| stay unchanged?                                                       |
+------------------------------------------------------------------------+

Axis 1: Text Correctness (The OCR Axis)

The benchmark runs PaddleOCR's PP-OCRv5 on text-region crops. Each video is routed to a Latin, Chinese, Japanese, or Cyrillic recognizer according to the source text's Unicode blocks. Recognized boxes below confidence 0.30 are dropped, and strings undergo Unicode normalization, case folding, and whitespace and punctuation removal.

To reduce confounding when the source text is obscured or blurred, correctness is scored on source-detectable frames: frames where source OCR matches the annotated source string with similarity of at least 0.5. Of the 157 evaluation clips, 152 contained such frames. Five had none under this OCR criterion and were excluded from SeqAcc and CharAcc means; that does not establish that their text was unreadable to a person.

The authors compare strings using substring edit distance ($d_{\text{sub}}$). Unlike standard Levenshtein distance, unmatched prefixes and suffixes of the candidate OCR string are free. If the target is "BIG" and OCR returns "ABIGA", the target appears exactly inside the candidate, giving a distance of 0.

• SeqAcc (Primary, $\uparrow$): The fraction of source-detectable frames whose normalized output OCR contains an exact substring match for the target.
• CharAcc ($\uparrow$): One minus substring edit distance divided by target length, providing partial credit for near-matches.
• TTS (Temporal Text Stability, $\uparrow$): The fraction of adjacent source-detectable frame pairs with identical decoded output strings. Videos without an eligible pair are omitted.

Reported means are computed at the video level. Crucially, TTS measures decoded-string stability, not correctness: the unedited source scores 0.760 TTS despite never making the requested edit. Its score below 1 also shows that OCR itself can be unstable across frames.

Axis 2: Visual and Temporal Quality (The Motion Axis)

Scoring temporal quality requires distinguishing raw frame changes from residual differences after accounting for source motion.

For Warp, the benchmark computes RAFT optical flow on the source video. It uses that flow to backward-warp the edited frame at $t+1$ toward the edited frame at $t$, then measures Mean Absolute Error (MAE) over valid flow pixels.

• Warp_c (Primary, $\downarrow$): Warp inside a stationary bounding box enclosing the entire text-mask trajectory, expanded by a 16-pixel margin. In the authors' human study, it had a stronger Spearman correlation with temporal ratings ($\rho = -0.40$) than full-frame Warp ($\rho = -0.20$), supporting its choice as the temporal primary.
• Flicker_f / Flicker_c ($\downarrow$): Raw adjacent-frame differences without motion compensation.
• MUSIQ_f / MUSIQ_c ($\uparrow$): No-reference estimates of image perceptual quality, measured over full frames or text crops.

Full-frame MUSIQ uses all frames; crop MUSIQ uses source-detectable frames. Flicker and Warp can decrease under smoothing or nearly constant outputs, so low values must be read alongside text correctness and perceptual quality.

Axis 3: Edit Locality (The Preservation Axis)

To measure changes outside the editable region, the benchmark constructs a locality-only comparison frame: predicted pixels remain outside the mask, while source pixels replace the prediction inside it. That frame is compared with the source.

• DreamSim-loc (Primary, $\downarrow$): A learned perceptual distance used to assess differences attributable to the surrounding scene.
• PSNR-loc and SSIM-loc ($\uparrow$): Pixel-based similarity measures.
• LPIPS-loc ($\downarrow$): A learned perceptual distance complementary to the pixel-based measures.

These metrics assess preservation from different perspectives. None alone establishes successful text editing.

How the Authors Built 230 Paired Training Videos

A central obstacle in video scene text editing is the limited availability of paired real-video data. Obtaining two otherwise identical recordings with different text on the same surface is difficult.

The authors addressed this with a model-assisted, human-in-the-loop pipeline that produced 230 paired training videos from Panda-70M and InternVid footage.

1. Mask Generation: An annotator placed positive and negative keypoints using a custom SAM 3 web interface. SAM 3 propagated the mask through the video; the annotator inspected it and added corrections as needed. A $25 \times 25$ pixel elliptical kernel then dilated the mask for three iterations.
2. Target Prompting: Qwen3-VL-32B-Instruct read a first-frame crop and proposed a different target string with approximately matching length and suitable scene context. Its prompt requested avoidance of offensive, political, and trademark content, and an annotator audited the candidates.
3. Background Erasing: Removal-1.3B, a fine-tuned Wan2.1-VACE-1.3B model released with PISCO, generated a clean background video by removing text and associated effects such as shadows and highlights.
4. First-Frame Synthesis: Google's Gemini 3 Pro Image ("Nano Banana Pro") rewrote the first frame. Failed edits were retried and manually checked. A second SAM 3 pass extracted the new text patch ($p_1^{\text{new}}$).

The authors then used two paths to place that patch into the cleaned video:

• Strategy A (Alpha Composition): Applicable when the text region remains static throughout the clip. This strategy supplied 56 of the final training pairs.
• Strategy B (PISCO-Based Insertion): A fine-tuned PISCO-14B inserter used the first-frame patch as a reference, with depth from Depth Anything 3 providing geometric priors. It supplied 174 training pairs.

Those counts are construction choices, not static-versus-dynamic video counts. Dynamic videos used Strategy B exclusively; static videos were processed with both strategies, and the higher-quality result was retained.

The result is ViTeX-Dataset: 387 source clips, with 230 paired training examples and 157 frozen evaluation inputs without paired edited references. Across all 387 videos, typography is approximately 23% printed, 44% handwritten, and 33% artistic. Source strings range from 1–41 characters; target strings range from 1–36.

The paired edits are reviewed training references, not unique ground-truth renderings: the instruction does not uniquely determine font, weight, color, or lighting interaction.

ViTeX-Edit-14B: Anchoring Moving Characters with Glyph Videos

Using the 230 paired training videos, the authors adapted Wan2.1-VACE-14B into an open-source reference editor: ViTeX-Edit-14B.

Alongside target-text embeddings and source-video conditioning, it receives an explicit visual guide that follows the source text's motion.

Source Video (V) + Dilated Mask (M) ---> Video Condition Unit (VCU)
                                                        |
Target String (stgt) ---> uMT5-XXL ----------------------+
                                                        |
Source Text Region ---> Font Selection + EasyOCR        |
                    ---> CoTracker3 Tracking             |
                              |                         |
Target String ---> Render + Tracked Homography Warp     |
                    ---> Glyph Video (Gvid)              |
                    ---> Wan VAE Latents                 |
                    ---> 64 Pooled Glyph Tokens --------+
                                                        |
                                             Wan2.1-VACE-14B
                                                        |
                                                  Edited Video

The Glyph-Video Pipeline

1. Font Matching: Qwen3-VL inspects the first-frame source-text region and selects a typeface from a curated Latin font library. Non-Latin scripts use script-specific defaults.
2. Trajectory Tracking: EasyOCR detects the source-text quadrilateral in frame 1, and CoTracker3 tracks it across the remaining 119 frames.
3. Synthetic Rendering: The target string is rendered white on black at twice the detected bounding-box resolution. Framewise homographies warp it according to the tracked quadrilateral, supplying source-aligned position, scale, and perspective.
4. Conditioning Injection: A frozen Wan VAE encodes the glyph video ($G_{\text{vid}}$). Patch embedding and cross-attention pooling with 64 learnable queries produce a fixed-length glyph-token bundle ($E_G$).
5. Architectural Hook: Added condition cross-attention layers inject those tokens into eight VACE blocks at trunk layers 0, 5, 10, 15, 20, 25, 30, and 35. Zero-initialized residual output projections preserve the backbone output at initialization.

Training freezes the main DiT trunk, text encoder, and VAE, while updating the VACE branch and new conditioning modules. Flow-Matching supervised fine-tuning runs in two stages: five epochs at $720\text{p} \times 49$ frames, followed by two epochs at $720\text{p} \times 121$ frames. The reported cost is 576 GPU-hours on eight H100 80GB GPUs.

Benchmark Results

ViTeX-Edit-14B combines the highest mean CharAcc among the evaluated video-native editors with the lowest comparable text-crop Warp among raw editor outputs:

• CharAcc: 0.688, compared with VideoPainter's 0.619, Wan2.1-VACE-14B's 0.298, and Kling's 0.208.
• SeqAcc: 0.341, below VideoPainter's 0.364 among video-native editors. Several per-frame image editors also score higher.
• Warp_c: 1.53, compared with FLUX-Text's 13.01, Kling's 2.90, and the unedited source's 1.27.

The ViTeX-Edit-14B and VideoPainter correctness intervals overlap; these are differences in observed means, not established pairwise statistical significance. VideoPainter's temporal scores are excluded from ranking because its adaptation uses linear frame blending, which changes the measured residuals.

The reference editor therefore provides a useful operating point, rather than a clean sweep across the benchmark.

The Composite Control: Don't Re-Invent Background Pixels

One practical insight comes from Composite, a deterministic, training-free post-processing wrapper applied to every editor.

Some methods reconstruct entire frames, while bounding-box-local editors already copy most exterior pixels directly from the source. Composite helps separate text synthesis from background restoration through three steps:

1. Construct a surrounding band by dilating the mask with a $41 \times 41$ structuring element and subtracting the original mask.
2. Use source and prediction statistics in that band for Reinhard mean–variance color transfer in CIELAB space.
3. Blend the corrected prediction onto the source with a 4-pixel feathered boundary derived from distance transforms.

The paper reports approximately five minutes to process the 157 evaluation videos using eight CPU workers. For ViTeX-Edit-14B, Composite raised PSNR-loc from 29.08 to 42.95 dB and reduced DreamSim-loc from 0.024 to 0.002. SeqAcc changed slightly, from 0.341 to 0.345.

Across all eight baselines, Composite brought locality PSNR to approximately 43 dB. Much of this gain comes from restoring source pixels. It is not evidence that each underlying editor learned better background preservation.

Method	Family	SeqAcc ($\uparrow$)	CharAcc ($\uparrow$)	TTS ($\uparrow$)	Warp_c ($\downarrow$)	MUSIQ_f ($\uparrow$)	PSNR-loc ($\uparrow$)	DreamSim-loc ($\downarrow$)
Source Video	—	0.000	0.317	0.760	1.27	70.33	$\infty$	0.000
FLUX-Text	Image	0.528	0.737	0.326	13.01	70.26	31.49	0.012
TextCtrl	Image	0.475	0.734	0.511	2.09	70.32	41.14	0.004
RS-STE	Image	0.354	0.626	0.534	1.81	69.57	37.00	0.007
AnyText2	Image	0.280	0.633	0.382	3.95	66.68	25.56	0.043
TextCtrl + AnyV2V	Prop.	0.057	0.308	0.257	3.97	69.41	21.08	0.073
Wan2.1-VACE-14B	Video	0.000	0.298	0.689	1.56	70.54	35.21	0.007
VideoPainter	Video	0.364	0.619	0.606	3.35†	67.16	28.56	0.024
Kling Video 3.0 Omni	Video	0.000	0.208	0.641	2.90	72.23	21.18	0.061
ViTeX-Edit-14B	Video	0.341	0.688	0.648	1.53	69.64	29.08	0.024
ViTeX-Edit-14B (Composite)	Post	0.345	0.689	0.666	1.56	70.27	42.95	0.002

Source is an unranked reference, and Composite is a separately reported post-processing control. † VideoPainter's temporal values are excluded from ranking because its adaptation from 8 to 24 fps uses linear blend interpolation. Baseline Composite correctness was checked only on a sample; only the ViTeX-Edit-14B Composite pair was fully re-scored.

Why It Matters: Real-World Takeaways for Document AI and OCR

For teams interested in precise visual text editing, ViTeX-Bench suggests four practical lessons. Their reach should remain tied to what the benchmark actually tests.

1. In Generative Pipelines, OCR Can Help Judge the Result

ViTeX-Bench uses OCR as an evaluator of generated text. Kling's high MUSIQ scores coexist with zero SeqAcc, illustrating why general visual quality cannot substitute for checking the requested characters.

OCR is not an infallible judge, however. On source-detectable frames, the recognizer achieved 0.851 exact-match accuracy and 0.966 CharAcc against the existing source text. A method-blinded human transcription study agreed strongly with OCR's method-level CharAcc ranking (Spearman $\rho = 0.95$), but absolute scores differed.

The practical lesson is to combine character checks with calibration and human inspection. The paper supports OCR-anchored evaluation, not replacing every other form of assessment.

2. Multi-Metric Evaluation Matters

Wan2.1-VACE-14B belongs to the benchmark's raw-output Pareto set alongside FLUX-Text, TextCtrl, RS-STE, and ViTeX-Edit-14B—even with SeqAcc 0.

That is possible because no included competitor matches or improves on all three of its primary mean scores while strictly improving at least one. Pareto membership describes a trade-off; it does not certify a successful edit.

If an evaluation rewards preservation and stability without checking target text, unchanged outputs can look competitive. Correctness, temporal quality, and locality need to be interpreted together.

3. Separate Text Synthesis from Background Restoration

The Composite experiment demonstrates a substantial locality benefit from restoring source pixels around the generated region. It gives developers a concrete control for distinguishing synthesis quality from the consequences of whole-frame reconstruction.

It does not prove that end-to-end training is inefficient, or that compositing is universally preferable. Color transfer, feathering, and encoding can alter readability, and complete post-Composite text and text-crop temporal scores were not reported for every baseline.

For localized edits, the supported takeaway is narrower and useful: evaluate what source-pixel restoration contributes before attributing preservation gains to the model.

4. Text Editing Needs Evaluation Across Time

A correct screenshot cannot establish a successful video edit. The FLUX-Text results show how strong frame-level text accuracy can coexist with large temporal residuals, while the propagation baseline shows how an initially correct edit can drift.

This makes video-level evaluation essential for the task studied here. It also suggests questions for related OCR workflows, but ViTeX-Bench is an editing benchmark—not a direct test of document extraction, mobile scanning, or recognition systems in deployment.

Its evidence concerns whether a requested replacement remains correct and visually coherent while the original scene is preserved.

Honest Limitations: What Still Needs Work

While ViTeX-Bench provides a reproducible framework, the authors are candid about its boundaries:

• Heavy Latin-Script Bias: Of the 157 evaluation clips, 150 use Latin script. The remaining seven comprise four Chinese, one Japanese, and two Cyrillic clips. AnyText2 leads this small pooled slice with 0.295 CharAcc; the other eight raw editors, including ViTeX-Edit-14B, score below 0.19. This is a limited diagnostic, not enough for broad per-script conclusions.
• Difficult Geometry and Motion: Dense layouts, curved surfaces, severe occlusion, and extreme motion are sparsely covered. Ranking stability within the sampled clips does not establish performance beyond that distribution.
• Pipeline-Generated Training Data: The 230 paired references may retain rendering errors. Their OCR CharAcc is 0.790, while human transcription on calibration crops reaches 0.917. These measurements indicate both OCR limitations and remaining uncertainty about rendering quality. Initial target resampling and first-frame retry counts were not logged.
• Limited Independent Validation: One author conducted the method-blinded transcription study. Three non-author raters assessed 70 outputs, with much lower agreement on locality (ordinal $\alpha = 0.37$) than text correctness or temporal quality. Independent mask re-annotation covered 12 clips using the same model-assisted pipeline.
• Training Cost and Unresolved Components: Fine-tuning required 576 GPU-hours, and inference uses a 50-step pass. The paper does not report inference latency or establish real-time suitability. Controlled component ablations and training-scale studies remain future work.

What ViTeX-Bench Establishes

ViTeX-Bench gives video scene text editing a dedicated dataset and a concrete measurement protocol. Across the evaluated systems, correct characters, temporal stability, and scene preservation remain difficult to achieve together. Polished video can contain the wrong text, and accurate individual frames can still form an unstable clip.

Motion-aligned glyph conditioning gives ViTeX-Edit-14B a useful balance: the highest mean CharAcc among the evaluated video-native editors and the lowest comparable text-crop Warp among raw outputs. It does not eliminate the trade-offs or establish which architectural component causes each gain.

The paper reports releases of the dataset, evaluation code, and reference-model weights on Hugging Face and GitHub. Code and weights are listed under Apache-2.0; the dataset is CC-BY-NC 4.0 for non-commercial research, with additional upstream ShareAlike obligations for redistributed InternVid-derived portions. Together, these resources provide a reproducible starting point for studying precise text replacement in real video.