The Hidden Capture-Device Bias in an OCR Benchmark: What VeriCam Measures Across Devices

A pre-trained PARSeq-tiny model in the VeriCam paper reached 95.98% whole-plate accuracy in its intra-device LPLCv2 evaluation. On “Perfect” plates, it reached 99.01%; on “Illegible” plates, 68.18%.

In the paper’s cross-device evaluation, where the test capture devices were not present during training, those figures changed. Overall plate accuracy was 94.26%, while performance on Illegible plates was 50.76%: a 17.42-percentage-point difference from the intra-device result.

Those experiments are not a production deployment test, and the paper does not isolate the precise visual cues responsible for the gap. But they do support the authors’ central warning: capture-device contamination can skew evaluation in traffic-surveillance OCR datasets.

A paper from the Federal University of Paraná (UFPR) and the Paraná Military Police, titled VeriCam: A Verification Baseline for the Classification of Unknown Data (arXiv:2608.31107, accepted at SIBGRAPI WIP 2026), studies this issue directly. Authors Lucas Wojcik, Gabriel E. Lima, Sergio M. Silva Jr., Eduil Nascimento Jr., and David Menotti argue that LPLCv2 carries an inherent capture-device bias that poses a cross-device generalization challenge for downstream license-plate recognition tasks, including OCR.

Their proposed pipeline, VeriCam, trains a verification model to compare image pairs, then builds a graph and clusters unknown capture-device classes without requiring test-time class labels. In the cross-device scenario, the paper reports an F1-score of 93.45 for verification. Table V reports a V-Measure of 80.14 for its Leiden clustering setup without known labels.

Capture-device bias inside surveillance datasets

The paper builds on prior work by Laroca et al. that found a very small CNN could identify which dataset a traffic image came from in a standard classification task. VeriCam examines a related problem within one dataset: capture-device contamination between partitions.

When an image-level split places images from the same capture device in training, validation, and testing, the evaluation tests unseen images from known devices. A device-level split instead holds out capture devices entirely, making validation and testing images come from unknown devices.

The authors use LPLCv2, a traffic-surveillance dataset with 37,099 images. Of those, 34,760 are annotated with camera IDs. After retaining devices associated with at least ten images, their working dataset contains 33,668 images across 612 devices.

The dataset is substantially imbalanced:

• Concentration: Around 50% of the working dataset, or 16,644 images, comes from 15% of its classes: 90 devices.
• Long tail: 158 devices have 10–19 images each, while 62 devices have 100 or more images.

The paper’s objective is to identify capture-device groups dynamically, enabling a more device-disjoint benchmark without requiring capture-device labels at inference time.

Why fixed-label classification and OOD detection are insufficient

The paper frames its problem as classification when the number of target classes is unknown and not fixed in advance.

1. Fixed-label classification has a fixed output space: Standard classifiers map inputs to a predefined number of classes. They do not directly assign samples to previously unknown classes.

2. OOD detection does not discover unknown classes: OOD methods can extend an (N)-class problem to an (N+1) setting by flagging samples outside known classes. But detecting an unknown sample is different from grouping multiple unknown samples into their underlying classes.

3. Fine-grained separation remains difficult: The authors argue that image and text foundation models, as well as vision-text hybrids, may lack the specialized representational power needed when visually similar classes differ in fine-grained details.

Other zero-shot approaches may transfer shared features or use class prototypes. VeriCam instead reframes the task as local pairwise verification followed by clustering.

Reframing classification through verification

Rather than asking a model to select one capture-device class from a fixed list, VeriCam estimates whether two images belong to the same latent class.

The approach draws on verification-based feature learning used in face recognition. VeriCam uses a Vision Transformer as a feature descriptor, estimates pairwise relationships, and turns those relationships into a graph.

[ Unseen Traffic Images ]
           │
           ▼
[ ViT-B/16 Verification Model ]
           │
      (Feature Vectors)
           │
           ▼
[ Pairwise Cosine Similarity ]
           │
      (Affinity Graph)
           │
           ▼
[ Naive or Leiden Clustering ]
           │
           ▼
[ Discovered Capture-Device Classes ]

The metric-learning backbone

The authors train a ViT-B/16 architecture from scratch on the verification task using triplet loss and dynamically generated training and validation triplets.

Images are resized to fit a (224 \times 224) square while preserving aspect ratio, centered, and padded with gray pixels.

The paper defines cosine distance between feature vectors as:

$$\text{CosDist}(V_1, V_2) = 1 - \text{CosSim}(V_1, V_2) = 1 - \frac{V_1 \cdot V_2}{|V_1| |V_2|}$$

Cosine similarity ranges from (-1) to (1); the corresponding cosine distance ranges from 0 for equal vectors to 2 for completely different vectors.

The reported training configuration includes:

• Maximum epochs: 1,000.
• Batches per epoch: 120.
• Early stopping: Patience of 40 epochs, monitored by validation accuracy.
• Optimizer: Adam.
• Initial learning rate: (1\times10^{-4}), with a 0.75 reduction factor after five epochs of patience and a minimum of (1\times10^{-6}).
• Compute: NVIDIA RTX 6000 GPU.

From pairwise similarities to graphs

After feature extraction, VeriCam represents the image set as an undirected graph (G=(V,E)). Each vertex represents an image, and edge weights represent cosine similarity between connected image pairs.

The authors note that spectral clustering appears prohibitively expensive for their testing sets, which contain more than 6,000 images. They therefore evaluate two approaches.

1. The naive algorithm

The naive algorithm processes instances sequentially.

1. It compares each new image with images in the known set.
2. It calculates the mean similarity between that image and each known class.
3. If every class mean is below the threshold, the image initializes a new class.
4. Otherwise, it is assigned to the class with the highest mean similarity.

The threshold is set to 0.6. The algorithm can begin with known instances and labels or with no prior class information.

2. Leiden graph clustering

The second approach uses the Leiden algorithm for graph clustering. The paper uses the Constant Potts Model quality function with a resolution parameter of 0.8.

The authors selected this setup because they expect tightly connected communities when verification accuracy is high. As with the naive method, the task is treated as label-agnostic clustering.

The experimental setup: image-level versus device-level splits

The 33,668 working images are divided using 60/20/20 training, validation, and testing partitions.

Scenario	Partition	Instance Count	Device Count
Intra-Device	Training	20,200	612
	Validation	6,734	612
	Testing	6,734	612
Cross-Device	Training	20,914	368
	Validation	6,477	122
	Testing	6,277	122

In the intra-device scenario, each device’s images are divided across the three partitions. The images are unseen at test time, but the devices are known.

In the cross-device scenario, splitting occurs at the device level. Devices in validation and testing are not present in training, so those partitions contain unknown capture devices.

The numbers: high homogeneity, lower completeness

The verification model was evaluated on a static set of 50,000 random test-image pairs, balanced between genuine and impostor pairs.

Verification baseline results

Scenario	Accuracy (%)	Precision (%)	Recall (%)	F1-Score
Intra-Device	98.13	98.87	97.36	98.11
Cross-Device	93.79	97.83	89.52	93.45

Cross-device recall fell from 97.36% to 89.52%. The paper identifies the resulting false negatives as an important limitation: genuine pairs from the same device are more often rejected when those devices were not encountered during training.

Naive algorithm performance

Scenario	Initialization Mode	V-Measure (%)	Correct Attributions (%)
Intra-Device	None	89.68	64.52
	Training Set	92.30	82.09
Cross-Device	None	73.45	43.10
	Training Set	72.51	33.90

For known devices in the intra-device scenario, initial training information improved the naive algorithm’s results. In the cross-device scenario, the same initialization reduced both V-Measure and correct attributions. The authors attribute this to the test distribution falling outside the range of the training set.

Leiden algorithm performance

Scenario	Known Labels Provided	Homogeneity (%)	Completeness (%)	V-Measure (%)
Intra-Device	None	99.22	81.11	89.25
	Training Set	99.26	81.17	89.31
Cross-Device	None	99.18	67.23	80.14
	Training Set	95.34	63.66	76.35

In the cross-device, no-prior-information setting, the Leiden result exceeds the naive result: 80.14 versus 73.45 V-Measure.

The difference between homogeneity and completeness is central. Cross-device Leiden clusters were highly homogeneous: they rarely mixed images from different capture-device classes. But completeness was lower, meaning images from the same device were often split among multiple clusters.

The paper’s qualitative analysis connects this fragmentation to false negatives in the verification stage. Missing same-class relationships leave classes divided into separate groups, even when those groups remain internally pure.

The OCR stress test with PARSeq

The authors also evaluate PARSeq-tiny on LPLCv2. They train both from-scratch and pre-trained versions for the intra-device and cross-device scenarios, then report plate and character accuracy across the dataset’s plate-wise legibility labels.

Full PARSeq-tiny OCR evaluation

Setup	Legibility Tier	From Scratch: Plate (%)	From Scratch: Char. (%)	Pretrained: Plate (%)	Pretrained: Char. (%)
Intra-Device	Perfect	98.73	99.72	99.01	99.78
	Good	94.43	98.72	95.17	98.96
	Poor	84.82	97.21	87.12	97.63
	Illegible	65.91	91.02	68.18	91.34
	Overall	95.29	99.00	95.98	99.15
Cross-Device	Perfect	98.72	99.78	99.08	99.84
	Good	94.29	98.94	96.16	99.25
	Poor	84.52	97.06	85.60	97.38
	Illegible	56.06	85.61	50.76	82.90
	Overall	93.41	98.66	94.26	98.80

Three reported patterns stand out:

1. Perfect plates changed little across scenarios: Plate accuracy was 98.73% versus 98.72% for the from-scratch model, and 99.01% versus 99.08% for the pre-trained model.

2. Overall plate accuracy declined across devices: The from-scratch model fell 1.88 percentage points, from 95.29% to 93.41%. The pre-trained model fell 1.72 points, from 95.98% to 94.26%.

3. Illegible plates were most affected: The from-scratch model fell 9.85 points, from 65.91% to 56.06%. The pre-trained model fell 17.42 points, from 68.18% to 50.76%.

The authors specifically note that low-resolution, poorly readable plates were the most affected group. Their study measures the resulting performance differences; it does not identify the particular visual features responsible for them.

What the study does—and does not—say about OCR benchmarks

VeriCam’s direct evidence concerns traffic-surveillance imagery in LPLCv2 and PARSeq-tiny license-plate recognition. It does not evaluate receipts, invoices, identity documents, commercial OCR APIs, or document-scanning hardware.

Within its stated scope, however, it establishes several useful benchmark-design observations.

1. The split protocol changes what is being measured

An intra-device split tests new images from devices already represented in training. A cross-device split tests unknown capture devices.

For the LPLCv2 OCR experiment, those two protocols produced different plate-recognition results, particularly for the Illegible subset. The paper presents dynamic device recognition as a step toward constructing fairer, less contaminated benchmarks.

2. Unknown capture-device groups can be discovered without test labels

VeriCam’s pipeline uses supervised verification training, then clusters target images in a label-agnostic way. In the tested cross-device setup, Leiden clustering achieved high homogeneity but lower completeness.

That result does not establish a general solution for every OCR corpus. It does show, in this traffic-surveillance setting, that a verification-and-clustering pipeline can recover useful capture-device structure without requiring the target classes to be known in advance.

3. Plate and character metrics tell different parts of the story

For the cross-device pre-trained PARSeq result, overall character accuracy remained 98.80% while overall plate accuracy was 94.26%. On Illegible plates, the corresponding values were 82.90% character accuracy and 50.76% plate accuracy.

The table therefore shows why both character-level and complete-string metrics are relevant when interpreting this particular OCR evaluation.

Hard limitations and the road ahead

The paper is explicit about its current limitations:

• Cluster fragmentation from lower recall: In the cross-device Leiden setting, completeness was 67.23% despite 99.18% homogeneity. Same-device images were often divided among multiple clusters.

• Sensitivity to underrepresented classes: The authors report that lesser-represented classes remain difficult even after restricting the dataset to devices with at least ten images.

• Need to correct past decisions: Future work includes improving the naive approach so it can use incoming test-time information to correct previous mistakes.

• Limited validation scope: The method is validated in the street-surveillance domain on LPLCv2. The authors identify expansion to novel features and other datasets as future work.

The authors have open-sourced their implementation at github.com/lmlwojcik/VeriCam.

Final thoughts

VeriCam does not establish that every OCR failure is caused by capture-device bias, nor does it test document-AI workloads beyond license-plate recognition. Its result is narrower and still important: on LPLCv2, device-disjoint evaluation produces lower PARSeq plate accuracy than image-level splitting, with the largest differences on difficult plates.

The paper’s contribution is a verification-based baseline for grouping unknown classes, alongside a label-agnostic method for identifying capture-device groups. For its traffic-surveillance setting, that provides a concrete route toward testing whether a benchmark is measuring generalization to unseen capture devices rather than only performance on new images from known ones.