Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models

2026-05-29

AI Summary

研究通过引入FaithC4基准测试，评估了视觉语言模型在处理不完美文本时的转录准确性问题，发现其倾向于重写文本而非忠实转录。

Abstract

Vision Language Models (VLMs) are increasingly used in place of traditional OCR pipelines for document understanding. In this paper, we show they do not always act as faithful transcribers: when text is imperfect, they often tend to rewrite it into a more plausible form - a behavior that clean-text OCR benchmarks cannot detect. We introduce FaithC4, a multilingual perturbation benchmark of 1,455 single-page documents (English, Chinese, Korean) with three perturbation families: scramble, random substitution, and visually similar substitution. We use the benchmark to evaluate 15 systems spanning general-purpose VLMs, OCR-specialized VLMs, and traditional OCR pipelines. These three categories differ in WER degradation under perturbation: general-purpose VLMs degrade by up to 4.5 points, OCR-specialized VLMs by 0.2-2 points, and traditional OCR by less than 0.6 points on English. Probing Qwen3-VL-4B layer-by-layer, we identify a consistent pattern: rewriting fires only when a perturbed word's final layer FFN representation stays close to the original encoding; when the representation diverges sufficiently, the model transcribes faithfully. Word length affects rewriting rate: short words (4-6 characters) are rewritten up to 10% of the time, with a sharp cutoff at 8 characters above which rewriting drops to 0%.

Problem

• Vision Language Models
• Transcription Faithfulness
• Perturbation Benchmark
• OCR Pipelines

Innovation

• FaithC4
• Transcription Faithfulness
• Vision-Language Models
• Perturbation Benchmark