VACoT: Rethinking Visual Data Augmentation with VLMs

2025-12-02

AI Summary

Existing vision-language models (VLMs) rely heavily on large-scale supervised training while neglecting the role of visual enhancement during inference, resulting in limited robustness on fundamental perception tasks—particularly OCR—as well as on out-of-distribution and adversarial examples. To address this, we propose VACoT, a novel inference-time framework that for the first time integrates large language model–guided chain-of-thought reasoning to dynamically select optimal post-processing enhancements (e.g., denoising). VACoT combines a local-global transformation strategy with a conditional reward-driven proxy reinforcement learning mechanism to enable precise and parsimonious enhancement decisions—without requiring additional fine-tuning. Extensive experiments demonstrate state-of-the-art performance across 13 diverse benchmarks. Furthermore, we introduce AdvOCR, a new adversarial OCR evaluation benchmark, which validates VACoT’s strong generalization under challenging, perturbed conditions.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

While visual data augmentation remains a cornerstone for training robust vision models, it has received limited attention in visual language models (VLMs), which predominantly rely on large-scale real data acquisition or synthetic diversity. Consequently, they may struggle with basic perception tasks that conventional models handle reliably. Given the substantial cost of pre-training and fine-tuning VLMs, continue training on augmented data yields limited and diminishing returns. In this paper, we present Visual Augmentation Chain-of-Thought (VACoT), a framework that dynamically invokes image augmentations during model inference. By incorporating post-hoc transformations such as denoising, VACoT substantially improves robustness on challenging and out-of-distribution inputs, especially in OCR-related adversarial scenarios. Distinct from prior approaches limited to local cropping, VACoT integrates a structured collection of general visual augmentations, broadening the query image views while reducing training complexity and computational overhead with efficient agentic reinforcement learning. We propose a conditional reward scheme that encourages necessary augmentation while penalizing verbose responses, ensuring concise and effective reasoning in perception tasks. We demonstrate the superiority of VACoT with extensive experiments on 13 perception benchmarks and further introduce AdvOCR to highlight the generalization benefits of post-hoc visual augmentations in adversarial scenarios.

Problem

• Enhances VLM robustness via dynamic inference-time visual augmentations
• Reduces training complexity and overhead with agentic reinforcement learning
• Improves performance on adversarial and out-of-distribution perception tasks

Innovation

• Dynamic inference-time image augmentation chain
• Post-hoc transformations like denoising boost robustness
• Conditional reward scheme for concise effective reasoning