To Deceive is to Teach? Forging Perceptual Robustness via Adversarial Reinforcement Learning

2026-01-24

AI Summary

This work addresses the susceptibility of multimodal large language models (MLLMs) to perceptual fragility and hallucination in complex visual scenes, as well as their reliance on static and costly training data. To this end, the authors propose the Adversarial Opponent Training (AOT) framework, which introduces self-play reinforcement learning into MLLM robustness training for the first time. AOT dynamically generates adversarial examples through the co-evolution of an image-editing attacker and an MLLM defender, establishing a scalable training loop. Combined with supervised fine-tuning and a large-scale adversarial dataset, AOT-SFT, the approach significantly enhances the model’s perceptual robustness in complex scenarios and effectively suppresses hallucinations, demonstrating the effectiveness and scalability of the AOT framework.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Despite their impressive capabilities, Multimodal Large Language Models (MLLMs) exhibit perceptual fragility when confronted with visually complex scenes. This weakness stems from a reliance on finite training datasets, which are prohibitively expensive to scale and impose a ceiling on model robustness. We introduce \textbf{AOT-SFT}, a large-scale adversarial dataset for bootstrapping MLLM robustness. Building on this, we propose \textbf{AOT (Adversarial Opponent Training)}, a self-play framework that forges MLLM robustness by creating its own training data. Our method orchestrates a co-evolution between an image-editing Attacker and a Defender MLLM, where the Attacker generates a diverse and dynamic curriculum of image manipulations, forcing the Defender to adapt and improve. Extensive experiments demonstrate that AOT enhances the Defender's perceptual robustness and reduces hallucinations, establishing a scalable paradigm for training more reliable MLLMs.

Problem

• Multimodal Large Language Models
• perceptual fragility
• visual complexity
• training data limitation
• model robustness

Innovation

• Adversarial Reinforcement Learning
• Multimodal Large Language Models
• Self-play Training
• Perceptual Robustness
• Adversarial Dataset