Reinforced Attention Learning

2026-02-04

AI Summary

This work addresses the challenge that multimodal large language models often fail to improve—or even degrade—their perceptual capabilities during reinforcement learning–based post-training due to generating excessively verbose reasoning trajectories. To overcome this limitation, the paper introduces a novel paradigm termed “attention policy,” which, for the first time, applies policy gradient reinforcement learning directly to optimize the model’s internal cross-modal attention distributions rather than its output sequences. The approach further incorporates online attention distillation to enhance modality alignment. Evaluated across multiple image and video benchmarks, the proposed method significantly outperforms strong baselines such as GRPO, demonstrating consistent gains in both perception and reasoning abilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Post-training with Reinforcement Learning (RL) has substantially improved reasoning in Large Language Models (LLMs) via test-time scaling. However, extending this paradigm to Multimodal LLMs (MLLMs) through verbose rationales yields limited gains for perception and can even degrade performance. We propose Reinforced Attention Learning (RAL), a policy-gradient framework that directly optimizes internal attention distributions rather than output token sequences. By shifting optimization from what to generate to where to attend, RAL promotes effective information allocation and improved grounding in complex multimodal inputs. Experiments across diverse image and video benchmarks show consistent gains over GRPO and other baselines. We further introduce On-Policy Attention Distillation, demonstrating that transferring latent attention behaviors yields stronger cross-modal alignment than standard knowledge distillation. Our results position attention policies as a principled and general alternative for multimodal post-training.

Problem

• Multimodal LLMs
• Reinforcement Learning
• Attention Optimization
• Post-training
• Perception

Innovation

• Reinforced Attention Learning
• attention optimization
• multimodal alignment
• policy gradient
• attention distillation