LookThere! Sparse Vision by Reinforced Selection

2026-09-04

AI Summary

本文通过端到端强化学习框架LookThere解决了视觉变换器在极端稀疏条件下的高效计算问题，选择任务相关输入以减少计算量，提高性能。

Abstract

Vision transformers typically treat every image token as equally important, yet for most tasks in computer vision only a fraction are needed. Adaptive computation methods accelerate inference by choosing which tokens to process, but existing methods struggle at extreme sparsity and require heuristics that may not generalize like token diversity and attention scores. We address these limitations with LookThere, achieving a new pareto frontier in performance-compute trade-offs through an end-to-end reinforcement learning framework that jointly trains a shallow input selector and a deep representation extractor. The selector learns where to look and the extractor learns what to see, together saving computation by selecting only what is worth processing for a given task without relying on auxiliary signals. We show that LookThere only selects the task-specific input, excelling at sparse recognition in high-resolution settings (traffic signs, billiards), and maintaining accuracy with as little as 0.2% of the input. It generalizes across tasks and models, including global recognition (ImageNet classification), local recognition (ADE20K segmentation), zero-shot classification (by distillation), and regression (counting). Across all settings, LookThere surpasses state-of-the-art selection to provide a general and scalable framework for specialized and efficient adaptive computation.

Problem

• Vision Transformers
• Adaptive Computation
• Sparse Recognition

Innovation

• reinforcement learning
• adaptive computation
• sparse vision
• end-to-end training
• efficient processing