Learning to Zoom Efficiently with a Contrastive Curriculum

2026-09-02

AI Summary

本文提出了一种新的无需额外标签或预热微调的内在奖励方法，通过对比训练信号有效学习使用缩放工具，提高了处理高分辨率图像任务的效率。

Abstract

Using a zoom-in tool is an important foundational part of modern visual agents, because it allows to efficiently handle tasks involving high-resolution images. Most previous methods need an extensive warm-start supervised fine-tuning phase for teaching models zoom-in. We show that this is not necessary by proposing a new intrinsic reward for learning tool use in MLLMs without the need for additional labels or warm-start SFT. Our InfoNCE-style reward uses a curriculum of increasingly hard negative tool calls as a contrastive training signal. Empirical experiments on $V^*$, HRBench and MME-RealWorld show that our approach is competitive while being more efficient. When used as a drop-in replacement for SFT, we even outperform all baselines. To directly measure the zoom-in ability of models, we further introduce the scalable synthetic Muffin&Chihuahua (M&C) dataset. Each image consists of a grid with every cell either showing a muffin or chihuahua. Leveraging the M&C dataset's unique region of interest labels, we find that recall is the metric that most strongly correlates the zoom-in region with final task performance. Our model and code for reproduction is publicly available under https://github.com/UKPLab/emnlp2026-zoom-in

Problem

• zoom-in tool
• high-resolution images
• supervised fine-tuning
• intrinsic reward
• contrastive training

Innovation

• Contrastive Curriculum
• Intrinsic Reward
• InfoNCE-style reward
• Muffin&Chihuahua dataset