HeatTok: Enhancing Remote Sensing Image Understanding via Thermodiffusion-based Tokenization

2026-08-23

AI Summary

为解决遥感图像中物体分割不准确的问题，提出基于热扩散聚合的HeatTok方法，生成语义独立、对象对齐的不规则token，并设计G-MRoPE增强空间感知。

Abstract

Current visual tokenizers in Multimodal Large Language Models (MLLMs) predominantly rely on patch-based partitioning, which causes severe semantic mixture and object fragmentation in remote sensing imagery due to the irregular contours of geo-objects. Moreover, existing adaptive methods struggle to extract precise object-level tokens and lack dedicated geometric positional encodings for irregular regions. In this paper, we propose HeatTok, a semantic-aware tokenizer driven by thermodiffusion aggregation. Inspired by the physical principles of heat conduction, HeatTok adaptively merges adjacent homogeneous regions to generate semantically independent, object-aligned irregular tokens. To enable MLLMs to perceive these irregular shapes, we design the Gaussian Multimodal Rotary Positional Embedding (G-MRoPE), which models token spatial distributions via 2D Gaussians and explicitly injects center, scale, and orientation cues. Extensive evaluations on the VRSBench and EarthVQA datasets demonstrate that HeatTok effectively preserves object-level semantic integrity and achieves state-of-the-art performance under a reasonable token budget. The code is available: https://github.com/YingyingYan1/HeatTok.

Problem

• visual tokenizer
• remote sensing imagery
• semantic mixture
• object fragmentation
• geometric positional encodings

Innovation

• Thermodiffusion-based Tokenization
• Semantic-aware Tokenizer
• Gaussian Multimodal Rotary Positional Embedding
• Object-level Semantic Integrity