AeroReformer2: Spoken-Query Referring Segmentation for Aerial Images

2026-08-09

AI Summary

This work addresses the limitation of existing referring expression segmentation benchmarks for remote sensing imagery, which support only textual queries and are thus ill-suited for voice-based interaction. To bridge this gap, we present the first benchmark for aerial image segmentation driven by spoken referring expressions, incorporating diverse accents and realistic acoustic noise. We propose an efficient dual-path network that integrates boundary-preserving visual encoding with token-conserving speech encoding, coupled with a kernelized linear cross-modal attention mechanism that avoids explicit dense alignment matrices. This design enables effective multi-scale feature fusion and high-resolution boundary refinement. Our method achieves 62.09% mIoU and 68.22% oIoU on clean test data, substantially outperforming the strongest baseline, and maintains superior performance under noisy conditions with 54.09% mIoU.

Abstract

Spoken language offers a natural, hands-free interface for specifying an arbitrary target in dense remote-sensing imagery, yet existing referring remote-sensing image segmentation benchmarks accept only written expressions. To bridge this gap, we introduce \dataset, a spoken-query benchmark derived from RISBench that adds accent- and voice-diverse speech while preserving the original image, mask, and data splits. Its hard evaluation sets combine rotor, wind, and mixed interference with three signal-to-noise levels. We also propose \model, an efficient bilateral network that combines a boundary-preserving visual path with token-preserving speech encoding, kernel linear cross-modal attention, and a resolution refinement head. The design conditions visual features at two scales without materializing a dense speech--visual affinity matrix, then restores fine boundaries using high-resolution visual features. On the clean test split, \model with Swin-Base achieves 62.09\% mean intersection over union (mIoU) and 68.22\% overall intersection over union (oIoU), outperforming the strongest audio-adapted remote-sensing baseline by 5.38 and 2.08 percentage points, respectively. It retains the best hard-set mIoU at 54.09\%. To the best of our knowledge, this is the first benchmark and model study of full-sentence spoken-query referring segmentation for remote-sensing imagery. The code will be made publicly available.

Problem

• spoken-query
• referring segmentation
• aerial images
• remote-sensing
• speech-language interface

Innovation

• spoken-query referring segmentation
• aerial image segmentation
• cross-modal attention
• boundary-preserving network
• remote-sensing benchmark