🤖 AI Summary
Existing training-free methods struggle to accurately parse referring expressions in remote sensing images that involve spatial, comparative, and ordinal relations (e.g., “the largest ship”). This work proposes GeoSelect, a training-free referring segmentation framework that leverages a frozen text-to-text language model to compile such expressions into typed spatial programs. These programs undergo well-formedness validation before being executed by a deterministic interpreter. GeoSelect integrates continuous geometric fields with discrete set and sorting operators, yielding interpretable intermediate outputs, and employs a reliability ladder mechanism to guarantee a valid segmentation for any input. Evaluated on RRSIS-D and RISBench, GeoSelect achieves mIoU scores of 58.86 and 55.27, respectively—more than doubling the performance of the current best training-free approaches—while requiring no referring supervision and operating on a single GPU.
📝 Abstract
Referring remote sensing image segmentation isolates the object named by a natural-language expression in an aerial image. Existing training-free methods resolve the expression through implicit vision-language activations or region-text similarity, which gives weak control over the spatial, comparative, and ordinal relations that dominate aerial referring: they cannot represent constructions such as the largest ship or the second court from the left. We propose GeoSelect, a training-free pipeline that reframes referring as the execution of a typed spatial program. A frozen, text-only language model synthesises the expression into a small domain-specific language, a well-formedness checker accepts the program, and a deterministic executor runs it. The central abstraction is a single scored candidate set type under which every operator composes: continuous geometric fields realise position and proximity as dense pixel-level maps, while discrete set and order operators add the extremum, ordinal, counted-union, and relational constructions that fields alone cannot express. Because execution is explicit, every intermediate program, field, and ranking is inspectable, and a reliability ladder degrades any failing program to a field-only special case, so every expression returns an answer. GeoSelect attains 58.86 mIoU on RRSIS-D test and 55.27 mIoU on RISBench test, more than twice the best prior training-free method on RRSIS-D, with no referring supervision and on a single GPU. A controlled comparison with candidates and segmenter fixed attributes the gain to explicit execution, not the backbone; an oracle decomposition localises the residual gap to detection recall on RRSIS-D and selection on RISBench, and an exposure audit confirms robustness to pretraining leakage. Code will be released upon acceptance at the project page https://avalon-s.github.io/GeoSelect/.