Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions
To address the challenge of retrieving complex interactive objects (e.g., “a numbat digging on the ground”) that are difficult for non-native speakers or vocabulary-limited users to name yet sketchable—or describable yet hard to sketch, this paper introduces Composite Sketch-Text-Based Image Retrieval (CSTBIR), a novel cross-modal retrieval task. We present the first large-scale CSTBIR benchmark comprising 2 million composite queries and 108K images. To tackle this task, we propose STNet, a multi-modal Transformer architecture optimized via multi-objective learning: it jointly models sketch–image local alignment, text–image joint embedding, and hierarchical losses for object localization, cross-modal matching, and interaction modeling. Extensive experiments demonstrate that STNet significantly outperforms state-of-the-art text-based (TBIR), sketch-based (SBIR), and hybrid retrieval methods on composite queries. Both code and dataset are publicly released to advance fine-grained cross-modal understanding research.