SHREC 2025: Retrieval of Optimal Objects for Multi-modal Enhanced Language and Spatial Assistance (ROOMELSA)

2025-08-12

AI Summary

This paper addresses the challenging problem of fine-grained 3D model retrieval from panoramic indoor images using natural language queries in realistic, complex scenes. To this end, we introduce ROOMELSA—the first benchmark unifying scene-level language grounding and 3D object retrieval—comprising over 1,600 apartments, 5,200 rooms, and 44,000 natural language queries. Methodologically, we propose a lightweight multimodal framework built upon CLIP, jointly modeling material properties, part-level structure, and spatial context to achieve cross-modal fine-grained alignment. Experiments reveal that while coarse-grained retrieval has matured, fine-grained recognition remains highly challenging. Our method consistently achieves state-of-the-art performance across most evaluations, significantly outperforming CLIP and other baselines. This demonstrates the effectiveness of tightly coupled vision-language modeling augmented with explicit spatial awareness.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Image and Video Retrieval

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent 3D retrieval systems are typically designed for simple, controlled scenarios, such as identifying an object from a cropped image or a brief description. However, real-world scenarios are more complex, often requiring the recognition of an object in a cluttered scene based on a vague, free-form description. To this end, we present ROOMELSA, a new benchmark designed to evaluate a system's ability to interpret natural language. Specifically, ROOMELSA attends to a specific region within a panoramic room image and accurately retrieves the corresponding 3D model from a large database. In addition, ROOMELSA includes over 1,600 apartment scenes, nearly 5,200 rooms, and more than 44,000 targeted queries. Empirically, while coarse object retrieval is largely solved, only one top-performing model consistently ranked the correct match first across nearly all test cases. Notably, a lightweight CLIP-based model also performed well, although it struggled with subtle variations in materials, part structures, and contextual cues, resulting in occasional errors. These findings highlight the importance of tightly integrating visual and language understanding. By bridging the gap between scene-level grounding and fine-grained 3D retrieval, ROOMELSA establishes a new benchmark for advancing robust, real-world 3D recognition systems.

Problem

• Evaluate 3D retrieval in cluttered scenes with vague descriptions
• Bridge gap between scene-level grounding and fine-grained 3D retrieval
• Advance robust real-world 3D recognition systems

Innovation

• Benchmark for natural language interpretation in 3D retrieval
• Lightweight CLIP-based model for object retrieval
• Integration of visual and language understanding