Evolving Symbolic 3D Visual Grounder with Weakly Supervised Reflection

2025-02-03

AI Summary

To address challenges in 3D visual grounding—including difficult cross-modal alignment, scarce and costly annotations, and high inference overhead of existing LLM/VLM approaches—this paper proposes EaSe, a training-free symbolic framework. Methodologically, EaSe introduces (1) an evolvable symbolic grounding paradigm that models spatial relations as executable code; (2) tight coupling between an LLM—generating spatial reasoning code—and a VLM—providing cross-modal semantic grounding; and (3) automated code evaluation with weakly supervised reflection, enabling dynamic optimization without any training. Evaluated on Nr3D and ScanRefer, EaSe achieves state-of-the-art zero-training performance with 52.9% and 49.2% Acc@0.25, respectively, while significantly reducing inference latency and computational cost compared to prior LLM/VLM-based methods.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

3D visual grounding (3DVG) is challenging because of the requirement of understanding on visual information, language and spatial relationships. While supervised approaches have achieved superior performance, they are constrained by the scarcity and high cost of 3D vision-language datasets. On the other hand, LLM/VLM based agents are proposed for 3DVG, eliminating the need for training data. However, these methods incur prohibitive time and token costs during inference. To address the challenges, we introduce a novel training-free symbolic framework for 3D visual grounding, namely Evolvable Symbolic Visual Grounder, that offers significantly reduced inference costs compared to previous agent-based methods while maintaining comparable performance. EaSe uses LLM generated codes to compute on spatial relationships. EaSe also implements an automatic pipeline to evaluate and optimize the quality of these codes and integrate VLMs to assist in the grounding process. Experimental results demonstrate that EaSe achieves 52.9% accuracy on Nr3D dataset and 49.2% Acc@0.25 on ScanRefer, which is top-tier among training-free methods. Moreover, it substantially reduces the inference time and cost, offering a balanced trade-off between performance and efficiency. Codes are available at https://github.com/OpenRobotLab/EaSe.

Problem

• 3D Visual Localization
• Data Scarcity
• Computational Efficiency

Innovation

• EaSe
• LLM-generated code
• VLM-assisted localization