FARM: Find Anything using Relational Spatial Memory

2026-06-13

AI Summary

This work addresses the challenge of accurately localizing target objects in object-rich home and warehouse environments based on natural language instructions that encode semantic, visual, and spatial relationships. The paper introduces the first open-vocabulary, object-level memory system that integrates vision-language models (VLMs) with explicit symbolic representations of spatial relations. By jointly parsing semantic and spatial constraints from linguistic queries, the system enables efficient and robust real-time object retrieval. Evaluated on 44k language queries across 67 scenes, the approach substantially outperforms end-to-end and scene-graph baselines, achieving 164% and 224% relative improvements in Recall@5 and Recall@10, respectively. Further VLM-based reranking yields a 35% gain in Accuracy@1. The system has been successfully deployed on a quadruped robot to perform closed-loop manipulation tasks.

Abstract

Robots operating in homes, warehouses, and other object-rich environments need memory systems that can find specific object instances on demand. Object-level memory alone is often insufficient: scenes contain many plausibly matching objects, and users refer to the target through relations to landmarks and surrounding objects (e.g. ``the tall lamp below the dartboard and to the left of the poster''), demanding a relational spatial memory that supports retrieval through semantic, appearance, and spatial predicates over objects. To achieve this, we present FARM (Find Anything using Relational Spatial Memory), which builds, in real time at 5-10 Hz, a compact, open-vocabulary, object-level memory with geometry, visual-language descriptors, and viewpoint evidence. At query time, FARM uses VLMs to parse the query and score visual evidence, while grounding spatial constraints explicitly through object symbols and relational predicates. This structured use of VLMs enables more accurate and robust retrieval than end-to-end reasoning over frame histories or scene-graph context. In experiments on 44k language queries spanning 67 indoor and outdoor scenes, ranging from 15 to 15,000 m^2, FARM improves Recall@5 and Recall@10 over prior methods by 164% and 224%, and a final VLM reranking stage improves Accuracy@1 by 35%, while running in real time. We further demonstrate closed-loop deployment on a quadrupedal robot using onboard sensors and compute.

Problem

• relational spatial memory
• object retrieval
• visual-language models
• spatial reasoning
• robotic memory systems

Innovation

• Relational Spatial Memory
• Visual-Language Models
• Object Retrieval
• Real-time Scene Understanding
• Robot Memory Systems