Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Robotics

2026-07-14

AI Summary

This work addresses the challenge of “perceptual saturation” in traditional 3D scene graph construction methods on edge robots, where redundant perception undermines real-time performance and low-latency requirements for long-horizon tasks. To overcome this limitation, the authors propose JITOMA, a novel task-driven, on-demand activation framework that leverages task heatmaps to guide perceptual streams and dynamically trigger dense descriptions and functional reasoning over local subgraphs. By tightly integrating task reasoning, perception, and memory in a closed loop—augmented with lightweight anchor maintenance and large language model–driven cognitive query parsing—JITOMA substantially reduces both active graph size and image captioning latency. The approach sustains stable processing performance across diverse, long-duration multitask scenarios and is accompanied by the release of JITOMA-Bench, a new evaluation benchmark.

Abstract

While 3D Scene Graphs (3DSGs) provide crucial structured representations for embodied agents, conventional Ahead-of-Time, build-everything-then-filter pipelines conflict with the real-time, low-latency demands of edge platforms, inducing a perceptual saturation effect via severe observation redundancy. To resolve this, we present JITOMA (Just-In-Time On-demand Memory Activation), a closed-loop framework that unifies task reasoning, perception, and memory into a just-in-time growth process. Instead of exhaustively mapping the entire environment, JITOMA leverages a top-down task heatmap at the frontend to filter continuous observations, routing minimal streams to maintain a global foundation of low-cost, dormant anchors. Upon a cognitive query, the backend Large Language Model (LLM) parses the robotic intent to dynamically awaken task-relevant anchors, triggering resource-intensive operations -- such as dense node captioning and functional inference -- exclusively within the activated local subgraph. To evaluate these dynamic capabilities and study perceptual saturation trade-offs, we introduce JITOMA-Bench, a comprehensive suite for long-horizon multi-tasking and complex multi-step reasoning. Extensive experiments demonstrate that JITOMA substantially reduces active graph size and captioning latency, while maintaining stable processing time under long-horizon task switching.

Problem

• perceptual saturation
• 3D Scene Graphs
• long-horizon robotics
• real-time perception
• observation redundancy

Innovation

• Just-In-Time Scene Graph
• Perceptual Saturation
• On-demand Memory Activation
• Task-driven Perception
• Large Language Model (LLM)