Accelerating Physical Property Reasoning for Augmented Visual Cognition

2025-11-05

AI Summary

This work addresses the high latency (10–20 minutes) and poor deployability of conventional methods for vision-guided physical property reasoning—hindering real-time visual cognition, especially on wearable devices. We propose an algorithm-system co-optimization framework integrating rapid 3D geometric reconstruction, cross-view semantic feature alignment, and parallel multi-view encoding, augmented by eye-tracking to enable target localization under sparse visual input. Evaluated on the ABO dataset, our method achieves end-to-end inference latency under 6 seconds—accelerating prior approaches by 62.9× to 287.2×—while matching state-of-the-art accuracy in object-level physical property estimation and surpassing it in material segmentation and voxel-level inference. In real-world deployment at an IKEA store using Meta Aria smart glasses, the system demonstrates robust, low-latency on-site physical property estimation.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

This paper introduces sysname, a system that accelerates vision-guided physical property reasoning to enable augmented visual cognition. sysname minimizes the run-time latency of this reasoning pipeline through a combination of both algorithmic and systematic optimizations, including rapid geometric 3D reconstruction, efficient semantic feature fusion, and parallel view encoding. Through these simple yet effective optimizations, sysname reduces the end-to-end latency of this reasoning pipeline from 10--20 minutes to less than 6 seconds. A head-to-head comparison on the ABO dataset shows that sysname achieves this 62.9$ imes$--287.2$ imes$ speedup while not only reaching on-par (and sometimes slightly better) object-level physical property estimation accuracy(e.g. mass), but also demonstrating superior performance in material segmentation and voxel-level inference than two SOTA baselines. We further combine gaze-tracking with sysname to localize the object of interest in cluttered, real-world environments, streamlining the physical property reasoning on smart glasses. The case study with Meta Aria Glasses conducted at an IKEA furniture store demonstrates that sysname achives consistently high performance compared to controlled captures, providing robust property estimations even with fewer views in real-world scenarios.

Problem

• Accelerates vision-guided physical property reasoning pipeline
• Reduces latency from minutes to seconds for efficiency
• Enables robust property estimation in real-world scenarios

Innovation

• Rapid geometric 3D reconstruction for acceleration
• Efficient semantic feature fusion optimization method
• Parallel view encoding reduces system latency