6-DoF Object Tracking with Event-based Optical Flow and Frames

2025-08-20

AI Summary

To address frame-rate limitations and motion blur in real-time 6-degree-of-freedom (6-DoF) pose tracking of high-speed objects, this paper proposes a tightly coupled event-RGB camera fusion framework. The method estimates instantaneous motion velocity from asynchronous event streams and refines global 6-DoF pose predictions from single RGB frames, integrating both modalities via a unified sensor-fusion and state-integration pipeline. Crucially, it is the first to deeply integrate event-based optical flow with RGB-driven global pose estimation within a single coherent architecture, thereby overcoming dynamic performance bottlenecks inherent to conventional frame-based vision. Extensive experiments on synthetic and real-world high-speed datasets demonstrate that the approach achieves superior tracking accuracy over state-of-the-art frame-based methods, extends operational dynamic range beyond 1000°/s, and reduces latency by approximately 60%, enabling high-precision, low-latency 6-DoF tracking under extreme motion conditions.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Tracking the position and orientation of objects in space (i.e., in 6-DoF) in real time is a fundamental problem in robotics for environment interaction. It becomes more challenging when objects move at high-speed due to frame rate limitations in conventional cameras and motion blur. Event cameras are characterized by high temporal resolution, low latency and high dynamic range, that can potentially overcome the impacts of motion blur. Traditional RGB cameras provide rich visual information that is more suitable for the challenging task of single-shot object pose estimation. In this work, we propose using event-based optical flow combined with an RGB based global object pose estimator for 6-DoF pose tracking of objects at high-speed, exploiting the core advantages of both types of vision sensors. Specifically, we propose an event-based optical flow algorithm for object motion measurement to implement an object 6-DoF velocity tracker. By integrating the tracked object 6-DoF velocity with low frequency estimated pose from the global pose estimator, the method can track pose when objects move at high-speed. The proposed algorithm is tested and validated on both synthetic and real world data, demonstrating its effectiveness, especially in high-speed motion scenarios.

Problem

• Tracking 6-DoF object pose under high-speed motion
• Overcoming motion blur and frame rate limitations
• Integrating event-based optical flow with RGB pose estimation

Innovation

• Event-based optical flow for motion measurement
• RGB global pose estimator integration
• 6-DoF velocity tracking for high-speed objects