Event-based Photometric Bundle Adjustment

2024-12-18

arXiv.org

AI Summary

This work addresses photometric bundle adjustment for pure-rotation event cameras. We propose the first end-to-end photometric BA method that jointly optimizes the camera rotation trajectory and semi-dense scene radiance directly on intensity images, leveraging a physics-based event generation model to formulate photometric residuals—without requiring intensity image reconstruction or prior initialization. The method employs Levenberg–Marquardt nonlinear optimization and introduces a sparsified Jacobian matrix explicitly designed to exploit the spatiotemporal sparsity and structural properties of event streams. Experiments demonstrate up to 90% reduction in photometric error, significantly improving rotation estimation accuracy and scene detail reconstruction quality. The approach supports high-resolution panoramic imaging and is compatible with IMU measurements as well as existing event-based rotational estimation algorithms.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: State Estimation
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

We tackle the problem of bundle adjustment (i.e., simultaneous refinement of camera poses and scene map) for a purely rotating event camera. Starting from first principles, we formulate the problem as a classical non-linear least squares optimization. The photometric error is defined using the event generation model directly in the camera rotations and the semi-dense scene brightness that triggers the events. We leverage the sparsity of event data to design a tractable Levenberg-Marquardt solver that handles the very large number of variables involved. To the best of our knowledge, our method, which we call Event-based Photometric Bundle Adjustment (EPBA), is the first event-only photometric bundle adjustment method that works on the brightness map directly and exploits the space-time characteristics of event data, without having to convert events into image-like representations. Comprehensive experiments on both synthetic and real-world datasets demonstrate EPBA's effectiveness in decreasing the photometric error (by up to 90%), yielding results of unparalleled quality. The refined maps reveal details that were hidden using prior state-of-the-art rotation-only estimation methods. The experiments on modern high-resolution event cameras show the applicability of EPBA to panoramic imaging in various scenarios (without map initialization, at multiple resolutions, and in combination with other methods, such as IMU dead reckoning or previous event-based rotation estimation methods). We make the source code publicly available. https://github.com/tub-rip/epba

Problem

• Bundle adjustment for purely rotating event cameras
• Direct photometric error using event generation model
• Handling large variables with sparse event data

Innovation

• Uses event generation model for photometric error
• Leverages sparsity for efficient Levenberg-Marquardt solver
• Directly processes brightness map without event conversion