Cyclops: LiDAR as a Camera That Dreams in Color

2026-08-17

AI Summary

This study addresses low-light perception degradation and the cross-modal gap by proposing Cyclops, a framework that translates sparse LiDAR intensity into RGB video for all-weather, camera-free perception. The method employs a densification module and latent bridge matching for image generation, while integrating temporal attention and differentiable rewards to optimize the ODE velocity field. Experimental results demonstrate that the synthesized RGB data enables standard models to significantly outperform traditional LiDAR and camera baselines in tasks such as semantic segmentation. By effectively overcoming perception bottlenecks in near-dark conditions, this approach achieves high-quality cross-modal translation and robust perception, offering a viable solution for reliable environmental understanding where conventional optical sensors fail.

Abstract

Conventionally, robotic perception relies heavily on cameras due to the rich semantic texture they provide. However, their performance degrades significantly in low-light or high-dynamic-range environments. Conversely, while Light Detection and Ranging (LiDAR) captures illumination-invariant geometric and intensity properties, the resulting data are typically single-channel and sparse, creating a significant modality gap when applying vision models pre-trained on RGB datasets. In this paper, we propose Cyclops, a framework that translates sparse Non-Repetitive Scanning LiDAR (NRS-LiDAR) intensity into RGB video, enabling camera-free inference for all-day perception tasks. Our approach first converts sparse LiDAR intensity projections into dense representations via a frozen pre-trained densification module, serving as a geometrically rich source condition. The dense intensity latent is then transported toward the target RGB distribution through Latent Bridge Matching (LBM) with a learned velocity field in a few ODE integration steps. To mitigate inter-frame flickering, we inject prior-frame context via temporal attention layers and further formulate the velocity field as a policy optimized by a differentiable terminal reward that encourages terminal fidelity through backpropagation along the ODE trajectory. Extensive experiments demonstrate that the synthesized RGB, including those generated under near-dark conditions, enable standard RGB-based perception models to substantially outperform both LiDAR baselines and conventional cameras on semantic segmentation, lane detection, and point cloud colorization across diverse lighting conditions.

Problem

• Robotic Perception
• LiDAR-RGB Translation
• Modality Gap
• Low-light Perception
• Camera-free Inference

Innovation

• LiDAR-to-RGB translation
• Latent Bridge Matching
• Non-Repetitive Scanning LiDAR
• Temporal attention
• Differentiable ODE optimization