ResFlow: Fine-tuning Residual Optical Flow for Event-based High Temporal Resolution Motion Estimation

2024-12-12

arXiv.org

AI Summary

Addressing the dual challenges of lacking high-temporal-resolution (HTR) ground-truth flow labels and inherent event sparsity in event-camera-based optical flow estimation, this paper proposes a residual two-stage paradigm: first estimating global linear motion, then refining HTR residual flow via a shared Refiner network. To enable domain-intrinsic self-supervised training, we innovatively model residual distributions using region-wise noise injection, facilitating effective transfer from low-temporal-resolution (LTR) supervision to HTR inference. Our method integrates residual decomposition, parameter-sharing refinement, region-aware noise augmentation, and event-driven optimization. Evaluated on both LTR and HTR benchmarks, it achieves state-of-the-art performance, significantly improving accuracy and robustness of optical flow estimation under sparse, asynchronous event inputs.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: State Estimation
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Event cameras hold significant promise for high-temporal-resolution (HTR) motion estimation. However, estimating event-based HTR optical flow faces two key challenges: the absence of HTR ground-truth data and the intrinsic sparsity of event data. Most existing approaches rely on the flow accumulation paradigms to indirectly supervise intermediate flows, often resulting in accumulation errors and optimization difficulties. To address these challenges, we propose a residual-based paradigm for estimating HTR optical flow with event data. Our approach separates HTR flow estimation into two stages: global linear motion estimation and HTR residual flow refinement. The residual paradigm effectively mitigates the impacts of event sparsity on optimization and is compatible with any LTR algorithm. Next, to address the challenge posed by the absence of HTR ground truth, we incorporate novel learning strategies. Specifically, we initially employ a shared refiner to estimate the residual flows, enabling both LTR supervision and HTR inference. Subsequently, we introduce regional noise to simulate the residual patterns of intermediate flows, facilitating the adaptation from LTR supervision to HTR inference. Additionally, we show that the noise-based strategy supports in-domain self-supervised training. Comprehensive experimental results demonstrate that our approach achieves state-of-the-art accuracy in both LTR and HTR metrics, highlighting its effectiveness and superiority.

Problem

• Estimating high-temporal-resolution optical flow from sparse event data
• Addressing absence of ground-truth for high-temporal-resolution motion estimation
• Mitigating optimization difficulties in event-based flow accumulation paradigms

Innovation

• Residual-based paradigm for HTR optical flow
• Shared refiner for LTR supervision and HTR inference
• Noise-based strategy enabling self-supervised training