Real-time Motion Segmentation with Event-based Normal Flow

2026-02-24

AI Summary

This work addresses the challenge of motion segmentation from event cameras, whose sparse output complicates direct and efficient implementation. To overcome this, the authors propose using dense normal flow—learned from local event neighborhoods—as an intermediate representation, formulating motion segmentation as an energy minimization problem solved via graph cuts. They further introduce an iterative optimization scheme that alternates between normal flow clustering and motion model fitting, along with a novel initialization strategy for motion models based on normal flow, which enables accurate estimation of independently moving objects using only a small set of candidates. Evaluated on multiple public datasets, the method achieves high accuracy while running nearly 800 times faster than the current best open-source approach, substantially meeting real-time requirements.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Learning on the Edge & Model Compression
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Event-based cameras are bio-inspired sensors with pixels that independently and asynchronously respond to brightness changes at microsecond resolution, offering the potential to handle visual tasks in challenging scenarios. However, due to the sparse information content in individual events, directly processing the raw event data to solve vision tasks is highly inefficient, which severely limits the applicability of state-of-the-art methods in real-time tasks, such as motion segmentation, a fundamental task for dynamic scene understanding. Incorporating normal flow as an intermediate representation to compress motion information from event clusters within a localized region provides a more effective solution. In this work, we propose a normal flow-based motion segmentation framework for event-based vision. Leveraging the dense normal flow directly learned from event neighborhoods as input, we formulate the motion segmentation task as an energy minimization problem solved via graph cuts, and optimize it iteratively with normal flow clustering and motion model fitting. By using a normal flow-based motion model initialization and fitting method, the proposed system is able to efficiently estimate the motion models of independently moving objects with only a limited number of candidate models, which significantly reduces the computational complexity and ensures real-time performance, achieving nearly a 800x speedup in comparison to the open-source state-of-the-art method. Extensive evaluations on multiple public datasets fully demonstrate the accuracy and efficiency of our framework.

Problem

• event-based vision
• motion segmentation
• real-time processing
• sparse events
• dynamic scene understanding

Innovation

• event-based vision
• normal flow
• motion segmentation
• graph cuts
• real-time performance