SalM$2$: An Extremely Lightweight Saliency Mamba Model for Real-Time Cognitive Awareness of Driver Attention

2025-02-22

AI Summary

Existing driver attention estimation methods neglect driving-scene semantics and suffer from excessive model parameters and poor real-time performance. This paper proposes the first Mamba-driven, lightweight saliency modeling framework tailored for driving scenarios: it jointly encodes visual and driving-semantic cues and introduces a semantic-aware dynamic saliency architecture; it leverages the state-space model (Mamba) to model temporal attention dynamics and integrates a task-driven, lightweight spatiotemporal feature encoder. The resulting model contains only 0.08M parameters—0.09%–11.16% of state-of-the-art (SOTA) models—yet achieves over 98% of SOTA accuracy while maintaining real-time inference capability, significantly outperforming comparably sized models. The core contribution lies in the first application of Mamba to driver attention prediction, achieving a unified design that simultaneously ensures semantic guidance, extreme parameter efficiency, and high prediction accuracy.

Technology Category

• Computer Vision: Large Vision Models
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Driver attention recognition in driving scenarios is a popular direction in traffic scene perception technology. It aims to understand human driver attention to focus on specific targets/objects in the driving scene. However, traffic scenes contain not only a large amount of visual information but also semantic information related to driving tasks. Existing methods lack attention to the actual semantic information present in driving scenes. Additionally, the traffic scene is a complex and dynamic process that requires constant attention to objects related to the current driving task. Existing models, influenced by their foundational frameworks, tend to have large parameter counts and complex structures. Therefore, this paper proposes a real-time saliency Mamba network based on the latest Mamba framework. As shown in Figure 1, our model uses very few parameters (0.08M, only 0.09~11.16% of other models), while maintaining SOTA performance or achieving over 98% of the SOTA model's performance.

Problem

• Real-time driver attention recognition
• Minimizing model parameter complexity
• Enhancing semantic information processing

Innovation

• Lightweight saliency Mamba model
• Minimal parameter usage
• Real-time cognitive awareness