Deformba: Vision State Space Model with Adaptive State Fusion

2026-05-20

AI Summary

Existing visual State Space Models (SSMs) rely on fixed image scanning orders, limiting their ability to model complex geometric structures and lacking effective mechanisms for multimodal interaction, which hinders their application in tasks such as multi-view 3D perception. This work proposes Deformba, the first SSM framework to incorporate deformable spatial sampling for context-adaptive spatial structure modeling, alongside a cross-attention-inspired mechanism enabling cross-modal state fusion. Deformba achieves significantly enhanced visual modeling capabilities while maintaining linear computational complexity. It provides a unified architecture for both 2D and 3D vision tasks, delivering state-of-the-art performance across multiple benchmarks—including image classification, object detection, segmentation, and bird’s-eye-view (BEV) perception—demonstrating its effectiveness and broad applicability.

Abstract

State Space Models (SSMs) have emerged as a powerful and efficient alternative to Transformers, demonstrating linear-time complexity and exceptional sequence modeling capabilities. However, their application to vision tasks remains challenging. First, existing vision SSMs largely depend on manually designed fixed scanning methods to flatten image patches into sequences, which imposes predefined geometric structures and increases the complexity. Second, the broader adoption of vision SSMs is hindered in domains that require query-based interactions between distinct information streams. This is a result of the inherently causal and self-referential nature of SSMs designed for 1D sequence modeling tasks. This fusion mechanism is indispensable for critical perception tasks such as multi-view 3D fusion. To address these limitations, we propose Deformba, a context adaptive method that dynamically augments the spatial structural information while maintaining the linear complexity of SSMs. Deformba also allows multi-modal fusion like cross attention. To demonstrate the effectiveness and general applicability of Deformba, we test its performance on general 2D vision tasks such as image classification, object detection, and segmentation, as well as 3D vision tasks like BEV perception. Extensive experiments show that Deformba achieves strong performance across various visual perception benchmarks.

Problem

• Vision State Space Models
• Fixed Scanning
• Multi-modal Fusion
• Query-based Interaction
• 3D Perception

Innovation

• State Space Model
• Adaptive State Fusion
• Vision Modeling
• Multi-modal Fusion
• Linear Complexity