Mamba-3: Improved Sequence Modeling using State Space Principles

2026-03-16

AI Summary

This work addresses the trade-off between inference efficiency and modeling capacity in existing linear sequence models, which often sacrifice state tracking performance and struggle to realize theoretical linear complexity on real hardware. From an inference-first perspective, the paper introduces three key innovations within the state space model (SSM) framework: a highly expressive recurrence derived from SSM discretization, a complex-valued state update mechanism enabling richer dynamics, and a multi-input multi-output (MIMO) architecture that incurs no decoding latency. Evaluated at 1.5B parameters, the proposed model outperforms the strongest baseline, Gated DeltaNet, by 0.6 accuracy points on average, with the MIMO variant yielding an additional 1.2-point gain (total +1.8%). Notably, it matches Mamba-2’s perplexity using only half the state size, substantially advancing the Pareto frontier of performance versus efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While the current Transformer-based models deliver strong model quality, their quadratic compute and linear memory make inference expensive. This has spurred the development of sub-quadratic models with reduced linear compute and constant memory requirements. However, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice. Guided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule that enables richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation for better model performance without increasing decode latency. Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficiency Pareto frontier.

Problem

• inference efficiency
• state tracking
• linear models
• compute complexity
• memory requirements

Innovation

• State Space Model
• Linear-time Inference
• Complex-valued State Update
• MIMO Architecture
• Efficient Sequence Modeling