[ArXiv 2026] Mamba-3: Bridging the Gap Between Linear Efficiency and Transformer Quality

TL;DR

Mamba-3 redefines the landscape of sub-quadratic sequence modeling by addressing the two "elephants in the room": the lack of specialized expressivity (like state-tracking) and low hardware utilization during decoding. By evolving the core State Space Model (SSM) math into a complex-valued, second-order discretized, and MIMO-capable system, Mamba-3 pushes the Pareto frontier, achieving Transformer-level quality with the efficiency of a recurrent model.

Problem & Motivation: The "Efficiency-Quality" Tradeoff

While Transformers dominate in quality, their quadratic complexity is a deployment nightmare. Sub-quadratic models like Mamba-2 and Gated DeltaNet (GDN) offer constant memory usage, but they historically struggle with:

1. State-Tracking: Tasks like parity or modular arithmetic are impossible for real-valued SSMs.
2. Hardware Underutilization: During token generation (decoding), the GPU is often waiting for memory (memory-bound) rather than calculating (compute-bound), leading to wasted FLOPS.
3. Discretization Heuristics: Prior Mamba versions relied on "Exponential-Euler" approximations that lacked strong theoretical grounding for time-varying systems.

Methodology: The Three Pillars of Mamba-3

1. Exponential-Trapezoidal Discretization

Unlike the first-order Euler method used in Mamba-1/2, Mamba-3 uses a second-order Trapezoidal rule. This provides a more accurate approximation of the underlying continuous-time ODE.

• The Insight: This recurrence can be viewed as an implicit convolution on the input, allowing the model to capture local patterns more effectively without requiring an external 1D convolution layer.

2. Complex-Valued SSMs and the "RoPE Trick"

To solve the state-tracking problem, Mamba-3 introduces complex-valued state transitions.

• The Math-to-Physics Intuition: Complex eigenvalues allow the hidden state to "rotate" in the latent space. This rotation is essential for tasks like counting or parity.
• Efficiency: The authors prove that a complex SSM is equivalent to applying data-dependent Rotary Positional Embeddings (RoPE). This allows them to use high-performance RoPE kernels to implement complex dynamics.

Figure: Comparison between Mamba-2 and Mamba-3 architectures, highlighting the new discretization and RoPE integration.

3. Multi-Input, Multi-Output (MIMO) Formulation

To fix the low arithmetic intensity of decoding, Mamba-3 moves from a Single-Input Single-Output (SISO) to a MIMO setup.

• The Benefit: By updating the state using matrix-matrix multiplications instead of outer products, the model performs 4x more FLOPs per byte of memory moved. Because decoding is memory-bound, these extra calculations come "for free" without increasing the actual wall-clock time.

Experiments & Results: Pushing the Pareto Frontier

Mamba-3 was tested across a variety of scales, from 180M to 1.5B parameters.

• Language Modeling SOTA: At 1.5B, Mamba-3 MIMO achieves a significant +1.8 point gain in downstream accuracy over GDN.
• The Pareto Shift: One of the most striking results is that Mamba-3 with a smaller state size (64) outperforms Mamba-2 with a larger state (128). This means you get better performance at lower latency.

Figure: The Pareto frontier showing Mamba-3 variants achieving lower perplexity at higher inference speeds.

State-Tracking Performance

As predicted by the theoretical inclusion of complex values, Mamba-3 solves the Parity task with 100% accuracy, whereas Mamba-2 fails completely (random guessing).

Analysis & Conclusion: Why It Matters

Mamba-3 is more than a simple iteration; it's a fundamental shift toward inference-first architecture design.

Key Takeaways:

• SSM Math is Back: The return of complex values (once discarded for simplicity) proves that mathematical expressivity is required for "reasoning-lite" tasks like state tracking.
• Efficiency via Compute: By increasing arithmetic intensity (MIMO), we can make models "smarter" without making them "slower" at inference time.
• Hybrid Future: In long-context retrieval, hybrid Mamba-3/Transformer models show incredible promise, matching the best retrieval capabilities of pure Transformers while maintaining sub-quadratic speed.

Limitations: MIMO models are roughly 2x slower during the training/prefill phase due to the increased compute overhead. However, for many production applications, the trade-off for significantly better decoding quality is well worth it.