mamba

# Mamba

![Mamba](assets/selection.png "Selective State Space")
> **Mamba: Linear-Time Sequence Modeling with Selective State Spaces**\
> Albert Gu*, Tri Dao*\
> Paper: https://arxiv.org/abs/2312.00752

![Mamba-2](assets/ssd_algorithm.png "State Space Dual Model")
> **Transformers are SSMs: Generalized Models and Efficient Algorithms**\
>     **Through Structured State Space Duality**\
> Tri Dao*, Albert Gu*\
> Paper: https://arxiv.org/abs/2405.21060

![Mamba-3](assets/mamba3.png "Inference-first State Space Model")
> **Mamba-3: Improved Sequence Modeling using State Space Principles**\
>     **Through Structured State Space Duality**\
> Aakash Lahoti*, Kevin Y. Li*, Berlin Chen*, Caitlin Wang*, Aviv Bick, J. Zico Kolter, Tri Dao†, Albert Gu†\
> Paper: https://arxiv.org/abs/2603.15569

## About

Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers.
It is based on the line of progress on [structured state space models](https://github.com/state-spaces/s4),
with an efficient hardware-aware design and implementation in the spirit of [FlashAttention](https://github.com/Dao-AILab/flash-attention).

## Installation

Install PyTorch first. By default, `mamba-ssm` installs the core package without compiling the
`selective_scan_cuda` extension and without downloading cached CUDA-enabled wheels.

| Install mode | Command | Behavior |
| --- | --- | --- |
| Core package | `pip install mamba-ssm --no-build-isolation` | Installs `mamba-ssm` without `selective_scan_cuda` and does not compile CUDA extensions. |
| Core package plus causal-conv1d | `pip install "mamba-ssm[causal-conv1d]" --no-build-isolation` | Installs the core package and the `causal-conv1d` extra, still without `selective_scan_cuda`. |
| Force local core package build | `MAMBA_FORCE_BUILD=TRUE pip install mamba-ssm --no-build-isolation` | Builds the default pure Python wheel locally, still without `selective_scan_cuda`. |
| CUDA selective scan opt-in | `MAMBA_KEEP_CUDA_BUILD=TRUE pip install mamba-ssm --no-build-isolation` | Installs `selective_scan_cuda`; pip first tries a matching prebuilt CUDA/HIP wheel, then compiles locally if no wheel is available. |
| Force local CUDA selective scan build | `MAMBA_FORCE_BUILD=TRUE MAMBA_KEEP_CUDA_BUILD=TRUE pip install mamba-ssm --no-build-isolation` | Skips cached wheels and compiles `selective_scan_cuda` locally. |

`--no-build-isolation` is required for CUDA builds so that pip uses your existing CUDA-enabled
PyTorch instead of installing torch-cpu in an isolated build environment.

Source installs use the same defaults and opt-in flags:

| Install mode | Command |
| --- | --- |
| Source default, no `selective_scan_cuda` | `pip install . --no-build-isolation` |
| Source default from GitHub, no `selective_scan_cuda` | `pip install git+https://github.com/state-spaces/mamba.git --no-build-isolation` |
| Source forced loc